Crowd wisdom vs. Peer review (or the State of our ‘Knowledge’)

Rethinking some taken-for-granted assumptions on what counts as ‘real’ knowledge

At a recent work event on scholarly activity – during which we discussed what exactly counts as ‘scholarly activity’ (specifically in EAP, but we touched on other disciplines, too) – someone briefly mentioned that blogs could not be counted as ‘academic’ because they are not subject to rigorous peer-review; somebody else briefly raised a caution to this and said that the open public scrutiny that blogging is subjected to can provide similar guarantees to peer reviewing, which means bloggers and tweeters (those who want to be considered credible by their academic communities) do take care of their evidence and tone.

I admit that I am here reporting the spirit of what was broached the other night, rather than the letter. Things weren’t actually articulated as such because this wasn’t the focus of the event – however, my critical ears pricked up and have been burning ever since, so I am using my own (non-academic, non-peer reviewed) blog to indulge in some poetic licence and to imagine what would have been said if we could have discussed the (de)merits of peer-review (if those present are reading this, please speak up and peer-review me!)

Firstly, I would have liked us to have flagged up instances where ‘proper’ peer-review has been outrageously and clamorously discredited. The most famous case is the Sokal Affair in which a bunch of self-assured and well-known postmodernists fell for the most notorious academic imposture of all time. So smug were they to have had a ‘proper’ physical scientist arguing that knowledge is ‘constructed’ and therefore ‘relative’, ‘subjective’ and ‘masculine’, that their own critical reading radars were switched right off and Sokal was published, no questions asked (he himself had to reveal the hoax a few weeks later; had he not done so, Sokal would still be part of collective postmodern epistemologies).

Secondly, somebody else in our room would have raised a more recent hoax, the one in which 120 non-existent and entirely computer-generated scientists dazzled the peer-review panel at Springer, and made their grand entrance onto the ‘academic’ stage.

Thirdly, my other imaginary conversants would have pointed to evidence suggesting that the cumulative effect of crowd wisdom actually weeds out factual errors and bad arguments (all of which fall under the remit of a peer reviewer). Pat Thomson also deals with this topic here and reveals her penchant for wikipedia (note that my own references for Sokal come from wikipedia).

Finally, we would have openly and generously reflected on what all of this means in relation to the reliability of knowledge and of information (not the same thing!), of whose knowledge it is and how it got there, of critical engagement with good ideas, rather than an unquestioning and deferential reliance on the authority of editorial panels. We would then have pondered on where we draw the line between what counts and does not count as academic, and I would not have to spend years doing a PhD on it!

But this conversation never happened, it is all in my imagination.