Archive for November 22nd, 2009


Is Peer Review Unbiased?

November 22, 2009

A lot of┬ácantankerous finger-pointing has been going on recently, mostly over the CRU hack. You can read all about it below. (The comments thread at the Feeding Frenzy post gives my read on the allegedly incriminating e-mails. Short summary: There’s no there there.)

One topic that keeps popping up is the claim that this shows the bias in peer review, thereby somehow undermining the research.

Newsflash: Peer review is not unbiased. Never has been; never will be. As a standard, peer review ought to be unbiased, and referees ought to do what they can to expunge bias from their determination of what gets published and what doesn’t, but nobody — and I mean, nobody — is capable of distancing themselves entirely from biases. Paradoxically, that’s why we have institutions like blind peer review.

The difference here rests on an important distinction between regulative ideals and standards of practice,where the former relate to what the latter ought to be structured around. In other words, bias-free evaluation is a regulative ideal that should, everything else equal, govern the actions of all practitioners within a given context C. As a result, practitioners put into place standards that will help better meet this regulative ideal.

That the ideal is not, and cannot be, met — it’s an ideal, remember — is a presumption of the standards of practice. The standards are in place in order to steer the discussion away from less messy matters. True, practitioners should be striving to meet the ideals independently of the standards of practice. Supposing that the standards allow some infiltration of bias, the practitioner ought to take extra steps to eradicate this bias. So there’s a natural tension there. This tension exists in all scientific disciplines; and it exists even in the humanities. It exists any time there is a discursive contradiction or dialogical conflict between two or more parties.

Curiously, it is precisely this tension that gives rise to some of the banter in the e-mails. Journal X appears to have B sort of bias. Journal X conforms with the standards of practice, but by having B bias, doesn’t meet with the regulative ideal. Ergo, we ought to do what we can either to modify the standards of practice of to forcefully expose this bias.┬áThis kind of reasoning applies to all parties here, to the skeptical and the dedicated crowd.