Curly Wants His Money BackOctober 29, 2009
Steve McIntyre gives the impression in his recent post that I somehow think peer review is a closed system. He doesn’t say as much, but I gather that he assumes I don’t attribute much weight to the role of blogs in the peer review system. Here’s what he has to say, specifically regarding my comments:
Roger Pielke Jr had opined hopefully that this concession would finally settle at least one small point in paleoclimate. Pielke said that “it looks like this dispute will in fact be resolved unequivocally through the peer-reviewed literature, which for all of its faults, is the media of record for scientific claims and counterclaims”. Pielke was obviously aware of the role of blogs (both Climate Audit and in Finland) in this dispute and was here focusing more on the fact that Kaufman was admitting the upside down use in a formal venue, rather than the role of the journals in extracting the admission from Kaufman. This point was misconstrued by Ben Hale here who interpreted Roger’s post as evidence that the Kaufman error had been detected and resolved by journal peer review and due diligence, when that’s not what happened at all. (I posted a comment at Hale’s to this effect.)
I added the colorful language. More after the jump…
Just to set the record straight, it’s not that I think that the appearance of the correction happened strictly through closed, peer review channels and through due diligence, but that the system isn’t broken. Scientists correct themselves, and do so in the peer-reviewed literature.
This is important, because when it comes to establishing some sort of standard that laypeople — like me, and presumably all but a select few — can trust, the peer-reviewed literature is that standard. When a correction appears in the peer-reviewed literature, as far as I’m concerned, it is something that we can reliably trust, and it is an open question as to how it got there. For all I care, it could’ve appeared to the author, apparition-like, during a particularly hideous methamphetamine-induced nightmare. If somehow the correction makes its way through and into the literature, this is something that we can put our money on.
In this case, that’s what happened. The correction will appear. The system isn’t broken. It moves — maybe thanks, in part, to Steve McIntyre’s relentless criticism.
The important point is not what appears in the peer-reviewed literature, nor the means by which it appeared in the peer-reviewed literature, but that the peer-reviewed literature is open to amendment and is not closed. That something new appears at all is evidence that there are external factors moving the literature along. Blogs obviously play a role in this, as do good editors, fair judgment, due diligence, and other external, irrelevant factors: school-boy politics, sexiness, money, nepotism, and so on.
Now then, just as we can impugn the truth of a claim in an article, so too can we impugn the procedures by which such a claim was adopted. In this case, maybe we want to suggest that the blogoverse shouldn’t have played such an influential role in Kaufman’s decision to offer a correction; or maybe we want to take the alternate position, to say that it is complete bullshit that external political pressures were brought to bear on Kaufman, thus forcing him to offer his correction. Ultimately, these grievances have to be adjudicated and assessed through channels — some formal, some less so — similar to those that establish what makes it into the peer-reviewed literature.
Before readers go screaming about the implausibility of this view, consider that it is entirely possible that Big Joe Curly could grease the palms of editors to get his article in the peer-reviewed literature. Let’s assume he does. If Big Joe Curly slips several Benjamins in the pocket of the editor, and then suddenly his article appears, it is not Big Joe Curly, but we, who are saddled with the responsibility of demonstrating that Big Joe Curly has engaged in such duplicity. Maybe that seems unfair, but all is not closed to us.
We can impugn the credibility of his article in a variety of ways. We can do so on our blogs, we can do so by inaugurating a public smear campaign, or we can do so by affixing creepy video technologies to the trees outside his house. Some of these approaches are better than others, obviously.
An important second observation, therefore, is not only that we have ways of challenging Curly’s article, but also that when challenging it, we can say anything at all. We can do anything. We can say that Big Joe Curly likes to torture snail fetuses, for instance, and we can post sicko-pictures of him wearing a hoodie and slurping butter off the backs of baby escargot. The world is our mollusca. So even here we need a way to ensure that our criticisms are not entirely misguided, are not fallacious. We need a way to ensure that our criticisms hit up Big Joe Curly and not Soft Larry Cuddly.
Now, as I’ve said numerous times in the past, in the case of climate science, this is the system of peer review. When something appears in a respectable, reputable location, that’s something you can put your money on. Fortunately for all of us, there are multiple respectable locations, assorted around a range of scientific concerns, and they are not limited to the top journals in climate science. Blogs, unfortunately, aren’t those locations. Blogs are you and me tittering to one another over brandy and chocolate about Peter Pan’s panty lines.
So to put this differently: in the end, it does, of course, matter how X is uttered, and it also matters whether it is truth-apt, but when it comes to ruling on the strength of the claim, we still have to defer to what’s in the literature, regardless of how it got there. The important thing in this case is that criticism of what’s in the literature goes through channels that make it to authors and editors to alter what’s in the literature. If we later want to impugn how it got there, we can and obviously should do that. But just as we have to rely on the peer-reviewed literature for our scientifically established claims, we also have to rely on other procedures to establish the strength of our claims about the procedural legitimacy of their appearance. Just as I am required to withhold judgment on an accused’s innocence or guilt until they are proved guilty “in a court of law,” so too with the claims in the peer-reviewed literature.