Curly Wants His Money Back
October 29, 2009
Steve McIntyre gives the impression in his recent post that I somehow think peer review is a closed system. He doesn’t say as much, but I gather that he assumes I don’t attribute much weight to the role of blogs in the peer review system. Here’s what he has to say, specifically regarding my comments:
Roger Pielke Jr had opined hopefully that this concession would finally settle at least one small point in paleoclimate. Pielke said that “it looks like this dispute will in fact be resolved unequivocally through the peer-reviewed literature, which for all of its faults, is the media of record for scientific claims and counterclaims”. Pielke was obviously aware of the role of blogs (both Climate Audit and in Finland) in this dispute and was here focusing more on the fact that Kaufman was admitting the upside down use in a formal venue, rather than the role of the journals in extracting the admission from Kaufman. This point was misconstrued by Ben Hale here who interpreted Roger’s post as evidence that the Kaufman error had been detected and resolved by journal peer review and due diligence, when that’s not what happened at all. (I posted a comment at Hale’s to this effect.)
I added the colorful language. More after the jump…
Just to set the record straight, it’s not that I think that the appearance of the correction happened strictly through closed, peer review channels and through due diligence, but that the system isn’t broken. Scientists correct themselves, and do so in the peer-reviewed literature.
This is important, because when it comes to establishing some sort of standard that laypeople — like me, and presumably all but a select few — can trust, the peer-reviewed literature is that standard. When a correction appears in the peer-reviewed literature, as far as I’m concerned, it is something that we can reliably trust, and it is an open question as to how it got there. For all I care, it could’ve appeared to the author, apparition-like, during a particularly hideous methamphetamine-induced nightmare. If somehow the correction makes its way through and into the literature, this is something that we can put our money on.
In this case, that’s what happened. The correction will appear. The system isn’t broken. It moves — maybe thanks, in part, to Steve McIntyre’s relentless criticism.
The important point is not what appears in the peer-reviewed literature, nor the means by which it appeared in the peer-reviewed literature, but that the peer-reviewed literature is open to amendment and is not closed. That something new appears at all is evidence that there are external factors moving the literature along. Blogs obviously play a role in this, as do good editors, fair judgment, due diligence, and other external, irrelevant factors: school-boy politics, sexiness, money, nepotism, and so on.
Now then, just as we can impugn the truth of a claim in an article, so too can we impugn the procedures by which such a claim was adopted. In this case, maybe we want to suggest that the blogoverse shouldn’t have played such an influential role in Kaufman’s decision to offer a correction; or maybe we want to take the alternate position, to say that it is complete bullshit that external political pressures were brought to bear on Kaufman, thus forcing him to offer his correction. Ultimately, these grievances have to be adjudicated and assessed through channels — some formal, some less so — similar to those that establish what makes it into the peer-reviewed literature.
Before readers go screaming about the implausibility of this view, consider that it is entirely possible that Big Joe Curly could grease the palms of editors to get his article in the peer-reviewed literature. Let’s assume he does. If Big Joe Curly slips several Benjamins in the pocket of the editor, and then suddenly his article appears, it is not Big Joe Curly, but we, who are saddled with the responsibility of demonstrating that Big Joe Curly has engaged in such duplicity. Maybe that seems unfair, but all is not closed to us.
We can impugn the credibility of his article in a variety of ways. We can do so on our blogs, we can do so by inaugurating a public smear campaign, or we can do so by affixing creepy video technologies to the trees outside his house. Some of these approaches are better than others, obviously.
An important second observation, therefore, is not only that we have ways of challenging Curly’s article, but also that when challenging it, we can say anything at all. We can do anything. We can say that Big Joe Curly likes to torture snail fetuses, for instance, and we can post sicko-pictures of him wearing a hoodie and slurping butter off the backs of baby escargot. The world is our mollusca. So even here we need a way to ensure that our criticisms are not entirely misguided, are not fallacious. We need a way to ensure that our criticisms hit up Big Joe Curly and not Soft Larry Cuddly.
Now, as I’ve said numerous times in the past, in the case of climate science, this is the system of peer review. When something appears in a respectable, reputable location, that’s something you can put your money on. Fortunately for all of us, there are multiple respectable locations, assorted around a range of scientific concerns, and they are not limited to the top journals in climate science. Blogs, unfortunately, aren’t those locations. Blogs are you and me tittering to one another over brandy and chocolate about Peter Pan’s panty lines.
So to put this differently: in the end, it does, of course, matter how X is uttered, and it also matters whether it is truth-apt, but when it comes to ruling on the strength of the claim, we still have to defer to what’s in the literature, regardless of how it got there. The important thing in this case is that criticism of what’s in the literature goes through channels that make it to authors and editors to alter what’s in the literature. If we later want to impugn how it got there, we can and obviously should do that. But just as we have to rely on the peer-reviewed literature for our scientifically established claims, we also have to rely on other procedures to establish the strength of our claims about the procedural legitimacy of their appearance. Just as I am required to withhold judgment on an accused’s innocence or guilt until they are proved guilty “in a court of law,” so too with the claims in the peer-reviewed literature.
Cruel Mistress 
Ben, as far as the utility of peer review in the paleoclimate literature, a test is underway, courtesy of Steve McIntyre and your blog.
Mann et al (2008) Figure S8a.
The link is to a comment I put up on your post “Peer Review. Game On.” McIntyre and others have now proven that the Lake Korttajarvi varve proxies were used blindly and wrongly in Mann et al. Among other things, this finding strongly suggests that there is something very wrong with Figure S8a, which was intended to address the quality of the data Mann used, and of the CPS method used to analyze them.
The message of Figure S8a is, “our findings are robust.”
* Did the reviewers understand what the figure purports to show? Did they look into it enough to give it a passing grade?
* Why did the reviewers approve this figure in its current form?
* What does the green line in Figure S8a represent?
* Given the misapplication of the four Lake Korttajarvi varve proxies, how can the near-identity of the green and black lines be explained?
* What’s next, in terms of how this potentially-severe methodological problem is handled by PNAS? Will an explanation or correction be freely offered? Will the editors demand one? Or will the status quo simply persist?
A Delphic remark by the authors that a criticism is “bizarre” should not stand as the last published word.
The question of “proof” is obviously what’s at issue here. The standard I’m applying is one that restricts proof to appearance in the peer-reviewed lit, rather than opens it to an external body of non-peer reviewers. This is more-or-less pragmatic: Unless you’re qualified to judge — which you may be, I have no way of knowing — you should probably also apply that standard. If you are qualified to judge, however, you should maybe help get the criticism in the peer-reviewed literature. If not, I recommend withholding judgment until it appears.
McIntyre and bender are not peers of Mann. We’re amateurs. The process that is solving this problem is amateur-review, not peer-review.
That’s the problem. (No offense, Bender.) I’ll assume that McIntyre’s more of a peer than you are, but again, I don’t know.
The problem is that you can lodge maybe 1, 10, 500 criticisms of the paper, some of which may be right on, some of which may be way off. Since you’re uniquely caught up in your own little circle, it’s hard for anyone to judge the legitimacy of your claims.
How to handle that? Let qualified peers evaluate their strength. If and when they make it through to appear on the pages of a peer-reviewed publication, that will count as strength-giving evidence that your criticism has merit.
It’s not that your criticism won’t have truth-merit independent of the peer review, it’s that these are incredibly complex questions and only peer-review can provide a standard of validation that will satisfy the question of whether your criticism is to be believed.
No offense taken. But you are wrong. The elites are extremely well-equipped, in theory, to judge the merits of the amateur’s argument. The thing they’re lacking – as you yourself surely know – is time.
Or people who disagree with the result can do new science. That’s how things work most of the time, and ought to work if there’s any value in the concept of scientific progress, although I realize it’s no fun at all for people like you.
Yes, we know you loathe the detection and correction of errors.
In benderworld, people would spend most of their time endlessly attacking and defending picayune aspects of the science. It would be a world with less scientific progress, albeit a more fun one for bender personally.
Steve Bloom,
You seem pretty well-informed on paleoclimate issues, and well-disposed toward the work of Mann’s group.
If you’re so inclined, I’d be interested to hear your take on five statements I made as comment #1 at W. Connolley’s Stoat post “Tiljander.” They concern the Tiljander varve proxies and their use in Mann et al (2008).
Thanks
Peers believe that audit-style review is beneath them. Amateurs are more willing to do hard work, and have not the insight or motivation to pick the fruit of the elitist academic’s tree. Amateurs have a role to play, despite what the elites say. Welcome to a flatter, better connected humankind. A Farewell to Kings.
No offense taken. But you are wrong. The elites are extremely well-equipped, in theory, to judge the merits of the amateur’s argument. The thing they’re lacking – as you yourself surely know – is time.
Ben Hale wrote (11:27am) —
> Unless you’re qualified to judge — which you may be, I have no way of knowing…
I am qualified to ask the questions about Fig. S8a that I asked in the linked comment. I am qualified to a present a plain-English narrative on this subject, in a cautious way, being open to corrections and insights.
You have no way of knowing? To the contrary, you are qualified to form an opinion as to whether I have done what I set out to do, and whether or not Mann et al should address the issues I raise.
Are scientists too specialized/exalted/delicate to be expected to explain and, as needed, defend their work outside of their professional comfort zone?
If so, then they start to resemble priests. That’s not good for the search for Truth. Or for the process of crafting science-based public policy.
I can, of course, form my own opinions, but I generally do this by relying on a variety of credentialing schemes. As far as I know, your name is AMac, you’re reasonably well informed about some of these issues, and you’re writing comments on my blog.
Ben Hale (12:19pm) —
I rely on credentials, too, though it seems that they matter more to you. A chacun son gout.
My comment on Fig. S8a is 18 paragraphs long, with most covering one point. These points don’t rely on behind-firewall material or on complex maths or computer code. I think many readers of this blog could read through para by para, evaluating each one.
* Assertion of fact, application of logic, or opinion?
* If assertion of fact: True? False? Don’t Know?
* If application of logic: Correct? Wrong? Don’t Know?
* If assertion of fact or application of logic: Important? Trivial?
This small exercise doesn’t require that the reader be credentialed, and doesn’t depend on the credentials of the writer.
It would be great if well-informed Mann supporters would undertake this exercise and share their criticisms. One way that science advances is by engaging in such a marketplace of ideas.
More, less, not sure. They matter, is all. I’m happy to discount the dribblings of a well-credentialed nincompoop’s bad argument, so I don’t want to give the impression that I rely on credentials too heavily.
The same goes for what you’re suggesting. You maybe don’t have the credentials, but maybe you’ve got the knock-down criticism. That’s cool by me. I can accept that that may be true.
I’m intentionally not getting involved in the substantive back-and-forth because (a) I’m not qualified to judge, (b) I’m not terribly interested in spending my weeks learning the background material required to make myself qualified to judge, and (c) I trust that there are mechanisms for criticisms like yours to rise to the surface.
Those same mechanisms, Ben, are what allow the separation of wheat from chaff on the amateur blogs.
Yeah, except that, as one who is not qualified to judge, I can’t distinguish between wheat and chaff. I rely on the peer reviewed journals for that.
That’s where Eli Rabett actually makes an excellent point, below. Closed peer review is slowly evolving toward internet-based open peer + non-peer review (such as at Climate of the Past Discussions). This is going to further blur the distinction that you are trying to make. I rely on peer review too. But that is changing.
Indeed, procedural openness and transparency is a great thing for peer review. Problem is, the openness comes with its own share of problems. The filters play a deliberative role too.
Let me first state in general terms I agree with you regarding the appearance of peer-reviewed scientific papers in literature. The problem comes in when a branch of science (in this case climatology) gets politicized with massive amounts of money to promote one point of view, and the majority of journals taking the same point of view. When a journal does not enforce its own documentation/archive rules for a paper it agrees with, the process becomes effectively broken by preventing verification/duplication by independent parties. Had Mann’s or Briffa’s data been released at the time of publication, an analysis would have rapidly followed and flaws would have been exposed, hopefully with corrections to follow in the peer-reviewed literature. Instead, the data was hidden for about a decade and was the basis for other peer-reviewed papers, which showed a preponderance of evidence, all based on bad science, or at least analysis. How does an independent critical analysis get into peer-reviewed literature when the journal doesn’t enforce its own standards, and has a political position sympathetic with the original (faulty) paper?
I may not have been entirely clear about this above with my Curly example, but my view is that there are other peer-reviewed outlets to take up concerns like those that you’re noting. People like Roger do exactly that by looking at the politics of science and science policy, and then publishing their findings or their arguments in reputable journals.
Again, the issue isn’t solely with the strength of the claims, so much as with the extent to which the claims have been run through and met with standards acceptable to a body of qualified reviewers.
As to your concern that starts “When a journal does not enforce its own documentation/archive rules…” I think we’ve addressed that before. I’m not persuaded that the journal didn’t enforce its own standards. But more importantly, if we’ve addressed it before, or if it’s been discarded by others who may be familiar with the standards of that journal, then that’s a filtering mechanism that sorts out good criticism from bad criticism.
I disagree with bender’s assessment that he and SM are amateurs vs. Mann et. al. being professional peers. In fact, the Team members are not strong in statistics, which is the tool that is being used in their “climate science”. If they are mis-using this tool in their research – a point well established by Wegman – it is their professional duty to seek out the right expertise. The Team steadfastly refuses to do this since they know what would happen. The fact is that legions of persons in other fields are more qualified to critique this sort of analysis.
what Tom C said. Climate Science desperately needs experienced and independent statistical reviewers.
The fact is that this is a science dominated by people who got into it on a mission to save the earth from global warming. They tend toward passion rather than objective analysis. They are not good basic scientists but rather synthesizers of other disciplines.
A chaperone is needed to rationalize this field.
Yes! To paraphrase a former head of the Luftwaffe, whenever we see some science we don’t like we need to reach for our rationales.
Lets take another example – Jolliffe’s criticisms of the PCA analysis used in Mann’s earlier papers. Joliffe is a recognised authority in using PCA, and he broadly said that Mann did not describe his analysis correctly and did not use PCA in a form that was recognised.
These comments took 10 years to come out and cannot be found anywhere in the peer reviewed literature. They are buried in an Open Thread on the Open Mind blog.
Concerning the narrow issue of the use of the Lake Korttajarvi varve proxies by Mann et al (2008), William Connolley has responded to my five claims on the matter (Comment #1 of Stoat post Tiljander).
My reflections (and tentative conclusions) are way downthread, here in summary form and in the following comment in detail. Stoat slightly mangled my [blockquote] html; the longer argument is more readable as the mirrored version at ClimateAudit.org.
This rather small and restricted issue is quite relevant to the topic of this Cruel Mistress post. Employment of these lakebed sediments in Mann et al’s paleoclimate reconstruction is a restricted and technical question that can be definitively addressed by a layperson who is willing to scan only one paper beyond Mann et al and its related materials.
To date, the manner in which PNAS has handled this modest challenge involving the publication of mishandled data in a peer-reviewed article has been woefully deficient. The authors have apparently chosen to stonewall, and the editors seem okay with that.
In my opinion.
Darn, I messed up a link in the preceding 1:36 pm comment. It should have read, “the longer argument is more readable as the mirrored version at ClimateAudit.org.”