Is Peer Review Unbiased?
November 22, 2009
A lot of cantankerous finger-pointing has been going on recently, mostly over the CRU hack. You can read all about it below. (The comments thread at the Feeding Frenzy post gives my read on the allegedly incriminating e-mails. Short summary: There’s no there there.)
One topic that keeps popping up is the claim that this shows the bias in peer review, thereby somehow undermining the research.
Newsflash: Peer review is not unbiased. Never has been; never will be. As a standard, peer review ought to be unbiased, and referees ought to do what they can to expunge bias from their determination of what gets published and what doesn’t, but nobody — and I mean, nobody — is capable of distancing themselves entirely from biases. Paradoxically, that’s why we have institutions like blind peer review.
The difference here rests on an important distinction between regulative ideals and standards of practice,where the former relate to what the latter ought to be structured around. In other words, bias-free evaluation is a regulative ideal that should, everything else equal, govern the actions of all practitioners within a given context C. As a result, practitioners put into place standards that will help better meet this regulative ideal.
That the ideal is not, and cannot be, met — it’s an ideal, remember — is a presumption of the standards of practice. The standards are in place in order to steer the discussion away from less messy matters. True, practitioners should be striving to meet the ideals independently of the standards of practice. Supposing that the standards allow some infiltration of bias, the practitioner ought to take extra steps to eradicate this bias. So there’s a natural tension there. This tension exists in all scientific disciplines; and it exists even in the humanities. It exists any time there is a discursive contradiction or dialogical conflict between two or more parties.
Curiously, it is precisely this tension that gives rise to some of the banter in the e-mails. Journal X appears to have B sort of bias. Journal X conforms with the standards of practice, but by having B bias, doesn’t meet with the regulative ideal. Ergo, we ought to do what we can either to modify the standards of practice of to forcefully expose this bias. This kind of reasoning applies to all parties here, to the skeptical and the dedicated crowd.
Cruel Mistress 
The way round bias is to have promotion of more than one school of thought – like in economics. What some of these e-mails suggest is attempts to suppress papers that are contrary to the consensus.
The mistake made at the outset was to institutionalise this bias in the IPCC and other organisations.
You still face a discretion problem. We do have multiple different schools of thought, and multiple journals that take up multiple different schools of thought. Even still, every journal must exercise some judgment in their determination of whether a given article is worthy of publication. Your solution will not solve the problem of bias.
i think we must read what Dr.Judy Curry has written inhttp://www.climateaudit.org/?p=7826 She is really a scientist and shi is not a “denier”
Yeah, that’s interesting. Curry is right, of course, both about transparency and about climate tribalism. But those are different matters than the questions about whether (a) this condemns the research, (b) there is a conspiracy to manipulate data, or (c) illegal activities have taken place. This is primarily a political problem for climate science, not an indictment of climate science, and not necessarily an indictment of the scientists who have been sending the e-mails. Again, look only at the e-mails. Disregard other things you think you know.
I should maybe add that there are multiple forces that may shape a regulative ideal, some of which are internal to the science itself, and some of which are external. Curry’s argument is that there are external, political reasons to promote transparency. Those aren’t internal science-based reasons.
Assertions like this demonstrate Curry’s utter naivete:
“The quickest way for HADCRU et al. to put Climateaudit and the rest of this tribe out of business is make all climate data and metadata public and make every effort to improve the datasets based on all feedback that you receive.”
And this is also a good point. It is probably true that giving over the data wouldn’t ever satisfy the climate auditors, as they’d be hounded and the data picked over until CRU could do no research. I’m of the mind that, generally speaking, greater transparency is better politically speaking, but there is certainly a case to be made that this is an empirical matter; and on this particular question, the answer is that it wouldn’t be better politically speaking.
Having said that, look at the fine political mess that the CRU hack has generated. Hard to tell at this point if it’ll last. For the time being, however, I’ve gotta imagine that it’s a tremendous headache.
I agree. It’s important to point out that the denialists only focus on aspects of the science where there’s enough detail and scope for interpretation to allow for virtually endless argument. “Auditing” the lake/ocean sediment studies would be easy by comparison, but their noses would be rubbed unavoidably in the fact that their long-cherished targets of the surface record and dendrochronology are no longer very policy-relevant. I think I’ll email Judy and suggest that she challenge McIntyre to do that.
[Ben: Fixed]
first please take note that english is not my “native tongue”, so i’ll made a lot of mistakes.
second, i,as Dr.Curry, think that all the data must be in public domain, so if there is an error it can be detected. peer review is o.k. but reviewers are not infallible… and they are not idle people, they have a lot of things to do. maybe someone can see an error not seen by the reviewer, it is no a “sin”.
i also know that some of the “auditors” wouldn’t be satisfied, that is not the problem. the problem are the laymans. i do not know a lot of statics, but if someone not only do not show me the data, but refuse to do that… well, he lost my confidence my “trust”. and that is the point, the confidence the “trust” of the citizens with the scientists.
i do not think there are conspiracies, illegal activities or global warming is wrong. i think that there was errors (minors or enormous) lack of transparency and a lot of brillant peoples that been brillant forget that they are humans beings and made a lot of stupids things…
One real problem here, however, as has been noted before, is that some data is very expensive or very hard to come by, and so it cannot be a general rule of scientific research that all data be available to all, unless you want to make it the case that a general coordinating body — a world government, say — purchases and owns the rights to all of the research.
mr.hale: yes i know “some data is very expensive” but you must remember that “hardcrut” is not a private institutions, it is a branch of United Kingdom’s Ministry of Defense, paid by taxes of its citizens. i think that they sold some of its data to bussiness groups, so it cannot be published immediately. but a 10 years delay?
right now it is 2 A.M. in my country!!! good by!!
Well, I’m only talking about the generalizability of a rule requiring complete transparency. Even if it’s the case that, say, L laboratory in C country is funded completely by C, that doesn’t entail that there is a prima facie duty of L to share their data either with non-residents of C or all residents of C, particularly if those residents of C may distribute the data beyond borders. Quite a bit more has to be in place for this to be true.
sorry, me again!!!
mr hale: if i read in N.Y.Times that Dr.D. from Kenya says that product P cures leukemia, he must support his claim with “something”. i think that “something” is a paper published and peer reviewed somehow, somewhere, with all the necessary data. if we could not see his data and verify it… well, his claims are not valid. and i’m not saying he is cheating us, but us Dr.Curry says: “Sometimes the professional ego piece gets out of control (we saw some of that in the CRU emails), or the social responsibility piece is too ardent. We are all humans and decision making under uncertainty is a very challenging endeavor.”
again, excume my bad english
oops!!! i forgot to post this frase from Dr.Curry: “If they wanted to decrease the relevance of SteveM, they should have given him the data.”
That’s demonstrably false, Jorge. There are many proprietary datasets in science. Even some longitudinal studies maintain very tight lock on their data. The Demographic Surveillance Study in Bangladesh at the ICDDR,B has been going on since the sixties, for instance, has very robust data. They are very tight about who has access to the data, and I think they control most of what gets published using their data too. Does that call into question the integrity of their work? Perhaps; perhaps it raises questions about the reliability of the dataset. But more than anything, it should lay to rest this silly notion that somehow all data must be, as a matter of law, made available to everyone, as well as your claim that not releasing the data to a broad set of scrutinizers somehow “invalidates” the research.
Ben,
Private data is all well and good for private use but what happens when it is being used to set public policy? We have to have a process that allows outsiders that are affected by the public policies developed based on data to verify that the claims are actually supported by the data. Just ‘trust me’ is not an acceptable answer.
The DSS at ICDDR,B is being used to set public policy.
Ross McKitrick wrote an essay in 2004 that addresses this problem unavoidable bias in science.
http //www.uoguelph.ca/~rmckitri/research/McKitrick-PubSci-essay.pdf
I think we need to reform organizations like the IPCC along the lines suggested by Ross.
Right after we put all the economists in jail. Ross can make suggestions from his cell.
There are middle grounds, for example NASA satellite data belongs to the investigators for a fixed period (one year) and then becomes public.
The case of the Yamal data set is different because it has taken decades to put the set together and the people who put it together and their chosen collaborators need a longer exclusivity period to repay their effort.
[…] good discussion wrt climate science here and here, […]
The best discussion of the leaked emails here, with over 1,500 comments.
Read the article and comments, and you will be up to speed on the issue.
Those apologists for hiding data from the people who paid for it are simply showing that their personal politics overrides their personal ethics.
First, no one has to submit papers for review by anyone. That is the decision of those writing the papers. But when they do take a position putatively based on science in the public interest, they are obligated by the Scientific Method to make their raw data and methodologies available to anyone who requests them. This was routinely done using copiers and the mail, before the internet made transparency as easy as a few mouse clicks. Yet they still stonewall their raw data. Why?
Their excuses fail the smell test. What they are doing is trying to sell a pig in a poke, and the buyer – who who is expected to pay for the pig sight unseen – is not allowed to inspect it. The clear message is: “trust us.” But should we?
As we see in these voluminous emails, no one in their right mind would trust this clique. They have hijacked the climate peer review process, as Mann forthrightly admits in his email. They are getting paid enormous amounts of money by entities such as the Grantham Foundation, which has a very heavy AGW agenda, to support the AGW cause. And he who pays the piper calls the tune.
Finally, bunnyboi’s apologia for the Yamal trickster, in which only twelve (12) trees were carefully hand-selected from an area with literally millions of trees, is typical of the alarmist contingent’s devious methods. Briffa [who amazingly comes across in the exposed emails as the least unethical of their clique] used one tree [YAD061] to create his alarming hokey stick. But if that single tree were replaced at random, Briffa’s hockey stick would disappear.
If these corrupt scientists had simply followed the Scientific Method, they would not have pocketed nearly so much taxpayer loot. But their ethics would be unchallenged. We can now see that their decision was to take Mammon’s path.
Smokey, no offense but you don’t know what you are talking about.
You talk as if the lack of transparency is endemic to the field, yet your hero, McI has been complaining about 2 data sets: the Yamal dataset used by Briffa and CRUTEM V.3. Both of these were covered by intellectual property agreements. There are hundreds of sites on the internet housing extensive caches of raw climate data. I suggest you visit Open Mind and click on the “Climate Data Links” link at the top of the page.
Of course, I am quite sure that even if you were in possession of all the data you could want, you would still have no idea of what to do with it.
About Yamal. It was 17 live tree samples and several hundred sub-fossil samples. Dropping YAD061 from the set did not affect the recon, Tom P showed that over at CA, but I’m sure that bender’s insane shouting drowned out any perspective which might have been gained from that observation.
The rest of your statements are just as silly, but I don’t have time to deal with them. Read the two rather long threads over at RC. Pay attention to Gavin’s replies.
Smokey: No offense, but you misrepresent the practical presuppositions of the scientific method.
I misrepresent nothing. You can see the corruption of the Scientific Method throughout Karl Popper’s rules for same:
You can not test a hypothesis [such as AGW] unless all raw and adjusted data and methods are provided transparently to other scientists. But as we see, those requests are routinely stonewalled. Thus, AGW is simply untestable propaganda, not science.
As Richard Feynman explained regarding the Scientific Method, even the proponents and inventors of a hypothesis have a duty to try to falsify it in every way they can. And they have a duty to share all of the data and methodologies that went into formulating their hypothesis with anyone who asks.
But the Jones/Mann/Schmidt clique still stonewalls requests for their data. You can read their scheming and maneuvering to deny data requests in their exposed emails. Thus, they violate the Scientific Method.
Finally, Phil Jones alone has received more than £13.7 million [$22 million] in grants to ‘study global warming’, from such entities as the Grantham Foundation, which has a heavy AGW agenda.
Jones is on the public payroll. He can not serve two masters. Tell us, who is being cheated? Grantham? Or the taxpayers?
Hmm, I don’t see transparency in that list.
Are we back on Popper? What’s up with you and Popper? Falsifiability is a deeply flawed theory about the nature of a scientific (versus a non-scientific) theory in principle. You don’t seem to understand that.
As it turns out, you can, in principle, test the hypothesis of AGW, so it’s a falsifiable and, ergo, a scientific theory, even using Popper’s criteria.
Rattus:
Transparency means opening your work to others to falsify it, if they can. But the bovine fecal purveyance specialists shoveling their CO2=CAGW globaloney consistently refuse to allow others to see how they arrived at their outrageous conclusions. Can’t blame ’em. As Popper points out, “Some genuinely testable theories, when found to be false, are still upheld by their admirers.” That’s another way of saying CO2=CAGW true believers suffer from cognitive dissonance.
And Ben Hale, it is you who has the problem with Karl Popper, not me. Falsifiability is what got Western civilization to the point where you go to the hospital to get a disease treated, instead of going to your local witch doctor.
To falsify something you need to test it; testability is absolutely essential to falsifying any real world hypothesis like AGW. Yet you claim, without providing any evidence, that falsifiability is a ‘deeply flawed theory.’ Only in the minds of non-scientists. That’s why no one in their right mind would ask a philosopher to build a bridge – because apparently they actually believe that the necessity of testing is a ‘deeply flawed’ concept.
If a hypothesis can not be falsified, because its proponents refuse to share the data they used to construct the hypothesis in order to allow testability of its veracity, then the hypothesis becomes nothing but rank speculation; a baseless conjecture.
You can hypothesize all you want that there’s a monster living under your bed. But if you won’t permit falsification of your hypothesis, by allowing those you’re trying to convince to turn on the light and look for themselves, your hypothesis fails.
CO2=CAGW fails for the same reason. Its purveyors are saying, “Trust us.”
As if.
It is an astonishing but little known fact that we had no doctors until Karl Popper came along in the late 1950s to dispel the mysticism in hospitals (theretofore: voodoo shrines) with his well-regarded and doctrinal theory of falsifiability. Far be it from me to rob him of this crown. No matter. Why bother criticizing something I hardly understand?
Just out of curiosity, what are your thoughts on W.V.O. Quine’s work, particularly in “Two Dogmas”? I’ve always found the implications for knowledge somewhat compelling, but perhaps you have stronger views. Or maybe, instead, you’d just like to comment on his collaborative thesis with Pierre Duhem?
Ben Hale me boy, I can’t even honestly say: “nice try.”
First, perhaps you would like to explain exactly how ‘falsifiability is a deeply flawed theory’. No need to link to others, if you can think for yourself. Otherwise, go ahead and cut ‘n’ paste.
And of course you do know that you made a logically flawed rejoinder, pretending that I said Popper is the reason for not resorting to witch doctors, when I was referring to the principle you find so flawed. You could even look it up ^.
Finally, since I wrote my last post, I ran across this article. Interesting, in that the article and comments restate almost exactly what I had written. Why not make a comment there yourself, and see how long it takes before your own personal deconstruction starts? Seeing as how your critical thinking is so fuzzy and all.
I predict several slap-downs within ten minutes. Go ahead, give it a try.
I’m going out, this is too easy. You can have the last rant.
That paper is on verification and reproducibility, not falsifiability. Again, it should be clear that you misunderstand the concepts miserably, which makes it hard to explain to you what’s wrong with them.
We’ve already had this conversation about Popper, several times. I’ve already explained to you that falsifiability refers to the status of a theory, as to whether it is scientific or non-scientific. A theory, if falsifiable in principle, is a scientific theory. If it is not in principle falsifiable, then it is a non-scientific theory. Freud’s psycho-analytic theory is in principle not falsifiable (because we can’t get into anybody’s head), so therefore it’s a non-scientific theory. Einstein’s theory of relativity in principle is falsifiable, though it hasn’t been falsified yet (to my knowledge). Hilary Putnam’s theory that if you put a sack of flour on your head and rap the table 99 times then a demon will appear is also falsifiable, and so therefore, is also a scientific theory (on this view).
The problem with asserting that falsifiability applies in practice is that you have an infinite range of observation errors that interfere with an assessment of whether a given observation in fact falsifies your claim. There are also an infinite number of alternative explanations that will allow a scientist to explain away the falsification. As a general rule on this view, you can never falsify even a falsifiable theory (cf. Pierre Duhem). (If you want the practical line, check out Richard Cole’s 1968 article in Mind)
At any rate, no matter. You fail your philosophy paper for not distinguishing between verifiability and falsifiability.
First off, Ben, you are wrong when you say that Karl Popper’s paper paper “is on verification and reproducibility, not falsifiability.”
The title of the paper is: Science as Falsification. Good thing for that tenure racket, eh?
And thank you for claiming that the
theorythehypothesisthe failed conjecture of CO2=CAGW [catastrophic anthropogenic global warming] is not falsifiable, and therefore it is unscientific [I disagree, BTW.] CO2=CAGW has been repeatedly falsified; and the planet herself laughs at the hubris of those Elmer Gantry-style charlatans shilling CAGW, by cooling for the past decade – as the entirely beneficial and harmless trace gas CO2 continues to rise. Good. The planet needs more CO2. Plants thrive on it, and it’s otherwise completely harmless. Take it from a retired scientist.According to Harvard: “Karl Popper is generally regarded as one of the greatest philosophers of science of the 20th century.” Yet you denigrate him. So let’s see what the ‘consensus’ thinks of your boys, per google:
Quine: 181,000
Duhem: 87,600
Popper: 290,000
Karl Popper gets more followers than the other two combined.
I know that’s not scientific. But it’s certainly more accurate than the outright fraud being perpetrated by the rent-seeking CRU scoundrels cashing in on their fake global warming scare. They refuse to show their data. Why? It’s not like the weather is a national defense secret. They hide their data because if they disclosed it, their conjecture would go up in flames.
And rattus me boy, it looks like your guy Gavin [and maybe others] may now be facing legal action for [allegedly] using their taxpayer-paid time to run their realclimate blog during working hours. For years. Those time/date stamps are a bitch, aren’t they? Today I hear RC stopped taking comments, and started advertising for a moderator. Does this mean Gavin has to actually work, instead of play blogger?
Hey, now maybe some philosopher can philosophize to us hoi polloi about how alarmist climate trickery is A-OK.
I wasn’t talking about Popper’s paper. I was talking about the article you linked to in your post above titled “Verification and Validation Through Replication.” Sorry if that wasn’t clear.
Also, “Harvard” who? I suspect those are citations, not followers. There’s nothing about those numbers that says “followers” to me. That a particular article gets some large number of citations may speak more to its influence than to its rightness. Newton gets a lot of citations too, and he was very smart, but he was also wrong. I never denied that Popper was influential and smart. I denied that his theory of falsifiability is very wrong. And I also asserted that you don’t understand it.
Finally, with regard to the legal action, it’s a suit. Anybody can file suit against anybody else for all manner of things. We’ll see if they have standing. We’ll see how far they get. I suspect they won’t get far.
As in evolutionary biology and economics, superior outcomes result when we get a multiplicity of behaviours, actions, appearances etc. Science is the same. The “fittest” hypotheses will survive.
But don’t fall for the old trap. Darwinian “fittest” means “most competitively advantageous for the prevailing environment”. If, in an field of science the prevailing environment is one that allows only one slant and also constrains alterntive outlets, peer review could fail.
I think you are right in the end. Such a scenario (i.e. one where access to publication is strictly controlled) will contain the seeds of its own destruction and the better ideas will out in the end.
But “the end” might be a long time coming. And that could be very costly.