h1

Cherry Ping-Pong

October 13, 2009

There’s a lot of tit-for-tat going on in the blogosphere over the alleged cherry-picking of data (also here and original, criticized post here).  I’ll remain agnostic on the empirical question, as all usual caveats apply.  But what exactly is cherry picking?  Is it ever okay to select data?

Let’s be clear on one thing: deliberative and judicious selection of data is not equivalent with cherry picking.  A cherry picking charge is considerably more severe.  Scientists, like technicians, select out data using criteria that “seem to fit” their view of what is happening.  There’s nothing suspicious about this.  It’s what all specialists do: scientists, academics, politicians, lawyers, policy makers, businessmen, and so on.  We select out relevant data and discard the irrelevant stuff by using our professional judgment.

Selection becomes a problem, however, when we discard and/or discriminate against relevant data that either does not support our position or that contradicts our position.  Cherry picking is an informal fallacy of relevance.  (There are other, related, fallacies of induction; but cherry picking, as I understand it, is a fallacy of relevance.)

A clear implication of this fallacy is that the charge of cherry picking cuts both ways. The charge applies to anyone who chooses to select data that fallaciously demonstrate her position. The nature of the dispute over whether someone has cherry picked, in other words, must be over the relevance of the data and not the mere existence of contravening data.

Relevance is key.

In Chip Knappenberger’s Guide to Cherry Picking, Chip artfully tries to show that depending on your stopping and starting points, you can end up with one or the other conclusions about the warming or cooling of the earth.

…the answers [about whether the climate is warming or cooling] depend on several things, among them the dataset you want to use and the time period over which you examine—i.e., which cherries you wish to pick.

But this is a distortion of what we mean by “answer”…

What the answer actually depends on is the climate, and whether the climate is in fact warming or cooling.  The job of the scientist is to try to get at this answer by using strong scientific and inductive reasoning.  Depending on how she approaches the problem, depending on the strength of her reasoning, she may give you a slightly different answer.  If she uses poor selection procedures in choosing her data, for instance, she’s liable to get the wrong answer.  If she actively disregards contravening data, she is then manipulating the answer.

Chip Knappenberger, who originally wrote the cherry-picker’s guide, doesn’t give us this discussion.  He just gives us the data, which he pairs with several answers spinning around the blogosphere.   We are left to draw our own conclusions.  Again, I’m not a scientist, so I’m not suited to judge the relevance of the data. But I can definitely judge the argument; and the argument in Knappenberger’s case doesn’t even aim at demonstrating relevance.  It simply shows that that there are other data.  One would expect more on relevance conditions from a cherry-picker’s guide.

Here’s another set of data to add to his chart.

The temperature in my downstairs guest room stays roughly 65 degrees, year-round, give or take ten degrees.  Thanks to the insulating nature of Colorado’s soil, our indoor air temperature on our bottom floor doesn’t fluctuate much.  If I include such data in my calculations of the climate, I’m likely to see that, over the past decade, temperatures have not changed.  They’re flat at 65 degrees.

Fortunately, it’s easy for most people to see how irrelevant my basement temperature is to a broader discussion of climate.  It’s irrelevant because it is not a good measurement of climate.  It’s just as irrelevant as the temperature in my refrigerator or the CPU temperature of my computer.  What is less easy to see, however, is that this relevance is not a given.  A scientist has to first identify this information as irrelevant and then filter it out of her calculations.

Scientists don’t waste much time discussing the climate of their basements.  Most everyone knows that basement climates are irrelevant. Scientists do spend time discussing, however, other possibly-relevant noise that only may or may not have something to do with the correct answer about to whether the climate is moving up or down.

As I said, Chip Knappenberger does not show, or even discuss, whether the data chosen to demonstrate a given point of view were either relevant or irrelevant.  He does not offer up the reasoning.  He simply shows us that different graphs yield different answers, paying little attention to the correctness of the answer.

Finally, a note of caution that should cast doubt on this guide.  Knappenberger has the following to say for himself, in his second paragraph, typed in italics:

What I can say for certain, is that the recent behavior of global temperatures demonstrates that global warming is occurring at a much slower rate than that projected by the ensemble of climate models, and that global warming is most definitely not accelerating.

This is an exceptionally strong claim.  It is not just moderately strong. It is exceptionally strong. Even knowing nothing about climate science, as is my case, I am far less likely to put stock in a claim posed in the positive, about what one knows “for certain,” than a claim posed in the negative, about what we “cannot know for certain.”  This is not because some things cannot be known for certain, but rather because when there are reliable and expert sources that say otherwise, said certainty can readily be called into question.  If someone says that they can say for certain that X did not cause Y, and yet there are many reliably expert people who beg to differ, then you should question not those who beg, but those who proclaim certainty.

It appears that some scientists, particularly over at RealClimate, beg to differ.  They write:

Even the highly “cherry-picked” 11-year period starting with the warm 1998 and ending with the cold 2008 still shows a warming trend of 0.11 ºC per decade (which may surprise some lay people who tend to connect the end points, rather than include all ten data points into a proper trend calculation).

It is clear even to a non-scientist like me that looking at a short timespan and drawing a conclusion about the certain state of the earth’s climate is a form of cherry picking the data.  Stefan is correct to point this out.  There is natural variability over any set of years, and if this natural variability is at all to be taken seriously, as Stefan gives feasible argument that it should, then we need to look across a longer time horizon.  Unlike what we’ve been exposed to in the cherry-picker’s guide, we do get here, in short form, an appeal to relevance.

About these ads

50 comments

  1. What? Are you growing Yamal larches in your basement guest room?
    =====================================


  2. Some of the comments from the link:

    • For the past 8 years (96 months), no global warming is indicated by any of the five datasets.

    • For the past 5 years (60 months), there is a statistically significant global cooling in all datasets.

    • The globe has been cooling rapidly for the past 8 years. [Using the CRU and satellite records]

    There are arguments on both sides, indicating that the science is far from being settled. That is the reason there is such desperation by those heading to Copenhagen to pig out on lobster, brie, caviar and champagne, as they try to get a commitment from the U.S. to spend $Trillions of taxpayer dollars on a repeatedly falsified conjecture.

    As beneficial CO2 continues to rise, the planet’s temperature continues to fall: [link] The planet itself is falsifying the conjecture that CO2 has any meaningful effect on the climate.

    So who should we listen to? Conniving political appointees promoting an agenda? Or planet Earth?


  3. The problem is that there are better and worse arguments for some positions. Ascertaining the betterness or worseness of arguments is a pretty difficult task, irrespective of Copenhagen. It is irrelevant to the climate question whether people in Copenhagen will eat lobster or whether they will eat shoelaces. What is relevant is whether the correct data are being used to support the correct position. Simply pointing out that there are differences in positions between parties isn’t itself enough to undermine the correctness of one party’s position over another’s. Adding information about the reliability of sources does, however, call into question the strength of an argument that proclaims “certainty.”


    • Ben,

      Please try to understand how the scientific method works.

      For over a century the accepted theory has been that the climate fluctuates above and below its long term trend line on a multi-decaldal time frame. The gradual warming trend line goes back to the Little Ice Age [LIA], and to the last great Ice Age before that, with shorter periods of warming and cooling riding above and below the long term trend line.

      The planet has been warming naturally in fits and starts since the LIA. There is no dispute that there is global warming. But now a new hypothesis has emerged, which claims that carbon dioxide [specifically, human produced CO2] is the cause of global warming. This is the CO2=AGW hypothesis.

      According to the scientific method, this new hypothesis must explain reality better than the existing theory. As climatologist Roy Spencer puts it: “No one has falsified the theory that the observed temperature changes are a consequence of natural variability.”

      The AGW believers are claiming that the current climate, which is very benign and well within historical parameters, is not natural. But their claim lacks proof, or even empirical evidence. It is based solely on computer models, which are programmed by people who probably in all sincerity believe in AGW.

      The long-accepted theory of natural climate fluctuations within defined parameters can not be the result of CO2, since CO2 has been many times higher in the past without causing runaway global warming. In fact, numerous times in the past when CO2 was thousands of parts per million [compared with today's very low 387 ppmv], the planet has descended into Ice Ages repeatedly.

      In the scientific method, scientific skeptics have nothing to prove. Every good scientist is a skeptic. It is the alarmist crowd that has the entire burden of showing that CO2=AGW. They have failed pretty spectacularly. Not a single GCM [computer model] was able to predict the past very severe northern hemisphere winter, and not one even came close to predicting the declining global temperature over most of the past decade. They predicted warming instead. Yet computer model predictions are the basis of the CO2=AGW hypothesis.

      The scientific method requires that an upstart hypothesis has the burden of showing that it is valid. Normally this would be accomplished by the new hypothesis predicting the climate — something it has failed to accomplish. Even hindcasting the models by inputting all current known data fails to predict today’s climate.

      Keep in mind that scientific skeptics have nothing to prove. They are simply skeptics. A new hypothesis must prove its validity. With the scientific method it’s not: “My theory versus your hypothesis.” Rather, it is: “You have the burden of showing that your new hypothesis explains reality better than the existing theory.” The AGW folks have never been able to do that.

      Not only is CO2=AGW an increasingly questionable hypothesis, but the planet continues to cool while CO2 continues to rise. The AGW folks are trying to explain that away by citing a mysterious, hidden “heat in the pipeline.” But they have been unable to identify where that supposed heat is located, and it can not be measured. The ARGO network, with thousands of deep sea buoys, shows deep ocean cooling.

      Every contention made by the believers in CO2=AGW has failed to pan out. Every one. Yet the same people insistently demand that we must immediately agree to spend $Trillions on their repeatedly failed hypothesis.

      Directing so much money into climate science and unnecessary CO2 mitigation, at the expense of all other science, will starve those other science projects of funding. In fact, it is already occurring.

      Doesn’t it make more sense to pause before spending enormous amounts of our national wealth, until we understand the climate better?

      That is the critical question. But we can see that the Copenhagen steamroller, run by political appointees, is desperate to hobble the West based on a failed hypothesis, before that failure becomes apparent to the general public.

      So the question is: must we act right now? Or should we take the time to learn more? Which path do you choose?


      • Smokey, thanks for the very succinct summary!!!

        Bruce


    • Ben, do you read ClimateAudit.org?

      It is extremely helpful in “ascertaining the betterness or worseness” of many of the arguments. Steve McIntyre, in my opinion, does a wonderful job of making some fairly complicated statistical issues easy to understand.

      And there are a number of highly qualified mathematicians and scientists that post there and participate in the comments.

      Given the current politicized nature of the issue, I think it would be very difficult to come to a rational, informed opinion about the AGW debate without first spending a fair amount of time at ClimateAudit.org. It’s been a real eye-opener for me.


  4. I have a fairly good understanding of how the scientific method works, both conceptually and practically. Falsifiability is a pretty strict standard, even for the staunchest advocates of the strictest versions of the scientific method. There are an infinite range of implausible or incorrect views that have not been falsified.


  5. And your last question is a different question, but one I’m happy to take up at some point soon.


  6. Your basement is not such an irrelevant example. Some of the longest instrumental records include early periods where the temperatures were measured inside rooms. How to treat those in house measurements, e.g. how to use them, is an issue of not little interest.


    • Fascinating. I didn’t know that. But the temperature of my CPU is probably not of great interest. At the very least, records of my CPU temperature only date back two years.


      • Those interested might want to read Manley (1974) and learn how issues of micro-siting, urban heat islands, TOBS, etc. have been considered by climatologists long before Pielke Sr., McIntyre and Watts claimed original discovery:

        http://www.rmets.org/pdf/qj74manley.pdf


      • BTW: Watts didn’t clam “original discovery”. Watts noted that there were siting issues. He raised the issues with the folks at USHCN-NOAA. USHCN responded that their software adjusts for siting issues and that UHI isn’t real. Watts & volunteers did (are doing) a commendable job of documenting site issues and are providing important QA that ought to have been done by USHCN-NOAA.

        Bruce


  7. The ability to pick and choose which samples to use in an analysis is an advantage unique to dendroclimatology.
    =
    Dendroclimatologists don’t gather and report enough meaningful metadata to make an a priori decision what constitutes a meaningful sample. Therefore we reserve the right to define those criteria a posteriori, or not define them at all – making subjective choices based on undisclosed reasoning.
    .
    I can not believe that Jan Esper does not regret saying this. If he does not now, he soon will. If he were to clarify what he meant, that would be most welcome. I still find it hard to believe he meant what he said.


    • Yes, no doubt Esper will start regretting it any moment now, especially because of all the media attention and public opprobrium he’s been subjected to ever since McI broke this story… in September 2005 regarding a comment made in 2003. It’s like you’ve stopped even trying, bender.

      Ben, the quote can be found in its context on page 92 of this paper.


      • Hey, the worm is turning, Steve; too many climatologists were awfully casual about their science and their statistics, and look at the mess we’re in. It is not amusing, Pal.
        ================================


      • And the context of the quote matters how, exactly? Are you suggesting the meaning changes out of context? That he dared to put these words in print is laughable. That it passed review is telling. And sad.
        .
        I had, inded, always assumed the words were spoken in an offhand way, perhaps in a Q&A after a conference speech. In fact they were not. They were extremely deliberate. They’re so crazy that I never searched the literature for them. Never imagined someone could be so transparently bone-headed in print. Thanks, Bloom.


      • Tell the truth, Bloom: when did you come across the article? And what led you to search for it? Did you think I was misquoting because you yourself couldn’t believe your eyes? Tell the truth.


  8. “I’ll remain agnostic on the empirical question”

    Come on Dr. Hale, you’re being too coy here by half. You are not agnostic because you clearly disparage Knappenberger and tout Stefan. Maybe look up in your philosopher’s dictionary the word “underdetermination” and see how it might apply here. Your views are far too blackandwhite, rightandwrong for any sort of nuanced philosophical discussion, tho’ they are well suited for a political mash up on a blog.

    Knappenberger’s point is that there are multiple explanations consistent with the evidence, but there are no explanations of the evidence consistent with observations exceeding the model ensemble.

    Should you wish to leave your protective cocoon of the standard disclaimer and actually look at some data, you could make such judgments yourself, rather than subcontract them out based on factors other than the facts of the matter.


  9. Steve’s link:

    http://www.wsl.ch/staff/jan.esper/publications/TRR_2003.pdf


    • What do you think, LB – do you agree with the Esper et al. quote? (Any idea who actually penned it? Was it Esper, or a co-author?)


  10. I do agree with Steve that you are misrepresenting and framing the quote out of context.

    Regardless, you have already impeached yourself as a rational and objective party with this little bit of uncharacteristic honesty:

    I’m here to rip the funding out from under you and/or the folks you defend because it’s a sham the precision that is guys proclaim.

    Your a priori criteria is an aggressive troglodyte political agenda and has nothing to do with science.

    GFYS


    • What exactly am I misunderstanding or misrepresenting about the quote? I want to understand. And when can I expect to debate you at CA?


      • You are assuming it is a justification for his argument.


    • On the contrary, it is BECAUSE of your science – your failure to deliver a robust global climate reconstruction – that I conclude that your funding needs to be pulled. This was not a prior bias, but an a posteriori conclusion. No funding until the Esper principle gets sorted out. A principle that you seem to be avoiding.


      • McIntyre failed to falsify the robustness of MBH98. Every reconstruction since has added to the robustness of the conclusion that 21st century GSTs are higher than any in the last, now, nearly two millenia, with or without tree rings.

        You are a mad dog barking at the moon.


      • Let us debate that question at Climate Audit.


      • As for who is “mad”, I think it is Esper, if he still stands by his words.


      • Mad dog bender,

        I’m not avoiding the ‘Esper principle’. you are avoiding reasoned and informed comprehension of what is a parenthetical off hand remark in favor of your pre-prejudiced argumentum ad ignorantium interpretation by;

        1.) Not reading the paper itself, but making context free and erroneous conclusions as to its purport.

        2.) Not having the contexual background knowledge and experience in dendrology to understand its particular meaning. (Hint: it isn’t about data analysis, it’s about data collection.)

        I have no interest in going into a long pedagogical explanation that you are so evidently inclined to reject out of hand or just plain ignore. You will, as all good science students should, have to investigate the matter for yourself.


      • So you refuse to explain what you think the context adds to the quote?


      • If you were to explain how I’m misrepresenting Esper’s view, I would back off the issue. Why do you want the issue to go unresolved?


      • “advantage unique to dendrochronology”.

        Who needs context? The word ‘unique’ is damning.
        ==============================


      • I gave you a clue, doofus. Show me you can think or STFU.

        Unique relative to what? In what context?


      • You have no defense for ‘unique’. The dendros just think they have a medium which allows them to bend the rules of science. They, and you, are wrong.

        Read the discussion at climateaudit.org headed ‘Upside Down Mann’. Why am I getting an image of Michael Piltdown Mann getting a swirlie?
        ===================================


      • Ben Hale, please consider reprimanding “luminous beauty” for his less-than-luminary, less-than-beautful retorts. Two strikes now. Do we really need a third?


      • Challenged thusly by luminous beauty:

        “Show me you can think or STFU.”

        it behooves me to mention that I don’t see as an “advantage” the *burden* of having to choose sites in particular, hard-to-find circumstances, where. moreover, no experiments have ever been done to calibrate ACTUAL responses to temperature in a controlled experiment.

        To further suggest that this burden of selecting the correct sampling method (site and species combination) is “unique” to one domain is non-sensical. The only sense in which it is unique is that you must GUESS at the species-site combination that you PRESUME will give a strong response.

        This is not an “advantage”. It is a *disadvantage*.

        So, naturally, the reader is led to contemplate: what advantage could he possibly be thinking of? The advantage of substiting “unresponsive” chronologies for “responsive” ones? Hmmm. That would be statistical suicide; yet it seems this is precisely what Briffa has done. Did Briffa incorrectly invoke LB’s interpretation of the Esper principle? Did he invoke my interpretation instead?

        The world awaits Dr. Briffa’s detailed reply.


      • LB: Can I speak now?


      • …no experiments have ever been done to calibrate ACTUAL responses to temperature in a controlled experiment.

        False.

        …you must GUESS at the species-site combination that you PRESUME will give a strong response.

        Over a hundred years of well replicated and progressively refined research demonstrates you don’t know what your talking about.

        You are free to blather your ignorance, but you are well advised it is better to keep silent and be thought a fool than to speak and remove all doubt.


      • LB?


    • You see what brings out the vitriol? “Say whatever you like, but don’t threaten our funding.” Until then it’s all chuckles. Chuckles and demonizing.


  11. Those wanting a debate about the details of dendro science are more than welcome at delayed.oscillator, Deep Climate and Open Mind (although I’m afraid the tolerance of each for drunk commenting — and at your age, too, bender! — has sharp limits). McI has pretty much used up his 15 minutes.


    • And you, of course, are welcome to comment at Climate Audit – if you can stick to the science.


  12. Hello from Russia!
    Can I quote a post in your blog with the link to you?


  13. Open Mind is not… open. See the discussion where Tammy banned Lucia for being, well, right. He didn’t like having someone question his precious “two box model” calculations, so he banned her. Turns out, Tammy was simply wrong about his model and couldn’t stand that someone pointed it out.

    Real Climate is not open. It is an echo chamber. “Guests” regularly have comments snipped that don’t toe the party (sorry, Team) line. Folks who regularly question the science (as should happen in science) get banned. It’s not a formally stated ban (Gavin doesn’t send a message saying, “You’ve been banned”, they set their software to automatically block comments from certain users. The users don’t know that they’ve been blocked. Nice and open!!!

    On the contrary, the more commonly viewed as “skeptic” blogs have much more open communication.

    So, where would you choose to go to learn something new???

    Bruce


  14. “Luminous Beauty” argues by name-calling. Like this:

    “GFYS”, “Mad Dog Bender”, “troglodyte”, “GFYS”, and so on.

    I have found this to be the routine, typical name calling that passes for debate on climate alarmist blogs — when they don’t simply censor comments outright that refute their AGW belief system. Any commentator who feels they must resort to insults is doing it for only one reason: they lack a coherent, credible argument.

    The fact is that the basis for the CO2=AGW conjecture comes entirely from always-inaccurate computer climate models. There is zero empirical data measuring the “A” part of AGW. That may be because AGW doesn’t exist, or it may be that it is so small and insignificant that it cannot be measured. In either case, there is no rationale for spending $Trillions based on speculation.

    There is no empirical fact set supporting AGW. None. There are GCMs [models], but a model is only a tool; it is not data. And without data, AGW is mere speculation.


    • smokey, you missed “STFU”.


    • Smokey: You forgot to add, “…and the models don’t work.”

      People are told (with a remarkably straight face) that the models don’t work for medium term predictions (more than 2-3 weeks.. time scales that could readily be tested and dis-proven), but are super-duper-great for time scales of 50+ years!

      Does this sound logical?

      Bruce


      • It may not sound logical to everyone, but there is a logic. Not saying it’s correct, mind you. Just saying that there is a logic.


  15. Ben,
    I see a very basic aspect of science that you are missing here.

    Please explain your statement that scientists “select out data using criteria that ‘seem to fit’ their view of what is happening.”

    I agree that they DO get to select data collection methods and parameters that fit their view of what is happening.

    But once the data pours in from the experimental method, they have to live with it.

    Data selection criteria can’t be based on the data.

    There has to be a physical (i.e. non-data-based) theory/hypothesis/method behind the data selection criteria.

    Medical testers can’t remove data.
    Chemists can’t remove data.
    Nuclear physicists can’t remove data.
    Biologists can’t remove data.
    Surveyers can’t remove data.

    etc etc etc.

    Yet this practice has apparently become SO common in this arena of science that it is simply accepted!


  16. [...] Maybe someone can explain to me what McIntyre is after, but from my vantage, it looks a lot like he misses the point. The point should be a simple one: if he wants to demonstrate a failure in the science, he has to do so through the formal, albeit flawed, channels. Very few readers of his blog are qualified to judge whether what he’s saying makes sense; and it is definitely true (a) that there are some qualified people who object to his methods, and (b) that there are some people in the world who are meticulous enough to back up crazy ideas with lots and lots of numbers. Both of these facts cast suspicion on what he, or anybody else for that matter, writes. Thus, we have formalized systems of peer comment and review.  You can read all about it at the three, relatively hot, hockey stick threads here, here, and here. [...]


  17. [...] hominem is one of many fallacies of relevance. I spoke earlier about fallacies of relevance in the Cherry Ping Pong thread. Ultimately, fallacies of relevance are tricky li’l buggers, because some claims that [...]



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 28 other followers

%d bloggers like this: