29 November 2009

CRU on Global Temperature Data

The Times had an article yesterday reporting the old news that CRU did not have in its possession the original station data from some locations that comprise its global temperature index. I am quoted in the Times article as follows:
“The CRU is basically saying, ‘Trust us’. So much for settling questions and resolving debates with science,” he said.
The quote comes from a blog post I put up last August when CRU announced that it did not have some of the original station data. Here is the full context of my quote:
CRU has in response to requests for its data put up a new webpage [NOTE: Apparently this page is no longer up on the CRU emergency server] with the following remarkable admission (emphasis added):
We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.
Say what?! CRU has lost track of the original data that it uses to create its global temperature record!? Can this be serious? So not only is it now impossible to replicate or reevaluate homogeneity adjustments made in the past -- which might be important to do as new information is learned about the spatial representativeness of siting, land use effects, and so on -- but it is now also impossible to create a new temperature index from scratch. CRU is basically saying, "trust us." So much for settling questions and resolving debates with empirical information (i.e., science).
Today I received an email from a climate scientist of CRU-email fame complaining about my quote in the Times. He says that the national meteorological services have the original data, suggesting that I was misrepresenting the situation. I replied to him as follows:
I would suspect that there are some very profound disciplinary differences in the handling of data here between the community I am from and yours. If, for instance, an economic research unit were releasing analyses of global economic activity in support of policy claimed to not hold the original country data -- instead saying, well the countries have it -- that would be highly problematic.

My advice to you and your colleagues is that the defense that you present in your email to me is not a very good one. Rather, I suggest instead being open and simply saying that in the 1980s and even 1990s no one could have known that maintaining this data in its original form would have been necessary. Since it was not done, then efforts should be made to collect it and make it available (which I see CRU is doing). Ultimately, that will probably mean an open-source global temperature record will be created. If you believe -- and I see no reason to suspect otherwise -- that such an open-source analysis will confirm the work of Jones et al., then you should be welcoming it with open arms.
Obviously, CRU should have taken these steps long before the present circumstances, but regardless, they are now moving towards greater responsiveness and transparency. When the data is available in its original form those skeptical of climate science can then do the temperature math themselves out in the open where everyone can see their work. If the global numbers come out as CRU has presented over the years, then it will strike a blow to skepticism about global temperature trend records produced by CRU and restorse a good deal of credibility to this area of climate science. At that point, the fellow who emailed me and his colleagues can rightly boast of their integrity and say "told ya so." Until then, a defensive, circle-the-wagons approach is probably not the best course of action. But old habits die hard.


  1. Is long term CO2 historical data as important as temperature data for justifying decarbonization? If so, has that data been subjected to more QA procedures than the temperature data?

  2. For people who come close to admitting fraud, you sure seem to give them a huge benefit of a doubt.

  3. Roger
    Thanks for the clarification. Gavin at RC indicated that the individual NWSs have the data - but as you say given that CRU left everybody with the impresssion that they have this data by virtue of their responses to FOI requests and the claims that they could not release the data because of prior agreements with these NWS, then I agree that CRU should be expediting the reassembly of this data. Until that data is available and has been reassessed it is hard to see how any research based on this data or derivatives thereof can be used. This is the price that Jones et al has imposed on all other climate researchers.

  4. First, the CRU statement is nonsense.

    The required data has not been available in useful form for all these years.

    The plain fact of the matter is that CRU did not publish, and refused to make available (even through FOI requests) the identification of the temperature series used in their index, or their algorithm.

    The fact that a data series is publicly available for download is not helpful absent the knowledge that a particular series was used in the CRU index.

    The extraordinary simplicity of making this information available (a simple list of the temperature series used) is all the more reason to question why it was not.

  5. Has anyone here commented yet on “New Zealand’s Climategate”?

    Has Roger’s faith in government CO2 intervention been shaken yet?

  6. Second,

    If the temperature data is made publicly available, and it sensibly replicates the CRU temperature indexes, it will not "strike a blow to skepticism about global temperature trend records" because as of this moment, the dominant assumption is that the CRU records DO represent an accurate depiction of an index based on the available data.

    Measurement difficulties, and the sudden disappearance of more than half of the temperature stations during the past 20 years will continue to be an issue.

    But I do not see any respected skeptics predicting that the actual data will fail to replicate the actual index.

    There is no upside here for the CRU researchers, and they should be embarrassed to even considering putting out a press release announcing that their results are not the product of some sort of systematic fraud.

    Their role, at this point, should be remedying as many of the deficiencies in openness and transparency as they can as quickly as they can, and letting the scientific community judge the quality of their work (while they sit quietly in the corner).

  7. This underlines the importance of multiple independent confirmation of key findings, in this as any other science. NOAA and GISS have long maintained separate data series and analysis (conveniently grouped online by realclimate.org), which I assume rely on much the same raw NWS data but are separately analyzed. Not enough? Better still are data from sources altogether separate from near-surface air temperature measurements, such as borehole temperatures (relying on soil/rock storage of heat) and seawater temperatures (after all, 90% of the heat added to the system winds up promptly in the oceans). Let's be clear: the key question for skeptics in all this mess is (somehow, still) whether the planet has in fact been warming up rapidly since the 1970s, as was predicted (since the 1970s) from greenhouse calculations. If we ask what all this means for policy, that is truly the question in all this. To answer that, besides the direct temperature measurements we have measurements of ice and glacier disappearance, spring budbreak, migration and distribution of birds, etc. etc. In short, there is no reason whatsoever to question the main conclusions of the CRU analsis, since we would have arrived at the same conclusions if CRU had never existed.

  8. Spencer,

    The problem is not whether we expected the temperatures to go up. The question is how much have they gone up and how much of that is due to C02? If they have gone up less than currently claimed because of dubious analyses by CRU the that would likely imply that the CO2 is not a crises and there is no justification for committements to rapid emission reductions. A slower, practical, technology focused approach would likely be better.

    What I find frustrating is so many alarmists do not understand is it is the 'alarm' which most sceptics dispute. Not the warming or the abstract goal of reducing CO2 emissions when/if it can be done in an economically rational way.

  9. -7-Spencer,

    1) The satellite data do not support your claim that “the planet has in fact been warming up rapidly since the 1970s”.

    The 1998 peak resulting from el Nino warming appeared more dramatic when compared against the Mt. Pinatubo cooling. Temperatures then stabilized around 2002 and have been flat to down ever since.

    Further debunking your distorted impression of temperature trends:

    A) Examine the ongoing, uninterrupted 10,000 year cooling trend at both the Greenland ice sheet and Vostok, Antarctic.

    B) Click here and here for the citation links and more details.

    2) We have peer reviewed science (from the Pielkes themselves) demonstrating a warming bias from the surface station data (and demonstrating that the satellite data are FAR more reliable).

    3) Even NOAA admits that 15 years without any warming would (with 95% certainty) invalidate the IPCC computer models:

    “The [computer model] simulations rule out (at the 95% level) zero trends [in global temperatures] for intervals of 15 yr or more, suggesting that an observed absence of warming of this duration is needed to create a discrepancy with the expected present-day warming rate.”

    4) This paper is one of many suggesting that we’ll see at least 20 years without any warming.

    Personally, I expect (purely on the basis of historical precedent) that we’ll see 30 to 40 years of cooling much like the cooling seen in the USA from 1934 to 1979.

    5) Click here and here and here to debunk your concerns over “ice and glacier disappearance”.

    6) Click here for some general commentary about the (endlessly ironic) general hysteria over anything which represents “change” (beneficial or otherwise).

    7) Click here for some basic climate change science.

  10. Got to love Spencer's arguments. We know that "adjustments" add almost all the warming in the GISStemp database. We know that the urban heat island influence is laughably underestimated. We know that NZ and Australia are just as bad. As the NZ scientists responded, their adjustments are the same used by all the other AGW true believer scientists.

    Nothing to see here. Move along. Our religion is correct and the conclusions are settled.

  11. All roads lead to CRU.


  12. What is the point of representing to the world that you have a massive collection of data related to an issue of supreme importance, then insist that your institution not be required to actually hold the data?

    Instead we are "sold" this idea that there is such a thing as "value added" data such that it is better data than the raw data and makes the data that went into the "value add" obselete.

    I can't help but wonder if there is a now lost record set of experimental trials relating to determining if witches indeed float. Perhaps there, instead, is a "value added" set of data from an authority on witches that was based on such a record. Such "value added" data would certainly tell us that witches float. Perhaps we would then be obligated to accept that witches do float and that witch trials by ordeal be reintroduced to society.

  13. @ SBVOR -

    How do you account for the 3+mm per year rise in ocean surface levels in the last 15 years. My back-of-the-envelope calculations suggests that is about a 30 cubic kilometer ice melt in the last 15 years unless there is some other source of water out there.

  14. -13-Adam,

    1) Since you failed to cite a source, I don’t take your assertion seriously.

    2) Click here and get some perspective on ever changing sea levels.

    3) The ice has been melting (this time) for the last 20,000 years. So what?

    Change is the ONLY constant in an ever changing multiverse. Get used to it.

  15. You have a splling mistak in the title.

  16. Not credible excuses. I am becoming more and more comfortable in my role as wild eyed, 'science made to fit the government policy' conspiracy theorist.

    The emails portray a highly pro active partisan attitude. That isn't surprising. It's why those individuals were employed. Thatcher appointed John Houghton, Houghton appointed Phil Jones and the rest of the staff.

    The same pattern will be found in Rupert Murdoch newspapers. Senior staff will have remarkably similar poltical views to their proprietor.

    Further, the fundamental fact of the economic system is that the vast majority are employees, with their job as their only means of survival. In those circumstanceas, it is very easy for management to apply pressure.

  17. Spencer and Raven,

    Furthermore, whether or not we expected temperatures to do something, and whether or not the temperatures were due to increased CO2, there remains the question of the "unprecedented" rise in temperatures, and the doom that is predicted to follow.

    If the temperature rises and life as we know it improves (or stays about the same), why blow trillions of dollars to make ourselves worse off? If the case is rather that the warming is somewhat similar to what happened in the past, when things seem to have gotten demonstrably better in some ways (MWP, RWP), then the whole climate intervention argument falls apart.

  18. So GISS doen't have the raw data too? Or is it a subset of CRU?

    I recall CA going thru much analysis of how the raw data was processed in different parts of the world (I seem to remember something about looking for Waldo)

  19. Roger,

    The author of the Times story should have made clear he was quoting an old blog post of yours. It's highly misleading otherwise.

  20. -19-keith

    Yes, the entire story is kind of odd, as the "news" is 3 months old -- hence my post.

  21. -19-Keith & -20-Roger,

    Noting the date and context of Roger's quote would have made for slightly better journalism. But, I fail to see how failing to do so is "highly misleading".

    We all know that CRU's refusal to share their data is a story which predates the revelation of the CRU e-mails. But, it is also a story more strongly and more directly reaffirmed by said revelations.

    So, while the article might falsely lead the reader to conclude that Roger’s quote was a reaction to the posting of the CRU e-mails, the reality is that ClimateGate -- especially that aspect of ClimateGate relating to the refusal to share data -- is an evolving story which predates the posting of the e-mails. Given that fact, I don’t think the context and date of Roger’s quote matters much within the context of this particular article.

  22. SBVOR -

    Since you felt the need to state 2 and 3 it seems you did take 1 seriously.

    Nonetheless, here is one of many sources on ocean level rise.


    Roger runs one of the few blogs where people from the full spectrum of opinion and experience related to climate change and policy can discuss things with little hyperbole. So doing things like bolding a statement that few are going to argue with (you've described entropy, for onw thing) isn't going to win folks over. Nor is trying to reduce the discussion to two binary camps - that is where just about every other climate blog is.

  23. SBVOR (22)

    The quote from Roger is "highly misleading" because, as Andrew Sullivan points out,
    "It reads as if it were offered yesterday in response to 'news' that UEA had somehow destroyed its original data."

  24. -21-Adam,

    After correcting your URL from .COM to .GOV, I failed to find data supportive of your claim of a “3+mm per year rise in ocean surface levels in the last 15 years”.

    Furthermore -- on this NOAA page -- a simple average of the “MSL Trend” column “for all data to 2006” produced an annual average of 1.68mm per year. A simple extrapolation over the next 100 years would suggest a sea level rise of 168mm (6.6 inches). That is not exactly the kinda stuff I tend to loose sleep over.

    Even the ever alarmist IPCC suggestion of 7 to 23 inches of sea level rise by 2100 fails to generate any sense of panic.

    (See Table 3.1 on page 45 from this rather large and slow loading IPCC PDF file.)

    Additionally, simple extrapolation of a 3mm per year trend (IF it were true) would only produce 300mm (11.8 inches) of sea level rise over a period of 100 years (well within the range of the IPCC suggestions).

    Meltwater Pulse 1A sounds kinda dramatic. But, we’ll have to blame that one on Mother Nature. And, we won’t see anything like that again until we complete and exit from the next glacial cycle.

    Again, click here and get some perspective on ever changing sea levels.

  25. To say that the CRU has decided to release the data is sub optimal. That means the same crew will be involved. Frankly, at this stage, they need to have an independent IT audit of the CRU, and then all data is released through the FOI apparatus, along with a full log since July, of accesses and file changes for the CRU computing environment with auditor signoff.
    To leave these people where they are, and still in charge of data access merely entrenches the bureaucratic excuse making Roger mentions above.

  26. Cum hoc ergo propter hoc. So Mr Pielke if the temperature rise is authenticated, that "proves" "it", because that correlates with rise in Man-made CO2 and this is scientific evidence of causal link?

    Science is in a bad way.

  27. adam,
    Substitute 'AGW skeptics' for 'witches', and you will be spot on.

  28. Hmm, this is irrelevant to the thread, but I just noticed there are two "Keiths." For the record, the "Keith" who wrote comments 19 and 24 belong to kkloor.

    I'll be sure to use that hence forward.

  29. There is a good article in this weeks ECONOMIST. It's called "A Heated Debate". It seems to be premium article so no link. Here is a snippet:

    This newspaper believes that global warming is a serious threat, and that the world needs to take steps to try to avert it. That is the job of the politicians. But we do not believe that climate change is a certainty. There are no certainties in science. Prevailing theories must be constantly tested against evidence, and refined, and more evidence collected, and the theories tested again. That is the job of the scientists. When they stop questioning orthodoxy, mankind will have given up the search for truth. The sceptics should not be silenced.

  30. What strikes me the most about the CRU gang in all of this is their political naïveté. They act as if they were quiet academics investigating some obscure point of arcanum of interest to a few, as opposed to scientists attempting to influence trillions of dollars worth of public policy decisions resulting in very close scrutiny both scientific and political.

    There is an email in the CRU dump were somebody complains about how they want to ditch all this scrutiny, and all of these pesky people asking for data, and "get on with my own research agenda." My jaw dropped. How could anyone be that clueless? How could anyone believe they could, potentially at least, impact so many important areas of political and economic life and not be put under the microscope? The political reality is going to mean their "life's work" is going to be conducted in a fishbowl, in full view of anyone who wants to examine it. Science shouldn't have a problem with that. Hell, science should THRIVE in such an environment. Instead, for some reason, these folks have opted for secrecy instead of transparency. (Maybe they simply didn't like the level of criticism any work with political/economic implications is liable to generate, but, I'm sorry, that's tough. Its the nature of the beast. Either get used to it, or go do something else.) As a result they have become a cabal, and as a result the science has suffered. Imagine if ten years ago temperature data had been released and people found out the troubles inherent in the collection and homogenization process. Might we not be well on our way to a much better set of data? Wouldn't the state of the science be improving as a result?

    Granted, none of this would make the political component of all this go away, but nothing can make the political component go away. To wish otherwise is nothing but an exercise in utopian dreaming. However, the secrecy has bought them nothing but trouble. A) The science has suffered, and brought the entire discipline into ill repute, and B) It has brought the motivations of scientists into question. Are they being driven by a fear of criticism? An ideological agenda? A quest for power and influence? All of those possibilities and more are brought into play by their actions.

    My own interpretation of their motivation lies somewhere in-between the benign and the diabolical, but the point is the "benign" interpretation is enough to make this a scientific scandal of the first order, and they STILL don't seem to get that.

  31. @Iconic MW
    Note that climate scientists of Phil Jones' generation started out in a dull, irrelevant, tiny corner of applied physics. They never got training in media, law, or politics. Climate change grew into big science and big politics well after their formative years.

    This is no excuse, of course.

  32. -31- Richard,

    You are undoubtedly right, but its not much of a comfort. These are important people and they seemingly have no clear conception of how politics in our society works. Its hard to imagine they have any sympathy for it as a result. The more I look at this stuff the more I become convinced that Ortega y Gasset was right when he talked about the "barbarism of specialisation." We have whole generations of enfants terribles deciding our fate.


  33. Richard Tol

    As I said earlier, Margaret Thatcher set up the CRU and Hadley for purely political reasons (the war against coal and coal miners). There is no way Thatcher would spend a penny of taxpayer's money on creating something that wouldn't provide a return.

  34. Roger,

    I am intrigued by the connection between two recent stories that I haven't seen linked anywhere. In NZ, it was shown that all the warming trend came from adjustments to the raw data. This shouldn't have been much of a surprise, the same thing is true of the warming in the US. In NZ, scientists defended their adjustments by claiming that they simply do the same type adjustments that are employed by everyone else.

    AGW defenders have responded to the CRU mess by claiming that it doesn't matter because their graphs roughly agree with other datasets.

    My point -- if all the warming comes from adjustments and everyone uses the same adjustments in their datasets, why should their agreement be proof of anything? it's just as plausible that they are making the same mistakes.

  35. -33- Stan,

    There are many legitimate (or at least scientifically defensible) reasons to adjust data in order to facilitate comparisons over time. The NZ data looks weird because most of the adjustments you see worldwide involve reducing modern raw data downward to account for increased urbanization, but in NZ the older data was skewed down to make it cooler in the past then the raw data would indicate. Now, there are some exceptional circumstances where that would be justified, but it certainly wouldn't be the norm. If the NZ team is correct, and the general skewing of older data downward is "normal operating procedure" well then there would be a problem.

    From what I've read it looks like the NZ data is just a mess, but I don't think anything nefarious was going on.

    Hey maybe in the next stimulus package the Dems could fund an army of researchers to go and make a usable data set. Just think of all the grunt-work jobs that could create!

  36. Just a note for Adam. (re "How do you account for the 3+mm per year rise in ocean surface levels in the last 15 years. My back-of-the-envelope calculations suggests that is about a 30 cubic kilometer ice melt in the last 15 years unless there is some other source of water out there.")

    Most of the sea level rise is believed (including by the IPCC) due to the ocean expanding due to its increasing temperature. Some contribution from glaciers etc but probably not so significant, and as you can see from the IPCC 4th assessment report (p.46), the error bars on the largest ice sheets in the world are larger than the values - meaning it's not so easy to work out:

    "The estimated range in mass balance for the Greenland Ice Sheet over the period 1961 to 2003 is between growth of 25 Gt yr–1 and shrinkage by 60 Gt yr–1 (–0.07 to +0.17 mm yr–1 SLE). Assessment of all the data yields an estimate for the overall Antarctic Ice Sheet mass balance ranging from growth of 100 Gt yr–1 to shrinkage of 200 Gt yr–1 (–0.27 to +0.56 mm yr–1 SLE) from 1961 to 2003, and from +50 to –200 Gt yr–1 (–0.14 to +0.55 mm yr–1 SLE) from 1993 to 2003."

    What to take away from this?

    It does show that the ocean has been heating up and most people (including me) believe that the world has warmed up in the last century.

    But those who like to understand science want to analyze exactly how much, which is what this blog post is about.

    I'm surprised at the reaction of what we might call the AGW camp (even though that might imply only 2 positions on this subject).

    Defend, defend, defend. The data is out there already. The data isn't needed. These people asking for the data are only trying to damage the proven case for ...

    All the press releases in all the world aren't going to convince those who are skeptical that there isn't some kind of error or, even worse, systematic bias in the temperature records.

    The more press releases I see saying how everyone is making a mountain out of a molehill, the more it seems like there is something interesting worth finding!

    Give us the data! The real data.

  37. -34 eric144
    In the early days, climate change was the perfect environmental problem for politicians. One could claim a visionary care about the planet and the future, while knowing that painful measures to reduce emissions were a few elections removed.

  38. Everything I've seen shows that adjusting the past to be cooler and adjusting the last 30-40 years to be warmer is standard operating procedure. For all the databases. They downplay UHI, reduce the use of stations in higher altitudes or latitudes, smear warmer temps around, and generally play games with the data.

    Does anyone really believe that Manhattan has less UHI impact today than 100 years ago? Before the cars and highways, air conditioning, massive sprawl, and the addition of many skyscrapers?