22 December 2009

Peer Review in the IPCC

The IPCC has long expressed a strong preference for relying on peer-reviewed scientific literature in its reports (PDF) :
Contributions should be supported as far as possible with references from the peer-reviewed and internationally available literature, and with copies of any unpublished material cited.
However, the IPCC has evolved such that it increasingly relies on "grey literature" in its reports. Its guidelines (PDF) explain the need for additional procedures to handle grey literature:
Because it is increasingly apparent that materials relevant to IPCC Reports, in particular, information about the experience and practice of the private sector in mitigation and adaptation activities, are found in sources that have not been published or peer-reviewed (e.g., industry journals, internal organisational publications, non-peer reviewed reports or working papers of research institutions, proceedings of workshops etc) the following additional procedures are provided.
The IPCC asks its authors to be very discerning in what grey literature to include:
Critically assess any source that they wish to include. This option may be used for instance to obtain case study materials from private sector sources for assessment of adaptation and mitigation options. Each chapter team should review the quality and validity of each source before incorporating results from the source into an IPCC Report.
The IPCC has strict guidelines for obtaining and making available any source from outside the peer reviewed literature.

Obviously, the IPCC's claim to authority rests in its claims to have a very rigorous process for vetting information and including only that which the scientific community finds to be accurate and reliable. A former director of the IPCC explained that the report was "probably one of the most peer-reviewed documents you could ever find." A few weeks ago in Copenhagen the current head of the IPCC touted its rigor while explaining the need to act decisively to reduce emissions (PDF):
The IPCC assessment process is designed to ensure consideration of all relevant scientific information from established journals with robust peer review processes, or from other sources which have undergone robust and independent peer review. The entire report writing process of the IPCC is subjected to extensive and repeated review by experts as well as by governments. In the AR4 there were a total of around 2500 expert reviewers performing this review process.
Given the claims made on behalf of the IPCC, finding flawed information in the report should be cause for serious concern. I have documented how the IPCC has systematically misrepresented the science of disasters and climate change here on various occasions, and it appears that these sorts of errors are not unique.

Consider the case of the melting of Himalayan glaciers as discussed in Chapter 10 of the IPCC WG II report (PDF). The IPCC claimed that Himalayan glaciers could be mostly gone by 2035, prompting much concern since the report was released in 2007. For instance, CNN reported in October of this year:
The glaciers in the Himalayas are receding quicker than those in other parts of the world and could disappear altogether by 2035 according to the 2007 Intergovernmental Panel on Climate Change (IPCC) report.
It turns out that the 2035 value is not just wrong, but when confronted with the error, the IPCC leadership apparently has refused to look into, clarify or even admit that there may be a problem in its report.

In a blog posting today John Nielsen-Gammon, an atmospheric scientist at Texas A&M, confirms claims first raised by J. Graham Cogley, a glaciologist in the Department of Geography at Trent University (which were reported on my father's blog and then by the BBC). Here is an excerpt from Nielson-Gammon's posting:
To recap, the available evidence indicates that the IPCC authors of this section relied upon a secondhand, unreferreed source which turned out to be unreliable, and failed to identify this source. As a result, the IPCC has predicted the likely loss of most or all of Himalaya's glaciers by 2035 with apparently no peer-reviewed scientific studies to justify such a prediction and at least one scientific study (Kotlyakov) saying that such a disappearance is too fast by a factor of ten!

This could have been a small, inconsequential error. The WG2 Chapter 10 authors did not highlight the prediction as a key finding in their executive summary, nor does it appear in the summary for policymakers. But such an astounding prediction could not help but attract attention. And it has long since become effectively common knowledge that the glaciers were going to vanish by 2035.

The Indian environment ministry released a report in November by Vijay Kumar Raina that concluded that Himalayan glaciers on the whole were retreating, but not at an alarming rate or any faster than glaciers on the rest of the globe. According to The Guardian, countryman Rajenda Pachauri, the head of the IPCC, was furious.
Pachauri dismissed the report saying it was not "peer reviewed" and had few "scientific citations".

"With the greatest of respect this guy retired years ago and I find it totally baffling that he comes out and throws out everything that has been established years ago."

Given the nature of the peer review and scientific citations in the IPCC report, we have here a case of the pot calling the kettle black.
The BBC also reported on the IPCC reaction:

When asked how this "error" could have happened, RK Pachauri, the Indian scientist who heads the IPCC, said: "I don't have anything to add on glaciers.". . .

Murari Lal, a climate expert who was one of the leading authors of the 2007 IPCC report, denied it had its facts wrong about melting Himalayan glaciers.

This situation highlights the problem of the "laundering of grey literature" associated with IPCC reports, which occurs when an analysis or claim occurs outside the peer review literature and is subsequently cited in the assessment report. Because the IPCC is widely viewed to have been reviewed at such a high standard, the presumption is that it is very unlikely that simple errors with enormous consequences would remain in the final version of the report. Thus, claims reported by the IPCC are subsequently cited as a "peer reviewed source" which could very easily give a claim that originated in the grey literature a status that it never would have held without the IPCC's stamp of approval. This is of course why the IPCC requires that its authors evaluate the "quality and validity of each source" that it wishes to include. The IPCC authors are in effect serving as peer reviewers of any grey literature that it includes in its report, warranting its accuracy based on their expert evaluation. If this process breaks down, it would be very easy for false or misleading claims to be represented as authoritative.

In the case of melting glaciers in the Himalayas, the IPCC 2035 claim has led to, in Nielsen-Gammen's words, an egregious mistake becoming "effectively common knowledge that the glaciers were going to vanish by 2035." Like the common (but wrong) knowledge on disasters and climate change that originated in the grey literature and was subsequently misrepresented by the IPCC, on the melting of Himalayan glaciers the IPCC has dramatically misled policy makers and the public.

That the IPCC has made some important mistakes is very troubling, but perhaps understandable given the magnitude of the effort. Its reluctance to deal with obvious errors is an even greater problem reflecting poorly on an institution that has become too insular and politicized.


  1. Not to be needlessly repetitive, but the science could only be helped by posting all the literature references (including gray) and asking people to weigh in on the facts found, and the logical link to conclusions drawn.

    Scientists should, and do usually, have vast areas of disagreement; the beauty of science should be its disciplined approach to finding facts. By making it a social process (getting "scientific consensus"), we can't help but have an opinion poll of scientists rather than science. In my opinion.

  2. Is the peer-reviewed literature really any better? Mann and the rest of the hockey team got their studies into the IPCC assessments without audit or replication. When we get a look at Rahmstorf's silly 2007 study or Steig's Antarctica temperature smear, we see how really bad their analysis is. Why should we place any more credence in any of the other studies which continue to lack the transparency to be checked?

    The problem isn't grey literature or the IPCC's refusal to handle this particular correction properly. The problem is that the entire climate science "consensus" is based on a bogus process. The climate models are not verified or validated. They shouldn't have any place in an assessment of the settled science. The databases have been shown to lack serious quality control and have been adjusted in all kinds of ways which no one has reviewed. There are very serious questions about them. The self-checking function of science has clearly broken down. The whole process is fatally flawed.

    There is no point trying to sort out what little might be salvaged from the IPCC. Just put it out of its misery and start over. Let competing groups put together their own assessments and force them to support their conclusions. Replicate the finding of key studies and build an understanding of what we do know one solid brick at a time.

  3. The UN will never take action to reform the IPCC. What is needed is a NGO of scientists to become a shadow IPCC. I noticed this group http://www.nipccreport.org/ . I have no idea how much creditability they have. But it's the right concept. The NGO would need some big bucks backing them to get their ideas across in the media. With $20m media budget and help from dissenting scientists the IPCC could be brought to its knees in 6 months.

  4. Calling what the IPCC did as "mistakes" is rather charitable. Intention use to be the dividing line between mistake and intended action. One of the emails refers to "dirty laundry": www.eastangliaemails.com/emails.php?eid=347&filename=1059762275.txt

  5. The curious — I hate to impute motives, so just "curious" — thing I noticed when I was looking at this before is that the WWF report is very carefully sourced and cited except for the "2035" citation, which doesn't appear in the bibliography and which turned out to be very hard to track down.

  6. Refusal to correct highlights that the UN-IPCC pursuing it's own self interest. It needs alarmism for it's continued existence.

    Sen. Inhofe has it right when he states that the UN-IPCC is only interested in Hollywood Science. 2035 is "The Day After Tomorrow" in climatic terms.

  7. The World Wildlife Fund isn't a neutral party. It has an agenda. Moreover, it identifies its agenda as being diametrically opposed to the agenda(s) pursued by, say, the petroleum and coal industries.

    No one - not most mainstream scientists and certainly not the WWF - would consider it appropriate to use research authored solely by an industry source as the basis of any claim in an IPCC report.

    So how have we arrived at the point where the opposite not only take places but people don't even blush at this fact?

  8. Some other bloggers have pointed out that it appears that the trail of references eventually lead back to a 1996 UN hydrology report by V.M. Kotlyakov, Variations of Snow and Ice in the past and at present on a Global and Regional Scale.

    See http://unesdoc.unesco.org/images/0010/001065/106523E.pdf

    In that report, on page 66 he says that extrapolar glaciation is expected to shrink from 500,000 sq km down to 100,000 sq km. The Himalays are specifically mentioned as one of the areas that glacier are expected to STILL exist in 2350.

  9. I have some quantitative data on the proportion of grey literature in the IPCC 2001 report (I have 2 articles on the disciplinary bias and the (inter)disciplinary structure in the IPCC reports). I was surpriced to find so much grey literature as the IPCC claim to only use peer review, which is far from the truth. I see the IPCC peer review claim as just another way to be dishonest by overstating the objectivity on science.

    I have raw data (takes some time to quantify, but its a possible article to write on that) but not percent numbers. But as a rought guess:
    10-20 % grey literature in the earth science references in the IPCC report (the lowest numbers are found here) and 30-70 % in some chapters on economics and societal issues (the highest numbers are found here).

    Please e-mail me if anyone find this interesting and want more info or perhaps do some calculations and writing: andreas.bjurstrom@globalstudies.gu.se