Peer Review in the IPCC

The IPCC has long expressed a strong preference for relying on peer-reviewed scientific literature in its reports (PDF) :

Contributions should be supported as far as possible with references from the peer-reviewed and internationally available literature, and with copies of any unpublished material cited.

However, the IPCC has evolved such that it increasingly relies on "grey literature" in its reports. Its guidelines (PDF) explain the need for additional procedures to handle grey literature:

Because it is increasingly apparent that materials relevant to IPCC Reports, in particular, information about the experience and practice of the private sector in mitigation and adaptation activities, are found in sources that have not been published or peer-reviewed (e.g., industry journals, internal organisational publications, non-peer reviewed reports or working papers of research institutions, proceedings of workshops etc) the following additional procedures are provided.

The IPCC asks its authors to be very discerning in what grey literature to include:

Critically assess any source that they wish to include. This option may be used for instance to obtain case study materials from private sector sources for assessment of adaptation and mitigation options. Each chapter team should review the quality and validity of each source before incorporating results from the source into an IPCC Report.

The IPCC has strict guidelines for obtaining and making available any source from outside the peer reviewed literature.

Obviously, the IPCC's claim to authority rests in its claims to have a very rigorous process for vetting information and including only that which the scientific community finds to be accurate and reliable. A former director of the IPCC explained that the report was "probably one of the most peer-reviewed documents you could ever find." A few weeks ago in Copenhagen the current head of the IPCC touted its rigor while explaining the need to act decisively to reduce emissions (PDF):

The IPCC assessment process is designed to ensure consideration of all relevant scientific information from established journals with robust peer review processes, or from other sources which have undergone robust and independent peer review. The entire report writing process of the IPCC is subjected to extensive and repeated review by experts as well as by governments. In the AR4 there were a total of around 2500 expert reviewers performing this review process.

Given the claims made on behalf of the IPCC, finding flawed information in the report should be cause for serious concern. I have documented how the IPCC has systematically misrepresented the science of disasters and climate change here on various occasions, and it appears that these sorts of errors are not unique.

Consider the case of the melting of Himalayan glaciers as discussed in Chapter 10 of the IPCC WG II report (PDF). The IPCC claimed that Himalayan glaciers could be mostly gone by 2035, prompting much concern since the report was released in 2007. For instance, CNN reported in October of this year:

The glaciers in the Himalayas are receding quicker than those in other parts of the world and could disappear altogether by 2035 according to the 2007 Intergovernmental Panel on Climate Change (IPCC) report.

It turns out that the 2035 value is not just wrong, but when confronted with the error, the IPCC leadership apparently has refused to look into, clarify or even admit that there may be a problem in its report.

In a blog posting today John Nielsen-Gammon, an atmospheric scientist at Texas A&M, confirms claims first raised by J. Graham Cogley, a glaciologist in the Department of Geography at Trent University (which were reported on my father's blog and then by the BBC). Here is an excerpt from Nielson-Gammon's posting:

To recap, the available evidence indicates that the IPCC authors of this section relied upon a secondhand, unreferreed source which turned out to be unreliable, and failed to identify this source. As a result, the IPCC has predicted the likely loss of most or all of Himalaya's glaciers by 2035 with apparently no peer-reviewed scientific studies to justify such a prediction and at least one scientific study (Kotlyakov) saying that such a disappearance is too fast by a factor of ten!

This could have been a small, inconsequential error. The WG2 Chapter 10 authors did not highlight the prediction as a key finding in their executive summary, nor does it appear in the summary for policymakers. But such an astounding prediction could not help but attract attention. And it has long since become effectively common knowledge that the glaciers were going to vanish by 2035.

The Indian environment ministry released a report in November by Vijay Kumar Raina that concluded that Himalayan glaciers on the whole were retreating, but not at an alarming rate or any faster than glaciers on the rest of the globe. According to The Guardian, countryman Rajenda Pachauri, the head of the IPCC, was furious.

Pachauri dismissed the report saying it was not "peer reviewed" and had few "scientific citations".

"With the greatest of respect this guy retired years ago and I find it totally baffling that he comes out and throws out everything that has been established years ago."

Given the nature of the peer review and scientific citations in the IPCC report, we have here a case of the pot calling the kettle black.

The BBC also reported on the IPCC reaction:

When asked how this "error" could have happened, RK Pachauri, the Indian scientist who heads the IPCC, said: "I don't have anything to add on glaciers.". . .

Murari Lal, a climate expert who was one of the leading authors of the 2007 IPCC report, denied it had its facts wrong about melting Himalayan glaciers.

This situation highlights the problem of the "laundering of grey literature" associated with IPCC reports, which occurs when an analysis or claim occurs outside the peer review literature and is subsequently cited in the assessment report. Because the IPCC is widely viewed to have been reviewed at such a high standard, the presumption is that it is very unlikely that simple errors with enormous consequences would remain in the final version of the report. Thus, claims reported by the IPCC are subsequently cited as a "peer reviewed source" which could very easily give a claim that originated in the grey literature a status that it never would have held without the IPCC's stamp of approval. This is of course why the IPCC requires that its authors evaluate the "quality and validity of each source" that it wishes to include. The IPCC authors are in effect serving as peer reviewers of any grey literature that it includes in its report, warranting its accuracy based on their expert evaluation. If this process breaks down, it would be very easy for false or misleading claims to be represented as authoritative.

In the case of melting glaciers in the Himalayas, the IPCC 2035 claim has led to, in Nielsen-Gammen's words, an egregious mistake becoming "effectively common knowledge that the glaciers were going to vanish by 2035." Like the common (but wrong) knowledge on disasters and climate change that originated in the grey literature and was subsequently misrepresented by the IPCC, on the melting of Himalayan glaciers the IPCC has dramatically misled policy makers and the public.

That the IPCC has made some important mistakes is very troubling, but perhaps understandable given the magnitude of the effort. Its reluctance to deal with obvious errors is an even greater problem reflecting poorly on an institution that has become too insular and politicized.