[UPDATE: March 11, 2010. A letter signed by over 150 scientists released today includes this claim:The reference list of the AR4 contains about 18,000 citations, the vast majority of which were published in peer-reviewed scientific journals.Really?]
Andreas Bjurström, of Göteborgs Universitet (Sweden) has summarized some of his ongoing research on the IPCC in a guest post below. The focus is on the Third Assessment Report (TAR) of the IPCC, published in 2001.
Gray literature in the IPCC, A guest post by Andreas Bjurström
In the wake of Climategate, several controversies related to the IPCC have followed. Most notable are the errors regarding the Himalayan glaciers and the arrogant statements in media by the IPCC chair.
Whether the literature that the IPCC rely upon is submitted to peer review is a crucial component in these controversies, since the IPCC derive much authority from the peer review process. I will therefore describe bibliometric data from the IPCC third assessment report to nuance the common claim that the IPCC rely mainly on peer review journal articles. (Unfortunately neither I nor anyone else has bibliometric data from the fourth assessment report. An educated guess is that the third and fourth assessment report is similar. This will also be assumed in the following).
Table 1. Basic bibliometric properties of IPCC TARIPCC TAR consists of 14,000 references, of which 62 % are journal references (see Table 1). The difference between working groups (WG) in the types of references used is large. The natural science dominated WG1 has 84 % journal references. The social science oriented WG3 has only 36 % journal references.
The majority of references in WG1 and WG2 are peer-reviewed scientific journal articles. That WG3 rely mainly on peer-reviewed scientific journal articles is simply false.
There is consequently no doubt that this is overstated: “The IPCC cites 18,000 references in the AR4; the vast majority of these are peer-reviewed scientific journal papers” (www.realclimate.org). Nevertheless, such statements is a crucial component in the justification of the consensual standpoint.
TAR consists of 1100 different journals, but the reference frequencies decline exponentially. As few as six journals stand for one third of all journal references. 60 journals stand for two thirds of all journal references. Analysis of the politicization of peer review should be concentrated to these core journals (Climategate reveal the intention to orchestrate politicized peer review, but we don´t know whether this is common behaviour among the infuential group of climate researchers that control the core journals).
The proportion of gray literature in the IPCC is difficult to estimate given my empirical material. Non journal references (my technical term) includes everything that is not peer review journal articles; previous IPCC assessment reports, books, reports, conference papers and articles in the daily papers and magazines. Some of these are submitted to scientific peer review whereas others are not. Moreover, peer review literature, published literature and gray literature has fuzzy boundaries.
The proportion of gray literature is obviously low for WG1. Non journal references include mainly books, reports and conference papers. The proportion of non journal references diverge slightly for chapter 13 (Climate Scenario Development) with one third non journal references.
The proportion and the variety of non journal references are similar in most chapters of WG2. A few chapters contain a majority of non journal references. Chapter 8 (Insurance and Other Financial Services) contain the highest proportion of non journal references and references to the daily papers and magazines are more common here as well.
It seems that books are much more common in WG3 compared with WG1 and WG2 whereas conference papers are not more common. Also reports (including not just research reports, but also reports from governments and business) seems to be more common in WG3. Chapter 1 (Setting the Stage: Climate Change and Sustainable Development) and chapter 3 (Technological and Economic Potential of Greenhouse Gas Emissions Reduction) contain three quarters non journal references.
This rough analysis of non journal references indicates some well-known facts; the natural sciences have a strong preference for journal articles whereas the social science has a significant publication in book form. Research results from the natural sciences are usually submitted to peer review. The social sciences (one may include many non social science studies on impact, adaptation and technical issues here as well) are submitted to peer review less often and refer in many instances to sources of dubious scientific standards. The reason for this is obvious. The natural sciences, especially the earth sciences, dominate climate research whereas the social sciences is marginal and lack funds. Only 12 percent of the total number of references in the third assessment report belongs to the social sciences.
This blog post shows that the claim that the IPCC rely mainly on peer review scientific articles is true for working group 1, partly true for working group 2 and false for working group 3.
In the battle over credibility, where peer review journal articles are hard currency, there are literally thousands of references to gray literature in the IPCC assessment reports. Let us all be honest about this, whatever your views on the climate debate.
