05 March 2010

Gray Literature in the IPCC TAR, A guest post by Andreas Bjurström

[UPDATE: March 11, 2010. A letter signed by over 150 scientists released today includes this claim:
The reference list of the AR4 contains about 18,000 citations, the vast majority of which were published in peer-reviewed scientific journals.

Andreas Bjurström, of Göteborgs Universitet (Sweden) has summarized some of his ongoing research on the IPCC in a guest post below. The focus is on the Third Assessment Report (TAR) of the IPCC, published in 2001.
Gray literature in the IPCC, A guest post by Andreas Bjurström

In the wake of Climategate, several controversies related to the IPCC have followed. Most notable are the errors regarding the Himalayan glaciers and the arrogant statements in media by the IPCC chair.

Whether the literature that the IPCC rely upon is submitted to peer review is a crucial component in these controversies, since the IPCC derive much authority from the peer review process. I will therefore describe bibliometric data from the IPCC third assessment report to nuance the common claim that the IPCC rely mainly on peer review journal articles. (Unfortunately neither I nor anyone else has bibliometric data from the fourth assessment report. An educated guess is that the third and fourth assessment report is similar. This will also be assumed in the following).
Table 1. Basic bibliometric properties of IPCC TAR

IPCC TAR consists of 14,000 references, of which 62 % are journal references (see Table 1). The difference between working groups (WG) in the types of references used is large. The natural science dominated WG1 has 84 % journal references. The social science oriented WG3 has only 36 % journal references.

The majority of references in WG1 and WG2 are peer-reviewed scientific journal articles. That WG3 rely mainly on peer-reviewed scientific journal articles is simply false.

There is consequently no doubt that this is overstated: “The IPCC cites 18,000 references in the AR4; the vast majority of these are peer-reviewed scientific journal papers” (www.realclimate.org). Nevertheless, such statements is a crucial component in the justification of the consensual standpoint.

TAR consists of 1100 different journals, but the reference frequencies decline exponentially. As few as six journals stand for one third of all journal references. 60 journals stand for two thirds of all journal references. Analysis of the politicization of peer review should be concentrated to these core journals (Climategate reveal the intention to orchestrate politicized peer review, but we don´t know whether this is common behaviour among the infuential group of climate researchers that control the core journals).

The proportion of gray literature in the IPCC is difficult to estimate given my empirical material. Non journal references (my technical term) includes everything that is not peer review journal articles; previous IPCC assessment reports, books, reports, conference papers and articles in the daily papers and magazines. Some of these are submitted to scientific peer review whereas others are not. Moreover, peer review literature, published literature and gray literature has fuzzy boundaries.

The proportion of gray literature is obviously low for WG1. Non journal references include mainly books, reports and conference papers. The proportion of non journal references diverge slightly for chapter 13 (Climate Scenario Development) with one third non journal references.

The proportion and the variety of non journal references are similar in most chapters of WG2. A few chapters contain a majority of non journal references. Chapter 8 (Insurance and Other Financial Services) contain the highest proportion of non journal references and references to the daily papers and magazines are more common here as well.

It seems that books are much more common in WG3 compared with WG1 and WG2 whereas conference papers are not more common. Also reports (including not just research reports, but also reports from governments and business) seems to be more common in WG3. Chapter 1 (Setting the Stage: Climate Change and Sustainable Development) and chapter 3 (Technological and Economic Potential of Greenhouse Gas Emissions Reduction) contain three quarters non journal references.

This rough analysis of non journal references indicates some well-known facts; the natural sciences have a strong preference for journal articles whereas the social science has a significant publication in book form. Research results from the natural sciences are usually submitted to peer review. The social sciences (one may include many non social science studies on impact, adaptation and technical issues here as well) are submitted to peer review less often and refer in many instances to sources of dubious scientific standards. The reason for this is obvious. The natural sciences, especially the earth sciences, dominate climate research whereas the social sciences is marginal and lack funds. Only 12 percent of the total number of references in the third assessment report belongs to the social sciences.

This blog post shows that the claim that the IPCC rely mainly on peer review scientific articles is true for working group 1, partly true for working group 2 and false for working group 3.

In the battle over credibility, where peer review journal articles are hard currency, there are literally thousands of references to gray literature in the IPCC assessment reports. Let us all be honest about this, whatever your views on the climate debate.


Jason S said...

How does the fourth assessment compare?

My first thought in hearing these numbers is that TAR is old news.

Andreas Bjurström said...

Unfortunately, I don´t have anything on the fourth assessment, but it is unlikely that the general picture is much different. Probably a trend towards higher % peer-review for WG3 and the social sciences. I would be very surpriced if my conclusions in the blog post doesn´t hold also for the fourth assessment.

I studied the references to be able to quantify disciplinary bias and interdisciplinarity (relations between disciplines, using statistical analysis). I started with that 2005 (2 years before AR4 was released). I have an article in press on that (the journal Climatic Change).

Since there is so much debate now about peer-review in the IPCC, Pielke wanted me to use my material to say something on that. That is why I (we) are interested in this "old" material.

Read the comment from TripodGirl at another post here She have done a count on one chapter in AR4. Tols post "Bias in IPCC WGIII":

I've done a blog post that expands on some of Dr. Tol's concerns. Short version: Chapter 11 of Working Group 3's report cites 330 sources in its list of references. Fully 42% of those are grey-literature rather than peer-reviewed. This is despite the protests of Tol and others in their capacity as IPCC expert reviewers.


UAN said...

Andreas, thank you for sharing your work! I've also enjoyed reading your contributions here and at a couple of other sites (RC comes to mind).

A quick question on your statement:

The social sciences (one may include many non social science studies on impact, adaptation and technical issues here as well) are submitted to peer review less often and refer in many instances to sources of dubious scientific standards. The reason for this is obvious. The natural sciences, especially the earth sciences, dominate climate research whereas the social sciences is marginal and lack funds.

Outside of climate research, do the social sciences publish in peer-reviewed journals at a much higher rate? I'm curious to what extent lack of funds impacts the usage of peer-review literature in WG3, and how much, if any, is it reflective of the general practice of the social sciences in general.

One observation from the figures you provide. It seems that much of the debate going on around AGW may be running at cross purposes with really ill-define terms and misunderstood concerns. That is, much of the skepticism towards the "consensus" may be directed at WG3, but it's the natural scientists from WG1 who are responding, and from their point of view, the consensus is pretty robust. They may also be assuming that the science presented in WG2 and WG3 is as "settled" as in the natural sciences. My experience has been that there is a level of ambiguity in the soft sciences that the those from the hard sciences don't really appreciate. (Not to mention that folks from the soft sciences often feel that their research is as predictive as what's found in the hard sciences. :-)

Luke Lea said...

And in terms of policy relevance, WG3 > WG2 > WG1. Or am I mistaken?

Richard Tol said...

"The social sciences [...] submit to peer review less often"

Let's be clear that this is a description of the social sciences as applied to climate change and policy.

This is not a description of the social and behavioural sciences in general.

There is a measurement problem here. There is a difference between "This paper critically evaluates government policy ([as describe in] Government Report, 2010)" and "Government policy would make everyone happy ([as claimed in] Government Report, 2010)". The former is a valid but necessarily gray reference.

Luke Lea said...

And if the conclusions of WG2 and WG3 are driving the funding of WG1, mightn't this be a motive for WG1 not to criticize too closely WG2 and WG3? (Or the executive summaries of WG1 for that matter?)

Raven said...

Can I get some opinions on whether these statements are true:

The lead authors cannot easily ignore a paper appears in the peer reviewed literature that goes against the biases of the lead authors. They must explain why it was excluded.

No such requirement exists for grey literature. The lead authors are free to pick the grey literature that suits their biases and are under no obligation to explain why other sources were left out.

This means that allowing grey literature has the effect of amplifying the biases of the lead authors.

Richard Tol said...

See my two "bias" posts.

You are right that it is harder to ignore or misrepresent a well-known paper.

bernie said...

Nice job. Perhaps a la Lucia you could run a book on the % for AR4.
Also could you add another row for the number of journals that require and enforce the posting/archiving of data and analysis methods/code? I would be happy with a sample of 30 randomly selected journals for each segment.

UAN said...

...and refer in many instances to sources of dubious scientific standards.

This is the statement that really bothers me and perhaps could be expanded on. Citing "dubious" sources should have nothing to do with funding--though there may be a dearth of literature to draw on for WG3 due to the lack of funding of research in specific areas.


I agree in principle with the distinction you make between valid and invalid gray references. But there's a difference between "sounds right" and "is right", not to mention how far from the source one eventually gets (i.e., how much the gray literature is relying on other sources, and so on).

Malcolm said...

Conclusion: The soft-science contributions and the grey literature weigh more heavily than the actual hard-science.

That is very revealing!!!!!!!!!!!!!!!!!

Reiner Grundmann said...

UAN, Richard

the fact that there is less social science research published in the peer reviewed literature reflects, first of all, the lack of interest of several relevant fields (sociology, anthropolgy, even public policy). In contrast, economics, law, and international relations have been much quicker, historically speaking. Sociologists have made an 'alarmed discovery' of the climate change problem (very much mimicking the mass media).

Policy relevance of WG3>W2>W1
this is true, if ironic. Before the W2 scandals broke, only Wg1 got the attention of media and decision makers. The drama and story line was: will the next IPCC report (meaning: model predictions of WG1) be more alaraming than the previous or not?

TripodGirl said...

I've chosen (totally at random) another chapter from the AR4 Working Group 3 report to examine. I'm working on a new blog post which I hope to finish by tomorrow.

Sneak peak: in that case it's even worse than the Chapter 11 Dr. Tol discusses. Only 24% of the citations are from peer-reviewed journals. And, despite the protests of the IPCC's own expert reviewers, 2 press releases made it into the final list of references.

My post will be available here:


bernie said...

There may also be a far more significant issue with WG3 since it tends to employ almost exclusively observational rather than experimental studies -http://www.americanscientist.org/science/pub/everything-is-dangerous-a-controversy

The actual podcast is fascinating.

I found the link at Matt Briggs' site where he started a discussion on bad study designs - http://wmbriggs.com/blog/?p=2043

boballab said...

On the WG3>WG2>WG1 as to policy revelance it is true if you just apply a little logic to it. First WG1 is the straight science aspects that turn into model "projections" of x amount of increase in temperature. Now here is the rub, if that is all you get out of it is an increase in temperatures then there is no need for any action to be taken. That is why there is a WG2. Without the proposed impacts from the projected rise of temps in WG1, there is nothing to base a policy on. That is also why the hit to WG2 is actually very damaging. If it is shown there is little to no harmful impacts then there is no need for WG3 and the policies outlined. WG2 and WG3 has always been the IPCC's achilles heel besides the models and their built in assumptions.

Richard Tol said...

That post is now up. It is scary.

bernie said...

Very clear statement of the issue. The question becomes why such a disconnect between the avowed policy and the actual practice?

Is anonymity still of value?

Reiner Grundmann said...

"Based-on-peer-review-only" has become the mantra to keep the so called sceptics out. This was the one big stick to wield: "you don't publish in the peer reviewed literature (but in the Wall Street Journal etc). That this was a pretext can now be seen by the fact that politically "welcome", un-peer-reviewed material had an easy ride.

Andreas Bjurström said...

My text is foremost a description of the IPCC report. I don’t have any explanatory variables in the material, but I speculate a bit on causes. I agree that this can be questioned and improved.

It is important to not confuse the IPCC and research, for example, I don´t even know the rate of peer-reviewed journals within social science climate research.

I think this is important:
1: the general research practice of different disciplines
2: the total amount of published results relevant to climate change of diferent disciplines
3: The behaviour of the IPCC in choosing what to refer to.

1: The practise differ, bibliometrics know lots on this, all I know is the general trend that social sciences produce more books in the ratio of books over journal articles.

2: Funds is one aspects. The more funds, the more results. However, the interest for climate change within each discipline is also important (see Grundmanns comment).

3: IPCC refer mainly to peer review articles in WG1 because there is so many available. IPCC more or less must refer to gray literature in WG3 because of the lack of studies. However, there is no clear boundary for what should be included or excluded in WG3 (for WG1 this is rather clear). For example, IPCC can include general social theory to sthrengten the assessment if they wish.

My forthcoming article in Climatic Change is a through description of the representation of different disciplines in IPCC. Still, I don´t know whether there is systematic bias in what is included and excluded since I don’t have a material from the research context to compare with.

Reiner Grundmann said...

Frank Furedi has an article on Spike along the lines of my previous post (thanks to Hans who put it on Klimazwiebel!)

Andreas Bjurström said...

I think that the general outline (of gray literature in the IPCC) is well established. One could quantify the full AR4 report, but it´s very hard work, and the results will likely confirm what we already know. Better is to take a sample of a few chapters, e.g. like TripodGirl have done. Depending on aim, one can pick chapters to confirm the general characteristics, or outliers
with likely low or high ratio of gray literature for in-depth studies.

9 Bernie, not sure what you ask for (codes, methods, etc). Please explain again and I will try to answer, but it might be complicated to explain technical details. E-mail me if you want some data.

7 Ravens point is important. To what extend can IPCC ignore and pick references to ones liking?
That would be the most valuable in depth study. There is most likely processes of inclusion and exclusion that are important. The IPCC report is not mere a mirror with the same biases as the scientific community. Some knowledge on the peer reviewed literature in the domain of each chapter studied is needed, to be able to find relevant articles that are not refered to in the IPCC report. I think it is rather easy to find that, but I have not tried. Pielke can probably give examples for some chapters and Tol from other chapters. Also how the content are misrepresented.

Dean said...

I don't see where the idea that the importance of WG3 to policy is more important than WG1 is justified. WG1 is most important if the question is whether anything needs to be done. WG3 gains in importance if you support doing something and are debating what we should do. WG3 is very important to Roger's position about how decarbonization should be tackled. But for the many commenters here who think that we don't need to do anything, at least yet, WG1 is the most important driver.

Andreas Bjurström said...

Building concensus and legitimating policy:
WG1 -> WG2 -> WG3

Policy relevance/usefulness: WG3 > WG2 > WG1

boballab said...


Dean WG1 dosn't justify anything, all it does is say that we have had x amount of temp increase, which they attribute to risisng CO2. From that they project using models that there will be Y amount of warming out to 2100. So just becasue CO2 and temperature are rising does not,in and of themselves, call for a policy. You have to demonstrate that there is more negative impacts to those rises then positive, but WG1 doesn't address that, that is what is in WG2.

WG2 the impact section is where the IPCC says what all the negative consequences from the rise of CO2 and Temps, thus requiring we do something about it. However they don't mention the postitive side which has been recently highlighted in recent weeks, such as the forest growth in the Eastern US has been greater then expected due to rising CO2 and tmeps. Now with WG2 the IPCC says we have shown you the reason you need to take action (Himalayan glacier melt, rising sea level, less rainfall in Africa,and the melting artic icecap that seems to always be melting since 1922)but not what action, that is what WG3 is for.

So no WG1 is not the driver for taking action WG2 is and is also where the IPCC got caught with their pants around their ankles.

Mark B. said...

From my grad school days:

"There is no science in social science."

Dean said...

It is a common misconception that only climate models are used to project impacts. Study of past climate change epochs (ice cores, boreholes, etc) and what happened with them is more than adequate to demonstrate serious impacts.

Nor does the use of gray literature _necessarily_ undermine a particular section. The term is quite broad and many cases are perfectly justified to use.

langmann said...

You may also wish to consider constructing funnel or galbraith plots to rule out publication bias.

Andreas Bjurström said...

We have had a bit of discussion over at realclimate, about my blog post here, and this comment is very interesting (I post it here and hope that CM don´t mind be doing that):

594 CM says:
If anyone’s still reading this thread, I’ve been trying my hand at a simple Perl script to parse AR4 references and do a rough type count, in case it could shed some light on the discussion. After a day’s work, it still obviously doesn’t parse WG3 well and isn’t proven on WG2, so I don’t expect it to settle anything.

But summarizing 19,040 references from 44 chapters in 16 seconds is fun, and it did OK analyzing a WG1 chapter (ch. 4, snow and ice), which I also hand-counted for verification. The script correctly found 257 references and suggested 82-86% of these might be journal articles (against 84% in my count – including the non-peer-reviewed but tried and tested Stinemann (1980) in Creative Computing…).

Neither Andreas nor Gavin saw any reason why AR4 would be dramatically different from TAR with regard to journal references, though Gavin expected an increase. My very preliminary results agree. They suggest 90-92% journal articles in AR4 WG1, and the lower end of a 61-67% range for WG2 (cf. Andreas’s TAR figures of 84% and 59%, respectively). I’m not taking any bets on WG3 yet. If I manage to narrow down the errors, I’ll post what I find, though the anti-IPCC crowd will probably have hand-counted every chapter by then already (good, keep them busy).

With regard to “gray” literature, Marco’s and Gavin’s points are clearly the more important. As I said, 84% of the references in AR WG1 ch.4 (the snow and ice chapter) turn out to be journal articles, which is less than in the other AR4 chapters bar the first one.

I have classed the remaining 41 references as follows (hand count):
- 14 proceedings papers,
- 16 chapters in edited collections,
- 5 books, and
- 6 technical reports.

Generally these were published by serious-sounding scientific bodies or publishers, so I assume they have been accepted, edited, and in some cases refereed, by experts. The percentage of references that have passed scrutiny equivalent to peer review could well be in the high nineties. I agree with Marco: The journal article count alone does not tell the whole story.

That bears repeating, because Andreas started out (#571) referring to his “thorough analysis of gray literature” on Pielke’s blog, claiming “(t)here are thousands of references to gray literature in the IPCC reports”. He later backtracked, admitting he had no numbers on “gray” literature as such, only on journal and non-journal publications (#580), but still claimed to have scored a point against RealClimate on a technicality, tch tch.

Aldalin said...

What is the best way of managing Grey Literature.
What I mean is storage and how it may affect copyright.

Post a Comment

Note: Only a member of this blog may post a comment.