05 May 2010

Richard Tol's Draft Submission to IAC IPCC Review

Richard Tol has asked for your comments on his draft submission to the InterAcademy Council Review of the IPCC. His draft is below, and your comments on it are encouraged.
Richard Tol's Draft Submission

I was a lead author of two chapters in AR2's Working Group 3 report (1995), convening lead author in the Special Report on Regional Impacts published in 2001 (Working Group 2), contributing author of one chapter in AR3's Working Group 1 report (2001), lead author of one chapter in AR3's Working Group 2 report (2001), and contributing author of one chapter in AR4's Working Group 2 report (2007).

The IPCC is a victim of its own success. Policy makers trust the IPCC reports as neutral and authoritative assessments of climate research. Therefore, people with a political agenda have tried to influence the IPCC. Such attempts were largely in vain in AR2 and AR3, but this is not true for AR4. Working Group 2 systematically portrays climate change as a bigger problem than is scientifically acceptable. Working Group 3 systematically portrays climate policy as easier and cheaper than can be responsibly concluded based on academic research. These biases can be found in the chapters, the technical summaries, the summaries for policy makers, and the synthesis report.

The most important problem of the IPCC is the nomination and selection of authors and Bureau Members. Experts are included or excluded because of their political allegiance rather than their academic quality. The “right” authors are put in key positions with generous grants to support their IPCC work, while the “wrong” authors are sidelined to draft irrelevant chapters and sections without any support.

IPCC assessment reports do not cover the entire literature. For instance, the IPCC does not fully assess the literature on trade-offs between greenhouse gases and international environmental treaties. Both areas are actively researched and conclusions are typically critical of current policy, but were largely omitted from the IPCC reports. The literature on comparing the impacts of climate change to the costs of emission reduction falls outside WG2 and WG3, and thus outside the Synthesis Report as well. There is a risk that the nascent literature on post-hoc evaluations of climate policy will be ignored in AR5.

AR5 should be put on hold until the IPCC can prove that its author teams indeed have the relevant expertise, and its authors are at least as qualified as their peers. If needed, the IPCC should request additional nominations. The IPCC should submit the proposed outline for AR5 for peer review, and ensure that it covers all aspects of climate change and climate policy, including those parts of the academic literature that are potentially embarrassing to policy makers.

The IPCC member states are represented by their environment departments. For AR6, this responsibility should be transferred to their research departments or their academies. The selection process for authors and Bureau Members should be made transparent. The IPCC should commission a scientometric assessment of its Bureau Members, authors, and chapters.

The IPCC Plenary has delegated its supervisory powers to the IPCC Bureau, which is also its executive. This implies that there is no mechanism to correct the IPCC Chair if he oversteps his mandate and gives policy advice on behalf of the IPCC, if he says embarrassing things to journalists, or if he uses the IPCC to raise funds for his home institution. There is no mechanism to reconcile differences between IPCC Working Groups on the treatment of interdisciplinary topics (e.g., global warming potentials). Therefore, the Chairs of the IPCC and its Working Groups should leave the IPCC Bureau, and the Bureau should adopt a supervisory role under a strong and independent chairperson.

The procedures for drafting IPCC reports are fine, but their enforcement is not. Review Editors should be more independent. The Review Editors rather than the Convening Lead Authors should have the final say about chapters. Review Editors should actively seek the input of referees. Review Editors should have the right to reject sections of the report that do not meet basic quality standards at the deadline.


  1. My suggestion is to include some graphics. Compare the present IPCC process: http://www.ipcc.ch/publications_and_data/publications_and_data.htm to "new and improved."

  2. The last paragraph begins, "The procedures for drafting IPCC reports are fine,..."

    That seems somewhat contradictory to the problems cited in the previous paragraphs.

  3. Peer review is meaningless unless the authors are forced to address all comments rather than just ignoring the inconvenient ones.

  4. -2/3
    I was running out of steam in the final paragraph.

    How about this:
    In contrast to the procedures for governing the IPCC and selecting its members, the procedures for drafting IPCC reports are fine. However, these procedures were not enforced in AR4 WG2 and WG3. Some chapters rely heavily on gray literature but ignore peer-reviewed literature on the same matter. Other chapters cite papers published after the deadline. Incomplete drafts were sent for peer-review. Substantial material was added after the final review. In order to avoid repetition, Review Editors should be more independent, and return chapters for further revision if needed. The Review Editors rather than the Convening Lead Authors should have the final say about chapters. Review Editors should actively seek the input of referees. Review Editors should have the right to reject sections of the report that do not meet basic quality standards at the deadline.

  5. Richard Tol -4

    In my opinion, perhaps "acceptable on paper but not in practice" in lieu of "fine." IF the procedures were not followed, then, there was something inherently flawed in the process structure.

  6. Excellent article Richard!
    Explains a few things

  7. Although I am not sure the ultimate product is worth the painfully laborious and bureaucratic governance needed to be more objective, I would add that open review on the internet of the selections of individuals and certainly public posting of all information used and conclusions on the internet with opportunity for public comment would be illuminating and also a great joint science education experience.

  8. -7- Sharon
    The selection process cannot be fully opened because of privacy and libel laws. Selection does rely on arguments such as "Ms Jones is smarter than Mr Smith but she is not a team player." Both could sue.

    During the last couple of weeks, I have been working on a thought experiment "if I were convening lead author of a chapter in AR5". I would start a blog and put every subsection up for public comment.

    I would not use a wiki, primarily because of my bad experience with Wikipedia.

    I am reluctant to recommend this for the entire IPCC, because there are many colleagues who do not know how to handle the blogosphere.

  9. Much ado has been made of the IPCC's "Review Process" - described by one AR4 Lead Author as an "incredibly intense peer review process" - the purpose of which is purportedly:

    "to ensure that the Reports present a comprehensive, objective, and balanced view of the areas they cover."

    In order to meet this standard:

    Responses to the Reviewer Comments should be both identifiable and readily quantifiable.

    Currently, they are neither. While the Reviewer is identified [with the exception of Govt. Reviewers who are identified only by country], the responder is not. Is this not the *reverse* of "peer-review"?!

    Considering that authors often cite their own work in the WG reports - and respond to Reviewer Comments on same - the potential for conflict of interest suggests that more transparency is definitely required.

    Quantifiability can be readily achieved if the responses are standardized (e.g. Accepted, Partially Accepted, Noted, Rejected etc.) and not included in the same column as explanatory notes/reasons.

    For verification purposes, it would also be useful to include the (Chapter) Section numbering along with current Page/Line identifiers - so that it will be easier to compare reviewed text with final (web) version.

    When "grey literature" is used, the IPCC's rule for identifying such material must be respected and adhered to. In AR 4, only 6 of the non-peer-reviewed References were appropriately designated as such.

    P.S. I also agree with Craig 1st #5 and Sharon F. #7

  10. I think the clue to all the problems are in the title.

    Inter-governmental (politicised) Panel (exclusive) Climate Change (ideological) Assessment Report (partisan).

    You need to go from such a discredited organisation and flawed format to one which is transparently and truly international, inclusive, scientific and non-partisan.

    I do not have faith that the UN can or wants to reform the IPCC in this way. We know the IAC will only produce a whitewash report.

  11. Richard Tol #8

    "start a blog and put every subsection up for public comment" and "I am reluctant to recommend this for the entire IPCC [...]"

    Perhaps this would be a good way to implement greater transparency, verifiability and quantifiability in the Review Process.

    The technology certainly exists to create a "bloglike" web application with controlled permissions for posting [e.g. authors, reviewers, editors) - which would minimize the need for moderation. Could even have links to underlying data for tables, references, etc :-)

    Each section could have link to more open blog for "public" comments with moderation level determined in accordance with author's preferences.

  12. You write:

    "The “right” authors are put in key positions with generous grants to support their IPCC work, while the “wrong” authors are sidelined to draft irrelevant chapters and sections without any support."

    Who provides these grants? I thought that authors are not paid directly by the IPCC but continue to receive their institutional salaries. If so, the IPCC is not responsible for different levels of support, but I may well have misunderstood.

  13. I would also add a sentence to the penultimate paragraph saying that the IPCC should have clear guidelines on conflicts of interest of its officials (similar to the UN's own guidelines) as discussed in this earlier post and others:


  14. -12- Ruth
    Officially, all IPCC authors etc are volunteers. However, the IPCC is part of the professional duties for civil servants and people from NGOs and companies. Some academics work on IPCC matters during the weekend and holidays. Other academics get a government grant for something climaty with remarkably few obligations.

    The IPCC is a bit like the Olympics in the old days. Amateurs only.

  15. Second draft is here: