07 July 2010

The Muir Russell Review

The Muir Russell Review is out (here in PDF), and it has plenty in it for everyone on all sides of the debate over the East Anglia emails to crow about and to complain about. It has some strong rebukes of the scientists involved in the emails and dismisses many, but accepts some, of the criticisms raised by their strongest critics.

In this post I want to highlight a very puzzling statement in the report. The Muir Russell report characterizes the IPCC as follows (p. 41):
The IPCC produces assessments of the current state of understanding of climate change, its causes and implications. Its approach is to produce the most probable account of these issues; together with their uncertainties, and to identify where there is insufficient evidence to discriminate between different interpretations of a phenomenon. Its purpose is to produce a "best estimate" of what is currently understood, through the work of a group of scientists chosen for their expertise and experience to make reasoned assessments on the balance of evidence. It is not to produce a review of the scientific literature.
The idea that the IPCC presents a "best estimate" understanding based on the views of a selected group of scientists is completely contrary to how the IPCC characterizes its own work. To suggest that the IPCC is "not to produce a review of the scientific literature" is just plain wrong.

Here is how the IPCC WG I (the relevant working group for the MR inquiry) characterizes the scope of its own assessment process (emphasis added):
All chapters undergo a rigorous writing and open review process to ensure consideration of all relevant scientific information from established journals with robust peer review processes or from other sources which have undergone robust and independent peer review.
Note that it says "all relevant scientific information" -- it says nothing about a "best estimate." The IPCC states very clearly in its principles for report preparation (PDF) that its reports are supposed to,
present a comprehensive, objective, and balanced view of the areas they cover
And it explains that authors of IPCC reports,
should clearly identify disparate views for which there is significant scientific or technical support, together with the relevant arguments
IPCC reports are further supposed to
represent the latest scientific, technical and socio-economic findings and are as comprehensive as possible . . . [and] provide a balanced and complete assessment of current information
The notion that IPCC reports are supposed to present a selective view of climate science, representing the judgments of a select group of experts is in fact contrary to the mission of the IPCC.

The Muir Russell mischaracterization of the IPCC becomes relevant in the report when it uses the characterization as a criterion for evaluating the efforts revealed in the emails to minimize or exclude certain perspectives. For instance, the Muir Russell report explains with respect to one alleged instance of exclusion of peer reviewed literature from IPCC drafts that (p. 76):
Those within the [IPCC] writing team took one view, and a group outside it took another. It is not in our remit to comment on the rights and wrongs of this debate, but those within the team had been entrusted with the responsibility of forming a view, and that is what they did.
This speaks directly to problems of the IPCC, revealed to some degree by the emails, but of much broader concern. The IPCC is supposed to "identify disparate views" not hide them from view or take the side held by the author team. Had the Muir Russell review actually taken an accurate view of the IPCC, it is likely that its judgment about the appropriateness of the behaviors revealed by the emails would be considerably different.

It is not the job of the IPCC authors to serve as selective arbiters of the peer reviewed literature and judge which peer reviewed science they agree with and disagree with. This only invites extra-scientific considerations into the assessment process and a cherrypicking of the literature, rather than a considered assessment. The job of the IPCC should be exactly as it says it is -- to produce a comprehensive, balanced and complete review of the relevant literature. If the IPCC finds itself in a situation where its author team reflects a perspective represented by only a subset of the literature, then the IPCC has a problem.

The released East Anglia emails -- for better or worse -- revealed some problems associated with in-group control of parts of the IPCC. Muir Russell's sanctioning of in group behavior in the preparation of IPCC reports is a notable shortfall in what otherwise appears to be a nuanced and comprehensive assessment of the implications of the East Anglia emails.

21 comments:

Hector M. said...

The view expressed by the Russell report in this regard, is probably the view conveyed to them by the CRU scientists. They have also repeatedly behaved as if they believed their mission was to adopt a view they regard as the best estimate, disregarding other views they see as inferior or wrong, without feeling obligated to perform a balanced examination of the evidence (contained in peer reviewed literature). The paragraphs cited by Roger, besides, do not mention the fact that the IPCC is supposed to provide a dispassionate scientific account of climate change, without any policy recommendations (and implicitly: without presenting only such evidence as supports their preferred policy).

Simon said...

Well summarised. What has been identified as a flaw in the IPCC's process - in that its procedures in this regard were open to abuse, and apparently were abused by those at the CRU - Sir Muir has somehow identified as a feature.

That's quite an impressive failure to get a handle on one of the main rubbing points for climate science reformists. One cannot help but wonder if Sir Muir should actually have interviewed key dissenters like McIntyre after all.

One has to wonder who's testimony/ies led Sir Muir and his team to understand that the purpose of the IPCC WG1 was different from that defined by the IPCC itself.

Reiner Grundmann said...

Roger, this is interesting. In a paper I am trying to publish I quoted one of your statements (here on the blog) about the selective omission of data by the IPCC. One reviewer objected to this, as follows: "in fact this is what IPCC authors are charged to do - they have to select published material they deem to be relevant - hence, by definition, they omit other papers. Nothing sinister per se in this."

Richard Tol said...

Well spotted.

Roger Pielke, Jr. said...

-3-Reiner

I'd ask your reviewer (via the editor) to explain how such selectivity is consistent with the IPCC's mandate to be "comprehensive" and to "clearly identify disparate views".

This is not "sinister" just an open invitation to fudging and cherrypicking the literature.

Reiner Grundmann said...

Roger, this may be difficult given that the paper has been rejected. Will have to try again with another journal. The section on the IPCC will certainly gain in depth!

Stan said...

Roger,

Now that you have a post on it, I'd like to repost this comment I placed on the dutch report thread today --

The Muir Russell report has been released. Once again, an 'official investigation' has been conducted with only one side being interviewed. I have a question for you -- I have to think that that the sloppy, one-sided efforts are deliberate. Do you think that the blatantly obvious shortcomings of the process are really a signal by the investigators?

I'm serious. Assume that the people involved all understand their marching orders are to produce a whitewash. Having some shred of integrity, they deliberately avoid making the effort of dotting all the 'i's and crossing all the 't's that a thorough quality effort would require. Since the 'investigation' is clearly a coverup to anyone looking at the incompetence of the process, they are signaling that they aren't happy with what they have had to do. It's subtle sabotage.

Why else would they produce such obviously inadequate investigations? They don't even bother to go through the motions.

jgdes said...

The report states that the raw data was archived in 2009. However this was the "value-added" data. The raw data is still somewhere out in the fens waiting to be discovered by our descendents.

Since Jones already admitted in a previous inquiry that the raw data was lost there's no excuse for such a gross error so it must be just plain dishonesty.

And what is a representative of BP doing there having the cheek to judge anyone else?

And I always look out for the phrase "we find no evidence" as opposed to "there is no evidence". The former means that they didn't bother to look and if you don't look you don't find it but yet remain weaselly covered. Of course the evidence of dishonesty is right there in the emails - prima facie evidence at that! Of course it was the BP man it seems who was charged with trawling the emails and reporting back to the others. What he'd consider to be honest rather than just standard PR trickery is quite different from most folk.

j ferguson said...

These reports seem a product of apprehension about killing off the AGW industry which already exists.

It may be that the climate fright business including advertising, education, wind and solar power, publishing, the research groups and universities has passed the tipping point where a recall is politically possible.

Assuming that these activities are not insignificant to the economies of some western states, is it so surprising that no-one would want to do anything which might deflate them?

Frontiers of Faith and Science said...

Until a review of CAGW/AGW/Crugate/IPCC includes enforceable demands for full disclosure and actually hears from skeptics, the so-called reviews will be nothing more than whitewashes, and poor ones at that.

Frontiers of Faith and Science said...

Roger,
'Nuanced'?
Please. This is as nuanced as a Papal review of Luther coming to the conclusion that he is indeed a baseless heretic deserving of the fires of hell.
You and others granting goodwill to those who are clearly acting in cynical bad faith is not earning you their respect.

dagfinn said...

They just redefined the IPCC into something no rational person would want. I doubt that they understand the consequences.

Brian H said...

"notable shortfall in what otherwise appears to be a nuanced and comprehensive assessment" -- since aggressively selective group-think is at the core of the problems, I'd say that the shortfall is fatal, not just notable. What value the rest of the "comprehensive assessment" if thoroughgoing gatekeeping is blinked at?

hro001 said...

Not only that, but this report attempts to diffuse responsibility for the IPCC chapter product to the "chapter team" (as opposed to the Lead Authors, who IIRC, do in fact, have the final say - and who were instructed by the TSU to not include their names in their responses, so as to reflect the view of the entire "chapter team"). There's also a rather curious elevation of the actual role of the Review Editors.

Not to mention that wrt Ch 6, while Muir Russell made much ado over whether or not Wahl & Amman (2007) could be included in accordance with some apparently very elastic "IPCC Rules",

[see http://hro001.wordpress.com/2010/05/24/ipcc-monopoly-gatekeeper-rules-very-elastic/ for example]

they completely ignored/overlooked the more serious (and more frequently commented) objection:

"The statement: McIntyre and McKitrick (2003) reported that they were unable to replicate the results of Mann, et al. is a misrepresentation"

http://hro001.wordpress.com/2010/01/18/the-climate-change-game-monopoly-the-ipcc-version/

I'm only at p. 86 ... but my overall impression thus far is that I'm reminded of the response of over-protective, over-bearing parents summoned to the Principal's office because the behaviour of their bullying little brat was causing a lot of difficulty in the schoolyard:

"Oh, but all children act this way, and you shouldn't pick on our little Johnnie because whatever else he might have done could not possibly have been his fault".

hro001 said...

Ooops ... hit Post too soon ... further to my observations re Ch 6, I would note that if any of the Muir Russell team took the time and trouble to read Montford's "The Hockey Stick Illusion" they must have done so with eyes wide shut.

Alternatively they have chosen to hide behind the shield of: "oh, but it's not peer-reviewed"

If the latter, it doesn't say much for their understanding of the shortcomings of "peer-review" - or for their supposed recognition of "The Changing Forum for Debate and the Blogosphere".

eric144 said...

In layman's terms, they were guilty of manipulation of data and the peer review process, but most of all, of extreme bias and collusion.

One good thing came out of it. When Jones' colleagues and the Guardian hung him out to dry, he spilled the beans on global warming to the BBC. I suspect Jones set the questions himself.

Do you agree that from 1995 to the present there has been no statistically-significant global warming

Yes, but only just. I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level. The positive trend is quite close to the significance level. Achieving statistical significance in scientific terms is much more likely for longer periods, and much less likely for shorter periods.

C - Do you agree that from January 2002 to the present there has been statistically significant global cooling?

No. This period is even shorter than 1995-2009. The trend this time is negative (-0.12C per decade), but this trend is not statistically significant.

Do you agree that according to the global temperature record used by the IPCC, the rates of global warming from 1860-1880, 1910-1940 and 1975-1998 were identical?

An initial point to make is that in the responses to these questions I've assumed that when you talk about the global temperature record, you mean the record that combines the estimates from land regions with those from the marine regions of the world. CRU produces the land component, with the Met Office Hadley Centre producing the marine component.

Temperature data for the period 1860-1880 are more uncertain, because of sparser coverage, than for later periods in the 20th Century. The 1860-1880 period is also only 21 years in length. As for the two periods 1910-40 and 1975-1998 the warming rates are not statistically significantly different (see numbers below).

I have also included the trend over the period 1975 to 2009, which has a very similar trend to the period 1975-1998.

So, in answer to the question, the warming rates for all 4 periods are similar and not statistically significantly different from each other.

http://news.bbc.co.uk/1/hi/sci/tech/8511670.stm

Fred said...

And we all know about how robust that "robust peer review" really is and how much non-scientific/gray literature is used by the IPCC.


Gray literature is also known as agit-prop from the greenie industry and the IPCC reprts are full of such clap-trap.

eric144 said...

The Guardian has lauded and supported the results of the Muir Russell Review and its (basic) vindication of British Ministry of Defence employee, Professor Phil Jones.

However, they seem to think that Chilcot enquiry into the Iraq war is a whitewash. Isn't that strange ?


**

Now we have a better idea of what we are not being told. The Iraq inquiry has published a list of witnesses seen in private (as it insists on describing secret hearings) during the period when things went very quiet during the election. There were a staggering 35 of them. All were, apparently, official witnesses and none appear to have been whistleblowers. A few new transcripts involving junior officials will be published, but most of what we really need to know will remain secret.


http://www.guardian.co.uk/commentisfree/libertycentral/2010/jul/08/chilcot-inquiry-iraq-secret-witness

DavidH said...

I am sorry to get to the party late, but congratulations Roger, you have identified a flaw in the Russell Report as bad as the Himalayan Glaciers and the several other mistakes in AR4. But, as been said, if you only listen to the accused and not their accusers you are not likely to come to the right conclusion.

Most people seem to confuse the IPCC with the IPCC process, the IPCC Bureau and the IPCC network and very few seem to know the Principles and Procedures.

In her presentation to the InterAcademy Council on 14 May 2010 the IPCC Secretary clearly understood what the IPCC is but thought the second draft of the IPCC Report was just an opportunity to make sure the Expert Reviewers’ comments on the first had been properly dealt with. The Chairman did no better claiming that “every stage of the drafting of our report is peer reviewed, and whatever comments we get from the peer review process are posted on the website of the IPCC, and the reasons why we accept or reject those comments are clearly specified.” Yey right. Who reviewed that WWF citations in WGII CH10? No one they were added after the review stage.

The rules are spelt out in just 20 pages of Principles and Procedures, yet Sir Muir and his team relied upon Review Editor John Mitchell to tell him whether the Wahl and Ammann paper was justifiably included in AR4 Chapter 6. Mitchell – remember he said he only did the job in a personal capacity and kept no working papers (see CA) – did not deny Wahl and Ammann missed the deadline but on page 82 of the Report says he was not aware of the issue until after the event. Well read the emails. He was the only one that did not know of the problem.

So Mitchell says the Lead Authors had every right to cite the up to date literature and by implication suggests that the many new citations added months after the second and final Expert Review stage was within the rules. Well if that was within the rules then the two citations Steve McIntyre requested under the “new guidelines” issued after Bergen (Wegman and NRC 2006) should have been used or the Lead Authors should, as the Chairman said, have given and published the reasons that they were not. They were not because WGI TSU never forwarded them to UEA. UEA released the Post Bergen reviewer’s comments in January (Keith Briffa found them on a memory stick). McIntyre’s comment, which was acknowledged by the TSU, is not there!

Now are you getting an idea of why two world class scientists might take the potentially career ending step of discussing the deletion of information that just two days previously I requested in a formal EIR request? Now why do you think that on page 125 my submission is the only one not published without a clear statement of why?

Sir Muir’s legal advice was that it was potentially defamatory. Any accusation is potentially but only actually so if untrue. It was Sir Muir’s job to establish if it was true or not. The documents speak for themselves and the individuals could be asked to account for their actions. That’s what an Enquiry is about. As it is, one of the most discreditable of the emails remains unexplained and John Mitchell has said that Wahl and Ammann was fine. End of story.

The Wahl and Ammann issue is one that UEA will fight with libel writs and my pockets are not deep enough. However, if anyone wants a confidential – not to be published or distributed – copy of the “banned” evidence, email me at crusub(at)tesco.net

David Holland

Bob Denton said...

This is a summary of Brian Hoskins, Review Editor of Ch 3, to the review panel.


"11. It is important to note however that the IPCC science process is one of assessing science,
not reviewing it. The levels of confidence and uncertainty reflected in the drafts were
based on the consensus of a group of CLAs and LAs who were chosen for their expertise
and experience in relevant fields. Irrespective of whether a paper is published in a peer
reviewed journal, it is the responsibility of the whole team to assess whether a paper’s
conclusions are robust and justify its arguments carrying weight in the assessment.
These decisions for each draft were taken in plenary sessions of the whole team. It is
inconceivable that a paper making significant claims relevant to the work of IPCC and
the Chapter 3 team, would not be considered by the team as a whole."

He, clearly, thinks the IPCC process is one of assessment not review.

Gordon said...

In contrast to IPCC, the Yucca Mountain Project required that every statement of fact come from original sources, all of which were cataloged and housed in the project library. To verify, YMP employed many scientists in the role of checkers. These men and women dredged up every paper or document cited by every author and verified their authenticity and that the authors had correctly interpreted and utilized the works. Result? In one fell swoop, Harry Reid and President Obama, on the basis of pure politics, killed YMP. We spent $11 Billion over 30 years to study this "postage stamp" with rigor, and political ideologues killed it. On the other hand, we have spent only about 3x this amount over 20 years to study the entire planet and its interactions with the universe, which is a bit larger and more complex than a 1,400 acre patch of mountain in Nevada. The climate work, as we see again here, is done with limited rigor, not remotely approaching the level of YMP, yet the politicians and AGW promoters wholeheartedly support it and prophesy a coming apocalypse. Before the developed world jumps over the brink (with $Trillions at stake), I demand that more than one, independent and well-funded "YMP" spring up to check and verify every shred of information, conduct their own, ongoing literature analyses and interpretations and build new, independent GCMs "from scratch." Any climate researcher that would disagree with this idea must suffer from severe myopia.

Post a Comment

Note: Only a member of this blog may post a comment.