11 August 2011

How Many Findings of the IPCC AR4 WG I are Incorrect? Answer: 28%

I suspect that headline will raise some eyebrows.

In a paper just out in Climatic Change today Rachael Jonassen and I perform a quantitative analysis of all 2,744 findings found in the three 2007 assessment reports of the Intergovernmental Panel on Climate Change.  Here is the abstract of our paper:
Jonassen, R. and R. Pielke, Jr., 2011. Improving conveyance of uncertainties in the findings of the IPCC, Climatic Change, 9 August, 0165-0009:1-9, http://dx.doi.org/10.1007/s10584-011-0185-7.

Abstract Authors of the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) received guidance on reporting understanding, certainty and/or confidence in findings using a common language, to better communicate with decision makers. However, a review of the IPCC conducted by the InterAcademy Council (2010) found that “the guidance was not consistently followed in AR4, leading to unnecessary errors . . . the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented.” Our comprehensive and quantitative analysis of findings and associated uncertainty in the AR4 supports the IAC findings and suggests opportunities for improvement in future assessments.
The paper characterizes the various findings of the report in terms of the uncertainty guidance used by the IPCC.  The paper includes various summary statistics and discussion.

The answer to the provocative title of this post is found in the following part of the paper:
If we confine our attention to those findings that refer to the future, one can ask how many IPCC findings can be expected to become verified ultimately as being accurate? For example, if we consider findings that refer to future events with likelihood in the ‘likely’ class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be appropriate to conclude that as many as a third can be expected to not occur. More generally, of the 360 findings reported in the full text of WG1 across all likelihood categories and presented with associated measures of likelihood (i.e., those summarized in Table 2 below), then based on the judgments of likelihood associated with each statement we should logically expect that about 100 of these findings (~28%) will at some point be overturned. 
A footnote to this paragraph explains: "This calculation assumes that each finding can be treated independently. If the findings are not independent (e.g., they are cumulative) then this calculation would result in a higher estimate."  Since this is just mathematics following from the IPCC uncertainty guidance, it should be obvious, but appears never to have been actually calculated.

What does it mean?  Nothing too interesting, really -- science evolves and any assessment is a snapshot of knowledge in time. However, I suspect that some people will get excited or defensive to learn that by the IPCC's own logic, the report's future-looking findings could include 28% or more that will not stand the test of time. Of course, such excitement and defense are part of the context that the IPCC and its critics have together created, which has led to incentives to hold the IPCC up as some sort of sacred text or to denigrate it as a sham. Our work suggests neither. Instead, from the perspective of its assessment products it is a valuable if imperfect organization.

Our paper concludes:
Although the IPCC has made enormous contributions and set an important example for global assessment of a vexing problem of immense ramifications, there remain clear opportunities for improvement in documenting findings and specifying uncertainties. We recommend more care in the definition and determination of uncertainty, more clarity in identifying and presenting findings and a more systematic approach in the entire process, especially from assessment to assessment. We also suggest an independent, dedicated group to monitor the process, evaluate findings as they are presented and track their fate. This would include tracking the relationship of findings and attendant uncertainties that pass up the hierarchy of documents within AR5. Strict rules for expressing uncertainty in findings that are derived from (possibly multiple) other findings are needed (see, e.g., the second example in the Supplementary Material).

It is not the purpose of this note to discuss other, related scientific assessments of climate change knowledge; but, we do note that our preliminary analysis of the U.S. Global Change Research Program Synthesis and Assessment Products suggests a far less systematic application of the guidance supplied to authors of those documents and far less consistent application of the defined terms. We believe that the concerns we have expressed here, and the resulting recommendations, apply more broadly than the IPCC process.
You can find the full text here.  Our full dataset of IPCC findings is online here.  Comments welcomed!

52 comments:

James Annan said...

Your headline is of course utter nonsense.

If I say (correctly) that a single roll of a fair die is "likely" (83%) to turn up 1-5, then throwing a 6 does not make my statement "incorrect". If I repeat the statement (and experiment) 100 times, then 16% of my statements will not have been incorrect either.

Roger Pielke, Jr. said...

-1-James Annan

Thanks for the comment. If the IPCC statements were in reference to die rolls you'd be correct. But they are not.

A finding of the IPCC such as the following will either be right (it will verify) or it will be wrong (it will not), the IPCC handicaps it at 90% certain:

"It is very likely that hot extremes, heat waves and heavy precipitation events will continue to become more frequent."

There is a 10% chance, according to the IPCC, that the statement will not verify.

There are not 100 die rolls here, just one future.

pjb253 said...

There are some relevant statistics [here]

James Annan said...

Why does your analysis not apply to a roll of a die? If it helps you, assume that after the first throw, a magpie swoops down and steals it. The throw is not repeatable. Would you claim I had been incorrect if I had thrown a 6?

The statements you analyse are still probabilistic statements irrespective of repeatability, and as such can be neither absolutely verified nor refuted in isolation by the future. If 10% of their very likely statements turn out to not occur, this would simply mean their assessment was well calibrated.

Therefore your usage of "incorrect" (which interesting does not appear in the paper, only the blog post) is nonsensical.

I think you know this and are playing at rabble-rousing. My eyebrows are raised, but only a little, at this behaviour.

Roger Pielke, Jr. said...

-4-James Annan

If I ask you to predict the next roll of a die and you predict that it will be in the range 1-5, an it turns up 6, then you would be wrong.

You could hem and haw about 100 rolls and such, but I would explain that I was not asking for a discourse on probabilities but a simple prediction of the next roll.

Now we may both wish that the IPCC was more sophisticated in its discussion of probabilities and futures, and that is a subject of our paper.

Operational weather forecasters will no doubt understand this situation quite well. To suggest that there is no such thing as being incorrect, probably does not help the credibility of climate science.

Thanks again!

James Annan said...

But if, rather than making a definitive prediction that it *will* be in the range 1-5, I merely state that this is "likely" to occur, will you still say I was wrong if it comes up a 6?

It seems to me that the IPCC was rather more sophisticated in its usage of probability, than you are in understanding it.

Roger Pielke, Jr. said...

-6-James Annan

Yes, a good question.

Perhaps there is some insight to be had from looking at how weather forecasters verify probabilistic forecasts.

Answer: Empirically. Se e.g.,

http://www.swpc.noaa.gov/forecast_verification/Assets/Bibliography/i1520-0493-119-07-1590.pdf

Do you have any evidence to suggest that the IPCC was not making statements about the actual climatic future that would be realized?

That would be news indeed.

James Annan said...

I note that you ducked my question about the roll of a die. I would still appreciate an answer to it. Would you claim I had been incorrect if I had made that probabilistic statement, and yet rolled a 6?

Richard Tol said...

Roger and James are both right. Roger is right if one assumes that the IPCC predicts events, James is right if one assumes that the IPCC predicts probability density functions. Having been involved since 1992, I don't know what the IPCC predicts.

Like James, I had hoped for a different paper after reading the abstract and the title. The current paper is useful as an exercise in applied probability, but it does not validate (or otherwise) the IPCC as a model to predict the future.

Reading some of the "predictions", I am struck how trite many are. "There will be impacts." I would say "certain" (forget about the "virtual").

Here's a prediction: The Jonassen and Pielke paper will be remembered for its innovative use of Semantic Wiki (high confidence, low agreement, no evidence).

Frontiers of Faith and Science said...

James,
You are proposing the equivalent of "the dog ate my homework" in a very novel and interesting way.
In 2009 the typical AGW attack on skeptics would include something to the effect that, "The IPCC is the gold standard of climate science", and tell the skeptics (denialist) to come up with something better before daring to challenge the IPCC.
It seems to me that James is simply using a more convoluted defense of the IPCC that still boils down to the circular reasoning that since the IPCC said it, that settles it.
Sort of like the fundamentalist assertion that "The Bible says it, and that settles it".
But the magpie eating the dice is a novel one.
Thanks for an early morning chuckle.
Roger,
Your continued defense of the IPCC along the lines of "X is wrong but the science is sound", or "Y person is corrupt but their work is to be trusted", and now, "The IPCC is flawed but they are an important example" is not going to work out well.
The IPCC will be used as an example of what not to do by the time this winds up.

Matt said...

The probabilities themselves are basically untestable, as far as whether they reflect reality, vis a vis the "real" PDF, unless there is a 1 or a 0 in there somewhere.

The typical probabilistic statement is probably too vague to really be able to pin it down and decide what it really means. As always, the real sticking point comes down to the assumptions behind the probabilities.

Of course, there's also the fact that (from what I've seen) many of the probabilities are about parameters, not actual data, so the predictions probably don't mean what most people will think they mean.

Roger Pielke, Jr. said...

-8-James Annan

On your question about a die, it is a bad analogy.

A die has a known PDF, So if you say, something like "a die has 6 sides with each side having equal probability of turning up" that is by definition a correct statement. If you instead say "I'll give you 5 to 1 odds that the next roll is not a six" then that is a statement that can be verified by experience, and money will change hands.

Of course, with respect to the IPCC PDFs are not known, and the report is written about the climate future, of which we will have only one.

Following your logic the IPCC can never be evaluated empirically about anything expressed probabilistically. This is silly. Probabilistic statements are made to add information to decision making, not to subtract from accountability.

If a weather forecaster offers an 80% chance of rain the next day and does so for 360 days, then I can evaluate that forecast by converting it into a deterministic prediction that should verify 80% of the time, if it is well calibrated. Similarly, if the IPCC offers up 360 statements expressed with various certainties, then (if its judgments are well-calibrated) we can expect that the scientific community will collectively judge 28% of those statements to be incorrect in the future.

Roger Pielke, Jr. said...

-9-Richard Tol

"Having been involved since 1992, I don't know what the IPCC predicts."

Oh come now, the IPCC predicts the climate future conditional upon various scenarios.

Buck said...

Why the IPCC don't simply use error bars in their charts is a mystery to me. It is good that they have guidance on how to express uncertainty. It's better that someone keeps them on their toes when they fall from that guidance.

I understand why the paper treats findings in the aggregate and yet I'm unconvinced that the aggregate is as important as the individual findings that are (likely) wrong. An example might serve to illustrate better than this abstract statement.

If we have 3 findings:
1) Sea level will rise 1 metre
2) Wheat production will decrease 5%
3) Orchid production will decrease 5%

And we are told that 33% of these findings are wrong I'm not sure I'd be too worried if it turned out that their orchid finding was it.

Still, studying the IPCC findings' language is very useful, even in the aggregate. Thank you! Hopefully, the next round of papers to come out of the IPCC will be clearer yet thanks to the work of people like you.

Richard Tol said...

-12 Roger
The IPCC (conditionally) predicts a wide variety of things: events, PDFs of events, convolutions of events, tendencies, qualities; and freely mixes statements about the real world with statement about our knowledge of the real world and the state of research to improve such knowledge.

Here are selected "predictions":
Nature-based tourism is one of the booming industries in Asia, especially ski resorts, beach resorts and ecotourist destinations which are likely vulnerable to climate change.

Sea-level rise, warming sea temperatures and extreme weather events are likely to have impacts on the regions' islands and coasts which attract considerable number of visitors from countries such as Japan and Taiwan.

Toxic algal blooms are likely to become more frequent and to last longer due to climate change. They can pose a threat to human health, for both recreation and consumptive water use, and can kill fish and livestock.

Tourism along the Mediterranean is likely to decrease in summer and increase in spring and autumn. Winter tourism in mountain regions is anticipated to face reduced snow cover.

Recreation preferences are likely to change (more outdoor activity in the north, less in the south).

The proportion of the population over 65 years of age in the EU15 is expected to increase from 16% in 2000 to 23% in 2030, which will likely affect vulnerability in recreational and health aspects.

What does it mean to say that these things are likely? These are not predictions like "the sun will rise tomorrow at 5:46 am", "the global mean temperature will rise by 3K if the atmospheric concentration of carbon dioxide doubles", or "the annual probability of a class 5 hurricanes in the Caribbean will increase by 10% in 2050".

Many of the IPCC statements are too vague to know what they mean even before you attach a likelihood to them.

Roger Pielke, Jr. said...

Over at James Annan's blog he provides a telling statement:

"The obvious elephant in the room that Roger cannot bring himself to acknowledge is that the statement is correct irrespective of the outcome of the roll."
http://julesandjames.blogspot.com/

In other words, by equating IPCC predictions with rolls of a die, he is implying that whatever happens in the real world, the IPCC is correct, about everything.

Infallibility doesn't even work for the Pope.

Roger Pielke, Jr. said...

-15-Richard Tol

Agreed, but this is why the 28% number was calculated with respect to WG I, which are far more conventional predictions.

Matt said...

-14- Roger,

He would have been more accurate to have said, "the statement may be correct irrespective of the outcome of the roll."

You can still argue that such and such a prediction, given what we knew as of AR4 was X% likely to come true. So whether or not it comes out correctly, we can't really say that whether X was a good predictor or not, especially for these one time events.

I would say that you could take a look, after the fact, at how many of these predictions were accurate against the predicted confidence as expressed in the probabilities. If the numbers are close, then we'd probably say that they had reasonable skill at predicting the confidence in their outcomes.

Richard Tol said...

-17 Roger
I take my words back then.

Roger Pielke, Jr. said...

-18-Matt

Thanks, yes that statement would have been better.

But what seems to be missed here is that the IPCC is not engaged in rolling dice, but in making scientific judgements. And it will be fairly easy to assess whether certain judgements are still held or they change over time.

Hector M. said...

Actually, the IPCC offers PROJECTIONS, not predictions. Projections are the result of running models based on a certain scenario, or "storyline", up to a certain point in the future: if the storyline turns up to be correct, AND the model actually embodies the climatic implications of the storyline, then the projection would agree with observation. Logically this does not imply that the storyline is correct, and that's the reason why several such storylines (scenarios) are used. To the uncertain storyline one has to add the uncertainties in the models themselves: they could be structurally wrong, or some parameter of the model may have been wrongly quantified (e.g. climate sensitivity).

Probabilities refer always to collections of events, not to specific events. They make statements about the relative frequency of an outcome among the various possible outcomes, over a large number of observations. They can be falsified by collection of events, not by individual events. The 1/6 probability of getting a six in one throw of a die is not falsified by the next throw not showing a six, or even by several throws failing to show a six. Twenty or fifty (or more) throws without a six are definitely possible. It will not be confirmed by getting a six in the next throw. But a large number of throws (by the Law of Large Numbers) will show 1/6 of the cases were sixes. This hardly applies to climate projections, because only ONE future would be actually observable, and that will happen in the future and is thus no help to evaluate the projections in the present.
Most of the "likely" or "very likely" assessments in the IPCC reports are based on ensembles of models, each with its particular quirks and peculiar quantification of parameters. If 66% of those model runs show a certain result (e.g. average temperature rise above 2°) this is not translatable into 66% probability of such warming to occur: other scenarios and models, or other quantifications of the models, would yield different results.

One must also distinguish probability and strength of belief, though in some context (especially those using Bayesian approaches) the two are often not distinguished. If a die fails to show a six after fifty throws, our belief in the fairness of the die starts to falter, but perhaps the die is fair after all, and we are also facing a long succession of no-sixes which is entirely compatible with the die being fair.

The likelihood attributed to future events in IPCC reports are more akin to strength of belief (among IPCC authors) than to probabilities.

Hank Roberts said...

>> I don't know what the IPCC predicts. -- Tol
> the IPCC predicts the climate future -- Pielke

Citation needed. There's no single "IPCC" statement I can find that even confirms that "the IPCC predicts" -- which you both take for granted.

Searching: http://www.google.com/search?q=site%3Aipcc.ch+"the+IPCC+predicts"

Weeeelllll, isn't that an interesting result?
It points to just a single page, and you should look at the page to see what it really says.

Try the IPCC's own search tool:
http://www.ipcc.ch/sphider/search.php?query=the+IPCC+predicts&type=phrase&results=20&search=1

Even more interesting, eh?

So _what_are_you_talking_about??

Citations please? You're professionals.

As an interested lay reader, I look for what a paper's authors give as their definitions.

Where papers are pulled together in a summary, as from the IPCC's working groups (or any edited work collecting other work), I look to the editors.

There's no single top editor for the IPCC.

Here's one set of definitions, not the only set:

http://www.ipcc.ch/ipccreports/tar/wg2/index.php?idp=125
-----------
..."projection" is used in two senses in this chapter. In general usage, a projection can be regarded as any description of the future and the pathway leading to it. However, a more specific interpretation was attached to the term "climate projection" throughout the Second Assessment Report (SAR) to refer to model-derived estimates of future climate.

Forecast/Prediction. When a projection is branded "most likely," it becomes a forecast or prediction...."
-----------

NOTE: it says "in this chapter"

Each chapter may differ.

The definitions are among the details.
Have either of you compiled all of them, or found a compilation in one of the IPCC reports?

Pointer please. Why do you both state that "the IPCC predicts" above?

Roger Pielke, Jr. said...

-22-Hank Roberts

You can find the answer to your question in the IPCC glossary.
http://www.ipcc.ch/pdf/glossary/tar-ipcc-terms-en.pdf

Here are entries for climate prediction and climate projection:

"A climate prediction or climate forecast is the result of an attempt to produce a most likely description or estimate of the actual evolution of the climate in the future (e.g., at seasonal, interannual, or long-term time-scales)."

"A projection of the response of the climate system to emission or concentration scenarios of greenhouse gases and aerosols, or radiative forcing scenarios, often based upon simulations by climate models. Climate projections are distinguished from climate predictions in order to emphasize that climate projections depend upon the emission/concentration/radiative forcing scenario used, which are based on assumptions, concerning, for example, future socio-economic and technological developments that may or may not be realized, and are therefore subject to substantial uncertainty."

A climate projection is a conditional climate prediction.

Hank Roberts said...

> A climate projection is a conditional
> climate prediction.

Wait, don't you have that backwards?

As I read it -- as an amateur reader,
trying to get the terms right:

Projections are based on simulations, repeated runs.

Projections fix one thing -- e.g. "business as usual" -- where the big CO2 control knob is set -- and hold that constant.

They then run multiple scenarios -- projections -- and the outputs of the model runs vary because the other conditions (albedo, humidity, rainfall, ocean currents, ice melting) vary over time as the model runs -- generating a bundle of projections. Each time the model is run, those various conditions interact somewhat differently.

Of those many projections, the IPCC authors may pick one as their prediction/forecast for what happens under, e.g., BAU.

So the prediction is a conditional climate projection.

Roger Pielke, Jr. said...

-24-Hank Roberts

In IPCC parlance a projection is based upon an emissions scenario, which is itself dependent upon assumptions about population, economy, technology. The IPCC SRES went to some length to explain that its scenarios were considered equal probability.

Here is the IPCC's general definition of "projection":

"A projection is a potential future evolution of a quantity or set of quantities, often computed with the aid of a model. Projections are distinguished from “predictions” in order to
emphasize that projections involve assumptions concerning, for example, future socio-economic and technological developments that may or may not be realized, and are therefore subject to substantial uncertainty."

I have always thought that Joel Cohen's description of "conditional predictions" is exceptionally clear:

http://books.google.com/books?id=TDe9Vp0dNUgC&pg=PA14&lpg=PA14&dq=joel+cohen+%22conditional+prediction%22&source=bl&ots=e7wG9hh2VM&sig=36ZU83Kkv7pverflsk3z7mBzho4&hl=en&ei=QGpFTtOQB6KJsQKEqLmSCA&sa=X&oi=book_result&ct=result&resnum=1&ved=0CBkQ6AEwAA#v=onepage&q&f=false

Your discussion in #24 raises the notion of ensemble prediction, which is a different issue.

DeWitt said...

Roger Pielke, Jr.,
"The IPCC SRES went to some length to explain that its scenarios were considered equal probability."

When I look at those scenarios closely, particularly on fossil fuel production, it's pretty clear that the probability is vanishingly small.

Hank Roberts said...

> ensemble prediction

Hmmmm, who would know about that?

http://www.google.com/search?q=site%3Aipcc.ch+"AR4"+%2B"ensemble+prediction"

Hank Roberts said...

> Here is the IPCC's general definition of "projection":

Are you quoting from here (AR4 WG1)? Or somewhere else?

http://www.ipcc.ch/publications_and_data/ar4/wg1/en/annexessglossary-p-z.html

Roger Pielke, Jr. said...

IPCC Glossary here:

http://www.ipcc.ch/publications_and_data/publications_and_data_glossary.shtml

Roger Pielke, Jr. said...

From Judy Curry's blog post on this paper, here is an excellent comment:

Fred Moolten | August 12, 2011 at 2:49 pm | Reply

I thought the exchange between Annan and Pielke Jr was amusing, because Pielke’s headline – 28% of the WG1 findings are wrong – was tongue in cheek. He doesn’t really believe that (as his text itself reveals), but was just being provocative. In a broader sense, the argument reminds us that “probability” is not an inherent property of a system but a measurement of our ignorance about the system. In the case of climate, as we learn more about particular phenomena, the probability of their occurrence will change, even though our learning process hasn’t had any impact on how the natural world behaves. A future event that is 72% probable now may become 99% probable in the future (or 1% probable) even if we have done nothing to change its behavior.

(A weather prediction is a familiar example. The forecast may predict rain two days from now as 75% probable. Tomorrow it may become 40% probable although the forecaster hasn’t changed the weather but only the forecast.)

Roger Pielke, Jr. said...

Link:

http://judithcurry.com/2011/08/12/two-new-uncertainty-papers/#comment-97878

Alex Harvey said...

I am struck by how happy James Annan is with the IPCC's so-called "predictions" or "projections" apparently being - like the six-sided die comparison - completely unfalsifiable.

Alex Harvey

Sam said...

Probabilities used in projections are “a measure of the degree to which a particular level of future climate change is consistent with the information used in the analysis” (UKCP09 2009) and thus the outcome is highly conditional on the methodology and statistical methods used to derive them as well as assumptions underlying then (Hall 2007) - e.g. emissions scenarios which ascribe not likelihood to different pathways.

Projections are closer in their nature and function to scenarios than predictions. Bray and von Storch 2009, outline this well.

ticobas said...

It is ironic that an author contributing to a paper that concludes by saying, among other things, that "there remain clear opportunities for improvement in documenting findings" decides for such a scientifically unwarranted, provocative headline in the knowledge that it may well be, and indeed has been, used by climate change denial propagandists happily ignoring what the paper actually said.

ticobas said...

-32- Alex Harvey

The implicit assumption of throwing a six-sided (fair) dice is that each side has the same probability of coming up. That obviously can't be verified by throwing the dice once, only by throwing it a large number of times. Likewise, individual IPCC projections are not falsifiable but by pooling those within the same likelihood category and time frame and comparing realised outcomes (i.e. verification / falsification can not happen until time of projection has arrived) against what was projected as 'likely' or 'most likely' to happen, one can judge the skill of the projections. Simplifying things a little: consider 100 projections referring to conditions in 2030s with a likelihood in the >66% class. If approximately 66 or more out of these 100 projections came to pass, then it would be justified to call the set of projections as a whole verified.

Roger Pielke, Jr. said...

-34-ticobas

Thanks for the comment ... so you are concerned that my admittedly tongue-in-cheek headline will give some sort of points (or whatever metric you use) to the "denial proagandists" in the great and meaningless war between so-called deniers and so-called alarmists?

Interesting. This post seems to have the added and unexpected benefit of smoking out the high strung ;-)

What did you think of the paper?

Roger Pielke, Jr. said...

-35-ticobas

You write: "If approximately 66 or more out of these 100 projections came to pass, then it would be justified to call the set of projections as a whole verified."

Not exactly. In a case that you describe the set of projects would be "well calibrated" not "well verified".

Thanks.

ticobas said...

-34-Roger

I have just gone through the paper (thanks for sharing, my University does not hold this journal). It's a fairly interesting paper that I think AR5 authors would benefit from reading. For me the most insightful section is the paragraph on pg. 8/9 ["The modal class of likelihood .. hinder decision making."] So far for the positive part.
I thought it funny that this blog-post is based on the one single paragraph [pg. 7 "If we confine .. be overturned."] that, if I was asked to review this paper as it currently is, would stand between revision and accepting for publication.

For clarity the relevant paragraph in full here:

"If we confine our attention to those findings that refer to the future, one can ask how
many IPCC findings can be expected to become verified ultimately as being accurate? For
example, if we consider findings that refer to future events with likelihood in the ‘likely’
class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be
appropriate to conclude that as many as a third can be expected to not occur. More generally, of the 360 findings reported in the full text of WG1 across all likelihood categories and presented with associated measures of likelihood (i.e., those summarized in Table 2 below), then based on the judgments of likelihood associated with each statement we should logically expect that about 100 of these findings (~28%) will at some point be
overturned."

1 I object against the use of 'as many as' in the second sentence as this constitutes an implicit exaggeration; it omits to make clear that 'a third' is actually the upper limit ('likely' represents >66%).
2 The phrasing of the second and third sentences reflect the misunderstanding inherent in the headline of this blog post and in the following discussion with James Annan in the comments. I think what is meant to say in the second sentence is: "For example, if we consider findings that refer to future events with likelihood in the ‘likely’ class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be appropriate to conclude that the realised outcome for up to a third of the projections was unlike the projected scenario considered 'likely' (>66% likelihood)."
To which any (climate) scientist / meteorologist would (should) say ... 'doh' as, it is re-phrasing what should be (blatantly) obvious anyway.
3 It is important to emphasize that a 'finding' comprises the full range of possible outcomes for a given event / condition: i.e. that what is 'likely' as well as that what is 'not likely' to occur. What it therefore categorically does NOT mean is that a third of the findings (or 28% altogether) is 'incorrect' or 'will at some point be overturned'. On that basis I would also recommend rewriting the second part of the third sentence

Final point, although you do not go into this in the paper, I presume one of the reasons you urge IPCC authors to be more clear, consistent and cautious about statements relating to (un)certainty, is that equivocal use of certainties relating to future conditions / events could be potentially confusing and misleading to policy makers, right? You imply that the headline of this blog post was intended to be tongue-in-cheek but my impression is that is part of 'Operation Damage Control', presumably after the exchange with James Annan. The main reason I'm not convinced is because the basis for the misunderstanding can actually be found in the paper as I elaborated on above. However, if I am mistaken on that and you truly intended it to be tongue in cheek then I would still counter that a blog post title containing a categorical falsehood asserted with a high degree of certainty on a Professor's blog equally has a lot of potential to mislead.

Roger Pielke, Jr. said...

-38-ticobas

Thanks ... a few replies

1."as many as a third ..." is the direct English translation of ">66%"

2. Huh?

3. Disagree. Though I suspect the nature of our disagreement lies in the imprecision of the IPCC's uncertainty guidance rather than different understandings of the math of probability.

This is quite similar to confusion in day-to-day weather forecasts. If your weather forecaster predicts that it is likely to rain tomorrow in your region do you know what that means? 66% of the region will experience rain with 100% probability? Every point has a 66% chance? Some points have a 90% chance others a 40% chance and the areal average is 66%? Or something else? Alan Murphy had a great paper on misunderstandings of probability forecasts.

4. Perhaps you missed this part of the blog post, which I will repeat:

"What does it mean? Nothing too interesting, really -- science evolves and any assessment is a snapshot of knowledge in time. However, I suspect that some people will get excited or defensive to learn that by the IPCC's own logic, the report's future-looking findings could include 28% or more that will not stand the test of time."

Arguing about blog post headlines in somewhat less interesting than arguing about newspaper headlines which is itself somewhat less interesting than arguing about angels on the head of a pin;-)

Just to be clear about terminology though:

If a finding is that, just to invent an example, "There is a 90% chance of more hot days" and future science actually leads us to that 10% side where future expectations are actually of constant or less hot days, then this finding, obviously, will not survive the test of time. Future assessments will say something different. For the finding to actually survive the test of time then the only thing that would change is the expressed uncertainty, e.g., "There is a 99% chance of more hot days"

Hope this makes more sense, thanks.

bernie said...

ticobas:
You argue above(#35):
"The implicit assumption of throwing a six-sided (fair) dice is that each side has the same probability of coming up. That obviously can't be verified by throwing the dice once, only by throwing it a large number of times. "
Is this really true? Are there not other ways of determining that a six-sided dice is fair? Surely there are. Just as you do not have to roll a dice to determine whether it is likely to be fair or not. The reason is that we have a physical model that determines whether or not the die is physically symmetrical or not. The problem as I see it with this discussion is that the probabilities in the IPCC report are based on very incomplete or potentially inaccurate physical models. Isn't this what is behind the Ernest Rutherford quote: "If your experiment needs statistics, then you ought to have designed a better experiment."

ticobas said...

-39- Roger

1 re: "as many as a third ..." is the direct English translation of ">66%".

No. You may want to check it out here: http://bit.ly/pah8LU (pg. 146, Unit 73C).
'Up to' should have been used instead.

2/3 From the 'Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties':

A statement that an outcome is “likely” means that the probability of this outcome can range from ≥66% (fuzzy boundaries implied) to 100% probability. This implies that all alternative outcomes are “unlikely” (0-33% probability).

It is conceivable that out of many different events, for each of which one particular outcome was classed as "likely", up to a third of the realised outcomes actually comprise what were previously considered 'alternative outcomes'. That does not mean that a third of the findings (or 28% altogether) is 'incorrect', 'will at some point be overturned' or 'will not stand the test of time'. This really goes back to James Annan's point on verifying probabilistic statements in comments 1,4 and 6:

"If I say (correctly) that a single roll of a fair die is "likely" (83%) to turn up 1-5, then throwing a 6 does not make my statement "incorrect". If I repeat the statement (and experiment) 100 times, then 16% of my statements will not have been incorrect either."

Your response to that was:

"A finding of the IPCC such as the following will either be right (it will verify) or it will be wrong (it will not), the IPCC handicaps it at 90% certain:

'It is very likely that hot extremes, heat waves and heavy precipitation events will continue to become more frequent.'

There is a 10% chance, according to the IPCC, that the statement will not verify."

You are mixing up the probabilistic statement with the realised outcome. If hot extremes, heat waves and heavy precipitation events indeed become more frequent then it does not necessarily follow that the IPCC's statement
'It is very likely that hot extremes, heat waves and heavy precipitation events will continue to become more frequent.' has verified. Only multiple future realities (as in throwing a dice multiple times) would be able to verify a probabilistic statement like this. I'm sorry but if you don't understand this you have to read/ponder this again until you do. Likewise, if hot extremes, heat waves and heavy precipitation events did not become more frequent then it does not follow that the IPCC's statement 'It is very likely that hot extremes, heat waves and heavy precipitation events will continue to become more frequent.' has not verified, just that what was prior deemed the 'unlikely' outcome became reality.

4 I did pick up on that. I happen to take the view that it is entirely justified and relevant to question the choice of headline, particularly in this situation. Copy editors for tabloids choose sensationalist headlines, presumably, because they boost sales. In academia, accuracy and representation are valued over sensationalism. The striking contrast between the 'nothing-too-interesting' conclusion in the paper and the provocative headline makes me wonder about your motive(s) for choosing the headline even more.
You state in the abstract that "... the impression was often left, quite incorrectly, that a substantive finding was being presented.” Don't you think it is testimony of double standards that you write a blog post with a title that does exactly what you criticise IPCC authors of doing, particularly when considering that some parts of the blogosphere are all too eager to run with a headline like this for political purposes, not bothered so much about nuance expressed further on in 'the small print' of the blog, let alone in the paper?

ticobas said...

-40- bernie

"Are there not other ways of determining that a six-sided dice is fair?"
No. When we talk about whether a dice "is" fair that relates to each side having the same probability of coming up. Of course there will be a range of attributes of die that can be controlled that would make it 'virtually certain' (note, probabilistic!) that a die is fair. However, a true determination of fairness requires actually throwing the dice, then applying a statistical test such as the Chi Square test to see if the distribution significantly differs from the expected theoretical distribution under the null hypothesis.
http://statisticshelp.com/one-way-tables

"The problem as I see it with this discussion is that the probabilities in the IPCC report are based on very incomplete or potentially inaccurate physical models."

The IPCC recognises events for which there is limited evidence and low agreement, see Table 1 here:
http://www.ipcc-wg2.gov/meetings/CGCs/Uncertainties-GN_IPCCbrochure_lo.pdf

and factors this in when assigning uncertainties for future events.
I don't know the context in which Rutherford made that statement but I think it's a dangerous strategy to use quotes as some kind of broad scientific compass. I personally attach more value to the legacy of Ronald Fisher.

Roger Pielke, Jr. said...

-41-ticobas

Thanks ... a few replies:

1. The link you provide says no results found for "as many as" -- but as this is now about semantics I plead nolo contendre

2/3 The guidance for the Fifth AR is not relevant to our paper as it is focused on the AR4:
http://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf

Don't worry, I fully understand verification of probabilistic forecasts;-) To invoke "multiple future realities" as a basis for claiming that the IPCC cannot be wrong (or logically right) is to advance a reductio ad absurdum. There will be but one future and we will compare how it evolves to what the IPCC has projected. If global temperatures cool for the next 20 years, good luck claiming that in 95 other realities temperatures went up;-)

I discuss many similar issues in this paper, just FYI:

http://sciencepolicy.colorado.edu/admin/publication_files/resource-2786-2009.47.pdf

4. On the headline, you don't like it, I hear ya, but ... it's a blog, lighten up;-)

ticobas said...

-43-

Roger, if you think that:

re: 1) not bothering to scroll down to the pg. (146) and section (Unit 73C) I indicated is a valid way of dismissing a point

re: 2/3a) casually suggesting that the quote lifted from the AR5 Guidance Note provides a specific example, not relevant to the paper, is a valid way of ignoring criticism of your claims regarding verification of probabilistic forecasts as displayed here in the comments

re: 4) as a scientist there is one set of rules when it comes to writing papers / IPCC reports and another when it comes to writing blogs

then at least you've managed to convey the message there's no reason for me to carry on commenting here.

Good luck with your research and [snark] coming up with sensationalist, factually incorrect headlines based on it for your blog posts.[/snark]

models-methods-software.com said...

" Only multiple future realities (as in throwing a dice multiple times) would be able to verify a probabilistic statement like this. "

Wow !! You can't be serious. You must be joking. WT... are you saying.

The pleading for special exemptions in Climate Science continues unabated and has now reduced The Science to nothingness. Climate Scientists have researched, developed and implemented technical evaluation criteria metrics that have no information content. None. The criteria have no useful purpose.

bernie said...

ticobas:
I believe that you have an overly narrow notion of determining a fair dice. The fairness of a dice or a coin is a characteristic of the physical properties of the object. Surely you would know a dice is not fair if it was visibly asymmetrical in some way? Moreover if you discovered that a dice was unfair you would identify the physical property of the dice that made it unfair. While throwing the dice may be a useful diagnostic test to determine its center of gravity, it is nothing more than one form of the physical examination of the symmetrical nature of the dice. I am sure that in the manufacturing of dice, attention is primarily paid to the manufacturing process so as to ensure the homogeneity of the material, the symmetry of the molds and the smoothness of the facets and edges. Rolling the dice is not necessary to determine that a dice is fair if you can adequately control the manufacturing process.
This video on how to make trick dice by changing its center of gravity, I believe, helps make my point: http://www.youtube.com/watch?v=dURhrY9Z-xg

ticobas said...

-45-

You and other commenters would do yourselves a favour by reading some elementary science textbook like for example 'Understanding philosophy of science' by James Ladyman (2002). It's available as a .pdf here: http://bit.ly/qHsVvt

Section 3.5a (page 81) deals with verification of probabilistic statements. As some people seem to have problems finding pages even when the url and the pg. number are given I will quote here in full:

---

3.5 Problems with falsificationism

There are several problems with Popper’s account of falsificationism. Some of these are specific to the details of the theory Popper first elaborated and so may be avoided by a more careful formulation or by revising some of the details. However, some are quite general and challenge the fundamental idea that it is possible to give an account of the scientific method without endorsing any kind of inductive inference. Below, some of the main criticisms of falsificationism are briefly explained.

(1) Some legitimate parts of science seem not to be falsifiable

These fall into three categories.

(a) Probabilistic statements

Science often seems to issue statements about the probability of some occurrence. For example, modern physics tells us that the half life of uranium 235 is 710,000,000 years, which means that the probability of one atom of uranium decaying in 710,000,000 years is one-half or that it is highly probable that if one starts with 1 kg of uranium then in 710,000,000 years 500 g of it will have decayed. However, such statements cannot be falsified because an experiment may produce an improbable outcome and that is consistent with the original statement – improbable things are bound to happen sometimes. Any statement about the probability of a single event is not falsifiable, so, for example, the probability that a particular coin toss will land heads is 1/2, but we cannot falsify that hypothesis by tossing the coin because the fact that the probability is 1/2 is consistent with the coin landing heads or tails on that occasion. This problem does not arise for probabilities that are defined over large populations; hence, the statement that the probability that a particular coin will land heads 50 per cent of the time during a million tosses would be considered refuted if the coin landed tails 90 per cent of the time.

---

That should clear up the key issue at stake here.

Roger Pielke, Jr. said...

-47-ticobas

You are obviously new around here. . . I suspect that you will love the "consist-with chronicles" ;-)

http://cstpr.colorado.edu/prometheus/archives/prediction_and_forecasting/#001416

Robert said...

Ticobas - As a couple of commenters touched on above (Matt in #14; Fred Moolten as quoted by Roger in #30), there is a subtle but distinct difference between probabilistic statements (e.g., odds of rolling a 1 on a die) and statements regarding our degree of confidence in some finding or prediction. The IPCC's use of likely, very likely, etc. regarding various preditions of findings relate to the degree of certainty of the prediction or finding, not on the probability of it occurring. In other words, the degree of certainty statements reflect on the confidence in our state of knowledge rather than probability of an event or finding coming to pass. Our state of knowledge will advance and so our degree of certainty will change.

Roger - Unfortunately, when I tried open the full text pdf in Adobe Reader X, I got a message that the file was corrupted and could not be recovered. Don't know if it was on my end or a problem with the source file. Also, though I think I get your point regarding quantitative uncertainty estimates and the changing state of knowledge, couldn't this paper have been done about any topic where scientist have attempted to quantify degrees of certainty and thus, is not necessarily an indictment of the IPCC process.

models-methods-software.com said...

The subject is not philosophy of science, and especially not philosophy of Climate Science. The subject is about the usefulness of the projected outcomes that have been presented. I think the subjects are verification and validation of computational science and engineering.

Recall that there will be but a single physical realization of the future state of the Earth's climate systems.

If the projected responses cannot be determined to be valid, of what use are they. If the validity of the projected responses is already known to be indeterminate, they cannot be presented to policy-makers and stakeholders as guidance for planning for future outcomes. http://tinyurl.com/3dyk2u8 or http://preview.tinyurl.com/3dyk2u8

Additionally, the procedures used to arrive at the presented numbers are such that classical statistics and probability cannot be applied to them.

While we're giving reading assignments:

http://tinyurl.com/4xl54rw

http://tinyurl.com/3vau7yb

markbahner said...

-9-Richard Tol

"Having been involved since 1992, I don't know what the IPCC predicts."

-17-Roger Pielke Jr.

"Oh come now, the IPCC predicts the climate future conditional upon various scenarios."

Ooh, ooh! This gets to something that seems to me to be very important, but is essentially never discussed. Let's take as a given that the "...IPCC predicts the climate future conditional upon various scenarios"...

...what about the problem that the IPCC makes no effort at all to make their scenarios likely to approximate future conditions?

For example, wouldn't you both agree that the the IPCC dramatically overpredicts likely sulfur dioxide emissions in the 21st century (a cooling forcing) but also dramatically overpredicts likely black carbon emissions (a warming forcing)?

So isn't it really the case that the IPCC can not be shown wrong, because they can always point to ways in which their scenarios did not match the real world?

Isn't this analogous to what has happened to James Hansen's "predictions" in 1988:

http://cstpr.colorado.edu/prometheus/archives/climate_change/000836evaluating_jim_hanse.html

?

GoRight said...

As you may know William Connolley was piling with James Annan over at his blog. I thought that you and your readers might get a chuckle out of a recent exchange I had there with Connolley concerning this thread. See the following snapshot (comments 15 and 21):

http://www.webcitation.org/614Ug189s

Note also that Connolley has subsequently deleted comment 21 presumably because he found it to be "inconvenient". :)

http://www.webcitation.org/618PZGcPS

This is the typical modus operandi of Connolley when he can't make a substantive response. He just edits the storyline to look like the inconvenient comments never existed. That's why I take the snapshots using Webcite.

Typical head in the sand behavior.

Post a Comment

Note: Only a member of this blog may post a comment.