27 May 2010

The Significance of Climate Model Agreement: A Guest Post by Ryan Meyer

Ryan Meyer is a PhD student at ASU's School of Life Sciences where he is writing a dissertation about US federal climate science research and its relationship to policy making. He is spending a yer in Australia on a Fulbright and he blogs at Adapt Already.

If four out of five global climate models (GCMs) agree on a result, should we feel more confident about that result? Does agreement among models constitute increased certainty about the models’ basis in reality? My colleagues and I wondered about this a few years ago when we started noticing that many climate scientists seem to adopt this logic without any explanation or justification. They claim, for example, that we should be more worried about drought in the southwestern US because 18 out of 19 models predict a transition to a more arid climate over the next 50 years. Or they pick a subset of models to represent a particular process such as the Asian monsoon, and then point to agreement among those models as significant.

If 18 models get the same result, is that better than just one? Why? Climate science should provide a thorough explanation for this, especially if climate models are to begin informing policy decisions.

We argue in a paper now available in Environmental Science and Policy (PDF here) that agreement is only significant if the models are sufficiently independent from one another. The climate science community has mostly ignored the crucial problem of model independence while taking advantage of a tacit belief in the force of model agreement. To quote from our introduction:
GCMs are commonly treated as independent from one another, when in fact there are many reasons to believe otherwise. The assumption of independence leads to increased confidence in the ‘‘robustness’’ of model results when multiple models agree. But GCM independence has not been evaluated by model builders and others in the climate science community. Until now the climate science literature has given only passing attention to this problem, and the field has not developed systematic approaches for assessing model independence.
To some these arguments may seem like nitpicking. Or they might seem better suited to the pages of some technical journal where modelers work these things out for themselves. But we strongly believe that this extends beyond methodology, and is in fact a policy question. It relates to the kind of investments we can and should be making in climate science.

The question of independence is one small piece of a much needed broad discussion about climate science policy. What kinds of knowledge are most helpful in crafting a response to climate change? What institutions, disciplines, and tools are best suited to deliver such knowledge? Such crucial questions of science policy tend to be ignored. We argue about what "the science" says, rather than how it works and how it could work better for the needs of decision makers.

In our paper, we take it as a given that governments will continue to fund large and complex models of the climate and related systems. (A broader discussion about the merits of this investment is important, but we do not directly address it). But how should they be funded? Who should decide the most important questions to pursue? In the past, we have tended to let climate scientists sort that one out. They are, after all, the experts. But they are certainly not unbiased participants in this discussion. Are they asking the most important questions, or just the ones they find most interesting?

It is important to recognize that there are many possible trajectories for our climate science knowledge. We may not know exactly where each one leads, but we can still make wise, informed choices. This is why the independence problem is important, not just for climate modelers, but for science policy makers, potential users of climate science, and advocates for climate change adaptation.

We have three basic recommendations related to the independence problem:
  1. Climate modelers should be wary of overselling results based on model agreement.

  2. The climate science community should begin to better address the independence problem.

  3. Science policy decision makers should take this problem into account when building strategies for climate modeling, and climate science more broadly.