10 January 2012

Bad Economics at NOAA

UPDATE 1/12: See this interesting follow up.

The National Oceanic and Atmospheric Administration is a federal agency that does a lot of excellent work related to weather, climate and the oceans. In fact, it is the primary sponsor of CIRES here at the University of Colorado where I serve as a Fellow. However, NOAA has been publishing information related to disasters that is extremely misleading and scientifically inaccurate.

The graph above shows NOAA's tally of "billion dollar disasters" which NOAA defines as "the 1980-2005 events which resulted in at least $1 billion in overall damages/costs at the time of the event" (emphasis added, source here in PDF).  The bolded part of that sentence is where NOAA's methodology has a serious flaw, as $1 billion does not mean the same thing today as it did in 1980.  In fact, adjusting just for inflation means that $1 billion today would have been the equivalent of $400 million in 1980. And that is not all, because there has been considerable development across the nation since 1980, meaning that there is more property and wealth to be damaged, $1 billion in damage today is actually equivalent to about $170 million in 1980.

Events which would have caused $1 billion in damage today, but did not when they occurred are not included in the NOAA listing. So by focusing on a $1 billion threshold, as $1 billion comes to represent less and less over time, NOAA has built in a strong bias in their analysis which creates the illusion of trend. If the point of the analysis is to say something about trends in weather, it will always be better to look at weather data, not damage data. 

Think of this analogy: Imagine if you had a shopping cart and added up the number products in your basket costing more than $5. In 1980 you would not have very many, and today you'd have a lot. Would a index of "number of products costing more than $5 in a basket" tell you anything about the overall cost of food?  No.

The New York Times recently published an editorial that serves as a good example of how NOAA's data is misleading (Nature however is not fooled):
 A typical year in the United States features three or four weather disasters costing more than $1 billion. In 2009 there were nine. Last year brought a dozen, at a cost of $52 billion, making it the most extreme year for weather since accurate record keeping began in the 19th century.
To be sure, 2011 saw some very extreme events in the United States. But was it the "most extreme year for weather" since the 19th century? Not by a long shot, whether the metric is dollar damage or loss of life.

The NOAA "billion dollar" data has only 1 event from 1980 costing $1 billion, a major drought.  I have quickly compiled a list of other events that would  have certainly resulted in more than $1 billion in damage were they to occur today and a second list of events that lack detailed accounting, but would be worth a further look (note that a rigorous analysis would implement the methodology of economic normalization that we have applied in a range of contexts).
1980 Disasters Greater than $1 Billion in Normalized 2011 Dollars But Not on the NOAA List

Certain additions:

Hurricane Allen - Aug 9, 1980 - $2.0 billion
Grand Island Tornadoes - June 3, 1980 - $1.7 billion
Western Wisconsin Derecho -- July 16, 1980 - $3.8 billion
California/Arizona Floods -- February 13-21, 1980 -$2.0 billion

Other candidates for inclusion but lacking a rigorous quantitative accounting:

Hawaii storms -- January
Hampton Roads "Circus" Blizzard --  January
Texas/Alabama/Louisiana Storms -- May 
Midwest Floods -- June
Panorama Fire -- December
There are thus 4 events that clearly would have been $1 billion events had they occurred in 2011 and I can find 5 other candidates for which data is lacking, but which could possibly have reached $170 million in damage in 1980 (especially if data collection were as intensive as today).  Regardless, of whether the total is 4 or 9 missed events or somewhere in the middle, NOAA's data misses at least 80% of the billion dollar disasters in 1980. Not good. No doubt that a reanalysis of the years 1981 to present would turn up many more such events that failed to meet the contemporary billion-dollar threshold but would certainly do so today.

It is extremely misleading to use economic impacts as the basis for making claims about weather and climate. NOAA should take immediate steps to improve the scientific quality of its tabulation of "billion dollar disasters" lest it find itself accused of misleading the public and decision makers with scientifically unsound information.