Forecast Evaluation Not Wanted

Forecasting the future is fraught with difficulties.  It can be even more challenging if people pay attention to your forecasts and use them in decision making.  So it is perhaps understandable that some forecasters seek to avoid accountability for their predictions.

This is apparently the case at the UK Met Office, according to documents uncovered by the BBC that appear to show the agency taking steps to reduce the ability of the public to evaluate its forecasts.

The BBC explains that after its embarrassing 2009 prediction of a "barbeque summer" was followed by one of the wettest summers in a century,

its seasonal prediction for last winter was also awry, failing to signal sufficiently the long and severe cold spell.

An internal executive paper noted the impact as follows:

"Unfortunately, less 'intelligent' (and potentially hostile) sections of the press, competitors and politicos have been able to maintain a sustained attack on the Met Office ... The opprobrium is leaking across to areas where we have much higher skill such as in short range forecasting and climate change - our brand is coming under pressure and there is some evidence we are losing the respect of the public."

This report argued that one downside of the seasonal forecasts was that they remained on the website and could easily be later compared to reality. It said:

"One of the weaknesses of the presentation of seasonal forecasts is that they were issued with much media involvement and then remain, unchanged, on our website for extended lengths of time - making us a hostage to fortune if the public perception is that the forecast is wrong for a long time before it is updated."

In contrast it noted that the "medium range forecast (out to 15 days ahead) is updated daily on the website which means that no single forecast is ever seen as 'wrong' because long before the weather happens, the forecast has been updated many times."

The idea that forecasts, once made, leave an agency "hostage to fortune" is no doubt true, but it can also be crucially important information for the public to evaluate the reliability of future forecasts.  The criticism that the Met Office has faced in recent years results from too much credibility being placed in their forecasts.  Highlighting the true state of predictive capabilities can help decision makers to understand better the uncertainties in forecasting, even if that means low credibility for the forecasters.

I saw this exact same dynamic at work in the US NWS in 1997 when I served on the agencies Disaster Survey team following the floods in the Red River of the North (for details see this paper in PDF).  For the previous 16 years the NWS had issued a seasonal forecast for spring floods, and at Grand Forks the average error in this outlook had been 10%.  But the NWS didn't tell anyone about its forecast accuracy.  At the time NWS officials told me that they didn't advertise this information (e.g., by putting it on the web) because they didn't want the public to lose confidence in their forecasts by seeing their track record.  In 1997 the seasonal outlook at Grand Forks was for a 49 foot river crest.  The city build its defenses to 51 feet.  The flood crested at -- you can't make this up -- 54 feet, or 10% higher than the prediction and almost exactly the average error of the past forecasts.  The levees were overtopped and massive flooding resulted (the entire story is told here in PDF).

Believe it or not, but an understanding of forecasts "busts" can be a decision makers best friend.  In our book on Prediction we argued that weather forecasts are so valuable not because they are always correct, but because decision makers inevitably have so much experience with them that they understand and can calibrate their accuracy based on experience.  Perhaps the inevitability of such evaluations based on experience helps to explain why the weather community has fully embraced forecast evaluation in a way that the climate community has not.  Such experience is not possible with seasonal forecasts (or those forecasts made on longer time scales) making understanding and communication of forecast uncertainties (and areas of ignorance) all the more important.

As difficult as it might seem, forecasters should embrace open evaluation of their forecasts, even (and especially) when they go bust.  Better use of their products -- and potentially better decisions -- will result.