Hiding Uncertainties Doesn't Work

The US government commission looking back at the BP oil spill (officially the National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling) has released a working paper (PDF) that critiques the government's handling of estimates of oil flowing into the Gulf during the spill and later estimates about the fate of the oil. The report paints a damaging picture of political officials trying to shape public perceptions by hiding uncertainties in their estimates, favoring lower estimates of both the spill magnitude and oil remaining the the Gulf.

The report summarizes its findings as follows:

The federal government‟s estimates of the amount of oil flowing into and later remaining in the Gulf of Mexico in the aftermath of the Macondo well explosion were the source of significant controversy, which undermined public confidence in the federal government‟s response to the spill. By initially underestimating the amount of oil flow and then, at the end of the summer, appearing to underestimate the amount of oil remaining in the Gulf, the federal government created the impression that it was either not fully competent to handle the spill or not fully candid with the American people about the scope of the problem.

The details are remarkable.  The report explains that the government's interagency committee established to provide an estimate of the flow of oil into the Gulf (chaired by the director of the USGS who also was science advisor to the Secretary of the Interior) produced a range of estimates of the possible flow rate.  However, only the lower bound information was released to the public:

The Flow Rate Group published its first estimate on May 27, 2010, noting that “[t]he only range of flow rates that is consistent with all 3 of the methods considered by the [the Group] is 12,000 to 19,000 barrels per day. Higher flow rates [of up to 25,000 bbls/day] are consistent with the data considered by [the Plume Team].”55 The Group‟s press release contained little information as to how each of the three Flow Rate Group teams calculated those ranges, other than to note that the Plume Team‟s range of 12,000-25,000 bbls/day was “an initial lower bound estimate.”56

On June 2, 2010, the Flow Rate Group released a three page Summary Preliminary Report that explained the May 27, 2010 estimate in more detail.57 That document noted that the Plume Team produced “a range of lower bounds” of 12,000 to 25,000 bbls/day (±40%), but did not elaborate on the underlying data or calculations.58 Moreover, the June 2, 2010 report did not include the upper ranges of the Plume Team‟s estimates because “[t]he experts concluded that the effect of the unknown unknowns made it more difficult to produce a reliable upper bound on the flow rate.”59 It is the Commission staff‟s understanding that the “lower bound” range was simply a collection of the minimum estimates produced by each of the Plume Team members. A few members had also produced maximum estimates, several of which were in excess of 50,000 bbls/day, but this upper bound was not released.60 Further, the Plume Team‟s report originally contained appendices that revealed some divergence of opinion within the Team.61 The appendices were not publicly released.

When NOAA sought to release some of its analysis indicating possible worst-case scenarios (which ultimately proved far more accurate than initial government estimates), the agency was denied permission by the Office of Management and Budget, presumably at the political level:

The Commission staff has also been advised that, in late April or early May 2010, NOAA wanted to make public some of its long-term, worst-case discharge models for the Deepwater Horizon spill, and requested approval to do so from the White House‟s Office of Management and Budget.47 Staff was told that the Office of Management and Budget denied NOAA‟s request.48

The White Paper suggests that the Oil Spill Commission might want to consider recommending greater transparency in releasing scientific information, especially information about uncertainties:

The Commission may wish to consider recommendations that encourage government responders to disclose information about the scenarios under which they are operating—in this case, the operational worst-case discharge estimates. Putting aside the question of whether the public had a right to know the worst-case discharge figures, disclosure of those estimates, and explanation of their role in guiding the government effort, may have improved public confidence in the response. Instead, government officials attempted to assure the public that they were not basing operations on the official flow-rate estimates, while not stating what they were basing operations on instead. That lack of information may have contributed to public skepticism about whether the government appreciated the size of the Deepwater Horizon spill and was truly bringing all of its resources to bear. Moreover, the national response may have benefited early on from a greater sense of urgency, which public discussion of worst-case discharge figures may have generated.49

Uncertainties and plain old ignorance are facts of life.  They are real and often highly meaningful  Trying to hide them or betting that they will break your way sets the stage for bad decision making.  It is a also a surefire recipe for a loss of trust in institutions of science and government.