29 September 2011

Selective Importance of Science Integrity Guidelines

The EPA Inspector General has found that the agency failed to comply with the Administration's requirements for peer review in the preparation of its endangerment finding with respect to carbon dioxide (report can be found here in PDF).

The IG report has no direct bearing on climate policy either in EPA or beyond. What has been interesting has been to see various statements by observers about the significance of the IG report.  I'd speculate that these observers would have had different reactions had this report been requested by Henry Waxman in 2006 about the last administration's EPA.

For instance, Andrew Dessler, a climate scientist, finds the worry about process a distraction:
This is a battle of lawyer versus lawyer. The issues here are not scientific. If we start taking scientific advice from lawyers, we are in deep trouble.
 Kevin Trenberth, another climate scientist, thinks that the concern about process is all political:
"This has nothing to do with the science that justifies the endangerment finding and everything to do with politics," Trenberth said, adding that the IG's criticisms focused only on process and not the quality of science EPA is using. "There is nothing here that undermines the EPA's way forward."
Francisca Grifo of the Union of Concerned Scientists explains that the breakdown in process is not a big deal:
The process matters, but the science matters more and in this endangerment finding, the science is accurate.
Of course, during the Bush Administration concern about processes to ensure scientific integrity were all the rage. At that time it was generally understood that process matters, not simply because it helps to improve the quality of scientific assessments, but also because it helps to establish their legitimacy in the political process.  One sneers at process at some risk.

Of course, had the EPA endangerment finding gone through a more rigorous peer review, misleading and sloppy arguments might have been identified and corrected -- such as found in this example.

18 comments:

  1. Your father submitted comments for review of the endangerment finding and was essentially blown-off by the EPA. I can't wait to see his posts on this subject.

    ReplyDelete
  2. -1-Sean

    There is no guarantee than a peer review process will work (see, e.g., IPCC on disasters) but it is almost certain that a flawed peer review process will be a political liability.

    ReplyDelete
  3. I always wondered what the IG in the IG Nobel prizes stood for...

    (ps -- they're tonight http://improbable.com/ig/)

    ReplyDelete
  4. This is not about science, per se. It is about a bureaucracy, with a granted authority, imposing progressive obstacles in the way of private enterprise.

    They pursued the process through litigation and experienced varying degrees of success and profitability. They pursued the process through popular opinion and ultimately failed. They pursued the process through legislation and failed. They are attempting to pursue the process through regulation, and should have expected a response in kind.

    Also, since when did people accept scientific outcomes without peer review? Suddenly, the notion of competing interests holding each other accountable has found disfavor.

    Selective importance, selective science, selective history... A selective reality has become tres chic. It is also counterproductive and utterly ridiculous.

    ReplyDelete
  5. People don't realize how thin is the science behind most of the EPA's scientific findings. I went through the scientific basis for the new power-plant regulations, and discovered that for many of their claims the science was incredibly threadbare. For example, for the claim of the number of asthma exacerbations that would be prevented, they were relying from two tiny and questionably relevant studies, which after processing, gave a +/-2 sigma range of something like 4000 - 400,000. A journal referee, looking at this, would say 'collect ten times more data and submit again, when your error limits are sane.'

    The CO2 stuff is actually quite well supported, by EPA standards.

    ReplyDelete
  6. The responses by Trenberth and Dessler are very insightful. But not in a good way.

    ReplyDelete
  7. " What has been interesting has been to see various statements by observers about the significance of the IG report. "

    Agreed. But that is true on both sides, is it not?

    "This report confirms that the endangerment finding, the very foundation of President Obama's job-destroying regulatory agenda, was rushed, biased, and flawed," Sen. James Inhofe (R-Okla.) said in a statement."

    The IG report is not "confirmation" of a flawed finding, but a statement about a flawed process.


    Not worthy of your comment, Roger?

    ReplyDelete
  8. -7-Joshua

    Thanks ... I did comment, but perhaps too subtlely:

    "The IG report has no direct bearing on climate policy either in EPA or beyond."

    This is a post about science policy not climate politics, a distinction I often try to make, even though most folks want to talk about the horse race.

    ReplyDelete
  9. Roger -

    The other comments you highlighted were made by scientists. But their comments are inextricably linked to the larger political climate. That context also includes the political background behind the IG report.


    For example, Trenberth's statement that it "has nothing to do with the science" is obviously too categorical. Concerns about a flawed process are not inherently political per se, but I think it is important to quantify to what degree the concerns about process are political.

    ReplyDelete
  10. -9-Joshua

    Thanks ... I don't know if you have read my commentary going back to 2004 on the "politicization of the politicization of science" but if not, I am very much of the view that nowadays most (i.e., almost all) concern expressed about process related to science policy is tactical in pursuit to some other end (Chris Mooney's arguments are the canonical example, e.g., "who is worse, my tribe or their?"). This is particularly the case related to climate, but also seen in other areas.

    To see prominent climate scientists so readily dismissive of process in the aftermath of all the trouble that the IPCC has had seems self-destructive. Inhofe could not hope for a more agreeable opposition.

    In this context, the proper response to Inhofe is not to argue substance, but rather to simply agree that the process should indeed be respected, and fight the substantive battle in another setting.

    ReplyDelete
  11. @10, Roger -

    I am trying to post a response but seem to be having trouble getting it to go through (I'm not getting the comment that my post will be moderated. Is there a length restriction to comments, or some general restriction that keeps posts from going into moderation?

    ReplyDelete
  12. -11-Joshua

    There is a length limit. Try breaking it up. That said, Blogger has been known to be a pain. If all else fails, email me and I'll post it for you, but try a series of shorter posts first. Thx!

    ReplyDelete
  13. -10, Roger-

    In case the problems was a length restriction, I will try dividing my response into two.

    ====

    I am generally aware of your position re the political overlay on the climate debate.

    I mostly agree with your last post - including the observation that the nature of tribalism in the climate debate is not unique, but can be seen as a skirmish in (and I would add is mostly a proxy for?) a larger political battle.

    But, IMO - it's a bit early to judge the long-term impact of climate scientists being dismissive of what they see as politicization of process. Would a more nuanced or strategically-moderated approach have had much greater impact given the current political and economic context? I tend to doubt that. The nature of the political opposition (to the extent that it can be distinguished from scientific opposition) indicates that inherently, no matter what form of advocacy for AGW-related policy had taken place, efforts would have been made to marginalize it as extreme, regardless; it follows from the political nature of some of the opposition.

    I think that ultimately, we will either see dramatic enough climate change that “climate change skepticism” opposition fails to garner sufficient political support, or we won’t. Until such time, the politically-driven opposition to climate related policy will, as it has in a host of ideologically linked debates, remain strong enough to prevent substantial policy modifications.

    In the event that dramatic climate changes become overwhelmingly obvious, I would guess that categorical dismissal of the political underpinnings of “skepticism” might well prove to be advantageous for those on the other side of the political divide, in that those who share the political framework of the “skeptics” will be marginalized. Of course, it follows then that the opposite will be true if dramatic climate change does not take place. The tribalism will remain either way, but it will be that much easier to marginalize depending on what takes place, likely, decades out.

    ReplyDelete
  14. -10, response part 2.

    In the meantime, does (arguably inaccurate) categorical attribution of perspective on the science to underlying political motivations make a difference along the edges of energy policy? I think that is the case, and I see such attribution as being more harmful to the “pro-consensus” side than the “anti-consensus” side, (although from what I’ve seen, such inaccurate attributions take place on both sides – I’m not sure whether it is more characteristic on one side as opposed to the other) . I certainly know that I cringe, at a personal level, when I see scientists who are trained and experienced in analysis of determining causation in the presence of correlation, fail to apply their training and experience uniformly.

    But: (1) while we don’t know what the long-term developments will tell us about the impact of the current politicization of the debate, it stands to reason that the impact of other developments long term will swamp the impact of short-term politicization and, (2) I think that within reason, it is always important for everyone of influence to be painstakingly careful to identify politicization across the board, and that trying to tease out some distinction between political influences among scientists from the larger political context amounts to chasing down a false dichotomy: By the very nature of a human tendency towards tribalism and motivated reasoning, and in particular when viewed within the larger current political climate, such a distinction doesn’t exist.

    But perhaps your recommendation for a more effective response to Inhofe, and an extension of that recommendation as a more general principle, would create better outcomes: It would certainly fit with my view of how to effectively debate the merits of one’s position within an academic framework, if not necessarily within a political framework. I would certainly like to see changes made to see if your recommendations would result in better outcomes. But on the other hand, I would argue that the very same logic would extend to how we all comment as observers, or participants, on the “politicization of the science,” i.e., that focusing specifically on the political advocacy of climate scientists without identifying and quantifying the full context of the political antecedents and political advocacy of their interlocutors will in the end only feed tribalistic energy - by solidifying and energizing the politically motivated “skeptics,” and by further entrenching the political motivations of the “pro-AGW” tribalists.

    ReplyDelete
  15. Given that Dressler just got a paper published in a peer review journal without real peer review, his dismissal of correct procedure is not that surprising, is it???

    ReplyDelete
  16. Joshua said... 13

    Would a more nuanced or strategically-moderated approach have had much greater impact given the current political and economic context?

    Governance only works because a broad cross section of the population has trust in the institutions of Government.

    Let's take the simple case of speed limits.

    Some regulatory agency has the authority to set speed limits. They have a specific set of criteria they have to follow to set speed limits. There is no 'law' that sets the speed limit on my street specifically at 25 MPH. A traffic safety engineer applied a set of criteria to determine the speed limit.

    It would be impossible for elected officials to get anything done if a public hearing and votes had to be taken on the speed limit for each individual street.

    It would also be impossible for courts to process all the speeding cases if the agency charged with setting speed limits were occasionally be found to have 'just made it up'.
    Every defense lawyer would demand a full hearing as to whether the agency responsible for setting speeding limits had followed the proper procedures. Instead of a 'speeding trial' taking 15-20 minutes they would all take a week.

    Environmental Regulation just got harder and more expensive to enforce because every two-bit polluter is going to demand a review as to whether or not EPA followed the proper procedures in establishing that regulation.

    ReplyDelete
  17. If a clique of guys with similar science degrees happen to share the same policy preferences does that mean their position is based on science?

    Dr. Trenberth apparently believes that no procedures are required if the clique agrees with the policy outcome, no procedure is acceptable if it it does not.

    Maybe somebody needs to explain to Dr. ("Kim Il") Trenberth that the Administrative Procedures Act is all about accountability and transparency, topic areas which have not been the strong suit of the paid climate science community. He should therefore leave that sort of procedural consideration persons more qualified and less overtly partisan than himself.

    ReplyDelete
  18. You have to wonder what is going to happen to public perception of science as a whole if climate change does not pan out.

    ReplyDelete