NOTE: THE WORD CLOUD ABOVE IS OF THE PRESIDENT'S MARCH, 2009 MEMO]
Some of my questions below have been clarified in the recently published guidelines; the next post in this series will be deconstructing the proposed guidelines themselves.
(7) By this memorandum, I assign to the Director of the Office of Science and Technology Policy (Director) the responsibility for ensuring the highest level of integrity in all aspects of the executive branch's involvement with scientific and technological processes.Again, I am not sure if the term “integrity” is used in its sense as “wholeness” or its sense as “moral.” Certainly it is a word that it is hard to be against, because it means generally “good.” But note that this is the “executive branch’s involvement with scientific and technological processes.”
What exactly are “scientific and technological processes?” Let’s list them. First, there is determining the science and technology budget, and what goes to which agencies for what kind of research. Then there is the process of determining what is studied, which disciplines are included, how the questions are generally framed, and running the panels to determine who gets the grants. Finally, there is checking on the validity of the scientific products through peer review and other processes.
But again, I thought the problem that initiated all this work was about how science is used in policy processes, not about how “scientific processes” themselves are run. If we are going to “let the punishment fit the crime,” seems like the first step is clarifying the crime.
The principles (8):
(a) The selection and retention of candidates for science and technology positions in the executive branch should be based on the candidate's knowledge, credentials, experience, and integrity;This statement sounds to me vaguely like the old Data Quality Act, which also required small armies of GS-14s and 15’s in D.C. to write guidelines for each agency. The purpose of the Data Quality Act was to "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies". Here we see the word “integrity” again, but used in a different way.
In addition to my other questions in the previous post about “what kind of integrity,” it’s not clear to me what a “science and technology position” is. There are many, many positions throughout the agencies that require a technical background of some kind. Are “science and technology” positions those related to administering and conducting research only, or to application of different scientific fields to the mission of the agencies? Where do you draw the line, or do you need to?
(b) Each agency should have appropriate rules and procedures to ensure the integrity of the scientific process within the agency;
See (7).
(c) When scientific or technological information is considered in policy decisions, the information should be subject to well-established scientific processes, including peer review where appropriate, and each agency should appropriately and accurately reflect that information in complying with and applying relevant statutory standards;
In the Wikipedia cite to the DQA we find this quote:
The DQA has been criticized by the scientific community and journalists as a ploy of corporations and their supporters to suppress the release of government reports contrary to their economic interests.I remember when we were working on the DQA responses within agencies, we also attempted to deal with the question David Bruggeman raised in his comment here:
"As subsequently interpreted by the Bush administration . . . the so-called Data Quality Act creates an unprecedented and cumbersome process by which government agencies must field complaints over the data, studies, and reports they release to the public. It is a science abuser's dream come true" (Chris Mooney, The Republican War on Science [New York: Basic Books, 2005], p. 103).
“Much research utilized to inform policy will have been conducted with no knowledge of how to inform policy or intent to inform policy. Can we realistically expect the kind of peer review (essentially a second round of it) and QA/QC that you would like to see, as it would have to be done retroactively?”The problem remains the same problem as when we were dealing with the DQA. If you have quality considerations for the use of science in policy, it would restrict the science used to that which had been designed to be used in policy. I suppose instead you could have two separate classes of scientific information, “designed to be used for policy” and “other” with different guidelines for the use of each class.
I also question the words “appropriately” and “accurately.” I think anyone in the trenches of dealing with this recognizes that using any piece of research “appropriately” and “accurately” is a value judgment. I think it’s a great science education opportunity to have public discussion about what is appropriate and accurate for any piece of scientific information with regard to a particular piece of policy. Yet the principle as articulated here seems to assume that there is one right answer. This is the linear model of research to policy, which, of course, scientists who study science policy have shown to be incorrect. At least I think that is an “appropriate” and “accurate “reflection of the available information ;).
d) Except for information that is properly restricted from disclosure under procedures established in accordance with statute, regulation, Executive Order, or Presidential Memorandum, each agency should make available to the public the scientific or technological findings or conclusions considered or relied on in policy decisions.This is pretty much the fourth principle of my four science-to-policy principles found below.
e) Each agency should have in place procedures to identify and address instances in which the scientific process or the integrity of scientific and technological information may be compromised.What does it mean to “compromise the scientific process?”. Hopefully this does not mean “reducing funding for research”. Selecting only your buddies for reviews or review panels? And again, you could argue that “the integrity of scientific and technological information” was also sought by the DQA; what is the difference? Have we learned something from agency efforts under the DQA? Also, I don’t see why each agency needs to develop these separately (unless it’s part of a larger jobs program for mid-level guideline writers ;)). Can you imagine the potential train wreck when a regulatory agency, regulating a land management agency, operates from a different worldview with different procedures? And there’s the possibility that a research agency, regulatory agency and management agency could all have different approaches to determine the integrity of the research agency’s research. In my view, clarity is not achieved by asking many agencies to do something fuzzy.
(f) Each agency should adopt such additional procedures, including any appropriate whistleblower protections, as are necessary to ensure the integrity of scientific and technological information and processes on which the agency relies in its decision making or otherwise uses or prepares.Again, does this make sense to do separately by agency? Many policy decisions use research derived from different sources and agencies. In fact, many use research developed outside the federal government. Should we expose that work to a series of quality reviews before we use it? (This was a discussion point on the DQA). Do these outside sources need to have their whistleblower procedures in place?
Summary of Memo:
1. What if we were to apply the ideas espoused in the memo to the promulgation of the memo (as it is policy) itself? We might expect a section describing how the work of noted science policy experts was used in the development of the memo, with peer-reviewed citations. I’d expect to see Jasanoff, Sarewitz and Pielke, Jr., at least, cited.
2. Here are my four principles for improving the use of information in policy, (1) joint framing and design of research with policymakers (2) explicit consideration of the relevance of practitioner and other forms of knowledge (3) quality measures for scientific information (including QA/QC, data integrity and peer and practitioner review), and (3) transparency and openness of review of any information considered and its application to policy.
3. If the DQA and the “Integrity” work are seen to be the result of inchoate longings by many for an improved “science to policy” process; and if they seem each to have become, instead, weapons to slime the opposing political party, then why not establish a bipartisan commission on improving the use of scientific and technical information in policy? Science policy experts would advise the commission, and the deliberations would be transparent and open to public comment. The terrain to be explored would include my four principles above, and add considerations of involving citizens more directly in working with the relevant Congressional committees in developing federal research budgets and priorities.