False Positive Science

Writing in the journal Psychological Science, Simmons et al. (2001, here in PDF) identify a problem in the psychological literature which they call "false positive psychology." They describe this phenomenon as follows:

Our job as scientists is to discover truths about the world. We generate hypotheses, collect data, and examine whether or not the data are consistent with those hypotheses. Although we aspire to always be accurate, errors are inevitable.

Perhaps the most costly error is a false positive, the incorrect rejection of a null hypothesis. First, once they appear in the literature, false positives are particularly persistent. Because null results have many possible causes, failures to replicate previous findings are never conclusive. Furthermore, because it is uncommon for prestigious journals to publish null findings or exact replications, researchers have little incentive to even attempt them. Second, false positives waste resources: They inspire investment in fruitless research programs and can lead to ineffective policy changes. Finally, a field known for publishing false positives risks losing its credibility.

In this article, we show that despite the nominal endorsement of a maximum false-positive rate of 5% (i.e., p ≤ .05), current standards for disclosing details of data collection and analyses make false positives vastly more likely. In fact, it is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis.

Why does this phenomenon occur?

The culprit is a construct we refer to as researcher degrees of freedom. In the course of collecting and analyzing data, researchers have many decisions to make: Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both?

It is rare, and sometimes impractical, for researchers to make all these decisions beforehand. Rather, it is common (and accepted practice) for researchers to explore various analytic alternatives, to search for a combination that yields “statistical significance,” and to then report only what “worked.” The problem, of course, is that the likelihood of at least one (of many) analyses producing a falsely positive finding at the 5% level is necessarily greater than 5%.

This exploratory behavior is not the by-product of malicious intent, but rather the result of two factors: (a) ambiguity in how best to make these decisions and (b) the researcher’s desire to find a statistically significant result. A large literature documents that people are self-serving in their interpretation of ambiguous information and remarkably adept at reaching justifiable conclusions that mesh with their desires (Babcock & Loewenstein, 1997; Dawson, Gilovich, & Regan, 2002; Gilovich, 1983; Hastorf & Cantril, 1954; Kunda, 1990; Zuckerman, 1979). This literature suggests that when we as researchers face ambiguous analytic decisions, we will tend to conclude, with convincing self-justification, that the appropriate decisions are those that result in statistical significance (p ≤ .05).

Ambiguity is rampant in empirical research.

The problem of "false positive science" is of course not limited to the discipline of psychology or even the social sciences. Simmons et al. provide several excellent empirical examples of how ambiguity in the research process leads to false positives and offer some advice for how the research community might begin to deal with the problem.

Writing at The Chronicle of Higher Education, Geoffrey Pullam says that a gullible and compliant media makes things worse:

Compounding this problem with psychological science is the pathetic state of science reporting: the problem of how unacceptably easy it is to publish total fictions about science, and falsely claim relevance to real everyday life.

Pullam provides a nice example of the dynamics discussed here in the recent case of the so-called "QWERTY effect" which is also dissected here. On this blog I've occasionally pointed to silly science and silly reporting, as well as good science and good reporting -- which on any given topic is all mixed up together.

When prominent members of the media take on an activist bent, the challenge is further compounded. Of course, members of the media are not alone in their activism through science. The combination of ambiguity, researcher interest in a significant result and research as a tool of activism makes sorting through the thicket of knowledge a challenge in the best of cases, and sometimes just impossible.

The practical conclusion to draw from Simmons et al. is that much of what we think we know based on conventional statistical studies published in the academic literature stands a good chance of just not being so -- certainly more than the 5% threshold used as a threshold for significance. Absent solid research, we simply can't distinguish empirically between false and true positives, meaning that we apply other criteria, like political expediency. Knowing what to know turns out to be quite a challenge.

Is Manufacturing Special?

ManpowerGroup has just released their latest employment outlook survey looking to the next quarter (April-May-June). In the graph above I show the report's numbers for companies in the US expecting to hire more employees over the next three months (calculated as the percent expecting to add minus the percent expecting to cut, US data here in PDF).

I have highlighted the manufacturing sectors in red (durable and nondurable goods). They show significant expectations for hiring, but less than leisure & hospitality, mining and professional & business services. With professional & business services currently employing about 18 million people, as compared to about 12 million in manufacturing, someone will have to remind me why manufacturing is supposed to be a special sector and not not professional and business services.

Paul Nurse's Dimbleby Lecture

A few weeks ago, Sir Paul Nurse, president of the Royal Society and Nobel Laureate, gave the Richard Dimbleby Lecture for the BBC (here in PDF). In it he presented a rather simplistic and two-dimensional picture of the relationship of science and decision making.

He first conflates scientific judgments with judgments about action:

It is impossible to achieve complete certainty on many complex scientific problems, yet sometimes we still need to take action. The sensible course is to turn to the expert scientists for their consensus view. When doctors found I had blockages in the arteries around my heart I asked them for their expert view as to what I should do. They recommended a bypass, I took their consensus advice, and here I am. That is how science works.

As the doctor metaphor is a common one in this context, I have written about it on numerous occasions to illustrate that consulting a medical expert is not as simple as the patient doing whatever the doctor says. For instance, here is what I wrote in The Climate Fix (p. 215):

So your child is sick and you take him to the doctor. How might the doctor best serve the parent’s decisions about the child? The answer depends on the context.

• If you feel that you can gain the necessary expertise to make an informed decision, you might consult peer-reviewed medical journals (or a medical Web site) to understand treatment options for your child instead of directly interacting with a doctor.
• If you are well informed about your child’s condition and there is time to act, you might engage in a back-and-forth exchange with the doctor, asking her questions about the condition and the effects of different treatments.
• If your child is deathly ill and action is needed immediately, you might ask the doctor to make whatever decisions are deemed necessary to save your child’s life, without including you in the decision-making process.
• If there is a range of treatments available with different possible outcomes, you might ask the doctor to spell out the entire range of treatment options and their likely consequences to inform your decision.

Even in the superficially simple scenario of a doctor, a parent and child, it’s clear that the issues are complicated. Understanding the different forms of this relationship is the first step toward the effective governance of expertise.

Nurse asserts that we must resolve science questions prior to resolving political questions:

Today the world faces major problems. Some uppermost in my mind are food security, climate change, global health and making economies sustainable, all of which need science. It is critical for our democracy to have mature discussions about these issues. But these debates are sometimes threatened by a misinformed sense of balance and inappropriate headlines in the media, which can give credence to views not supported by the science, and by those who distort the science with ideology, politics, and religion.

From the very beginning of science there have always been such threats. When Galileo argued that the earth orbited the sun, the Inquisition did not argue back with science, they simply showed him the instruments of torture. It is very important that we keep such influences separate from scientific debate. The time for politics is after the science not before.

Nurse chooses (unwisely) to illustrate his point with genetically modified crops:

It is time to reopen the debate about GM crops in the UK but this time based on scientific facts and analysis. We need to consider what the science has to say about risks and benefits, uncoloured by commercial interests and ideological opinion. It is not acceptable if we deny the world’s poorest access to ways that could help their food security, if that denial is based on fashion and ill-informed opinion rather than good science.

Good luck separating science and politics in that debate, much less getting the science before the politics! Debates over GM crops at times involve questions that might be resolved through the tools of science, but more often, such debates involve questions of values grafted on to issues of risk and benefits, as if science might resolve them.

Nurse ends his speech with a passionate and conventional plea for more government support for science and more autonomy for researchers:

We need more science in Government, the boardroom, and public services, we need more funding for science, we need greater engagement with the public and a society comfortable with science, we need to convey the wonder of science, and what it contributes to our culture and our civilization.

But other than broad generalities, he does not address what science is needed (all of it I suppose), how it is to be paid for or what returns ought to be expected. Such hand waving is of course common place in science policy debates.

In an editorial last week, Nature politely took issue with Nurse's arguments about the relationship of science and society, and offered a somewhat more nuanced extension and re-characterization of Nurse's remarks:

[A]lthough political (and religious) ideology has no place in deciding scientific questions, the practice of science is inherently political. In that sense, science can never come before politics. Scientists everywhere enter into a social contract, not least because they are not their own paymasters. Much, if not most, scientific research has social and political implications, often broadly visible from the outset. In times of crisis (like the present), scientists must respond intellectually and professionally to the challenges facing society, and not think that safeguarding their funding is enough.

The consequences of imagining that science can remain aloof from politics became acutely apparent in Germany in 1933, when the consensus view that politics was, as Heisenberg put it, an unseemly “money business” meant that most scientists saw no reason to mount concerted resistance to the expulsion of Jewish colleagues — regarded as a political rather than a moral matter. This 'apolitical' attitude can now be seen as a convenient myth that led to acquiescence in the Nazi regime and made it easy for German scientists to be manipulated. It would be naive to imagine that only totalitarianism could create such a situation.

The rare and most prominent exception to apolitical behaviour was Einstein, whose outspokenness dismayed even his principled friends the German physicists Max Planck and Max von Laue. “I do not share your view that the scientist should observe silence in political matters,” he told them. “Does not such restraint signify a lack of responsibility?” There was no hint of such a lack in Nurse's talk. But we must take care to distinguish the political immunity of scientific reasoning from the political dimensions and obligations of doing science.

Nature is on target. Ultimately, the question facing scientists is not whether to engage in the political arena, but how - and resolving the science before the politics is just not an option in all but the most simplistic of decision settings.

How Good Science Makes One a Darling of the Deniers

[UPDATE 3/14: To get a sense of how the game is played by climate scientists -- and they do indeed treat it like a game -- see this query of mine to Richard Somerville asking about comments attributed to him and his response.]

In December I was contacted by a journalist named Christine MacDonald who wanted to ask me a few questions about extreme events and climate change:

Hi Prof. Pielke,

I am writing an article for E Magazine on climate change and extreme weather. I read about your concerns that some scientists are overstating the connections between hurricanes, cyclones and some other weather events and climate change. I wanted to ask you what you think of Kevin Trenberth's proposal about shifting the burden of proof so that we assume climate change is involved in all usual events unless proven otherwise.

I chatted with her, giving what is by now a very familiar set of points related to a wide range of peer-reviewed science, including the most recent IPCC SREX report. She responded with this email to check quotes:

Hi Roger,

Here's what I was going to use. Please take a close look, particularly at this part: events such heatwaves, droughts, floods and hurricanes. Is that accurate to your views? if not please let me know and I will fix.

“It stretches the science to the point of breaking and provides skeptics with fuel. So why go there?” says Pielke of his colleagues efforts to discern global warming “fingerprints” on events such heatwaves, droughts, floods and hurricanes.

“It’s more of rhetorical interest than anything scientific,” Peilke says of Trenberth’s proposal. “The science is not there because if it were he’d be making the case with numbers.”

Thanks.

Christine

I responded with the following clarifications:

Thanks Christine ... two very important changes (delete between stars):
--------------------------

“It stretches the science to the point of breaking and provides skeptics with fuel. So why go there?” says Pielke of ***DELETEhis colleagues**** efforts to discern global warming “fingerprints” on RECENT INDIVIDUAL events such heatwaves, droughts, floods and hurricanes.

“It’s more of rhetorical interest than anything scientific,” Peilke says of Trenberth’s proposal. “The science is not there because if it were he’d be making the case with numbers.”

---------------------------
Explanation:

1. It is not my colleagues efforts (maybe one or two, but the current phrasing is too broad a brush) but mainly political advocates and certain media ...

2. The issue is not with the longer-term climate time scale studies (over 30-50 years and longer) but with attribution of specific events

Please note the misspelling of my name, Thanks!

Please ask if anything is unclear ... RP

She wrote back to accept the changes.

Here is what just now appeared in her story for E: The Environmental Magazine:

One obstacle to exactitude is that extreme events, by their very nature, are infrequent and leave few clues for scientists to follow, says Kevin Trenberth, senior scientist at the National Center for Atmospheric Research. Trenberth caused a stir last year by suggesting that the time had come to flip the burden of proof and assume climate change plays a role in all weather today.

“Of course there is large weather and natural variability. And even with climate change, most of the time it falls within the bounds of previous experience. But increasingly it doesn’t, and records are being broken that are consistent with a human influence of warming,” he wrote in an e-mail exchange

It’s a controversial idea that has attracted criticism from other climate scientists and has been assailed by climate deniers and a few prominent bloggers like Roger Pielke, Jr.

“It’s more of a rhetorical question than anything scientific,” says Pielke, a professor of environmental studies at the Center for Science and Technology Policy Research at the University of Colorado at Boulder.

Pielke, a darling of the denialist community for his criticism of Al Gore, Trenberth and others, has accused climate scientists of overreaching to make weather-climate change connections, allegations that have annoyed Somerville and other top scientists who consider Pielke, a political scientist, an unqualified gadfly on matters of climate science.

Apparently Ms. MacDonald is a follower of the "don't tell the whole truth" school of environmental journalism ;-)

The Richer Rich in the United States

The figure above comes from a column by Allan H. Metzler in today's WSJ. He argues:

Regardless of one's economic philosophy, the public deserves an accurate presentation of the reasons for the change in income distribution. The change is occurring in all the developed countries. The chart shows that policies that redistribute wealth and income have at most a modest effect on income shares.

While he is correct that gross trends are similar across countries, he underplays the rather remarkable acceleration of income share captured by the top 1% of earners in the United States, which has continued since the end of the data period shown in the graph. Is that acceleration a function of policy decisions? Of course. On that point Metzler is letting his economic philosophy speak rather than looking closely at what the graph actually suggests.

Gasoline, Energy and the Economy

Continuing the ongoing discussion of gasoline prices and the economy, the figure above shows gasoline expenditures as a fraction of personal consumption expenditures for 1959 through 2011 (Data: BEA). You can see that gasoline ended 2011 at about the 4% level. Gasoline spending was actually higher in September 2011 (4.1%) than in December (3.8%).  The increase of 1% -- from 2.8% in May 2009 to 3.8% in December 2011 -- coincided with a 70% increase in the price of gasoline (data: BLS).

The graph below shows total energy spending (where energy is defined as the sum of expenditures on gasoline, heating oil, electricity and natural gas) from 1959 through 2011, and it shows that total energy spending is just under 6% of personal consumption expenditures. To some degree, higher priced gasoline has been offset by lower spending on heating oil and electricity.

The third graph, below, shows gasoline spending as a percent of total personal energy consumption expenditures. The graph shows a remarkable degree of consistency over many decades with gasoline comprising between 50% and 60% of total energy expenditures. Gasoline has increased above the 60% threshold as gasoline prices have increased and other energy expenditures have remained constant or declined.

So what does all of this data mean? Here is my interpretation:

1. The US economy is not yet in a danger zone with respect to gasoline prices -- $4 dollar per gallon gasoline is not what it used to be. What would it take for gasoline to exceed 5%, 6% or more of personal consumption expenditures? It is not easy to answer with any confidence, but if the increase in price of $1.50 per gallon or 70% from May, 2009 to September, 2011 resulted in an increase in personal gasoline consumption expenditures of 1%, then assuming linearity (caution: rarely a good assumption, but lets play around), then we'd have to see gasoline prices of more than $5 per gallon to reach that 5% threshold. There are good reasons to think that such an outcome is not imminent.

2. The overall impact of gasoline prices is somewhat muted in the context of total energy expenditures because of the falling price of natural gas. To the extent that natural gas can replace gasoline (not easy in the short term) this trend will be accelerated. Meantime, consumers will see a bit of an offsetting balance in their total energy expenditures due to lower electricity and gas prices. A hot summer could easily lead to increased expenditures as air conditioning is cranked up, and summer is traditionally when there is more demand for gasoline.

3. Why did things appear to change in US gasoline spending starting 2002? My guess is China, which joined the WTO in 2002 and subsequently saw its oil consumption increase by almost 90% from 2002 to 2010, a period of remarkable economic growth. Over the same time period the United States saw its oil consumption decline by more than 3% (Data: BP). Such trends are likely to continue, making it more important for the US to reduce the gasoline intensity of its economy and/or develop new resources. The pressure to drip and pipe will only increase.

4. In the context of such trends, it should be a matter of policy to seek to reduce the share of economic activity expended on gasoline in particular and energy in general. The good news is that there does seem to be a consensus, even in these most partisan of times, on what makes for a common sense energy policy -- The Economist makes a good start.

Here is President Obama just yesterday responding to a Fox News reporter who asked if he really -- deep down inside -- wants to see gasoline prices go higher. The President responds with his verison of the iron law:


H/T: Daniel Ahn and James Hamilton for useful analyses along similar lines.

Are Media Apologists Good for Climate Science?

[UPDATE 3/7: In a move rather ironic given this post, Michael Mann writes on his Facebook page that I have "teamed up" with Marc Morano to produce a "dishonest smear." How have I "teamed up" with Morano?  Apparently Morano linked to this post. Mann appears challenged with the truth.

Also, on Twitter (gotta follow lots of media these days) Mann suggests that his advocacy for the Hockey Stick was in one part of the IPCC report not another. My reply -- So what? The Gillis NYT article mentioned and linked to the entire IPCC report. In how many places was the Hockey Stick featured in the 2001 IPCC report? At least in 6 places (PDF).]

Yesterday, Michael Lemonick asked of journalists at Climate Central, "should we tell the whole truth about climate change?" His answer is to ask "So where’s the right balance between telling the whole truth and being truthful in an effective way?" (have a look at the link title as well). For some journalists a desire for "effectiveness" trumps "truth."

Journalists, like everyone else, have their biases and perspectives. And on the issue of climate change journalists are as prone as any of us to the seductive siren of tribalism, with good guys on one side and evil ones on the other. But does this framing actually serve the interests of the broader climate science community?  I think not.

Here is an example (and to be clear, it is one example that could be selected from many). In today's New York Times, Justin Gillis (who has made clear on several occasions that he is no fan of mine) offers a favorable review of Michael Mann's new book. That Gillis likes Mann's book is not at all problematic. However, the fact that in his review Gillis includes a statement that is demonstrably false in his defense of Mann is problematic and arguably reinforces the polarization around the debate.

Gillis writes of the so-called "hockey stick" graph produced by Mann and highlighted by the IPCC in its 2001 report (emphasis added):

The graph of reconstructed temperatures is called a hockey stick because the right-hand side shows temperatures veering sharply upward in the last century. The paper and its graph, along with subsequent studies by Dr. Mann and several other scientists, suggest that this recent warming is anomalous, at least over the past millennium. Through no choice of Dr. Mann’s, the graph became a symbol of modern climate science when it was featured prominently in a 2001 report by a United Nations panel.

Now you don't have to be a climate insider to know that Mann was in fact actively involved in promoting his own graph to be featured by the IPCC in its 2001 report. and that Gillis' statement is flat out wrong. You don't have to believe me (or Steve McIntyre, who Gillis denigrates but does not cite by name), but Mann himself. Mann was unequivocally a vigorous advocate within the IPCC promoting his own work. When the NYT gets such a basic fact wrong it not only makes them look bad, but has collateral damage for the climate science community.

The realities of the most intensely contested aspects of the climate debate are that there are human beings on both sides -- complex, contradictory, red-blooded, imperfect human beings. When the media places scientists up on a pedestal and does so via the spinning  of untruths, they simply set the stage for a bigger fall when the scientists cannot live up to their adulatory press coverage. And besides, many of us know better. The media should cover science in three dimensions, and eschew the two-dimensional fiction of good vs. evil, even if that means exploring nuance and contradiction.

Rick Santorum, Higher Ed and American Values

UPDATE 3/5: The WSJ reports that Rick Santorum says "never mind" on the "snob" comment. He must have read my piece ;-)

I've got a short piece in the Boulder Daily Camera today on Rick Santorum's comment last week about that "snob" in the White House who wants to see more college graduates.  Here is how it ends:

Given how much Americans like to make money, remain employed, benefit from ongoing innovation and participate in the annual March Madness basketball selection pools, good advice for any aspiring presidential candidate would be to not put yourself on the wrong side of core American values.

See the whole thing here.

Invitation for Input on a Second Edition of The Honest Broker

I am currently mulling over the idea of working on a second edition of The Honest Broker (CUP willing;-). I have received a lot of excellent comments, suggestions and critique on the original, and there has also been discussion in the academic literature. Currently I am thinking about adding three chapters:

• a discussion of how policy makers should use experts (so, the flip side of the main perspective advanced in the book)
• a discussion of best practices in "honest brokering" from around the world
• a case study of the climate debate using the framework of the book, with a focus on science, the media, and decision makers (unsure about this one as it might distract from everything else)

What would you like to see in a revision? Advice and suggestions welcomed!