Evaluating Predictions of the UK General Election

The United Kingdom is going to have a general election on May 7th, just over five weeks from now. There are high stakes and a lot of uncertainties, probably more so than usual. Yesterday, David Cameron (Conservative and Prime Minister) and Ed Miliband (Labor and party leader), squared off in parallel interviews with interviewer Jeremy Paxman in a broadcast watched by 12% of the UK TV audience.

These days, where there are elections there are also election forecasters. But political scientists have been doing this for a long while. Back in the early 1990s, when I was in graduate school in political science, I wrote a seminar paper on methodologies of election forecasting, which at that time I took a pretty dim view of (I still do!).

Where prediction is concerned, it is always worth evaluating our forecasts, lest we trick ourselves into thinking we know more than we actually do. So for fun, I am going to evaluate predictions of the upcoming UK elections.

Courtesy Will Jennings, a political scientist at the University of Southampton @drjennings, via Twitter, below is a summary of various predictions of the outcome of the upcoming election.

The 12 predictions span a huge range, +/-33 seats for the Conservatives and +/-25.5 for Labor. Six of the 12 forecast Labor holding more seats than the Conservatives, and 6 forecast less. With such a wide spread, it is mathematically safe to say that some predictions will be better than others.

I am going to evaluate 2 questions using this data after the results are in.

1. Which forecast showed the most skill?
2. Does the collection of forecasts demonstrate any skill?

To evaluate #1, I will use a naive baseline as the basis for calculating a simple skill score. The naive baseline I will use is just the composition of the current UK Parliament.

Party	Seats
Conservative	302
Labour	256
Liberal Democrat	56
Democratic Unionist	8
Scottish National	6
Independent	5
Sinn Fein	5
Plaid Cymru	3
Social Democratic & Labour Party	3
UK Independence Party	2
Alliance	1
Green	1
Respect	1
Speaker	1
Total number of seats	650
Current working Government Majority	73

It is important to note that there are, effectively, a limitless number of ways that a forecast evaluation might be structured, with different results as a consequence. Always beware post hoc forecast evaluations. I have no horse in this race, so I am producing a very simple evaluation, based on methods I have used before on many occasions. These choices could of course be made differently.

Some methodological details:

• I am evaluating predictions of actual seats, not percentage gains or losses.
• I am counting all seats equally.
• I am not evaluating the prediction of specific seats, but overall parliamentary composition. Yes, this means that skill may occur for spurious reasons.
• Yes, there are other, likely "better," naive baselines that could be used (e.g., using recent opinion poll results). Such a choice will reflect upon absolute skill, but not relative skill.

Given that there is a wide spread among multiple forecasts, we have to very careful about committing the logical fallacy of using the election outcomes to select among forecasts. This is of course a very common problem in science (which i described in this paper in PDF in the context of hurricane forecasts in reinsurance applications). I have described this problem as the hot hand fallacy meets the guaranteed winner scam. It is easy to confuse luck with skill.

So I will also be evaluating the forecast ensemble. I will do this in 2 ways. I will evaluate the average among forecasts and I will evaluate the distribution of forecasts, both against the naive forecast as well as the election outcome. We can expect to be able to conclude very little about the skill of a forecasting method (as compared to a specific forecast) because we are looking at only one election. So my post-election analysis will necessarily include the empirical and the metaphysical. But we'll cross those bridges when we get there.

This exercise is mainly for fun, but because my new book has a chapter on prediction (that I am in the midst of completing) it is also a useful way for me to re-engage some of the broader literature and data in the context of a significant upcoming election.

Comments, suggestions most welcomed from professionals and amateurs alike!

The University of Colorado-Boulder's New "Degree in Three" Initiative

The University of Colorado-Boulder has started a new initiative, called "Degree in Three." The Daily Camera today has some background:

The University of Colorado is launching a new initiative for cost-conscious and decisive undergraduate students who want to finish their degree in three years.

Traditionally, students and parents have thought of college as a four-year experience, but that doesn't always need to be the case, said Michael Grant, CU-Boulder vice provost and associate vice chancellor for undergraduate education.

The goal of "Degree in Three" is to make students aware that it's possible to finish all the requirements for a bachelor's degree in three years, an effort that could save them money and help move them along to the next step in their life.

Over the past year, I have been working with Michael Grant, CU-Boulder vice provost and associate vice chancellor for undergraduate education, to roll out the initiative. From the Camera article:

Roger Pielke Jr., a faculty member who has been working with Grant on the initiative, said he thinks of it as an "experiment" to see what kind of demand exists.

"It seems that there's space for expanding the options that are available to students these days, with concern about the cost of college and the job market and so on," he said.

Though students may need to pay for additional courses during the summer, he estimated that finishing in three years could save a student 10 to 20 percent on the total cost of their degree—every little bit counts, he said.

Currently, undergraduate tuition in the College of Arts and Sciences is $9,048 for in-state students, $31,410 for out-of-state students and $32,910 for international students. Students in other colleges and schools pay different tuition rates.

Even if students take slightly longer than three years to finish, that's still a win for the campus.

It's also a win for students and their families. If you are interested in the initiative, please check out its new website - Degree in Three. And please feel free to email me with any questions.

New Review of Disasters & Climate Change

Here’s my review of new book by @RogerPielkeJr in @weeklystandard An even shorter review: I like it. http://t.co/cXYGXRT31v
— Robert Bryce (@pwrhungry) March 20, 2015

The Weekly Standard has a positive review of Disasters & Climate Change, by Robert Bryce. Here is an excerpt:

In The Rightful Place of Science, Pielke acknowledges the massive challenges, and inherent conflicts, in the energy/climate debate—and in so doing, he reveals himself to be both rationalistic and humanistic. I’ll take those stances over religious zealotry every day of the week. 

See the whole review here. Other reviews can be found here.

My Review of Galileo's Middle Finger

In this week's Nature, I have a review of Galileo's Middle Finger: Heretics, Activists, and the Search for Justice in Science, by Alice Dreger, a medical historian who has spent much of the past decade studying controversies in science.  My review can be found here and here in PDF.

The book is engaging and eye-opening. It chronicles series of issues, mainly related to the science of sexuality and gender, in which activists, a category which also includes academics, engage in professional and personal attacks of scholars whose work that they find to be inconvenient, unhelpful or just offensive to their sensibilities about how the world should work. Most academic work is like the proverbial tree falling in the forest, but every so often (and probably more often than many of us would like to think), scholarship becomes the focus of a political battle.

I should know. My review was completed and filed the week before I was targeted with an "investigation" by a member of Congress for the audacity of testifying before that august body with the results of peer reviewed, government-funded research, widely accepted as scientific consensus. But even before that, my own career has led me to be sympathetic to Dreger's arguments.

I have lots of experience with personal and professional attacks based on my research and advocacy. For instance, it was one year ago today that I published a piece at FiveThirtyEight on that same research, which prompted a social and mainstream media campaign to have me fired for voicing such heresies. The Guardian, New York Times, Slate, Salon and even the American Geophysical Union all joined the campaign. Unsurprisingly, FiveThirtyEight succumbed to the pressure, explaining "Reception to the article ran about 80 percent negative in the comments section and on social media. A reaction like that compels us to think carefully about the piece and our editorial process."

So, scientific consensus vs. Facebook likes - Guess who won?

The pressures on academics to conform (or not deviate) is very high. But Dreger is an example of the rare scholar willing to take the heat for taking a good hard look at the "wisdom" of the crowd. She shows that in some instances the crowd is just a mob, utterly indifferent to evidence and scholarship. Dreger has taken many lumps for her work, but I have little doubt that Galileo's Middle Finger will secure her role as a champion of the integrity of evidence, despite the various efforts to delegitimize her and her work, which will probably continue as well.

Here is an excerpt from my review:

Even as her “stomach hurt from the thought of the backlash”, Dreger published her findings (A. Dreger Arch. Sexual Behav. 37, 503–510; 2008). She faced online accusations and e-mails about her funding and politics; ethics charges were filed against her with her dean. Ultimately, however, she won a Guggenheim Fellowship to look at other conflicts involving scientists and activists.

Dreger ends this powerful book by calling for her fellow academics to counter the “stunningly lazy attitude toward precision and accuracy in many branches of academia”. In her view, chasing grants and churning out papers now take the place of quality and truth. It is a situation exacerbated by a media that can struggle when covering scientific controversies, and by strong pressures from activists with a stake in what the evidence might say.

She argues, “If you must criticize scholars whose work challenges yours, do so on the evidence, not by poisoning the land on which we all live.” There is a lot of poison in science these days. Dreger is right to demand better.

If you are interested in the politics of science in the 21st century and the challenges faced by scholars who do work deemed politically taboo, then you will benefit from Dreger's engaging writing and exploration of numerous cases. You'll also learn something about human sexuality and its many complexities. Galileo's Middle Finger will be on my fall syllabus in my graduate seminar in Science & Technology Policy.

Read my full review here in PDF.

A Thank You to Rep. Raul Grijalva, Narrative Killer

With this post I'd like to express a sincere Thanks to representative Raul Grijalva (D-AZ). As most readers here will know, Rep. Grijalva is "investigating" me based on his belief that I do research and public service as a consequence of shadow payments from fossil fuel companies. Ridiculous, I know.

I'm thanking Rep. Grijalva not for the media exposure (e.g., NPR, NYT) nor for the bump in sales of my books (e.g., THB, TCF, D&CC), and not even for the many bits of fan mail via email and Twitter from the fringes of the climate debate. Rather, I am thanking Rep. Grijalva for doing more than his part in helping to kill a narrative.

For more than a decade, leading elements of the science and media communities have advanced a narrative which said that conservatives were stupid and/or evil and were singularly responsible for pathologically politicizing science.  Reality, as the saying goes, has a liberal bias. It turns out that concerns over the "politicization of science" were themselves subject to politicization.

I wrote about this in 2003:

Politicization of science is a problem irrespective of the ideology of those doing the politicizing. Our scientific enterprise is too important to allow putative concerns about the politicization of science to become just another weapon in partisan battle.

And in 2005:

It is clear that there is an ample supply of people willing to use concern over the politicization of science as a political bludgeon to score points on the Bush Administration. It is also clear that there are plenty of others aligned with the Bush Administration willing to do exactly the opposite. The question I have is, where are the analysts (including reporters) who care about the politicization of science irrespective of possible advantages that are lent to today’s partisan political battles?

A decade ago the face of the "Republican War on Science" narrative was a journalist named Chris Mooney, then a fresh-faced 20-something who had capture the zeitgeist in a book by the same title. I offered a detailed critique of of the "War on Science" framing in 2005.  I think that critique stands up pretty well.

For his part, Mooney followed up his "War on Science" book with a bizarre book on eugenics, claiming that US conservatives were somehow genetically inferior.  Mooney turned his prominent role in Republican-bashing into a spot on the board of directors of the American Geophysical Union (I kid you not), as an "expert" in science communication hired by the National Science Foundation to tour the country, training young scientists (still not kidding), and ultimately as a reporter for The Washington Post. Not a bad resume for an English major who has dabbled in eugenics.

This critique is less about Mooney, who I met once and seemed a nice fellow, and more about the power of a narrative. One that has been so fully accepted and reinforced by significant parts of the science and media communities. Mooney captured that narrative and went along for the ride. One day, hopefully, we'll look back at this era and ask "What the hell were we thinking?"

Writing in The New Republic last week Erik Nisbet and Kelly Garrett offered a welcome tonic to the "war on science" meme and a good indication that perhaps, just perhaps, that narrative has reached its sell-by date:

[P]olitical journalism too often treats science like a political issue to be debated by non-experts in televised partisan theater. This type of media coverage about scientific issues often obscures the actual scientific evidence and consensus and unfortunately only deepens polarization by providing partisan cues for both conservatives and liberals.

Our study’s findings suggest that such intensive, polarizing media attention depresses the public‘s confidence in the scientific community for liberals and conservatives alike.

The second lesson is that that science communicators who target conservatives specifically as somehow uniquely deficient when it comes to understanding science turn the focus to a clash of ideologies and away from promoting communication that bridges ideological gaps about science issues—and yes we think such gaps can be bridged!

Demonizing a third of the population in science policy debates by claiming they have an insurmountable psychological deficit does nothing to promote a solution to the challenges of effective science communication—and unfortunately represents our human biases at work.

Nisbet and Garrett are reporting on research which provides a solid empirical basis for rejecting the politicization of the politicization of science as a way of doing business in science or in journalism. It is neither accurate nor effective. Other scholars doing excellent work in this area include Dominique Brossard, Brendan Nyhan, Dan Kahan. Dietram Scheufele, Matt Nisbet, among others. But despite all this good empirical, historical and political research, the "war on science" narrative still has deep roots and fervent adherents.

Which brings me all the way back to Rep. Grijalva. In his "investigation" of me -- someone who probably shares many of his policy preferences, including on climate -- Rep. Grijalva has admitted to not liking my peer reviewed research (and logically the assessments of the IPCC). It is hard to maintain a uniquely Republican "war on science" meme with this type of high profile nonsense going on. Of course, in his Washington Post column, Mooney hasn't acknowledged Rep. Grijalva's "witch hunt." Probably just something deficient in his partisan brain.

As I have often written, there is no "war on science" being conducted by Republicans or by Democrats. There is however plenty of politics. Politics can be conducted in ways that contribute to common interests, or in ways that are pathological. The science community has tried the latter for a decade. It is time to move beyond the toxic partisanship of the most recent science wars. Oddly enough, Rep. Grijalva's overreach helps us to move in that direction.

Kenneth Prewitt on Science and Congress

Kenneth Prewitt, of Columbia University and social scientist extraordinaire,  has an interesting paper out in the January, 2015 Annals of the American Academy of Political and Social Science.  The paper is titled, "Who Is Listening? When Scholars Think They Are Talking to Congress." 

Here is a neat excerpt:

Scientists, social or otherwise, err in claiming that members of Congress are hostile to science or are anti-science when they vote contrary to scientific warnings about climate change or side with Biblical literalists and intelligent design proponents or express misgivings about genetic modification of crops. Members of Congress can decide that they or their constituents have sound economic reasons for digging and burning coal or have religious reasons for embracing intelligent design or place the precautionary principle line on genetic modification more conservatively than what is warranted by the preponderance of science. A member of Congress might take one of these positions, but not the other two; or take all three but at the same time vote for a defense budget that includes spending on missile defense research or vote to double the NIH budget.

To return to a point emphasized in Using Science as Evidence in Public Policy, a lot goes into the policy mix. When there is scientific evidence on the probable outcome of pushing the X button rather than the Y button, we can hope that it is taken into account. The bridge will be more likely to withstand the earthquake, the missile more likely to hit its target, and the school reform more likely to reduce dropout rates. Scientists can and should vigorously assert that scientific evidence makes for more efficacious policy. But scientists—speaking as scientists—cannot argue that economic interests, ideological preferences, or political considerations have no place in policy choices.

When you label someone as "anti-science" because they support different policy choices than you prefer, you are indeed arguing "that economic interests, ideological preferences, or political considerations have no place in policy choices."

It is probably past time for the rest of us to take science back from those who expect it to carry all the weight of politics on highly contentious societal issues.

Running Updates on the "Witch Hunt"

This post will serve as a running update on the so-called "investigation" of my research on disasters and climate change at the University of Colorado. I will update it as warranted, with newer stuff at the top. Pointers and tips welcome in the comments.

That's It

5 March - I'm moving on with my life, hope you are too! If there is anything really significant to add to the updates below, I'll do so, but meantime, I'm going back to writing my book.

Updates

• The lead editorial in Nature this week is about what they call a "fishing expedition" by Rep. Grijalva, saying that his actions send "a chilling message to all academics and to the wider public." Well said.
• Just yesterday, my 2013 Senate testimony that prompted Rep. Grijalva's attack was cited in a Senate hearing. Senator Jeff Sessions asked EPA administrator Gina McCarthy about trends in drought and hurricanes, citing my earlier testimony before that committee. On hurricanes McCarthy responded, "I cannot answer that question. It's a very complicated issue." The answer is of course "They have decreased in frequency and intensity by 20% since 1900." So long as Democrats find it impossible to hear what the data say on extreme events, the topic will remain deeply politicized. (The same is true for Republicans on other issues, hence the pathologically politicized climate issue.) I missed all the mainstream media coverage of McCarthy's struggles with the science.
• In the WSJ, Richard Lindzen of MIT, and another target of Rep. Grijalva, has a hard hitting op-ed in which he discusses the "witch hunt" and connects it to the overall political battle over climate policy and climate science.
• At Science Insider, a blog by Science magazine, I am interviewed by Eli Kintisch (@elikint) about COI, the "witch hunt" (my words) and my Ferrari, with plates that read "KochBros." 
• The American Geophysical Union has reversed course, after first offering support to Representative Grijalva's "investigation." The AGU now says "The rules of transparency affect us all equally, and therefore must be applied equally."  Better late than never, I suppose.
• I just did an interview with AM630 KHOW in Denver with Mandy Connell. We discussed science and politics, and it was another good conversation. You can listen to it below, starting at 72:40.
• The Boulder Daily Camera has a new article up, noting Rep. Grivalja has admitted to "overreach" but hasn't modified his request to my university. The University of Colorado states that it will be responsive to the request. I also am quoted in the article.
• I just did a lengthy, and I think pretty substantive, interview with Garret Lewis at KNST in Tuscon (in Rep. Grijalva's district). We discussed the "investigation," Al Gore, a carbon tax and other things. Lewis was well informed. You can listen to it below.
• Rep. Grivalja has walked back his requests, according to Ben Geman at the National Journal: Climate Letters Went Too Far. Since Rep. Grivalja already has complete access to all my financial COI disclosures, I guess we now know that the letter was an unnecessary stunt designed to smear. Nice.
• The Denver Post (March 3) has an editorial titled "CU rightly defends Roger Pielke Jr. against political bully" -- key quote: Rep. Grijalva's "gambit amounts to a bold, abusive assault on academic freedom."
• Mark Steyn asks why no reporters appear to be interested in the fact that an anonymous person had forewarning of Rep. Grivalja's "investigation" and used a fake email account from a Russian server to taunt those who would later receive letters from the Congressman. Bizarre to be sure, and if I hadn't seen the taunting email in advance I might not have believed it. But as I've said, nothing surprises me in the climate debate anymore.  
• To believe that Rep. Grijalva's "investigation" has merit, you have to believe either (a) in a shadowy conspiracy of fossil fuel interests funneling me (and others) money under the table to produce certain research results and testimony, which have somehow mysteriously passed peer review and been accepted by the IPCC, or (b) there is no such conspiracy, but I (and others) need to be falsely accused and smeared in order to remove us from the debate. Tin foil hat or unethical campaigner? Not a great choice. 
• Several reporters have asked me why I testify before Congress if I know that my results will be used by Republicans. Aside from the interesting framing of this question, I have written on my views of providing testimony here in PDF (following the testimony that I am now being investigated for) and please take careful note of the "To Avoid Any Confusion" bullets on p. 2 of my testimony here in PDF.  

Original bullet points

• This week I have been invited to do various interviews for print/online and radio. I'll update here when these are available and I have more details.
• 9 News in Denver had a excellent story, shown in the video above and online here.
• For those interested in my actual research on climate please head over to this summary in the final post at my climate blog, The Climate Fix.
• A group called the Energy & Environment Legal Group has filed state freedom of information act requests modeled on the Rep. Grijalva letter with 4 universities (Colorado, Arizona, Delaware, Georgia Tech) requesting funding information from 5 researchers. This is obviously a retaliatory act, legitimized by Rep. Grijalva's campaign. It is just as wrong-headed.
• Here in PDF is that strongly worded letter from the American Meteorological Society to Representative Raul Grijalva (D-AZ) warning that he is sending "a chilling message" to all researchers.
• Yesterday I had a nice chat with Representative Jared Polis (D-CO) who represents my district here in Boulder. What we said will stay between us, but it was very a positive conversation.
• Also yesterday @EricHolthaus - a widely read scientist and climate activist - taunted me with the following bizarre Tweet: "It’s getting harder and harder for @RogerPielkeJr to remain relevant." Upon later learning that I'm no longer doing climate change research Holthaus Tweeted that his earlier taunt was no longer relevant. Great evidence that a lot of this is about eliminating unwelcomed voices in the debate.
• At The Breakthrough Institute, Michael Shellenberger and Ted Nordhaus take the high road and argue that political intimidation of academics in unacceptable, defending both me and Michael Mann.