A Case Study in (how not to do) Quantitative Policy Analysis

Writing in the Boulder Daily Camera yesterday Tom Rohrer identifies a howler of a mistake in a recent report by the City of Boulder on "Safe Streets." The city has been implementing "flashing crosswalks" around town which light up to let motorists know that a person or bike is about to cross the street. In theory, the signals are supposed to lead the drivers to stop the pedestrians or bikers to cross, and everyone to go on their way.  In practice, the "flashing crosswalks" have been the location of some pretty nasty car-pedestrian accidents.

So it is a surprise that the city has issued a report that claims that the "flashing crosswalks" are safer than non-flashing crosswalks. Rohrer explains where the city's analysis is flawed:

[T]he city recently released their Safe Streets Boulder Report, in which they quite correctly note that a very large number of the accidents between motor vehicles and either a pedestrian or a bicyclist occurred in crosswalks at intersections, and a rather smaller number occurred in the flashing crosswalks. Moreover, I don't doubt their arithmetic was correct when they converted those accident counts to percentages.

On the face of it, the report's finding that 37 percent of the pedestrian/bicyclist-vehicle accidents were in crosswalks at intersections while "only" 6 percent of such accidents were in flashing crosswalks seems to be evidence that the flashing crosswalks are statistically more safe than anyone thought, and that if anything we should be worried about the crosswalks at our city's intersections.

Unfortunately those numbers are profoundly misleading.

You see, the report doesn't mention that there are only a few flashing crosswalks in the entire city (only 18 have ever been installed). However, there is a very large number of the crosswalks at intersections -- four at almost every intersection, totaling hundreds if not thousands across the city. So it is not terribly surprising that the city found a smaller number of accidents in a very small number of flashing crosswalks, while there were a larger number of accidents in the much larger number of crosswalks at intersections.

To correctly compare the safety of flashing crosswalks with those at intersections, one needs to know the accident rates for each type of crosswalk.

Under a more appropriate methodology the proper conclusion is that the "flashing crosswalks" have a much higher accident rate than non-flashing crosswalks -- the exact opposite of the City's conclusion. Thus, a poor quantitative analysis can do more to mislead than to clarify.  This clear and simple example will be part of my graduate seminar quantitative methods of  policy analysis the next time I teach it.