26 September 2011

Gatekeeping at GRL? You be the Judge

So imagine that you are an editor at Geophysical Research Letters, a middle tier scientific journal.  Let us further suppose that you receive a very straightforward analysis of trends in tropical cyclone landfalls around the world which finds no upwards trends. As editor you decide to send the paper our for peer review and you get the following responses from two reviewers, both of whom find the paper publishable (emphases added):
Reviewer #1

This paper presents a global database of hurricane-force, landfalling tropical cyclones. Construction of such a database is of significant importance and publication is recommended for this paper after consideration of some relatively minor issues, which are detailed below.

Reviewer #2

The work seems essentially sound and useful to the community but lacks in-depth analysis and illustration. It does confront the issue of continued misrepresentation by some of the impact of “climate change” on presently experienced insurance and other losses from tropical cyclones. For that reason it is perhaps (just) publishable but claims of a new homogeneous database (based on JTWC outside of the US) are grossly over-stated as there is much work needed before that can be genuinely claimed. This is especially so in regard to intensity, which the authors treat fairly simplistically in any case. I would like to see that aspect down-played and perhaps the title adjusted to read “Towards a homogeneous database …” or some such.
Both sets of general comments were followed by about a dozen or so minor recommendations on typographical errors, requests for small points of clarification and word emphasis changes.

So you have two reviews that find the paper publishable, one recommending publication and the other coming down on the side of finding the paper "publishable" but certainly not enthusiastically.

As the editor what would you do?
A) Provisionally accept the paper pending a revision that meets the editor's judgment of responsiveness
B) Provisionally accept the paper pending re-review by the two reviewers
C) Reject the paper
D) Reject the paper and tell the authors that any reconsideration of the paper would have to be accompanied by a detailed response to the two reviewers followed by selection of new reviewers and a restart of the review process
If you picked (D) then you too can be an editor at GRL.  Here is what our editor, Noah Diffenbaugh at Stanford, said about our paper in rejecting it:
Thank you for submitting the manuscript "A homogeneous database of global landfalling tropical cyclones" (2011GL049419) to Geophysical Research Letters. Based on the review, I believe that the article requires a major revision, and therefore I cannot accept this version of the manuscript for publication (please see Editorial Policies for major revisions at http://www.agu.org/pubs/pdf/Editorial_GRL.pdf).

You will see that the review (comments enclosed below) found promise in the manuscript, and indicates that a revised manuscript might indeed meet GRL criteria.
Diffenbaugh explains that he "believes" that the paper needs a "major revision" but did not explain what he meant by that.  Further, Diffenbaugh has chosen not to respond to my emails asking for clarification as to what he sees as needing "major revision." However, the chief editor for GRL, Eric Calais of Purdue University, was very responsive when I contacted him and agreed to take a look at this case.  Here is what I sent to him (I fixed a typo, and emphasis added):
Here for ease of access are the reviews to our paper (below and attached).

The first reviewer says: "publication is recommended for this paper after consideration of some relatively minor issues" -- clearly no "major issues" there. The second reviewer asks for a change in the title and then offers 13 specific suggestions for punctuation, wording or emphasis. Nothing has been raised remotely in line with the GRL criteria for "major revision" by either reviewer.

The editor, Noah Diffenbaugh, mischaracterizes the reviews in his letter to us as follows: "You will see that the review (comments enclosed below) found promise in the manuscript, and indicates that a revised manuscript might indeed meet GRL criteria."  This is incorrect in two ways -- First, there are two reviews not a single review, and second, neither review says anything close to his statement "that a revised manuscript might indeed meet GRL criteria."  To the contrary, both reviews found the paper publishable.

Having served for many years one the editorial board of various journals, I am aware that editors can make mistakes for many simple reasons. In this case an error seems to have been made, and I ask that you take a close look at the details of this situation and provide me with your independent judgment -- not on the general GRL process, but on the substance of the matter, and specifically the editor's invocation of the notion of "major revisions" to reject a paper that received two judgments of publishable from the reviewers and only minor suggestions for changes.

Many thanks from Boulder,

 Calais explained the decision to me as follows:
Dear Roger,

I reviewed in detail the editorial process your paper went through and found that the process went according to GRL policies. The editor's decision was to reject your paper with encouragement to resubmit it, after revisions that account for the reviewers' comments. GRL has not been using "major revisions" for several years now, please read our 2010 EOS editorial at http://www.agu.org/pubs/pdf/Editorial_GRL.pdf. Also, the decision letter you received from the editor is the standard one we send to all authors for similar decisions.

Thank you for your interest in GRL.

I responded to his suggestion that our paper was rejected based on a need for "major revisions" by asking for specific clarification on this point, as I could identify no such recommendation from the two reviewers, nor did the editor tell us what such revisions might entail (emphasis added):

Many thanks for your prompt reply ...  but I remain confused.

The EOS editorial that you point to explains a major revision as "if the reviews point to a need for additional analyses, simulations, or other significant changes to support purported high-impact results or implications."  No such requests were made of our paper by the reviewers.  The paper was in fact judged publishable by both reviewers who together recommended only very minor changes, yet it was rejected anyway.

As I explained to Noah, responding to the reviews would take about a day (and perhaps even less time).  He did not provide any guidance to us whatsoever on what "major revisions" he would expect from us based on the reviews.

Can you specifically point  to the "major revision" that GRL sees as necessary based on the reviews?  We will be in a better position to understand what is meant by a "major revision" if you would specifically point to which of the reviewers' recommendations qualify as "major."

Many thanks,

Calais once again promptly responded to my request.  His response however was a bit odd.  He cited the reviewers request that we change the title by adding "Toward" to it as the basis for rejecting the paper as needing major revisions.  After reciting part of Reviewer #2's comments Calais explains to me:
After consulting again with my editor for Climate, I believe that the additional work needed to address the reviewer's comments are beyond minor revisions. Given the significant changes expected, a revised version of your paper would need to be reviewed again. Hence the decision to reject and encourage resubmission. Again, this is a fairly common procedure at GRL.
I responded as follows:
Thanks Eric,

The reviewer suggested in the context of this comment that we change our title by adding "Toward ..." at the beginning.  Not so major ...  A far more appropriate response given the reviews would have been a provisional acceptance pending a re-review by the two reviewers.

In any case, we shall take our work elsewhere, as I have lost all confidence in GRL as a scientific journal. Just FYI, in the interest of transparency, I shall soon publish the reviews and your several responses to my concerns on my blog so that this sort of behavior can get a broad airing.

Thanks again for your prompt responses,

Calais responded:
You are most welcome, Roger.

We will of course be submitting our paper elsewhere, and I suppose I won't be invited to any more AGU meetings ;-)  Here is the abstract of our paper.  If you'd like a copy, just send me an email:
A homogeneous database of global landfalling tropical cyclones

Jessica Weinkle* and Roger Pielke, Jr.
Center for Science and Technology Policy Research, University of Colorado, 1333 Grandview Ave, Campus Box 488, Boulder, Colorado 80309

In recent decades, economic damage from tropical cyclones (TCs) around the world has increased dramatically. Scientific literature published to date is strongly suggestive that the increase in losses can be explained entirely by increasing wealth in locations prone to tropical cyclone landfalls. However, no homogenized dataset of tropical cyclone landfalls has been created. We have constructed such a homogenized global landfall TC database. We find no long-term global trends in the frequency or intensity of landfalling TCs for the period with reliable data, providing very strong support for the conclusion that increasing damage around the world over the period(s) of record can be explained entirely by increasing wealth in locations prone to TC landfalls, and adding confidence in the fidelity of economic normalization analyses.