26 September 2011

Gatekeeping at GRL? You be the Judge

So imagine that you are an editor at Geophysical Research Letters, a middle tier scientific journal.  Let us further suppose that you receive a very straightforward analysis of trends in tropical cyclone landfalls around the world which finds no upwards trends. As editor you decide to send the paper our for peer review and you get the following responses from two reviewers, both of whom find the paper publishable (emphases added):
Reviewer #1

This paper presents a global database of hurricane-force, landfalling tropical cyclones. Construction of such a database is of significant importance and publication is recommended for this paper after consideration of some relatively minor issues, which are detailed below.

Reviewer #2

The work seems essentially sound and useful to the community but lacks in-depth analysis and illustration. It does confront the issue of continued misrepresentation by some of the impact of “climate change” on presently experienced insurance and other losses from tropical cyclones. For that reason it is perhaps (just) publishable but claims of a new homogeneous database (based on JTWC outside of the US) are grossly over-stated as there is much work needed before that can be genuinely claimed. This is especially so in regard to intensity, which the authors treat fairly simplistically in any case. I would like to see that aspect down-played and perhaps the title adjusted to read “Towards a homogeneous database …” or some such.
Both sets of general comments were followed by about a dozen or so minor recommendations on typographical errors, requests for small points of clarification and word emphasis changes.

So you have two reviews that find the paper publishable, one recommending publication and the other coming down on the side of finding the paper "publishable" but certainly not enthusiastically.

As the editor what would you do?
A) Provisionally accept the paper pending a revision that meets the editor's judgment of responsiveness
B) Provisionally accept the paper pending re-review by the two reviewers
C) Reject the paper
D) Reject the paper and tell the authors that any reconsideration of the paper would have to be accompanied by a detailed response to the two reviewers followed by selection of new reviewers and a restart of the review process
If you picked (D) then you too can be an editor at GRL.  Here is what our editor, Noah Diffenbaugh at Stanford, said about our paper in rejecting it:
Thank you for submitting the manuscript "A homogeneous database of global landfalling tropical cyclones" (2011GL049419) to Geophysical Research Letters. Based on the review, I believe that the article requires a major revision, and therefore I cannot accept this version of the manuscript for publication (please see Editorial Policies for major revisions at http://www.agu.org/pubs/pdf/Editorial_GRL.pdf).

You will see that the review (comments enclosed below) found promise in the manuscript, and indicates that a revised manuscript might indeed meet GRL criteria.
Diffenbaugh explains that he "believes" that the paper needs a "major revision" but did not explain what he meant by that.  Further, Diffenbaugh has chosen not to respond to my emails asking for clarification as to what he sees as needing "major revision." However, the chief editor for GRL, Eric Calais of Purdue University, was very responsive when I contacted him and agreed to take a look at this case.  Here is what I sent to him (I fixed a typo, and emphasis added):
Here for ease of access are the reviews to our paper (below and attached).

The first reviewer says: "publication is recommended for this paper after consideration of some relatively minor issues" -- clearly no "major issues" there. The second reviewer asks for a change in the title and then offers 13 specific suggestions for punctuation, wording or emphasis. Nothing has been raised remotely in line with the GRL criteria for "major revision" by either reviewer.

The editor, Noah Diffenbaugh, mischaracterizes the reviews in his letter to us as follows: "You will see that the review (comments enclosed below) found promise in the manuscript, and indicates that a revised manuscript might indeed meet GRL criteria."  This is incorrect in two ways -- First, there are two reviews not a single review, and second, neither review says anything close to his statement "that a revised manuscript might indeed meet GRL criteria."  To the contrary, both reviews found the paper publishable.

Having served for many years one the editorial board of various journals, I am aware that editors can make mistakes for many simple reasons. In this case an error seems to have been made, and I ask that you take a close look at the details of this situation and provide me with your independent judgment -- not on the general GRL process, but on the substance of the matter, and specifically the editor's invocation of the notion of "major revisions" to reject a paper that received two judgments of publishable from the reviewers and only minor suggestions for changes.

Many thanks from Boulder,

Roger
 Calais explained the decision to me as follows:
Dear Roger,

I reviewed in detail the editorial process your paper went through and found that the process went according to GRL policies. The editor's decision was to reject your paper with encouragement to resubmit it, after revisions that account for the reviewers' comments. GRL has not been using "major revisions" for several years now, please read our 2010 EOS editorial at http://www.agu.org/pubs/pdf/Editorial_GRL.pdf. Also, the decision letter you received from the editor is the standard one we send to all authors for similar decisions.

Thank you for your interest in GRL.

Regards,
Eric
I responded to his suggestion that our paper was rejected based on a need for "major revisions" by asking for specific clarification on this point, as I could identify no such recommendation from the two reviewers, nor did the editor tell us what such revisions might entail (emphasis added):
Eric-

Many thanks for your prompt reply ...  but I remain confused.

The EOS editorial that you point to explains a major revision as "if the reviews point to a need for additional analyses, simulations, or other significant changes to support purported high-impact results or implications."  No such requests were made of our paper by the reviewers.  The paper was in fact judged publishable by both reviewers who together recommended only very minor changes, yet it was rejected anyway.

As I explained to Noah, responding to the reviews would take about a day (and perhaps even less time).  He did not provide any guidance to us whatsoever on what "major revisions" he would expect from us based on the reviews.

Can you specifically point  to the "major revision" that GRL sees as necessary based on the reviews?  We will be in a better position to understand what is meant by a "major revision" if you would specifically point to which of the reviewers' recommendations qualify as "major."

Many thanks,

Roger
Calais once again promptly responded to my request.  His response however was a bit odd.  He cited the reviewers request that we change the title by adding "Toward" to it as the basis for rejecting the paper as needing major revisions.  After reciting part of Reviewer #2's comments Calais explains to me:
After consulting again with my editor for Climate, I believe that the additional work needed to address the reviewer's comments are beyond minor revisions. Given the significant changes expected, a revised version of your paper would need to be reviewed again. Hence the decision to reject and encourage resubmission. Again, this is a fairly common procedure at GRL.
I responded as follows:
Thanks Eric,

The reviewer suggested in the context of this comment that we change our title by adding "Toward ..." at the beginning.  Not so major ...  A far more appropriate response given the reviews would have been a provisional acceptance pending a re-review by the two reviewers.

In any case, we shall take our work elsewhere, as I have lost all confidence in GRL as a scientific journal. Just FYI, in the interest of transparency, I shall soon publish the reviews and your several responses to my concerns on my blog so that this sort of behavior can get a broad airing.

Thanks again for your prompt responses,

Roger
Calais responded:
You are most welcome, Roger.

Eric
We will of course be submitting our paper elsewhere, and I suppose I won't be invited to any more AGU meetings ;-)  Here is the abstract of our paper.  If you'd like a copy, just send me an email:
A homogeneous database of global landfalling tropical cyclones

Jessica Weinkle* and Roger Pielke, Jr.
Center for Science and Technology Policy Research, University of Colorado, 1333 Grandview Ave, Campus Box 488, Boulder, Colorado 80309

Abstract
In recent decades, economic damage from tropical cyclones (TCs) around the world has increased dramatically. Scientific literature published to date is strongly suggestive that the increase in losses can be explained entirely by increasing wealth in locations prone to tropical cyclone landfalls. However, no homogenized dataset of tropical cyclone landfalls has been created. We have constructed such a homogenized global landfall TC database. We find no long-term global trends in the frequency or intensity of landfalling TCs for the period with reliable data, providing very strong support for the conclusion that increasing damage around the world over the period(s) of record can be explained entirely by increasing wealth in locations prone to TC landfalls, and adding confidence in the fidelity of economic normalization analyses.

58 comments:

boballab said...

Hmm lets see it's not acceptable for publication of your paper because of some typos and adding a word to the title.

However GRL finds it acceptable to publish online a paper with major errors in it and then let the author change the galley proofs afterwards:
He gets a ratio of about 20:1 for non-radiatively forced (i.e. non-cloud) temperature changes versus radiatively (mostly cloud) forced variations. If that 20:1 number is indeed good, then we would have to agree this is strong evidence against our view that a significant part of temperature variations are radiatively forced. (It looks like Andy will be revising this downward, although it’s not clear by how much because his paper is ambiguous about how he computed and then combined the radiative terms in the equation, below.)

But the numbers he uses to do this, however, are quite suspect. Dessler uses NONE of the 3 most direct estimates that most researchers would use for the various terms. (A clarification on this appears below). Why? I know we won’t be so crass as to claim in our next peer-reviewed publication (as he did in his, see The Ugly, below) that he picked certain datasets because they best supported his hypothesis.

http://www.drroyspencer.com/2011/09/the-good-the-bad-and-the-ugly-my-initial-comments-on-the-new-dessler-2011-study/

I wonder how that happened...

ABC NEWS WATCH said...

Try Remote Sensing!

ABC NEWS WATCH said...

Cartoons showing how the peer review now works available at:

http://abcnewswatch.blogspot.com/2011/09/peer-review-alive-and-well-despite-best.html

(done with apologies to the original artist "Nick")

Jonathan said...

This looks like cock up rather than conspiracy to me, but sadly it's too late to check now. The letters are phrased oddly to my eyes, but I think I would just have made the minor changes requested and resubmitted to see what happened. Not really an option any more...

PaulM said...

I think this journal just has a different approach from most. They want a fast turnround of papers. So if the changes are not minor they reject and ask you to submit again. I'm not defending their policy, but I think that's what it is.

Roger Pielke, Jr. said...

-5-PaulM

Thanks, I agree with their "major revisions" policy -- it makes sense. In this case there was nothing "major" about anything we were asked to do.

When I asked the editor to explain he did not reply, and when asked the chief editor pointed to the request to change the title as the basis for the rejection ... not good ;-)

KnR said...

How is it possible to need a major revision when they can offer no clear idea beyond a word change as to what needs changing. Surely that means it can't actual be major or the reason would be a easy to outline ?

Jacob B said...

I'm sorry but doesn't "A homogeneous database of global landfalling tropical cyclones" differ substantially from "Toward A homogeneous database of global landfalling tropical cyclones"?

The first implies a completed work having established a homogenous database. The second implies work still needs to be done and that the database is currently incomplete, which it appears to me is the gist of why they have booted you?

Roger Pielke, Jr. said...

-8-Jacob B

Maybe so, but if so they should have said so.

Instead, I am assuming that the reviewer is hinting at the broader need for a full-scale reanalysis of the data along the lines being done here in the US for the past decade or so, before one can make a claim of having a "homogenous database." Fair enough, if so. Hence the reviewer's request to redo the title was made, not to redo the analysis.

I don't think the title change would much bother us and it wouldn't make any difference for how the landfall data is evaluated in the context of economic normalization studies.

nigguraths said...

Journals are usually circumspect about publishing papers with a title such as yours, simply on the basis of concerns that it could become established, qua its place in the pantheon, on the strength of its title, irrespective of whether the content of the paper substantiates what the title claims.

The reviewer who recognized this, (is he correct - you be the judge of that), correctly recommended publication based on title changes.

If the editor rejected publication, then it suggests that he let his own biases influence his judgement.

Gerard Harbison said...

The editors' decision would certainly be anomalous in my field. Nature is sometimes this picky, but it can afford to be. GRL has an impact factor of 3.5, not high enough to get ideas beyond its station.

The editor backing up the associate editor is just standard bureaucratic solidarity.

Is there a box where the referees can indicate whether they ask for 'minor revisions' or 'major revisions'? I know this is often a fairly arbitrary distinction, but if one of them specifically said he was asking for a major revision, then that might be part of the rationale for re-review.

Roger Pielke, Jr. said...

-11-Gerard Harbison

Thanks ... if there is such a checked box it was not shared with us, nor did the editors mention anything of the sort to us.

Of course the text of the reviews says anything but "major revisions" so even if such a box existed and had been checked, I'd expect that the editors would either ask the reviewers to clarify (explain) or used their own judgment given the reviews.

In my experience, with one very positive review and one with some questions but coming down saying "publishable", the appropriate response would have been at worst revise and re-review by the same reviewers.

John M said...

My experience with peer review also is inconsistent with this behavior.

If I had been treated this way by an editor, my only conclusion would be that

1)I must have spilled coffee on him at a conference in the past

2)He or someone he knows just doesn't want the work published.

Further, I would have concluded there ain't no way, no how he would ever approve anything that was resubmitted.

As far as the chief editor's response?

What did those 19th century wagon trains used to do again? You know, when attacked?

Roger Pielke, Jr. said...

-13-John M

Maybe for fun we should revise and send it back to GRL ;-)

Carrick said...

Roger this is a tricky situation for the editors, because papers that involve to significant of revision (a judgement call by the editors and reviewers) might get allowed in without all of the changes going through the same rigor of peer review.

My recommendation would have been to modify it as suggested and send it back in for reconsideration.

n.n said...

The limited, circumstantial evidence would not support a conspiracy. However, as you are a professional in the field, you would know better if this behavior was pervasive and to what extent.

You should consider a market-based approach. You and others who feel there is an unjustified bias or otherwise inconsistent treatment should publish your own scientific journal. Inform your peers of your enterprise, its goals, and permit them to pass judgment on the value it presents.

If your perceptions about the existing market are correct, then it is likely others will support your efforts, and this will either force your competitors to change or be rendered obsolete.

Whatever the case may eventually prove to be, it would be helpful to gather evidence to either prove or disprove the anecdotal experience.

Of all human endeavors, it is highly undesirable for scientific enterprises to be subverted through special interests.

Richard Feynman, the Nobel, 'It's a pain in the neck.'

What do you think, is Feynman correct about the incestuous nature of "clubs"?

In any case, competing interests are the most natural and optimal mechanism to keep the honest people honest, and to hold others accountable for their actions.

Roger Pielke, Jr. said...

-16-n.n.

Thanks ... I have not alleged a conspiracy. The decision is odd, certainly. Maybe just bad luck.

But since GRL says the process worked as it was supposed to, then I am sure that they would have no problem allowing people a look inside the peer review process as I have offered here.

One would think that given the visibility that the journal has had of late that when the reject a paper because it needs "major revisions" they would at a minimum clearly explain what those needed revisions actually are, lest some blogger document the seemingly arbitrariness of their decision making.

But our paper is a good one, and it will get published eventually ;-)

James Annan said...

There is no question that the GRL editors lean towards the harsh side of review interpretation, not so clear to me that this is really out of the ordinary.

It may depend on what your anticipated revisions were: reviewer 2 says it would take a lot of work to live up to the promise of the title, were you happy to take the alternative route of downplaying it ("towards")? If so, I can't understand why you didn't just do that, and resubmit (perhaps with a covering letter arguing for the same reviewers).

Of course, downplaying the significance of the work would reduce its attractiveness to GRL - I've seen this catch-22 played out often enough before (where the reviewer says tone down the hype to make it acceptable, but the editor says it has to be sexy to be publishable in his journal).

Roger Pielke, Jr. said...

-18-James Annan

Hi James, thanks for stopping by ... an interesting spin you have.

There is nothing here about downplaying our results, clearly the fact that there are no trends in tropical cyclone landfalls worldwide is plenty sexy enough. The reviewer simply has a different standard of "homogeneous database" than we did -- fair enough in my view.

But here is a challenge for you -- please cite one example, just one, of a decision by GRL to reject a paper given two reviews that deem the paper publishable. Since you say this is not out of the ordinary, I'd guess you can support such claims pretty easily, since you've seen it often enough before, and with supporting evidence ...

Thanks!

n.n said...

-17- Roger Pielke, Jr.

It is not only odd, but it is also not helpful. Since your paper was not rejected by the initial reviewers, it would seem reasonable for the editors to at least document their objections. It would be prudent to do so in the interest of furthering scientific knowledge.

No, I would not make the allegation of conspiracy, nor would I attribute it to you. There is only limited, circumstantial evidence to support such a claim, and it is insufficient to justify an accusation. I would, however, be interested in characterizing this behavior. Especially in light of similar behavior within the journalistic arm of scientific enterprise and its institutions.

Well, I am not a member of your community; but, similar questions arise throughout society. There is no reason to assume that it does not occur within the scientific community. In fact, earlier anecdotal evidence was offered by Feynman, which, in addition to current observations, would, at the minimum, justify a skeptical outlook.

Whether you decide to challenge them or not, the market-based approach is an incredibly effective method for arbitration in any human endeavor. I'm sure you already know this, as I imagine it is in part the reason for this blog.

Good luck in your effort to be the honest broker. It is not an easy aspiration, but it is worth pursuing.

Salamano said...

I'm surprised they didn't have a problem with your use of the word "entirely" in the abstract...as few things in Climate Science have been demonstrably proven as 'entirely' explainable by any one single entity...or is this the first?

Richard Tol said...

@Roger
The verdict by Referee 2 is clearly negative: You oversell.

Overselling may be countered in less than an hour by changing a sentence here and there. However, the definition of "major revision" is in the substance of the revision, rather than in the effort made.

The editor's decision "revise-and-resubmit" is thus appropriate. As GRL does not do that, "reject-but-resubmit" follows.

Instead of rejoicing, you decided to pick a fight. The decision thus became "reject-and-never-come-back".

Jos said...

Hi Roger,

I have published several papers in GRL before and after some changes GRL made to speed up the review proces.

First of all, my experience is that recently papers are more often "formally rejected" based on review comments that do not take that much time to incorporate - and which before the changes to the editorial proces would have not resulted in a rejection.

Nevertheless, my experience is also that in all those cases after making some (necessary/requested) changes the resubmission quickly got accepted.

Now, the second referee notes that "homogenization with regard to intensity is done in a rather simplistic way". which appears a serious point of criticism even though the referee is not specifically asking to do some more work.

Obviously, if either the referee or the editor feels that there is a need to clarify this they should have asked you to either discuss why you think this is not so much of an issue (possibly including a summary of that discussion in the paper) or ask you to do some additional work on this issue. If I were the editor I would have asked you address this one way or another.

Arguably neither the editor nor the referee asked for this - even after your inquiries, hence you have a valid point.

Nevertheless, why not simply accept that this is a bit weird but do some additional work or discuss the issue of intensity in the paper - showing your willingness to seriously consider the referee comments - and then resubmit it?

J.

ob said...

OK, what does GRL say about its publishing

"High impact innovative results", "Results with immediate impact", "methods that make new science advance possible".

What does Rev2 say about your manuscript:

"lacks in-depth analysis and illustration" and "claims are grossly over-stated".

Hm. I guess, the editorial decision sounds reasonable.

Roger Pielke, Jr. said...

-22-Richard Tol

Thanks, interesting perspective. OK you are the editor, I ask you to explain what "major revisions" are necessary .. your response?

Roger Pielke, Jr. said...

-23-Jos

Thanks. My experience with GRL is different than yours, regrettably.

You ask a fair question: "why not simply accept that this is a bit weird but do some additional work or discuss the issue of intensity in the paper - showing your willingness to seriously consider the referee comments - and then resubmit it?"

The answer is simple -- I don't trust GRL and have no expectation of a fair re-review.

Why not? (a) Our editor (Diffenbaugh) wrote a sloppy letter to us and then refused to reply to my emails asking perfectly fair questions about structuring a revision, (b) the chief editor (Calais) when asked refused or was unable to clarify what "major revisions" were necessary.

Had either (a) or (b) turned out differently I suspect we would have simply revised and sent back in. Our decision not to resubmit for re-review had nothing to do with our unwillingness to consider the reviews (in fact, their useful comments have been incorporated into a revision!), rather it had everything to do with the weird editorial process.

I have no need to publish in GRL -- there are plenty of middle tier journals out there;-) If this sort of weird editorial process occurred at any other journal I'd walk away from there also. In my experience, which covers 100+ peer reviewed papers, this stands out as one of the weirdest, and that is enough for me.

Roger Pielke, Jr. said...

-21-Salamano

Neither reviewer had an issue with this, rightly so.

Roger Pielke, Jr. said...

-22-Richard Tol

Also, your definition of "major revision" is at odds with that used by GRL, which is indeed focused on the effort made:

"[M]anuscripts are routinely
declined if the reviews point to a need for
additional analyses, simulations, or other
significant changes to support purported
high-impact results or implications. However,
for those submissions that show promise
of reaching GRL’s criteria, authors are
encouraged to resubmit following necessary
revisions. While “resetting the clock”
on manuscripts that require major revisions
reduces the time-to-publication dates,
the policy is motivated not by a desire to
make the GRL editorial process appear as
rapid as possible but rather by a desire to
make the process be as rapid as possible."

Richard Tol said...

-25- Roger
There are all sorts of reasons why the editor did not reply. For all we know, he may on a field trip in hospital, or working on a proposal.

Roger Pielke, Jr. said...

-29-Richard Tol

Sure, maybe aliens picked him up.

But, once again you are wrong. Fortunately, aliens have not picked him up -- He has been in discussions with the Chief Editor on this matter, as you can clearly see from the exchange reproduced above. He simply chose not to respond to me.

Still awaiting your response to #25 above.

Richard Tol said...

-25, 30- Roger
Diffenbaugh is in "selective reply" mode. He's alive and online so, but you are clearly not his main priority.

I read the report of referee 2 as "oversell". That's a major revision in my journal, even though it takes little effort.

Roger Pielke, Jr. said...

-31-Richard Tol

Thanks for your walk back.

If an editor at a "rapid publication" journal does not have the time or priorities to respond to his authors within a week, then he should probably look for an editorial position at a "slow publication" journal. A non-response is simply unacceptable. I am surprised that you defend it.

Your journal (which is what?) has a different process and standards than GRL. Fine, but you err in applying your journal's standards to another journal and then criticizing me according to standards under which our paper was not judged.

Thanks!

Ric Werme said...

Being a fulltime software engineer and unemployable as a climate scientist, I must say I'm rather disappointed in Diffenbaugh's actions.

My first contact with him was direct - he was a speaker at American Mensa's Annual Colloquium a couple years ago. I went because Jim Hansen was the keynote speaker and I didn't want him recruiting intelligent but uninformed people to his army. Fortunately Hansen left for the Capitol Hill Power Plant protest a day early to miss the snowstorm. In Atlanta. At the beginning of March.

Diffenbaugh spoke about his modeling that projected that the American Southwest would undergo a change from a summertime semi-permanent low pressure system (caused due to heat) to a high pressure system, bringing drier summers and less monsoon rain. Near as I could figure, surrounding areas warming up might be able to produce descending air in the center, but I wasn't convinced that can happen.

Since then he's found a spot at Stanford's Woods Institute for the Environment, and found collaborators, and widescale impact - "Tropical regions may see the most dramatic changes first, but wide swaths of North America, China and Mediterranean Europe are also likely to enter into a new heat regime by 2070, according to the study.", see http://news.stanford.edu/news/2011/june/permanent-hotter-summers-060611.html

I'm still not convinced.

I am surprised that he's an editor at GRL. I'd expect that you would want people who have longer and wider experience, and have ground all their axes before being offered that sort of job. (Like I say, I'm just a software engineer, so I have no right to an opinion.) My guess is that the major revision Diffenbaugh wants would include a sentence like "The worldwide increase in storm intensity supports the expectation that tropical regions are seeing dramatic changes."

Personally, I'd rather read papers that simply report what is seen. (Like I say, ....)

D. Robinson said...

Roger,

From what I have read on the topic I agree with your conclusions. However, the 2nd review is very negative and just north of saying 'do not publish this paper'.

Parse it and read all the qualifiers: "The work seems essentially sound and useful to the community but lacks in-depth analysis and illustration. It does confront the issue of continued misrepresentation by some of the impact of “climate change” on presently experienced insurance and other losses from tropical cyclones. For that reason it is perhaps (just) publishable but claims of a new homogeneous database (based on JTWC outside of the US) are grossly over-stated as there is much work needed before that can be genuinely claimed. This is especially so in regard to intensity, which the authors treat fairly simplistically in any case. I would like to see that aspect down-played and perhaps the title adjusted to read “Towards a homogeneous database …” or some such" (emphasis mine)

The 2nd reviewer nuked you. This reviewer doesn't think you showed strong enough data, and he/she doesn't agree with your conclusions. The review is not unfriendly, but it is very negative.

No offense or negative judgement on the paper intended from me.

Roger Pielke, Jr. said...

-34-D. Robinson

Perhaps so, but the reviewer's rating of the paper -- 3A, strongly suggests otherwise.

What is a 3A?

The A:

"Presentation A : Manuscripts should meet ALL of the following:

*Abstract is succinct (< 150 words), accurate, and comprehensible to a non-specialist
*Manuscript is generally well-written, logically organized, and adequately illustrated
*Figures and tables are understandable and readable (when sized for GRL) English usage and grammar is adequate, with few spelling/typographical errors (please specify any minor fixes)
*Manuscript appears to to GRL's 4-page limit

The 3:

"Science Category 3: The paper is publishable in the refereed literature but is unlikely to become a Category 1 paper. For example:

*It is a scienti cally correct paper but not obviously a signi cant advance in a geophysical field
*A solid paper with little immediate impact on the research of others (e.g., a routine application of a standard research technique, or a new measurement/laboratory method with limited geophysical application)
*A good but basically incremental improvement to existing data sets, models, or instruments"

What is most odd is that the category of 3A basically means "no major revisions" -- the reviewer just wasn't too excited, but found nothing major wrong with it according to these criteria.

Both categories 2 and 4 imply a need for major revisions, yet the reviewer did not select these categories!

In case you are curious, the other reviewer gave us a 1B, where 1 means:

"Science Category 1: The manuscript meets one or more of the following criteria:
*Important new science at the forefront of an AGU discipline
*Innovative research with interdisciplinary/broad geophysical application
*Instrument or methods manuscript that introduces new techniques with important geophysical applications"

Any paper that gets a 1 and a 3 and an A and a B should not be rejected, that seems obvious.

D. Robinson said...

Re: Roger @ 35 - Interesting, the '3A' rating from the 2nd reviewer appears to be way less negative than the text of his/her review.

I would agree that as far as the ratings go there wouldn't seem to be any justification for requesting 'major revisions'.

Roger Pielke, Jr. said...

-36-D. Robinson

Indeed. But in the context of this subdiscipline the reviewer's comments are to be understood not as a critique of our methods, but his view of the state of the data in this field in the context of what he understands as a "homogenous database".

This is of course why he judged the paper publishable in light of his comments and recommended no major revisions.

Marlowe Johnson said...

"Instead of rejoicing, you decided to pick a fight. The decision thus became "reject-and-never-come-back"."

If it had been Gavin or Romm that said that it would be one thing....but Tol?

Ouch....

Doug Proctor said...

They don't have a problem with the paper. They have a problem with you.

I suggest a test: another paper without your name on it, some safe young guy. Then, when the paper is accepted and published, come out with the real authors.

Becoming persona non grata is often the result of asking the inconvenient questions too many times too loudly in too public a place. It's actually fairly easy.

Roger Pielke, Jr. said...

-39-Doug Proctor

You raise an interesting point. If someone accuses GRL of bias in its decision making, the arbitrariness of the process leaves them vulnerable.

A more rigorous (less arbitrary) process provides protection against such accusations.

I have not accused GRL of bias, just a sloppy and inconsistent process, coupled with poor communication in this instance.

Jonathan said...

-35-Roger

You are suddenly introducing very significant new information in this comment. It would have helped enormously if you had put these ratings in your head post.

Roger Pielke, Jr. said...

-41-Jonathan

Yes, editorial decisions get made ;-)

I am glad that you find the extra information useful.

Roger Pielke, Jr. said...

Quick note:

Comments that speculate about the bias or motivation of individuals will continue to be disallowed. Please just don't go there ... Thanks!

Roger Pielke, Jr. said...

In _19_ above I challenged James Annan to provide an example of a paper rejected by GRL that received two reviews saying publish. He is probably asleep by now, but I just received an email from someone who cited an example:

Chip Knappenberger shared information on a paper that was ranked 1A and 2A and was rejected by GRL.

I'll let him share whatever details he wants.

Chip said...

Re: 44,

A few more details…

The subject of our paper was reconstructing surface melt across Greenland back ~225 years. As Roger mentioned, both reviewers liked our initial submission (Reviewer 1: “An excellent paper. Publish almost as is.” Reviewer 2: “The paper is very well written, clear and concise, and makes its point very well. Other than needing to address the four issues below, I recommend that the paper be accepted and published in GRL.”) The editor, himself an expert of Greenland hydro/cryosphere, elected to reject the paper for reasons of his own. Which indicates that there was no real need for the editor to have sent it out for review in the first place.

We responded to the reviewers’ comments and the editor’s comments, and resubmitted our work as a new paper (the same thing Roger has been invited to do). The same editor handled the new submission. We got two new reviewers. One liked it: “I thus recommend the otherwise strong science paper for publication, provided that speculation is removed from the Conclusions section and that it is made clear the reconstruction s not continuous since 1784.” The other thought there was not enough “new” information “The paper is well written, the methodology is clear. But this paper is not enough original, compared to Chylek et al. (2007) and Fettweis et al. (2008), to be published in GRL” The editor decided again to reject the paper and this time we were not invited to resubmit it to GRL.

Ultimately, the paper was published (2 years later) in another AGU journal, Journal of Geophysical Research. When it was published, one of the JGR reviewers, claimed that the publication of the paper represented “the failure of the review process” at JGR (see here for the sordid details).

This shows the power of the Editor in charge of the paper.

-Chip

Jos said...

- 26 -

I can honestly say that I have been on the other side as well. Acting as referee, having serious questions and doubts about papers and then seeing that a paper nevertheless gets published because the authors are capable of convincing the editor that there is no problem without ever providing proper answers to my questions.

And then once published, try to correct things ... nearly impossible.

I have seen all kinds of weird things happen in peer review, much weirder than this - just ask Ross McKitrick - and have seen it happen everywhere. And one of my publications even caused some colleagues to think about "redefining what peer-reviewed literature is" (yes, one of those was mine). I could write a chapter about the unusual things I have experienced over the years (and that covers just about 35 publications).

The peer-review system is just not infallible, but that is nothing new.

Point is, over the years I have developed a pretty thick skin with regard to peer-review. I just don't get that upset anymore (and yes, I am currently in yet another battle) but take a constructive approach trying to figure out how to work - or beat - the system. Trying to fight the system just costs too much energy. As my PhD supervisor once advised: you have to pick your battles carefully.

J.

Jeff said...

Thanks for exposing this continuing behavior. I hope you do get the work published elsewhere soon.

Roger Pielke, Jr. said...

-46-Jos

Thanks ... I don't get much upset either, it is as you suggest part of the deal.

That said I also have no problem airing these issues in public, in fact, I'd hypothesize that the climate science community would be a lot healthier if more of the community would discuss such issues out in the open.

Based on the conversations I've had over the past 24 hours with colleagues about peer review and the many offers from leading experts to comment on our paper, I see no downside. In fact, our paper will be better for it, and with some luck GRL will benefit from a critique of their process.

Thanks!

federico said...

Business as usual, see Climategate. The Editor would risk his job if he accepts your paper, it's that simple.

If I understood correctly, your paper does not confirm the "consensus" view (cyclones getting wilder because of AGW), therefore it must stay out of the AR5 process.

It is clear that your only option is fighting your way through this; I admire your (and your father's) stamina and integrity in such a hostile and politicized research environment.

Jos said...

- 48 -

>>> I'd hypothesize that the climate science community would be a lot healthier if more of the community would discuss such issues out in the open.

I couldn't agree more ...

Good luck!

Jos said...

- 48 -

>>> I'd hypothesize that the climate science community would be a lot healthier if more of the community would discuss such issues out in the open.

I couldn't agree more ...

Good luck!

edwinkite said...

Hi Roger,

AGU wants its journals to turn around papers more quickly. They respond by rejecting papers that have more than slight problems (resetting the clock for processing time). Personally, I score these "administrative rejections" as revisions, rather than resubmissions. I think that is what is going on in your case.

Best wishes,

- Edwin Kite

EliRabett said...

While this is all very amusing, one point has slipped through. Anyone care to name another rapid publication journal in earth science that has a better reputation than GRL.

hro001 said...

Roger,

I stumbled across an interesting parallel experience, as described by Edward Carr:

"[A paper] that has been through 13-odd reviewers over the past 6 months? Well, it came back this morning with that least satisfying of responses: revise and resubmit. The reviewers disagreed on the strengths and weaknesses of the paper, but all of them wanted at least some small changes.

"Now, this is hardly the first time I’ve had to deal with this. Happens all the time. But this review, like a number of others I have encountered in my career, highlights for me the need for strong editors at journals. The email from the journal read, in part:

"[...] Please note that the editors are forwarding these comments as you might find them helpful; individual comments do not necessarily represent the views of the editors, or a consensus regarding the direction of any possible revision."

The latest installment on Carr's adventures in peer-review-land (with links back to the above) is available at:

http://www.edwardrcarr.com/opentheechochamber/2011/10/06/more-tales-of-publication-revision-negotiations-part-2/

And, at the risk of appearing to be shamelessly plugging my own blog, on a somewhat related note, I offer my thoughts on "journal retractions" (which may or may not be contributing to the problems you and Carr have experienced) and the IPCC at:

http://hro001.wordpress.com/2011/10/05/journal-retractions-some-lessons-for-the-ipcc-to-be-learning/

Tom Curtis said...

I'm late to the party, but my comments on this issue can be found here:
http://bybrisbanewaters.blogspot.com/2011/10/much-ado-about-nothing.html

I disagree with Roger Pielke Jnr on this issue, and am notifying him of my comments so he can attempt a rebutal if he wants to.

Roger Pielke, Jr. said...

-55-Tom Curtis

Thanks Tom for alerting me to this post. You have some basic facts wrong -- I dd not claim that my paper should be published, but rather, in the event that an editor judges a paper to require "major revisions" he has an obligation to explicitly state what such revisions would entail, especially if neither reviewer asks for "major revisions." That the editors were unable or unwilling to provide such guidance is the basis of my complaint. Thanks for your interest.

Ali said...

I had a totally different experience. One of the referees said the paper was very good. And the other said it is worthless. I think we have to face these difficulties, to get chopped as shown in the cartoon.

MM Ali

blamannen said...

Roger, contrast your experience with a paper that was published last year in GRL. ONE review was comissioned and a highly partisan one at that. This paper is far below the standards I would expect of GRL: most of the text is a description of satellite sensor characteristics. See: http://onlinelibrary.wiley.com/doi/10.1029/2012GL053611/full
I now wonder if GRL is worth reading.

Post a Comment

Note: Only a member of this blog may post a comment.