How to deal with bad reviews

Recently a paper I co-authored came back with three reviews from a reputable and respected journal, one of which was very bad. By which I don’t mean that the reviewer did not like the paper (although they absolutely did not), but rather that is was just a horribly written review. The entire review was less than 250 words long and consisted of eight bullet points which I will summarize below (and these genuinely are the extent of the reviewers critiques):

1) “Your hypothesis is flawed”. We had no hypothesis, which makes this statement a bit puzzling to say the least, particularly when no explanation whatsoever of those flaws were provided.
2) “Your work is not innovative” Again absolutely no explanation of why this is the case.
3) “This sentence is inconsistent” No explanation of inconsistent with what, or why it is inconsistent.
4) “You should discuss this thing” Which we already discuss in some detail.
5) “You failed to cite the right papers”. Without providing any idea of what the ‘right’ papers are how can the authors possibly know what the reviewer thinks is missing? Are we supposed to just randomly add new literature in the hopes of hit on those papers the reviewer (secretly) thinks are important?
6) “The use of the second person is sign of poor quality”. As opposed to a perfectly reasonable, and easily replaceable, stylistic choice?
7) “Don’t cite papers that are not published yet”. Despite the fact the Journal guidelines expressly allow you to do just that.
8) “Your reference list is incomplete”. Fair enough they got us on that one.

The reviewer recommended rejection. As it happens the other two reviews, while both disappointingly brief, liked the paper and recommended very minor revisions. However, it is entirely possible that based on this ’review’ that this paper could have been rejected. I believe there is something fundamentally wrong a paper being rejected based on the subjective, unsupported assertions of a very unprofessional reviewer.

So how do we deal with this? Well for this paper I think the solution is easy, we simple state to the handling editor that we do not believe it is possible to meaningfully engage with such an unconstructive assertions and therefore, apart from fixing the reference list, we will simply ignore this review (it deserves nothing more). In more general terms I think we need a change in peer-review culture.

When a handling editor receives a review like this I believe the correct thing to do is thank the reviewer, but clearly state to the reviewer that the review is not of sufficient rigour or quality (explaining why) and therefore will not be used in the peer-review process.

With reviewers I think that all is really needed is for us to think about what sort of reviews we want to receive and make sure that if we accept the invitation to review a paper that we provide one of the type of quality we ourselves would wish to receive. My personal checklist for a quality review would include:

1) Don’t make arguments by assertion. If, for example, you think a paper is not novel enough point out where the ideas have been presented before, just saying the paper is “not novel” is entirely unconstructive and provide no help for the authors in improving the manuscript.
2) If you think literature is missing say what that literature is and why it matters.
3) Acknowledge that just because you don’t like a particular approach that does not make it wrong.
4) Provide examples. If, for example, you don’t think the conclusions match the findings, provide illustrative examples of where this is the case.
5) Don’t ask the authors to add lots of new text/references, when the paper is already at the word count/reference count limit.
6) Don’t ask the author to cite your papers, unless doing so would change the paper in some substantive way. If your work is cited but the paper is fine without it, sorry but that is just tough cheese. You should not try and strong arm others into cite your work.

I would be particularly interested to hear others reviewing does and don’t.

2 thoughts on “How to deal with bad reviews

  1. Dave, in many ways you are of course absolutely right. That review should not have been written, once it had been written then it should not have been forwarded to you by the handling editor, and it certainly should not have been the basis of the journal’s decision.

    Three times “should”, just like in a number of places in the Paris Agreement – in places where we would have wanted to see “shall” instead.

    However, as a journal editor, I ask you to consider also the following:

    We all have mostly clear views about academic standards, and, besides differences in details, there is a very solid basis of agreement among us about these. In fact, indirectly, your post is a good expression for just that agreement. Our current peer review system is an expression of it.

    But then, there are these situations… It’s late [in the year][in the day][with respect to some deadline]. We have perhaps not done everything we [wanted][needed] to do. And we have agreed to [be editor of][review] this paper. The deadline has passed, we feel bad because some managing editor has already sent us two reminders, and [he][she] also rather would have wanted to do other things than reminding us.

    You see where I am getting to. That review may not actually get done [with appropriate standards][ever].

    A number of colleagues automatically now turn down all review requests they are getting, even from journals edited by their friends. Sometimes it’s understandable: they are world-famous in some field, and they probably get several requests per day, so it would be physically impossible.

    But others are less overbooked. They turn down the invitations because they [need][want] to finish their own papers, thereby mounting the pressure on the poor buggers who still do reviews with two levers: by not reviewing, and by submitting ever more papers.

    To actually get two reviews for a paper, I often have to send seven or eight invitations.

    So, as others, including Jörn, have pointed out earlier, we are heading towards systemic failure here. I, with my 600+ submissions per year, certainly do not have a near-term solution, and it worries me.

    The only thing I want to ask you here is to sometimes have some [mercy][understanding] for a poor person who did a bad and ill-tempered review some time late at night after a long day somewhere. My suggestion to you would be to politely write back to the handling or chief editor and ask him whether [he][she] really thinks this review did the journal any good. While also overworked, these editors often are very reasonable people (certainly the ones from Regional Environmental Change are, every single one of them).

    PS: Oh, and one more thing from an editor’s point of view: The number of truly bad submissions you receive is staggering. And this year so far, I have received two manuscripts that were fully plagiarized carbon copies of somebody else’s papers. I only mention that in order to help you understand that an editor sometimes touches [his][her] limits…

    • Dear Wolfgang,

      I entirely agree with everything you wrote.I would add that poor quality reviews add a further burden to already over burdened journal editors. I guess my position is that if you are not inclined, or unable to provide a review of the quality you would hope to receive for your own work don’t offer to review the paper. In This particular case, the handling editor clearly put in a lot of work, providing their own constructive comments on the paper and how it could be improved. I would argue that if they had received better reviews this would have been an additional job the editor could have avoided.

      Regarding poor quality submissions, I feel your pain. I do a fair bit of reviewing (I think 24 papers this year, plus all the revisions), and a fair bit of what I review is quite poor*. It is tempting when receiving a poor submission to review to provide a cursory review (why should I put in effort to review a paper that seems to have little effort put into it?). However, I try and provide the same detail of critique for every paper I review. How else can we expect better quality submissions if we don’t provide constructive feedback (even for very poor submissions)?

      Regarding the systemic failure of the peer-review system, it does indeed seem to heading that way and I also am unsure of how to steer it into a more sustainable direction.

      * I think this is in part due to my relatively low position in the academic pecking order (the more established you are the more likely that you will be asked to review ‘important’ or ‘good’ papers). It would be interesting to know how this dynamic plays out. For example, is it mainly junior researchers reviewing the work of other junior researchers?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s