Recently a paper I co-authored came back with three reviews from a reputable and respected journal, one of which was very bad. By which I don’t mean that the reviewer did not like the paper (although they absolutely did not), but rather that is was just a horribly written review. The entire review was less than 250 words long and consisted of eight bullet points which I will summarize below (and these genuinely are the extent of the reviewers critiques):
1) “Your hypothesis is flawed”. We had no hypothesis, which makes this statement a bit puzzling to say the least, particularly when no explanation whatsoever of those flaws were provided.
2) “Your work is not innovative” Again absolutely no explanation of why this is the case.
3) “This sentence is inconsistent” No explanation of inconsistent with what, or why it is inconsistent.
4) “You should discuss this thing” Which we already discuss in some detail.
5) “You failed to cite the right papers”. Without providing any idea of what the ‘right’ papers are how can the authors possibly know what the reviewer thinks is missing? Are we supposed to just randomly add new literature in the hopes of hit on those papers the reviewer (secretly) thinks are important?
6) “The use of the second person is sign of poor quality”. As opposed to a perfectly reasonable, and easily replaceable, stylistic choice?
7) “Don’t cite papers that are not published yet”. Despite the fact the Journal guidelines expressly allow you to do just that.
8) “Your reference list is incomplete”. Fair enough they got us on that one.
The reviewer recommended rejection. As it happens the other two reviews, while both disappointingly brief, liked the paper and recommended very minor revisions. However, it is entirely possible that based on this ’review’ that this paper could have been rejected. I believe there is something fundamentally wrong a paper being rejected based on the subjective, unsupported assertions of a very unprofessional reviewer.
So how do we deal with this? Well for this paper I think the solution is easy, we simple state to the handling editor that we do not believe it is possible to meaningfully engage with such an unconstructive assertions and therefore, apart from fixing the reference list, we will simply ignore this review (it deserves nothing more). In more general terms I think we need a change in peer-review culture.
When a handling editor receives a review like this I believe the correct thing to do is thank the reviewer, but clearly state to the reviewer that the review is not of sufficient rigour or quality (explaining why) and therefore will not be used in the peer-review process.
With reviewers I think that all is really needed is for us to think about what sort of reviews we want to receive and make sure that if we accept the invitation to review a paper that we provide one of the type of quality we ourselves would wish to receive. My personal checklist for a quality review would include:
1) Don’t make arguments by assertion. If, for example, you think a paper is not novel enough point out where the ideas have been presented before, just saying the paper is “not novel” is entirely unconstructive and provide no help for the authors in improving the manuscript.
2) If you think literature is missing say what that literature is and why it matters.
3) Acknowledge that just because you don’t like a particular approach that does not make it wrong.
4) Provide examples. If, for example, you don’t think the conclusions match the findings, provide illustrative examples of where this is the case.
5) Don’t ask the authors to add lots of new text/references, when the paper is already at the word count/reference count limit.
6) Don’t ask the author to cite your papers, unless doing so would change the paper in some substantive way. If your work is cited but the paper is fine without it, sorry but that is just tough cheese. You should not try and strong arm others into cite your work.
I would be particularly interested to hear others reviewing does and don’t.