Everyone gets rejected. And it never stops being painful not matter how successful or how long you have been in the business. Some of this is inevitable; not everyone is above average. But some of it isn’t. I thought that I would offer some ‘dos’ and ‘don’ts’ for reviewers out there to improve the process and save some hurt feelings, when possible. Some are drawn from personal experience; others, more vicariously. I have done some of the “don’ts” myself, but I feel bad about it. Learn from my mistakes.
First, and I can’t stress this enough, READ THE F*CKING PAPER. It is considered impolite by authors to reject a paper by falsely accusing it of doing THE EXACT OPPOSITE of what it does. Granted, some people have less of a way with words than others and are not exactly clear in their argumentation. But if you are illiterate, you owe it to the author to tell the editors when they solicit your review. It is okay – there are very successful remedial programs they can recommend. Don’t be ashamed.
Second, and related to the first, remember the stakes for the author. Let us consider this hypothetical scenario. In a safe estimate, an article in a really top journal will probably merit a 2-3% raise for the author. Say that is somewhere around $2000. Given that salaries (except in the University of California System) tend to either stay the same or increase, for an author who has, say, 20 years left in his/her career, getting that article accepted is worth about $40,000 dollars. And that is conservative. So you owe it more than a quick scan while you are on the can. It might not be good, but make sure. Do your job or don’t accept the assignment in the first place. (Sorry, I don’t usually like scatological humor but I think this is literally the case sometimes.)
Third, the author gets to choose what he/she writes about. Not you. He/she is a big boy/girl. Do not reject papers because they should have been on a different topic, in your estimation. Find fault with the the paper actually under review to justify your rejection.
Fourth, don’t be a b*tch. Articles should be rejected based on faulty theory or fatally flawed empirics not a collection of little cuts. Bitchy grounds include but are not limited to – not citing you, using methods you do not understand but do not bother to learn, lack of generalizability when theory and empirics are otherwise sound. The bitchiness of reviews should be inversely related to the audacity and originality of the manuscript. People trying to do big, new things should be given more leeway to make their case than those reinventing the wheel.
Fifth, don’t be an a**hole. Keep your sarcasm to yourself. Someone worked very hard on this paper, even if he/she might not be very bright. Writing “What a surprise!”, facetiously, is a dick move. Rejections are painful enough. You don’t have to pour salt on the wound. Show some respect.
Sixth, remember that to say anything remotely interesting in 12,000 words is ALMOST IMPOSSIBLE. Therefore the reviewer needs to be sympathetic that the author might be able to fix certain problems where he/she given more space to do so. Not including a counterargument from your 1986 JOP article might not be a fatal oversight; it might have just been an economic decision. If you have other things that you would need to see to accept an otherwise interesting paper, the proper decision is an R&R, not a reject. Save these complaints for your reviews of full-length book manuscripts where they are more justifiable.
Seventh, you are not a film critic. Rejections must be accompanied by something with more intellectual merit than “the paper did not grab me” or “I do not consider this to be of sufficient importance to merit publication in a journal of this quality.” This must be JUSTIFIED. You should explain your judgment, even if it is something to the effect of, “Micronesia is an extremely small place and its military reforms are not of much consequence to the fate of world politics.” Even if it is that obvious, and it never is, you owe an explanation.
Excellent stuff here. The only additions I would make are: do it in a timely manner (I was tempted to follow the old PS article joke and send a paper of mine a birthday card as it was under review for more than 12 months); and send clear signals. If the article is not redeedable in its current form, reject, not R&R. If the paper is very good and only needs a few tweeks, accept, not R&R. R&R should be applied to situations where the paper should be revised and should be resubmitted. It is not just a legitimate default answer. Three possible outcomes (A, R&R, R)–pick one.
Yes, great point. No one likes a paper tease.
great post. Your suggestions are all the more applicable given the anonymity that the reviewer enjoys. Anonymity is not a licence to say things that you wouldn’t normally say directly to somebody.
That’s a really important point. I still read through every review before sending and think, How would I seriously feel if I received this in response to my most recent submission? That tends to remove the temptation to write ‘This article was both interesting and innovative. Sadly the sections that were interesting were unoriginal and the sections that were innovative were analytically limited and methodologically flawed…’
I would also add that the review should be constructive — meaning that for each major criticism, you should offer some idea of how it could be fixed. Another thing I would recommend is to start the review with a long paragraph in which you (the reviewer) summarizes the objective and argument of the article. This demonstrates that you have carefully read the article and will also serve as a quick check for editors to see if you in fact understood the intent of the article (per Brian’s excellent first point above).
In my five years of journal editing, I have seen the whole spectrum of reviewers. Sad to say, a sizeable majority are “lazy reviewers”. You know the ones – they give you 1-2 short paragraphs that amount to “love it – publish” or “hate it – REJECT.” These are useless reviews and sometimes have to be thrown out because it’s so obvious that the reviewer did not bother to read the article. This is not just insulting to the author of the article. It is insulting to the editors of the journal, who then have to go back and beat the bushes to get another review, lengthening the review time in the process. This may actually be worse, in my opinion, than the “it’s all about ME” reviewers (the ones who spend most of the review complaining about not being cited).
I’m curious: how long are your reviews for articles of 8k-12k words? Mine tend te be 3-5 pages, single-spaced, which may be overkill. But I am very sensitive to Brian’s points above about justifying as criticism and praise. What is an appropriate amount of feedback?
Brian, another great post. I have another important thought though, which is that knowing the temptation to review poorly, we can take steps to incentivize ourselves to review well by being selective in what we agree to review in the first place (again, your point number one). We should be prepared to say no to review requests (and refer editors to people who are a better fit) when we know for a fact we do not have the time or interest in doing them justice. Although I know a lot of faculty who have a hard time saying no to review requests and are not sure under what conditions this is acceptable, my opinion is that a norm of saying ‘no’ is actually a public good for the discipline because the reviews we do will be of better quality.
So how many reviews is your fair share? My rules of thumb is a) I need to do three times as many reviews as I send articles out per year (because three people review each of my articles). And which do I pick? Those in which I actually have a stake in the paper being published, not vetted. I look for research that is clearly within my current area of research interest (not what I might have been interested in five years ago). This means I’m likely to a) want to read it closely b) have appropriate familiarity with the area to judge its quality and c) be engaged enough to give sufficient feedback and interested enough in seeing it published to give constructive feedback.
Plus I’d go one step further than Brian and say it’s important to read the ABSTRACT the editor sends you as closely as you read the paper. The reviews I’m the least proud of are of articles that I end up with that I should never have said yes to, because I failed to give sufficient consideration to the quality and substance of the abstract – and the editor’s deadline – before I said clicked “yes.” We could save ourselves a lot of time, authors a lot of grief and editors a lot of annoyance if we took the time to carefully consider requests on their merits before blindly accepting.
There might be one potential problem with your criteria of picking up which articles to review, namely if you define current research agenda not rather broadly you will run into papers that you are familiar with in previous incarnetions such as conference papers and thus most likely know the authors with a high frequency. This would further reduce the pool avail. for you to review. Not a great concern of course but one potential issue.
Excellent! I would add that reviewers should not point (or fault authors for not citing/reading) to journal articles in a foreign language that the author cannot be reasonably expected to know. Next can we have advice for editors? Like don’t hold papers for three months only to desk reject them (name of prestigious journal withheld…)
In the forthcoming ‘advice to editors’ can we please request universalisation of the process that occurs in most but not all prestigious journals: the actual use of desk reject? I have been sent some serious tripe by some top journals over the years that made applying the above advice (accurate as it is) very challenging…
Brian – best post yet. It would be nice if journals exercised some quality control and refused to disseminate substandard reviews to authors. Alas, many editors care more about pruning the pile of papers than they do about really aiding scholarship…
Most journals do use screening (desk rejection) extensively. At RIPE we screen close to 50 percent, but only after each of the six editors has weighed in. We make these decisions quickly and try to suggest alternative outlets.
Each year, ISA holds a roundtable of IPE editors where we talk about all of this. Definitely an interesting conversation and useful especially for new scholars just getting started in publishing.
Kate, your comments are welcome and quick desk reject with suggestion of alternative outlets would be a very good thing in some cases, but I think you miss the point of Martin’s suggestion. He wants editors to keep unhelpful REVIEWS from AUTHORS. Having been driven to gnashing of teeth by reviews from people who clearly haven’t read the paper or understood the most basic point, I can see some merit in this.
We do this as well. But we also exercise screening options so that we don’t waste social capital with the scarce lot of good reviewers we have.
I have enjoyed reading this article – and although mainly being on the other side of things as an author, I will keep your advice in mind for my next review.
I recently had to deal with an obviously bitchy reviewer – quoting passages out of context to discredit (reject) the whole article. The editor proposed R&R to me, but since his own position was not clear to me at all, I decided to resubmit elsewhere.
My question to you: Why do journal editors accept peer reviews they solicit even if they are obviously biased and let it affect their editorial decision?
And why do editors most of the times (in my experience, at least) just forward the peer reviews to the author, but not any kind of editorial statement?