Somehow I managed to delete my mediocre post on peer review. The gist: peer review is arbitrary and capricious; summary rejections offer a cosmetic fix; we need to reduce our reliance on counting peer-review journal articles as a basis for evaluating scholarly worth.
Jarrod Hayes commented:
I am struck by the arbitrariness of peer-review. I have at least once asked to come in as a reviewer when two reviews reached polar opposite conclusions about a manuscript. Both reviews were careful and conscientious, and I found merit in both when I saw them in the decision letter to the author. How could they both be right?
To which I respond: at least they were both conscientious! I sometimes play a game with a friend called “guess which review was mine?” It isn’t much of a game, to tell the truth. Both of us tend to produce long reviews, often with full references and explanations for why citing particular omitted work matters, and that seldom use denigrating language like “this is obviously a seminar paper.”
But Jarrod raises an absolutely crucial point: publish or perish depends a great deal on luck of the draw. Many of the “top journals” in our field engage in a form of “peer-review triage” in which all of the reviews have to be at least fairly strong R&Rs (“revise and resubmit”) to avoid rejection. This means that it is quite possible for a manuscript to accumulate more “accepts” than “rejects” and never see the light of day–at least as at a “prestigious” journal. Is that evidence of a functional system for allocating status and success?