Brad Delong calls this “hoisted from the archives,” which is clearly a better term for what I’m doing. But, as that’s taken and I’m not as smart as the great economics professor, I guess I’ll just have to stick with this alternative.
Peer reviewing: a call to arms (updated)
From: 22 April 2009
I just turned down a request that I review for a journal because, in part, they failed to send me an anonymized copy of the decision letter the last time I reviewed for them. And this despite the journal using an electronic review system that automates the process.
I can think of a number of reasons why all peer-reviewed journals should be required to supply reviewers with copies of their decision letters. In no particular order:
(1) It provides closure to the reviewer.
If I invested–at minimum–a few days in carefully reading an article and writing a review of anywhere from two to six pages, it seems like basic courtesy to let me know what the editors decided to do with the manuscript.
(2) It helps improve the quality of reviews.
I find reading other reviews helpful in assessing my own. Did I miss something important? How much of my opinion was shaped by my prior commitments? Did I otherwise do an adequate job of providing feedback? Was my review helpful to the editors? If I split with the other reviewers, was I able to swing the editors around to my point of view or not?
(3) It helps me with my own work.
About 50-60% of the reviews I do involve papers that intersect in some non-trivial way with my own research and writing (this is how peer-review is supposed to function). This means that I have some interest in gauging how reviewers will react to certain kinds of arguments and warrants for them. Reading the other peer reviews helps with this. And even if the manuscript isn’t related to my own areas of research, I find I still learn things about the process that can be quite helpful down the road.
UPDATE: a reader emails me a fourth reason:
(4) It keeps editors honest.
One other important reason why reviewers should see the other reviews: it keeps the editors honest. Some journals never communicates with their reviewers about the fate of manuscripts, and certainly never send around the other reviews because, if they did, then reviewers might more openly question the decision-making of the journal. Don’t want to be circulating positive reviews when a manuscript was rejected for other reasons [I’ve edited the email to eliminate references to a specific journal as an exemplar of these practices].
I think that’s right; for some journal editors, the arguments I made above amount bugs, not features, of providing reviewers with decision reports.
Almost all of the major North American journals in political science provide decision letters to reviewers.
The sociology journals I’ve reviewed for do as well, but, somewhat puzzlingly, send the letters via snail mail.
The European journals are much spottier in this respect. Some (*cough* Millennium *cough*) won’t even send these materials–unless requested to do so–when asking for a second-round review!
But, regardless, given the almost universal use of electronic systems for submission and review, there is simply no excuse for not providing anonymized decision letters to peer reviewers.
It seems to me that there’s only one way to ensure that journals “do their duty” on this front: refuse to review for them unless they do.
So I’m calling–right here, right now–for reviewers to boycott the holdouts.
Peer-reviewers of the world unite! You have nothing to lose but lack of closure!
I don’t believe either IS or SS provide decision letters…
I’ve usually gotten them from Security Studies. During my twelve years as an active peer reviewer, I’ve reviewed exactly *one* article for International Security, but I don’t recall getting a decision letter. Since they don’t ask me to review, though, I can’t turn them down ;-).
SS gave me the final letter also.
I completely agree Dan! I think the fourth point is perhaps the most important. Editors have entirely too much power to curate perfectly good articles out of their issues when they don’t have to justify rejecting an article that was accepted by all reviewers (which I think we’ve probably all either experienced or heard of happening). Maybe a boycott would change the way some of the least transparent journals work (and I think we probably all know which ones are have the worst reputation in this regard).
I think it’s pretty inconsistent, even within a single journal. In my own reviewer experience, getting decision letters has been pretty uncommon. I fear that I can count the number of times I’ve gotten copies of other reviewers’ reviews (so useful!) on one hand. Or nearly one hand.
Writing as a journal editor, I think this practice should be required. I have my journal’s (Problems of Post-Communism) online system set to automatically send links to the decision letter and reviews when the decision letter goes to the author. It’s just as easy as Daniel indicates.
It strikes me that this is tremendously important, but difficult for individuals to do? I think a public resource like a wiki would be the way to go.