I understand that there’s been some recent blog-chatter on one of my favorite hobbyhorses, peer review in Political Science and International Relations. John Sides gets all ‘ruh roh’ because of an decades-old old, but scary, experiment that shows pretty much what every other study of peer-review shows:


Then, perhaps coincidentally, Steve Walt writes a longish post on “academic rigor” and peer review. Walt’s sorta right and sorta wrong, so I must write something of my own,* despite the guarantee of repetition.

What does Walt get right?

Third, peer review is probably overvalued because reviewers’ comments are often less than helpful and rarely decisive. By the time most articles are submitted for publication, they’ve usually been presented at academic seminars and have gone through multiple drafts in response to suggestions from the authors’ friends and colleagues. I’ve occasionally gotten useful suggestions from an anonymous reviewer’s report, but I’d say that more than half the comments I’ve received over the years were of no value at all and I simply ignored them. Indeed, a dirty little secret is that a lot of “peer reviews” are no more than a couple of cursory paragraphs along with a recommendation to publish, reject, or revise and resubmit. If that’s the reality of the review process, then why do we fetishize publication in “peer-reviewed” journals as much as we do? In other words, knowing that something got published in the American Political Science Review,World PoliticsInternational Organization, or International Security doesn’t tell you very much about its real value. You have to read it for yourself to make a firm judgment [emphasis added].

The abysmal quality of a large percentage of peer reviews is an open secret in the field. Part of the problem is structural: as the field has placed greater and greater weight on the publication of peer-reviewed articles (in leading journals) for hiring and promotion, scholars have followed the rational course of increasing the number of articles that they submit for review. Indeed, this has been going on for decades. But individual scholars can proliferate their submissions much more easily than the field can generate more qualified reviewers. The math is simply dreadful: the effectiveness of peer review diminishes extremely rapidly as its importance increases.

But a lot of the problem is of our own making. Many scholars refuse to review manuscripts. For example, in 2012 about 65% of the scholars contacted by International Studies Quarterly eventually submitted a review (PDF). Reviewers often, as Walt notes, put little obvious effort into this aspect of their craft. I’ve noticed that reviewers almost never engage in an even cursory check of referenced material to ensure that an author accurately represents sources (and nobody really cares, anyway). There’s plenty of blame to go around, of course. Lack of transparency when it comes to the quantitative data used in manuscripts doesn’t exactly ensure rigorous peer review. Inaccessible primary-source material creates similar problems in qualitative work.

Moreover, Walt notes that he often ignores reviews. And given the quality of most reviews, combined with the stochasticity of the process and the number of journals out there, this makes sense. But it also diminishes the incentives for a reviewer to put in the effort. I once wrote a ~2,500 (sympathetic) review of a manuscript. Along the way I pointed out that the author was making claims to novelty that didn’t make sense, as some of the works cited advanced similar arguments. I recently re-reviewed the same piece for a different journal, and the author hadn’t even made the small effort required to correct this problem. So, yeah, hard to feel like putting in that much intellectual equity into the process.

We’ve been trying to figure out how to address these issues when we take over ISQ. We think that tougher screening of manuscripts might help a little bit — fewer pieces going out for review means less burden on reviewers — and I’m considering promulgating some formal guidelines for reviewers. We also hope that more extensive data collection might at least shed some light on the process. But this is pretty weak tea.

Anyway, I part company with one of Walt’s conclusions:

 I am not suggesting that academia discard peer review and discourage scholars from publishing in prestigious journals. Rather, I’m suggesting that the social sciences would be more useful andmore rigorous if members of these disciplines adopted a less hidebound approach to the merits of different types of publication. “Should it really be the case,” Bruce Jentleson correctly asks, “that a book with a major university press and an article or two in [a refereed journal] … can almost seal the deal on tenure, but books with even major commercial houses count so much less and articles in journals such as Foreign Affairs often count little if at all?”

There are good reasons to discount publications in outlets such as Foreign Affairs that have nothing to do with the unreliability of peer review.

First, what makes something scholarship isn’t the peer-review process, but the mode, style, and form of argument. Foreign Affairs pieces seldom, if ever, reflect the norms, standards, and purposes of writing in the scholarly vocation. And they aren’t supposed to. By design. We can argue over whether this makes them better or worse — and I would say “just different” — but I don’t think that we can argue that they’re the same kind of work.

Second, who gets to publish in  outlets such as Foreign Affairs isn’t precisely a function of position in specific elite networks, but it is heavily inflected by relations of friendship, influence, and patronage. Now, academic publishing in International Relations is far from a strict meritocracy. Some journals are arguably even clubbier than their non-academic counterparts. But we almost certainly overvalue those journals, and we should certainly not compound our mistake by assigning more value to outlets whose editorial decisions are guided by concerns, norms, and incentives very different from those we aspire to as scholars.

So, yes, we should wean ourselves from the organized hypocrisy surrounding peer review. But that doesn’t mean granting me promotion for blogging, nor granting someone tenure for getting some pieces into Foreign Policy. Those activities amount to “service,” and should be treated as such.

What else? Nothing for now. I’m late on a review.

——————————–

*Remember when XKCD was funny?