Ah, the viciously gate-keeping peer review, the contemporary equivalent of burning the heretics at the stake. Receiving such a review seems to be a rite of passage in virtually every academic field and discipline of which I am aware, and though the details change, the general theme of such a review is always something like: “Author didn’t make the argument I would have made or have made, or even worse, Author directly challenged the argument I made or would have made; hence, since I am protected by a veil of anonymity, I will take my vengeance on Author by taking Author to task for being different from me and therefore wrong. I will thus ensure that Author’s manuscript will never see the light of day.” Because, of course, it’s the peer reviewer’s responsibility to enforce the orthodoxy, and to discipline anyone who would dare to question it, even and perhaps especially if this means completely misrepresenting the argument of the manuscript under review.

Anyone who has ever been on the receiving end of such a piece of nastiness (I hear the hands shooting up out there) is perhaps immediately sympathetic with Arthur Stinchcombe’s quip (which I first heard from Dan) that “everyone who has had a referee get the argument of his or her paper directly backward has wondered about calling it ‘peer’ review.” And I suspect that a notion like that discussed in this New York Times article — an open peer-reviewing standard where manuscripts are posted on a website for comment — might sound appealing. Public and non-anonymous comments, after all, would cut down on some of this gatekeeping, wouldn’t it?

But in this case I think that the cure might be worse than the disease. Anonymous peer review, for all of its flaws (and it has many, many flaws), does hold the potential for something that crowdsourced wikiality does not: radical challenge. Realizing that potential requires editors to take greater care in their use of reviewers, but it can be done.

The basic problem with an open reviewing standard, I think, it that it presumes that all of the reviewers have the same basic underlying assumptions, and that those are continuous with the assumptions of the author. While this might work reasonably well in some fields and disciplines, it certainly won’t do a very good job in any field where the fundamental assumptions are themselves an object of contestation. A number of people sharing similar goals and backgrounds can probably use a system of open review to come to a consensus, but that consensus is importantly underpinned by their prior agreement on the basics. Even in such a field, a completely radical challenge would likely not meet with a lot of success even in an open review system, because it would diverge from what “everyone” already knows. The mathematicians discussed in the article likely already shared many important assumptions about what constituted a valid proof, so it’s not surprising that they were able to reach consensus pretty quickly about whether a novel proof was valid or not.

In fact, my main worry about a lot of these crowdsourcing solutions to thorny problems is precisely that they presume a background consensus on goals, definitions of the outcome, and the like. If one of the complaints about anonymous peer review is that a small secretive cabal can dominate a field by refusing to acknowledge radical challenges, I fail to see how open critique will do much better — and indeed it might even be worse, because the putative challenger can actually see how marginal her or his views are when compared with the clear majority of everyone else. Similarly, the defenders of the orthodoxy can actually appeal to one another openly, invoking and mobilizing their shared assumptions to very effectively torpedo anything different. To believe otherwise is to believe that a broad consensus reflects not shared assumptions but some measure of Truth: that lots of people agreeing on something is a good sign that it is correct. Are we really all that willing to elevate the conventional wisdom that far?

The great potential of anonymous peer review is precisely that a savvy editor can send an anonymous manuscript not to people who are guaranteed to hate it — this is pretty easy to do, especially if the manuscript directly criticizes Scholar Z’s approach: send it to Scholar Z, who is pretty likely to produce a review containing some variant of “Author doesn’t agree with me and is therefore wrong” — but to people who are in principle open to a challenge to the disciplinary conventional wisdom. There are fewer of these people in the scholarly community than you might think. There’s this silly idealized notion that a scholar presented with an argument refuting claims to which they are sympathetic will say “wonderful, that’s been refuted! Publish this piece at once!” instead of getting defensive about those claims; in my experience this happens pretty much never — which is why we keep getting endless debates in IR about whether realism is generating any new knowledge, debates in which everyone makes the same arguments as they made in the last Forum and nothing gets resolved. (Perhaps this is because the controversy is about value-laden assumptions, and not about empirical matters on which we can come to fact-based agreement precisely to the extent to which we all agree on what constitutes a “fact” in the first place?)

But provided that one can find appropriately open-minded reviewers, anonymous peer review gives an editor the opportunity to have a radical challenge looked at without anyone feeling a need to moderate their comments due to the prestige of the author, or conversely, feeling no necessity to even be polite because the author is a nobody in the discipline. So the reviewers are thus freed to actually focus on the argument, and perhaps to help the author make the argument better by giving them constructive feedback — including, perhaps, the feedback that the author has to re-think the putatively radical challenge because a) it’s not all that radical, or b) it’s been said, and said better, by other people, or c) the challenge doesn’t actually point the way towards anything else, i.e., the implications of the challenge are not clear. The editor can then go ahead and advise the author appropriately, and publish the piece (maybe after revisions and more reviews) if in her or his professional judgment it deserves to be published.

We mistake the point of peer review if we simply regard it as an opportunity to see whether some claim meets with the approval of a jury of one’s peer reviewers. If that were the point, then both the anonymous and the open versions of peer review would amount to the same thing: poll some people and see if they approve. And the problems involved in a defense of orthodoxy would run rampant in either system. But peer review is not necessarily that kind of process; used properly, it can be an opportunity for improving arguments and — if used by a savvy editor — can be a way to preserve space for radical challenges to dominant conventions. Professional judgment is called for here, not a mechanical aggregation of the conventional wisdom. My biggest worry with an open system of critique is that it eliminates that space for such professional judgment and discretion, a space which in my view ought only — only! — to be used to publish a piece over the objections of reviewers who are gate-keeping and defending the status quo, and ought never — NEVER — be used to overrule reviewers who think that a piece ought to be published. To eliminate that space is to make a very profound bet: that the conventional wisdom is right, or at any rate, sufficient. I am not willing to make that bet, and so despite its problems, I’d rather have the system of anonymous peer review than a wikified alternative.