Tag: peer review (page 1 of 2)

IR, it’s time to talk about what the “multiple comparisons issue” really is

As a reviewer and recipient of reviews, I’ve noted a recent trend among IR papers. A study uses cross-national data with regression analysis, and runs multiple models with different variables or sub-sets of the data. Sometimes the results are consistent, sometimes they aren’t. But often a reviewer will object to the study’s validity, pointing to the “multiple comparisons” issue. Multiple comparisons can be a real problem in quantitative IR studies, but I worry we’re mis-diagnosing it.

What do I mean? Imagine we’re writing a paper on interstate conflict. We could measure conflict onset, duration or intensity. We could measure intensity by an ordinal scale, the number of deaths, or other more complicated measures. We could include all dyads, politically-relevant dyads, dyads since 1945, 1989, etc. Ideally these choices would be based on our theory, but our theories are often not specific enough to specify such choices.

Continue reading
Share

Want to fix peer review? Standardize appeals

It’s happened to all of us. You get that email “Decision on Manuscript…,” open it with a bit of trepidation, just to find a (hopefully) politely worded rejection from the editor. Sometimes this is justified. Other times, however, the rejection is due to the legendary “Reviewer #2,” a cranky, ill-informed, hastily written rant against your paper that is not at all fair. The details can vary–they don’t like your theoretical approach, don’t understand the methods, are annoyed you didn’t cite them–but the result is the same: thanks to a random draw from the editor’s reviewers list you’ve got to move on.

We all seem to agree this is a problem. Peer review is finicky, and often relies on gate-keepers who can fail to objectively assess work. The pressure to publish for junior faculty and grad students is immense. And editors are over-worked and overwhelmed. Dan Nexon provided a great service recently by writing a series of posts on his experience at International Studies Quarterly. This gave a lot of insight into this often opaque process, and got me thinking about what to do with the above situation.

Continue reading
Share

The Decision Letter, Part I

Public Domain — From Pixabay

For caveats and background, see my introductory post.

Editors write a lot of decision letters. At high-volume journals, editors write so many decision letters that it can become a tedious grind. For authors, though, the information communicated in decision letters matters enormously. It can affect their job prospects, salaries, and chances of advancement. Of course, authors, especially in the moment, overestimate the significance of any single journal decision. But receiving a rejection, revise-and-resubmit invitation, or an acceptance can certainly feel like a defining event. This is especially the case for graduate students and junior academics, who are less experienced in, and more vulnerable to, the vagaries of the review process.

This makes decision letters the single most consequential way that editors communicate with authors. The same is true for referees. We don’t spend a lot of time teaching academics how to craft referees reports. There is, at best, limited consensus about what makes for a good review. So decision letters also become an important way to send cues to referees about the quality of their reports.

If you think about it, all of this places a heavy burden on editors. That burden only seems heavier when we consider how arbitrary and capricious the peer-review process can be

Yeah. Okay. I’m being a bit melodramatic. Editors don’t perform literal surgery. They don’t design airplanes. The stakes are what they are. But I stand by the underlying sentiment: editors have a responsibility to take decision letters very seriously.

In this post, I’ll focus on general issues. In Part II, I’ll elaborate on them in the context of the specific kinds of decision letters.

Continue reading
Share

Reflections on Journal Editing: Caveats

Josh asked me if I would write a series of posts at the Duck of Minerva reflecting on my time editing International Studies Quarterly (ISQ). I agreed.

This post is less a reflection that some background and caveats. I figure that by collecting them in a single post, I won’t have to junk up subsequent entires in this series. I’ll just refer back to what I’ve written here.

Background. I formally edited ISQ from 2014-2018, although my team started to handle new manuscripts in October of 2013. I headed up a very large team. At peak, it included as many as fourteen academic editors and two managing editors. So my job was as much about oversight as about handling specific submissions. I won’t bore readers with a long discussion of process. You can read about our procedures in our annual reports.

ISQ is the “flagship” journal of the International Studies Association (ISA). This matters for three reasons.

First, “association journals” (such as ISQ) are answerable to external leadership. Their editors depend on the explicit or tacit support of that leadership when it comes to journal policy. Some policies are mandated by the association.

Second, association journals have a duty to the various constituencies of their parent organization. In principle, ISQ should be open to any of the kind of work produced by ISA’s intellectually and geographically diverse membership. It therefore has a responsibility to represent different methods, theoretical frameworks, and substantive areas of research.

Third, although ISQ has middling rankings in some indices—such as the infamous “Impact Factor”—it scores well on subjective rankings of prestige and enjoys significant visibility.

The combination of ISQ‘s relative pluralism and its visibility mean that, as far as I know, it receives more submissions than any other peer-reviewed journal in international studies. But it also has a lot of space, so while it received 650+ submissions in my final year as lead editor, our acceptance rates hovered around 10-12%.

Some Caveats. My observations about the peer-review process and journal publishing are based on a single journal in a single field. They also come from a discrete slice of time. Overall submissions at international-studies journals continue to increase. The field continues to globalize. Expectations for scholarly publishing continue to evolve. All of this means that while some of my views may remain relevant for years, others are likely to become quickly outdated.

In my next post, I’ll start talking substance.

Share

Editors, we need to talk about robustness checks

It’s happened to all of us (or least those of us who do quantitative work). You get back a manuscript from a journal and it’s an R&R. Your excitement quickly fades when you start reading the comments. One reviewer gives a grocery list of additional tests they’d like to see: alternate control variables, different estimators, excluded observations. Another complains about the long list of robustness checks already in the manuscript, as it obscures the important findings. Sometimes both of these reviewers are the same person.

And it gets even more complicated if the article ends up rejected and you send it to another journal. Now that list of robustness checks–some of which were of questionable value–expands under a new set of reviewers’ comments. And those reviewers irritated by voluminous appendices get even more annoyed by all the tests included with little clear justification (“another reviewer told me to add this” not being an acceptable footnote).

Continue reading
Share

Under Review: Cite This!

The boon and bane of our academic enterprise is that we get feedback all the time on our work.  Our work is better for it–that the hack-iest stuff I read is always stuff that is not submitted to any kind of refereeing process and relies instead on editors who seem to be blind to the hack-ness.   The bane is that, well, rejection and criticism can not only delay publication but also hurt feelings.  When well done, reviews further the enterprise.  However, sometimes, reviews seem to make the authors dance in relatively unproductive ways.  There have been lots of tweets and posts complaining about robustness checks–that reviewers have been asking authors to submit dozens (sometimes hundreds) of additional analyses.

My grievance du jour is something else–reviews that focus on stuff that one “should have cited.”

Continue reading

Share

Mocking the Reviewers

As our august leader here at the Duck is putting on his editorial robes, I thought a bit of fresh perspective on the review process is in order.

A fun take on the review process!

Share

The Three Commandments for Editing a Journal

Typesetters Sorting CaseThis brief post started life as a comment on a Facebook discussion thread about peer reviewing practices but I thought it might deserve a wider readership. The question was raised: is it kosher for a journal editor to request information about good reviewers from the author of the manuscript? The general consensus, with which I agree, was that it’s fine to request those names because editors are always looking for qualified reviewers, and the author’s list might provide names that the editor might not have thought of. Of course, the editor need not be bound by that list, and shouldn’t be, but sometimes people in a subfield (or a sub-subfield) know the specific lay of their part of the intellectual landscape better than an editor does.

That said, this is the kind of thing that can be easily abused, as people game the system and give names of people who are most likely to give their manuscript a thumbs-up. And the peer review system is a creaky beast in any event, so I thought I’d lay down a few imperative commands to journal editors, mainly in order to provoke discussion but also to summarize in concise form my own experience both as a journal editor and as an author and reviewer:

Continue reading

Share

(Peer/Non) Review

I understand that there’s been some recent blog-chatter on one of my favorite hobbyhorses, peer review in Political Science and International Relations. John Sides gets all ‘ruh roh’ because of an decades-old old, but scary, experiment that shows pretty much what every other study of peer-review shows:


Then, perhaps coincidentally, Steve Walt writes a longish post on “academic rigor” and peer review. Walt’s sorta right and sorta wrong, so I must write something of my own,* despite the guarantee of repetition.

Continue reading

Share

Talking Academic Journals: Publishing the “Best Work”

Note: this is the second in a series of posts opening up issues relating to journal process for general discussion by the international-studies community.

All journals commit to publishing “the best work” that they receive within their remit. All journals aspire to publish “the best work,” period, within their specialization. This raises special challenges for a journal such as the International Studies Quarterly, which constitutes the “flagship” publication of the International Studies Association (ISA). The ISA is incredibly diverse. It includes members from all over the world–nearly half are based outside of North America–who work in different disciplines and within heterogeneous research cultures.  Continue reading

Share

Talking Academic Journals: Collecting Data

Note: this is the first in what I hope will be a series of posts opening up issues relating to journal process for general discussion by the international-studies community.

Although many readers already know the relevant information, let me preface this post with some context. I am the incoming lead editor of International Studies Quarterly (ISQ), which is one of the journals in the International Studies Association family of publications. We are planning, with PTJ leading the effort, some interesting steps with respect to online content, social media, and e-journal integration–but those will be the subject of a later post. I have also been rather critical of the peer-review process and of the fact that we don’t study it very much in International Relations.

The fact is that ISQ by itself–let alone the collection of ISA journals and the broader community of cognate peer-reviewed publications–is sitting on a great deal of data about the process. Some of this data, such as the categories of submissions, is already in the electronic submission systems–but it isn’t terribly standardized. Many journals now collect information about whether a piece includes a female author. Given some indications of subtle, and consequential, gender bias, we have strong incentives to collect this kind of data.

But what, exactly, should we be collecting?
Continue reading

Share

Thomas Oatley on Peer Reviewing

Tom responds to Peter Henne’s questions.

1. Is there an objective standard for “so what?” No, there is not. Yet, this doesn’t make it fully subjective. Any good paper will explain why what it reports matters, and few papers under-sell their findings. A reviewer’s job is to evaluate the degree to which those assertions are warranted. A good rule of thumb here might be, if you are struggling to decide whether a paper you are reviewing is important, it isn’t.
2. When is a methodological flaw a disqualifier? Always. Whether such flaws warrant a reject or an R&R depends upon the severity of the flaw and the potential significance of the results once the flaw is corrected.
Share

Thoughts on Peer-Review Standards from a Junior Reviewer

peerreviewThis is a guest post by Peter S. Henne. Peter is a doctoral candidate at Georgetown University. He formerly worked as a national security consultant. His research focuses on terrorism and religious conflict; he has also written on the role of faith in US foreign policy. During 2012-2013 he will be a fellow at the Miller Center at the University of Virginia.

So I’ve reviewed several manuscripts for journals in the past year, probably the result of always saying yes. Now, don’t get me wrong, I’m glad to do it, and actually enjoy it—at least for now.  But I’ve had some concerns about the standards by which I make a decision. Namely, what constitutes a publishable paper, what necessitates revise and resubmit, and when is a rejection fair? I’d like to think it’s more than whether or not I like the paper/it supports my research/it cites my research.

But as a dutiful ABD, I’ve naturally thought through this. And as an aspiring publicly-engaged academic, I’ve presented below some of the questions with which I struggle, and tentative answers, for the consideration of readers who’ve been reviewing manuscripts much longer than I have. Finally, as someone who likes getting a lot of comments on blog posts, I’ve framed them as questions…

Continue reading

Share

Blog Archeology: No Decision Letter, No Peer Reviewing

Brad Delong calls this “hoisted from the archives,” which is clearly a better term for what I’m doing. But, as that’s taken and I’m not as smart as the great economics professor, I guess I’ll just have to stick with this alternative.


Peer reviewing: a call to arms (updated)

From: 22 April 2009

I just turned down a request that I review for a journal because, in part, they failed to send me an anonymized copy of the decision letter the last time I reviewed for them. And this despite the journal using an electronic review system that automates the process.

I can think of a number of reasons why all peer-reviewed journals should be required to supply reviewers with copies of their decision letters. In no particular order:

(1) It provides closure to the reviewer.

If I invested–at minimum–a few days in carefully reading an article and writing a review of anywhere from two to six pages, it seems like basic courtesy to let me know what the editors decided to do with the manuscript.

Continue reading

Share

Kant vs. Peer Review

From Department of Omnishambles

More good stuff where this came from (via Jairus Grove).

Share

‘Bleg’: How Long are your ‘Revise & Resubmit’ Letters to the Editor?

editor

I have been asked to revise and resubmit an article submitted for an IR journal. But it’s a big r&r; the editor even said it would be “a great deal of work” (groan). While I must make the changes to the ms, I must also submit a letter to the editors and reviewers to explain my changes. That’s normal of course, but I wonder how the community would appraise the proper length of a letter to the editor for a major r&r? In my last r&r, thankfully a minor, I wrote 2-3 pages. But for a major r&r that “needs a great deal of work’’, I was thinking around 10 pages. Is that too much? Would that you bore you to tears ? (Actually, don’t answer that.)

More generally, I think this is an interesting, undiscussed question for the field, because I have no idea if there are any norms at all on this. I can’t recall discussing this issue ever in graduate school (probably because I couldn’t have gotten an r&r anyway and didn’t even know what r&r meant). Nor can I recall seeing anything on this in all those journals we get from APSA (so many…). So whadda ya think?

Cross-posted on Asian Security Blog.

Share

Professionalization and the Poverty of IR Theory

[I wrote the bulk of this post very late at night while suffering a bout of insomnia. In the end, I ran out of energy and called it quits. Thus, I’ve edited the post for the purpose of clarity and style. Major content updates are in blue text (bad idea, now abandoned].

[2nd Update: I called this post the poverty of IR Theory not the poverty of IR. There’s a difference.]

PM’s post on getting into political-science PhD programs continues to provoke spirited debate. Of particular note is reaction to his claims (echoing Dan Drezner) about the importance of mathematical and statistical skills. As “Evanr” writes:

It sounds like you think these people emerge ‘ex nihilo’ as scholars at the top of the field. At one point and time they were ‘new graduate’ students too, and were very much made the way they are by virtue of their training. I don’t think Wendt would have produced the scholarship he did without the non-mathematical influence of Raymond Duvall [nb: Bud started out his career doing statistical work; Alex was also trained by David Sylvan, whose work extend to agent-based modeling]. Is it not worthwhile to look at the training of top scholars to see how we should shape current students? 

There may be a vast consensus – I’m not sure if there is – that specific forms of training are indispensable to graduate students, but this consensus may be wrong. It sounds like your recommendations are more about reproducing the orthodoxies of the field to make oneself a marketable candidate than they are to intended to produce thoughtful, innovative scholarship. In the short term this may give you an edge in entering the field, but it may also make for lackluster career advancement. With the exception of certain ‘citation cartels’, thinking like everyone else is not a great way to get published.

Having spent far too many years on my Department’s admissions committee–which I currently chair–I have to agree with part of PM’s response: it is simply now a fact of life that prior mathematical and statistical trainings improves one’s chances of getting into most of the first- and second-tier IR programs in the United States. But that, as PM also notes, begs the “should it be this way?” question.

My sense is that over-professionalization of graduate students is an enormous threat to the vibrancy and innovativeness of International Relations (IR). I am far from alone in this assessment. But I think the structural pressures for over-professionalization are awfully powerful; in conjunction with the triumph of behavioralism (or what PTJ reconstructs as neo-positivism), this means that “theory testing” via large-n regression analysis will only grow in dominance over time. I’d also caution some of my smug European and Canadian friends that the writing is on the wall for them as well… albeit currently in very small print.

I should be very clear about the argument I develop below. I am not claiming that neopositivist work is “bad” or making substantive claims about the merits of statistical work. I do believe that general-linear-reality (GLR) approaches — both qualitative and quantitative — are overused at the expense of non-GLR frameworks–again, both qualitative and quantitative. I am also concerned with the general devaluation of singular-causal analysis.

Indeed, one of my “problems” in IR is that I am probably too catholic for my own good, and thus don’t have a home in any particular camp. My views are heavily inflected by my time with high-school and college debate, which led me to a quasi-perspectivist view of theoretical explanation: different kinds of work involve different wagers about what “counts” as instruments, knowledge, and results. Different kinds of work can and should be engaged in a debate about how these wagers, but that doesn’t mean we shouldn’t also evaluate work on its own terms. Thus, I get excited about a wide — probably too wide — variety of scholarship.*

What I am claiming is this: that the conjunction of over-professionalization, GLR-style statistical work, and environmental factors is diminishing the overall quality of theorization, circumscribing the audience for good theoretical work, and otherwise working in the direction of impoverishing IR theory. As is typical of me, I advance this claim in a way designed to be provocative.

1. Darwinian Pressure

We currently produce more PhDs than there are available jobs in IR. We produce far more PhDs than exist slots at the better-paying, research-oriented, well-located universities and liberal arts colleges in the United States. Given this fact of life, consider the following two strategies:

  1. Challenge your professors; adopt non-standard research designs; generally make trouble. 
  2. Focus on finding out the template for getting the best job; affirm your professors; adopt safe research designs.

The first strategy can work out; it sometimes works quite well. But it more often fails spectacularly. Given some of the trends in the field (reinforced themselves by over-professionalization — we face a series of feedback loops here), the first strategy is much riskier than it was fifteen years ago. PhD students may be a variety of dysfunctional things, but stupid generally isn’t one of them. It isn’t much of a surprise, then, that an ever-growing number of them choose the second pathway.

2. Large-N Behavioralism Triumphant

Consider, for a moment, the number of critical, post-structuralist, feminist, or even mainstream-constructivist scholars who hold tenure at A-list and near A-list IR programs in the United States. How many of these programs have more than one tenured professor doing these kinds of work? Still thinking, I bet.

How many of them have multiple tenure-track professors working in this idiom? I can think of a few, including Cornell, George Washington, Ohio State, and Minnesota. But that’s not a lot.

How many have multiple tenure-track professors doing quantitative work, particularly in open-economy IPE (PDF) and JCR-style international security? There’s no point in enumerating them, as virtually every program fits this description.

How many exclusively qualitative scholars–critical, neopositive, or whatever–have gotten jobs at A-list and near A-list schools in the last five years? Not many.

Now recall my stipulation that most PhD students in IR aren’t stupid; most figure out pretty quickly that failure to develop strong quant-fu immediately (1) precludes one from getting a significant number of jobs but (2) closes the door on very few job opportunities. After all, very few members of search committees will say “well, that applicant’s dissertation involves a multivariate regression, I think multivariate regressions aren’t proper social science, so I’m going to block him.” But, I’m sad to say, many members of search committees will refuse to seriously entertain hiring someone who doesn’t use lots of numbers–unless some sort of logroll is underway.

Now add the fact of exponentially increasing computing power. Combine that with (1) nifty statistics packages that do a lot of the work for you; (2) data sets that, although often junk, are widely accepted as “what everyone uses”; and (3) the “free pass” we too often give to using inappropriate-but-currently-sexy statistical techniques. What we’ve got is a recipe for monoculture and for the wrong kind of innovation in statistical methods, i.e., innovation driven by latest-greatest fever rather than thinking through how particular approaches might either shed new, and important, light on old problems or open up new problem areas.

That’s not to say that you can’t “pick wrong” on the quantification front. Some people think statistical inference via sampling techniques is worthless and that only experiments tell us anything interesting. Others think experiments never say anything worthwhile about ongoing political processes. And game-theorists, who do use math, just aren’t getting the kind of traction that proponents of the approach thought they would in the 1990s.

I’m not going to bash large-N or other quantitative studies. Like many Ducks, I don’t find the distinction between quantitative and qualitative research particularly helpful. But I will claim that the triumph of general linear reality (GLR) models in the form of multivariate regression has reinforced small-c conservative tendencies within the field in a variety of ways.

Many quantitative GLR acolytes are convinced — or, at least, publicly express conviction — that they are on the correct side of the demarcation problem, i.e., that. they. are. doing, S-c-i-e-n-c-e. Normal science. Not that stupid “paradigms” wars that wasted our time in the 1980s and 1990s, and certainly not journalism, political theory, non-falsifiable parable telling, or any of that other stuff that is most. definitely. not. Science. And is therefore not simply a waste of our time, but also a shot fired directly at the heart of progress. As in: trying-to-drag-us-back-to-the-dark-ages evil.

My rhetoric may be over the top, but I am not joking. Many perfectly nice, very interesting, extremely smart, and otherwise generous people really do believe that, in blocking the advancement of “alternative” approaches, they are fighting the good fight.** In this paradigm, innovation takes the form of technical improvements; competent work on topics that some percentage of peer reviewers believe to be interesting should be published; and, to be frank, a certain scholasticism winds up prevailing.

Would this be different if another “movement” currently enjoyed an advantage? Probably not. But I do think there’s something — as PTJ has written about — at work among those self-consciously committed to “Science” (and before this was about quantitative methods is was about quasi-statistical qualitative, which should put to rest the notion that we’re talking about numbers) that makes monoculture more likely. I’d feel less worried about this if I saw more persuasive evidence of cumulative-knowledge building in the field — rather than “truths” that are established and upheld exclusively by sociological processes — and if scholars doing even non-standard GLR work had an easier time of it.

3. The De-intellectualization of Graduate School

So what happens when students who enter graduate school:

  • With most of the methods training they will need;
  • Have strong incentives to adopt the “template” strategy for getting a job;
  • Confront a publishing and hiring environment in which methodological deviance is a liability;
  • Receive instruction from at least some instructors who are convinced that there’s a “right way” and a “wrong way” to do social science; and
  • Train in Departments under intense pressure from Graduate School administrators to reduce the time-to-completion of the PhD? 

Answer: an increasing risk of getting an IR degree as a time of intellectual closure; a perfectly rational aversion to debates that require questioning basic assumptions. In short, a recipe for impoverished theorization.

4. Damnit, Don’t We Know Better?

The good news — as many angry graduate students who post on Political Science Job Rumors fail to understand – is that-most of the “better jobs” escape scholars who, no matter how many publications they have, aren’t producing solid middle-range theory. If your work consists of minor tweaks to existing democratic-peace models or throwing variables into a blender and reporting results, then, well, don’t assume that there’s some sort of conspiracy at work when an apparently under-published ABD gets the position that you think you deserve.*** The bad news is that you are increasingly more likely to get hired at a significant subset of institutions than creative scholars who don’t deploy multivariate regression… even if doing so would have been wildly inappropriate given available data and/or the nature of their puzzle.

A number of dynamics work here, but the most distressing involves key dimensions of organized hypocrisy in the field. In particular:

  • We all know that peer reviewing is stochastic–governed by, for example, a surfeit of mediocre reviewers, their transient mental states (‘may your reviewers never read your manuscript right before lunch’), and overwhelmed editors. But we still treat the successful navigation of the slings and arrows of a few prestigious journals as the leading indicator of scholarly quality. Because, after all, why use your own brain when you can farm out your judgment to two or three anonymous reviewers?
  • We all know that quality is not the same as quantity, yet we still wind up counting the number of journals articles as an indicator of past and future scholarly merit.
  • We all know that it is nearly impossible to make an innovative argument and provide empirical support for it, yet we continuously shrink the length of journal articles, demand that the latter accompany the former, and discount “pure theory” articles — thus making it even more difficult to publish innovative arguments. 
  • We all know that the peer-review process is already biased against controversial claims, yet more and more journals default to single-reviewer veto–a decision that makes it even harder to publish innovative work, let alone innovative theory.

These dynamics do, of course, sometimes let innovative arguments through. But it too often distorts them into conformist shadows of their former selves. Note again that these tendency reinforce orthodoxy — whatever that orthodoxy is at the moment.

    5. Conclusion

    I’ve completely lost track of where I began, what the point was, and where I intended to go. But this is a blog, and I have tenure, so I can yell at the kids to get off my lawn… and otherwise rant the rant of the aging curmudgeon. And, just in case you aren’t clear about this: I am overstating the case in order to push discussion along. Get that?

    And, if you didn’t get the moral of this story: I question the judgment of anyone who gets a PhD without developing statistical skills and being able to provide some evidence to committees that he or she has those skills. It. Just. Isn’t. Worth. It.

    Does that make me part of the problem? Maybe. But I think one can hardly look at my record and come to that conclusion.

    Manual trackbacks: James Joyner, Steve Saideman, Erik Voeten.

    ———
    *I do have a pet peeve, however: scholarship that combines multivariate regression with selections from a small menu of soft-rationalist mechanisms… when we are expected to accept the mechanism(s) simply because of widespread invocation in the field. See the overuse of audience-cost mechanisms in settings where the heroic assumptions required for them are simply not credible (get it?).
    **The amount of emotional energy invested on all sides of these disputes is, to be frank, absolutely shocking and appalling.
    ***But, let’s face it, you are correct. Clique dynamics matter a great deal in getting a first job; and given the massively uneven distribution of resources among US colleges and universities, that first job may very well have long-term downstream effects. Of course, we tend to confuse a field in which scholars are frequently born on second base and then advanced to third by a walk with “strict meritocracy,” but that’s another matter. That being said, almost no one actually cares about your “proof” that we’ve gotten the coefficient on the interaction term between trade and democracy slightly off — even if it did land in a “top” journal because, well, see my point number four.

    Share

    Peer Review Bites (and Quacks)

     Apropos Brian’s justified rant against peer-review practices in our field, I thought I’d remind readers (or let new ones know) that the pathetic state of the peer-review system in political science is something of a running theme at the Duck of Minerva.

    Samples include: Brian’s call to “read more and write less,” a note on the impact of one-strike rules given the stochastic quality of peer review, Laura’s discussion of anonymity, PTJ’s thoughtful thoughts on the subject, Bill Petti’s sharing of a Hitler peer-review video, my call to refuse to review for journals that don’t send decision letters to reviewers, a list of things not to say in a peer review, and five reasons academic peer review doesn’t work.

    Looking back over those posts makes clear to me that we’ve never produced a comprehensive indictment of the state of peer review. Perhaps one will be forthcoming. But anecdotal indictments often serve just as well. Via Henry Farrell, just such an illustrative example involving an attempt to replicate findings purporting to demonstrate ESP:

    Here’s the story: we sent the paper to the journal that Bem published his paper in, and they said ‘no, we don’t ever accept straight replication attempts’. We then tried another couple of journals, who said the same thing. We then sent it to the British Journal of Psychology, who sent it out for review. For whatever reason (and they have apologised, to their credit), it was quite badly delayed in their review process, and they took many months to get back to us.

    When they did get back to us, there were two reviews, one very positive, urging publication, and one quite negative. This latter review didn’t find any problems in our methodology or writeup itself, but suggested that, since the three of us (Richard Wiseman, Chris French and I) are all skeptical of ESP, we might have unconsciously influenced the results using our own psychic powers. The story behind this is that Richard has co-authored two papers where he and a believer in psi both did the same experiment, and the believer found positive results but he didn’t. However, the most recent time they did this – which was the best-controlled and largest size – neither found results. This doesn’t exactly give hugely compelling evidence for an ‘experimenter effect’ in psi research, in our opinion. Here’s that last paper, if you’re interested.

    Anyway, the BJP editor agreed with the second reviewer, and said that he’d only accept our paper if we ran a fourth experiment where we got a believer to run all the participants, to control for these experimenter effects. We thought that was a bit silly, and said that to the editor, but he didn’t change his mind. We don’t think doing another replication with a believer at the helm is the right thing to do, for the reason above, and for the reason that Bem had stated in his original paper that his experimental paradigms were designed so that most of the work is done by a computer and the experimenter has very little to do (this was explicitly because of his concerns about possible experimenter effects). So, after this very long and unproductive delay, we’re off to another journal to try again. How frustrating.

    Share

    Dear Reviewers, a Word?

    Everyone gets rejected. And it never stops being painful not matter how successful or how long you have been in the business. Some of this is inevitable; not everyone is above average. But some of it isn’t. I thought that I would offer some ‘dos’ and ‘don’ts’ for reviewers out there to improve the process and save some hurt feelings, when possible. Some are drawn from personal experience; others, more vicariously. I have done some of the “don’ts” myself, but I feel bad about it. Learn from my mistakes.
    First, and I can’t stress this enough, READ THE F*CKING PAPER. It is considered impolite by authors to reject a paper by falsely accusing it of doing THE EXACT OPPOSITE of what it does. Granted, some people have less of a way with words than others and are not exactly clear in their argumentation. But if you are illiterate, you owe it to the author to tell the editors when they solicit your review. It is okay – there are very successful remedial programs they can recommend. Don’t be ashamed.

    Second, and related to the first, remember the stakes for the author. Let us consider this hypothetical scenario. In a safe estimate, an article in a really top journal will probably merit a 2-3% raise for the author. Say that is somewhere around $2000. Given that salaries (except in the University of California System) tend to either stay the same or increase, for an author who has, say, 20 years left in his/her career, getting that article accepted is worth about $40,000 dollars. And that is conservative. So you owe it more than a quick scan while you are on the can. It might not be good, but make sure. Do your job or don’t accept the assignment in the first place. (Sorry, I don’t usually like scatological humor but I think this is literally the case sometimes.)
    Third, the author gets to choose what he/she writes about. Not you. He/she is a big boy/girl. Do not reject papers because they should have been on a different topic, in your estimation. Find fault with the the paper actually under review to justify your rejection.
    Fourth, don’t be a b*tch. Articles should be rejected based on faulty theory or fatally flawed empirics not a collection of little cuts. Bitchy grounds include but are not limited to – not citing you, using methods you do not understand but do not bother to learn, lack of generalizability when theory and empirics are otherwise sound. The bitchiness of reviews should be inversely related to the audacity and originality of the manuscript. People trying to do big, new things should be given more leeway to make their case than those reinventing the wheel.
    Fifth, don’t be an a**hole. Keep your sarcasm to yourself. Someone worked very hard on this paper, even if he/she might not be very bright. Writing “What a surprise!”, facetiously, is a dick move. Rejections are painful enough. You don’t have to pour salt on the wound. Show some respect.
    Sixth, remember that to say anything remotely interesting in 12,000 words is ALMOST IMPOSSIBLE. Therefore the reviewer needs to be sympathetic that the author might be able to fix certain problems where he/she given more space to do so. Not including a counterargument from your 1986 JOP article might not be a fatal oversight; it might have just been an economic decision. If you have other things that you would need to see to accept an otherwise interesting paper, the proper decision is an R&R, not a reject. Save these complaints for your reviews of full-length book manuscripts where they are more justifiable.
    Seventh, you are not a film critic. Rejections must be accompanied by something with more intellectual merit than “the paper did not grab me” or “I do not consider this to be of sufficient importance to merit publication in a journal of this quality.” This must be JUSTIFIED. You should explain your judgment, even if it is something to the effect of, “Micronesia is an extremely small place and its military reforms are not of much consequence to the fate of world politics.” Even if it is that obvious, and it never is, you owe an explanation.
    Share

    Mortenson declines Education Grawemeyer

    In addition to filling an open faculty line in international relations (IR), I was hired in 1991 by the University of Louisville with the idea that I would eventually direct the Grawemeyer Award for Ideas Improving World Order. The World Order award was then one of four Grawemeyer Awards and at the time I was hired, I knew virtually nothing about any of them. The prize was worth $150,000, making it the largest award in Political Science. Nonetheless, it was not especially well-known even within the discipline, nor much publicized outside of it, though the earliest prizes were awarded to prominent IR scholars and Political Scientists like Samuel Huntington, Robert Jervis, Robert Keohane and Richard Neustadt.

    The annual awards in Education and Religion were also relatively unknown. The award in Music Composition, however, apparently became a major global award and typically receives media coverage in the New York Times and other global outlets. The award amount eventually increased to $200,000 (though it decreased after the 2008 stock market dip) and a fifth award in Psychology was added in 2001. Sporadically, awards other than Music Composition have received a modicum of publicity.

    The World Order Award winner received a great deal of publicity in 1994 when Mikhail Gorbachev visited Louisville to speak and collect his payment. Though this selection occurred just before I assumed leadership of the World Order Award, I recall that most of the coverage concerned his missing pants truly. While I have never believed that the lack of media interest in the World Order winners reflected anything in particular about the field or the winning ideas, it can be frustrating laboring in relative obscurity. Many people reading this post have perhaps reviewed for the award in the past — and I know that many had never really heard about the prize until I asked them to read for it.

    In any case, there are clearly far worse fates than being unknown to the wider world. Earlier this year, on April 14 — after months of delay and behind-the-scenes negotiation — the Education winner for 2011 was announced: Greg Mortenson, author of the bestselling book, Three Cups of Tea.

    Was this the academic equivalent of the Grammy award for Milli Vanilli?


    A few days later, “60 Minutes” ran the famous story questioning his honesty, humanitarianism, and research integrity. A couple of days after that story broke, best-selling author Jon Krakauer published a digital book slamming Mortenson for lying and losing “his moral bearings.”

    Needless to say, this created a publicity nightmare for the University of Louisville and for my colleagues in the School of Education, who administer the prize. Time magazine ran a piece detailing the trouble and the story of the university’s apparent gaffe made more news than most of the awards ever have.

    This weekend, roughly one week before Mortenson was scheduled to visit Louisville, speak, and collect his prize, the University announced that Mortenson had decided not to accept the award.

    “We, like millions of others, have been inspired by Greg’s work and we share his commitment to education and to his belief that we can provide a more peaceful future for all our children through knowledge and friendship,” [Provost Shirley] Willihnganz said.

    While UofL will not give the 2011 Grawemeyer Award in Education, Willihnganz said the university will provide $50,000 in privately funded scholarships (unrelated to the Grawemeyer endowment) to students who decide to major in education and agree to teach in Louisville’s poorest schools.

    I have watched this affair unfold with both a sense of distance and uncomfortable proximity. Most of what I know about the Mortenson case has been learned by reading the newspapers and press releases. Each of the awards is quite distinct and I rarely see the faculty involved in the other awards — Psychology is a bit of an exception since it is part of Arts & Sciences. However, Education, Music, and Religion are located in completely different colleges within the University organizational chart.**

    For months, people in Louisville and fellow scholars have asked me about Mortenson because they assume my involvement in World Order grants me access to the inside scoop. That is not the case.

    Over the years, as you might expect, the World Order award has received nominations supporting fairly prominent political figures. Most of them, like Gorbachev, have baggage associated with their work even if they are best known for remarkable ideas or (more likely) for engineering dramatic political changes. Reviewers and the screening committee are supposed to focus on the nominated material, but these external issues inevitably loom in the background — and press against the foreground. I have no doubt that some ideas were considered more seriously at some steps in the multi-stage selection process precisely because they emanated from famous figures.

    The Grawemeyer review process involves nearly a full year of hard work to select a single work — and awarding the prize to a well-known figure can bring immediate attention to the entire effort. I do not believe that the Education committee selected Mortenson because of his name recognition. However, I do think that the selection serves as a cautionary tale for anyone involved in the review process. It could be read, in fact, as another point in favor of blind review.

    Before closing, I should note that I resigned my position directing the Grawemeyer Award for Ideas Improving World order in the spring, effective June 30, 2011. Our department chair had announced his intention to depart for another university, the faculty elected (and the Dean selected) me to succeed him, and I had a one semester sabbatical coming in fall 2011 that I did not want to interrupt. It seemed like a good time for a transition. As it happens, the chair of our Political Science department serves on the Final Selection Committee for the World Order award, meaning that I will again have some important Grawemeyer duties in fall of 2012.

    ** Correction/note: The Religion award is administered by a University faculty committee in conjuction with the Louisville Presbyterian Theological Seminary. The long-time coordinator of the award, Susan R. Garrett, holds a faculty position at the Seminary.

    Share
    « Older posts

    © 2019 Duck of Minerva

    Theme by Anders NorenUp ↑