Awhile back, when cross-posted here and at Lawyers, Guns and Money to harp on the Game of Thrones denouement, LGM Commenter “Dogboy” clicked a link in that post to this article by Stanford researcher Scott Sagan (with Benjamin Valentino), purporting to show (via survey experiment) that Americans would be fine carpet-bombing civilians in Iran. Dogboy’s rightful reaction: “WTF, WHY DID I CLICK THE LINK?” To which I was happily able to reply, “Don’t worry, I’ve studied this data and the authors are wrong. Stay tuned for my follow-up essay in the next few days.” 

It has taken many days to issue my follow-up, partly because, while I was busy completing replications on the Iran study and preparing a rebuttal for publication, Sagan and his team published another similar study on North Korea just before Trump headed over there.

This time, their survey respondents were not asked to saturation bomb an Iranian city (a flagrant violation of the Geneva Conventions), but to violate the UN Charter through a preventive strike on North Korea, with nuclear or conventional weapons depending on your treatment group, weighing the strategic gains against various game-theoretic likelihoods of various levels of civilian casualties in both North and South Korea, ranging from 15,000 to 1.5 million. 

Naturally, of course, the media reported on this study, published in the Bulletin of the Atomic Scientists and titled “What Do Americans Really Think About Conflict With Nuclear North Korea? The Answer is Both Reassuring and Disturbing” by focusing on the DISTURBING: the quote from the article most frequently mentioned in the media is this one:

“As we have previously found, the U.S. public exhibits only limited aversion to nuclear weapons use and a shocking willingness to support the killing of enemy civilians.” 

Well, having just replicated the original Iran study,* and also looked closely at this new North Korea one, I can tell Dogboy and everyone else that we can (mostly) relax. Here’s why: what Sagan’s team calls “disturbing” is not really that disturbing, and even if it is, the “Reassuring” way outranks the “Disturbing” in statistical terms. Below are three reasons why you shouldn’t worry too much about the blood-thirstiness of your fellow Americans.

“A Large Hawkish Minority Lurks.” Basically, Sagan’s team found Trump supporters are happy to bomb foreigners. Why the authors want to focus on (or have the media focus on) the minority of voters who would support war crimes over the large majority of Americans that don’t is unclear. But what they find “disturbing” is these voters also appear “appear insensitive to informational cues that most security experts would expect to reduce such levels of support.”

Voters may ‘appear’ that way, however, because Sagan and his team did not include ‘informational cues’ on knowledge or exposure to the Geneva Conventions in their experiment – something lots of ‘security experts’ including my co-author and I found matters tremendously in such matters. Indeed, when we replicated the original Iran Study we found that providing these cues reverses the result on saturation bombing:

What this means is that in real life, where international and domestic human rights groups (plus generals) would invoke the Geneva Conventions or UN Charter, support for these acts would be much lower than a carefully controlled survey experiment might suggest.

“A Shocking Willingness to Support Killing Civilians.” No. Absolutely not. What they found is that a large majority of Americans (77%) opposed killing civilians, whether or not this was done through conventional bombing or nuclear weapons. Again, this is “reassuring” not “disturbing” and it is also entirely consistent with the Geneva Conventions, which prohibit killing civilians no matter what weapons you use.

This new North Korea finding is actually consistent with what Alex Montgomery and I found on our replication of the original Iran study, as reported in this companion piece in Foreign Policy on audience reactions to the firebombing of King’s Landing: the vast majority of Americans believe it’s wrong to target civilians under any circumstances.

“A Strong Retributive Streak?” Sagan and his team also write that, even though “the majority of Americans do not want President Trump to return to threats to attack North Korea,” there is a “strong retributive streak in US public opinion.” 

This is a stretch. On the original Iran study, the authors developed this claim, because they found that among those willing to bomb the city, some Americans used a sort of “they deserve what they get” or “bomb them all” kind of explanation. On our replication, we found that only a minority of Americans really preferred to target civilians once you control for framing effects embedded in the original prompt, and of those that only a tiny minority (12%) evinced this sort of mentality when we studied the open-ended comments explaining the answers. This dropped to 6% if we gave an open-ended version of the question itself, rather than forcing respondents to choose between terrible options:

Now that’s an augmented replication of the original Iran study. In the new North Korea study, Sagan and his team argue death penalty support predicts retributive attitudes toward civilians and maybe so (we haven’t explored that), but this is hardly a “disturbing” finding about Americans, since death penalty support is at historic lows.

The Media Does Like “Disturbing” Things.

Whether these findings are legitimately disturbing or not, the media sure latched on to the claim that they were, and circulated the erroneous conclusion that this means Americans are happy to kill civilians – just as they did when the original (and flawed) Iran study came out.

This is a shame because what Americans think about what other Americans think can actually influence what Americans think – and that can influence what policymakers do. If the media’s misappropriation of this survey finding, due to a carelessly worded title in a research paper, leads Americans to think many of their countrymen are fine disregarding the UN Charter and the Geneva Conventions, that really is terrifying. Because research shows those rules are sometimes the thin red line encouraging restraint in war.

*Our paper is not yet published, but if you want to read more about the replication I’ve developed with Alexander Montgomery, read this description of our methods at Open Global Rights. If you want to learn more about how Sagan’s team conducted the original Iran study go here. If you want to ponder what’s at stake when pollsters invite Americans to express preferences for war crimes, read this DISTURBING new article by my brilliant PhD student Alexandria Nylen.

Cross-posted at Lawyers, Guns and Money.