[cross-posted at SSSpew]
When Political Scientists Do Not Understand Political Science …. they get published in the New York Times.
I tried, I really tried, to ignore the screed at the NYT against political science (especially of the quant variety), but Jacqueline Stevens’s rant is such a poor effort that I know it will be widely read and influential. Why? Because bad ideas often spread further and faster than good ones (see Clash of Civilizations).
There are so many things wrong with this piece that it is hard to know where to start. First, I am , of course, much of what this women hates about political science. I have actually worked not just with Department of Defense dollars but actually in the Pentagon and, dare I say it?, liked it. I have taken National Science Foundation [NSF] money–about $4,000. I have used …. data!
Ok, with that disclaimer aside, I guess the only way to address this piece is to go through it from the top.. Otherwise, I might write something as incoherent as Stevens’s piece. Ok, one more disclaimer, I am mighty miffed to see a left-wing political scientist end up being a fellow traveler with the right-wing ones that are trying to de-fund NSF’s political science program. I don’t know this person as her work is in political theory, a subfield that I do not know well.
Stevens argues that “it’s an open secret” that the creation of “contrived data sets” has failed to produce “accurate political predictions.” Oh, really? Yes, anyone creating a data set understands that coding political behavior means making assessments and assumptions. But any other methodology that seeks to generalize about politics also has to make assessments and assumptions. So, quantitative work will vary, just as qualitative work will, in how well they are performed. Yes, there are alternatives to using the past in either slices of numbers or in case studies, such as experiments and surveys and game theory–but they will have the same problems. So, either we go ahead and try to test our hypotheses and figure out whether there are generalizable dynamics or we don’t. If we don’t, then we don’t need federal grant money or any funding, as we can just think and write without doing the hard work of gathering data via coding or via interviews or whatever.
The second problem with this sentence is this: most of us do not aspire to provide accurate predictions of single events. Most of us seek to understand the causes of outcomes, which leads us t be able to predict that y is more likely or perhaps only possible if x is present (which she ultimately condemns in her conclusion). This can lead to predictions. Indeed, having more understanding should allow us to develop expectations. Having less understanding or no understanding is probably not the pathway to predicting anything.
Ok, let’s go to the examples. First–the end of the cold war. Oy. Yes, we didn’t have too many scholars predicting the Soviet Union would fall apart. There are a lot of reasons for this including a status quo bias where our theories have often tended to predict continuity and not change. Second, our work is not designed to produce predictions of when a country fails apart. Third, the accusation that International Relations failed misses the target as the Soviet Union collapsed due to domestic political processes with external forces filtered through these domestic dynamics. Sure, that just shifts the target from IR to Comparativists, but I would also say that our understanding of how authoritarianism works has always been behind democracy. Data limitations, for one, made it hard to figure out. Also, much of our work is based on how institutions work, but institutions tend to be less binding where rule of law is absent. Fourth, there were lots of elements to the end of the Soviet Union that we did understand pretty well, such as ethnic identity can generate conflict.
The really funny thing about Nancy Reagan’s astrologer is that we don’t know what she said, so we cannot evaluate her predictions.
Stevens goes on to say that political science didn’t get how Al Qaeda changed global politics and that we didn’t get Arab Spring. Um, I guess Stevens doesn’t read much stuff, as there was plenty of work out there on globalization, including of violence, that would alter the conduct of international relations–written before 9/11. On Arab Spring, no, folks didn’t predict Egypt’s fall, but plenty of NSF funded research on dissent and repression can make sense of what happened including that democracy only seems to have flowered in one spot. Again, we, aside from a few, do not say x will happen in 2011.
The op-ed then goes onto to say that our errors are caused by adopting the priorities of government by sucking up for money from DoD and elsewhere. Please. Yes, we are fad-driven–that we shift some of our attention based on what will get us money–security issues in the 1960’s, oil and interdependence in the 1970s; security during the resurgence of the Cold War in the early 1980’s; the European Union in the late 1980’s; ethnic conflict in the aftermath of the Cold war; terrorism and then counter-insurgency in the 2000’s. So, yes, some of the field and some of the funding does shift–but those are shifts in topics, not so much in what our answers are. That is, some of us will shift our research to study that which the public wants to know more about (the government is a public institution, right?), but we in general do not shift our answers or game the models to give answers that we think folks will want. More importantly, isn’t this point a contradiction for Stevens? She starts off by saying that our work does not provide for the public good because we cannot predict but then questions why we might be asking questions to which the public might want answers?
The next paragraph shows how little Stevens understands quantitative political science or modern political science in general, as she cites a few articles with which I am most familiar. Fearon and Laitin’s (F&L) piece has faced much criticism, including by me, arguing that ethnic grievances matter less and civil wars mostly result from weak states.
First, to be clear, their model does, ahem, predict which countries will have civil wars pretty well–in general, not specific ones at specific times. Second, arguing that subsequent work disagrees with F&L is not a condemnation of political SCIENCE but misses the point, as science of any kind requires a give and take. Every prominent piece of work that is published is not venerated, does not settle the question. The Cederman, Weidmann, Gleditsch [CWG] piece (which I happened to review for the APSR) is an advancement precisely because it was inspired by the F&L article. Fearon and Laitin made others think harder and have to provide evidence to make their cases better than if no one had written that piece or if it was not based on some hunk of evidence based on reality. The really amusing thing is that CWG make their case using … data. They coded inequalities and made assumptions about what their indicators measured to argue that ethnic grievances do matter. CWG did rely on heaps of public funding (European, I think, and some NSF) to collect a heap of data to test assertions about various factors and which ones are associated with a higher probability of violence. It is not a perfect piece, but it causes us to have to re-think the conventional wisdom.
The point here is THIS IS HOW SCIENCE WORKS. If you want to make general claims (which you must if you want to make any kind of prediction), you need to develop a set of hypotheses based on your logics and considering how others have addressed the question, you need then to develop some sort of test to see if the various hypotheses are reflected in the real world, and then you assess what you found. The process is part of an ongoing social engagement with other folks studying the same stuff–which means arguing.
The slam she adds is that our conclusions about grievances could be found in the NYT. Lovely, yes, because everything published in newspapers is right, including this piece? Um, HA!
Stevens then cites Tetlock’s work on experts, saying that we don’t do better than chimps throwing darts. Actually, the line she writes is chips and darts “would have done almost as well as the experts.” Oh, so we actually do make predictions that are better than random. That is good, right? Even if we are not perfect? [I have not read Tetlock’s work so I cannot speak to it directly but others can and have.]
Government can and should assist political scienitsts, especially those who use history and theory to explain shifting contexts, challenge our intuitions and help us see beyond daily newspaper headlines.
Um, isn’t that what Fearon and Laitin did? That they pointed out that state weakness is problematic, perhaps more so than ethnic grievances? Didn’t this explain the challenges we face today given the number of weak states in the world? Didn’t it challenge our intuitions by suggesting that it is not just about ethnic hatreds (oh, and yes, if we want to talk about crap in the newspapers, let’s start with ancient hatreds)? Isn’t state weakness not something that appears in most newspapers? Fearon and Laitin were not entirely right, but they were not entirely wrong either.
She writes that, “Research aimed at political prediction is doomed to fail. At least if the idea is to predict more accurately than a dart-throwing chimp.” Does she read her own stuff? She said that chimps throwing darts did “almost as well as the experts.” So, the advantage still goes to the experts. What is better? A lack of expertise?
She concludes by saying that NSF money should be allocated by lottery. Anyone with a PhD and a “defensible” budget could apply. This is utterly ridiculous. How about we then have lotteries for who gets their articles into the prestigious journals and books into the prestigious presses? After all, if having review processes taint who gets money, won’t it taint who gets published? Of course, we had that debate in the 1990s–perestroika it was called. An effort to rebuild political science to do away with the hegemony of quantitative analyses. That is Stevens’s goal: witness her last sentence which is not about point prediction at all: “I look forward to seeing what happens to my discipline and politics more generally once we stop mistaking probability studies and statistical significance for knowledge.” Aha! See, this was not about predicting stuff, it was about quantitative work. Her animus is not really about failing to predict the end of the cold war–which qualitative analysts also did not predict. Her target is, despite the token lines in the conclusion about getting some data, quantitative work.
What upsets me she that she condemns political science in general and lines up with those who are ideologically motivated to de-fund political science (because it does bad things like study accountability). But the NYT would not publish a piece if it said “I was on the losing side of a battle ten years ago to re-structure political science because I don’t like quantitative work.”
The funny thing is that these folks did win in a way–mixed methods is the way of the 21st century–methodological pluralism is the dominant path. You use the methods that work the best to answer your questions–at least in IR and Comparative. In American Politics, quant work is still pretty hegemonic, I guess.
Anyhow, the point is that Stevens does not understand contemporary political science, which makes here a mighty poor advocate for her position. I guess if political science is being attacked from the far right and from the far left, it must be doing something right.
[For more and better takes on this, see pieces elsewhere, including here and here.]
One of the reasons I’m sometimes reticent about voicing concerns about the quantification of the field is that, well, I don’t want to be associated with this sort of thing.
Epic Steve. Thanks. I had some, but not all those thoughts. Thanks for putting these in print. I rarely get angry reading an op ed, but I did in this case. Just so aggressively ignorant.
I started writing a commentary on the Stevens op-ed that was going to take her to task for conflating three different points — an objection to quantification, and objection to prediction, and an objection to Popperian falsification perversely masquerading as an embrace of Popperian falsification (this last point is more starkly in evidence on Stevens’ blog post elaborating on the op-ed) — but I ran out of steam and patience about half-way through. Frankly, I am just not interested in doing all the work it would take to straighten out Stevens’ conflations and elisions, even though they seem to be the kind of difficulties widely shared in the field when it comes to discussion of these issues. our dominant vocabulary is just not adequate for such controversies, because “we” still think that notions like “quantitative political science” and “qualitative political science” form meaningful methodological positions. They do not.
One quibble with Steve, though: “methodological pluralism” is NOT “the dominant path” in US Political Science, and the only kind of “mixed methods” that gets much of a hearing is the large-n “quantitative” statistics plus small-n “qualitative” statistics blend that, methodologically speaking, is not two things but one thing: neopositivism. It’s easy to combine two variants of the same basic thing, and no “pluralism” is required to do so. The sleight of hand here is the claim that hypothesis-testing to refine nomothetic generalizations is “HOW
SCIENCE WORKS” [Steve’s all-caps], when it’s more accurate to say that this is one way that science works. Sadly, the US discipline of Political Science seems convinced that only hypothesis-testing and nomothetic generalization counts as science, and this — not the red herrings of quantification and prediction — is what Stevens’ op-ed should have been about.
Perhaps unintentional, but writing “this woman” reads painfully.
Wow, someone hit a nerve huh? Clearly you have a lot to say about Stevens’ NYT piece. Somehow the sarcastic, condescending, and defensive tone of it all gives Stevens’ argument more credence than you might have wanted. You berate this scholar through your entire response- falling only slightly short of outright calling her stupid. Its a woman, and she just doesn’t understand quant methods. That’s really what your argument seems to come down to. For me, it doesn’t counter her core argument, that the dominant methods and approaches that political scientists use at the moment are not really that USEFUL. That is, quant methods don’t deserve the religious status they have received in many (particularly US academic communities. I agree with her on this and I think its a conversation that needs to happen.
Two smaller points- your comment “but we in
general do not shift our answers or game the models to give answers that
we think folks will want” hardly is a comfort.
Also, you base most of your argument on your claim that Stevens doesn’t understand politics and methods, yet you admit that you don’t understand her ‘field’- political theory. Irony? Maybe you just don’t get it either.
Except that there isn’t such a thing as “quant methods.” The Stevens piece is unclear about its target, and about its indictment of that target; hence it is not a very useful place to begin a discussion about the relatively unreflective dominance of neopositivism in US PoliSci and IR. So instead of meaningful conversation, we get inexact, imprecise, scattershot mudslinging. From all sides.
I’m sympathetic to the argument that US political science is over-quantified, but that doesn’t blind me to the fact that Stevens’ piece is embarrassingly incoherent. She sites Tetlock against “expert opinion,” but Tetlock’s argument suggests that qualitative experts are less reliable than straightforward statistical models. As PTJ noted above (as has Henry), what she’s written on the philosophy of science is nonsensical. And it is, to put it mildly, disingenuous to attack quantitative work on civil wars by citing an extremely sophisticated quantitative argument as evidence on behalf of your position. That these arguments are put at the service of the conservative attack on independent social science makes them more than merely face-palming: it makes them on dangerous.
I hate the accusation that someone is offensive. If someone attacks you and your kind, aren’t you supposed to either (a) ignore it; or (b) defend. Anyhow, my point is that I don’t have the arrogance to say what is good political theory since I do not have enough experience/training to really say what is good political theory. Stevens makes some huge claims based mostly on … air. The focus on prediction is really a facade for a broader attack on all quant work as her conclusion indicated. Her letter indicates that she should be far more humble about that which she knows not well.
Are quant methods useful? Certainly. Are they always the best tool? Absolutely not. Has the focus on high tech quant methods sometimes reached a fetish that is unwise? Absolutely. But the NSF funds data collection more than it funds advanced methods. Lots of interesting and important work has been done in Comparative and IR (and I assume American politics but do not read much of that stuff) using datasets that were funded by the NSF. Moreover, most of her claims would apply as much to qualitative work, so her argument might just be with positivist political science. But that would not play in the NYT.
I would not be so offended/horrified if Stevens made an informed argument about methods. Instead, she deceptively cites one quant piece to attack another. If she thinks that a sustained discourse among folks in the field of civil war does not lead to better understanding, she should have said that instead of saying that one set of authors dismissed Fearon and Laitin for doing stats.
Finally, I am offended at your implication that I am sexist. You should see my posts about the men who have attacked the NSF. I am equal opportunity Spew-er. If people argue that we should know less about how the world works, I will accuse them of supporting ignorance, gender be damned.
Oh, one more thing — Huntington’s “clash of civilizations” argument didn’t spread so widely because it was a bad argument. It was, but that’s irrelevant. What mattered is that Huntington captured the common sense of a certain part of the US foreign policy establishment and the wider US public, updating the containment strategy for US citizens facing a world without the USSR. If Stevens’ piece gains similar traction, it will be because of contextual factors like that, not because it’s a bad or a good argument.
I didn’t really mean that bad ideas spread further because they are bad, more of a frustration that bad ideas do seem to spread further than they should.
In terms of neopositivist domination, well, yeah, I have presented limited view of what political science is. I was staying within that limited view to say that conventional mainstream political science has value even if it does not always predict right. To argue against Stevens by talking about post/non-positive approaches would have been a non-sequitor in my mind since she was not blasting such stuff.
Boy, Stevens really got you boys (and girls) fired up. Actually, I think she hit so close to home that all of these political scientists (who can’t predict) went into fits of apoplexy. The folks at Monkey Cage need to chill, frankly. It’s just an op-ed. Here’s my piece, FWIW. Political science has way bigger problems than a horrible predictive track record: ‘Prediction in Political Science is Doomed to Fail’.
The funny thing is the para you like the most is the most deceptive. Stevens cites approvingly one quantitative piece to criticize another, omitting the fact that the approved piece is just as sinful. It is not that she hit close to home but that she got so much wrong. The other funny thing is that if we didn’t react, people would think that we have nothing to say, that we know that she is right. Damned if you, damned if you do not.
Why do we have to knock Huntington, though Clash of Civilizations was simply copying the “common sense” part of US foreign Policy (FP), its required reading in several FP, Terrorism, and ethnic conflict classes. Secondly, it’s sad political science has become so heavily quantified. Lack of sufficient quantitative methods skill will equivalent to no job in academia.