Tag: methodology (page 1 of 3)

Mechanical Turk and Experiments in the Social Sciences

Amazon created a platform called Mechanical Turk that allows Requesters to create small tasks (Human Intelligence Tasks or HITs) that Workers can perform for an extremely modest fee such as 25 or 50 cents per task.* Because the site can be used to collect survey data, it has become a boon for social scientists interested in an experimental design to test causal mechanisms of interest (see Adam Berinsky’s short description here). The advantage of Mechanical Turk is the cost. For a fraction of the expense it costs to field a survey with Knowledge Networks/GfK, Qualtrics, or other survey companies, one can field a survey with an experimental component. Combined with other low-cost survey design platforms like SurveyGizmo, a graduate student or faculty member without a huge research budget might be able to collect data for a couple of hundred dollars (or less) instead of several thousand. But, storm clouds loom: in recent weeks, critics like Andrew Gelman and Dan Kahan have weighed in and warned that Mechanical Turk’s problems make it an inappropriate tool, particularly for politically contentious topics. Are these criticisms fair? Should Mechanical Turk be off limits to scholars?

Continue reading

FacebookTwitterGoogle+TumblrRedditShare

Call for Participants: Interpretive and Relational Research Methodologies

“Interpretive and Relational Research Methodologies”
A One-Day Graduate Student Workshop
Sponsored by the International Studies Association-Northeast Region
9 November, 2013 • Providence, Rhode Island

International Studies has always been interdisciplinary, with scholars drawing on a variety of qualitative and quantitative techniques of data collection and data analysis as they seek to produce knowledge about global politics. Recent debates about epistemology and ontology have advanced the methodological openness of the field, albeit mainly at a meta-theoretical level. And while interest in techniques falling outside of well-established comparative and statistical modes of inference has been sparked, opportunities for scholars to discuss and flesh out the operational requirements of these alternative routes to knowledge remain relatively infrequent.

This ninth annual workshop aims to address this lacuna, bringing together faculty and graduate students in a pedagogical environment. The workshop will focus broadly on research approaches that differ in various ways from statistical and comparative methodologies: interpretive methodologies, which highlight the grounding of analysis in actors’ lived experiences and thus produce knowledge phenomenologically and hermeneutically; holistic case studies and forms of process-tracing that do not reduce to the measurement of intervening variables; and relational methodologies, which concentrate on how social networks and intersubjective discursive processes concatenate to generate outcomes.

Continue reading

Society for the Prevention of Cruelty to Methodological Tools: Social-Network Analysis Alert

Is the society depicted in this film historically accurate?
Let’s perform a social-network analysis!  

Here’s a helpful hint: the “realism” of social networks in the Iliad, Beowulf, and the Tain tell us squat, zero, nothing, zilch, not a bit about their historicity.

From the New York Times (h/t Daniel Solomon):

Archaeological evidence suggests that at least some of the societies and events in such stories did exist. But is there other evidence, lurking perhaps within the ancient texts themselves? 

To investigate that question, we turned to a decidedly modern tool: social-network analysis. In a study published in Europhysics Letters, we use a mathematical approach to examine the social networks in three narratives: “The Iliad,” “Beowulf” and the Irish epic “Tain Bo Cuailnge.” If the social networks depicted appeared realistic, we surmised, perhaps they would reflect some degree of historical reality. 


 Social networks have been widely studied in recent years; researchers have looked at the interconnectedness of groups like actors, musicians and co-authors of scientific texts. These networks share similar properties: they are highly connected, small worlds. They are assortative, which means that people tend to associate with people like themselves. And their degree distributions are usually scale-free — a small number of people tend to have lots of friends.

Shorter version: “if the social networks depicted in a cultural epic appear realistic, then the social networks depicted in that cultural epic appear realistic.”*

Once upon a time, Cosma Shalzi wrote an excellent post on physicists and social-network analysis. However, in this case, the problem seems not to be a failure to familiarize with the existing literature, but idiocy.

*For theoretically specified values of “realistic.”

Terrorism and Terrorists: Political, Analytical, and Methodological Issues

Some commentators have suggested posts that pose questions to our readers. I think that the discussion on Peter Henne’s piece, “A Modest Defense of Terrorism Studies,” provides just such an opportunity.

In Remi Brulin’s most recent comment, she asks:

… I am very much interested in better understanding why Peter (and others of course) do believe that the distinction between state and non-state “terrorism” is so important and necessary from an analytical point of view. 

For my part, I would tend to think that it could in fact add a lot to our understanding of “terrorism”, of the non-state or state variety. But even if it were not so, even if such difficulties do appear: that is a problem that scholars would deal with at their micro level, at the level of their case studies, of their datasets. I donot see how this can possibly be a reason or argument for defining a whole field of research and expertise.

My flip answer to Brulin is that there’s a significant literature on subjects such as the of targeting civilians, state repression, and mass violence that already engages with “state terrorism.” Some of that literature, I believe, extends its purview to non-state actors. Nevertheless, I think it worthwhile to begin with a premise, disaggregate some issues, and then throw things open to our readership for their opinions.

Let’s begin with a definition: terrorism is a strategy that seeks to instill fear in non-combatants for coercive purposes. This definition faces problems: what is fear? what is a non-combatant? But, for the sake of argument, let’s begin with a definition that does not render all violence in warfare as terrorism, yet is broad enough to include such disparate activities as nuclear deterrence, torture, collective punishment, and blowing up cafes.

So what is at stake — from an analytical and methodological perspectives — in limiting study to non-state actors that engage in terrorism? Will we learn more or less if we include every possible instances of terrorism in our universe of cases, or will we efface causal processes specific to different kinds of actors and contexts?

PS: for additional related arguments, see Phil Arena’s post on the matter.

Notes: First, Morning Linkage regularly runs Monday-Saturday, but only occasionally on Sundays. Second, due to Labor Day and the start of school last week, there will be no podcast this weekend. Podcasts will resume next week.

Field Reports

I spent last week doing “field research” – that is, participant-observation in one of the several communities of practice whose work I’m following as part of my new book project on global norm development. In this case, the norm in question is governance over developments in lethal autonomous robotics, and the community of practice is individuals loosely associated with the Consortium on Emerging Technologies, Military Operations and National Security. CETMONS is an epistemic network comprised of six ethics centers whose Autonomous Weapons Thrust Group collaboratively published an important paper on the subject last year and whose members regularly get together in subset to debate legal and ethical questions relating to emerging military tech. This particular event was a sponsored by the Lincoln Center on Applied Ethics, which heads CETMONS and held at the Chautauqua Institution in New York.

There among the sailboats (once a game-changing military technology themselves), smart minds from law, philosophy and engineering debated trends in cyber-warfare, military robotics, non-lethal weaponry and human augmentation. Chatham House rules apply so I can’t and won’t attribute comments to anyone in particular, and my own human subjects procedures prevent me from doing more than reporting in the broadest strokes about the discussions that took place in my research, foreign policy writing or blog posts. Nor does my research methodology allow me to say what I personally think on the specific issue of whether or not autonomous lethal weapons should be banned entirely, which is the position taken by the International Committee on Robot Arms Control and Article 36.org, or simply regulated somehow, which seems to be the open question on the CETMONS-AWTG agenda, or promoted as a form of uber-humanitarian warfare, which is a position put forward by Ronald Arkin.* 


However, Chatham House rules do allow me to speak in generalities about what I took away from the event, and my methodology allows me to ruminate on what I’m learning as I observe new norms percolating in ways that don’t bleed too far into advocacy for one side or the other. I can also dance with some of the policy debates adjacent to the specific norm I’m studying. And I can play with the wider questions regarding law, armed conflict and emerging technologies that arise in contexts like this. 

My posts this week will likely be of one or the other variety. 
____________
*Not, at least, until my case study is completed. For now, regarding that debate itself, I’m “observing” rather than staking out prescriptive positions. My “participation” – in meetings like these or in the blogosphere or anywhere else these issues are discussed – is limited to posing questions, playing devil’s advocate, writing empirically about the nature of ethical argument in this area, exploring empirical arguments underlying ethical claims on both sides of that debate, clarifying the applicable law as a set of social facts, and reporting objectively on various advocacy efforts.

No More Cups of Tea: Terrorism Research and the Law

This is a guest post from Tanisha Fazal, a professor of political science at Columbia University, and Jessica Martini, a human rights and international trade attorney based in New York City.

To conduct research on terrorism and insurgency, it’s best to be able to talk to people.  Combing through incident reports is helpful, but often an informal conversation over a cup of tea is as, if not more, illuminating.  But according to ban on providing “material support” (18 United States Code (U.S.C.) 2339B), buying a cup of tea for a terrorist can land you in [US] jail.  In 1996 the Antiterrorism and Effective Death Penalty Act (AEDPA) prohibited providing “material support or resources” to terrorists, which included providing goods and financing, in addition to intangibles such as training and personnel.  This was expanded in 2001 in the wake of the September 11th attacks, as part of USA PATRIOT Act, and subsequent court decisions interpreting this law, to include “expert advice and assistance” and coordinated advocacy.

As part of the government’s broader counterterrorism strategy, The Departments of Defense, State, and Homeland Security all have major initiatives and funding today to develop and promote better research on terrorism.  But another element of US counterterrorism – the material support ban – not only directly hinders the conduct of exactly this type of research, but also puts scholars in a position where they risk being fined or even imprisoned for researching terrorism and/or insurgency.
According to the American Bar Association, the material support ban

prohibits “providing material support or resources” to an organization the Secretary of State has designated as a “foreign terrorist organization.” The material support ban was first passed as part of the Antiterrorism and Effective Death Penalty Act of 1996 (AEDPA). The provision’s purpose is to deny terrorist groups the ingredients necessary for planning and carrying out attacks. Congress was concerned that terrorist organizations with charitable or humanitarian arms were raising funds within the United States that could then be used to further their terrorist activities. The provision outlawed any support to these groups, irrespective of whether that support was intended for humanitarian purposes.

The list of foreign terrorist organizations, or FTOs, contains many groups whose members scholars would like to interview to further their own research.  In addition to the restriction on contacts with FTOs and other entities listed on a number of other US Government lists, there are restrictions on bringing the modern tools of research, such as laptop computers and cell phones – into sanctioned countries like Syria or Iran due to trade sanctions and  export controls.

Prominent NGOs such as Human Rights Watch, The Carter Center, and the International Crisis Group and academic centers such as Notre Dame’s Kroc Institute have protested these restrictions, specifically by submitting amicus briefs (see more such briefs here, here, and here) in Holder v. Humanitarian Law Project, which was an unsuccessful test case challenging the constitutionality on First Amendment grounds of the material support ban.  Ambiguity in the Holder decision creates uncertainty about what is legal when conducting research involving people who may be affiliated with terrorists.  Any resources transferred to these groups – be it a discussion of your broader research that could be translated into advice, or buying lunch for a subject to thank them for taking the time to speak with you – could, in theory “free up other resources within the organization that may be put to violent ends,” according to the majority opinion of the court.

The Holder decision is an issue not just for academics, but also for journalists and activists.  Many of the groups co-sponsoring the amicus briefs were engaged in peacebuilding activities with groups such as the LTTE in Sri Lanka.  But the court’s ruling was that training members of these groups in international human rights law was illegal.

The material support ban and export control restrictions serve an important purpose. Terrorists are a proven threat to the US, and we shouldn’t abet them.  But in restricting resource transfers wholesale, we limit our ability to understand and help these groups find alternative means to achieve the ends they currently seek violently.  There are, in other words, important unintended consequences to the law and to the subsequent decision on its constitutionality.

The main danger for scholars is the vagueness of both the law and the court’s decision.  Insofar as academic research tends to stay within the academy, it’s highly unlikely that a terrorism scholar will be prosecuted for buying a cup of tea for an interview subject on the FTO.  But to the extent that scholarship makes it up to the levels of policy debate – which is partly the point of government programs such as the Minerva initiative, as well as foundation and university initiatives such as the Bridging the Gap program – these laws make conducting research on terrorism and insurgency even riskier than it already is.

– Cross-posted from The Monkey Cage

The Fallacy of Own-Termism

A standard critical argument in my field looks something like this:

1. Phenomenon X involves A assumptions about the world;
2. Approach Y contains assumptions inconsistent with A; therefore
3. Y cannot be used to understand X.

In some instances, and given some specific conditions, this can be a persuasive argument. But it is clearly not a priori true; articulated in the form above, I submit, it is a logical fallacy–one often found alongside, but distinct from, genetic fallacies.

Thus, I will call this the “own-termism fallacy” until someone finds a better–or, at least, preexisting–name for it.

UPDATE: some have asked me for an example. As I’ve written about, this kind of reasoning is extremely common in the “secular bias” literature, which often claims that “secular” theories and methods born of the enlightenment cannot possibly make sense of religious politics.

The Trouble with Combining, or Why I’m Not Touting the Global Peace Index

 The Institute for Economics and Peace is making a big splash today with the release of the 2012 edition of its annual Global Peace Index (GPI)—“the world’s leading measure of global peacefulness,” according to its web site. The launch event for the 2012 edition included several people whose work I respect and admire, and the Institute identifies some of the heaviest hitters in the global fight for peace and human rights—Kofi Annan, Desmond Tutu, and the Dalai Lama, for crying out loud—as “endorsers” of the GPI.

I really want to like this index. I’m a numbers guy, and I’ve spent most of my career analyzing data on political violence and change. But, the closer I look, the less I see.

The basic problem is one that confounds our best efforts to develop summary measures of complex concepts in many fields. Complexity implies multi-dimensionality; the complex whole is composed of many different parts. As a result, no single indicator will capture all of the elements we believe to be relevant.

To try to overcome this problem, we can mathematically combine measures of those separate elements in a single scale—an index. Unfortunately, with truly complex phenomena, those parts do not always move in lock step with each other. As a result, we often wind up with a summary measure that obscures as much as it clarifies because it blinds us to those tensions. In some cases, we can see changes in the index, but we can’t tell what’s driving them. In other cases, the index doesn’t budge, but that doesn’t necessarily mean that there haven’t been significant changes that just happened to cancel each other out. In both of these scenarios, we’ve got a number, but we’re not really sure what it means.

We can see this dilemma clearly when we look closely at the GPI. According to the Institute’s documentation (PDF), the Global Peace Index represents a weighted combination of 23 indicators in three concept areas: 1) ongoing domestic and international conflict; 2) societal safety and security; and 3) militarization. The index includes so many things, we are told, because it aims to get simultaneously at two distinct ideas: not just “negative peace,” meaning the absence of violence, but also “positive peace,” meaning the presence of structures and institutions that create and sustain the absence of violence.

Some of the indicators are inherently quantitative, like counts of deaths from civil conflict and number of jailed population per 100,000 people. Others, such as “perceptions of criminality” and “military capability/sophistication,” are qualitative concepts that are scored by Economist Intelligence Unit staffers. All 23 are converted into comparable five-point scales and then aggregated according to an algorithm that involves weights assigned by an expert panel at the level of the individual indicator and at the level of two sub-component indices having to do with internal (60%) and external (40%) peace. Here’s a complete list of the 23 components:

  • Number of external and internal conflicts fought in the past five years
  • Estimated number of deaths from organized conflict (internal)
  • Estimated number of deaths from organized conflict (external)
  • Level of organized conflict (internal)
  • Relations with neighboring countries
  • Perceptions of criminality in society
  • Number of refugees and displaced people as a percentage of the population
  • Political instability
  • Level of respect for human rights (Political Terror Scale)
  • Potential for terrorist acts
  • Number of homicides per 100,000 people
  • Level of violent crime
  • Likelihood of violent demonstrations
  • Number of jailed population per 100,000 people
  • Number of internal security officers and police per 100,000 people
  • Military expenditures as a percent of GDP
  • Number of armed services personnel per 100,000 people
  • Volume of transfers (imports) of major conventional weapons per 100,000 people
  • Volume of transfers (exports) of major conventional weapons per 100,000 people
  • Budget support for UN peacekeeping missions: percentage of outstanding payments versus annual assessment to the budget of the current peacekeeping missions
  • Aggregate number of heavy weapons per 100,000 people
  • Ease of access to small arms and light weapons
  • Military capability/sophistication

That’s a long list with a lot of very different elements that don’t always move in unison. More problematic in light of the GPI’s additive approach to combining them, those elements don’t always point in the same direction.

Take military expenditures and deaths from external conflicts. International relations scholars would tell you that countries can sometimes avoid wars by preparing for them; rival states are less likely to pick fights with armies they can’t easily beat. Most people would probably think of the avoidance of war as a peaceful outcome, but the GPI casts the preparations that sometimes help to produce that outcome as a diminution of peace. In an ideal world, disarmament and peace would always go together; in the real world, they don’t, but the index’s attempt to combine measures of negative and positive peace muddles that complexity.

The same goes for internal affairs. Imagine that a country is suffering a high homicide rate because of rampant criminal violence (Mexico? Venezuela?). As the GPI implies, that’s not a particularly peaceful situation. Now imagine that that country’s government invests heavily in policing to fight that crime, and that the expanded police presence leads to a decline in the homicide rate and to higher incarceration rates as criminals are arrested and imprisoned. According to the GPI, the gains in peacefulness realized by stopping the wave of murders would be (at least partially) offset by the increases in the size of the police force and the prison population. A change most citizens would regard as an unmitigated good gets washed out by the supposition that the means used to reach that end are detrimental to positive peace.

Now, put both of those problems and several others like them into a single box and shake vigorously. Instead of an elegant simplification, we end up with a complex tangle, simply represented. We see echoes of this problem in summary measures of democracy, like the 21-point Polity scale, which aggregates across several dimensions in ways that sometimes obscure differences of great importance and interest.

For an index to improve on its parts, it should capture something important that we miss when look at the components individually. In my opinion, one of the best examples of this is the Heat Index, which combines air temperature and relative humidity into a single number that we really care about: how hot it actually feels to us humans. The Heat Index is really useful because it gets at something we miss if we look at air temperature alone. The whole illuminates something that the single components can’t show.

Unfortunately, this is hard to do. In many situations, the individual components will offer sharper and more transparent measures of specific dimensions, and we’ll see more when we juxtapose instead of combining them. When we want to explore how these components relate to each other, we can start with two- or three-dimensional scatter plots, which quickly reveal interesting cases of reinforcing or competing tendencies. For more complex problems, multivariate models that relate the components to some observable ground truth (e.g., the absence of deaths from violent conflict) will often work better than indices that use expert judgment to assign weights and directionality.

In the case of the Global Peace Index, I think the starting point for a more useful set of measures would be to construct separate indices for positive vs. negative peace. From my reading of their project, this distinction is more relevant to their objectives than the internal vs. external peace distinction for which they currently report sub-indices, and these are the dimensions along which changes are most likely to be offsetting. This could be done separately for internal and external peace, producing four indices along which levels and movement could be compared and contrasted. Two-dimensional scatter plots could be used to compare countries overall (with positive and negative peace as the axes) or separately for domestic or international peace. To compare a few countries on all dimensions or to illustrate changes within countries over time, radar charts would be useful.

As I hope that last bit of constructive criticism makes clear, I don’t mean to knock the creators of the Global Peace Index for their thoughtful attempt to grapple with a very hard problem. I’d like to see them succeed; I just don’t think they have…yet. More generally, I think the ways in which their current effort falls short illustrate some common dilemmas of measurement that most social scientists face at one time or another.

This is a cross-post from my solo blog, Dart-Throwing Chimp.

What’s So ‘Institutional’ @ Historical Institutionalism?

brrokings

NOTE: The following was actually written before Dan Nexon posted a good piece on exactly the same essay. I’m not sure if that coincidence means anything, but here’s my take:

—————————————–

So I just read Orfeo Fioretos’ “Historical Institutionalism in International Relations” (IO 65/2, 2011). It’s very good – erudite and sophisticated, the kind of dense, abstract writing that makes me wonder if I can keep up in our uber tech-y scientistic field. In it (fn. 18), he defines ‘institution’ as “rules and norms that guide human action and interaction, whether formalized in organizations, regulations, and law, or more informally in principles of conduct and social conventions.” Wikipedia has the nice, punchy: “An institution is any structure or mechanism of social order and cooperation governing the behavior of a set of individuals within a given human community.”

So here is my question: What is really ‘institutional’ about these definitions? Aren’t they staying that pretty much an human behavior that occurs more than once can be an ‘institution’? And isn’t that counter to common-language usage?

I don’t mean to single out Fioretos in this discussion. His essay is excellent, as is Dan Nexon’s response. This definition is widely accepted in IR I think, and the nomenclature has long confused me since graduate school. But when I think of institutions I think of organizations with some kind of charter or formal guidelines, probably with a big building somewhere, where people do their jobs with bosses they don’t like and sit in cubicles all day, with schedules, meetings, deadlines, and cocktail hour. Like a university, or a police department, or the Congress. (The pic above is the Brookings Institute.) Isn’t that more intuitive? Isn’t that what our students and parents think we say ‘institution’ to them?

But in IR/social science, it seems like we can call almost any patterned behavior an institution, which seems pretty definitionally broad. Isn’t patterned human behavior pretty much everything and what all of us in the social sciences study? For example, I think Fioretos’ definition means that we could call the Cold War an institution, or my relations with my nieces, or my unpaid bar tabs. Does this really work? Does it really seem reasonable to call the Cold War, filled with paranoia, suspicion, and proxy wars, an institution? Does it make sense to use the term ‘institution’ in private settings that also have expectations of regularized conduct? I guess you could say the Cold War sorta became a ‘regime’ during détente. But an institution? Would non-IR readers really grasp that?

Fioretos goes on to note there is ‘rational choice institutionalism’ and ‘sociological institutionalism’ too in IR. So I guess my next question is, does that mean pretty much everyone in IR is institutionalist? Now I’m pretty sure I know what rat choice is (cost-benefit analyses, logic of consequences, human robots who would defect on their mother, etc.), and I think get the basic idea of sociology in IR with the logic of appropriateness, culture, constructivism, and hippies. But how does appending ‘institutionalism’ to this help me grasp this better? What exactly is a ‘rat choice institution’ and how is that different from just saying ‘actors using rationalism in making decisions’? When I hear ‘rational choice institutionalism,’ what would easily come to mind are institutions that use rational choice, like maybe the World Bank exporting rationalism to LDCs or something. It takes a conscious effort to force myself to see that ‘rational choice institutionalism’ as something else – and I still don’t really know what that is, or more precisely, how that differs from just plain old rat choice.

So I admit I don’t get it. Is the ‘institutional turn’ in the social sciences just a fancy way of saying we look at patterns over time, and is that just a fancy way of saying ‘history’? I don’t mean to be trite; I know I’m not a good methodologist as my reviewers always tell me. But I don’t really see why we don’t just call ‘historical institutionalism’ ‘history.’ Path dependence, temporality, sequencing – that’s all stuff historians have been doing for awhile, no?

Ok, now that everybody thinks I got my PhD at Walmart, tear me to pieces…

Cross-posted on Asian Security Blog.

Winecoff vs. Nexon Cage Match!

Kindred Winecoff has a pretty sweet rebuttal to my ill-tempered rant of late March. A lot of it makes sense, and I appreciate reading graduate student’s perspective on things.

Some of his post amounts to a reiteration of my points: (over)professionalization is a rational response to market pressure, learning advanced methods that use lots of mathematical symbols is a good thing, and so forth.

On the one hand, I hope that one day Kindred will sit on a hiring committee (because I’d like to see him land a job). On the other hand, I’m a bit saddened by the prospect because his view of the academic job market is just so, well, earnest.  I hate to think what he’ll make of it when he sees how the sausage actually gets made.

I do have one quibble:

While different journals (naturally) tend to publish different types of work, it’s not clear whether that is because authors are submitting strategically, editors are dedicated to advancing their preferred research paradigms, both, or neither. There are so many journals that any discussion of them as doing any one thing — or privileging any one type of work — seems like painting with much too wide a brush.

Well, sure. I’m not critical enough to publish in Alternatives, Krinded’s not likely to storm the gates of International Political Sociology, and I doubt you’ll see me in the Journal of Conflict Resolution in the near future. But while some of my comments are applicable to all journals, regardless of orientation, others are pretty clearly geared toward the “prestige” journals that occupy a central place in academic certification in the United States.

But mostly, this kind of breaks my heart:

I’ve taken more methods classes in my graduate education than substantive classes. I don’t regret that. I’ve come to believe that the majority of coursework in a graduate education in most disciplines should be learning methods of inquiry. Theory-development should be a smaller percentage of classes and (most importantly) come from time spent working with your advisor and dissertation committee. While there are strategic reasons for this — signaling to hiring committees, etc. — there are also good practical reasons for it. The time I spent on my first few substantive classes was little more than wasted; I had no way to evaluate the quality of the work. I had no ability to question whether the theoretical and empirical assumptions the authors were making were valid. I did not even have the ability to locate what assumptions were being made, and why it was important to know what those are.

Of course, most of what we do in graduate school should be about learning methods of inquiry, albeit understood in the broadest terms. The idea that one does this only in designated methods classes, though, is a major part of the problem that I’ve complained about. As is the apparent bifurcation of “substantive” and “methods of inquiry.”And if you didn’t get anything useful out of your “substantive” classes because you hadn’t yet had your coursework in stochastic modeling… well, something just isn’t right there. I won’t tackle what Kindred means by “theory-development,” as I’m not sure we’re talking about precisely the same thing, but I will note that getting a better grasp of theory and theorization is not the same thing as “theory-development.”

Anyway, I’ll spot a TKO to Kindred on most of the issues.

Labels and tribes

In the Matrix, it’s trivial to specify the underlying
data-generating process. It involves kung fu.

 Given PTJ’s post, I wanted to clarify two points from my original post on Big Data and the ensuing comment thread.

I use quantitative methods in my own work. I’ve invested a lot of time and a lot of money in learning statistics. I like statistics! I think that the development of statistical techniques for specifying and disciplining our analytic approach to uncertainty is the most important development in social science of the past 100 years. My objection in the comments thread, then, was not to the use of statistics for inference. I’m cautious about our ability to recover causal linkages from observational data, but no more so than, say, Edward Leamer–or, for that matter, Jeffrey Wooldridge, who wrote the first econometrics textbook I read.

My objection instead is to the simple term “inferential statistics,” because the use of that term to describe certain statistical models, as opposed to the application of statistical models to theoretically-driven inquiry, often belies an unconscious acceptance of a set of claims that are logically untenable. The normal opposition is of “inferential” to “descriptive” statistics, but there is nothing inherently inferential about the logistic regression model. Indeed, in two of the most famous applications of handy models (Gauss’s use of least-squares regression to plot asteroid orbits  and von Botkiewicz’s fitting of a Poisson distribution to data about horses kicking Prussian officers), there is no inference whatsoever being done; instead, the models are simply descriptions of a given dataset. More formally, then, it is not the case that “inferential” describes a property of statistical models, but rather should be taken strictly to refer to their use. What is doing the inferential work is the specification of parameters, which is why it is sometimes entirely appropriate to have a knock-down fight over whether a zero-inflated negative binomial or a regular Poisson is the best fit for a given test of a given theory.

So, my objection on this score is narrowly to the term “inferential statistics,” which I simply suggest should be replaced by something slightly more cumbersome but much more accurate: “the use of statistics for inference.” What this definition loses in pedantry it gains in accuracy.

The second point is that my post about Big Data was meant to serve as a warning to qualitative researchers about what could happen if they did not take the promise of well-designed statistical methods for describing data seriously. My metaphor of an invasive species was meant to suggest that we might end up with a much-impoverished monoculture of data mining that, by dint of its practitioners’ superior productivity, would displace traditional approaches entirely. But the proper response to this is not to equate the use of statistical methods with data mining (as I think a couple of commenters thought I was arguing). Quite the contrary: It would be much preferable for historians to learn how to use statistics as part of a balanced approach than for historians to be displaced by purely data miners.

This is all the more relevant because the flood of Big Data that is going to hit traditionally qualitative studies will open new opportunities for well-informed and teched-up researchers who can take advantage of the skills that leverage the availability of petabytes of data. After all, the real enemy here for qual and quant researchers in social science is not each other but a new breed of data miner who believes that theory is unnecessary, a viewpoint best expressed in 2008 by Chris Andersen in Wired:

But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. … There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

I feel confident that no reader of the Duck wants to see this come to pass. The best way to head that off is not to adopt an thinking anti-statistical stance but rather to use those methods when proper in order to support a deeper, richer understanding of social behavior.

Experiments, Social Science, and Politics

[This post was written by PTJ]

One of the slightly disconcerting experiences from my week in Vienna teaching an intensive philosophy of science course for the European Consortium on Political Research involved coming out of the bubble of dialogues with Wittgenstein, Popper, Searle, Weber, etc. into the unfortunate everyday actuality of contemporary social-scientific practices of inquiry. In the philosophical literature, an appreciably and admirably broad diversity reigns, despite the best efforts of partisans to tie up all of the pieces of the philosophy of science into a single and univocal whole or to set perennial debates unambiguously to rest: while everyone agrees that science in some sense “works,” there is no consensus about how and why, or even whether it works well enough or could stand to be categorically improved. Contrast the reigning unexamined and usually unacknowledged consensus of large swaths of the contemporary social sciences that scientific inquiry is neopositivist inquiry, in which the endless drive to falsify hypothetical conjectures containing nomothetic generalizations is operationalized in the effort to disclose ever-finer degrees of cross-case covariation among ever-more-narrowly-defined variables, through the use of ever-more sophisticated statistical techniques. I will admit to feeling more than a little like Han Solo when the Millennium Falcon entered the Alderaan system: “we’ve come out of hyperspace into a meteor storm.”

Two examples leap to mind, characteristic of what I will somewhat ambitiously call the commonsensical notion of inquiry in the contemporary social sciences. One is the recent exchange in the comments section of PM’s post on Big Data (I feel like we ought to treat that as a proper noun, and after a week in a German-speaking country capitalizing proper nouns just feels right to me) about the notion of “statistical inference,” in which PM and I highlight the importance of theory and methodology to causal explanation, and Eric Voeten (unless I grossly misunderstand him) suggests that inference is a technical problem that can be resolved by statistical techniques alone. The second is the methodological afterword to the AAC&U report “Five High-Impact Practices” (the kind of thing that those of us who wear academic administrator hats in addition to our other hats tend to read when thinking about issues of curriculum design), which echoes some of the observations made in the main report on the methodological limitations of research on practices higher education such as first-year seminars and undergraduate research opportunities — what is called for throughout is a greater effort to deal with the “selection bias” caused by the fact that students who select these programs as undergraduates might be those students already inclined to perform well on the outcome measures that are used to evaluate the programs (students interested in research choose undergraduate research opportunities, for example), and therefore it is difficult if not impossible to ascertain the independent impact of the programs themselves. (There are also some recommendations about defining program components more precisely so that impacts can be further and more precisely delineated, especially in situations where a college or university’s curriculum contains multiple “high-impact practices,” but those just strengthen the basic orientation of the criticisms.)

The common thread here is the neopositivist idea that “to explain” is synonymous with “to identify robust covariations between,” so that “X explains Y” means, in operational terms, “X covaries significantly with Y.” X’s separability from Y, and from any other independent variables, is presumed as part of this package, so efforts have to be taken to establish the independence of X. The gold standard for so doing is the experimental situation, in which we can precisely control for things such that two populations only vary from one another in their value of X; then a simple measurement of Y will show us whether X “explains” Y in this neopositivist sense. Nothing more is required: no complex assessments of measurement error, no likelihood estimates, nothing but observation and some pretty basic math. When we have multiple experiments to consider, conclusions get stronger, because we can see — literally, see — how robust our conclusions are, and here again a little very basic math suffices to give us a measure of confidence in our conclusions.
But note that these conclusions are conclusions about repeated experiments. Running a bunch of trials under experimental conditions allows me to say something about the probability of observing similar relationships the next time I run the experiment, and it does so as long as we adopt something like Karl Popper’s resolution of Hume’s problem of induction: no amount of repeated observation can ever suffice to give us complete confidence in the general law (or: nomothetic relationship, since for Popper as for the original logical positivists in the Vienna Circle a general law is nothing but a robust set of empirical observations of covariation) we think we’ve observed in action, but repeated failures to falsify our conjecture is a sufficient basis for provisionally accepting the law. The problem here is that we’ve only gotten as far as the laboratory door, so we know what is likely to happen in the next trial, but what confidence do we have about what will happen outside of the lab? The neopositivist answer is to presume that the lab is a systematic window into the wider world, that statistical relationships revealed through experiment tell us something about one and the same world — a world the mind-independent character of which underpins all of our systematic observations — that is both inside and outside of the laboratory. But this is itself a hypothetical conjecture, for a consistent neopositivism, so it too has to be tested; the fact that lab results seem to work pretty well constitute, for a neopositivist, sufficient failures to falsify that it’s okay to provisionally accept lab results as saying something about the wider world too.
Now, there’s another answer to the question of why lab results work, which has less to do with conjecture and more to do with the specific character of the experimental situation itself. In a laboratory one can artificially control the situation so that specific factors are isolated and their independent effects ascertained; this, after all, is what lab experiments are all about. (I am setting aside lab work involving detection, because that’s a whole different kettle of fish, philosophically speaking: detection is not, strictly speaking, an experiment, in the sense I am using the term here. But I digress.) As scientific realists at least back to Rom Harré have pointed out, this means that the only way to get those results out of the lab is to make two moves: first, to recognize that what lab experiments do is to disclose cause powers, defined as tendencies to produce effects under certain circumstances, and second, to “transfactually” presume that those causal powers will operate in the absence of the artificially-designed laboratory circumstances that produce more or less strict covariations between inputs and outputs. In other words, a claim that this magnetic object attracts this metallic object is not a claim about the covariation of “these objects being in close proximity to one another” and “these objects coming together”; the causal power of a magnet to attract metallic objects might or might not be realized under various circumstances (e.g. in the presence of a strong electric field, or the presence of another, stronger magnet). It is instead not a merely behavioral claim, but a claim about dispositional properties — causal powers, or what we often call in the social sciences “causal mechanisms” — that probably won’t manifest in the open system of the actual world in the form of statistically significant covariations of factors. Indeed, realists argue, thinking about what laboratory experiments do in this way actually gives us greater confidence in the outcome of the next lab trial, too, since a causal mechanism is a better place to lodge an account of causality than a mere covariation, no matter how robust, could ever be.
Hence there are at least two ways of getting results out of the lab and into the wider world: the neopositivist testing of the proposition that lab experiments tell us something about the wider world, and the realist transfactual presumption that causal powers artificially isolated in the lab will continue to manifest in the wider world even though that manifestation will be greatly affected by the sheer complexity of life outside the lab. Both rely on a reasonably sharp laboratory/world distinction, and both suggest that valid knowledge depends, to some extent, on that separation. This impetus underpins the actual lab work in the social sciences, whether psychological or or cognitive or, arguably, computer-simulated; it also informs the steady search of social scientists for the “natural experiment,” a situation close enough to a laboratory experiment that we can almost imagine that we ran it in a lab. (Whether there are such “natural experiments,” really, is a different matter.)
Okay, so what about, you know, most of the empirical work done in the social sciences, which doesn’t have a laboratory component but still claims to be making valid claims about the causal role of independent factors? Enter “inferential statistics,” or the idea that one can collect open-system, actual-world data and then massage it to appropriately approximate a laboratory set-up, and draw conclusions from that.

Much of the apparatus of modern “statistical methods” comes in only when we don’t have a lab handy, and is designed to allow us to keep roughly the same methodology as that of the laboratory experiment despite the fact that we don’t, in fact, run experiments in controlled environments that allow us to artificially separate out different causal factors and estimate their impacts. Instead, we use a whole lot of fairly sophisticated mathematics to, put bluntly, imagine that our data was the result of an experimental trial, and then figure out how confident we can be in the results it generated. All of the technical apparatus of confidence intervals, different sorts of estimates, etc. is precisely what we would not need if we had laboratories, and it is all designed to address this perceived lack in our scientific approach. Of course, the tools and techniques have become so naturalized, especially in Political Science, that we rarely if ever actually reflect on why we are engaged in this whole calculation endeavor; the answer goes back to the laboratory, and its absence from our everyday research practices.
But if we put the pieces together, we encounter a bit of a profound problem: we don’t have any way of knowing whether these approximated labs that we build via statistical techniques actually tell us anything about the world. This is because, unlike an actual lab, the statistical lab-like construction (or “quasi-lab”) that we have built for ourselves has no clear outside — and this is not simply a matter of trying to validate results using other data. Any actual data that one collects still has to be processed and evaluated in the same way as the original data, which — since that original process was, so to speak, “lab-ifying” the data — amounts, philosophically speaking, to running another experimental trial in the same laboratory. There’s no outside world to relate to, no non-lab place in which the magnet might have a chance to attract the piece of metal under open-system, actual-world conditions. Instead, in order to see whether the effects we found in our quasi-lab obtain elsewhere, we have to convert that elsewhere into another quasi-lab. Which, to my mind, raises the very real possibility that the entire edifice of inferential statistical results is a grand illusion, a mass of symbols and calculations signifying nothing. And we’d never know. It’s not like we have the equivalent of airplanes flying and computers working to point to — those might serve as evidence that somehow the quasi-lab was working properly and helping us validate what needs to be validated, and vice versa. What we have is, to be blunt, a lot of quasi-lab results masquerading as valid knowledge.
One solution here is to do actual lab experiments, the results of which could be applied to the non-lab world in a pretty straightforward way whether one were a neopositivist or a realist: in neither case would one be looking for covariations, but instead one would be looking to see how and to what degree lab results manifested outside of the lab. Another solution would be to confine our expectations to the next laboratory trial, which would mean that causal claims would have to be confined to very similar situations. (An example, since I am writing this in Charles De Gaulle airport, a place where my luggage has a statistically significant probability of remaining once I fly away: based on my experience and the experience of others, I have determined that CDG has some causal mechanisms and process that very often produce a situation where luggage does not make it on to a connecting flight, and this is airline-invariant as far as I can tell. It is reasonable for me to expect that my luggage will not make it into my flight home, because this instance of my flying through CDG is another trial of the same experiment, and because so far as I know and have heard nothing has changed at CDG that would make it any more likely that my bag will make the flight I am about to board. What underpins my expectation here is the continuity of the causal factors, processes, and mechanisms that make up CDG, and generally incline me to fly through Schipol or Frankfurt instead whenever possible … sadly, not today. This kind of reasoning also works in delimited social systems like, say, Major League Baseball or some other sport with sufficiently large numbers of games per season.) Not sure how well this would work in the social sciences, unless we were happy only being able to say things about delimited situations; this might suffice for opinion pollsters, who are already in the habit of treating polls as simulated elections, and perhaps one could do this for legislative processes so long as the basic constitutional rules both written and unwritten remained the same, but I am not sure what other research topics would fit comfortably under this approach.
[A third solution would be to say that all causal claims were in important ways ideal-typical, but explicating that would take us very far afield so I am going to bracket that discussion for the moment — except to say that such a methodological approach would, if anything, make us even more skeptical about the actual-world validity of any observed covariation, and thus exacerbate the problem I am identifying here.]
But we don’t have much work that proceeds in any of these ways. Instead, we get endless variations on something like the following: collect data; run statistical procedures on data; find covariation; make completely unjustified assumption that the covariation is more than something produced artificially in the quasi-lab; claim to know something about the world. So in the AAC&U report I referenced earlier, the report’s authors and those who wrote the Afterword are not content with simply content to collect examples of innovative curriculum and pedagogy in contemporary higher education; they want to know, e.g., if first-year seminars and undergraduate research opportunities “work,” which means whether they significantly covary with desired outcomes. So to try to determine this, they gather data on actual programs … see the problem?

The whole procedure is misleading, almost as if it made sense to run a “field experiment” that would conduct trials on the actual subjects of the research to see what kinds of causal effects manifested themselves, and then somehow imagine that this told us something about the world outside of the experimental set-up. X significantly covarying with Y in a lab might tell me something, but X covarying with Y in the open system of the actual world doesn’t tell me anything — except, perhaps, that there might be something here to explain. Observed covariation is not an explanation, regardless of how complex the math is. So the philosophically correct answer to “we don’t know how successful these programs are” is not “gather more data and run more quasi-experiments to see what kind of causal effects we can artificially induce”; instead, the answer should be something like “conceptually isolate the causal factors and then look out into the actual world to see how they combine and concatenate to produce outcomes.” What we need here is theory and methodology, not more statistical wizardry.

Of course, for reasons having more to do with the sociology of higher education than with anything philosophically or methodologically defensible, academic administrators have to have statistically significant findings in order to get the permission and the funding to do things that any of us in this business who think about it for longer than a minute will agree are obviously good ideas, like first-year seminars and undergraduate research opportunities. (Think about it. Think … there, from your experience as an educator, and your experience in higher education, you agree. Duh. No statistics necessary.) So reports like the AAC&U report are great political tools for doing what needs to be done.

And who knows, they might even convince people who don’t think much about the methodology of the thing — and in my experience many permission-givers and veto-players in higher education don’t think much about the methodology of such studies. So I will keep using it, and other such studies, whenever I can, in the right context. Hmm. I wonder if that’s what goes on when members of our tribe generate a statistical finding from actual-world data and take it to the State Department or the Defense Department? Maybe all of this philosophy-of-science methodological criticism is beside the point, because most of what we do isn’t actually science of any sort, or even all that concerned with trying to be a science: it’s just politics. With math. And a significant degree of self-delusion about the epistemic foundations of the enterprise.

Challenges to Qualitative Research in the Age Of Big Data

Technically, “because I didn’t have observational data.”
Working with experimental data requires only
calculating means and reading a table. Also, this
may be the most condescending comic strip
about statistics ever produced.

The excellent Silbey at the Edge of the American West is stunned by the torrents of data that future historians will be able to deal with. He predicts that the petabytes of data being captured by government organizations such as the Air Force will be a major boon for historians of the future –

(and I can’t be the only person who says “Of the future!” in a sort of breathless “better-living-through-chemistry” voice)

 — but also predicts that this torrent of data means that it will take vastly longer for historians to sort through the historical record.

He is wrong. It means precisely the opposite. It means that history is on the verge of becoming a quantified academic discipline. That is due to two reasons. The first is that statistics is, very literally, the art of discerning patterns within data. The second is that the history that academics practice in the coming age of Big Data will not be the same discipline that contemporary historians are creating.

The sensations Silbey is feeling have already been captured by an earlier historian, Henry Adams, who wrote of his visit to the Great Exposition of Paris:

He [Adams] cared little about his experiments and less about his statesmen, who seemed to him quite as ignorant as himself and, as a rule, no more honest; but he insisted on a relation of sequence. And if he could not reach it by one method, he would try as many methods as science knew. Satisfied that the sequence of men led to nothing and that the sequence of their society could lead no further, while the mere sequence of time was artificial, and the sequence of thought was chaos, he turned at last to the sequence of force; and thus it happened that, after ten years’ pursuit, he found himself lying in the Gallery of Machines at the Great Exposition of 1900, his historical neck broken by the sudden irruption of forces totally new.

Because it is strictly impossible for the human brain to cope with large amounts of data, this implies that in the age of big data we will have to turn to the tools we’ve devised to solve exactly that problem. And those tools are statistics.

It will not be human brains that directly run through each of the petabytes of data the US Air Force collects. It will be statistical software routines. And the historical record that the modal historian of the future confronts will be one that is mediated by statistical distributions, simply because such distributions will allow historians to confront the data that appears in vast torrents with tools that are appropriate to that problem.

Onset of menarche plotted against years for Norway.
In all seriousness, this is the sort of data that should
be analyzed by historians but which many are content
to abandon to the economists by default. Yet learning
how to analyze demographic data is not all that hard,
and the returns are immense. And no amount of
reading documents, without quantifying them,
 could produce this sort of information.

This will, in one sense, be a real gift to scholarship. Although I’m not an expert in Hitler historiography, for instance, I would place a very real bet with the universe that the statistical analysis in King et al. (2008) , “Ordinary Economic Voting Behavior in the Extraordinary Election of Adolf Hitler,” tells us something very real and important about why Hitler came to power that simply cannot be deduced from the documentary record alone. The same could be said for an example closer to (my) home, Chay and Munshi (2011), “Slavery’s Legacy: Black Mobilization in the Antebellum South,” which identifies previously unexplored channels for how variations in slavery affected the post-war ability of blacks to mobilize politically.

In a certain sense, then, what I’m describing is a return of one facet of the Annales school on steroids. You want an exploration of the daily rhythms of life? Then you want quantification. Plain and simple.

By this point, most readers of the Duck have probably reached the limits of their tolerance for such statistical imperialism. And since I am a member in good standing of the qualitative and multi-method research section of APSA (which I know is probably not much better for many Duck readers!), who has, moreover, just returned from spending weeks looking in archives, let me say that I do not think that the elimination of narrativist approaches is desirable or possible. Principally, without qualitative knowledge, quantitative approaches are hopelessly naive. Second, there are some problems that can only practically be investigated with qualitative data.

But if narrativist approaches will not be eliminated they may nevertheless lose large swathes of their habitat as the invasive species of Big Data historians emerges. Social history should be fundamentally transformed; so too should mass-level political history, or what’s left of it, since the availability of public opinion data, convincing theories of voter choice, and cheap analysis means that investigating the courses of campaigns using documents alone is pretty much professional malpractice.

The dilemma for historians is no different from the challenge that qualitative researchers in other fields have faced for some time. The first symptom, I predict, will be the retronym-ing of “qualitative” historians, in much the same way that the emergence of mobile phones created the retroynm “landline.” The next symptom will be that academic conferences will in fact be dominated by the pedantic jerks who only want to talk about the benefits of different approaches to handling heteroscedasticity. But the wrong reaction to these and other pains would be kneejerk refusal to consider the benefits of quantitative methods.

$h•! PTJ Says #3: protest banners vs. precise terms

 I am going to try writing down pieces of advice that I give to students all the time, in the hopes that they might be useful for people who can’t make it to my office hours.

“Many if not most of the terms we use to differentiate styles and traditions of scholarly inquiry are tools for positioning ourselves relative to other scholars. Names of schools of thought, incontrovertible assumptions that have to be agreed to in order to belong to a particular club, shorthand references to ‘great debates’ and ‘key controversies’ — treating these as though they had positive content is basically the same mistake as treating a nationalist claim to possessing some patch of ground from time immemorial as though it were a factual claim. Positioning can provide a helpful signal to other scholars, but but one should be careful not to go overboard in trying to give serious content to something that is basically a set of mapping coordinates.

“This is particularly problematic when we are discussing methodological terms, which are supposed to provide actual guidance for how to do good research. The ordinary academic machine that translates such terms into shibboleths and slogans does an immense disservice to anyone trying to figure out how to do, or to teach others to do, scholarly research, because if open is not careful one can easily find oneself trapped in a hall of mirrors. Perhaps the worst offenders nowadays are words like ‘qualitative’ and ‘interpretive,’ which seem to say something important about a style of research but actually don’t. Both are better thought of as hortatory protest banners: ‘qualitative’ means something like ‘you don’t have to use numbers in order to engage in systematic procedures of data-collection and -analysis’ and ‘interpretive’ means something like ‘get out of your office and go talk to some people, and not just in order to plug their responses into a regression equation’. Okay, fine, but this tells me basically nothing about how to actually do anything.

“Precise terms give us guidance about how to ‘go on’ in producing scholarship that is in some sense valid. Protest banners get our blood pumping and fuel our passion, and maybe get us out into the streets to complain about the lack of thinking space for our kind of work in our field or discipline, but that’s all they are good for. Don’t try to teach using them, and don’t spend too much time trying to give them positive meaning in your own work. Use them to carve out a little academic space for yourself, if you must, and then move on. Because at the end of the day, if you don’t show me the intellectual payoff of your conceptual apparatus, I am not sure what on earth it might possibly be for.”

Knowing and the Known

Although the majority of the offerings in the European Consortium on Political Research’s inaugural Winter School in Methods and Techniques (to be held in Cyprus in February 2012) are pretty firmly neopositivist, at the risk of sound like a shameless self-promoter I’d like to call your attention to course A6, “Knowing and the Known: Philosophy and Methodology of Social Science,” which I am teaching. The short description of this course is:

“The social sciences have long been concerned with the epistemic status of their empirical claims. Unlike in the natural sciences, where an evident record of practical success tends to make the exploration of such philosophical issues a narrowly specialized endeavour, in the social sciences, differences between the philosophies of science underpinning the empirical work of varied researchers produces important and evident differences in the kind of social-scientific work that they do. Philosophy of science issues are, in this way, closer to the surface of social-scientific research, and part of being a competent social scientist involves coming to terms with and developing a position on those issues. This course will provide a survey of important authors and themes in the philosophy of the social sciences, concentrating in particular on the relationship of the mind to the social world and on the relationship between knowledge and experience; students will have ample opportunities to draw out the implications of different stances on these issues for their concrete empirical research.”

Further details, including the long course description, below the fold.

======================

The First ECPR Winter School in Methods and Techniques – REGISTER NOW!
Eastern Mediterranean University, Famagusta, North Cyprus
11th – 18th February 2012

We are very pleased to announce that registration for the first Winter School in Methods and Techniques (WSMT) is now officially open!

This year’s school is being held at the Eastern Mediterranean University in the beautiful surroundings of Famagusta. Register for your course(s) before 1st November and you will receive a special ‘Early Bird Discount’. All information, including registration form, to be found via http://new.ecprnet.eu/MethodSchools/WinterSchools.aspx (move your cursor on “Winter School” –> “2012 – Cyprus” to consult the various pages.

The Winter School will be an annual event that is complementary to the ECPR’s Summer School and there will be a loyalty discount for participants who wish to take part in the 3 step programme at both of these schools (further details on the “course fees” page).
The comprehensive programme consists of introductory courses and advanced courses, in a one-week format, suitable for advanced students and junior researchers in political science and its adjacent disciplines. There will also be the opportunity to participate in one of the software training courses.
Intermediate-level courses will continue to be held at the 2012 Summer School in Methods and Techniques in Ljubljana, in either one two-week or two consecutive one-week courses.

If you have any questions or require any further information please contact Denise Chapman, Methods School Manager on +44 (0)1206 874115 or by email: dchap@essex.ac.uk

Best regards,
Profs. Benoît Rihoux & Bernhard Kittel, Academic convenors

========================

This course is a broad survey of epistemological, ontological, and methodological issues relevant to the production of knowledge in the social sciences. The course has three overlapping and interrelated objectives:

  • to provide you with a grounding in these issues as they are conceptualized and debated by philosophers, social theorists, and intellectuals more generally;
  • to act as a sort of introduction to the ways in which these issues have been incorporated (sometimes— often—inaccurately) into different branches of the social sciences;
  • to serve as a forum for reflection on the relationship between these issues and the concrete conduct of research, both your own and that of others.

That having been said, this is neither a technical “research design” nor a “proposal writing” class, but is pitched as a somewhat greater level of abstraction. As we proceed through the course, however, you should try not to lose sight of the fact that these philosophical debates have profound consequences for practical research. Treat this course as an opportunity to set aside some time to think critically, creatively, and expansively about the status of knowledge, both that which you have produced and will produce, and that produced by others.

The “science question” rests more heavily on the social sciences than it does on the natural sciences, for the simple reason that the evident successes of the natural sciences in enhancing the human ability to control and manipulate the physical world stands as a powerful rejoinder to any scepticism about the scientific status of fields of inquiry like physics and biology. The social science have long laboured in the shadow of those successes, and one popular response has been to try to model the social sciences on one or another of the natural sciences; this naturalism forms one of the recurrent gestures in the philosophy of the social sciences, and we will trace it through its incarnation in the Logical Positivism of the Vienna Circle and then into the “post-positivist” embrace of falsification as the mark of a scientific statement. Problems generated by the firm emphasis on lawlike generalizations through both of these naturalistic approaches to social science lead to the reformulated naturalism of critical realism, as well as to the rejection of naturalism by pragmatists and followers of classical sociologists like Max Weber. Finally, we will consider the tradition of critical theory stemming from the Frankfurt School, and the contemporary manifestation of that commitment to reflexive knowledge in feminist and post-colonial approaches to social science.

While not an exhaustive survey of the philosophy of the social sciences, this course will serve as an opportunity to explore some of the perennial issues of great relevance to the conduct of social-scientific inquiry, and will thus function as a solid foundation for subsequent reading and discussion—and for the practice of social science. Throughout the course we will draw on exemplary work from Anthropology, Economics, Sociology, Political Science; students will be encouraged to draw on their own disciplines as well as these others in producing their reflections and participating in our lively discussions.

New Statistics on Civilian Targeting

In a new paper, Michael Spagat and a number of collaborators explore the determinants of intentional civilian killing in war.

Using sophisticated regression analysis they claim to have found “four significant behavioral patterns”:

“First, the majority (61%) of all formally organized actors in armed conflict during 2002-2007 refrained from killing civilians in deliberate, direct targeting.

Second, actors were more likely to have carried out some degree of civilian targeted, as opposed to none, if they participated in armed conflict for three or more years rather than for one year.

Third, among actors that targeted civilians, those that engaged in greater scales of armed conflict concentrated less of their lethal behavior into civilian targeting ad more into involvement with battle fatalities.

Fourth, an actor’s likelihood and degree of targeting civilians was unaffected by whether it was a state or a non-state group.”

Now those who follow the literature on war law compliance will find a number of these arguments to be quite interesting, somewhat counter-intuitive, and highly policy-relevant. I’ll leave that discussion to comments (may even kick it off) but in the body of this post let me just say two things.

First, this paper is path-breaking not just for its findings but for the data it relies on. The authors are working with a new data-set based on combining three existing Uppsala data-sets: One-Sided Violence, Battle-Related Deaths and Non-State Conflict. Their aim is to disaggregated “intentional targeting of civilians” from wider civilian battle deaths, thus distinguishing for the first time I know of in a large-N study between civilian deaths caused on purpose and those caused incidentally from lawful operations. This is a fundamental distinction in international law that until now has until now been poorly reflected in the datasets on civilian deaths, making it difficult to track war law compliance, as I’ve argued here and here. Parsing existing data this way is a huge step forward for those of us trying to understand the effectiveness of the civilian immunity norm.

But that said I do see a limitation with the data coding. It seems the “battle-related civilian deaths” category ends up including both “collateral” deaths (which are legitimate under international law) and “indiscriminate deaths” which are not (see p. 12) and on the legal rule see AP 1 Article 51(4). So the “intentional targeting” category which is based on the one-sided-violence data, reflects only one type of war law violation rendering the data as currently coded only partially useful for tracking compliance with the Geneva conventions. A more versatile dataset would code each of these categories separately, so that scholars designing their research in different ways could choose to sum or disaggregate them in different combinations. Conflating indiscriminate attacks with collateral damage is both conceptually problematic and, I fear, risks leading to misunderstandings about the status of indiscriminate attacks in international law. I hope that continued work will be done on this dataset to correct for this problem as it will make the data far more useful to replicating and testing these and other hypotheses on civilian victimization.

$h•! PTJ Says #2: on the difference between assumptions and conclusions

I am going to try writing down pieces of advice that I give to students all the time, in the hopes that they might be useful for people who can’t make it to my office hours.

“Spelling out your theoretical and methodological assumptions — the contours of your conceptual equipment, so to speak — is a vital part of doing good social science, because if I don’t know what your assumptions are then I really can’t fairly evaluate your results. In fact, if I don’t know what your assumptions are, I probably have little choice but to apply my own standards, which may or may not be appropriate to your project. So being as clear as you can about your assumptions (with the caveat that it’s impossible to actually spell out *every* assumption that you’re making, both because that kind of self-awareness is a theoretical ideal rather than a live possibility, and because of the Wittgensteinian logical paradox involved in trying to endogenize every rule of a game) is critical.

However, spelling out your assumptions is not the same thing as establishing their validity or their value. Yes, your take on discourse is more pragmatic/Foucault than CDA/Wodak, but that’s not a conclusion of your research — it’s an assumption. Just like ‘individuals make rational choices under conditions of imperfect information’ or ‘human beings are meaning-making animals.’ The fact that you assume this tells me a lot about you, but basically zippo about whether you are right or, more to the point, about whether your assumption is a useful one for the research problem at hand. You can’t use a set of assumptions about discursive practices to conclude that discourse matters or that discourse works the way you think it does, because you already assumed that at the outset! Ditto assumptions about material factors, ideas, etc. “mattering.” You can and should be as detailed as you can be about your assumptions, but if you want anyone to appreciate them as anything other than an expression of your idiosyncratic aesthetic sensibilities, you need to show us what insight they generate in practice — and you have to refrain from overreaching and tautologically concluding that results generated by applying assumption X are an argument for the validity of assumption X. Those results might indeed contribute to an argument that it is useful to make assumption X when trying to explain what you’re trying to explain, but that’s as far as it goes.

Making ‘assent to assumption X’ a condition of membership in some fraternity helps you found or adhere to a school of thought, but whether it helps you explain anything is an entirely different issue. The fact that members of a school, like adherents of any other type of sect, will parade their results as if they constituted ‘evidence’ for their assumptions should be regarded in about the same spirit as any other testimonial, which is to say, compelling to believers but largely inscrutable to outsiders. Displaying your allegiance doesn’t contribute to knowledge, although it can get you into interesting conversations.”

Semantic polling: the next foreign policy tool

George Gallup –
what have you started?

The traditional methods for a state to know what overseas publics are thinking are changing. Instead of relying on your embassy staff’s alertness, your spies’ intelligence and the word of dissidents, we’re reaching the point where foreign policymakers can constantly monitor public opinion in countries in real-time. The digitization of social life around the world  – uneven yes, but spreading – leaves ever-more traces of communications to be mined, analysed and acted upon.  In a paper that Nick Anstead and I presented in Iceland this week, we called this ‘semantic polling’, and we considered the ethical, political and practical questions it raises.

Semantic polling refers to the use of algorithms and natural language processing to “read” vast datasets of public commentary harvested from the Internet, which can be disaggregated, analysed in close-to-real-time, and which can then inform policy. It can give a general representation of public opinion, or very granular representations of the opinion and behaviour of specific groups and networks. Multi-lingual processing across different media platforms is now possible.  Companies already provide this service to pollsters and parties in domestic campaigns, and NGOs make use of it for disaster response monitoring. Given how public diplomacy has adopted many techniques of the permanent campaign, it will be no surprise to see semantic polling become part of the foreign policy toolkit.


The semantic web is the standardization of protocols so that everything on the web becomes machine-readable. This means semantic polling is about more than reading social media data. In principle, our shopping, driving, social media, geolocation and other data are all searchable and analyzable. It is only a matter of computing power and integration of data streams for this method to profile to the individual behavioural level. This also enables predictive engagement: if Amazon thinks it knows what you want, then a state, with access to more data streams, might be use semantic polling and think it knows who will support an uprising and who will not.
Ethically, do people around the world know their tweets, public facebook data and comments on news blogs are being used to build a picture of their opinion? How should journalists report on this when it happens? Politically, how will states and NGOs use semantic polling before, during and after crises and interventions? Is it predictive, valid and reliable? Will semantic polling’s real-time nature further intensify the pressures on policymakers, since the performance, credibility and legitimacy of their policies can be visualized as they are enacted? Will publics resist and find ways to circumvent it? And given that it is barely regulated at the domestic level, how could or should it be policed in international affairs?
When we thought of this paper it seemed a little bit like an exercise in science fiction, but interviews with the companies, pollsters and social scientists driving this has convinced us this is developing quickly. Our political science audience in Iceland seemed positive about this – semantic polling offers relatively cheap, unobstrusive and ‘natural’ data that might provide insights into human behaviour existing methods cannot give. Perhaps a good first step would be for people around the world to understand how semantic polling works, so they can decide what they think about it, since it is their lives that are being monitored. 

Crunching Drone Death Numbers

The Monkey Cage has published a detailed guest post by Christine Fair on the drone casualty debate. Fair takes leading drone-casualty-counters (Bergen and Tiedeman’s New America Foundation database and new numbers out from the Bureau of Investigative Journalism) to task for their methodology, in particular focusing on their sourcing:

While these methodologies at first blush appear robust, they don’t account for a simple fact that non-Pakistani reports are all drawing from the same sources: Pakistani media accunts. How can they not when journalists, especially foreign journalists, cannot enter Pakistan’s tribal areas? Unfortunately, Pakistani media reports are not likely to be accurate in any measure and subject to manipulation and outright planting of accounts by the ISI (Pakistan’s intelligence agency) and the Pakistani Taliban and affiliated militant outfits.

Pakistani journalists have readily conceded to this author that perhaps as many as one in three journalists are on the payroll of the ISI. In fact, the ISI has a Media Management Wing which manages domestic media and monitors foreign media coverage of Pakistan. Even a prominent establishment journalist,Ejaz Haider, has questioned “What right does this wing have to invite journalists for ‘tea’ or ask anyone to file a story or file a retraction? The inquiry commission [to investigate the death of slain journalist Shehzad Saleem] should also look into the mandate of this wing and put it out to pasture.”

Pakistani journalists have explained to this author that, with respect to drone strikes, either the Pakistani Taliban call in the “victim count” or the ISI plants the stories with compliant media in print and television—or some combination of both. In turn, the western media outlets pick up these varied accounts. Of course the victim counts vary to give the illusion of authenticity, but they generally include exaggerated counts of innocents, including women and children. Of course as recent suicide bombings by females suggest, women should not be assumed innocent by virtue of their gender.

Thus, these reports mobilized by NAF and BIJ, despite the claims of both teams of investigators, cannot be independently verified. At best, their efforts reflect circular reporting of Pakistani counts of dubious veracity.

I think this is a really important analysis and share Fair’s concerns about the reliability and validity of these methods. I haven’t looked closely at BIJ’s dataset, but I’ve written previously about not only the sourcing problem, but also coding anomalies and conceptual problems with the NAF methods. The Jamestown Foundation, which has another drone-casualty dataset that Fair doesn’t address, has its own problems.

All that said, having made a clear case that we can’t really verify the numbers, I think it’s very strange that Fair arrives at the conclusion, by the end of her article, that drones must therefore be a pro-civilian technology:

U.S. officials interviewed as well as Pakistani military and civilian officials have confirmed to this author that drones kill very few “innocent civilians.” Indeed, it was these interviews that led me to revise my opinion about the drone program: I had been a drone opponent until 2008. I now believe that they are best option.

It’s hard to argue with her claims that drones might be more discriminate than ‘regular airstrikes,’ an argument that largely resets on her observation that the drone program is more highly regulated and this would be obvious to the public if the CIA didn’t have a variety of incentives to keep mum about the details. But in the absence of good data comparing the kill ratios – which we really don’t have for non-drone-strikes either – it’s hard to make this case definitively. Also, relative to what? A law-enforcement approach that involved capturing and trying terrorists rather than obliterating them might or might not be more ‘pro-civilian’ – though it would certainly be more costly in terms of military life and assets. We simply don’t know.

Regarding how we might know, I also don’t buy Fair’s argument that attempts to verify the civilian status of victims through interviews would be fallacious. This view is based on the assumption that the important question is whether a victim was actually a terrorist. But since extrajudicial executions of suspected criminals without due process is a violation of human rights law, that’s not the right question to ask. From a war law perspective, the right question is whether the individual was directly engaged in hostilities at the time of the attack.

Now, Stephanie may well jump in with some more nuance on percolating developments in war law, the notion that direct participation is being expanded in some ways to include off-duty terror leaders, etc; I agree this is a complex grey area in the law but it’s beyond the scope of this post. My point here is that the messiness of this debate is directly related to the conceptual muddiness introduced by the shift toward thinking about the combatant/civilian divide in terms suspects/non-suspects rather than participation/non-participation. Fair’s analysis is only another example of that.

I am not anti-drone. As I wrote earlier this year, I think there’s a fair amount of unwarranted hype about them, and that the real problems are how they’re used, not the nature of the weapons itself. But this tendency to use the drone discussion to legitimize a reconceptualizing of the very definition of ‘legitimate target’ is extremely problematic. And while I support Fair’s argument that some independent mechanism should be established for determining casualty counts and disaggregating the civilian from the combatant dead, I do not share her belief that this is impossible or that it hinges on the distinction between ‘innocent’ and ‘guilty’ – concepts that require due process to determine. If such fact-finding were done – and they should be, as the Oxford Research Group argues - I would support a coding method that reflects actual humanitarian law, not current US policy.

Here is another perspective for those following the issue. The Wall Street Journal also has a piece out today on the topic.

$h•! PTJ Says #1: justifying your theory and methodology

I am going to try writing down pieces of advice that I give to students all the time, in the hopes that they might be useful for people who can’t make it to my office hours.

“The fact that no one else has approached topic X with your particular perspective is not a sufficient warrant for approaching topic X with your particular combination of theory and methodology. In order to get the reader on board, you have to basically issue a promissory note with a grammar that runs something like:

‘Here’s something odd/striking/weird/counterintuitive about X. Other scholars who have talked about X either haven’t noticed this odd/striking/etc. thing at all, or they haven’t found it odd/striking/etc. Furthermore, they haven’t done so because of something really important about their theory/methodology that — even though it generates some insights — simply prevents them from appreciating how odd/striking/etc. this thing is, let alone trying to explain it. Fortunately, there’s my alternative, which I am now going to outline in a certain amount of abstract detail; but bear with me, because there’s a mess of empirical material about topic X coming after that, and I promise you that my theoretical/methodological apparatus will prove its worth in that empirical material by a) showing you just how odd/striking/etc. that thing is, and b) explaining it in a way that other scholars haven’t been able to and won’t be able to.’

Almost no one is convinced by theory and methodology, and absolutely no one is or should be convinced by a claim that existing approaches aren’t cool enough because they aren’t like yours. The burden is on you to give the reader reasons to keep reading, and at the end of the day the only reason for theory and methodology is to explain stuff that we didn’t have good explanations for before. So you have to convince the reader that other approaches *can’t* explain that odd thing about topic X. (And if you can do this without gratuitous and out-of-context references to Thomas Kuhn and being ‘puzzle-driven,’ that’s even better, because I won’t have to make you write an essay on why basically nobody in the social sciences actually uses Kuhn correctly.)”

Older posts

© 2015 Duck of Minerva

Theme by Anders NorenUp ↑