This is a guest post by Theo McLauchin (@TheoMcLauchlin), Assistant Professor of Political Science at the Université de Montréal
When is a norm not a norm? I ask this question when I read Colin Elman and Arthur Lupia’s vigorous defense of the Data Access & Research Transparency (DA-RT) initiative in APSA’s Comparative Politics section newsletter. I think Elman and Lupia try to have it both ways. Their piece argues, first, that journals need to adopt norms of openness. It then argues, in defense of DA-RT against a series of concerns that it will bring the editorial hammer down on many different forms of work, that DA-RT doesn’t change anything. Editors always could implement whatever policies they wanted to. But of course if the norms change, then the content of that editorial discretion – what decisions are actually made where the submission meets the desk – changes with it. Either DA-RT has an effect or it doesn’t; either the norms change or they don’t; either some articles become newly unpublishable at some journals or they don’t. If they don’t, then DA-RT cannot have the effect its creators hope for it.
For background on DA-RT, see Jarod Hayes’ post at the Duck of Minerva, as well as John Patty’s response to the petition to delay implementation (as well as its related website) and Jeffrey Isaac’s response to Patty and Isaac’s latest post. Roundups and responses abound.
I drafted a longer piece on DA-RT, but now realize that I will probably never finish it. So, instead, some brief comments:
- I have neither signed the DA-RT Journal Editors’ Statement (JETS), nor the acronym-challenged petition to delay it’s implementation. My basic reasons are straightforward. The journal that I edit, International Studies Quarterly, is a publication of the International Studies Association (ISA). Given the structure of the ISA, I do not believe that I enjoy the authority to make this kind of decision—although I should note that my view is not universally shared among ISA journal editors. ISA does have a replication policy that ISQ follows, but it currently only extends to the archiving of statistical materials. ISQ will follow whatever transparency policy the ISA deems appropriate for its journals.
The dynamics of the DA-RT dispute share some similarity to those associated with “wedge issues.” That is, it “divide[s] [political scientists] through code words [and] labeling” in ways I find deeply troubling.
- Participants in the debate over DA-RT need to make sure that they cultivate, or continue to display, intellectual empathy. In other words, opposition to DA-RT standards does not make you an opponent of social science understood correctly. Support for DA-RT standards does not make you a methodological puritan.
But the controversy does provide some people who hold these beliefs the opportunity to express them in unhelpful ways (cf. “wedge issue”).
This is particularly problematic, because there’s nothing wrong with agonistic and intense exchange on the issues.
If you have been living under a rock as I apparently have, then like me you may be unaware of the DA-RT controversy that is brewing in the American Political Science Association.* Turns out that some of our colleagues have been pushing for some time to write a new set of rules for qualitative scholarship that, among other things will require “cited data [be] available at the time of publication through a trusted digital repository” [This is from the Journal Editor’s Transparency Statement, which is what is being implemented Jan. 15]. The goal I gather is to enhance transparency and reproducibility. A number of journal editors have signed on, although Jeffrey Issac, editor at Perspectives on Politics, has refused to sign onto the DA-RT agenda.
There are a number of reasons to doubt that the DA-RT agenda will solve whatever problem it aims to address. Many of them are detailed in a petition to delay implementation (which I have signed) of the DA-RT protocol, currently set for January 15, 2016. To explore how posting data is more or less an optical solution that does little to enhance transparency or reproducibility, I want to run through a hypothetical scenario for interviews, arguably the most prone of qualitative methods to suspicion.
Regardless of the subject, IRBs nearly always insist on anonymity of the interviewees. Which means that in addition to scrubbing names and identifying markers, recordings of interviews cannot be made public (if they even exist, which many IRB decisions preclude). Therein lies the central problem—meaningful transparency is impossible, and as a result reproducibility as DA-RT envisions it is deeply impaired. Even if someone were interested in reproducing a study relying on interviews, doing so would be hindered by the fact that s/he would not be able to interview the same people as the person(s) who undertook the study (this neglects of course that the reproduction interviews could not be collected at the same time, introducing the possibility of contingency effects). In this very simple and nearly universal IRB requirement, there is fundamentally nothing to stop a nefarious, ne’er-do-well academic poser from completely fabricating the interview data that gets posted to the digital database DA-RT requires because there is no way to verify it (e.g. call up the person who gave the interview and ask if they really said that?!). Continue reading