If you have been living under a rock as I apparently have, then like me you may be unaware of the DA-RT controversy that is brewing in the American Political Science Association.* Turns out that some of our colleagues have been pushing for some time to write a new set of rules for qualitative scholarship that, among other things will require “cited data [be] available at the time of publication through a trusted digital repository” [This is from the Journal Editor’s Transparency Statement, which is what is being implemented Jan. 15]. The goal I gather is to enhance transparency and reproducibility. A number of journal editors have signed on, although Jeffrey Issac, editor at Perspectives on Politics, has refused to sign onto the DA-RT agenda.
There are a number of reasons to doubt that the DA-RT agenda will solve whatever problem it aims to address. Many of them are detailed in a petition to delay implementation (which I have signed) of the DA-RT protocol, currently set for January 15, 2016. To explore how posting data is more or less an optical solution that does little to enhance transparency or reproducibility, I want to run through a hypothetical scenario for interviews, arguably the most prone of qualitative methods to suspicion.
Regardless of the subject, IRBs nearly always insist on anonymity of the interviewees. Which means that in addition to scrubbing names and identifying markers, recordings of interviews cannot be made public (if they even exist, which many IRB decisions preclude). Therein lies the central problem—meaningful transparency is impossible, and as a result reproducibility as DA-RT envisions it is deeply impaired. Even if someone were interested in reproducing a study relying on interviews, doing so would be hindered by the fact that s/he would not be able to interview the same people as the person(s) who undertook the study (this neglects of course that the reproduction interviews could not be collected at the same time, introducing the possibility of contingency effects). In this very simple and nearly universal IRB requirement, there is fundamentally nothing to stop a nefarious, ne’er-do-well academic poser from completely fabricating the interview data that gets posted to the digital database DA-RT requires because there is no way to verify it (e.g. call up the person who gave the interview and ask if they really said that?!).
The problem for DA-RT in this scenario gets worse though. One might argue that even if audio cannot be posted, a transcription of the interview could be made available. Even if such an transcription is possible—again IRB may mandate that recordings not be made or the interviewer may not make recordings in hopes of eliciting as honest a response as possible (would DA-RT override the judgment of scholars in the field?)—depending on the subject, it may be impossible to post a complete transcript of an interview because doing so would provide enough contextual information to identify the interviewee. If the scholar has only handwritten or digital notes, the full interview may still be too revealing and even if it is not, the notes are not an unadulterated record of the interview, but rather the scholar’s distillation of the interview.
Which means we are left where we started, either a distilled record of the interview which may or may not be objectively accurate and is certainly not transparent or technically reproducible or pieces of an interview that the scholar analyzes in the context of the study. Sure, we can post these to a digital repository, but transparency and reproduction are not enhanced either because there really is not unadulterated data or unrevealed information in the repository. So what’s the point?
Practical points aside, there are broader methodological and disciplinary concerns. Political science needs greater intellectual diversity, not less. As Isaacs points out, the DA-RT protocol is suggestive of a epistemological and methodological disciplining move that will force scholars ever closer into a neopositivist straightjacket.
It also encourages us to continue to believe that quantitative approaches have cornered the market on transparency and reproducibility. But they haven’t. What do I mean? Transparency and reproducibility in quantitative studies largely address the process of analysis. If you want to run a regression on some data, you have to show your work. Other people run the same data on their computers making the same decisions and see if it checks out. Questions of transparency and reproduction lie in the analytical decisions, not in the data itself. No one[**] checks the decisions Polity or Correlates of War made in their coding of each datapoint, much less the data they were coding, much less that whiz-bang novel dataset that nobody but you is using. No one assembles a new Polity to see if they get the same numbers as existing Polity. And that is what true reproducibility would require. But we don’t see new Polity datasets because we aren’t really interested in data reproducibility, but rather analytical reproducibility—the weighting decisions, whether a spline was the appropriate mode of interpolation, whether there was a lag factor, and so on.*** But unlike quantitative methods, qualitative methods put the [interpretive] analytical process right up front, in the written piece. [Because the interpretive analytical move is usually made in the body of the text rather than in a data set or an unpublished regression algorithm] Qualitative scholarship already has analytical transparency and, insofar as it is possible, reproducibility.
I admit that the issues at play are more complicated than I represent here, and that my logic may be a bit slapdash as I try to write this and pack for ISA Northeast. For those interested in a nuanced and useful discussion of the issues, the Spring 2015 Qualitative and Multimethod Research newsletter is a good start.[****]
N.B. I have made a few minor edits to clarify points. Where I have done so the edits are in []
*Many thanks to Jelena Subotic and Kai Thaler for awakening me to this matter.
[**A good discussion on Twitter has made the case that I overstate this point. I accept that.]
***On the challenges of analytical reproduction, Nature has an interesting commentary.
[****In the interest of fairness, perspectives disagreeing with me can be found at Tom Pepinsky‘s and Thomas J. Leeper‘s blogs.]
It’s true that “there is fundamentally nothing to stop a nefarious, ne’er-do-well academic poser from completely fabricating the interview data”. But in the case of a requirement to deposit a transcript, it does substantially increases the costs (principally costs of imagination, but also costs in terms of time, which are costs shared by honest researchers) of fabrication. I think “will this increase the costs of fabrication” is perhaps a more useful lens than “while this prevent fabrication”.
Thanks Chris. That may perhaps be the case. But I doubt anyone willing to fabricate interviews would be all that deterred by a higher cost.
Two points. One, ethics in research is unavoidable, as we saw in the LaCour scandal. Formal rules are not sufficient.
Two, there has recently been a controversy about an ethnography carried out by Alice Goffman, who used ethnographic data built up from undergrad to publish a lauded book on police treatment of blacks in certain neighborhoods of Baltimore. Critics accused her of making up portions of her ethnography and hiding behind IRB rules.
One of her detractors decided to investigate and was able to identify some of her key participants, who confirmed her research. There were a few discrepancies, but that is not my point. My point is that even after Goffman anonymized and changed times/locations, etc to hide the identity of her participants, it was still possible to reconstruct her research. So this shows that there is some ‘reproducibility’ to even ethnographic research.
https://www.slate.com/articles/news_and_politics/crime/2015/06/alice_goffman_s_on_the_run_is_the_sociologist_to_blame_for_the_inconsistencies.html
I think the ‘transparency’ issue for qualitative data is similar to the voter fraud rationale for restricting votes. Its a power move to address a problem that doesn’t really exist, at least in any widespread manner that matters. Its bad enough that qualitative scholars have to jump through all of the IRB hoops, the DART rule will make a number of qualitative research projects impossible, as no one will participate in the research. This movement is misguided.
Also what Harry Stern said is correct. It takes a lot of work to get qualitative data, if it gets put up at the dissertation stage, advanced scholars are going to publish several articles before emerging scholars can get onto the publishing treadmill. Not fair.
“I think the ‘transparency’ issue for qualitative data is similar to the
voter fraud rationale for restricting votes. Its a power move to address
a problem that doesn’t really exist, at least in any widespread manner
that matters.”
And worth pointing out that in an era of increasing desk rejections, the DA-RT rules might provide a useful basis for editors to desk reject qualitative work.
It’s a bit late in the day to maintain that research quality/transparency is a “problem that doesn’t really exist,” nor is it clear why a rule on transparency would make qualitative research projects “impossible” (the statement directly acknowledges that many forms of data cannot be disclosed publicly).
I qualified my statements. Voter fraud exists in the US, but it is not significant and cannot be totally eliminated.
As Hanretty points out above, the proposal is not going to prevent the
problem that it addresses and it will significantly hinder qualitative
research. High quality qualitative research already discloses methods. I
fully support transparency for methods.
But any form of requirement for making transcripts public or even available to publishers (what if they are hacked!) is going significantly reduce the amount of qualitative research that is produced.
Say you are investigating Syrian refugees in Europe. Many of these people will worry that you work for the Syrian regime or some intelligence agency. Your only hope is that your commitment not to disclose any identifying information will get a few of them to talk, possibly after someone else they trust vouches for you. All it will take is a few researchers publishing transcripts or a hacking leak on topics such as these and you won’t be able to get people from NGOs to vouch for you.
Now say you are interviewing elite public figures. Many of these people will not talk to you on the record, should your research be limited to those that are willing to put up with public scrutiny?
I said it wasn’t a problem because I am talking about qualitative researchers, specifically ethnographers and interviewers.
Qualitative research is based on epistemology that does not find reproducibility relevant. A good portion of quantitative research in the social sciences is not reproducible, why have that standard for qualitative research?
I totally see the point about the need for privacy and confidentiality here. Yet I don’t see how that is an argument against DA-RT. The statement already expressly acknowledges that data can be “classified, require confidentiality protections, were obtained under a non-disclosure agreement, or have inherent logistical constraints.”
Why do you suppose that the transcripts in question wouldn’t be subject to one or more of those broad exceptions?
Think about this from another angle. Many datasets (from education to health to criminal justice) cannot legally be made public. Sometimes such datasets are literally required to be “destroyed” after use. Sometimes such datasets cannot even be accessed on a computer that is connected to the Internet, but must be walled off so that they can never be transferred anywhere. Sometimes the federal agency in question doesn’t even let ANY researcher have the dataset, but instead requires them to travel to a secure facility and exit with nothing more than statistical results.
So there are many cases where quantitative research cannot share data either. Yet I don’t see quantitative researchers saying that any transparency principles are somehow going to impinge on their research, or put their research at some sort of disadvantage.
Instead, quantitative researchers who work on education, health, crime, etc., seem to be without complaint. Given that no one has ever proposed transparency principles that would override legal or privacy restrictions, this is as it should be.
What makes qualitative researchers think that their research is more at risk than quantitative researchers whose data is every bit as confidential?
The problem I have is that qualitative researchers are already running uphill in the publishing game, and this provides another excuse for reviewers to disqualify qualitative research. And I think that some of the most important qualitative research has more capacity to destroy lives and even get people killed than the quantitative research you are describing.
But again, I don’t see what reason there is to think that any journals or reviewers could with the slightest plausibility rely on DA-RT to disqualify qualitative research, any more than they would use DA-RT to disqualify research on highly confidential IRS data.
“No one checks the decisions Polity or Correlates of War made in their coding of each datapoint, much less the data they were coding, much less that whiz-bang novel dataset that nobody but you is using. No one assembles a new Polity to see if they get the same numbers as existing Polity. And that is what true reproducibility would require. But we don’t see new Polity datasets because we aren’t really interested in datareproducibility, but rather analytical reproducibility—the weighting decisions, whether a spline was the appropriate mode of interpolation, whether there was a lag factor, and so on.”
I don’t think this is accurate. People notice coding problems or questionable cases that might drive the results all the time. I’ve certainly contacted the curators of big datasets to point out inconsistencies or ambiguities about how things are coded, and I’ve seen debates over results that hinge on how particular cases are treated. I think we’re interested in both data and analytical reproducibility. The LaCour scandal was certainly about data reproducibility.
But at any rate I don’t think this claim is critical for your broader point that there are legal and structural obstacles to posting qualitative data and that it’s worth thinking about how to handle this in an initiative like DART.
Thanks Terry. I was overstating the case obviously, in part to be provocative. I totally agree, that specific data points can and are often checked out. But I do think that singular or small numbers of data checking/confirmation does not constitute true replication in the sense that DA-RT seems to be advocating. And to the best of my knowledge there hasn’t been data reproduction on a scale that would count; I don’t know of anyone who has ‘reproduced’ Polity or other big datasets.
The objections here and elsewhere seem to miss the point. If greater transparency about data and code is a good thing (and it’s hard to argue that transparency is bad in and of itself), then it is not an argument against DART to point out that not everything can be made transparent.
Ethical concerns and legal concerns are everywhere. For example, it is not legal to publicly share many datasets on education, health, or criminal justice. Nonetheless, it is still good for scholars in those fields to be as transparent as they can — they can still post code, for example, and they can also post instructions for how other scholars could obtain the same data from the same agency or agencies. These are steps in the right direction, and given how much scholarship in so many fields has suffered from reproducibility problems when put to the slightest examination, the “just trust us” argument from Jeffrey Isaac doesn’t hold much weight.
There is also the issue of valuable, quasi-proprietary information. If I have taken months – maybe years – cultivating relationships in the field to conduct interviews, I want to milk those interviews for all that they are worth. They are what my career, esp. at the start, may depend on (not actually in my own personal case, because my work is mainly archival, but for many others). I do not want to make them available to all and sundry to use instead of me.
The top economics journal (AER) has a policy quite similar to DA-RT. The policy begins, “It is the policy of the American Economic Review to publish papers only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication. Authors of accepted papers that contain empirical work, simulations, or experimental work must provide to the Review, prior to publication, the data, programs, and other details of the computations sufficient to permit replication. These will be posted on the AER Web site. The Editor should be notified at the time of submission if the data used in a paper are proprietary or if, for some other reason, the requirements above cannot be met.”
Yet that journal regularly publishes articles whose data cannot be disclosed, not to anyone. Example: The Chetty/Rockoff/Friedman paper that looked both at school test scores and at tax records. https://www.aeaweb.org/articles.php?doi=10.1257/aer.104.9.2593
I have never heard any economist claim that a policy of data transparency at AER puts their research based on confidential data at a disadvantage.
So is it that political scientists do not trust each other to behave with the same common sense and decency as economists? That is, unlike economists, political scientists are worried that under a policy of data transparency, even one that makes broad exceptions for confidential data, their colleagues are going to use transparency as an excuse to kill papers with confidential data?
The more I think about the issue, the more I find myself worried about possible “chilling effects” that a norm of making interview transcripts publicly available could have. We’ve just seen the case of the IRA tapes at Boston College. Whatever else the case may or may not do, for sure it must send a very chilling message to any current or ex paramilitary/guerrilla/terrorist/criminal with regard to speaking with researchers. It will do so even if the researcher can plausibly assure the potential interviewee that nothing of the kind will happen to his transcripts: the interviewee will not be able to be sure that this assurance is reliable, since he/she will most likely lack the kind of deep knowledge of relevant legal & political matters required to really assess the interviewer’s assurances — but s/he will be aware of the Boston College precedent.
Much the same may happen with a norm of open transcripts. I’m just reading an extremely important paper on Chinese economic policy making, based on interviews with top-level officials. It is certain that none of these people would be willing to see transcripts of the interviews disclosed publicly. Now assume such a norm of transcript disclosure has actually become established. The researcher will of course assure them that nothing will be made public. But how can they really be sure? They’re vaguely aware of the foreigners increasingly having this norm of making transcripts public. A while ago there was a case of some unfortunate local official getting into real trouble when some interview transcript ended up online. They don’t know the details of the legal, institutional and political powers & frameworks in force in the researcher’s home country and university. They do know that researchers – esp. junior ones – are generally weak people (in the sense of having little or no political or institutional power — witness Boston College). They also know that any transcripts made from the interviews they were going to give the researcher would be widely read and re-used by other scholars. [I for one would love to read transcripts or notes from the interviews done for the paper I’m reading.] Will they still talk? Maybe. But it is to be expected that they will think even more carefully about whether they do do so, and what they say, because the promised confidentiality will have become even more attenuated.
Great points Harry. Thanks for reading and commenting!
The Boston College IRA incident seems like a complete red herring. The subject here is whether journals should promote a transparency policy that, as has been noted many times, makes numerous exceptions for confidential information. Yes, police might seek to subpoena tapes or transcripts about murder investigations, but they can do that regardless of whether journals adopt DA-RT or not.
So you are sure that DART will have no effect on the ability of researchers to have access to confidential information and that journals will be just as willing to publish research that doesn’t disclose its sources as research that does not?
That wouldn’t be a logical consequence, so the only way that would happen is if some political scientists are operating in bad faith. Once again, I would ask for even one specific example of anyone using a transparency requirement in bad faith.
In order for us to establish that, we need data about rejected research. So arguably, before DA-RT is implemented, journals should be required to be TRANSPARENT about what sorts of research doesn’t get published. You are chasing the wrong transparency goal, imo.
This is an interesting discussion, thanks.
I’d just like to put the perspective across of a grad student in political science.
I do qualitative work involving interviews around the topic of political violence in sensitive locations. To do this work I have had to become fluent in a difficult language, travel to dangerous places, put myself at personal risk, and protect my interlocutors.
I am not at the stage where I am finishing writing. Initiatives like DA-RT, as well as the general ‘mood’ in political science means my work will be far more difficult to publish in ‘good’ journals than somebody who has sat at my home institution collecting data from a desk, creating a dataset, and running analyses. I certainly don’t have any doubt about the value of their work but I am puzzled why my work will now essentially be harder to publish than theirs- my transcripts cannot be published, nor my field notes but- nonetheless- my research is most certainly replicable. But this initiative is just a symptom of the wider problem. For all the statements to the contrary it is self-evident that DA-RT does represent the discipline-wide embrace of a particular vision of what science should be. As I’ve made my way through grad school new professors have been hired at our department – more than one has expressed ‘concern’ that my work will never get me a job, apparently oblivious to the fact that this is only because they are replicating a certain point of view, and will do so when sitting on hiring committees. In the future, it is now unavoidable, scholars in political science will do what they do, basing their general theories (because – of course – theorizing itself is no longer valued in the discipline) on the insights of more eclectic, welcoming, and – let’s be blunt – interesting disciplines, before moving on to the brute number crunching. Consider – say – the introduction to Kalyvas’ logic of violence in civil wars. How many qualitative studies does he cite to get to his basic argument? How much knowledge is he relying on outside the reductionist model of science he embraces to make that argument? In the future none of those citations will be from within political science.
As for me, I’ve just applied for a grant with a professor from another department – sociology – that, I am hoping most fervently will be accepted. This will fund my research generously, alongside that of a few PhD students, but it will not be in the name of political science. And as I now submit articles to journals, I hedge my bets – submitting to sociology journals alongside those in political science.
All being well – for my personal future – I will soon no longer be a ‘political scientist.’
And I am not the only one – others in my cohort are doing the same – leaving the discipline that just a few years ago we were fascinated by. The loss will be felt for political science, alone.
“my work will now essentially be harder to publish than theirs”
If this were true, I would agree with you. But no one has produced any evidence or reason to think that this would be true.
This is not just about the DA-RT initiative, it’s in general – denying it’s getting harder to publish qualitative work that does not conform to a particular standard of science in political science is just ignoring the experience of many scholars. Believe me, why would I be trying to leave the discipline in which I am trained – which will make it harder, obviously, to be employed in another discipline, if I did not see myself as having no future in political science.
Perhaps I’m wrong, but along with others in the same position, we are not willing to ‘bet’ on political science remaining open to this kind of work any longer.
Dear Stuart,
Since – as far as I can gather from your website (https://about.me/stuartbuck) – you are not in fact a political scientist, in the sense of having neither done graduate training in a political science department (as opposed to a program on “education reform” housed in a department of public policy) nor currently teaching at a political science department, perhaps you should be careful about so easily dismissing the perspectives of those who are in fact political scientists?
Neither is there any “Harry Stern” who shows up on Google as affiliated with a political science department. Of course, I might have missed something, but no matter who you are, or if your name is even Harry Stern, that is irrelevant to the discussion. If you can make an argument (as opposed to ad hominems) that journals would actually use transparency as an excuse to disfavor interview-based research, I am happy to discuss those arguments with you or anyone else.
Rest assured that I am a Political Science PhD candidate in a top-ten program. Even if I travel by alias. As for ad hominems, the question is whether or not the characteristics of the person alluded to are relevant to the discussion. For instance, if I called you a rightwing hack, that would indeed be irrelevant. (You may be relieved to hear that I have no idea about, or interest in, your political leanings.) In the current case, I’m suggesting that given the absence of firm, quantitative data on either side, the insights and perceptions of those who are actually in the discipline and deeply exposed to its workings and trends might have greater evidential significance than the opinions of someone who appears to have only a passing acquaintance with it — like, perhaps, you. I for one can second every point made by Tim.
Great points, Harry.
I would add, however, that this debate is taking place among people already fairly entrenched within the discipline – Professors who have had the good fortune to advance within it. Things look a lot less clear-cut when one is a graduate student pondering the future while others – who have less to loose – decide what should be what…
Fair enough. But it is still necessary to make an actual argument or describe an actual experience. The point here isn’t whether qualitative research is getting harder to publish (I accept your testimony that it is), but whether such difficulties have anything to do with journal transparency requirements.
As I have mentioned upstream, economists have managed to figure out how to implement transparency requirements without penalizing research whose data is highly confidential. It doesn’t seem that difficult a problem, particularly if the transparency requirement has exceptions for confidential data (as DA-RT does).
If you or anyone else can point to any actual example of political science journals behaving with less common sense than economics journals on this point, that would be useful.
That’s not the general point.
The point is that DA-RT represents a particular understanding of science which is aside from issues of transparency. The fact that you are citing economics journals, of all possible options, as your example proves this point.
Also, just logistically, what you are now saying is that I will have to negotiate with editors about the possibility of submitting their before hand. This, in and of itself, makes my work harder.
There is already a bias against qualitative work in most political science journals, that bias just got worse.
Tim is on the money. I left Political Science/IR because of the bias against qualitative work. The people that are least likely to understand these issues are economists. Its comparing apples to oranges.
Well, my suggestion is this. Instead of running around the internet advocating for treating all research the same, you should be lobbying journals to publish data on the type of research they reject. I want to know whether it was qualitative or quantitative, the topic, and whether confidential data was involved.
What you don’t seem to get is that confidentiality is a mainstay of qualitative research, while it is a minority of cases in quantiative research. So any rule you apply to confidential info, disproportionally affects qualitative research.