Seven Reasons We Use Annotation for Transparent Inquiry (ATI)

15 May 2018, 1051 EDT

This is a guest post from Paul Musgrave, Assistant Professor of political science at the University of Massachusetts-Amherst, and  Sebastian Karcher, Associate Director of the Qualitative Data Repository at Syracuse University.

Recently, the Qualitative Data Repository launched “Annotation for Transparent Inquiry (ATI)” as a method to add transparency to scholarly research. ATI is a new approach to communicating scholarly evidence that employs electronic annotations to specific passages in scholarly articles—a sort of amped-up academic version of Genius.com’s annotations to song lyrics.

The goal of ATI is to facilitate future researchers’ work by enabling easier access to underlying data while enhancing research transparency by letting authors share specific justifications for interpretive or empirical judgments and linking them to the specific sections.

ATI builds upon but goes beyond Andrew Moravcsik’s proposal for active citation to include specific frameworks for data display, storage, and retrieval. QDR, partnering with the software nonprofit Hypothesis, is developing standards and software to support this initiative.

An initial nine sample annotated articles are now available, with more on the way. As part of this effort, QDR is currently soliciting applications to participate in a second round of pilot projects. Authors of selected projects will be invited to a workshop to help shape the future of ATI and receive a honorarium.

In this post, Paul Musgrave, whose annotated International Organization piece (co-authored with Dan Nexon) was one of the pilot articles, and Sebastian Karcher, one of ATI’s creators, reflect on some lessons from the first round of pilots.

  1. ATI lets authors share more research flexibly and comprehensively

Sebastian, one of the ATI creators: ATI annotations allow both author and reader to go deeper in the research process. Is there a complex backstory to obtaining a piece of evidence? Include it in an annotation. Would a longer excerpt from an interview or document than you have room for in your article bolster your claims? ATI annotations offer unlimited room for such excerpts. Is there a document that would be hard for readers to obtain but provides valuable context? You can include it as a data source.

At the core of ATI is the idea to showcase the depth of qualitative research. Research goes far beyond the 10,000 words of your article, and ATI gives authors a way to showcase this depth—not only to make the work more convincing but also to make it easier for readers to go beyond the article.

  1. ATI helps authors double-check their work

Paul, one of the authors in the pilot: Writing qualitative annotations really forced me to remember exactly why my co-author and I chose to include this piece of evidence instead of other ones as well as the interpretive debates that shaped my analysis. Even just a few months after writing a section, I might forget the precise details of where a quotation came from or how it fit in to a larger debate. Qualitative annotation made me fix all (well, at least most) of those details.

Had I kept better track of my analysis in my notes—rather than just the bits of evidence and quotations themselves—this would have been easier. One step that I’m considering as a result of this process is to start my own “lab notebook” to systematically capture as many of the intermediate steps in judgment as I can. As other people have noted, it’s never too early to develop good habits. Knowing that eventually justifications will end up in a qualitative annotation can help discipline writers.

  1. ATI lets authors respond to detailed critiques while respecting qualitative data

Paul: The peer review process privileges the scrutiny of outsiders—editors and reviewers—in a way that disadvantages qualitative work relative to quantitative scholars. Technical sophistication means that quantitative scholars have invented ways to display massive amounts of data in tables, charts, and appendices; a long-standing drive for replicability means that these efforts are increasingly expected, which helps the scholarly enterprise. But it’s much harder for scholars working in qualitative traditions to employ the same techniques because of the nature of our data.

There are obvious reasons why this system has evolved in this manner. But it’s time to level the playing field. Qualitative annotations (and other strategies, like online appendices) enable authors to demonstrate far more ways that we’ve interrogated our own material. That not only includes responding to outsiders’ scrutiny but also showing the structure of our arguments more fully, including the connections and perspectives that wouldn’t occur to someone who hasn’t thought about the material as deeply as we have. This “inside scrutiny” can be valuable for showing deeply engaged audiences exactly how we reached our decisions—and the requirements of transparency can force ourselves to be as honest and as documented as we can be in presenting these arguments.

  1. Displaying beautiful evidence

Sebastian: Qualitative research is full of fascinating encounters, be it encounters with people or with historical documents. One example: The Social Science Research Council has a photo competition (with some stunning entries) among its grant recipients, most of whom are doing qualitative or mixed-method research.

Yet our published work rarely reflects this excitement. While quantitative researchers produce evermore engaging visualizations, our articles are mostly text with the occasional diagram or table. ATI offers a way to bring the fieldwork into the article and to the reader. This can take many forms. Sometimes it could be longer text excerpts—perhaps both in the original language and in translation. Among the forthcoming ATI pilots, Ariane Tabatabai and Annie Tracy Samuels, use extensive quotes and translation from Farsi text to elucidate foreign policy debates within Iran.

Conservation researcher Marisa Rinkus, in another forthcoming project, added photos from her field site to provide texture to her description. To bring the accents of their participants in the Scottish village of Buckie to life, sociolinguists Jennifer Smith and Sophie Holmes-Elliott are able to provide recordings from interviews to accompany the phonetic transcripts included in their article.

But ATI annotations go beyond the mere analytic-aesthetic appeal of, e.g., graphs used to describe quantitative results. By inviting the reader into the research process and the underlying data, they invite active engagement with the material, especially among the most interested readers.

  1. Annotation provides deeper engagement for more interested scholars

Paul: People sometimes ask: Who’s going to read all of this? It’s a fair question! I asked it myself frequently while writing my notes. But I think it’s too pessimistic and proceeds from a bit of a false premise.

First, we should remember something effaced by the pressure to publish ever-thinner salami slices of work: scholarly communication isn’t supposed to be for wide audiences. It aims at the truth, not at popularity. And if sometimes the truth requires a few thousand words to rebut its rivals, well, that’s why researchers chose the vocation we did.

Second, as a practical matter, the who’s-gonna-read-it question exaggerates any given reader’s commitment to the text. Not everyone will read everything. And that’s fine! Unless my own reading habits are well off the mark, I feel confident in stating that most researchers, like me, haven’t digested every word in every article we’ve ever “read”. Some readers are already only skimming our arguments, even without annotation. So if some readers only read a few annotations, that’s still a win for that reader and for the author.

The most important readers from the perspective of truth-seeking, after all, are those readers who are best in a position to challenge our claims. And for them there is no limit to how much information we can provide about our projects. (Rarely, if ever, have I felt that the texts I seek to dispute or rely upon have provided me with too much information.) Qualitative annotation (and their kin, qualitative appendices) enable these most-invested readers, as well as apprentice scholars, with the opportunity to read our work as deeply as possible.

  1. Promoting “slow scholarship” and helping to prioritize depth of knowledge

Paul: We have to talk about the time commitment. Like the other participants at the recent ATI workshop, I found that composing my annotations took time—for our project, about 20 to 30 hours (Sebastian: that’s in line with other participants, most of whom report spending between 10 and 40 hs).

That’s probably misleading about how long it will take for the next time I use ATI. First, one big factor helped speed the process up: much of what I used had been written, or at least sketched out, for the manuscript itself, but the demands of structure and word limits meant that these arguments were taken out during revisions or even before submission. (Of course, I suspect other people will do the same!)

Second, a much more important point slowed it down: there were several months between our finishing the manuscript and writing up the annotations; if we’d done them at the same time, or at least during the same cycle, they would have gone much faster.

And, third, I’d been less systematic than I should have been in tracking my evidence; using Evernote and other tools even more would have made it much easier to find some of the trickier citations.

I’m of mixed feelings about this. Time is my most precious resource, and spending more of it on a project than I have to is something I aspire to avoid. On the other hand, the ATI annotations and qualitative appendix for our project reflects far better the work my co-author and I actually did. The marginal time I invested in annotating didn’t take away from anything so much as it enabled me to bring more to the conversation. Without the time and the technology, much of that work would have languished. (And, yes, I do think that readers, letter-writers, and colleagues should be impressed with that effort, just as I’m impressed by appendices that carefully lay out evidence and reasoning that supplement a text.) Promoting ways to bring the depth of scholarship to light is one way to bolster slow scholarship.

  1. ATI helps promote better planning.

Sebastian: The time required to generate ATI annotations or similar products also depends on how research was organized. Several researchers told us that keeping data and sources well organized dramatically reduced the needed effort:

“Having everything to hand in the way we did really underscored the value in keeping regular and up-to-date catalogues.”

Other researchers (most of the 20 participating authors, in fact) emphasized that better planning would have helped them dramatically reduce the time needed. Several mixed-method researchers contrasted this with already existing practices for the quantitative components of their study:

“We had fairly good practices in place for documenting our quantitative analysis but were much less systematic with respect to our qualitative data and source material. Going forward I will definitely put a lot more effort into generating digitals cans of primary material, cataloging quotes, and systematically organizing primary source material.”

Qualitative data sharing and transparency are much more recent developments than similar practices among quantitative researchers, so establishing best practices for organizing and training researchers to apply them during their fieldwork will not happen tomorrow. But doing so carries benefits that go beyond our ability to comply with transparency mandates. They make us better researchers.