This post was written by Simone Dietrich, Heidi Hardt and Haley J. Swedlund. Simone Dietrich is Associate Professor at the Department of Political Science and International Relations at the University of Geneva. Heidi Hardt is Associate Professor at the Department of Political Science at the University of California, Irvine; a member of the 2015 International Policy Summer Institute cohort, and a 2021 Council on Foreign Relations International Affairs Fellow. Haley Swedlund is Assistant Professor at the Department of Political Science at Radboud University and a member of the 2019 International Policy Summer Institute cohort.

For decades, many International Relations (IR) scholars portrayed experiments with foreign policy elites as too risky, too costly, or too difficult to implement. Faculty mentors discouraged graduate students from wasting their time. In a new article in European Journal of International Relations, we argue that elite experiments are not as difficult to implement as many believe they are. However, they do require careful planning in order to get elites on board.

When are elite experiments worth the costs? What are some tips and tricks for successfully carrying out this method? How might this approach be helpful in bridging the gap between IR and policy?

The Value of Elite Experiments for Scholars and Policymakers

Elite experiments involve modifying surveys and interviews of senior officials to retain control over random assignment of experimental conditions with the goal of uncovering causal relationships. Following Hafner-Burton et al, we define elites to be core decision-makers who (a) occupy “top positions in social and political structures,” (b) “have the highest indices in their branch of authority,” and (c) “exercise significant influence over social and political change.”

For scholars interested in bridging the academic-policy gap, elite experiments are particularly appealing because they allow scholars to test theories about policy-relevant behavior directly on elite samples. Instead of sampling non-elites (e.g. college students, mTurk individuals, etc.), elite experiments query foreign policy elites, such as members of parliament, senior judges, ambassadors, or military representatives. This allows scholars to test theories directly on the population of interest.

Elite experiments can also help with problems related to social desirability bias. As a method, elite experiments can provide insights into closed-door decision-making processes without having to directly ask elites about a current, politically sensitive matter (e.g. North Korea nuclear talks, China trade negotiations, etc.). Without an experimental design, which can be embedded in a survey or interviews, an elite may simply give the politically-correct answer.

Furthermore, elite experiments provide a platform for communication and exchange between academics and practitioners. Scholars gain from increased transparency into the policymaking process; elites gain from policy recommendations based on the experimental findings.

Best Practices for Carrying out Elite Experiments

That said, we know that foreign policy elites are busy. Sample sizes will, therefore, inevitably be smaller. Moreover, access barriers mean that employing the method requires significant upfront costs. So how does one practically implement an elite experiment in IR? In our article in EJIR, we provide numerous practical tips on how to implement elite experiments. Here, we want to highlight three:

First, we found it critical to be flexible about the method of recruitment. We employed a range of techniques from “cold-call” phone calls and emails to drawing on our own networks with the policy community. We also relied on colleagues, friends and mentors to provide e-introductions as an entry point into an organization. We are skeptical that random sampling is fully possible with elite populations. However, if one knows the limits of a population, it is possible to snowball-sample until the sample is representative. For example, for her book and article on institutional memory at NATO, Hardt was able to establish a representative sample of NATO’s North Atlantic Council by contacting all 28 (now 30) state delegations/missions.

Second, depending on framing, the initial contact can either make or break an elite’s willingness to participate. Elites may be suspicious of a scholar’s motives, including concerns about being treated a data point or having their identity exposed. Thus, it is important to tailor engagement appropriately. In our experience, some elites never responded or declined to participate. However, we broadly found that many elites were eager to participate from the start, and others, though initially hesitant, could be “won over” through clarifications about the research and how we would eventually use the data. We found elites to be more likely to participate if they felt safe (i.e., through assurances of anonymity and of IRB approval) and if they understood what the specific benefits were to participation (see below). 

Third, elite experiments involve many of the same barriers and risks associated with other types of fieldwork. Many elites share racial, gender, and other biases that may interrupt or negatively affect scholars’ efforts at recruitment and data collection. For example, we personally experienced both white privilege and gender discrimination in some of our interactions with elites. Therefore, we urge scholars, especially graduate students, to communicate with mentors before and during fieldwork, to review political science fieldwork advice, and to widely read and listen to stories and tips from diverse scholars on how to safely navigate field research.

Making the Value of Elite Experiments Apparent to Policymakers

When conducting an elite experiment, it is important for scholars to think through how they want to share results and findings with elite participants. Relative to non-elite subjects, elites are much more likely to have an interest in and knowledge of the scholar’s topic of research. We found that elites were often quite interested in participating in research and willing to make time to respond to our questions—if the link with the research and their work was clear and if they believed they or their organization might get something out of it.

In their initial contacts with elites, we recommend scholars be upfront about how long it may take for elites to reap these benefits. While scholars know studies can take years to get completed and published, not all elites are familiar with these timelines. Also, many elites may already have rotated on to a different organization by the time a study is published. Therefore, we suggest that scholars request a long-term means of contact, such as a non-government email address, a LinkedIn profile connection, or a Twitter handle, at the time of the experiment or as part of a follow-up thank you email. However, scholars should exercise caution before connecting on social media with elites working in or with authoritarian regimes and/or governments exercising significant censorship.

A priori, scholars should determine which benefits they are willing to offer. In the words of one elite, “if I participate, what can you do for me?”. In surveys with non-elites, scholars might give monetary compensation or Amazon gift cards. For some elites, this might still work. However, many government officials are not allowed to accept financial compensation. Free benefits—such as knowledge-sharing—can often be a sufficient incentive.  All three of us, for example, were able to carry out elite experiments without financial remuneration.

Scholars also should think through how they want to share their research with elite subjects after the study is complete. One option is to go public (e.g. blog, opinion piece, podcast, giving a talk at a think tank or a policy community). This option makes it easy for elites to circulate your materials and ideas within an organization. It also means that information you circulate is likely to be more general and less specific to an elite’s particular position.

A second option is to send individual emails to all elites who participated in the experiment. These emails can contain a one-page memo or a list of recommendations, along with a PDF of one’s study. (Paywalls will often prevent an elite from otherwise accessing an article online.) While more time intensive, individual e-mails can be tailored to the specific contact and can be useful for maintaining contacts for subsequent studies. Dietrich shared published work and short executive summaries with aid officials that participated in the research – a practice which then lead to invitations to present the research at different aid agencies. 

A third option is to travel to give a policy briefing. The latter might take place in the form of a large-group presentation that elites and staff of a particular organization can attend (e.g. book talk, speaker series); a small, targeted presentation to elites in a specific office or working on a specific issue; or a private briefing with organization leadership. (Given the current pandemic, an alternative to in-person briefings would be to offer synchronous remote presentation/s to elites at the organization in question.) Swedlund, for example, conducted joint donor-government workshops in Accra, Ghana and Dar-es-Salaam, Tanzania. These events were valuable for not just sharing her research but also for providing a space for participants to collectively brainstorm about how to address the challenges raised by the findings.

Finally, despite elites’ questions, one should avoid sharing the names of previous elite subjects who were guaranteed anonymity and avoid sharing preliminary results. The latter could bias other elites’ responses. We navigated these requests by mentioning the names of bureaus or offices without specifying individuals and/or sharing information that was not politically sensitive or already publicly available.

We have found that, when done ethically, politely and persistently, elite experiments can be a mutually beneficial means of producing original scholarship while connecting foreign policy elites with scholarly insights. We hope that, under the appropriate conditions, more scholars will consider this method as a complement to other research methodologies.