The Institute for Economics and Peace is making a big splash today with the release of the 2012 edition of its annual Global Peace Index (GPI)—“the world’s leading measure of global peacefulness,” according to its web site. The launch event for the 2012 edition included several people whose work I respect and admire, and the Institute identifies some of the heaviest hitters in the global fight for peace and human rights—Kofi Annan, Desmond Tutu, and the Dalai Lama, for crying out loud—as “endorsers” of the GPI.

I really want to like this index. I’m a numbers guy, and I’ve spent most of my career analyzing data on political violence and change. But, the closer I look, the less I see.

The basic problem is one that confounds our best efforts to develop summary measures of complex concepts in many fields. Complexity implies multi-dimensionality; the complex whole is composed of many different parts. As a result, no single indicator will capture all of the elements we believe to be relevant.

To try to overcome this problem, we can mathematically combine measures of those separate elements in a single scale—an index. Unfortunately, with truly complex phenomena, those parts do not always move in lock step with each other. As a result, we often wind up with a summary measure that obscures as much as it clarifies because it blinds us to those tensions. In some cases, we can see changes in the index, but we can’t tell what’s driving them. In other cases, the index doesn’t budge, but that doesn’t necessarily mean that there haven’t been significant changes that just happened to cancel each other out. In both of these scenarios, we’ve got a number, but we’re not really sure what it means.

We can see this dilemma clearly when we look closely at the GPI. According to the Institute’s documentation (PDF), the Global Peace Index represents a weighted combination of 23 indicators in three concept areas: 1) ongoing domestic and international conflict; 2) societal safety and security; and 3) militarization. The index includes so many things, we are told, because it aims to get simultaneously at two distinct ideas: not just “negative peace,” meaning the absence of violence, but also “positive peace,” meaning the presence of structures and institutions that create and sustain the absence of violence.

Some of the indicators are inherently quantitative, like counts of deaths from civil conflict and number of jailed population per 100,000 people. Others, such as “perceptions of criminality” and “military capability/sophistication,” are qualitative concepts that are scored by Economist Intelligence Unit staffers. All 23 are converted into comparable five-point scales and then aggregated according to an algorithm that involves weights assigned by an expert panel at the level of the individual indicator and at the level of two sub-component indices having to do with internal (60%) and external (40%) peace. Here’s a complete list of the 23 components:

• Number of external and internal conflicts fought in the past five years
• Estimated number of deaths from organized conflict (internal)
• Estimated number of deaths from organized conflict (external)
• Level of organized conflict (internal)
• Relations with neighboring countries
• Perceptions of criminality in society
• Number of refugees and displaced people as a percentage of the population
• Political instability
• Level of respect for human rights (Political Terror Scale)
• Potential for terrorist acts
• Number of homicides per 100,000 people
• Level of violent crime
• Likelihood of violent demonstrations
• Number of jailed population per 100,000 people
• Number of internal security officers and police per 100,000 people
• Military expenditures as a percent of GDP
• Number of armed services personnel per 100,000 people
• Volume of transfers (imports) of major conventional weapons per 100,000 people
• Volume of transfers (exports) of major conventional weapons per 100,000 people
• Budget support for UN peacekeeping missions: percentage of outstanding payments versus annual assessment to the budget of the current peacekeeping missions
• Aggregate number of heavy weapons per 100,000 people
• Ease of access to small arms and light weapons
• Military capability/sophistication

That’s a long list with a lot of very different elements that don’t always move in unison. More problematic in light of the GPI’s additive approach to combining them, those elements don’t always point in the same direction.

Take military expenditures and deaths from external conflicts. International relations scholars would tell you that countries can sometimes avoid wars by preparing for them; rival states are less likely to pick fights with armies they can’t easily beat. Most people would probably think of the avoidance of war as a peaceful outcome, but the GPI casts the preparations that sometimes help to produce that outcome as a diminution of peace. In an ideal world, disarmament and peace would always go together; in the real world, they don’t, but the index’s attempt to combine measures of negative and positive peace muddles that complexity.

The same goes for internal affairs. Imagine that a country is suffering a high homicide rate because of rampant criminal violence (Mexico? Venezuela?). As the GPI implies, that’s not a particularly peaceful situation. Now imagine that that country’s government invests heavily in policing to fight that crime, and that the expanded police presence leads to a decline in the homicide rate and to higher incarceration rates as criminals are arrested and imprisoned. According to the GPI, the gains in peacefulness realized by stopping the wave of murders would be (at least partially) offset by the increases in the size of the police force and the prison population. A change most citizens would regard as an unmitigated good gets washed out by the supposition that the means used to reach that end are detrimental to positive peace.

Now, put both of those problems and several others like them into a single box and shake vigorously. Instead of an elegant simplification, we end up with a complex tangle, simply represented. We see echoes of this problem in summary measures of democracy, like the 21-point Polity scale, which aggregates across several dimensions in ways that sometimes obscure differences of great importance and interest.

For an index to improve on its parts, it should capture something important that we miss when look at the components individually. In my opinion, one of the best examples of this is the Heat Index, which combines air temperature and relative humidity into a single number that we really care about: how hot it actually feels to us humans. The Heat Index is really useful because it gets at something we miss if we look at air temperature alone. The whole illuminates something that the single components can’t show.

Unfortunately, this is hard to do. In many situations, the individual components will offer sharper and more transparent measures of specific dimensions, and we’ll see more when we juxtapose instead of combining them. When we want to explore how these components relate to each other, we can start with two- or three-dimensional scatter plots, which quickly reveal interesting cases of reinforcing or competing tendencies. For more complex problems, multivariate models that relate the components to some observable ground truth (e.g., the absence of deaths from violent conflict) will often work better than indices that use expert judgment to assign weights and directionality.

In the case of the Global Peace Index, I think the starting point for a more useful set of measures would be to construct separate indices for positive vs. negative peace. From my reading of their project, this distinction is more relevant to their objectives than the internal vs. external peace distinction for which they currently report sub-indices, and these are the dimensions along which changes are most likely to be offsetting. This could be done separately for internal and external peace, producing four indices along which levels and movement could be compared and contrasted. Two-dimensional scatter plots could be used to compare countries overall (with positive and negative peace as the axes) or separately for domestic or international peace. To compare a few countries on all dimensions or to illustrate changes within countries over time, radar charts would be useful.

As I hope that last bit of constructive criticism makes clear, I don’t mean to knock the creators of the Global Peace Index for their thoughtful attempt to grapple with a very hard problem. I’d like to see them succeed; I just don’t think they have…yet. More generally, I think the ways in which their current effort falls short illustrate some common dilemmas of measurement that most social scientists face at one time or another.

This is a cross-post from my solo blog, Dart-Throwing Chimp.