Some Thoughts on Counting Terror, and Counting in General

This map of terror incidents across time and space in the United States (discussed more in detail here) is simultaneously fascinating and mind-boggling:

(This is only a screenshot: my efforts to embed the video itself failed miserably.)

It is impossible to look at this map and not immediately ask: “What exactly is a terror attack?” According to the research group at the University of Maryland that produced the map, “terror” is: 

The threatened or actual use of illegal force and violence by a non-state actor to attain a political, economic, religious, or social goal through fear, coercion, or intimidation….

Now that obviously can’t be entirely true. By this definition, every drive-by gang shooting would be an act of terror: a non-state actor using intimidation to secure economic gains. Arguably any act of domestic abuse, particularly in a semi-public or public place, could likewise be seen as an act of terror, since it represents an effort to use fear to advance a social goal (keeping women in a second-tier position). And so on. Simple armed robbery? Seems to fit. 

I think this map provides a clear example of a problem all too rarely discussed in statistics and the quantitative social sciences: no matter how hard we try make our models “objective,” at the end of the day the numbers we use in them are almost always quite subjective. Their "numberness" makes them seem objective, but they are not. And we need to either (1) openly note the underlying political/legal/moral assumptions that drive our numbers or—and this is my preferred option—(2) attempt to model more explicitly how our results vary with different political/legal/moral assumptions. 

In other words, there is no “objective” definition of terror, thus any measure of it we make will always be inherently political. If nothing else, note that the federal government uses a different definition than the one that undergirds this map: most tellingly, the federal definition drops “economic” from the justifications. Change the definition, change the results.

We see this in lots of situations. Illinois, for example, did not provide rape statistics to the Uniform Crime Reports for years because Illinois has used a different, broader definition of rape than the federal government. Or think about poverty: an income of $11,490 is poverty for one person, $15,510 for a family of two, and so on by $4,020 increments up the chain. That can’t be objectively “true”: why not increments of $5,020? Why not a baseline of $13,000?* Even murder: if you’re counting murder, should you count cases where someone kills another in self defense? Ostensibly, the person is still guilty of murder but has an affirmative defense. It may make sense to exclude that from a study of what-we-think-of-as-murder, but that’s a normative call.

 Yet social scientists almost never wrestle with these issues. We use the UCR’s rape or murder counts without asking if they really measure what we care about. And we insert “percent in poverty” into regressions all the time without carefully thinking about the host of normative and political decisions that go into that percentage and, more important, without thinking about whether those explicit and implicit assumptions are consistent with the research projects we are engaged in.

Sometimes our hands are tied: whoever gathers the statistics gathers them in a particular way, and it would be prohibitive to regather it ourselves. If the UCR excludes justifiable homicides from its murder stats, it is almost impossible for me to go back and collect that data myself. But sometimes it is laziness: poverty statistics are calculated from the CPS, and that is freely downloadable. I could compute my own poverty percentages if I wanted, but it would take time and energy, but it would force me to think more critically about what exactly I am measuring.

Yet even if it is impractical or inconvenient to recalculate the data, there are still steps we can take to better acknowledge the inherently normative content of the numbers we use. The first, and to me less appealing, appoach is to simply be more open about exactly what normative assumptions we—or the third party who gathered the data—are making when we arrive at the numbers in the data, and to defend those assumption.

But if the definition is ultimately normative or political, it really can’t be defended in any objective way: we choose it because we believe it, but we can’t “prove” that we’re right. Just like a utilitarian can’t “prove” a retributivist wrong and vice versa. That's what makes a "political" or "normative" issue political or normative in the first place. 

So the second, more appealing, approach is to model much more explicitly what turns on various political choices. Take the terror map. I would have loved to see three maps in a column, each showing the progression of terror attacks over time in sync, but using different definitions of terror. This would illuminate just how important the normative calls are, or perhaps for what questions they matter.

Stick with the terror example. Assume we all agree that Oklahoma City, 9/11, and a few other major events count as “terrorism.” Then it could be that our definition matters a lot for the number of terrorist events but is relatively unimportant for measuring the total number of terror-related deaths (since the clear-cut cases are few in number but dominate when it comes to human cost). So depending on what you are trying to count, the choice of definiton may be more or less important. That's really useful—essential—to know.

Yet we so rarely see this. The dominant approach is to pick a definition, perhaps—but generally not—defend it, and then treat it as the “right” definition, or at least the “best we can do” definition. But the fact that this is standard practice doesn’t mean it is the right way to go; it clearly isn’t. We need to acknowledge more openly the inherently political and normative nature of what we do, and we need to model much more honestly just how much turns on these normative and political choices.

And consumers of empirical work need to take care not to view the numbers in an empirical model as being perfectly objective measures of the phenomenon under question, at least most of the time. Perhaps if we want to measure “the total number of people in the room,” then an enumeration pulls that off in a purely objective way: either there are 10 people in the room or 11, and I don’t think we’ll fight over how to define “people” (but if I had said “adults”…?). But note that if we want “the total number of people in the country,” a number equally objective in theory and one that lacks the normative ambiguity of “poverty” or “murder” or “terror,” we face the fact that the decision to count rather than sample will produce different estimates. Unacknowledged subjectively lurks in almost every model; sometimes it matters, sometimes it doesn't, but the fact that it is never discussed is deeply troubling.

 

* Even worse, the poverty scheme was developed in the 1960s, and the only adjustment made to it since then is for inflation. So the standard seems likely to be out-of-date.