« “Why is a big gift from the federal government a matter of coercion? ... It’s just a boatload of federal money for you to take and spend on poor people’s health care” or the mysterious coercion theory in the ACA case | Main | Reflections on the Rhythm of Academic Life »

Wednesday, May 08, 2013

Thinking Carefully About Statistics: Gun Suicides (Updated)

There has been a lot of press attention on the CDC's recent report that gun-related suicides have jumped 28% between 1999 and 2010 for middle-aged people (35-64). Over at Atlantic Cities, Richard Florida attempts to explain the geography of suicides, tying it to gun laws: more access to guns, greater increase in suicides.

It's quite possible that gun laws and suicide are correlated, and I don't intend to get into a debate about gun control laws in this post. Instead, I want to draw attention to the fact that we too often attempt to force causal explanations onto data without first thinking carefully about the extent to which the results are just an artifact of baserates, regressions to the mean, etc., etc. And we perhaps too-readily overlook the risk of colinear or spurious relationships. That may be the case here--at least the risk is great enough that it demands attention.

The problem should be clear as soon as we look at the states with the ten largest increases in suicide rates:

From largest percent increase to tenth largest: Wyoming, North Dakota, Rhode Island, Hawaii, Vermont, Arkansas, Idaho, Indiana, Oregon, and South Dakota.

What immediately jumps out at me? It isn't economics (North Dakota is booming), it isn't politics (Wyoming and Rhode Island?), and it isn't geographic location. 

It's population size. Here is that list again, with each state's 2012 rank for population size:

Wyoming (50/50), North Dakota (48/50), Rhode Island (43/50), Hawaii (40/50), Vermont (49/50), Arkansas (32), Idaho (39), Indiana (16), Oregon (27), and South Dakota (46).

Only one state in the top half of population size is on the list (with one other close call), but the list does include the three smallest states and five of the ten smallest. Collectively, these ten states are home to just over 6% of the nation's popuation; the five smallest, just 0.01% (barely more people than Brooklyn, but without the bespoke mayonaisse). 

In other words, this is almost exactly the list we would expect to see if suicides increased by the same absolute amount across the country, since one more suicide in Wyoming (population 576,000) affects its suicide rate about 65 times more than one extra suicide in California (population 38 million).

And the baserates here are low: Wyoming's suicide rate of 31 translates into about 180 suicides per year, up from 100 in 1999. So it may not be surprising for absolute numbers to be similar across states, implying that the trends in suicide rates just reflect the differences in the population denominator.

Two immediate points:

1. Noise: To what extent is the top ten list simply a function of baserates, not of any policy decision?

2. Spurious correlation: If there is a causal link, it may be easy to think of one that could explain why small states simltaneously have weak gun control laws and high suicide rates without needing the former to cause the latter. I had thought that small states like ND might have the most lopsided gender ratios, which could foster both a pro-gun culture while leading to depression among men unable to compete in a limited marriage market, although the raw ratios from 2000 aren't compelling (but four states on the top ten list here are also in the top ten for highest ratios of men to women). But one can think of other explanations--people who like guns may like the freedom of sparse states but also find them too lonely when things go wrong.

The second point is well-known and is almost a parlour-game among empiricists: unless you have a clinical trial, we can all sit back and play "find the uncontrolled-for feasible cause you left out of your model."

But the first point is all too often overlooked when we discuss results like these. Sometimes results aren't tied to a particular causal story but are the product of baserates. Or, put more carefully, at least some part of the explanation is just baserates--nothing here should be read to say that gun laws are irrelevant (though they may be) or that they are an important causal factor (though they may be).

We should always try to account for these sorts of baserates issues before jumping into the more-complex causal story. It is imperative that empiricists and journalists/bloggers take them into account when generating or reporting results, and consumers of empirical results need to always keep them in mind as well.

UPDATE: A friend of mine pointed to a spurious-correlation possibility I had toyed with but hadn't pursued. Whites commit suicide at a much higher rate than blacks: they are about 2.5 times more likely to commit suicide. Which points to another good spurious correlation story, Sparsely-populated states tend to be very white states, so sparse population plus higher suicide rates for whites in general implies that small white states should see faster rates of suicide growth regardless of any other policy change.

Another way to think about all this is that we care about the marginal contribution of gun laws to suicide rates, and it is impossible to compute the marginal effect without first taking into account these sorts of baserate effects. 

Posted by John Pfaff on May 8, 2013 at 12:23 PM | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef019101e5df7c970c

Listed below are links to weblogs that reference Thinking Carefully About Statistics: Gun Suicides (Updated):

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

I'm not sure why the post focuses on statewide recent increases in suicide rates, which is not the focus of the Florida article, and would not be the way that one would test the relationship between firearms and suicide. Instead, you would examine suicide rates in households with and without firearms, no? The studies looking at that data have found the presence of a firearm in a household increases the likelihood of a firearm-related suicide. See Matthew Miller, Deborah Azrael & David Hemenway, Firearms and Violent Death in the United States, in Reducing Gun Violence in America 4, 11-13 (Daniel W. Webster & Jon S. Vernick eds., 2013).

Larry Rosenthal
Chapman University School of Law

Posted by: Larry Rosenthal | May 8, 2013 8:29:59 PM

I was incorrect to say that the CDC report was focused on gun-related suicides: it was just about suicides in general, so I've corrected that. But Florida *is* looking at gun laws, arguing that laxer gun laws lead to greater numbers of overall suicides.

But if we want to understand the relationship between firearms and suicide, we actually need to look at the overall suicide rate, not the firearm-related suicide rate. A common argument employed by those opposed to gun control legislation is to argue that you can't attribute killings to the guns, since people would just use something else to kill if the gun wasn't available. For things like drive-by shooting and mass shootings, the argument is preposterous. (Just compare the outcome of the knife attack that took place in a Chinese school the same day as the Newtown shooting: http://abcnews.go.com/blogs/headlines/2012/12/22-kids-slashed-in-china-elementary-school-knife-attack/.) It could even be implausible for things like heat-of-the-moment acts of domestic violence, which simply can't escalate in quite the same way with a knife as with a gun.

But for suicides the claim deserves a bit more attention. Suicide is, I believe, a much more intention act, and someone focused on killing himself will accomplish one way or the other. Guns are likely the easiest and most effective way, so it isn't surprising that more guns lead to more gun suicides. But do they lead to more suicides overall, or is it just a substitution effect?

In his article, Florida cites HSPH study that claims interstate variations in suicide are due primarily to gun suicides, but that doesn't clearly identify the role of guns. Would people in Wyoming kill themselves less often if they had fewer firearms? That they choose to kill themselves with guns when they make the decision to commit suicide doesn't actually answer the question, although that is all that the HSPH study establishes.

Ideally, I'd like to see if tighter gun laws lead to fewer overall suicides, since that would tell me that whatever substitution effect is out there isn't 1-1: some suicides are being actively prevented by limiting access to one particular way of killing oneself. (And there is certainly a viable story to explain that: the longer it takes someone to commit suicide, the more opportunities he has to change his mind.) But the gun-suicide rate doesn't tell me anything about such substitutions, and the weapon-substitution argument is strongest for suicides.

So from a Bayesian perspective, the HSPH study is certainly evidence in favor of the hypothesis that more guns lead to more suicide. But it's effect on whatever your priors are should be muted.

It just struck me looking at the list of states that fastest growth rates was highly correlated with small population size, which are states that are going to have faster moves in either direction simply because of smaller baserates. And such factors demand a certain amount of caution I didn't see in Florida's article.

Posted by: John Pfaff | May 8, 2013 10:51:49 PM

From the article I cited above (at pp. 4-5): "[T]he overall suicide rate in the United States is 1.6 times that of the average of other high-income countries, largely accounted for by a firearm suicide rate eight times the average of these countries."

As it happens, there is a wealth of public health literature addressing the question of substitution effects when it comes to suicide, and finding that it does not exist. This is largely a function of the fact that a suicide attempt with a firearm is far more likely to be fatal than an attempt using other leading methods.

Posted by: Larry Rosenthal | May 9, 2013 1:05:54 AM

I'm not at all surprised that there appears to be a causal link between guns and suicide levels, and that the substitution story can be oversold.

But in this case, there is still reason to be concerned about how strong the connection is between access to guns and the *rates of change* that Florida is looking at (as opposed to levels, which is what you point to). If the 10 highest increases were in Wyoming and then nine high-population states like Florida and Georgia (although with half the population crammed into just ten states, there aren't that many "high population states" out there), the story would be more convincing, and my emphasis on Wyoming being the least populated state woud seem nitpicky. But almost all of the largest increases are in very small states, and in general smaller states will exhibit larger percent changes simply because they are smaller, not because of any underlying policy differences.

In other words, there is no reason to assume that smaller states will higher levels of suicide than larger states simply because of math, although the variance in suicide rates should be larger. But if the nation as a whole is experiencing an upward trend, then it seems to me that the math alone increases the probability that smaller states will be the fastest growers simply due to size, due to the larger variance around the trend line.

That isn't to say that Florida is wrong, just that he is isn't giving enough attention to the role of baserates. And I'm not making this point because I have any particular issue with either Florida or the gun-control literature, but because this is a common failure of statistical thinking we see in lots of situations, and this was a useful casestudy to use to make the point.

Posted by: John Pfaff | May 9, 2013 8:20:55 AM

From 1999 to 2010, the overall suicide rate increased by 28.4% (95% confidence interval 25.7-31.2) while the rate from firearms increased by 14.4% (95% CI 11.0-17.8). The difference is statistically significant. One can only speculate about the reason but more stringent gun regulations is a possibility. Although stringent gun regulations may have contributed to the slower increase in firearm suicides they apparently had little effect on overall suicide rates.

What you overlook is that about half of all suicides involve firearms. I would suppose that people do not use assault rifles or guns with large ammunition clips to kill themselves so that restricting the sale of such weapons will have little to no effect. Also you and most other partisans in the gun control arguments overlook the fact that about two thirds of all firearm deaths are suicides. While people argue about gun control laws, 20, 000 people (virtually all of them gun owners or their family members) kill themselves with a gun every year. If laws will not prevent these deaths, what will?

Posted by: Ron Ratney | May 10, 2013 10:02:40 AM

Post a comment