« Greetings from the West Coast | Main | The Unappreciated Link between Health Insurance and Job Creation »
Friday, November 02, 2012
In Defense of Nate Silver
Glad to be back at prawfs. This month, I will be talking about the election (at least until November 6 and then I never want to talk about it again…seriously burned out) and then I will focus on the constitutionality of the Voting Rights Act. But first, let’s talk polls.
One thing that is notable about this election cycle is attempts by politically minded folks to either “unskew” polls that are unfavorable to their side or dismiss them outright because of “shady” methodology. Nate Silver, who runs the 538 blog at the New York Times, is the most recent victim of attacks from the right because his model gives President Obama a 80% chance of winning an election that everyone (including Silver) believes is a tie at this point. One of the reasons that President Obama fares so well under Silver’s model is because the model weighs state polls more heavily than national polls. Since Obama is ahead in battleground states like Ohio, Iowa, Michigan, New Hampshire, and Wisconsin and tied or slightly behind in others, he does exceptionally well in Silver’s model. Not surprisingly, this has generated a lot of outrage and accusations on the right, leading Silver to recently bet Joe Scarborough, former GOP congressman and current host of MSNBC’s Morning Joe, $1000 that Obama will win reelection next Tuesday (and by implication betting his reputation on the outcome of the election) after Scarborough criticized him (without mentioning Silver by name) on Morning Joe.
I am by no means a statistician, but like every one else that studies or writes on elections, I am obsessed with polls. However, I think that one thing that has gotten lost in these attacks on Nate Silver is that Silver never said that Romney had a 0% chance of winning the presidency. He is not saying that Obama is guaranteed to win next Tuesday. Why does this trouble me when it is probably a matter of semantics? Because even if Romney wins, it is not entirely clear to me why Silver’s reputation should be tarnished since his model does not predict a winner, just a probability. Silver gave Romney a little less than a 1 in 5 chance of winning; to listen to the Romney campaign and his surrogates, Romney’s chances far exceed 20%. In reality, I don’t think that anyone actually believes that this race is not tight (it is probably going to be a nail biter). But the interesting point in all of this is that Gallup has Romney ahead by 4-7 points nationally in recent polls but I have yet to hear anyone say Gallup’s reputation will be tarnished if Obama is reelected on Tuesday. Similarly, no one suggests that Rasmussen, also polling Romney favorably in many of the battlegrounds, would have to stop conducting polls if Romney does not pull out a win on Tuesday. In contrast, if Obama loses, there is a strong possibility that Silver’s reputation will not be able to recover because of all of the pushback that he is getting for the fact that his model gives Obama such a high probability of winning (rather than any actual flaws in his model, surprisingly enough).
In any event, I guess we will see who is right on Tuesday. Can’t wait.
Posted by Franita Tolson on November 2, 2012 at 12:14 AM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef017ee4a74b2e970d
Listed below are links to weblogs that reference In Defense of Nate Silver:
Comments
Anon: I was basing that statement on what I remembered. I fully accept that this isn't the best way to operate, but it also wasn't crucial to what I was trying to say. Happy to concede the point.
And as a side note, there's an interesting argument in Wired today that discusses something very similar--from a volcanologist's perspective.
Posted by: Eric | Nov 2, 2012 6:28:31 PM
It also begs to be mentioned that Silver performed quite unspectacularly in his 2010 predictions. I still do not understand why he has gotten so much fanfare.
Posted by: anon | Nov 2, 2012 1:44:58 PM
You may be misunderstanding the assertion. It is that among 100 amateur statistician bloggers using public polling data, you would expect about 20 to do as well as Silver did in 2008. It was competent but not spectacular work, and others did better but got less attention. (I can only speculate about the reasons for that.) Moreover, it has been reported that Silver had access to non-public polling data in 2008.
Posted by: statprof | Nov 2, 2012 1:28:28 PM
Statprof -- please explain your 20% chance claim -- it's not obviously plausible. And, in any case, isn't that a pretty strong performance -- doing what one had only a 1 in 5 chance of doing based on random assumptions? It's not like hitting the lottery, but to use some of the football analogies that have been thrown around on this subject this morning, it's statistically equivalent to coming back from a 7-point deficit with 3 minutes to go in the game.
Posted by: anon | Nov 2, 2012 12:36:24 PM
Nate Silver is doing a good, pragmatic, workmanlike job at simulating outcomes of the election based on polling data. He then reports the results of the simulations. It is not a difficult task. At best, he improves the results using some minor intuitive judgment that goes into his "proprietary model." At worst, he makes his predictions worse with the same intuitive judgments. Nobody, including Silver, knows which. He was among the better forecasters of the 2008 election, but he was not the best. His results in 2008 can be explained as roughly the result of a 20% chance, which is to say that one in five such bloggers could probably have been expected to do as well as he did, with the data he used, if they relied on randomness rather than claimed they were doing something thoughtful. His primary skill, like most popular bloggers, was gaining readership.
All told, I am baffled by the minute interest in day-to-day forecasting of the race. Unless you work for the campaigns or otherwise make decisions that depend on the perceived outcome of the race today as opposed to tomorrow or Tuesday, why do you spend so much time consuming this forecasting? Is the popular sentiment about the outcome of the 2008 election five days before it of any use or interest today? If not, why do you care so much about today's forecast of the 2012 election?
People like Silver expertly give you what you want, but the cycle is not helpful for any rational goal, and the "horse-race journalism" to which he contributes is lamentable. It seems fine to defend Silver from ideological, uninformed criticism, but be careful with hyperbolic, uninformed praise of his underlying work.
Posted by: statprof | Nov 2, 2012 12:15:21 PM
I just made the same point that Joseph did in longer form and with a link to a discussion of this on Deadspin:
http://prawfsblawg.blogs.com/prawfsblawg/2012/11/more-on-defending-nate-silver.html
Posted by: Howard Wasserman | Nov 2, 2012 11:22:07 AM
I agree with a lot of the above, but I will add that in many cases, the mainstream media and "pundits" generally are resistent to Silver's conclusions (even as properly understood) for self-interested reasons.
First, media outlets want to sell an "OMG it's SO CLOSE" narrative, because that's more likely to sell papers / get viewers to hang on the latest twist and turn. Saying, "well, anything's possible, but Obama is a 4-1 favorite at this point" doesn't help with that.
Meanwhile, pundits are rightly fearful that Silver may have come up with a prediction model that will make their more "from the gut" / "here's what I've learned from talking to a cabby or guy at an Appleby's buffet" sort of analysis irrelevant. It's much like the scorn the "moneyball" guys in baseball got from many of the baseball traditionalists; if someone has figured out a better way to use numbers and statistics to predict the future than the current crop of scouts/pundits, the scouts/pundits aren't going to like that.
Posted by: Joseph Slater | Nov 2, 2012 10:46:11 AM
Howard, Just trying -- perhaps a little too hard -- to be (or appear) appear non-partisan. There was a lot of liberal anger after 2000 (of which I was a part -- even making plans to go to Florida had SCOTUS not shut down the recount), but it was certainly of a different sort than we may see (but hopefully will not) if Obama wins the EC but loses the popular vote.
Posted by: Brian Clarke | Nov 2, 2012 10:22:06 AM
Franita, thanks for this post. I’m also troubled by the personal attacks on Silver— they seem to spring from a relatively thoughtless anti-academic/intellectual bias that has flowered in certain circles and about which we all should be concerned. I think this kind of rigorous analysis of polls and other election-related phenomena is fascinating and significant. It’s worth noting that Silver consistently has been among the more conservative of the serious people doing this kind of work. For example, Sam Wang at Princeton (http://election.princeton.edu/) currently projects the President’s odds of reelection at between 96% and 99.1%; Drew Linzer at Emory (http://votamatic.org/) has steadily projected more than 330 EVs for the President since September; and Jay DeSart at Utah Valley State (http://research.uvu.edu/DeSart/forecasting/october.html) has had the President’s reelection odds more than 84% for months (currently 87.05%). Beyond providing the kind of rationally defensible predictions that are the goal of modeling in a host of social sciences, Silver and other aggregators also have provided a valuable service by educating the poll-obsessed non-statisticians among us about how to better read and think about polls.
The point is that Silver, making very cautious assumptions, identifies the low end of the probability range – hardly a radical position. That he’s more visible doesn’t excuse knee-jerk trashing of the guy because he disagrees with others' preferred electoral outcome. It’s true that Silver’s and other models have consistently thrown cold water on the “Romney momentum” trope that everyone has seemed keen to foster since the first debate. But providing some evidence about reality in a campaign season that has been filled with confusion and misapprehension is objectively praiseworthy and a welcome antidote for those of us who have been infuriated by major media’s stubborn insistence on selectively covering certain polls to make the election seem as riveting as possible. (I'm hoping for a slightly less sinister explanation than Brian's, I guess.) That impulse alone is basis enough for a good reputation.
Posted by: Garrick | Nov 2, 2012 10:13:37 AM
Eric is right that the problem is that most people don't understand what Silver is doing--he is calculating odds, not predicting the score.
Michael: As I said in my comment above, the difference between criticizing a person and criticizing an organization, and challenging the reputation of a person as opposed to an organization, is crucial. Also, Intrade is not the same as Silver. It, as an organization, has no "reputation," beyond providing a forum for people willing to bet money to show what they think. It is individuals doing the work there.
Brian: You're right that this is what is going on for conservative and Republican critics. But the comparison doesn't hold. Most liberals (at least the ones who can be taken seriously) weren't upset that Bush lost the popular vote in 2000, other than in the course of questioning the wisdom of the Electoral College. They were upset over what happened in Florida and in SCOTUS. As someone (I forget who) wrote a few days ago, had Bush won Florida by 5000 votes--which means Bush v. Gore never happened, but Gore still won the national popular vote--no one would have questioned whether Bush "won", again, other than thinking the EC is stupid.
Posted by: Howard Wasserman | Nov 2, 2012 10:11:28 AM
Criticsm from the right as to Silver's "prediction" also ignores the nature of our electoral model -- likely intentionally. We all know that the winner of the popular vote and the winner of the electoral college need not be the same person. Given the electoral college math and the polling in the critical swing states -- plus Silver's analysis -- an 80% likelihood that Obama gets to 270 electoral votes on Tuesday seems about correct. It seems likely that conservatives are cynically setting the narrative for an attack on the legitimacy of an Obama second term along the lines of "He lost the popular vote and the liberals in California and New England have usurped the power of the rest of the country," thus a GOP controlled house and GOP senators are right to continue to block anything President Obama wants to get done. [Liberals certainly took a similar path in 2000, so this is not limited to conversatives, they just happen to be the ones doing it at the moment.]
Posted by: Brian Clarke | Nov 2, 2012 9:53:55 AM
Anon - As Silver said in the past, it's easier to predict the election now, when people are actually voting. He had a post a while ago that he takes greater pride in prediction a month or two or three ago.
If it's true that RCP and Intrade are predicting the same thing, then why aren't they coming under the same fire?
Posted by: Michael Risch | Nov 2, 2012 9:11:31 AM
I really dig Silver's statistical analysis. I agree that he isn't predicting a rout, and he is suggesting that far more potential states of the world exist in which Romney loses. More than this, however, his data shows that Obama is gaining likely routes to victory and Romney is losing them. This suggests that Obama has momentum and Romney does not. I suspect it is this last point that is driving some of the criticism.
Posted by: Matthew Bruckner | Nov 2, 2012 8:46:15 AM
I really dig Silver's statistical analysis. I agree that he isn't predicting a rout, and he is suggesting that far more potential states of the world exist in which Romney loses. More than this, however, his data shows that Obama is gaining likely routes to victory and Romney is losing them. This suggests that Obama has momentum and Romney does not. I suspect it is this last point that is driving some of the criticism.
Posted by: Matthew Bruckner | Nov 2, 2012 8:46:14 AM
Eric, curious what you mean "pretty much smoked all the odds makers in 2008." He predicted 49/50 states correctly. RealClearPolitics, which just aggregated polling, got 48/50 states correct. Intrade had Obama at well over an 80% favorite at this point four years ago. Silver's analysis was entirely consistent with the oddsmakers.
Posted by: anon | Nov 2, 2012 8:39:00 AM
Speaking of rout, from Eric's comment, there is a percentage likelihood of a rout in the model, and it is very low. I think people are also not understanding the methodology. He's not just calculating odds - he's running thousands and thousands of simulations to see what a distribution would look like if voters randomly fit the margins of error. This is different than just averaging polls, and a way more accurate way to look at possible outcomes. But, as others have noted, some of those outcomes have Romney winning.
Posted by: Michael Risch | Nov 2, 2012 8:15:59 AM
A recent post by Silver compares Obama's chances with a football team's chances when it's ahead by 3 with 3 minutes to go, and goes on to state that as it turns out, football teams in that situation win about that often. Anyone would say that a game of that sort was tight, and no one would criticize an odds-maker for giving strong odds to that team in such a situation. I think part of what is going on is that his critics are (either innocently or deliberately) misunderstanding exactly what his model is actually saying; 80% looks like he's predicting a rout, but that's not really what's going on.
As to why his reputation might be tarnished, he pretty much smoked all the odds makers in 2008, so now he has an unenviable reputation to uphold. I, for one, would not bet against him...
Posted by: Eric | Nov 2, 2012 3:15:22 AM
As to your penultimate paragraph: Some of that difference is individual--the reputation of a single individual (Silver) is much more susceptible to attack and destruction than that of an organization (Gallup). Some of it reflects partisan differences. On the other hand, if Gallup stays where it's at and Obama wins, rest assured that four years from now many on the left will argue that Gallup's polls should be disregarded.
Posted by: Howard Wasserman | Nov 2, 2012 12:37:41 AM
The comments to this entry are closed.