Monday, April 10, 2006
The Unbearable Lightness of Being a U.S. News Voter
The U.S. News rankings are out, and for the first time, I was complicit in the process. That’s because I controlled not just one but two ballots ranking all schools on their academic reputation. Every school gets four ballots, and among these, one goes to the most recently tenured faculty member and one to the chair of the school’s faculty appointments committee. I was both those things for Toledo last year (and I probably will be next year also). Thus, I got twice the “law porn,” because many schools apparently have separate lists of who occupies these slots, and they don’t cross-reference.
I understand that. What I don’t understand is how anybody could even try to do this ranking job even semi-competently. Are voters really expected to read and compare the glossy brochures we get from well over 100 schools? Go to websites and check out the scholarship of a significant percentage of the faculties of a significant number of schools? Survey multiple colleagues and friends at other schools as to how they think the University of X compares to Y College of Law in their areas? Making even a serious attempt at any of this would be a part-time job in itself.
Of course schools ranked in the top 20 have better reputations than schools in the middle of the pack, which have better reputations than schools at the bottom of the rankings. But the important part is not confirming what we’re all already supposed to know, it’s distinguishing among schools that are reasonably close in the rankings.
What I suspect actually happens is this:
People get positive or negative feelings about individuals in their own field. They know a few folks outside their field from conferences, the odd article people read in some other field, the occasional guest speaker, and maybe even those they encounter on blogs. But that is in no way representative. People also give higher votes, at the margins, to higher ranked or more famous universities (independent of the quality of the law school); places where they have friends, where they or their friends went to school; favored parts of the country, etc.
Also, I suspect there is probably a lot of strategic voting: at the margins: give lower votes to schools ranked right around where your school is ranked.
None of these methods would yield results that are, in any overall sense, fair or accurate. But I can’t think of a way to do this job well. Suggestions?
TrackBack URL for this entry:
Listed below are links to weblogs that reference The Unbearable Lightness of Being a U.S. News Voter:
Want a suggestion? Sell your vote(s). Instead of having schools waste their money sending you "law porn" that you toss in the trash can, just sell your two votes -- say, $50 for a "3", $100 for a "4", and $200 for a "5." Schools "buy" the US News rankings inputs in all kinds of ways -- by "buying" higher LSAT students with scholarships, by "buying" "academic reputation" through hiring senior scholars well past their teaching prime who may add a few lines to the faculty publications list but add little to student experience, and by "buying" more library volumes or teacher-student ratios. Since it's going on anyway, just make it more direct. Also, why should US News be the only one to profit off of its ranking system? You need to eat too.
Posted by: Geoffrey Rapp | Apr 10, 2006 10:35:57 AM
Geoffrey, that would be a wonderful system. Sadly, it wouldn't work because a school couldn't buy Slater off with any assurance that he'll fulfill his end of the bargain; Slater should solicit conflicting bribes from many schools and then vote in Toledo's self-interest.
That's the problem with a wholly self-interest-motivated system: information costs prevent us from becoming the libertarian nirvana we'd in which everything would be rational and efficient (and in which the US News rankings would be rendered worthless, unless US News chose to invest its rankings-issue profits by bribing Slater...).
Posted by: Scott Moss | Apr 10, 2006 11:54:44 AM
I'm wondering if a good alternative measure - that could be considered along with the reputational score - would be a citations/faculty or articles/faculty score so as to address faculty productivity. I don't think that this is already included in the rankings, but please feel free to correct me if I'm wrong. Of course, it could be argued that this is implicitely contained within the reputation score, but I think that you might find some interesting variations between the scores. This might also address some of the problems with rankings inertia.
Department of Political Science
University of Georgia
Posted by: Jeff Yates | Apr 10, 2006 1:50:16 PM
Jeff--I like your effort to suggest objectivity here, but those "impact rankings" can be abused, too. There's a great article in the Chron of Higher Education on this trend in the hard sciences: see
Posted by: Frank | Apr 10, 2006 6:09:06 PM
... and if "impact factor" becomes the metrick, Geoffrey will propose that for $5, he'll cite your article in his upcoming piece...
The problem is that any statistic, if it really starts to matter, can and will be "gamed." This is why there is no such thing as a rankings statistic that's good in the long-term. The best solution to gaming is to jumble the criteria, say every two years: now US News uses those dumb surveys; then they can use SSRN download stats to replace the surveys; then impact.... If US NEws keeps movign the target, they'll get quite hard to game.
Posted by: Scott Moss | Apr 10, 2006 7:08:49 PM
Certainly any ranking scheme can be gamed. Even the articles/faculty can be gamed, although it might need to be a top 20 articles/faculty. Of course, the sluggishness might continue as law review editors at top 20 begin to see merit in publishing their own faculty. (Perhaps a bit too Oliver Stone conspiracy theory on this last point). I like Scott's idea of jumbling criteria and providing a moving target. At present, the reputation scheme seems endogenous - prestige school X is highly ranked by peers, because, well... it's a prestigious school.
Jeff Yates - J.D., Ph.D.
Department of Political Science
University of Georgia
Posted by: Jeff Yates | Apr 11, 2006 9:11:09 AM