« Interesting essay in the Chronicle today | Main | Filibuster (III) »

Tuesday, April 26, 2005

The Uneasy Case for the US News Law School Rankings

Everybody loves to bash the US News rankings.  Especially Brian Leiter.  There is evidence that schools "game the system."  There are absurd results -- precipitous drops for University of Washington and University of Kansas.  There was even that dark time when the rankings placed NYU above Columbia -- sacrilege by any standards, and irrefutable proof of flawed methodology.  But even with all of its warts -- and they are many -- the US News list serves a valuable purpose.  It's cheap, accessible, and easily digestible, and it's right more often than not.  And frankly, it would be pretty ridiculous to expect much more from a $3.50 magazine.  With U.S. News, the reader gets exactly what she pays for. 

First, let's talk price.  The cover price of U.S. News and World Reports is $3.50 per issue.  Like many news magazines, it's often sold at a huge discount.  In fact, through the magazine's own website, it's as simple as a click to subscribe to it for about 50 cents an issue. 

And the reality is that 50 cents doesn't buy a whole lot.  Thus, U.S. News is not a scholarly journal.  Like the other news magazines, it specializes in taking complex national and global issues and boiling them down into 3-page stories with lots of glossy photos. 

In fact, you're not even really paying 50 cents.  After all, in its "Best Graduate Schools" issue, US News is also assessing business schools, medical schools, engineering schools.  Plus it's running its usual array of little stories about national events, health, the Middle East, etc. 

So in the end, you are paying about ten cents for US News's opinion of law schools.  And your ten cents won't buy you a double-blind, gold-plated, Brian-Leiter-stamp-of-approval study.  It just won't.  You pay your ten cents to US News, and you get the tabloid version.

Now, a corollary is that no sane person should base the decision about where to go to law school solely on US News.  That would be like reading their one-page blurb on some new cancer treatment, and immediately signing up for that treatment.  You just wouldn't do it.  You would talk to your doctor first, get some opinions, and so forth. 

On the other hand, tabloids have their uses.  Like I said earlier, they're easily digestible, and some of them are mostly right, and they serve as a good starting point.  (They're an awful lot like the Drudge Report, in fact).  And so, you put down your ten cents for the tabloid.  You read about the new cancer treatment, and maybe it piques your curiosity, and you follow up and ask your doctor about it.  And you proceed, not based on the tabloid, but based on what your doctor says.

US News law school rankings serve exactly that same purpose.  And it's useful, and necessary.  After all, it may be hard for legal academics to recall that knowledge of top law schools is not universal, and so to excoriate US News for its shallowness.  But the fact is that knowledge about law schools is often not accessible, and the ten-cent version is a great starting point.

And I can say this as someone who used US News as a starting point.

I worked hard in undergrad, took Honors classes, wrote a thesis.  I also didn't know the first thing about law schools.  I didn't really know many attorneys; I didn't have any family members who were attorneys; I'm the first from my family to attend law school.  And if you had asked me to pick out top law schools, I wouldn't have known where to start.  I probably would have picked a few Ivies, and Stanford; I'm certain I wouldn't have picked Michigan, NYU, Chicago, Virginia.  It just wasn't within my limited sphere of learning.

And there were a number of rankings available.  I quickly found some -- the Princeton Review is widely available, and so is US News, and so are some others.  The differences were immediately visible.  I don't recall all of the details, but as I recall, Princeton Review ranked Yale #1 and Virginia #2.  (That outfit hasn't gotten any better -- check out the latest rankings, which lists the schools with the "best faculty" as Washington & Lee, BU, and Kentucky.  Inconceivable.)  Many others were equally problematic.

And then there was US News.  As I did my research, US News looked more and more like a good assessment.  I concluded that Princeton was wrong to list Virginia as #2.  I concluded that, by and large, the US News list looked pretty accurate.  I used it as a starting point, and I drew up a list of schools that I wanted to investigate further.  I investigate these schools further, of course.  I read other material, and I made some decisions, and I sent out my applications.  And I think I did pretty well.

Since then, as I've become more immersed in legal academia, I've had moments where I've chuckled at a particular US News ranking.  But is it generally accurate? My impression as an applicant was that it was very helpful.  And I don't think I've changed my views all that much.

Compare the US News list to Leiter's list.  Leiter's top 5 (6, actually) are all accounted for within the first 6 spots of the US News list.  In the top 15 (16, actually, because of ties), there is a single school on Leiter's list not on the US News list, USC.  Skip down to the top 18, and again every school listed in the Leiter 18 is accounted for in the US News 18.

There are differences, to be sure.  US News lists Texas as 15th, while Leiter ranks it 8th; US News ranks Northwestern 10th while Leiter ranks it 14th.  But on a whole, the US News list seems to have gotten most of its data right.  It's not without individual anomalies -- see, e.g., Washington and Kansas.  It's also not without manipulability.  (But any ranking system is subject to manipulation; indeed, schools interested in improving their score in Leiter's own survey could try to manipulate the data by mass sending out lists of faculty publications to other leading law faculty who may be considered likely candidates for inclusion in Leiter's data pool).  But it's clear that the US News list is quite similar to Leiter's.  And the other tabloids are not, not even close.

As I've suggested earlier, I think that a 90% or 80% hit rate, is more than acceptable for a general-interest newsmag like US News.  Cheap access to a mostly-right chart is a good thing; US News is more of a positive force than not.  And yes, the list has flaws.  But let's face it -- if a law student is using US News as a sole data point for decisionmaking, then she has bigger problems than US News' methodology.  She is being spectacularly negligent in doing her homework.

Therefore, assume that every competent law student does more than simply say "Northwestern is #10 US News?  That decides it!"  Assume that all competent law students are doing their homework; some of them using US News as a starting point -- and hey, it's a hell of a lot better than some other starting points available  (e.g., Princeton Review) -- but all ultimately doing serious reading and research of schools.

Now that we remove the red herring of US News sending poor unwitting law students to their doom, we see that this is a non-issue.  The only person who takes US News seriously as a final arbiter is your aunt Hilda, who gets her magazine and then proudly tells her friends that her nephew is going to Columbia, and "they're the #5 law school in the country!"  (The dean at your law school knows this too, and she knows that sometimes Aunt Hilda pays the tuition bill, which is why she sometimes remarks about rankings.  But that's all her remarks mean -- they aren't directed at practitioners or academics, but at Aunt Hildas).

Could the US News rankings be improved?  Of course they could.  But even as is, are they providing a useful benefit to the general public?  Yes, they are. 

And certainly worth every penny of the ten cents you'll pay for them.

Posted by Kaimi Wenger on April 26, 2005 at 11:23 AM in Kaimi Wenger, Life of Law Schools | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef00d83425e60753ef

Listed below are links to weblogs that reference The Uneasy Case for the US News Law School Rankings:

» Law Review Citations and Law School Rankings from Concurring Opinions
There's no shortage of writing on law reviews or law school rankings, to say the least. So why not combine the two? Questions about law review ranking abound. How does one compare offers from journals at relatively equal schools? Is... [Read More]

Tracked on Dec 7, 2005 11:51:16 AM

» Using the web to measure university reputation from University Metrics
... an entirely new university ranking methodology based on the number of links between university websites measured by Google Search ... [Read More]

Tracked on Apr 7, 2006 6:33:27 PM

Comments

I e-mailed this post to Brian Leiter, and he was kind enough to reply and point out the obvious -- his own rankings are free.

Yet there are some important differences.

For one, Leiter does not rank schools past 45. Probably because the time and resources needed to do this are too great, also likely because the differences between these schools are less pronounced. Still, for the student deciding between, say, Pepperdine and Cal Western, Leiter's own rankings are unhelpful as a guide. Meanwhile, at US News, your ten cents gets you more than just data on Harvard and Yale, it also gets you data on Pepperdine and Cal Western.

Second, US News has filled an information vaccuum, in part by leveraging its institutional credibility as a news source. It has become the dominant player, and it's a pretty good one.

Would others -- such as Leiter -- be better? Probably. However, if US News fell off the face of the Earth tomorrow, there's no gurantee that the market would accept Leiter as the heir apparent. And here's the real benefit of US News -- as the 800-pound gorilla, it crowds out not just Leiter's rankings, but also the horde of seriously flawed rankings.

Don't believe me? Run a google search on "law school rankings." Directly behind US News is some outfit called the "Brennan Rankings." And the Brennan rankings list the top-10 schools as:

1. University of California Berkeley
2. Columbia University
3. University of Minnesota
4. Harvard University
5. New York University
6. University of Iowa
7. University of Texas
8. Ohio State University
9. University of Michigan
10. Southern University

I'm not kidding, alas.

US News imposes a pax romana. Their market domination marginalizes well-intentioned, serious scholars like Leiter. But their market domination also keeps out the hordes of much more serious charlatans.

If US News fell off the face of the earth tomorrow, the number one result for a search for "law school rankings" would be the Brennan list. Would that really be better than the status quo?

Posted by: Kaimi | Apr 26, 2005 1:00:44 PM

I think the true value of US News rankings is, as someone else wrote (maybe s/he even presented at that conference in Indiana or Iowa on the next generation of law school rankings), that the rankings serve as a device that forces the release of more information; the rankings induce the spread of much more information about schools than we would get otherwise. The increased transparency facilitates informed decision-making. Maybe it was Mitu (Gulati) who wrote this.

Posted by: Dan | Apr 26, 2005 1:37:15 PM

I agree with most everything positive that has been posted about US News rankings. However, I think the genuinely troubling (mis)use of US News data is not on the students' end (who cares what criteria uninformed applicants utilize... the nation has no deficiency of lawyers). The troubling scenario is if law schools themselves alter their admissions policies, public relations strategies, and worst of all, their pedagogical focus to accommodate the rankings monopoly that is US News. If (and of course we have only anecdotal evidence) schools are admitting students with an eye to affecting their rankings rather than assembling the best possible class, that's a troubling thing for anyone who still believes idealistically (naively) that the leaders in legal education should be relatively insulated from such politics.

Posted by: Kevin | Apr 27, 2005 10:16:50 AM

Hey, if all you want from the rankings is to be "cheap, accessible, and easily digestible," we all can produce those by hundreds -- randomly! But if you want something that's actually accurate, and also something that doesn't create perverse incentives for schools to change the JD/LLM ratios or to give their recent grads phony "employment" via $7/hour RA-ships to achieve high post-graduation employment rate, then, you'll have to engage in a complicated and time-consuming analysis that Leiter does.

Posted by: Kate Litvak | Apr 27, 2005 11:17:33 AM

I have serious trouble taking seriously the criticism of a complex system by someone who is unaware "any" is singular. It is not "any standards," but "any standard." Additionally, I am not persuaded by the rampant hyperbole, oversimplification, and otherwise highly controversial statements made in this piece which lack basis in fact. Who are you to say the dark days were when NYU was ranked above Columbia or that Chicago does not belong on the list? It certainly sounds like Ivy-League elitism dominates the thought behind this posting. In my view, this posting is utterly lacking in credibility. Some indicators of top performing lawschools are rates of preferred job placement across all grade strata, legal impact, faculty recruitment and accessibility, quality of life, and the strength of the alumni network. Much of this is perception-based. As such, how can you deny the validity of a largely perception-based survey and consider your views somehow superior to those of the aggregated views of hundreds of judges, professors, recruiters, and attorneys who evaluate students from these law schools on a regular basis? Ivy-League elitism, perhaps.

As a note, the top law school law reviews in the country in 2005, according to Washington & Lee's Rankings are: Yale L.J., Columbia L. Rev., Supreme Court L. Rev., NYU L. Rev., Cornell L. Rev./Stanford L. Rev., Va. L. Rev., Harvard L. Rev., Cal. L. Rev. The Ivy Leagues did not score so well on this ranking. It's time to deal in facts, not in opinions.

Posted by: Erik | Aug 1, 2006 10:09:37 AM

As for me, I have serious trouble taking seriously a critic who doesn't understand the basic details of a short blog post -- basic details such as the fact that I was myself criticizing Leiter's earlier criticism of the U.S. News rankings.

P.S. The very first example sentence under the dictionary definition of "any" (American Heritage College Dictionary, 4th ed.) is this: "Are there any messages for me?"

Posted by: Kaimi | Aug 1, 2006 8:30:46 PM

Post a comment