« A Modest Proposal Regarding Blind Grading | Main | That Feeling, Revisited »

Monday, April 20, 2009

U.S. News and "assessments"

I know we're not supposed to take notice of the U.S. News rankings (ed.:  so don't!), but . . . I continue to be struck by the facts that (a) very few of the "top 50" schools (only UCLA and USC, it appears) have "peer assessment" scores that are *higher* than their lawyers / judges score, (b) the schools with the largest gap between peer assessments and lawyer-judge assessments seem to be ones that are often said, with varying degrees of accuracy, to be "conservative" (e.g., George Mason, Brigham Young, Notre Dame), and (c), with very few exceptions, the peer-assessment number is so sticky.  (I realize I am treading very close to "sour grapes" ground, but the steadiness of my own school's number, assuming it is supposed to reflect a judgment about the faculty and their scholarly productivity, is unwarranted by the facts.) (ed.:  so quit whining, and get back to writing!) 

Posted by Rick Garnett on April 20, 2009 at 10:16 AM in Life of Law Schools | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef0115702ff8e7970b

Listed below are links to weblogs that reference U.S. News and "assessments":

» More: US News Specialty Rankings And Assorted Blog Commentary from The Faculty Lounge
The US News rankings of law school specialties appears to be here. It might fairly be entitled "Trading Places" because the number 1 and 2 ranked schools have switched places in the environmental, IP, international, and legal writing categories. FWIW.... [Read More]

Tracked on Apr 20, 2009 3:48:48 PM

Comments

Rick, I have the highest regard for you and your law school, but if you're suggesting that there's a political bias here, isn't the evidence ambiguous about which way the political bias points? It could be that (as I think you're suggesting) academics are biased against conservative schools, or it could be that lawyers and judges are biased in favor of them. The evidence gives us no basis for choosing between those two explanations, and there seems to me at least as much reason to believe the latter as the former. Or am I missing something?

Posted by: Sam Bagenstos | Apr 21, 2009 3:03:47 AM

anyone who's actually been exposed to legal academia knows that there's a definite leftward bias among law professors, for pete's sake.

Posted by: bill | Apr 21, 2009 9:50:14 AM

Sam -- an entirely fair point. If you look at my comment to Nadine's follow-up post, I refined my suggestion a bit, to say that there seems to be more of a "mark-down" effect for religiously-affiliated schools (by peers), whether or not they are plausibly characterized as "conservative". But, you are right -- it could be that judges and lawyers are overrating certain schools. I guess it seems more likely to me that academics would systematically underrate religiously affiliated schools than that lawyers and judges would over-rate them, though. Just a sense . . .

Posted by: Rick Garnett | Apr 21, 2009 5:30:11 PM

Hi Rick,
I wonder if there isn't a dynamic like this in effect with regard to religious law schools (other than, perhaps, Georgetown, which I think you'll agree is mostly nominally "religious", if that.) These school tend to restrict the faculty they hire to a smaller pool than do non-religious schools. Most of they don't _refuse_ to hire non-members (though some essentially do- BYU, for example), but many of them strongly favor either members of their own sect or else at least other self-professing Christians. Non-religious schools, other the other hand, make no such restrictions. Therefore, religious schools draw faculty from a smaller pool of applicants. Given this smaller pool, it's no wonder that they have, on average, somewhat less well regarded faculty than we'd otherwise expect for them given their other characteristics (student body, endowment, etc.) Sometimes, of course, a star is drawn by the nature of the school, but in general it seems like we'd except any institution that intentionally limits the pool it draws from to be weaker than one that draws from a bigger pool. This seems obvious to me. And, in the peer assessment score, those scoring are looking at faculty, so this is reflected without any need for bias on the part of those making the choice. But I think we get a somewhat different dynamic with students, where the self-selection process at least sometimes allows the _school_ to be more picky- the student population is much larger than the would-be professor population, but the choices of very good seriously catholic schools, or Mormon schools, are few. So, these schools can select the cream from these crops of students at a level hire than they otherwise would. This in turn explains the judge and practitioner scores- judges and practitioners are mostly focused not on the professors, but on the students. As many of these schools are able to be unusually selective of students, they have excellent graduates (good inputs making good outputs). These two dynamics together seem to me to pretty well explain the rankings of religious schools, and the disparity between the faculty and judge/practitioner rankings of them, without having to think there is any bias either way on either group of rankers. (Of course this doesn't rule out the rankings being unreasonably "sticky", but this seems to apply to religious and non-religious schools alike.)

Posted by: Matt | Apr 21, 2009 7:49:23 PM

Matt -- what you say is certainly reasonable. I see the faculty-hiring dynamic working differently, though. It's tough to speak about other schools, so I'll just talk about my own. It is obvious (to me, anyway) that Notre Dame gets better faculty than it otherwise could (given the givens) precisely because it is (or aspires to be) distinctively Catholic, in a broad way. And, *every* school (or almost every school) limits the "pool" from which it draws in any given year (politics, curricular needs, diversity concerns, etc.), so I'm not sure that a religiously affiliated law school -- which might, after all, be more open to a "best available athlete" approach -- should be assumed to be drawing from a consistently less-rich pool.

But, putting this sense of mine aside, I would hope that those filling out the US News surveys would not rely on an on-the-fly-and-perhaps-inaccurate sense that the schools are hiring from a smaller pool, but would instead look at the faculty and their records of publication. I am, again, painfully aware that the following observation can only sound like special pleading sour grapes, but . . . Notre Dame's faculty (and, I think, that of some other religiously affiliated schools) publishes in a way and at a level that is not commensurate with the (relatively low) peer-assessment score the school gets in US News. This number has been sticky for two decades now and, I fear, reflects an attachment to congealed and unreflective mis-judgments -- mis-judgments that, I suspect (without knowing for sure), reflected, at least in some cases, mistaken assumptions about religiously-affiliated law schools -- rather than to even a quick assessment of what's actually going on and getting written.

Posted by: Rick Garnett | Apr 21, 2009 10:51:15 PM

I'd not be surprised if unreflective misjudgment played an important role in the rankings. (I'd be a bit surprised if it didn't!) But I do think that, on average, any way in which a school restricts the applicant pool that it considered for hiring will tend to lead to a weaker faculty over-all, even if it also sometimes draws applicants that otherwise wouldn't be interested. How big an impact this is on any given school will be hard to know (it might even be a plus for Notre Dame, but I think that would be unusual, and even there, a small plus), but on average it would, I think, have to work against the average faculty quality for the school, no matter what the reason for limiting the pool of applicants considered.

Posted by: Matt | Apr 21, 2009 11:18:33 PM

Post a comment