« Duke Lacrosse II: First Amendment Implications? | Main | Confounding the Founding: Why Separated Powers? »
Wednesday, February 27, 2008
On "Deadwood"
The recent blogosphere discussion of the Green Bag's forthcoming so-called Deadwood Report, which by and large has been favorable, is no substitute for reading the editors' introductory report itself. Strip away your preconceptions -- including the assumptions that "more" "information" is always "better," that anything must be better than US News, and so on -- and read the report. Consider its claims. Consider its proposed methodology. Make up your own minds about it.
My own view, to be perfectly blunt about it, is almost entirely unfavorable. The Green Bag editors had the kernel of a good idea. It could have been presented with a due sense of its limitations; with a reasonable humility of spirit; with an appropriate warning that "objective" measures are only as good as what they purport to measure, and that the choice of what to measure is not itself necessarily objective. It could have reminded us of the essential fact that not everything worth measuring is objectively measurable, and not everything objectively measurable is worth measuring. The editors could have acknowledged that what they dismissively call the "tricky" questions -- "whether [a law school's] teaching is effective, whether the scholarship is sound, whether the service is in the public interest, and the like" -- may in fact be the most important ones. Instead, their idea has been tarted up and sent strutting onto the public stage in an editorial voice that reads like an unfortunate combination of P.T. Barnum, Sammy Glick, Franz Joseph Gall, Nancy Mitford, and Eddie Haskell.
The editors and their defenders set out to disarm or co-opt their expected critics by acknowledging that there may be some "grumbling" about methodology, but saying that "we are trying to do some good here." They let themselves off too lightly. It is not just that the methodology is seriously flawed -- which it is. It is that there are a variety of serious mismatches between what the report says it is doing and what it is actually doing.
I certainly do not want to align myself with the forces of darkness. I certainly agree that it is a fine thing for law schools to supply more, and more accurate, information about their doings. I acknowledge that the Green Bag editors are doing something toward this end. But one should keep in mind that rankings of this sort are supposed to aid you in making meaningful qualitative judgments about the "quality" of law schools and their faculty -- not to substitute for independent judgments of this kind. I fear that this is exactly what items like the Deadwood Report, wrapped up as they are in a pseudo-scientific appeal to "objectivity," will do.
I certainly don't suggest that law schools keep their heads in the sand. But I encourage everyone to read the editors' introduction for themselves, and to read the subsequent "Deadwood Report" with its obvious and serious limitations in mind. Keep in mind that, given its flawed methodology and questionable underlying assumptions, it is not a serious measure of "the relative quality of law schools," as it purports to be.
Keep in mind that even the very word "deadwood" is inaccurate. What the report will actually measure -- as the editors fairly clearly admit in the subsequent story in Inside Higher Ed -- is the accuracy of online self-promotion efforts by law faculties. That's a narrow but great thing to measure, and we could have been grateful to the Green Bag editors for doing just that. But that's not what the editors claim to be offering in their report, and Lord knows it's not how the thing will be used. How unfortunate that a report that offers itself up as an attack on law school puffery and deceptive marketing tactics offers up so much more of what it purports to condemn.
Posted by Paul Horwitz on February 27, 2008 at 11:50 AM in Life of Law Schools | Permalink
TrackBack
TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef00e5509b0ca18834
Listed below are links to weblogs that reference On "Deadwood" :
Comments
Paul -- what specific aspects of the methodology do you disagree with? The choice to cull data from law school websites? I'd be interested in hearing more about the specific methodological choices you're objecting to.
Posted by: Matt Bodie | Feb 27, 2008 4:02:32 PM
I'd hoped not to spend too, too much time on this, which is why I hesitated to post at all, but this is an entirely fair request, Matt. Do you mind if I just summarize, though? My complaint is both with the methodology and with some of the underlying assumptions that go into it and claims that come out of it, so I'll mix them all together here.
* The claim that the actual report will provide meaningful assumption about the "relative quality of law schools." The assumption that statements about relative quality are meaningful in the absence of some more qualitative statement about what quality is and means in an absolute sense.
* The ready ascription of motives to the AALS, the ABA, and pretty well everyone else.
* The assumption that "crudely measuring whether faculty members do the work that the law schools say they do" is even a crude measure of "law school faculty quality." As I said in my post, I've got nothing against monitoring law schools for puffery in promotion; but one does not equal the other.
* The utterly, utterly uncritical assumption that productivity should be measured by asking whether each individual faculty member combines teaching, scholarship, and public service," apparently on the assurance of "the law schools and many leaders of the profession" -- the very figures who are otherwise scorned by the editors -- that it is meaningful and/or important that each person do all three.
* Similarly, the Law Day-ish assumption that "the best lawyers of every stripe engage in service for the public good."
* The assumption that conclusions about faculty quality can reasonably be drawn via a review of school web sites.
* The still unknown "finer points of [ ] sorting and weighing" without whose transparency and rationality the whole project is rendered questionable.
* The assumption that a "moderately active teacher-writer" should be given more weight when it comes to making relative statements about faculty "quality" than, say, a mixed faculty of "hyper-teachers" and "hyper-writers."
* The statement that the editors will penalize the absence of accuracy and reward its presence -- a fine thing, to be sure, but not one that is relevant as such to "objective" judgments about faculty quality. As I said in my post, that's a measure of candor, not of "deadwood."
* The questionable relevance of how schools label "lecturers," "emeriti," and so on to the bottom-line question of faculty "quality." I grant you that this makes the editors' job easier, but it does not seem to me to go meaningfully to the underlying evaluation.
* The trite statement that a large-enrollment class is different from a highly specialized seminar, without actually saying which way the difference should cut and why.
* The apparent exclusion of book chapters, although I doubt they will be excluded in the final analysis.
* The exclusion of various other forms of scholarship, which may be justifiable but is not justified by the editors -- particularly in light of the fact that, except for the Caesar's-wife principle, the editors would apparently have counted publications in the Green Bag.
* The trite statement that "inbreeding is bad policy," which appears to mean that a publication by Professor Jones of the Podunk Law School in the Worse-Than-But-Different-Than-Podunk Law Review will count as less "deadwood" than a publication by Cass Sunstein in the Journal of Legal Studies, which is peer-reviewed but published at Chicago.
* The late-inning admission that the report is really measuring only the failure to perform as advertised, which is far different from any conventional understanding of the word "deadwood" -- but less sexy.
* The apparent assumption (unless it is nothing more than a stunt) that the proportion of professors attending graduation ceremonies says anything meaningful about how seriously professors take their teaching responsibilities or their students' accomplishments, let alone that this in turn says anything about relative faculty "quality." Let me add that I've attended all the graduations of schools where I've been a full-time faculty member. I've been happy to do it, and proud of my students -- although it's ironic that an editorial that bemoans the Lake Wobegon effect now treats simply graduating from law school as an "accomplishment." It's just that it doesn't measure anything, and certainly nothing that the report purports to measure, and certainly nothing that is both objectively measurable and meaningful.
* Finally, the use of the phrase "back 'em up" -- a verbal swagger that recalls nothing more than it does George Bush in a flight-suit.
I'm sure that's some of my list but not all of it, and I've focused on what I think is present but flawed rather than what I think is absent.
Posted by: Paul Horwitz | Feb 27, 2008 4:31:32 PM
Paul, I guess I'm a bit confused by the dissent here. I don't read the intent behind the Deadwood Report to claim to supplant all other rankings or to claim to be the only useful source of information; so if someone is going to make an informed choice about where to go to law school, what's so bad about having another source of information to help guide the perplexed?
Besides, there *is* a lot of puffery. This will encourage schools to keep websites clear and comprehensive, and schools will have an opportunity to address potential mistakes before they appear in final form. I don't mean to force you to justify each of the quibbles or real arguments you raise; but put aside a bit of the audacity in rhetoric in RD's ssrn piece, and looking at the goal as a whole, it seems like the project is worthwhile, which is not to say without flaws.
As you recognize in your post, there is a kernel of a good idea here. I don't think that the idea and the future effort to execute the idea are completely lamentable solely in light of the failure to make all the appropriate caveats and qualifications about the worth of the enterprise. But as always, you are right to warn us about over-reliance on it.
Posted by: Dan Markel | Feb 27, 2008 8:11:16 PM
Hi, folks! I agree that the Deadwood Report isn't going to cure all rankings ills, although if it keeps law schools from engaging in massive puffery, that's a good in itself. Frankly, law students already have a lot of factual info at their fingertips--that's what the joint ABA/LSAC publication of law school information is designed for--and yet they still seem to crave some sort of sorting function. I'm just opposed to any ranking system that pretends that there are real differences among tightly bunched groups of schools. It's like that great scene in THIS IS SPINAL TAP in which the mock rock star's amplifier _must_ be "one louder" because it "goes to eleven" and not just to "ten."
We have got to stop grading ourselves by ordinal rankings. It does very few of us much good and takes up way too much of our brainpower.
Posted by: Nancy Rapoport | Feb 28, 2008 1:44:29 PM
Thanks, Dan and Nancy (especially Nancy -- I love your own blogging and am graced by your presence here), for your comments. Nancy, I agree with all your points. I'd just reiterate that while preventing law school puffery is indeed a worthy end in itself, this isn't how the GB sold its project; I wish it had. I think that point applies also to Dan's second paragraph. As for your first paragraph, Dan, I kind of agree that the GB doesn't purport to supplant all other sources -- although the extent to which it sells itself as thoroughly objective, and pumps itself up by trashing the schools and a variety of other sources, certainly suggests that it is trying to assert some primacy here. Certainly nothing in the article tells folks who are looking for information that they ought to shop around, let alone that qualitative judgments are as or more important than, as Nancy puts it, ordinal rankings. As to what's so bad about having another source of information, I would agree with that notion in general terms, but in particular cases my concern is that more is not, in fact, always better. If the "information" is itself far less objective than it purports to be, if it is one more voice promising "objectivity" but not questioning the bases of that "objective" measure, if (see Nancy's comment) it encourages people to spend more time on "grading" than on real qualitative judgment -- let alone spending more time experimenting on innovations in teaching and scholarship at the institutional level -- then it raises genuine concerns, concerns that to my mind rise proportionately to the degree of puffery in the editors' own claims and the flaws or question marks in their methods.
Indeed, I admit that in this case I found it hard to put aside the audacious rhetoric in evaluating the GB article. And in my passion, I fear I used some audacious rhetoric of my own. The swagger in the editors' authorial voice suggests that they can take on all comers, so I am not especially concerned for their tender feelings. But in retrospect I believe I should have tempered my own language in criticizing their project -- not my conclusions, but my rhetoric. For that I apologize.
Posted by: Paul Horwitz | Mar 5, 2008 4:38:26 PM
The comments to this entry are closed.
