« 9/11/06 | Main | Some thoughts on letters of recommendation »

Monday, September 11, 2006

Law Reviews v765

We all know that law reviews lack any rigorous version of peer review.  And most of us think that is notable, if not altogether corrupt.  Even those who do not have PhDs or experience with academic publishing in non-law disciplines sense that something is awry.  Few tend to think it is a good thing that second-year students are the gate-keepers to our most important research vehicles -- and I doubt very many think it makes no difference to what gets published in the "top" journals.  But David Luban of Balkinization makes what I think is an astonishing claim:

The[re] are . . .  people who have always bridled at the idea that law students rather than peer reviewers should decide what counts as meritorious scholarship. I have never taken this problem seriously. Comparing the quality of articles in the top student-edited law reviews with the quality of articles in the top peer-reviewed philosophy journals (my own scholarly point of reference), I have never been able to detect superiority in the peer-reviewed philosophy journals. By and large, I think that law review editors – at least at the top law reviews, where the editors have an embarrassment of riches to choose from – have been pretty good gatekeepers.

Really?  Is the state of philosophy so bad?  My reference point is political science and I think it is obvious that the scholarship in the best journals is -- on average -- much more important and ground-breaking than the stuff in our best law reviews.  Do others have reference points they wish to share?   

Posted by Ethan Leib on September 11, 2006 at 09:46 AM in Life of Law Schools | Permalink

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef00d834b0ae9153ef

Listed below are links to weblogs that reference Law Reviews v765:

Comments

Jae, I think we're in agreement: the things you list on the right side of the versus are what I had in mind when I referred to "not moving the ball at all."

Posted by: Bruce Boyden | Sep 11, 2006 10:07:59 PM

I think Bruce misconstrues Ethan's comments. Ethan is not making a "big idea versus big idea applied to a small context" distinction. The operative distinctions in Ethan's comment are: scholarship versus sophistry, insight and depth versus empty charisma, substance versus packaging, originality versus a simple ignorance of/disregard for/misconstrual of what others have said on the topic, etc.

Posted by: Jae Lee | Sep 11, 2006 8:32:43 PM

Bruce, very nice connection to the Ian Shapiro critique of a lot of poli sci (re "Perestroika"). That's exactly what we have to watch out for if we fixate too quickly on "rigor"....these concerns can lead to scholasticism (in the worst sense of the word), where work is treated as an "advance" simply because it elaborates some extant architectonic (and that assessment of quality has nothing to do with the ultimate validity of the architectonic).

Posted by: Frank | Sep 11, 2006 7:12:48 PM

"Slickness rises to the top. Writing as if one is the first to write on a subject or writing as if your ideas are radically new rises to the top." Well, interesting. My impression from the two non-law fields I have any familiarity with is that we are trading sins here. Legal scholarship can perhaps get more fanciful than that in other fields, e.g. by practicing slap-dash economics, because all authors need to do (at least initially) is impress the students. Perhaps that's bad, but I think it can sometimes lead to avenues of inquiry that are fruitful.

In other disciplines I think what you tend to get is incremental moving of the ball. So in history, a lot of the articles coming out in the AHR or JAH in any given issue apply some general theory to some particular town or county, or one group in a given city in a short time-span, or even sometimes just one person. In (analytic) philosophy, articles tend to invent and play with a new hypothetical or 2 or 3 on an old chestnut problem. Obviously such a system has its benefits; incremental ball-moving is better than not moving the ball at all. But I suspect it has its disadvantages too. (E.g., the underlying causes of the "Perestroika" movement in political science.)

Posted by: Bruce Boyden | Sep 11, 2006 6:30:18 PM


Orly: Your point seems to be about who does a better job with EDITING, not with SELECTION. I thought we were talking about the latter. In any case, though, it is interesting information: perhaps sometimes the articles in the "better" law reviews are better (if they are) on account of the hard work by the law students at "better" schools (and, by implication, not necessarily the authors themselves). This is, of course, one of the concerns some raise about the process: better law reviews routinely accept half-baked articles by big names and then make them much, much better. That hardly makes one feel better about the selection process.

Frank: I don't think we need to get super-rigorous about these comparisons. I'm clearly drawing upon the anecdotal senses of those who publish and read in the peer review world. But your point, of course, has validity as far as it goes.

Greg: I DO contest that the best rises to the top. Slickness rises to the top. Writing as if one is the first to write on a subject or writing as if your ideas are radically new rises to the top. Careful and excellent work in less sexy fields routinely has difficulty placing in the "best" law reviews. Interesting insights that do not conform to form routinely get relegated outside "top 30" journals.

Posted by: Ethan Leib | Sep 11, 2006 5:27:05 PM

Competent law-econ or law-finance work is almost never published in law reviews. Whenever you see a decent law-econ article in a law review, it's almost invariably a dumbed-down spinoff from a real article published earlier in a peer-reviewed outlet.

Posted by: Kate Litvak | Sep 11, 2006 4:11:44 PM

If you compare what passes for "economics" in law reviews, with (for example) the papers in the American Economic Review, you will undoubtedly find that the stuff in AER is about 1000000 times better.

I disagree that law reviews cannot be compared to journals in other fields. That would be true if law reviews and law professors wrote about the law. That is not often the case, however. Instead, many articles seem tangentially related about the law, but in the end explore the author's interest in another field-- and it's not that hard to compare that article with articles in a journal peer-reviewed by experts in that other field.

Posted by: andy | Sep 11, 2006 2:18:13 PM

I problem I found as an editor, and a reason I think many law reviews fail the rigorous review threshold, is that the difference in status between 2Ls and 3Ls, on the one hand, and academics, on the other, makes it very difficult for both the students and the academics, especially when the academic is a professor at the law school the student attends. Even confident, competent students can buckle under the pressure of an academic that does not have the time or inclination to respond adequately to the students' concerns. In addition, this is often the editors' first exposure to the editing process of "serious" scholarship, and they are often under an enormous amount of academic pressure themselves, not to mention the pressure of securing a job, making the impulse to simply "shuffle paper" appealing.

Posted by: rex | Sep 11, 2006 1:56:46 PM

Being in the midst of first round revisions in response to a long memo from a law review, I must say that the level of commentary is far superior to many top peer review edits. These students invest so much time getting it right and they far outnumber the two or three peer reviewers that react substantively in most peer review. About 9 students read my piece and each made substantive suggestions. and this is just the first round.

Posted by: Orly Lobel | Sep 11, 2006 12:52:42 PM

For my money (I just got a Ph.D. in philosophy, but am teaching at a law school) I think that the stuff in philosophy journals is way, way better than the stuff in law reviews. But I'm looking mainly at metaphysics, epistemology, and language. For ethics, law reviews might benefit from being closer to actual tangible disputes.

Posted by: Chris | Sep 11, 2006 12:23:15 PM

I agree with "S.cotus". Further, for you to be correct, then you must believe the "lower" journals publish such ground-breaking scholarship that "higher" journals miss. Do you? If the author believes it's ground-breaking, then certainly the author will want to be published rather than withdraw the article completely.

As a former editor, I can tell you the best rise to the top. Any one committee could miss a great article, but it's doubtful that the top 5, 10, 20, 30, etc., law reviews would miss it. This numerosity makes up for lack of "scholarly" experience.

Posted by: Greg | Sep 11, 2006 12:02:55 PM

A corollary of Frank's point is that the plurality of legal scholarship means there's a much greater volume of it. Two hundred law faculties in the country with tens of faculty members on each of them means an onslaught of publications sent out to law reviews every season (as any former editor can attest). By contrast, in the humanities, journals are area-specific, and there's self-selection going on in the submission process because most of those journals require submission to only one at a time. The result is that legal academia may seem a more crowded marketplace of ideas, but I don't think that means it's worse. A senior colleague here dismissed concerns about the student-editor issue by observing that in his experience, the best articles tend to get taken by credible journals. The errors, in his view, run in the opposite direction (lower quality articles over-placing). But this seems like less of a problem to me because it's much easier to ignore an article that does get placed than to find out about a great article that fails to get published at all.

Posted by: Dave | Sep 11, 2006 12:00:40 PM

A corollary of Frank's point is that the plurality of legal scholarship means there's a much greater volume of it. Two hundred law faculties in the country with tens of faculty members on each of them means an onslaught of publications sent out to law reviews every season (as any former editor can attest). By contrast, in the humanities, journals are area-specific, and there's self-selection going on in the submission process because most of those journals require submission to only one at a time. The result is that legal academia may seem a more crowded marketplace of ideas, but I don't think that means it's worse. A senior colleague here dismissed concerns about the student-editor issue by observing that in his experience, the best articles tend to get taken by credible journals. The errors, in his view, run in the opposite direction (lower quality articles over-placing). But this seems like less of a problem to me because it's much easier to ignore an article that does get placed than to find out about a great article that fails to get published at all.

Posted by: Dave | Sep 11, 2006 12:00:37 PM

FWIW, some law reviews are doing informal peer review. My understanding is that when circumstances permit, Harvard LR is getting comments from scholars inside and outside HLS to issue comments on manuscripts that make it up the chain of review. Also, some faculty journals are not peer-reviewed, and thus decisions to publish function in a network-y kind of way; that's not to say the stuff they publish is crap. Far from it. Just that there's a mixed bag in lots of places--though perhaps peer reviewed places are less likely to publish crap. So, in the end I'm with Luban; at least on areas I know I'm not the skeptic Kate Litvak is (check out her UT law bio--she's got a cause!), at least with respect to most of the stuff published by much of the better law reviews.

Posted by: Dan Markel | Sep 11, 2006 11:52:42 AM

The problem is that we don’t have a definition of “good” that we can use to empirically verify which mode of selection is better.

Posted by: S.cotus | Sep 11, 2006 11:14:40 AM

I have many reservations about the law review process myself, but I don't see how you propose to make this comparison, Ethan. Legal scholarship, like poli sci, is itself deeply plural. How do we compare the "groundbreakingness" of incommensurable (and perhaps conflicting) research programs? Are there conceivably similar standards for rational choice, political philosophy, political theory, or area studies (to take but 4 of the 41 or so sections at the APSA)?

Similarly, can we compare the novelty or rigor of an article on cyberlaw with one on legal history? Can I say "Julie Cohen's recent piece on cyberspace as place is far more (or less) groundbreaking and rigorous than John Langbein's recent article on ERISA"?

I only bring up these points because I think it's meaningless to compare scholarship across disciplines, and often within a discipline. Rather, standards of rigor and novelty can probably only be applied within subfields. I have briefly pointed to evidence of that development in economics (based on Colander, et al., eds., The Changing Face of Economics) in my Information Overload piece. The sooner we can make it, the better.

Posted by: Frank | Sep 11, 2006 11:07:14 AM

Luban seems right to me.

Posted by: Sarah H. | Sep 11, 2006 10:59:46 AM

The comments to this entry are closed.