« Duke Lacrosse and Extra-Judicial Speech | Main | Weekend Trivia Challenge - Highest ranked non-U.S. journal »

Friday, April 11, 2008

...Because things are not so bad the way they are… (on the law review front)

In response to a lot of interesting and important thinking about law reviews, law schools and legal education this past week, for example here and here, I thought I would try and offer a partial defense of the-way-things-are-right now on the law review front:

  1. Don’t romanticize the alternative: When one begins to publish in the peer-reviewed world, the whole romantic notion of blind review becomes somewhat tainted -- in all of the fields I am familiar with -- psychology, economics, business, political science, history and sociology reviews (and also in the fields I am less familiar with, according to my hard scientist friends)  -- there is still A LOT of noise in the system: because of conferences, networks, SSRN and the like, it is VERY easy for reviewers to figure out who the authors are. There is a lot of tit-for-tat -- same folks publishing and editing in the A-journals in their fields. To think that a top name (the Cass Sunstein equivalent in any one of those fields) does not impact the decision of her peers is naïve. And even more so because the double blindness from the author's side is also often tainted -- at times, people can guess pretty easily (or hear about it in the process) who the reviewers are. At least with law review editors, the repeat players "problem" is ameliorated--but of course repeat players have many strengths as well in terms of expertise and knowledge about the field and quality of scholarship.
  2. There are in fact quite a few law or law related peer-reviewed journals so we are already in a world of choice and a world of competition.
  3. Law reviews themselves do consult quite frequently it seems with faculty advisors on accepting pieces. It is also my understanding that some of the top law reviews, like the Harvard Law Review, do really have blind review process, and their process is a multi-layered, multi-tiered serious one.
  4. The law review kids actually offer a lot of good editing after the article has been accepted, often better, and most of the time involving more eyeballs, than those of peer-reviewed journals.
  5. I suspect many law professors, like other faculty, would not like Ann’s suggestion to take away the prestige, the wins and losses of the game, of sending out one’s article to journals outside one’s own institution. Our work is intangible enough to not want to give away some passing rewards and signals of success however noisy they are. This is similar to suggesting that we do away with rankings – we should keep rankings, just make them far more meaningful. Remember, type A people strive on competition.
  6. There is something really cool about the speed and timeliness in which law reviews make decisions and then publish articles. Peer-review articles often take years upon years to come out and the review process may mean that things are not relevant by the time the article sees the light of day.
  7. There is also something cool about law review editors having some productive ignorance. It's ok if they sometimes get things “wrong” – its part of what shakes things up, puts some off ideas on the frontlines and disrupts some of the inner-breeding and self-referential work that at times stales the legal academy.

Bottom-line, the baby water and the baby have some advantages.

cross posted at Madisonian.

Posted by Orly Lobel on April 11, 2008 at 05:28 PM in Orly Lobel | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference ...Because things are not so bad the way they are… (on the law review front):


I appreciate your points, Bob, but I'm still a bit stuck on who loses when so-called "good" journals publish bad scholarship. Nobody likes to wade through bad writing and frivolous ideas during their research, but I doubt that the average spectator of law reviews--scholars--are hung up much by this. Most scholars are quite versed in their field and can typically spot poor work (whether it rises to the level of holocaust denial or not). The casual reader may is no doubt susceptible to shoddy work, but I have a tough time believing that there is such a thing as the casual reader of a law review. Thus, it seems to me like the only audience that law reviews have is an audience that is savvy to fake scholarship.

The one field where I will concede your point as to reliability is empirical work. Much of that is beyond the ken of most law professors and, unless the numbers appear outrageous, the work is likely to be taken as at least facially true. But the great majority of legal scholarship is normative work built on cases, history and theory, most of which the typical professor can sort out without too much trouble at all. It seems from the book you wrote that you have contrary evidence. I am certainly open to that. If the non-peer system really is causing harm, and a peer-review system can be adopted in a cost-efficient manner, then it should be adopted without question.

Posted by: Jack | Apr 13, 2008 9:58:05 PM

The law review problem, I would argue, is primarily one of the reviews themselves, because nearly all of what is published is not subject to any evaluation by people with knowledge of the field to which the submission claims to contribute. Those in the field of law are, by and large, the product of that field, in that many at the top of the legal profession served on law review as students (a high honor). So it’s only natural that credence is given to law reviews subsequently. The other point to make is that scholarship is not supposed to be like an unregulated open marketplace, where, say, the Holocaust deniers are somehow on a par with those who believe the Holocaust really occurred. Scholarship’s primary purpose is to advance knowledge, and that simply cannot occur in any sane way if there is no quality control, no evaluation of the quality of a piece of work based on its merits. Every field of study is advanced by specialists, such that even most people in a given field don’t have the tools to evaluate most work. That’s why peer review is indispensable. Without it, reliability is nearly impossible to determine. Law students, by definition, lack the ability to make any such evaluation. They’re students, not trained professionals. Peer review isn’t about tenure decisions, but about advancing knowledge.

Posted by: Bob Spitzer | Apr 13, 2008 6:48:15 PM

I'm a little scared to jump into this, but it seems like the critics of the laws reviews here--who make some fair points--are missing a key point. The problem is not in law reviews themselves, however naive their members may be, but in the credit we (members of the academy, mainly) give them. Law review editors may or may not be up to truly assessing the merits of an article. Some students certainly are, many certainly are not. But so what? It is a process for getting ideas out there. It's like SSRN, but without the typos and with everything hyperlinked and hyper-searchable.

Thus, the critics' point seems to be this: becuase law reviews cannot be trusted as proxies for intellecutal quality, it is unduly difficult to determine quality--whether for general research purposes or to determine whether to award tenure. But in an age where every library carries every journal simply by having an internet connection and westlaw subscription, why do we need others to make quality decisions for us? Just read the article and use your own brain to figure it out. Smae goes for those on a tenure review committee.

Perhaps this discussion could be furthered by the following thought experiment: what would happen to the hard sciences (both in terms of tenure review and the actual promotion of knowledge) if peer review were abolished. One big value that I can see to peer review is that tenure decisions would be less political. In a non-peer world, schools will still do a personal read of the professor's work, but if the junior professor publishes 5 great articles all in non-peer reviewed journals (because that is the norm), an enemy of the professor on the faculty might find it easier to blackball the tenure candidate if there are relatively few external, disintersted measures of quality. Peer review could thus silence, or weaken at least, those who are intent on IGNORING quality. For the rest of us--who are intent on FINDING quality--peer review helps realtively little. And to the extent it does, I believe that the law reviews currently operate as a rough--and I emphasize rough--proxy for quality.

under review cannot point to a neutral party to validate his work (i.e., the approval of a peer reviewed journal)

Posted by: Jack | Apr 13, 2008 5:38:25 PM

I’d like to respond to the April 11 post about law reviews, “Because things are not so bad the way they are. . .(on the law review front).”
Well, yes, they are, and it’s even worse than you may think.

Peer review, first, is not only the norm, but the gold standard for professional publications in literally every other academic discipline. The reason for this is fairly obvious: it is the best system yet devised to judge the merit of academic writing. Is peer review perfect or fool-proof? Of course not. But it has two unassailable advantages not found (with a literal handful of exceptions) in the world of law reviews: first, publication decisions are made by professionals with subject matter expertise, which means that the submitted work can be evaluated on its merits; and second, the review process is normally blind, meaning that the reviewers’ names are not known to the author, so that the reviewers (under professional editorial control) can offer full and frank evaluations. The primary check against possible abuse by vengeful or unfair reviewers is knowledgeable editors who can weigh the relative merits of reviewer comments, or solicit additional reviews if needed.

Compare this to student-run law reviews, where publication decisions are made by hard working and well-intentioned people, but who are students. That is, they do not possess, and cannot be expected to possess, the knowledge to evaluate submitted work on its merits. Yes, they can find missing punctuation and erroneous citations, but these minor technical errors pale in the light of the fact that they are unqualified to judge whether a submission makes a legitimate argument, reflects proper knowledge of the field, does or does not duplicate existing writing, or even presents arguments and facts fairly and correctly.

While it is at least possible for authors to figure out who a given reviewer might be, I know of no evidence to support the idea that such revelations are easy to discern or are frequently discerned. Quite the contrary. The most likely way that a reviewer’s identity would be discerned is if the reviewer deliberately reveals specific information in the review that allows the author to identify the person. But again, there’s simply no evidence to support the idea that this occurs beyond the rarest instances. And my own experience of publishing dozens of articles in peer reviewed journals, and reviewing many dozens of articles for many different journals, is that revelations just don’t occur.

Among the over 600 publications emanating from the nation’s 220 law schools, no more than a literal handful rely on peer review, and if I’m not mistaken, none of the top law reviews have peer review in the way that term is defined in every other academic discipline. Student law review editors certainly do consult with departmental faculty at times, depending on the publication and the institution, but this is not peer review, either.

It is certainly true that law reviews can publish quickly compared to other disciplines, but two considerations undercut this potential virtue. First, there are many ways to publish an important idea rapidly, either in professional publications that specialize in reporting new or important findings (most disciplines have such publications, including my discipline of political science), or as newspaper op-eds, magazine articles, or on the internet; second, and more important, the virtue of rapid publication is negated if what is published is defective or wrong. As others have noted, the maxim of the medical profession applies to scholarship, too: first, do no harm. Better to be slow and right than fast but wrong.

This new book (see link below; disclosure: I'm the author) argues that these very traits of law review publishing lead to the formation of wayward theories about the Constitution:


Posted by: Bob Spitzer | Apr 13, 2008 4:29:47 PM

Supremacy Claus:

Your last response is a series of dubious conclusory allegations. Responding at all probably isn't a good idea, and responding to each one would simply take too much time. Nonetheless, one of them bears a response just to show how little needs be done to bring down your house of cards.

The partial gist of your diatribe is that us law students have no experience with the law (contra medical students' experience with medicine) so that we shouldn't be selecting legal scholarship for publication because scholarship, in its own way, drives development of the law.

In fact, law students are quite well-exposed to the law outside of the academic environment. Most law schools have developed clinical externship programs, where students can represent clients and try cases. At my school and many others, students can represent clients ("the troubled parties mocked") in court. Outside of school, students, like myself this semester, can intern at public interest legal organizations or they otherwise gain experience through summer employment. And that says nothing about the experiences we take with us into law school from our previous lives.

Your disparaging remarks about student editors assume, without support, that law students operate in isolation from the law, which simply isn't the case. Not only do we spend a fair amount of time in class and studying, but we have experience outside of that that gives us some perspective on the law and scholarship.

Posted by: Adam Richardson | Apr 12, 2008 11:00:32 PM

So Supremacy Claus, it's "writ[ing] a demand letter in anger," then, that qualifies one to edit a law review? Just so we're clear here. (I assume it can't be "relat[ing] to the troubled parties mocked in the articles," because no one really seems to be submitting articles which mock troubled parties.)

Posted by: AE | Apr 12, 2008 3:19:23 PM

Post a comment