« JOTWELL: Epstein on Enns and Wohlfarth on swing justices | Main | Baseball rules, again »

Thursday, October 24, 2013

Wheat, Chaff, and Law Reviews

I really thought I could stay out of this Liptak-law-review discussion. I agree that Liptak's piece, with due respect to him, was not great. But there is always a risk that criticizing a piece leads to an over-defense of the current system of law reviews and the scholarship that appears in them. Preserving the baby doesn't mean we need to keep the bathwater, necessarily. There is a lot of bathwater in law reviews!

In any event, what moves me finally to write is Dan Solove's post in (partial?) defense of law reviews, and especially his responses to commenters. Dan writes in one comment: 

The issue is . . . whether the peer review system is worth the time and effort to implement in these days where there isn’t a scarcity of publishing opportunities.

Quite frankly, I’d rather spend my time reading pieces I know are good and working on my own scholarship than doing a lot of peer reviewing. I can tell rather quickly if a piece is going to be any good, and I can quickly move on if it isn’t. And if I don’t know whether a piece outside my field is any good, I can ask professors in that field whom I trust or see what pieces they recommend on Twitter and in blogs, etc. . . . Life is short, and I’d rather not spend my time this way, and I bet many law professors think the same. 

A commenter writes to chide him for this: "Peer review in other fields is considered one of the things that responsible academics do. Annoying and boring perhaps, but it helps separate the wheat from the chaff, and multiplied over all the people who have to do their own personal peer review every time they are looking for an article saves net time." Dan responds:

I think you’re missing my point. Peer review would matter if there were a scarcity of publishing opportunities. But there isn’t. So why bother putting so much time into front end review when there isn’t scarcity anymore and anyone can publish? Why not devote more time to other things?

The argument that responsible academics just do peer review doesn’t really answer my point about why pre-publication evaluation is important and why post-publication evaluation won’t be sufficient.

People can readily find the wheat from the chaff by looking at commentary on already-published works. Basically, being published just doesn’t mean much anymore, and I being published never was a particularly precise proxy for being good. It has been a proxy better than random, for sure, but not a precise one, and not one I’d rely heavily upon — even with peer review.

I think that most debates about peer review seem to be stuck in a bygone mentality where publishing opportunities were scarce. We’re living in a very different world now.

I agree with Dan that chaff is inevitable in scholarship and sorting is always necessary. I also agree that the lack of scarcity and the availability of venues like SSRN raises interesting questions. But I disagree with much else here. In particular, let me offer two comments.

First, I don't think the inevitability of chaff tells us enough about whether the current system is problematic or not. In particular, we might consider whether the current system overproduces chaff, and what systemic consequences that might have. I can think of several possible negative consequences. It raises the search costs, because of the time and effort involved in finding the few needles in an ever larger haystack. It's true that experts like Dan can do this efficiently, but not costlessly. Meanwhile, the law is full of generalists, especially in the judicial ranks, and the search costs for them have grown. It also increases the potential error costs: the possibility that some reader will give undue credence to a poor or erroneous piece. Even given that there is always lots of chaff and some wheat, we might still have reasons to want to avoid having too much junk out there. 

More interestingly, perhaps, the abundance of poorly policed publication opportunities may create or support bad incentives for law professors and the legal academy. I don't disdain theoretical or interdisciplinary work, but neither do I disdain straight doctrinal work. The plethora of publishing opportunities and the poor pre-publication gatekeeping may encourage more professors to do more "ambitious" and less doctrinal work, to focus on articles over treatises or PLI pieces or other doctrinal work. Moreover, because publication is treated as a core part of the job of full-time faculty, and standards of pre-publication judgment (not to mention tenure and promotion standards) are weak, it may lead to or at least enable not just an overproduction of non-doctrinal work, but an oversupply of full-time law faculty altogether, as opposed to adjuncts and other teachers with a stronger ongoing connection to particular practice areas. Perhaps if there were stronger pre-publication standards and a concomitant decrease in the number of seriously regarded publication slots, we would have a smaller, stronger professorial corps, or a more efficient distribution of resources in law schools, both between doctrinal and non-doctrinal professors and between full-time and part-time faculty.

In short, lack of scarcity is not everything. There are still independent reasons to want to separate the wheat from the chaff earlier in the process, and to encourage more faculty to work on more narrow and doctrinal pieces rather than take a flyer on more high-flying pieces (on the view that the piece will surely be published somewhere, and might just be published somewhere great).

I also think Dan shrugs off the point about peer review and related academic work too casually. For the reasons I've offered (and other reasons too, I'm sure)--e.g., the risk of overproduction of chaff and the associated risks of general readers overrelying on poor work--I think there are reasons "why pre-publication evaluation is important and why post-publication evaluation won’t be sufficient." And, both for practical reasons and more abstract, virtue-centered or vocational reasons, I think the commenter is right that services like peer review are a fundamental part of the academic's responsibilities and should not be dismissed as unnecessarily burdensome. Part of the academic's job is to make sure not just that her own work is sound, but that the general body of scholarship in her discipline is sound: that the well is not poisoned or diluted, that some level of minimal disciplinary standards are maintained and enforced across the board, and so on. She should also not want her colleagues, especially junior ones, to spend an undue amount of time on work that need not or ought not be done at all; she shouldn't want her junior colleagues to overproduce poor work, secure in the knowledge that someone will take it, when that scholar could be encouraged at an earlier stage to abandon a poor or unnecessary project and use her time and talents more usefully. I should add that I see no reason to doubt that Dan takes on his full share of these burdens, no matter how wearisome he may find it. But whether it's wearisome or not, it strikes me as serving practical purposes, as ultimately doing a kindness both to one's colleagues and to the overall body of scholarship, and as a meaningful part of the package of rights and responsibilities that make up the scholar's vocation.   


Posted by Paul Horwitz on October 24, 2013 at 10:51 AM in Paul Horwitz | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Wheat, Chaff, and Law Reviews:


I agree with Justin --- the American legal academy has an informal peer review system in that we often ask others in our field to read our manuscripts & then we improve upon them before submitting for publication. The people who take advantage of that informal system and improve their manuscripts accordingly, often find themselves benefitting from the reputational bump that Orin describes.

Also, to the extent people think that the peer review process is preferable b/c it would result in more specialized articles that do not contain significant background information, I don't think that would solve the problems that Liptak has identified. Namely, if we want law review articles to be more influential with judges and practitioners, then making them more specialized would make them less useful to generalists, i.e., non-law profs. Also, from my own entirely unscientific study, it seems to me that, when law review articles are cited in opinions, it is often a citation to the copious background information that some commenters think is so worthless.

To be clear, the conflict between peer review and relevance for generalists is a problem that originates with the Liptak article --- which was a weak effort for many, many reasons.

Posted by: carissa | Oct 28, 2013 2:31:52 PM

Brad, the argument about ex post review isn't about the accuracy of the description of existing law, which is the part that is likely to be of interest to practicing lawyers. That's one area in which student editors actually do a better job that peer reviewers: Student editors cite check the cases to make sure every line describing them is accurate, which faculty peer reviewers won't do. Instead, the argument about ex post review is about whether the broader argument made by the article is novel, makes sense, is realistic, is normatively appealing, etc.

Posted by: Orin Kerr | Oct 28, 2013 12:57:10 AM

Will, good point. I imagine the concept would transpose most easily to interdisciplinary work, particularly drawn from the social sciences---empirical legal studies, for example, has (or ought to have) pretty clear-cut methodological standards, and the same could be said at least for the more formal wing of L&E, etc. (Incidentally, I know I've seen "empirical" work from law professors that wouldn't even get sent out for review at a political science journal.) Doctrinal work could also benefit from at least basic soundness checks, i.e., to avoid blatant mischaracterizations/misunderstandings of law---not quite the same idea as PLOS one, but roughly analogous.

The other thing that I'd like to see in peer reviewed legal scholarship, perhaps combined with a PLOS model, and which could help dull the edge of methodological disagreements, would be open reviewing. Imagine the following: author submits article, reviewer gives very critical comments, and then author gets to decide whether to publish article with reviewer comments (and hence make the disagreement public and allow the academy as a whole to decide), or revise and resubmit.

Posted by: Paul Gowder | Oct 27, 2013 5:10:15 PM


I'm intrigued by your suggestion but I'd like to know more about how it might translate to law. It seems to me that in law there's frequently fundamental disagreement among members of the legal academy about what methodologies are sound, and hence I wonder how review for methodological soundness would work.


Posted by: Will Baude | Oct 27, 2013 11:06:48 AM

anon @ 2013-10-25 4:18:
No one contends that court decisions are quality controlled via amorphous post-hoc reference by other courts. There is an immediate process for appealing decisions by those most affected, and there is a great deal of process on the front end to ensure that the decision is right to begin with. It is for this reason that court decisions are considered at least persuasive unless specifically called into question by a court of the relevant jurisdiction.

If it's your contention that law review articles are entitled to the same default deference by dint of their selection by 2Ls, you are free to argue that. But many here and throughout the greater discussion, some have claimed that post-hoc peer review is an important part of the process. I'm just trying to figure out how that's supposed to work in practice.

Posted by: brad | Oct 25, 2013 5:48:58 PM

Re brad. Isn't this the exact same problem that lawyers deal with all the time in respect to cases from lower courts? Some lower court issues a decision. It gets cited rarely by other courts except in string cites. One or two courts (not in your jurisdiction) criticize it, but it still gets cited in your jurisdiction occasionally, although without much discussion. Do you cite it in your brief or not?

Lawyers handle this problem all the time with respect to caselaw, I guess I don't see why it is a qualitatively different matter with articles.

Posted by: anon | Oct 25, 2013 4:18:25 PM

I'm still puzzled by how this post-hoc peer review is supposed to be discovered and utilized by a reader.

Suppose an attorney is looking for research on a particular topic. He throws his search up into Westlaw and finds a likely looking article from last year published in a journal of a law school he's heard of. Is he supposed to look for citations? Ok, suppose he finds it's been cited once or twice with no discussion, just as part of a cite for a proposition. Does that mean the article is reliable? Unreliable? If there are no cites, does that mean the article flunked post-hoc peer review or that it was just never subjected to the process? Suppose there's one cite and it attacks the paper extensively. How is the attorney supposed to know whether or not this second paper is the work of a crank? Second level search for post-hoc peer review?

I don't say that peer review is perfect, but at least you know that someone, somewhere who is an expert at the topic at hand has the read the piece and signed off.

Posted by: brad | Oct 25, 2013 3:33:54 PM

Here's a question. Why don't we yet have something equivalent to PLOS One in law?

PLOS One: Open-access peer-reviewed in the hard sciences, where peer review is for methodological soundness, not impact or anything like that. Online only. So basically, SSRN but with a layer of peer review.

It strikes me that having something like that in law would be a great happy medium between the law journals and full peer-reviewed journals: it would still have the advantage of speed and of focusing primarily on post-publication review (and hence of maximizing the pipeline for work to get out there), but with the additional filter of peer review to eliminate the total garbage. And because the peer review model is relatively light compared to traditional journals, I imagine it's also less burdensome on reviewers, and also less likely to lead to the infamous "s/he didn't cite me, reject!" problem.

Posted by: Paul Gowder | Oct 25, 2013 12:35:40 PM

Orin Kerr's comment seems to offer a curious perspective on faculty -- a serious mistrust of their ability, or willingness, to approve of articles that differ from their own work. This may reflect a perception that legal scholars are not really scholars but are just advocates or guardians of the status quo, whichever you prefer, but I don't think there is much evidence for that bias outside of law, and it seems a deeply problematic view of one's colleagues, particularly when it is suggested that students can be better trusted -- which again seems quite odd given all the various studies regarding student bias in selecting articles based on institutional affiliation. There might be reasons to support student-edited law reviews but avoiding faculty bias in selection does not seem to be a compelling one.

Posted by: anon | Oct 25, 2013 8:24:55 AM

I am a European scholar, and can confirm that one of the consequences of peer review is the danger that articles developing very innovative ideas never get published. I have published my most innovative articles in the US, despite the fact that I could obviously not use the letter head of a top US law school. Also, it is hard to develop a very innovatice idea in a few pages, so, for some projects, it is necessary to have a limit higher than 10,000 words.

Posted by: Gil | Oct 25, 2013 5:57:50 AM

Anon Prof at 7:24 nails the problem, I think. The question is, would peer review make the problem better or worse? Students often maintain the status quo because they're focused on big names at top schools. But I'm not sure professors would be better: Too many professors judge the quality of work based on whether it approximates their own. So if the professor insiders were in charge of picking articles, I fear that echoing the views of the dominant insiders would become necessary to place articles. At least with students, outsiders can develop reputations over time. It's common for someone good to place their first few articles in a Top 50 journal, then the next few in Top 30, the next few in Top 20, etc; In effect, outsider authors can build the reputation that students use as a proxy through a history of prior publication. But if the dominant insiders could make the call, their relatively monolithic view could become the only view that anyone hears about in top journals. (I should add that the idea that the dominant insiders have a monolithic view depends on the field. I'm most familiar with subject areas where there is a specific set of accepted positions that are widely held in the field and essentially universal at the top schools, but that won't be true in all subject areas.)

Posted by: Orin Kerr | Oct 24, 2013 8:58:03 PM

The big problem I see with too much chaff is that it allows the dominant insiders to continue to drive the game.

How often do you see some bigger name scholar rehashing or reiterating points that were published elsewhere? Feels like this is happening more and more each year. (Yes, I realize the meta nature of this comment given that we're having this peer-reviewed vs. student-edited scholarship debate yet again & why this round was launched).

How easy is it for someone who is a complete outsider to the system to get published or read and have their work become part of the primary discourse in their field? In some highly specialized fields, newer or different voices can move or shape the dialogue or paradigm shift. But in increasingly more areas of law, the scholarship debate is captured by a very specific group of people with a narrower sense of what is contributing or offering new ideas and the only new voices accepted as even being part of the conversation are protégés of the insiders in the game to begin with.

When few kernels of wheat are buried by noisy chaff, everyone increasingly relies on the signaling that comes by having something published by someone they know or trust (or that their colleagues do), and it makes legal scholarship a venue that is less accessible to anyone else.

Lots of good stuff is out there that no one is reading. But the fact that it is out there doesn't mean that it is easy to find.

Posted by: Anon Prof | Oct 24, 2013 7:24:24 PM

I just want to add that if legal scholars appreciate the peer review that makes their books published with university presses (both Solove and Horwitz have them) as respected as those coming from scholars in other disciplines, I wonder why not apply the same logic to law review articles?

Also, as somebody familiar with the processes in a cognate discipline, other non-law academics also have fellow academics read and comment on their papers before submitting to a journal, in addition to workshopping or presenting them in various fora + peer review pre-publication as well as post-pub evaluations.

Posted by: anon | Oct 24, 2013 6:59:08 PM

At least for me, I find it highly beneficial to have others read and comment on the paper before I submit it to a journal. I also try to present my papers at 2 or 3 workshops before submitting them. (These are separate things. I can send the paper directly to people I know and trust for comments, usually before I start the workshop process. When appearing at workshops you have no control over who else will be there.) The additional improvements that result from receiving comments and presenting your paper at workshops improve the finished product substantially. In fact, I think this is the single largest difference between the articles I published as a practitioner versus those I have done as an academic.

Posted by: anon | Oct 24, 2013 5:41:18 PM

I'm confused by some of the comments above. Isn't it standard practice for American legal scholars to show drafts of work to peers prior to submission? Cynics suggest that this is done just so that the first footnote includes famous names, but nevertheless, these peers are reading the draft and offering comments, giving the author a chance to clarify, mitigate, or amend. Why is it more *scholarly* that this be done once with one set of reviewers as an explicit prerequisite to publication, then if denied, a year later with a different set, and so on, rather than just once and then published somewhere with every footnote checked (and yes, students really do do this)?

Posted by: Justin Long | Oct 24, 2013 5:32:20 PM

I am not entirely sold on the benefits of peer review (from the perspective of the author). I have submitted twice to peer-reviewed journals. The first time, the response I got was one paragraph long and wrong about the state of the law on the particular issue I was writing about. I obviously don't know who reviewed it, but I doubt it was an expert in the field. The second time, I waited nearly 9 months to get a "revise and resubmit." The comments I got back were marginally useful that time, but on the whole less useful than the comments I would get just workshopping the paper. At that point, I got tired of waiting and submitted it to mainstream law reviews, and it was picked up by a reputable one in a couple of weeks.

Which is not to say that I haven't had issues with student-edited journals. But so far, my experiences with student run journals have not been as bad as my experience with peer-reviewed ones. I can always delete the redundant/inane/wrong parentheticals that seem to be the most egregious thing that student run journals add during editing.

Posted by: anon | Oct 24, 2013 3:09:34 PM

A few thoughts:

First, I wanted to echo MS's comment about peer review contributing to better (and shorter) articles overall. I've recently gone through the peer review process for an article forthcoming in a Canadian journal that is peer reviewed, but student edited (an interesting hybrid that could work well in the US). The comments I got back in peer review were in general quite excellent and led to me clarifying a number of points, expanding on a few others, and even adding in some material that I had not thought to include in the first place, but which should definitely have been there. The comments from the student editors were also quite helpful, but in a different way—they focused on immediate clarity, rather than the broader argument, and, in at least a couple of places, suggested changes that would have resulted in an article that was less clear to a knowledgeable reader. While one might not always agree with one's peer reviewers, the process almost always makes for better scholarship.

It's also noteworthy that American law review articles are among the longest scholarly articles published. In most disciplines, and in law outside the United States, articles tend to be much shorter, with 20-30 pages the average. A lot of the difference between a 30 page article in Canada and a 50 page article in the US is the nigh-obligatory background section, in which the author relates everything that has ever happened/been written on the subject. While these are occasionally useful, there is wide recognition that they are in reality written solely to explain the issue to the student editors, who otherwise would not understand what is novel about the article or why it is important. Under peer review, these long background sections are unnecessary, because the individual accessing the article's quality already knows the field. (On a related note, peer reviewers tend to complain about long articles, whereas student editors often mistake length for quality.)

Second, while the existence of peer review in other disciplines is not an argument in and of itself for instituting widespread peer review in law, the lack of peer review does have serious reputational effects for legal scholars, especially interdisciplinary scholars. Though we tend to ignore the issue most of the time, our lack of peer review is often used as evidence by those in other disciplines (and other countries) that legal scholarship is not "serious." Moreover, many major grants, fellowships, etc., treat publications in non-peer-reviewed journals as inferior, regardless of how selective the journal is. Not only does this weaken the reputation and prospects of American legal academics, but it also weakens the journals themselves, since they lose many excellent submissions from international scholars and scholars in cognate disciplines (where, for instance, are all of the political scientists writing about legal institutions?).

As for changing the system, I agree with Bruce that it will be quite hard so long as the top general interest law reviews continue to accept multiple submissions without peer review. But, I suspect we are fast approaching a breaking point with the current system, where the volume of submissions is simply unmanageable, and at least one of the top journals will soon find it in its best interest to announce that it will only accept exclusive submissions on a going-forward basis, with publication decisions made by peer review. Maybe I'm wrong, but I suspect the journal making the change first will see immediate benefits—both a decline in the number of total submissions and an increase in their average quality. In the meantime, specialized peer review journals can continue to propagate and poach high quality articles from the general law reviews.

Posted by: Charles Paul Hoffman | Oct 24, 2013 2:40:31 PM

Paul, I agree with most of your post. The appeal to general scholarly norms, however, that the CO commenter relied on strikes me as fairly circular. Other fields that rely on peer review need a norm encouraging peer review in order to get people to do it. That doesn't tell us whether a field *without* widespread peer review should abide by the same norm. For that I think you need an argument that the benefits, all told, would exceed the costs.

I took Dan's response as basically making the cost-benefit point, and arguing that the costs would exceed the benefits. I'm not sure I agree for the reasons you state. But one problem is though, beneficial or not, I think it would be extraordinarily difficult to launch a peer-reviewed, exclusive-submission general-interest law review that competed for submissions with the top law school reviews. The existence of the current system saps the energy for change.

Posted by: Bruce Boyden | Oct 24, 2013 1:19:19 PM

I very much appreciated your defense of the peer review system and wanted to add just a couple of other factors. One that I know many find unpersuasive but has always seemed helpful to me is that if peer review has so little utility, why do all other disciplines use it? Why hasn't open source, which is basically what Dan Solove and others have proposed, taken over? I have to say it seems a bit absurd to say just read everything that comes out and determine as quickly as you can what is valuable. That's crazy. In other disciplines, the top peer reviewed journals are routinely read by informed academics, it is a filter and generally a good if imperfect one. In contrast, Law Reviews are still written as if someone is going to pick one up on their way home to read on the train from cover to cover. We are at least starting to get some discipline-specific peer reviewed journals but they are still too specialized for many articles. So I suppose one question is why is peer review appropriate for other disciplines but not law? (I had always assumed the answer was because of the abundance of footnotes in law articles which students are supposed to, but rarely do, check.) Peer review would also improve scholarship in at least two ways -- comments that peer reviewers provide can be very helpful, and they can also point out when someone has missed some relevant literature. Peer review would also likely make the articles much much shorter for it would no longer be necessary to explain every issue to uninformed students and finally we could eliminate the many articles that simply rehash (often unknowingly) what has come before since again an informed audience should know what has come before where students do not. On this last point though, I hold Professors rather than students responsible, as it is inappropriate to treat something as scholarship that might appear novel to students but which someone familiar with the literature is more apt to dismiss. Finally, if we took our professional obligations seriously, as you suggest and as they do in other disciplines, the problem of not having people willing to do peer review, which I think today is a potential problem, would quickly fade away.

Posted by: MS | Oct 24, 2013 12:57:34 PM

Post a comment