« Game theory post 6 of N: the anxiety of rationality | Main | In Defense of Students, OR: Student “Quality,” Student Engagement, Incentives, and the Fundamental Attribution Error »

Thursday, January 29, 2015

Open Thread: How do we Stop the Madness?

By "the madness," I mean this.  Opaque "submission seasons" and letterhead biases and footnote fetishes and massively multiple submissions (I kinda want to start an MMORPG called "World of Lawcraft," all about getting law review articles published) and all the other crazy pathologies of law review publishing.  

As Your GameTheoryBlogger, this seems to me like a classic strategic problem: nobody likes the system, it means huge amounts of work for the students, work that (time for Real Talk(TM)) probably impairs their educations, and most of the real benefit to them is just victory in an insane status arms race in which law review membership is a signal of smartness that law firms respond to; it also undermines the scholarly enterprise to have (Real Talk(TM)) scholarly reputations and their associated benefits depend (yeah yeah only in part post-publication review sure ok) on the judgment of 2Ls with like three minutes to read a paper.  Yet we are unlikely to be able to just replace the system whole-hog with peer review, because the individual costs of doing so are so high. (I confess I kinda miss the early days of Prawfs, where Kate Litvak was around and leading the mighty charge for peer review in the comments. Yes, I remember those days, back in like 2005---I think I even remember the first e-mail Dan sent around announcing this blog's existence!)

More broadly, we seem to have lots of collective action problems like this in legal education. Think of the pitiful death of the law clerk hiring plan. And of the way that we all bow and scrape to the almighty, but universally loathed, gods of U.S. News. Can we get better at it? How do we improve our institutional capacity for collective action? All ideas, no matter how crazy, welcomed in the comments.

Posted by Paul Gowder on January 29, 2015 at 04:45 PM in Law Review Review | Permalink


How about post-publication review as a measure of quality instead of journal "rank"? JOTWELL, anyone?

Posted by: Michael Froomkin | Feb 20, 2015 12:06:27 AM

Thank you for opening this thread, Paul. It's an important discussion.

I tend to think that pluralism is a good thing. I think it would be good if there were more peer reviewed law journals. At the same time, I think student-edited law reviews have certain strengths that we shouldn't undersell -- for example, students are sometimes more open to ideas that challenge conventional wisdom. I hear many stories from friends in other disciplines about a few senior scholars essentially exercising total control on what gets published in their area of expertise, which is a potential drawback of peer review. I like the hybrid peer review system that some student-edited law reviews have begun using, in which student editors solicit input from professors (at their home school and elsewhere) but continue to make the ultimate decisions. I wish online publication (everything from blogging to publication in online law review companions) was given more weight, as a lot of online writing is quite scholarly and online venues have a lot of advantages (faster publication being an important one but not the only one).

The key in all of this is that hiring and tenure committees would have to embrace, or at least accept, the idea that different venues all have different advantages and disadvantages, and that one is not inherently superior to another. That way professors could concentrate on writing a good piece and publishing it in the most appropriate venue for that particular piece.

Posted by: Nancy Leong | Jan 31, 2015 5:38:02 PM

peer review is not the great solution there is substantial bias by reviewers
once i was rejected within hours of submitting to a UK based peer review journal only to be accepted by a top line USA law school within days of submitting that is much better than the peer review
Also, I agree that the problem is authors have no idea how many slots are open there is great mystery surrounding who is pen and when etc
Some fault surely lies with journals who fail to communicate and also reserve slots for professor on staff who recommend their friends from other schools.

Posted by: rank matters | Jan 31, 2015 12:20:02 PM

Please don't even suggest doing away with simultaneous submissions! If you want to see the disruptive effects of single-submissions (or even limited-submissions) policies, just look at the problems with commercial publishing — especially, but not exclusively, commercial fiction and commercial serious nonfiction — which lead inevitably to (a) longer and longer review times, (b) even greater "letterhead bias" than in law journals, and (c) less and less risk-taking.

I've been in editorial capacities in both areas, and VASTLY prefer the chaos of the law journals. Yes, there are things that can be improved (more consultation of faculty — I'm against substituting formal peer review, having seen the damage that does in "opinion-based" journals like PMLA, Modern Fiction Studies, and virtually every history-based journal out there; greater honesty from everyone; and so on), but it sure beats having to have a tenured sponsor from the old-boy network to even get one's work read!

Perhaps Scholastica/bePress/another system can limit simultaneous submissions to, say, twenty per six-week window using their particular services... but much less than that will discourage writers from tailoring their submissions to appropriate markets, instead encouraging a mechanical top-to-bottom-of-general-reputation approach that will make the illusory nature of journal rankings even more distorted. Some kinds of articles SHOULD have a home in a nonflagship journal where they'll get the attention and context they need/deserve, and sometimes a flagship journal has a clear tradition in certain areas of law (e.g., just compare the article mixes at NYU and Columbia), which changes rather slowly over time.

In short, this is a false dichotomy, and overrestriction will make things worse (for all the handwringing about nobody paying attention to law-journal articles, there is even LESS attention paid to journal articles in vast swaths of the humanities... and substantially less so in even my lifetime!).

Posted by: C.E. Petit | Jan 30, 2015 1:01:33 PM

The only thing need to end the Madness is to eliminate the monopoly represented by the Bar and Law School certification. A first step would be to permit anyone to practice once having passed the bar exam, without requiring that he have graduated law school.

You can rise to the top in other professions like computers, aerospace and nuclear weapons design (Gates, Dell, Wozniak, Jobs, Snowden, Ford, Firestone, Carnegie, Edison) without diploma or certification. We citizens would still be living in the stone-age if other professions had bars to progress like the legal profession foists on us.

Posted by: Jimbino | Jan 30, 2015 11:39:59 AM

I don't know about peer review, but imposing rules that would cut down on the huge number of submissions would contribute to saner evaluation policies by law reviews. The best way to get something like that into place would be for AALS to convene representatives of the stakeholders and try and work something out that would be for the collective best. Some years ago, I proposed to the AALS that it undertake something like this and really make a contribution to legal education. It declined.

Posted by: Michael Green | Jan 30, 2015 11:16:15 AM

Jeff, I *love* that.

The thing with placement as a proxy for quality is not that the people who are so using it are stupid or too lazy to read papers on their own. (I mean, maybe sometimes...) It's that the decisions where quality matters are characteristically collective, and quality is characteristically an individual judgment. It's really costly for a tenure meeting or a hiring meeting or whatever to hash out individual disagreements over scholarship quality, and so a variety of proxies, including external peer review (which can also be characterized by disagreement) and including paper placement, have been developed to make it less costly. This is fine.

The problem is that the particular proxy we use in law is a really bloody noisy signal. I don't doubt that the median paper published in the Harvard Law Review is better than the median paper published in the Bob Jones Online Journal of Law, White Supremacy, and Quantity Surveying, but the variance is so high that we don't get a lot of reliable information. In other disciplines, we get more reliable information: a publication in, say, Phil. Review, or AER isn't a perfect signal, but it's a vastly better one than a publication in HLR.

(Don't get me started on APSR.)

Posted by: Paul Gowder | Jan 30, 2015 10:37:21 AM

Your last paragraph says everything. We are a profession mad for ranking and hierarchy - and we rely on proxies all over the place. I don't like the problem you describe, but I think I agree with the earlier commenters that placement as a proxy for quality is a bigger problem.

Posted by: Kristen | Jan 30, 2015 10:22:42 AM

How about requiring at least one peer reviewed piece as a condition of tenure? Even if a small fraction of the current submissions needed peer review to "count," wouldn't we see a clamor for more peer reviewed journals?

To Paul's point, if the market for hiring entry level professors were characterized by high demand and limited supply, there'd be a prisoners' dilemma - a significant downside if a school were to take unilateral action. But the market is precisely the opposite - limited jobs and lots of applicants. And the very small move of only requiring one such piece might be palatable.

I may have my concepts or terms a little off, but peer review is a derived demand. Compare the demand for automobile electric charging stations to the demand for gas stations. Shift the usage from gasoline-powered to battery-powered cars, and I suspect you'd see a relatively quick change, sort of like the charging stations for devices in airport gate areas (compared to the old days of staking out a seat next to the outlet for the vacuum cleaner).

Posted by: Jeff Lipshaw | Jan 30, 2015 10:00:56 AM

I've written about my views on a lot of this before, but since the internet has a short memory, let me say that: 1. We have a clear path to a coordinated solution for some problems, if only Bepress and Scholastica are willing to play. I have already written to both to ask them to force journals to honestly and consistently report how many slots are open for consideration at any given time, to no avail so far. Although Scholastica claims that they are doing more to get journals to announce when they are open for business.
2. Journal rankings are in many ways dumb (and I've written a few posts on how to improve them), but it is wrong to say they are useless. I was a clerk once (twice, actually), too. If I only had time to skim 3 articles, was I skimming the West Nyack Journal of Mall Law before the YLJ? No.

Posted by: BDG | Jan 30, 2015 9:12:45 AM

Two relatively easy moves would be (1) blind review (even if still by students) and (2) exclusive submissions (or at least a rule that you must accept the first journal offer given).

(1) would help even the playing field. It might also show that we really do need true peer review.

(2) would cut down on submissions. I have seen a few journals using exclusive submissions by posting a request on this or other blogs, so maybe there is already some movement in that direction.

Posted by: Haskell Murray | Jan 30, 2015 8:51:19 AM

It is quite true that law clerks and judges will indeed cite to an article if it supports the ruling even from a T150 school BUT there are 2 things that soften this: (1) as a former federal law clerk I can say that my fellow clerks would hesitate to recommend in a draft opinion to our Judges an article published in a lesser known journal - its true, a prestige school journal does carry more weight. Again, a lower ranked school CAN be cited but the chances are less. Indeed, there are rankings of journals per cites and you will see that judges do tend to cite to better schools' journals. Also (2) even assuming arguendo that T150 journals are "just as good" which may be correct, we are focusing on the academic world which factors in rank. Who will get the prestigious semester visitor appointment - a lecturer at school rank T14 or one from T100? You know the answer. Again as to cites, as federal judges are most usually products of high prestige academia (as are most law clerks who gets selected from T14 or T150?) perhaps there is a bias towards citing for example a T50 school journal over T150 school journals. This is the way it is rank does serve as a proxy for quality rightly or wrongly. Hey lawyer who got the date with the bombshell the guy with the corrola or the one in a Z4? The Z4 is a proxy for quality.

Posted by: rank matters | Jan 30, 2015 3:17:39 AM

I would've thought the first step on the path of sanity would be to stop posting anonymous and often inaccurate comments in the angsting threads. Or doing away with such threads, which contribute almost nothing to sum of information in the world but certainly make lots of people very anxious.

Posted by: dave hoffman | Jan 29, 2015 10:45:53 PM

If people stop treating placement as a proxy for merit the problems you describe mostly vanish. As "Lawyer" indicates we are already close to this solution. Even in the academic world most profs accept that placement is not a merit proxy -due to noisiness of the signal- and if they want to assess someones aptitude they read the article or ask others in the field to read it.

Posted by: Self Correcting | Jan 29, 2015 7:53:46 PM

I think it is interesting that we put so much emphasis on the "ranking" of the law review where an article is published. Professors scurry around playing one publication offer against another, and they agonize about "under-placed" articles hurting their chances for tenure. But, almost nobody in the real legal community knows where a particular law review is "ranked." And based on my years of experience practicing law and working for judges, almost nobody in the real legal community cares about the name of the law review where an article appears. What matters is whether the article is well written and helpful. The judges and lawyers conduct a search in Westlaw or Lexis, find an article, read it, and determine if it has value to the case at hand. If it does, then they cite it. They don't care whether it is published in a "top 25" or a "top 150" law review. The only people who care about that kind of stuff are insecure people in the academy who are looking for a way to validate themselves by saying "I published in a top 25 law review." I am all for legal scholarship, and I think it has great value. But, the quality of an article is in no way determined by the name of the law review in which it appears. And the academy would do well to stop pretending otherwise.

Posted by: Lawyer | Jan 29, 2015 5:45:02 PM

The comments to this entry are closed.