« How much time should a job offer be open to avoid being thought of as "exploding?" | Main | Why Declining Marginal Utility Matters »
Monday, December 07, 2009
Judging Scholarship, Or, Would You Kill For Blind Review?
Now that we’re in the middle of the law faculty hiring season, and so many of you out there are busily and conscientiously trying to gauge the comparative quality of candidates’ scholarship, here’s a plea: don’t treat all law review placements the same. Until, that is, the day that law reviews implement (and effectively enforce) a double-blind law review selection process (don’t hold your breath). Double-blind review means that the submitting author does not know the identity of those making the selection decision, and likewise reviewers do not know the identity of the author when deciding on publication offers.
There’s actually been a surprising amount of scholarship on the subject, and it all reaches the same conclusion: law review publication offers generally aren’t determined by quality, but by various proxies to quality that can be grouped under the general rubric of “letterhead bias” (such as the professor’s name, the law school at which the professor teaches, or where the professor has previously published). You’ll find a good, recent piece of this scholarship here (note, the citation is wrong; it actually starts on page 175). It contains citations to a lot more of the scholarship in this area in case you’re interested. Now, it may well be that letterhead bias is a roughly accurate barometer of quality, but it is at best only that, and it’s probably worse. (Particularly compelling is James Lindgren’s reported "nonscientific study" (see here, pg. 610) in which he submitted the identical article on Chicago-Kent and University of Chicago letterhead. His best offers were from Arizona, versus Penn and Northwestern, respectively.)
So here’s a plea: when weighing the quality of candidates’ law review placements, take into account the law school from which the candidates submitted. (Says me, writing from a fourth-tier school.)
There seems widespread dissatisfaction with this state of affairs, yet inertia continues to reign. Which is frustrating because this problem could (easily?) be fixed if only law reviews would adopt double-blind review, as recently advocated by Jonathan Gingerich, A Call for Blind Review: Student Edited Law Reviews and Bias, 59 J. Legal Educ. 269 (2009). If, say, the Top 20 law reviews got together and jointly adopted double-blind review (similar to the effort to shorten article length), would that be enough for the ethic to spread throughout the whole law review/journal world? (Note, Yale L.J. purports to already have it, see the first link above.)
Or, is there some other, better alternative to double-blind review?
Posted by Fabio Arcila on December 7, 2009 at 09:12 AM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef0120a71f3f83970b
Listed below are links to weblogs that reference Judging Scholarship, Or, Would You Kill For Blind Review?:
Comments
My paper from the Journal of Legal Education is available at http://ssrn.com/abstract=1524538.
Posted by: Jonathan Gingerich | Dec 16, 2009 6:48:26 PM
I am not sure that I agree with Matt that "in order to attain diversity... it does mean being conscious of the identity of the author while reviewing."
I suppose it depends on which elements of diversity you are trying to maximize. An article could be blind to letterhead bias, but still reveal race, for example.
Posted by: ctr | Dec 16, 2009 3:16:45 PM
As a current law review editor, I have to agree with John Pfaff's comment that blind review would have to be coupled with peer review, and perhaps with an entirely different system of article submission.
With the current bimodal calendar of article submission, we have approximately 30 days per cycle to review articles, and we receive approximately 1000 submissions per cycle. This works out to reviewing two articles per Articles Editor per day for a month. Unfortunately, this means—quite frankly—that it is impossible to review every piece on its merits, and the pieces we do review on the merits have to be reviewed _quickly_. Not only would our pipeline back up quickly if we did not breeze through pieces, but any good pieces in the pipeline would be taken by other journals.
In most areas of law—and in almost all interdisciplinary areas—students have no expertise. Combine this with our time constraints and, for better or worse, we have to resort to proxies, to some degree. Whether an author actually has any expertise in an area, whether they're published in any peer review journals, whether they're an authority in some field. There are dozens of fields of law, and when you combine these with psychology, economics, statistics, or any number of other fields, the chances that a board of 16 students can evaluate the work on its merits alone is, to some degree, laughable.
Additionally, because we are students, we do not know the "state of the art" in most fields. We don't know if an idea is new or not. And authors try to take advantage of this. Many authors will only very slightly alter an already-published article and resubmit it to other journals. You would be amazed how often this happens. My journal has accepted or come close to accepting articles, only to realize that essentially the exact same article was published somewhere else a year before. Granted our preemption checking should take care of this, but I return to my earlier point about time—we just don't have any. The quickest way to find preempting articles is to check authors' CVs.
Finally, whether or not you agree with them, some people actually make arguments _against_ blind review. It is, I think, a well-known phenomenon that the body of articles submitted to Expresso does not reflect the demographics of legal scholars as a whole. Some students feel a responsibility to improve the diversity of both the scholarship and the scholars published in law journals. This does not mean taking "inferior" scholarship in order to attain diversity (there are so many articles submitted that one can always find quality scholarship), but it does mean being conscious of the identity of the author while reviewing. Again, this is a very contested point on which reasonable people disagree, but it is out there.
Posted by: Matt S. | Dec 10, 2009 9:50:53 AM
Top 20 LR editor here, on the submissions committee. I agree with most of the commenters about blind review; there is no reason we need to be bombarded with the writer's CV and school affiliation before reading the article. (But we editors are maybe a little less naive than you think—before we selected a single article, we (the submissions committee) brought up the "letterhead bias" issue and to address it, committed ourselves to publishing at least one or two articles from professors at non-elite law schools. Not perfect, but it's a start.
Regarding students' ability to judge the quality of law review articles, sure, there are some obvious defects in the process. But a shocking number of submissions contain some or all of the following errors that student editors are competent in judging: general lack of citations, lack of pin cites in citations, relying on the same five or so sources throughout the entire article, terrible/incomprehensible writing, an unreasonable number of grammatical/typing errors, and logical fallacies/weak arguments.
Peer reviewers will be able to better judge complex/highly specialized work, but they also get to play gatekeeper, and there's a pretty robust and nasty history of the gatekeepers rejecting reasonable arguments that go against the reviewer's preferred orthodoxy. Student reviewers allow for more diversity of ideas/viewpoints. This must be considered when choosing between keeping our current system and changing to a peer review system.
While there are real problems, no system is perfect. Blind review is the easiest step and I'm in favor of taking it. Anyway, back to studying!
Posted by: GU | Dec 9, 2009 2:02:30 PM
Stuart, in your example, if the author knows who the reviewer is, the reviewer may be less inclined to be brutally honest about the piece. Scene at AALS meeting or conference. Author: "Nice you to meet you, I'm X." Reviewer: "Oh, nice to meet you, I'm Y." Author: "You son of a bitch; you trashed my essay."
Think of it this way:
Single blind (reviewer knows who author is, but not vice versa) - reviewer can be brutally honest, but might be biased by factors other than the merits of the piece.
Single blind (author knows who reviewer is, but not vice versa) - reviewer won't be biased, but may hold back because her identity is known to the author.
Double blind seems to ensure both candor and lack of bias.
Posted by: Jeff Lipshaw | Dec 9, 2009 1:43:20 PM
Ah. But in that case, the "double" part of "double-blind" isn't an improvement over just "blind." Again, in the medical context, not letting either the doctor or patient know what the treatment is means that neither person can be contaminated by the placebo effect. But with legal scholarship, it shouldn't matter one way or the other if the author knows who the reviewer is -- the reviewer's task will be completely unaffected (unless we're supposing some behind-the-scenes lobbying). Right?
Posted by: Stuart Buck | Dec 9, 2009 1:28:11 PM
Double blind is that the reviewer doesn't know who the author is, and the author doesn't know who the reviewer is.
You can have single blind review, in which the reviewer knows who the author is, but not vice versa (I have been on both sides of that).
Posted by: Jeff Lipshaw | Dec 9, 2009 11:31:01 AM
Hopefully not a dumb question, but what's double-blind review? Blind review I understand, but double? In the medical field, that would mean that both the doctor and the patient don't know if the patient is receiving the actual treatment or a placebo. What's the equivalent for legal scholarship -- the scholar doesn't know whether he or she wrote the article?
Posted by: Stuart Buck | Dec 9, 2009 9:12:58 AM
Law journals, I must say, are a bit of a joke. I'm an articles editor this year, which means I read a freakin' huge number of articles last spring, and picked five for publication. For me, letterhead bias did not exist, because one of the first articles I read was from some prof. at Yale, and it was gawdawful. After that I quit bothering with the letterhead and just got to the meat of the article.
After about two weeks, I had so many expedite requests that those were the only articles I read (which means I was basically using the Podunk Auto Diesel College and Law School Law Review as my first reader). The articles about subjects I had studied, or about history (in which I have an MA) I thought I could judge fairly well. The articles about patents or corporations or estate law, they were completely beyond me. I had no clue if they were good not, except that some were less mind-numbingly dull than others.
I did find out one thing about myself, though; I really liked articles that dealt with the intersections of medicine and the law. If you'd have asked me before the articles process if I had any interest in this field, I'd have said no, but I found it really fascinating.
So, on a personal growth level, law review was good for me; for the world of legal scholarship, the system appears to me to be totally rotten and, frankly, ridiculous.
Posted by: sehi | Dec 8, 2009 10:31:37 PM
Perhaps I should have been clearer on one aspect of my dismay. Bluntly, law review members who have "nonlaw expertise" often have enough expertise to reject certain varieties of "bad" material; for example, failure to even cite the two or three leading authorities on a matter revolving around developmental biology -- let alone take account of their views through secondary sources -- would be pretty damning to someone who had not just taken a developmental biology lab, but worked in a lab researching in that very area. That's not quite the same thing as I think Mr Pfaff is suggesting with the peer review process, which it seems to concern more the acceptance than the rejection side of things.
That said, peer review is no panacaea; I lost my first name to peer-reviewed journals, and things have gone downhill ever since. Further, since law writing is so data-independent (even in so-called "empirical" studies of legal contexts... which aren't even empirical), there is a real danger that law journals will evolve toward what has happened in literary studies, and that would be very, very bad indeed. I would support more consultation, but not universal (or even majority) peer review.
Posted by: C.E. Petit | Dec 7, 2009 2:58:55 PM
C.E. Petit, your example is interesting, but in the end I'm not wholly convinced. Here's why. The four of you had four very interesting backgrounds. But law covers far more than four disciplines. No matter what your training, there had to be papers on subject matters about which you knew little (the science background is great, for example, but how many of you knew con law or admin law deeply?). Peer review allows a journal to farm out every article to those most qualified to evaluate it.
In other words, no law review board, no matter how qualified, is ever really qualified to engage in the review process on its own. Just like no economics editorial board is. If the EIC of a major econ journal is an empirical macroeconomist, he likely shouldn't be deciding whether a theoretical microecon game theory paper is at the top of the field. Thus, even before we get to the 2L/3L point, the idea of a single board making all the calls doesn't hold up.
But by your own admission, your board was unique. I think that on average the interdisciplinary makeup of law review boards is rarely of that quality. And regardless, no law review board, no matter how well qualified, has the depth of experience for law-related issues that law professors have. By definition, law profs have been studying the subject longer and--God, I hope--understand the issues, the open questions, and the methodological approaches better.
And I'm not arguing that legal peer review should be confined to law professors--if someone is writing on law and genetics, by all means recruit a geneticist to decide whether the science is right. But I just can't see how relying on students can ever be superior to relying on more-fully trained professionals.
Posted by: John Pfaff | Dec 7, 2009 12:23:08 PM
I have to express some caution, and perhaps dismay, regarding Mr Pfaff's comment: His assumption that 2Ls and 3Ls necessarily "lack the depth of understanding needed to make good calls." This reflects an unconscious bias in the legal academy: That law review members (and, for that matter, members of the legal academy) went straight from undergraduate education in a law-cognate discipline to law school, with no intervening life or academic experience.
My own experience as a flagship law review articles editor (top-tier school) illustrates the converse. We had a "3+1" system: Initial review by the three Articles Editors, with resort to the EIC if there was a 2-1 split. This was at the beginning of the "interdisciplinarity means more than just law and economics" wave. We definitely constituted an atypical set of selectors: Three of the four of us had hard-science backgrounds; a different three of the four of us had advanced degrees; two of us were over thirty; and a different two of us had extensive editorial experience with "real" publishing operations.
Without naming any names, we rejected a number of articles eminating from top-tier schools on the grounds of bad science (or equivalent doctrine); we accepted and published several articles from lower-ranked sources that appear to have played a role in getting higher-ranked appointments for their authors (and continue to be cited, an academic eternity later), and lost only one offer to a higher-prestige journal. I won't claim that we were double-blind, but we did make a concerted effort to RTFM (read the frakking manuscript) before looking at anything accompanying it... like a letter or (disfavored, at least with us!) CV. So it can be done; it need not, and perhaps should not, involve peer review confined to law professors.
Posted by: C.E. Petit | Dec 7, 2009 11:49:19 AM
Actually, I'm not sure whether the volume of work published is an argument for or against the current system; my priors are that it may be an argument against. The wheat-to-chaff ratio is, I think, much worse in law than in other fields, partly due to the lack of effective review, but also due to the sheer volume of work out there. With so many journals looking to fill so many pages, anything can eventually get published somewhere. I think legal scholarship would be better off with fewer journals and fewer--or/and substantially shorter--articles.
Posted by: John Pfaff | Dec 7, 2009 11:28:25 AM
Josh: thank for chiming in and confirming Im not just crazy. Or merely cranky.
Posted by: Fabio Arcila | Dec 7, 2009 11:09:22 AM
John: I totally agree with you. I think this gets us to the crux of one of the primary arguments in favor of the current law review system: though quality might be spottier, a lot more scholarship gets published. To a meaningful extent, the debate is about whether to sort on the front end (through a more reliable mechanism like peer review), or on the back end (through community recognition of good scholarship after publication). The problem with the current back end route is that it takes much longer for newer faculty to get recognition, a delay that is exacerbated by letterhead bias. The problem with the front end is less scholarship, and a heavy up-front work investment by reviewers. Unlike other fields, law schools are not designed to reward the time investment in being a reviewer, which is much different from other fields, where being a reviewer is highly prestigious. That would have to change.
Posted by: Fabio Arcila | Dec 7, 2009 11:08:42 AM
As a former Law Review editor I know all too well how important the letterhead and C.V. of a prospective author are. Articles submitted by professors at lower ranked schools, or non-professors would be given less consideration than professors at better schools. I personally abhorred this policy, but as a matter of efficiency and expediency, it was a necessary sorting mechanism. Even if it is a weak proxy for quality, it is a stronger proxy for getting citations. An established prof is more likely to getter better cites, thus making our journal better.
A double-blind would also help non-Professors submit. As a recent law school grad and current law clerk, I know the bias my manuscripts receive. Fortunately, I've been lucky with my placements, but I am pretty sure if the same manuscripts were submitted with better letterhead, the placements would have likely been better. Some more thoughts here http://joshblackman.com/blog/?p=2951
Posted by: Josh Blackman | Dec 7, 2009 11:01:01 AM
Fabio, I think double-blind is a great idea, but I think the big stumbling block is that it needs to be joined with peer review. I would go further than Jeff and argue that both interdisciplinary and doctrinal works demands peer review--2Ls and 3Ls simply lack the depth of understanding needed to make good calls. In fact, I think it is the student-ness more than lack of blindness that is the real problem.
But peer review requires a major upheaval. Referee reports take a long time--I spend probably 10 or 12 hours on each report, often reading a paper three or four times before I'm done. So simultaneous submission would have to go away, since no one would invest the necessary time if racing against a clock. And thus so too would expediting. With peer review, you have to look at your paper and really ask "just how good is this? where does it truly belong?" (This is actually an argument FOR the long delays that Jeff laments--the Am. Econ. Rev., at least when I was in grad school, said that the average time to *rejection* was about 18 months, which was clearly designed in part to prevent people with unqualified papers from trying to hit the AER "jackpot" and thus overwhelm the high-quality referees that journal could enlist.)
Also, peer reviewing demands that law faculty play a much bigger, and often uncompensated, role in the reviewing process. Law profs I've talked to have often expressed displeasure at the idea of "wasting" time refereeing, so I think it is a cultural problem. Coming from a peer-reviewed field, I always viewed it simply as part of my job. But there is a self-interested appeal here, too--as a referee, I get to act as a gatekeeper, to play an important role in deciding what is considered "high quality" in my field. Framed this way, perhaps more law profs would be willing to spend the necessary time to really review papers.
Posted by: John Pfaff | Dec 7, 2009 10:59:25 AM
Mark: you are right. Navigating the Law Review Selection Process, to which I linked, conducted a survey of law reviews/journals, and found significant frustration with losing articles to the expedite process. Interestingly, it even found this frustration at top tier law reviews, with student editors (who Im guessing probably relied heavily upon the U.S. News law school rankings when deciding where to enroll) amazed that authors would abandon an offer for a placement deemed to be only marginally better, possibly by as few as 1 or 2 U.S. News spots. So frustration certainly exists on both sides of the selection process.
Posted by: Fabio Arcila | Dec 7, 2009 10:38:25 AM
James-hallelujah. I actually think there is some hope for your suggestion of not treating article placement as a proxy for quality. It seems like every year I hear more complaints about poor quality articles being placed in higher ranked journals, so this frustration should lead to more merit-based consideration. That being said, using placement as a proxy has a tremendous amount of inertia. It takes a lot of time to actually read and judge scholarship. Appointment committees, for example, face quite a strain when deciding on scholarship, and the attraction of relying at least to some extent on placement is both natural and undeniable.
Posted by: Fabio Arcila | Dec 7, 2009 10:35:12 AM
Thanks, Fabio - I was familiar with expediting up, and as a pseudo-auction mechanism I can see how it improves the appropriateness of placement. But it does seem to compromise the decision process at the top tier reviews (as you say, letterhead bias becomes an almost-necessary screening device), and it must also frustrate the lower tier reviews, the editors of which may get very excited at receiving a great paper, only to lose out to a better review.
Posted by: Mark D. White | Dec 7, 2009 10:29:51 AM
Or, is there some other, better alternative to double-blind review?
Yes. Readers of legal scholarship could stop treating journal placement as a proxy for quality, and could actually base their judgments on the articles themselves.
Posted by: James Grimmelmann | Dec 7, 2009 10:28:50 AM
Fabio - I'm glad that our students recognized a good piece when they saw one. ;)
Posted by: Michael Risch | Dec 7, 2009 10:24:31 AM
You are correct that the conventions for law review submissions are unlike those in the rest of academia. Though they cause other disciplines to look askance at legal scholarship, or at least its publication selection process, they are an accepted part of legal academia (though not happily accepted even in legal academia). Simultaneous multiple submissions are part of this accepted practice, and indeed for the vast majority of law profs it is a requirement, as it goes hand-in-hand with another weird, but accepted, convention: expediting up. Once one publication offer arrives, it is usually open for a period of time, and the accepted convention allows the author to seek expedited review at other (higher standing, more prestigious) law reviews. Through this process, an author can climb the ladder to obtain a higher-ranked offer. Given that the multiple submissions practice is joined with the possibility of expediting up, it is crucial to the appropriate placement and evaluation of manuscripts. Essentially, higher ranked law reviews have to deal with a higher level of submissions that lower ranked reviews. Thus, they use particular letterhead bias measures to decide where to start their review, while relying upon lower ranked reviews to identify other promising manuscripts, which will be brought to their attention through expedite requests. Authors who do not enjoy letterhead bias advantages would be left out of the selection process at higher ranked reviews were it not for the possibility of multiple submissions and expedited review. Not a great system, but the one we have.
Here is some anecdotal evidence. I write from a fourth-tier school. The best placements I have received have been the result of expediting up through about 15 publication offers per manuscript. On one of those manuscripts, the best offer I received when submitting from my home institution was from West Virginia Law Review. A fine offer, but not a top tier offer. I felt strongly that the manuscript deserved a better placement, so I held it, and resubmitted the virtually identical manuscript later the same calendar year on Fordham University Law School letterhead (a top tier law school), where I was visiting for the academic year. It then got placed in the Boston College Law Review, a top tier journal. So, in essence, my personal experience is consistent with James Lindgrens nonscientific study.
Posted by: Fabio Arcila | Dec 7, 2009 10:05:28 AM
I was asked to do a blind review of a piece for the Stanford Law Review last year (ironic, by the way, because I suspect that my own submission would be subject to the letterhead bias). So it's more than just Yale.
As to peer review, everybody knows what the downsides are: (a) length of time, and (b) the possibility that you will get back "revise and resubmit" or "rejected." Like many aspects of the legal academy, the law review system is an anachronism. Back when legal scholarship consisted of writing treatises, and law review articles dealt not only with doctrine, but often the real world application of doctrine (if you want to weep, read Samuel Williston's sad tirade against the adoption of the UCC in the Harvard Law Review back in the 1950s), student-edited law reviews may have actually made sense. If you are writing doctrinal material, it seems to me that the old system is still fine. If, however, one is writing cross-disciplinary, and particularly without an advanced degree in the other discipline, one ought to consider the problem of perceived dilettantism, and think seriously about submitting to blind peer review. When somebody asks in an interview, "what makes you think you have any business writing on X?" (or words to that effect), pointing to your publication in a blind peer reviewed journal is a pretty good answer.
Posted by: Jeff Lipshaw | Dec 7, 2009 9:53:19 AM
Please excuse my ignorance, but you seem to be pretty conversant in this literature (and thank you for the links) - what seems to be the consensus feeling about simultaneous multiple submissions, which, as a convention, seems unique to law reviews (though it is seen very occasionally in peer-reviewed journals in various fields)? Is this considered to have a positive or negative effect on appropriate placements, evaluations of manuscripts, etc.?
Posted by: Mark D. White | Dec 7, 2009 9:40:34 AM
The comments to this entry are closed.