« "Let 'em Play" and Hockey | Main | Signing Off (and Giveaway for IP Prawfs!) »
Tuesday, August 16, 2011
Stanford Law Review’s Peer Review Process
As part of our continuing “law review review” features, I spoke with Andrew Prout, a senior articles editor at the Stanford Law Review, about their peer review process. (For background on the peer review process, check out Brian's overview from last week.)
(1) How long has Stanford participated in the peer review program?
My understanding is that this is the third year of the program (i.e., Vol. 62 started this program).
(2) Do you share reviewers with the other schools, or do you find your own?
We use our own reviewers, although on maybe one or two occasions (when we were under a very tight deadline from another journal's exploding offer), we have reached out to professors who we knew had already reviewed the piece for another school.
(3) About how many articles do you send to be peer-reviewed each cycle (spring/fall)? How are these articles chosen?
This year [as of the spring], we've sent around 40 to 50 articles for peer review, and so far we've accepted 12 articles. Usually, we choose to send an article to peer review after several student readers have shown a very strong interest in the piece. We also turn to peer review when we have certain concerns that we feel reviewers can address, or when we discuss the article at the full committee level and decide that we need more information before we can make an offer.
We almost never send to peer review an article that is over our word limit (30,000 words, including footnotes, excluding appendices).
(4) How do the reviewers give their feedback? Do you use a system of categorization or a ratings system?
Their feedback is holistic. We ask reviewers for their perspective on the article's novelty, usefulness, and accuracy.
(5) Have you thought of formalizing the peer-review feedback and providing an (anonymous) copy to the author after the review?
We know that scholars in other fields can receive feedback from their peer reviewers once the process is complete, and we've considered whether we should do the same. Trouble is, many legal scholars writing about a niche topic are familiar with the same scholars who end up providing us with peer reviews. So even if we sent just the text of the review without any name attached, many could tell who had reviewed their article. This in fact happened at least once before when we used to share just the text of the peer review with authors. Because complete anonymity encourages many of our reviewers to be frank with us, we promise that we will never reveal their review to the authors, and we stick to that promise.
We could formalize the reviews, as you suggest, but that would require quite a bit of editing, and frankly, we don't have enough hours in our days to do that. We also are concerned that using a form or otherwise asking for more standardized responses from our reviewers would deter potential reviewers, most of whom we have to ask to respond within a short timeframe due to exploding offers.
(6) Do you always follow what the peer reviewer suggests? In other words, have you ever accepted an article that was given a poor peer review? Or have you ever rejected an article that got stellar peer reviews?
When the reviewers are nearly unanimous in their praise or disdain for a piece, we almost always follow their lead. But that's not always the case. If anything, we err on the side of rejection, so there have been times when peer reviewers liked an article and we rejected the article anyway, either because we changed our mind about how much we liked the article in the first place, or because the content of the review did not alleviate some concerns about the piece even if the reviewer liked the piece overall.
NB: Last year, we withdrew from ExpressO, so all the articles we receive are submitted directly through our website (www.stanfordlawreview.org).
Posted by Matt Bodie on August 16, 2011 at 08:44 AM in Law Review Review | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef014e8ab15f80970d
Listed below are links to weblogs that reference Stanford Law Review’s Peer Review Process:
Comments
For another approach to peer review in legal scholarship publishing, you may wish to refer to PRSM (the "Peer Reviewed Scholarship Marketplace"): http://www.legalpeerreview.org/index.php. As a consortium of law reviews, PRSM is able, I think, to avoid some of the potential problems presented by a single journal's efforts. Indeed, PRSM addresses a number of concerns raised in this thread. It is worth noting that the Stanford Law Review is also a PRSM member journal, and for good reason. Advancing peer review in the law is an important goal that should be approached in as many different ways as possible until one or more good solutions are found. I applaud the Stanford Law Review for their efforts.
For the sake of full disclosure, I also note that I was the founding Peer Review Editor at the University of South Carolina School of Law, which launched and continues to manage PRSM. You can find links to an essay describing our early experience with peer review at the PRSM site above. You can also find a link there to the foreword Judge Posner wrote for us in 2009.
Posted by: John Zimmer | Sep 19, 2011 2:56:34 PM
I also peer review for social science journals and university presses. I always submit my reviews within the 30 day period. My articles and books also have been promptly reviewed by others when I am the author. Perhaps economics is an outlier. I still say the current law review system in which the author's identity is known doesn't work. It also doesn't work when law review editors receive 2000 submissions per cycle. Hence, the reason for my suggestions. My point is simply the system outlined in the Stanford interview is problematic.
Posted by: anonLawProfPHD | Aug 17, 2011 1:28:45 PM
aLPPhd,
Thanks for your thoughtful response. With respect, I have to continue to disagree about limiting the number of targeted journals for submissions, for the same reasons I gave in my original comment.
I do think the idea of blind submissions is interesting, though I have some doubts as to its workability (probably the biggest one is that authors could easily signal, or just outright provide, their identity in articles' text--that's easy enough to do with "As I have argued elsewhere"-type phrasing).
I have one last note, concerning social science editors' apparent skill at reminding reviewers to finish reports within 30 days. That process is totally automated now, and at least for the vast majority of econ journals, it is virtually meaningless. There are exceptions, but in my experience as a referee, it is entirely possible to ignore multiple reminders from editors without having the paper pulled or suffering any apparent cost. One reason is that referees are poorly paid, if at all, so the only real threat the editor has against them is similar foot-dragging when the referee submits a paper to that same journal in the future. In a given year, I probably referee 5-10 times as many papers as I submit, and for a broad array of journals, so the odds are quite low that such a threat would actually harm me if it were, counterfactually, to be carried out. I do my best not to take too long, on principle, but if there were real costs or benefits, I am sure I would both referee fewer papers and try harder to be quick with those I did referee. I doubt I am unusual in any of these regards among econ people who referee frequently.
Posted by: jonah gelbach | Aug 17, 2011 10:27:01 AM
Mr. Gelbach:
I recommended limiting the number of law reviews to which an author may submit his or her article because the system in which law review editors now find themselves means that they receive far more submissions than they can actually read. As a result, they default to proxies for quality, especially the letterhead of the author. Maybe permitting the author to submit to 2 or 3 journals at a time is too few. Perhaps half a dozen would solve the problem you identify and also the problem I identify. Again, the unlimited submission universe in which law reviews currently operate is unworkable, which explains in part why some law reviews now strongly encourage exclusive submission (a welcome practice, in my opinion). I never said the social science process is perfect. That's why I suggested modifying it a bit. That said, editors of many social science journals are now very good at reminding reviewers to finish their peer evaluations within 30 days.
Posted by: anonLawProfPHD | Aug 17, 2011 7:17:59 AM
In response to anonLawProfPHD, I disagree strongly with this:
"prohibit the author from submitting the article to more than 2 or 3 law reviews at a time for 30 days or so (i.e., not quite the exclusive review demanded by social science journals, but pretty close)."
Whatever the problems with student-edited journals, at least authors know their pieces will be accepted within a short period of time (assuming anyone accepts them). That is an alluring prospect to an author. By contrast, the long lags that take place at social science journals, at least in economics, mean that a paper can take years to be accepted (never mind actually published). From my experience as both an author and (a much more frequent) referee for economics journals, I do not think it's an accident that econ journals both forbid multiple submission and also take forever.
Posted by: jonah gelbach | Aug 16, 2011 10:14:31 PM
This isn't "peer review" in the social science sense of the process, especially given how few articles are actually granted peer review by the student gatekeepers. I strongly suspect that the letterhead bias that has long impacted law review decisionmaking plays a strong role in determining which articles are sent out for peer review. Moreover, feedback _prior to publication_ is at the heart of real peer review. I think the only way to fix these problems is (1) to forbid the student gatekeepers from knowing the author's name and institutional affilation until _after_ the publication decision is made and (2) to prohibit the author from submitting the article to more than 2 or 3 law reviews at a time for 30 days or so (i.e., not quite the exclusive review demanded by social science journals, but pretty close).
Posted by: anonLawProfPHD | Aug 16, 2011 1:59:18 PM
Scholars can still benefit from the process when the article is accepted and published. We are working on finding a way to streamline giving feedback to all authors who are peer reviewed, but the peer reviews nevertheless guide our substantive edit comments for the articles we do publish. If you have any suggestions, feel free to contact us.
Posted by: Andrew Prout | Aug 16, 2011 12:28:48 PM
Matt, I'm glad you did this interview.
As an author, I take there to be two salient pieces of information: roughly and only 1/4 of the articles that make it to peer review get accepted, which means that authors should know that going to peer review at Stanford is still a relatively weak signal vis-a-vis other options that might be alive, and two, authors don't really benefit from this peer review process in terms of the way their scholarship can improve through this process. When Polsky and I went "up" for SLR peer review on our piece, Taxing Punitive Damages, we had been told we were going to get some form of redacted comments from the readers. The editors never delivered those comments, even though they claimed they would just after the height of the submission season when things slowed down but while we could still have revised had we been given feedback.
So, having peer review might be a net benefit for Stanford LR (if it's conducing to better quality publications in SLR generally), but it's not an benefit for the general class of authors. Maybe that's an overstatement: after all, I suppose someone might think of an acceptance at Stanford LR as equivalent to a peer-review journal now, but that strikes me as somewhat implausible given the gappiness of it all. In any event, and as the joke goes, some of my best friends publish at the Stanford Law Review...so they can't be all that bad, right? (Enter Kate Litvak for ritual spanking.)
Posted by: Dan Markel | Aug 16, 2011 9:34:32 AM
The comments to this entry are closed.