« More on Newbie Clerks vs. Old-Timers | Main | San Francisco Exceptionalism »

Thursday, February 23, 2006

The peer-review conundrum

Michael Heise over at the ELS blog asks a provocative question that I encourage other readers to answer: “[Do law professors] see (or perceive) shifts in overall law school culture or tenure expectations germane to scholarly publication options[?].  If so, when it comes to decisions about where to place scholarship how are folks assessing such options as traditional student-edited law reviews, faculty-edited peer-reviewed journals (in law or other fields), and university press books?”

It would be interesting to know the answer to this question. I suspect the norms vary pretty dramatically, even among elite schools.  For example, publishing in Top 10 law journals generally helps one's career.  But schools like Cornell, Northwestern, and Texas are trying to build empirical powerhouses.  Thus, insofar as peer-review becomes the preferred outlet at those law schools, I suspect that those expectations will influence the publication norms of the wider legal academy, especially with empirical work. 

I had this same question on my mind several months ago when I asked a prominent empiricist (an eminent social scientist who is now on the faculty of a Top 15 law school) whether it was better to publish empirical work in peer-reviewed journals rather than law journals.  [I am not naming names because the question was not asked in the context of posting the answer on a blog.]  I fully expected to get an answer to the affirmative.  But much to my surprise, she/he said that extensive workshopping was as good or better for quality control than the peer-review process.  So to his/her mind, student-edited journals were just fine.  The burden of quality control was on the author.

I think the issue here comes down to a signaling heuristic:  If you get published in JELS or JLS or ALER, then distinguished people in the field said it was good enough to publish.  From the perspective of a junior faculty person, that is pretty attractive.  There will be a presumption of quality that a negative review in your tenure file is less likely to rebut. That said, it is another matter entirely to believe that the peer-review process catches all significant mistakes.  The economics of refereeing a journal article are replete with agency costs, and I have collected anecdotes on this topic over the years.  But that is the topic for a separate post.

By the way, PrawfsBlawg alumni Joelle Moreno had some interesting thoughts on this topic a few months ago.

Posted by Bill Henderson on February 23, 2006 at 04:43 PM in Peer-Reviewed Journals | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef00d834ba3fcb69e2

Listed below are links to weblogs that reference The peer-review conundrum :

Comments

Bill,

Very interesting question. I wonder whether workshopping can sufficiently replicate the peer review process? I've gotten some really, really terrific feedback from workshops. But it's often of a vastly different kind of criticism than I get from peer reviews--which are almost always anonymous and, hence, likely to be more honest than workshop feedback. (I've also had some remarkably candid workshop feedback, but that's a story for another time.) Second, the typical law school workshop participant won't know much about the particular topic being presented. I've heard talks in recent memory on curricular reform in legal education, federal personal income tax policy, late nineteenth-century constitutional history, progressive-era criminal prosecutions, an empirical study of administrative agency ossification, voting coalitions in the Supreme Court in the last forty years, the gendered nature of judicial decisions regarding slavery, the development of professional responsibility codes, recent trends in clemency in capital punishment cases, and, well, the list could go on for a little while yet. But you get the picture. I was able to offer the kind of sustained critique from an informed perspective that you'd expect from a peer reviewer in only one (or maybe two) of those cases. And I suspect my experience is not atypical of law faculty. Third, I think workshops are becoming less about getting feedback on articles and books as they're in gestation and more about showing off completed work. I may be wrong about this, but I (interestingly enough) was just talking about this phenomenon with a colleague at another school yesterday.

In short, I think it's hard to substitute the kind of serious, searching reading that you get from peer reviewers through the usual faculty workshop process. You might, of course, simulate it through presentation to specialized workshops. (Though for specialized workshops, presenters may feel an even greater requirement that they present finished work.) However, even in those cases, I think one is unlikely to get the sustained reading that major university presses put manuscripts they're considering through. And for the Law and History Review (the peer-reviewed journal I know best), an article is typically read by the editor-in-chief, as well as by at least three peer reviewers who are experts not just in legal history but in the area of article under review. I suspect that the process is similarly rigorous at other major peer reviewed journals.

Posted by: Alfred L. Brophy | Feb 23, 2006 9:22:43 PM

Al,

Your experience with the peer-review process is a solid, informed data point that I respect. And I certainly agree that journal referees are much better positioned to offer critical feedback than a general workshop audience.

But one thing I don't understand is the theory that anonymity necessarily provides better feedback. Assuming the typical reviewer or workshop attendee has tenure, why does he or she need the additional shield of anonymity to provide serious, candid advice (and here I asking myself how I ought to behave)? The point is to advance knowledge, and good criticism does exactly that.

I know the analogy to the peer-review process is far from perfect, but anonymous posts on the Internet are generally more vitriolic and petty and off-point than non-anonymous posters. When you have to own your words, you are more careful in choosing them. (Bloggers learn this quickly.) On the other hand, if anonymity produces a mixed bag of valid criticism and self-indulgent digressions, then the author can sort that out during the revise and resubmit process, and the goal of better work product is achieved. So it might be a wash.

I need a few more years of experience to make up my mind on the pros and cons of blind review.

Posted by: William Henderson | Feb 24, 2006 1:00:40 AM

William,

good post, but I've got to defend peer-review.

Remember, peer-review isn't anonymous. It is just blinded between author and reviewer. The editor knows both, and reads paper and reviews. Vitriolic and sophomoric criticisms - as well as reviews that fail to pick up on weaknesses - will reflect badly on the reviewer. That is, if one of the other reviewers and/or the editor picks up on a weakness that the other reviewer didn't, then it looks bad. Thus, there's an incentive to be rigorous but not obtuse. Of course there are bad reviewers as well, but it is easier for an editor to pick up on which have merit than an author; after all, we're often slow to realize our mistakes.

Finally, workshops and peer-review aren't an either/or kind of thing. The best papers go through both. The burden of peer review actually provides _impetus_ for workshops rather than a substitute.

A couple more quick points. Many reviewers don't have tenure, and many are not yet full professors, but I don't think that career worries get in the way of candidness very often. Rather, it is often a small world, and you can review someone's work you know. Anonymity allows you to criticize friends without jeopardizing the friendship - remember, a recommending against publication affects promotion chances, you don't want to saddle yourself with that. It also allows you to read papers without pre-judging the author; it's harder to read things openly when you think an author's prior work sucks, and it's hard to read things critically when you think the author's prior work is brilliant. When I diagree with, say, Hume, I tend to think I'm missing something, not him. When I agree with, say, Derrida, I tend to think that we're both missing something. :)

Posted by: untenured | Feb 24, 2006 8:28:46 AM

Bill: the job of a referee is not only to provide feedback, but to help editors decide whether an article should be published. Killing someone's publication prospects is a major blow; few people would be willing to deliver that blow on the merits without the shield of anonymity. Tenure protects your job, but not your relationships with colleagues, not your future conference invitations, chances of co-authorship, prospects of lateral moves, reviews of your grant applications, referrals for consulting, and (of course) the fate of own submissions to peer-reviewed journals. It's bad enough that real anonymity of _authors_ is hard to maintain; we should at least strive to preserve the independence of referees.

P.S. Thanks for plugging Texas on "empirical powerhouse" aspiration! We've got four people with empirical papers at ALEA this year -- few schools can beat that.

Posted by: Kate Litvak | Feb 24, 2006 8:59:30 AM

Well, this thread at least proves (to my mind anyway) that blogs are not a complete distraction. Kate and untenured and Al marshaled some terrific, forceful arguments in favor of anonymous refereeing. I am persuaded. However, I still believe that hard, candid criticism in workshops is a worthy aspirational norm. Insofar as we bite our tongues to curry favor with colleagues, we are shirking at our jobs. It is (or ought to be) all about the ideas.

Posted by: William Henderson | Feb 24, 2006 9:34:32 AM

Bill,

Very intersting question about anonymity--and I like the analogy to the net.

I meant only to suggest that peer review provides feedback that can't easily be replicated through workshopping. But as to anonymity, I was thinking it makes it easier to give honest feedback. I think it's generally harder to tell people in a constructive manner in public about the problems with a work. The constraints of polite society constrain certain kinds of feedback. (This is an important reason why published book reviews are often discounted by hiring committees.) Come to think of it, there's probably a pretty interesting paper to write on social norms and faculty workshops....

And maybe this is just me, but I take quite seriously the anonymous reviews that I've received. There's something about the solemnity of a reader's report that causes me to give it great weight. But this also raises an important question: should peer be double-blind (as Law and History and the other history journals I'm familiar with do) or single-blind, as is more common with university presses?

Posted by: Alfred L. Brophy | Feb 24, 2006 6:04:36 PM

Post a comment