PrawfsBlawg

What is the Future of Empirical Legal Scholarship?

First, thanks to Dan and everyone else at Prawsblog for inviting me to post here this month. I'm really looking forward to it.

I found Jonathan Simon's recent posts about the future of empirical legal scholarship quite interesting. On the one hand, as someone in the ELS field, I found his general optimism about its future uplifting. But I'm not sure I share it, although my concern is more internal to the field itself than external. I'm going to be writing about this a lot this month, so I thought I'd use my first post to just lay out my basic concerns.

Jonathan focuses on trends outside of ELS: cultural and attitudinal shifts within the law as a whole, and more global changes in, say, economic conditions. And at one level I think he is right--the decreased cost of empirical work, the PhD-ification of the law, the general quantitative/actuarial turn we have witnessed over the past few decades all suggest ELS is here to stay. But there are deep internal problems not just with ELS, but with the empirical social sciences more generally, that threaten their future. I will just touch on the major points here, and I will return to all of these issues in the days ahead.

To appreciate the problem, it is first necessary to note a major technological revolution that has taken place during the past three decades. The rise of the computer cannot be overstated. Empirical work that I can do in ten minutes sitting at my desk would have been breathtakingly hard to do twenty-five years ago and literally impossible fifty years ago. The advances have been both in terms of hardware (computing power and storage) and software (user-friendly statistics packages). The result is that anyone can do empirical work today. This is not necessarily a good thing.

So what are the problems we face?

1. An explosion in empirical work. More empirical work is, at some level, a good thing: how can we decide what the best policy is without data? But the explosion in output has been matched by an explosion in the variation in quality (partly because user-friendly software allows people with little training to design empirical projects). The good work has never been better, and the bad work has never been worse. It could very well be that average quality has declined. Some of the bad work comes from honest errors, but some of it comes from cynical manipulation.

2. A bad philosophy of science. Social scientists cling to the idea that we are following in the footsteps of Sir Karl Popper, proposing hypotheses and then rejecting them. We are not. We never have. This is clear in any empirical paper: once the analyst calculates the point estimate, he draws implications from it ("My coefficient of 0.4 implies that a 1% increase in rainfall in Mongolia leads to a 0.4% increase in drug arrests in Brooklyn"). This is not falsification, which only allows him to say "I have rejected 0%." Social science theory cannot produce the type of precise numerical hypothesis that falsification demands. We are trying to estimate an effect, which is inductive.

3. Limited tools for dealing with induction. Induction requires an overview of an entire empirical literature. Fields like medicine and epidemiology have started to develop rigorous methods for drawing these types of inferences. As far as I can tell, there has been no work in this direction in the social sciences, including ELS, of any sort. This is partly the result of Problem 2: such overviews would be unnecessary were we actually in a falsificationist world, since all it takes is one black swan to refute the hypothesis that all swans are white.

As a result of these three problems, we produce empirical knowledge quite poorly. To reuse a joke I've made before and will likely make again at least a dozen times this month, Newton's Third Law roughly holds: for every empirical finding there is an opposite (though not necessarily equal) finding. 

With more and more studies coming down the pike, and with little to no work being done to figure out how to separate the wheat from the chaff, ELS could defeat itself. If it is possible to find any result in the literature and no way to separate out what is credible and what is not, empirical research becomes effectively useless. (This only exacerbates the problem identified by Don Braman, Dan Kahan and others that people choose the results that align with their prior beliefs rather than adjusting these beliefs in light of new data.)

So what is the solution? Empirical research in the social sciences needs to adopt a more evidence-based approach. We need to develop a clear definition of what constitutes "good" and "bad" methodological design, and we have to create objective guidelines to make these assessments. We have to abolish string cites, especially of the "on the one hand, on the other hand" type, and replace them with rigorous systematic reviews of the literature.

Of course, these guidelines and reviews are challenging to develop for the methodologically straight-forward randomized clinical trial that medicine relies on. In the social sciences, which are often forced to use observational data, the challenge will be all the greater. But, as I'll argue later this month, the rewards will be all the greater as well.

The use of systematic reviews is particularly important in the law, for at least reasons:

1. Inadequate screening. Peer review is no panacea by any means, but it provides a good front line of defense against bad empirical work. We lack that protection in the law. There are some peer reviewed journals, but not many. And the form of peer review that Matt Bodie talked about for law reviews recently isn't enough. The risk of bad work slipping through is great.

The diversity of law school faculties, usually a strength, is here a potential problem. Even theoretical economists have several years of statistics, so everyone who reads an economics journal has the tools to identify a wide range of errors. But many members of law school faculties have little to no statistical training, making it harder for them to know which studies to dismiss as flawed.

2. Growing importance of empirical evidence. Courts rely on empirical evidence more and more. And while disgust with how complex scientific evidence is used in the courtroom has been with us since the 1700s if not earlier, the problem is only going to grow substantially worse in the years ahead. Neither Daubert nor Frye are capable of handling the evidentiary demands that courts increasingly face.

Given that my goal here was just to touch on what I want to talk about in the weeks to come, I think I'll stop here. This is an issue I've been thinking about for a while now, and I am looking forward to seeing people's thoughts this.

Justice Scalia's One-Way Ratchet: Congress and Federal Habeas Jurisdiction

I've just posted to SSRN a draft of a new (and short) essay that is to be published later this summer by The Green Bag. The essay, titled "The Riddle of the One-Way Ratchet: Habeas Corpus and the District of Columbia," tries to shed light on a missing piece of the ever-ongoing debate concerning Congress's power over the habeas corpus jurisdiction of the Article III courts.

To spoil some of the fun, the essay's central argument is that statutes such as the Military Commissions Act are constitutionally problematic entirely because there are no other courts in which detainees may otherwise bring habeas petitions. Ex parte Bollman arguably prevents detainees from going straight to the Supreme Court, and Tarble's Case, for better or worse, prevents detainees from pressing their claims in state courts. None of that, of course, is new.

But it's an obscure provision of the D.C. Code (section 16-1901(b)), and not a Supreme Court decision, that closes off the last possible escape valve -- the D.C. Superior Court. As the essay explains, the D.C. Superior Court would otherwise likely have the authority to issue a common-law writ of habeas corpus against a federal officer, which would vitiate any Suspension Clause-based challenges to statutes such as the MCA. In other words, what makes the constitutional question so tricky when Congress attempts to constrain the habeas jurisdiction of the Article III courts is that Congress already has constrained the jurisdiction of the one court that would otherwise be open...

In his dissent in St. Cyr, Justice Scalia suggested that Congress's power over federal habeas jurisdiction was necessarily plenary: "If . . . the writ could not be suspended within the meaning of the Suspension Clause until Congress affirmatively provided for habeas by statute, then surely Congress may subsequently alter what it had initially provided for, lest the Clause become a one-way ratchet.” As the essay concludes, such a statement is absolutely true, but thoroughly incomplete. The ratchet results only from the fact that Congress has itself already precluded access to the common-law writ in the D.C. local courts.

On SSRN and Peer Review

I had a thought today on the perenially interesting subjects of SSRN and peer review.  I got to thinking about it because I recently sent an article out for peer review -- a paper I posted last month on SSRN. 

Despite the anonymity the journal tries to preserve, I wonder whether referrees for economics journal -- and, to a lesser extent, political science journals (because political scientists don't upload to SSRN as often, I think) -- check to see if a paper has already been posted on SSRN to get author information and a download number count (that moronic register of quality).  This issue might also arise with certain peer-reviewed law journals, since law professors (especially those in the ELS and L & E world) routinely upload circulation drafts.  Could SSRN be undermining double-blind peer review?

Of course, not that many would admit to the practice.  But feel free to offer anonymous comments if you've got good info. 

I should note that I don't mean to single out SSRN as the only problem -- one could just as easily google a title to get author info if the paper has been blogged about or workshopped.  But I would imagine that many many more papers get uploaded to SSRN and then go into the publication submissions cycle than get blogged about or workshopped.  Thoughts?

Publication Venue as a Lagging Indicator of Quality

Over at the ELS, Max Schanzenbach weighs in on the pros and cons of peer-reviewed versus student-edited law journals; Max essentially ends in equipoise.  I recently posted on the same issue.  I am wary of perpetual navel-gazing (especially my own, which ain't pretty), but this seems like an issue that is on the minds of many young academics.   I think some of this unease is traceable to a period of transition affecting both the law and social science academies.

Specifically, law is drifting toward the peer-review model because (a) empirical legal studies are in high demand because they supply hard facts in an academic discipline awash in normative polemics; (b) barriers to data collection and analysis have fallen rapidly, which means there is an abundance of important and tractable projects awaiting capable researchers; and (c) student editors have no special ability to assess the rigor of this work, and the lack of technical sophistication by most law faculty have sent them reeling for some external assurance of quality, which peer-reviewed journals provide.

Re the tumult in the social sciences, or at least economics and its various applied branches, the interminable delay of the peer-reviewed single submission process has been one of the primary drivers for the growth of SSRN (and the crossover for economics and finance explains why biz law professors dominate the SSRN law rankings).  A scholar can lay claim to an idea by posting a polished working paper; if it is any good, it will be downloaded, discussed, referenced, and presented at conferences before it appears in a journal.  A dossier of unpublished papers on SSRN is a good strategy for a junior social scientist (or law professor) to build his or her career as he or she waits for journal acceptance and publication–i.e, validation of quality.

But for both law and the social sciences, peer-review now operates on two levels: (1) established scholars, after a lengthy editorial process, accept an  article for publication; or (2) established scholars working in the same field gravitate toward the best work in pre-publication form.  Because of changes in the cost and flow of information, reputations are now being formed based on route #2.  In other words, publication venue is a lagging indicator of quality.

The peer-review conundrum

Michael Heise over at the ELS blog asks a provocative question that I encourage other readers to answer: “[Do law professors] see (or perceive) shifts in overall law school culture or tenure expectations germane to scholarly publication options[?].  If so, when it comes to decisions about where to place scholarship how are folks assessing such options as traditional student-edited law reviews, faculty-edited peer-reviewed journals (in law or other fields), and university press books?”

It would be interesting to know the answer to this question. I suspect the norms vary pretty dramatically, even among elite schools.  For example, publishing in Top 10 law journals generally helps one's career.  But schools like Cornell, Northwestern, and Texas are trying to build empirical powerhouses.  Thus, insofar as peer-review becomes the preferred outlet at those law schools, I suspect that those expectations will influence the publication norms of the wider legal academy, especially with empirical work. 

I had this same question on my mind several months ago when I asked a prominent empiricist (an eminent social scientist who is now on the faculty of a Top 15 law school) whether it was better to publish empirical work in peer-reviewed journals rather than law journals.  [I am not naming names because the question was not asked in the context of posting the answer on a blog.]  I fully expected to get an answer to the affirmative.  But much to my surprise, she/he said that extensive workshopping was as good or better for quality control than the peer-review process.  So to his/her mind, student-edited journals were just fine.  The burden of quality control was on the author.

I think the issue here comes down to a signaling heuristic:  If you get published in JELS or JLS or ALER, then distinguished people in the field said it was good enough to publish.  From the perspective of a junior faculty person, that is pretty attractive.  There will be a presumption of quality that a negative review in your tenure file is less likely to rebut. That said, it is another matter entirely to believe that the peer-review process catches all significant mistakes.  The economics of refereeing a journal article are replete with agency costs, and I have collected anecdotes on this topic over the years.  But that is the topic for a separate post.

By the way, PrawfsBlawg alumni Joelle Moreno had some interesting thoughts on this topic a few months ago.