A Summary, and a Small Point About Competence

Since I've been posting for a month now about problems with the production and use of empirical evidence, I thought I would take a moment here to (1) briefly summarize my arguments so far and (2) quickly address an issue of competence.

First, the points so far:

1. Empiricists have traditionally produced knowledge poorly, focusing too much on pioneering work and not enough on rigorous syntheses. This could be due in part to a general disdain for the review essay, but is surely also the result of a poor philosophy of science.

2. This failure to produce knowledge well is becoming untenable as the volume of empirical work--much of it of poor quality--explodes. Social scientists and courts alike are being forced to wade through ever deeper, and ever more incoherent, bodies of empirical claims.

3. Fortunately, empiricists* are developing increasingly powerful methods for determining what studies are of good quality and then synthesizing their findings in what are oftren referred to as "systematic reviews." This is at the heart of the Evidence Based Policy movement.

4. There is, however, much work still to be done. Most work on systematic reviews has focused on ranking and aggregating the findings from experimental studies. Almost no attention has been paid to observational studies, which is unfortunate: such studies are much more common (especially in litigation), much easier to produce, and much more vulnerable to (naive or cynical) errors.

5. Though only in its infancy, EBP provides a meaningful opportunity for reforming how the law uses science. Prior to EBP, the sciences generated knowledge through roughly-adversarial means, and thus they did not have a true alternative to offer the law. EBP represents a more collaborative and holistic approach, and thus a true alternative to adversarial knowledge production.

In my remaining posts on this topic, I will look at how to incorporate EBP and systematic reviews into an adversarial legal system. Some sort of reform is necessary. Mirjan Damaska, in Evidence Law Adrift, posits that the current scientization of the law is the greatest challenge it has faced since the end of the Middle Ages, and I think he is right. Given the profound changes that occurred then, we should expect--and thus should start think about--such change again.

But before that, I want to address one small, but important, point. It seems to me that the rise of EBP finally provides evidence for a claim that has long been asserted with little proof: juries and judges lack the epistemic competence to handle complex evidence.

Gathering data on this issue is difficult. Asking judges if they agree with jury conclusions, as some studies have done, seems to miss the point. If both judges and juries lack the necessary skills to understand scientific evidence, then asking judges is akin to asking one blind man how well a (slightly more?) blind man described a painting. And asking statisticians, epidemiologists, or others to review how well jurors do is perhaps too time intensive. It may be possible to test this with mock juries, and if there is work like that out there I would certainly appreciate any pointers.

But EBP itself provides evidence of epistemic incompetence. EBP is based on the idea that trained experts cannot properly synthesize a literature. Few would disagree with the claim that epidemiologists or economists or doctors with years, possibly decades of training are better able to draw conclusions from a literature than a jury--at most half of whom have graduated from college--or a judge, especially one whose academic training includes little math or science. Thus if EBP argues that the experts lack the necessary talents, then by the Transitive Property of Competence, judges and jurors must be even more impaired.

To call on judges to screen evidence or jurors to interpret it is thus to ask them to do something they simply cannot do. But let me be clear: I do not mean to denigrate judges or jurors in saying that, nor do I think that I do so. EBP, and the actuarial turn more generally, are forcing us--where "us" refers to everyone everywhere--to confront more honestly our epistemic shortcomings. 

Our growing appreciation of the limits of judges and jurors, combined with the upsurge in empirical work that they face, means that something must give. And what better time than now to think about bold and daring changes that we can make, when all the avenues still remain open to us?

* I should point out that I am using "empiricist" as economists, not philosophers, do. To me, an empiricist is someone who analyzes data, in contrast to the theorists. I appreciate that philosophers of science may first think of, say, the logical positivists like Carnap when seeing that term, but that is not how I intend it.