« Co-authoring Strategies; books, etc. | Main | A: "No, Senator, I don't know what Barbra Streisand's eighth album was." Q: Was it not, in fact, Color Me Barbra, sir?" »

Thursday, May 07, 2009

A Summary, and a Small Point About Competence

Since I've been posting for a month now about problems with the production and use of empirical evidence, I thought I would take a moment here to (1) briefly summarize my arguments so far and (2) quickly address an issue of competence.

First, the points so far:

1. Empiricists have traditionally produced knowledge poorly, focusing too much on pioneering work and not enough on rigorous syntheses. This could be due in part to a general disdain for the review essay, but is surely also the result of a poor philosophy of science.

2. This failure to produce knowledge well is becoming untenable as the volume of empirical work--much of it of poor quality--explodes. Social scientists and courts alike are being forced to wade through ever deeper, and ever more incoherent, bodies of empirical claims.

3. Fortunately, empiricists* are developing increasingly powerful methods for determining what studies are of good quality and then synthesizing their findings in what are oftren referred to as "systematic reviews." This is at the heart of the Evidence Based Policy movement.

4. There is, however, much work still to be done. Most work on systematic reviews has focused on ranking and aggregating the findings from experimental studies. Almost no attention has been paid to observational studies, which is unfortunate: such studies are much more common (especially in litigation), much easier to produce, and much more vulnerable to (naive or cynical) errors.

5. Though only in its infancy, EBP provides a meaningful opportunity for reforming how the law uses science. Prior to EBP, the sciences generated knowledge through roughly-adversarial means, and thus they did not have a true alternative to offer the law. EBP represents a more collaborative and holistic approach, and thus a true alternative to adversarial knowledge production.

In my remaining posts on this topic, I will look at how to incorporate EBP and systematic reviews into an adversarial legal system. Some sort of reform is necessary. Mirjan Damaska, in Evidence Law Adrift, posits that the current scientization of the law is the greatest challenge it has faced since the end of the Middle Ages, and I think he is right. Given the profound changes that occurred then, we should expect--and thus should start think about--such change again.

But before that, I want to address one small, but important, point. It seems to me that the rise of EBP finally provides evidence for a claim that has long been asserted with little proof: juries and judges lack the epistemic competence to handle complex evidence.

Gathering data on this issue is difficult. Asking judges if they agree with jury conclusions, as some studies have done, seems to miss the point. If both judges and juries lack the necessary skills to understand scientific evidence, then asking judges is akin to asking one blind man how well a (slightly more?) blind man described a painting. And asking statisticians, epidemiologists, or others to review how well jurors do is perhaps too time intensive. It may be possible to test this with mock juries, and if there is work like that out there I would certainly appreciate any pointers.

But EBP itself provides evidence of epistemic incompetence. EBP is based on the idea that trained experts cannot properly synthesize a literature. Few would disagree with the claim that epidemiologists or economists or doctors with years, possibly decades of training are better able to draw conclusions from a literature than a jury--at most half of whom have graduated from college--or a judge, especially one whose academic training includes little math or science. Thus if EBP argues that the experts lack the necessary talents, then by the Transitive Property of Competence, judges and jurors must be even more impaired.

To call on judges to screen evidence or jurors to interpret it is thus to ask them to do something they simply cannot do. But let me be clear: I do not mean to denigrate judges or jurors in saying that, nor do I think that I do so. EBP, and the actuarial turn more generally, are forcing us--where "us" refers to everyone everywhere--to confront more honestly our epistemic shortcomings. 

Our growing appreciation of the limits of judges and jurors, combined with the upsurge in empirical work that they face, means that something must give. And what better time than now to think about bold and daring changes that we can make, when all the avenues still remain open to us?

* I should point out that I am using "empiricist" as economists, not philosophers, do. To me, an empiricist is someone who analyzes data, in contrast to the theorists. I appreciate that philosophers of science may first think of, say, the logical positivists like Carnap when seeing that term, but that is not how I intend it.  

Posted by John Pfaff on May 7, 2009 at 04:57 PM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference A Summary, and a Small Point About Competence:



Thanks for the clarification.

While I appreciate the fact that you would not have us divorce empirical work from theory, I'll simply reiterate my point about how even "empirical" work is shot through with theory, even if unintentionally so....

So perhaps my point is that there's something amiss if we have some social scientists understanding themselves to be doing the "empirical work," while others are devoted to articulating the "theoretical work." That division of labor may work more or less for the natural sciences, but I suspect it has pernicious effects in the social sciences insofar as the latter are not slavisly beholden to the former.

I think Elster's work on the difficulty of determining the precise causal mechanisms at work in the social sciences is why we have so much fanciful speculation based on correlations although virtually everyone these days is aware of the dictum that correlation is not causation.

And I agree with you that "we need to accept that we don't [necessarily] test or evaluate our theories in a Popperian way."

In any case, I appreciate your efforts to publicly think through these questions.

Posted by: Patrick S. O'Donnell | May 12, 2009 10:55:21 AM

Patrick: Thanks for your comments. I've added a small "say" to my footnote to make it clear I wasn't focusing on a particular branch of empiricism but just drawing attention to the philosophy/social science distinction.

The problem with "positivist" is that, at least as used in the social sciences, positive analysis can be both theoretical or--to keep using the term--empirical (data-driven). Work falls into a two-by-two grid, with positive/normative on one axis and theoretical/empirical on the other (although I think the normative/empirical box should be left blank, since outcome-driven empirical work is unacceptable).

I know I need to read McCloskey's work, but I'm not sure how rhetorical "empirical" is here. Both theoretical and empirical work is essential in the social sciences, so to say I engage in empirical work isn't to argue that what I do is more important than what other social scientists do.

And I'm not advocating that we divorce empirical work from theory. Simply mining correlations is how we get the "disease of the week." But we need to accept that we don't test or evaluate our theories in a Popperian way.

Posted by: John Pfaff | May 12, 2009 10:11:06 AM

Any philosopher of science worth her salt may first think of empiricism (e.g., Locke), or logical positivism, or logical (more broadly, epistemic) empiricism (e.g., Hempel), pragmatic empiricism (van Fraassen), "post-positivism" (Quine), or contextual empiricism (Longino). In other words, in philosophy of science, empiricism comes in a variety of forms and is hardly limited to its logical positivist incarnation.

Without here explaining why (cf. Deirdre McCloskey's treatment of the rhetoric of economics in several well-known books), I think the methods you are referring to are better described as "positivist" (or 'post-positivis') rather than "empiricist." This certainly holds true for many economists, particularly those (the vast majority) committed to the methods and tenets of neoclassical economics. I suspect use of the word "empirical" serves a broad rhetorical appeal (who could be against the use of empirical methods in the social sciences?).

In any case, and sounding dogmatic, I doubt one can "analyze" anything in the social sciences without theory. Or more generously, it sounds as if we're regressing to a fact/value dichotomy if we believe analyzing data takes place without theory of some sort or, if you prefer, value and normativity permeate all experience, as the classical pragmatists held, and insofar as empiricism needs reference to experience (or 'data'), it is implicated in normative judgments (and not just ethical judgments, as Hilary Putnam reminds us, but judgments of 'coherence, 'plausibility' 'reasonableness,' 'simplicity,' and so on), and I think it is well-nigh impossible to have judgments of value of this type without some kind of background theory, however tentative or general (i.e., they don't operate in an epistemic vacuum) or unacknowledged.

Posted by: Patrick S. O'DonnellP | May 7, 2009 8:39:50 PM

The comments to this entry are closed.