Thursday, May 07, 2009
A Summary, and a Small Point About Competence
Since I've been posting for a month now about problems with the production and use of empirical evidence, I thought I would take a moment here to (1) briefly summarize my arguments so far and (2) quickly address an issue of competence.
1. Empiricists have traditionally produced knowledge poorly, focusing too much on pioneering work and not enough on rigorous syntheses. This could be due in part to a general disdain for the review essay, but is surely also the result of a poor philosophy of science.
2. This failure to produce knowledge well is becoming untenable as the volume of empirical work--much of it of poor quality--explodes. Social scientists and courts alike are being forced to wade through ever deeper, and ever more incoherent, bodies of empirical claims.
3. Fortunately, empiricists* are developing increasingly powerful methods for determining what studies are of good quality and then synthesizing their findings in what are oftren referred to as "systematic reviews." This is at the heart of the Evidence Based Policy movement.
4. There is, however, much work still to be done. Most work on systematic reviews has focused on ranking and aggregating the findings from experimental studies. Almost no attention has been paid to observational studies, which is unfortunate: such studies are much more common (especially in litigation), much easier to produce, and much more vulnerable to (naive or cynical) errors.
5. Though only in its infancy, EBP provides a meaningful opportunity for reforming how the law uses science. Prior to EBP, the sciences generated knowledge through roughly-adversarial means, and thus they did not have a true alternative to offer the law. EBP represents a more collaborative and holistic approach, and thus a true alternative to adversarial knowledge production.
Posted by John Pfaff on May 7, 2009 at 04:57 PM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference A Summary, and a Small Point About Competence:
Any philosopher of science worth her salt may first think of empiricism (e.g., Locke), or logical positivism, or logical (more broadly, epistemic) empiricism (e.g., Hempel), pragmatic empiricism (van Fraassen), "post-positivism" (Quine), or contextual empiricism (Longino). In other words, in philosophy of science, empiricism comes in a variety of forms and is hardly limited to its logical positivist incarnation.
Without here explaining why (cf. Deirdre McCloskey's treatment of the rhetoric of economics in several well-known books), I think the methods you are referring to are better described as "positivist" (or 'post-positivis') rather than "empiricist." This certainly holds true for many economists, particularly those (the vast majority) committed to the methods and tenets of neoclassical economics. I suspect use of the word "empirical" serves a broad rhetorical appeal (who could be against the use of empirical methods in the social sciences?).
In any case, and sounding dogmatic, I doubt one can "analyze" anything in the social sciences without theory. Or more generously, it sounds as if we're regressing to a fact/value dichotomy if we believe analyzing data takes place without theory of some sort or, if you prefer, value and normativity permeate all experience, as the classical pragmatists held, and insofar as empiricism needs reference to experience (or 'data'), it is implicated in normative judgments (and not just ethical judgments, as Hilary Putnam reminds us, but judgments of 'coherence, 'plausibility' 'reasonableness,' 'simplicity,' and so on), and I think it is well-nigh impossible to have judgments of value of this type without some kind of background theory, however tentative or general (i.e., they don't operate in an epistemic vacuum) or unacknowledged.
Posted by: Patrick S. O'DonnellP | May 7, 2009 8:39:50 PM
Patrick: Thanks for your comments. I've added a small "say" to my footnote to make it clear I wasn't focusing on a particular branch of empiricism but just drawing attention to the philosophy/social science distinction.
The problem with "positivist" is that, at least as used in the social sciences, positive analysis can be both theoretical or--to keep using the term--empirical (data-driven). Work falls into a two-by-two grid, with positive/normative on one axis and theoretical/empirical on the other (although I think the normative/empirical box should be left blank, since outcome-driven empirical work is unacceptable).
I know I need to read McCloskey's work, but I'm not sure how rhetorical "empirical" is here. Both theoretical and empirical work is essential in the social sciences, so to say I engage in empirical work isn't to argue that what I do is more important than what other social scientists do.
And I'm not advocating that we divorce empirical work from theory. Simply mining correlations is how we get the "disease of the week." But we need to accept that we don't test or evaluate our theories in a Popperian way.
Posted by: John Pfaff | May 12, 2009 10:11:06 AM
Thanks for the clarification.
While I appreciate the fact that you would not have us divorce empirical work from theory, I'll simply reiterate my point about how even "empirical" work is shot through with theory, even if unintentionally so....
So perhaps my point is that there's something amiss if we have some social scientists understanding themselves to be doing the "empirical work," while others are devoted to articulating the "theoretical work." That division of labor may work more or less for the natural sciences, but I suspect it has pernicious effects in the social sciences insofar as the latter are not slavisly beholden to the former.
I think Elster's work on the difficulty of determining the precise causal mechanisms at work in the social sciences is why we have so much fanciful speculation based on correlations although virtually everyone these days is aware of the dictum that correlation is not causation.
And I agree with you that "we need to accept that we don't [necessarily] test or evaluate our theories in a Popperian way."
In any case, I appreciate your efforts to publicly think through these questions.
Posted by: Patrick S. O'Donnell | May 12, 2009 10:55:21 AM