« Dave Matthews Band, Bootlegs, and Network Economics | Main | WiPping Your Articles Into Shape »

Wednesday, June 03, 2009

Sytematic Reviews and the Scientization of Law

So far I have looked at how to incorporate systematic reviews into our current legal framework, whether through court-apppointed Rule 706 experts or through special masters or technical advisors assisting judges in their Daubert or Frye decisions. In both cases, however, partisan experts remain. Rule 706 experts, for example, testify along side--not in place of--partisan experts; and special masters or technical advisors never testify at all, instead only helping to determine which partisan experts are allowed to testify. Can we go further, however? Is it even reasonable to ask whether we can abolish the partisan expert altogether and rely solely on the systematic review?

My focus in my next few posts is positive, not normative. My previous posts have laid out my argument for why I think it would be a good idea to get rid of the party expert. Here I want to ask whether it is even possible to adjust our adversarial procedures in such a way. Perhaps not surprisingly, I think the answer is yes.

There are two sources of my optimism. The first is inspired by a statement made by Mirjan Damaska, that the scientization of the law is the greatest challenge the law has faced since the Middle Ages. And the second comes from a recent history of the special master by Amalia Kessler. In this post I'll look at the point raised by Damaska, and in my next that raised by Kessler.

Damaska puts its bluntly (on page 151 of Evidence Law Adrift):

Let there be no mistake. As science continues to change the social world, great transformations of factual inquiry lie ahead for all justice systems. These transformations could turn out to be as momentous as those that occurred in the twilight of the Middle Ages, when magical forms of proof retreated before the prototypes of our present evidentiary technology.

Exactly so. Legal practices, and our definition of what constitutes proof, respond to fundamental changes in epistemology. Consider the Medieval forms of proof (here I draw on the work of Ellen Sward and Stephan Landsman): the ordeal, the oath, or the battle, to name a few. In the ordeal, the party bearing the burden of proof had to undertake some sort of physical challenge, such as dipping his hand in boiling water. If he emerged unscathed, he met his burden. Similarly, in the oath the party with the burden had to recite precisely a complicated oath to meet his burden. And in battle, the two sides or their proxies met in armed combat, with the victor in battle the victor in law.

All three approaches were united by a common epistemology: God would reveal the party who deserved to prevail. The Renaissance and the Enlightenment, however, replaced such faith in God with a faith in reason, in people's ability to think through a problem and reach the correct answer. Legal procedures changed--profoundly changed--accordingly.

The past few decades have witnessed a revolution of similar proportions. As I mentioned before, modern statistics is barely a century old. The idea that science is primarily about measurement and experimentation, as opposed to more logical (like mathematics) or classificationist (like zoology) endeavors, is a 19th century development. And our ability to meaningfully undertake such measurement did not arise until the mid- to late-20th century. Moreover, it is only within the past thirty years, with the rise of the actuarial turn, that we have begun to fully appreciate the limits on our ability to reason and the steps we must take to address these limitations. These three changes--a focus on measurement, a growing ability to measure, and a newfound appreciation of the limits of reason--add up to a Scientific Revolution of their own.

When confronted with significant epistemic shifts, the law adapts. Perhaps slowly, perhaps imperfectly, but it adapts. The changes over the past several decades are not small adjustments but a profound change in our understanding of what constitutes scientific knowledge, in our ability to produce the necessary information, and in our sense of how to compile that information into usable knowledge. It is hard to believe that the law will not adapt once more. 

Posted by John Pfaff on June 3, 2009 at 02:46 PM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Sytematic Reviews and the Scientization of Law:


For what's it worth, I think it is important to keep in mind the following observations about and characterizations of science in general:

1. "It is properly supposed that science can be distinguished from other modes of systematic inquiry but a distinctive *method.* This is not what is observed. The *techniques* used in scientific research are extraordinarily diverse, from counting sheep and watching birds to detecting quasars and creating quarks. The epistemic *methodologies* of research are equally varied, from mental introspection to electronic computation, from quantitative measurement to speculative inference." In addition, "'testability' is only one of the epistemic standards by which hypotheses are judged as they come to mind or are presented by others. Theoretical ideas are considered plausible on the basis of diversity of tacit, even contrary criteria. Sometimes scientists try to express their theoretical preferences in terms of 'generality,' 'specificity,' 'simplicity,' 'parsimony,' 'complexity,' 'rigour,' 'flexibility,' 'symmetry,' 'incongruity,' 'communicability,' 'subtlety,' 'accuracy,' 'scope,' 'fruitfulness,' or just 'elegance.' But these are essentially *heuristics*--helpful 'guides to discovery'--not recipes that necessarily guarantee a satisfactory product."

2. Every scientific discipline has its own methods, methodologies, regulative principles, epistemic norms, principles of rationality, forms of inductive reasoning and justification, heuristics or rules of thumb, maxims, and so on. Another way to put this would be to appreciate the fact that every scientific discipline has its own criteria of "scienticity" (or 'scientificity'). Standardization across the sciences is irreducibly and invariably both quantitative and qualitative.

3. "[W]hatever the other virtues of quantification, it cannot succeed in eliminating social or personal factors from the human sciences."

4. "It is a philosophical fantasy to suppose that a scientific [or empirical] 'fact' can be freed from the context in which it was observed. That context always contains both 'theoretical' and 'subjective' features, usually closely intertwined. A sophisticated instrument embodies many theoretical concepts. But those are only elaborations and extensions of the theories needed by a trained observer to 'see' what is scientifically significant in her personal experience of the world."

5. Formal criteria and classification schemes are soon made obsolete by scientific progress. What counts originally as an 'effect' become the basis of a novel method of observation which finishes up as a routine instrumental technology. At any given moment, a discipline or subdiscipline has its recognized research methodologies, its observational instruments, its experimental protocols, its standard scales of measurement and other conventions. But these are maxims, not rigorous regulations. They are not mandatory in principle and are often ignored in practice."

6. "Let us not doubt [that the sciences and the humanities] differ enormously in their subject matter, their intellectual objectives, their practical capabilities, and their social and psychic functions. Nevertheless, they belong to the same culture, and operate institutionally under the same ethos. As a consequence, the knowledge produced by the natural sciences is no more 'objective,' and no less 'hermeneutic,' than the knowledge produced by the social, behavioral and other human sciences. In the last analysis, they are all of equal epistemological weight."

7. Sciences can be either descriptive (e.g., anthropology) or analytical (e.g., physics), and the virtues of one don't necessarily apply to the other, in other words, neither is inherently more "scientific."

8. "As philosophers and other metascientists are coming to realize, theories are very like *maps.* Almost every general statement one can make about scientific theories is equally applicable to maps. They are representations of a supposed 'reality.' They are social institutions. They abstract, classify and simplify numerous 'facts.' They are functional. They require skilled interpretation. And so on. The analogy is evidently much more than a vivid metaphor. In effect, every map *is* a theory. An analysis of the most commonplace map explores almost all the metascientific features of the most recondite theory. From a naturalistic point of view, the London Underground map exemplifies these features just as well as, say, the 'Standard Model' of particle physics."

9. The historically recent "focus on measurement, a growing ability to measure, and a newfound appreciation of the limits of reason" is perfectly compatible with the recognition and appreciation of the fact that, "According to the circumstances, valid scientific reasoning may involve the evaluation of testimony, empathic understanding of human behavior, pattern recognition, category formation, classification, generalization, analogy unification and, above all, the grammar of natural language."

10. An explanatory (as opposed to a descriptive) theory in science requires a model. "This can take a variety of forms--scale models, mechanical model, analogical models, ostensive models, toy models, mathematical models, etc. Indeed, despite philosophical objections, the word 'model' is so widely used in scientific practice that it has become almost a synonym for a 'theory' [cf. the work of Ronald Giere, for instance, Science Without Laws, 1999]. Moreover, "even the most austerely 'scientific' models operate through *analogy* and *metaphor*" [on the former, see the work--often in collaboration--of Dedre Gentner, Keith Holyoak and Paul Thagard; for the latter, cf. the work of Ronald Giere, for instance, his Science Without Laws, 1999].

11. In effect, "scientific rationality is no more than *practical reasoning* [cf. some of the contributions in Peter Carruthers, Stephen Stich and Michael Siegal, eds., The Cognitive Basis of Science, 2002] carried out as well as possible in the context of research." Furthermore, "What counts as serious scientific discourse can be extraordinarily varied and heterogeneous in form and substance." Furthermore, "It is clear that scientific maps, models, metaphors, themata and other analogies are not just tools of thought, or figures of speech. They are of the very substance of scientific theory. As sources of meaning and understanding, they stand on an equal footing with explicit verbal and symbolic representations [such as those found in measurement]."

12. Finally, with regard to measurement and methodolgy in general, and given the recent infatuation with Bayesian formalism, we should be alert to the temptations incarnate in the what Nicholas Rescher dubs the "penchant for quantities," the "fetish for measurement:"

"People incline to think that if something significant is to be said, then you can say it with numbers and thereby transmute it into a meaningful measurement. They endorse Lord Kelvin’s dictum that ‘When you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.’ But when one looks at the issue more clearly and critically, one finds there is no convincing reason to think this is so on any universal and pervasive basis."

As Rescher reminds us, "…the things you cannot quantify in the context of an inquiry may well turn out to be the most important." And this brings us full circle to the "problem of induction," and thus John Norton's insight that "all inductions ultimately derive their license from facts pertinent to the matter of the induction," and thus, given the motley natural and human sciences, "the more universal the scope of an inductive inference schema, the less its strength."

Posted by: Patrick S. O'Donnell | Jun 3, 2009 9:07:06 PM

I forgot to mention the bulk of the quoted material above was from John Ziman's remarkbale book, Real Science... (2000).

Posted by: Patrick S. O'Donnell | Jun 3, 2009 9:10:31 PM

Erratum, no. 1 above: "...from other modes of systematic inquiry by a distinctive *method.*"

Posted by: Patrick S. O'Donnell | Jun 3, 2009 9:12:13 PM

In no. 10 above the second bracketed reference should not have been to Giere's book again but rather Theodore Brown's Making Truth: Metaphor in Science (2003).

Posted by: Patrick S. O'Donnell | Jun 3, 2009 11:45:05 PM

Post a comment