Wednesday, April 22, 2009
What is Quality Empirical Work, Part 1
In this post, I want to start to flesh out some answers to the challenges posed by evidence based legal empirical scholarship I raised earlier, or at least examine how to go about addressing them.
1. A randomized trial may be politically or ethically impossible. This is, of course, a problem that toxicologists face: they cannot randomly poison some people and not others (though, as evidenced by the Tuskegee experiments, this has happened in the past). But it is also hard--though not impossible--to randomized police interventions.
2. A randomized trial may be technically impossible. RCTs are impractical for studying, say, the effect of exposure to a particular drug or chemical over ten years. Maintaining control conditions for long periods of time is simply too hard a task, even if theoretically possible.
3. A randomized trial can measure only certain effects. As Heckman and Smith demonstrate, RCTs are effective at measuring the average difference between treated and untreated groups, but not other relationships of interest, such as the variation in response.
4. People know they're being tested. Levitt and List have shown that people, sensing that they are being experimented upon, consciously adjust their behavior. As a result, experimental of behavior are harder to interpret.
5. We just don't have time to wait. Experiments often take time to conduct. If we need to know what the relationship is between x and y, and we need to know it quickly, we need observational work if no experiment has already been conducted.
Posted by John Pfaff on April 22, 2009 at 03:43 PM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference What is Quality Empirical Work, Part 1:
Not sure if you're getting a chance to read the comments here, but there's an interesting recent paper that compares randomized experiments to non-randomized:
"Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments," William R. Shadish, M. H. Clark, Peter M. Steiner. Journal of the American Statistical Association.
Posted by: Stuart Buck | Apr 23, 2009 5:09:37 PM