Thursday, April 23, 2009
What is Quality Empirical Work, Part 2
In my previous post, I examined the difficulty of defining what quality means for the type of (observational) work that makes up much of empirical legal scholarship. Here I want to turn my attention to the problem of measuring quality even if there is an accepted definition.
1. First, that measuring the quality of observational studies is difficult, even if there is no disagreement over what "high quality" means.
2. Second--and perhaps more important--that this difficulty does not reflect a weakness of systematic reviews but a strength. If we cannot agree on how to measure certain quality components, that is important information for us to have--information we must have to appreciate the epistemic limits of what our models can tell us.
1. How to score a particular item. Do we adopt a binary approach ("good/bad") or something more continuous (1-5). But if something more continuous, how do we decide how many points to use, and what distinguishs a 2 from a 4? Binary approaches are blunter but more objective.
2. How to score a study. Some have advocated using a single aggregate quality score that can be used to weigh studies and results by quality. But others have pointed out several flaws with such an approach, arguing instead for a component-by-component score. But if the number of components is close to the number of studies, this could prove intractable.
3. How to empirically verify the guideline terms. As I've discussed before, this requires meta-evidence about whether particular quality terms in fact matter. In observational settings, the number of relevant quality criteria may exceed the number of studies, rendering it impossible to isolate individual effects of quality terms on the results.
4. What studies to include. Study quality will vary. Should all the studies be included but some how "weighted" to reflect quality, or should studies only above a certain line be used? And what would that line look like when using a multidimensional quality score? Like all decision rules, this is ultimately more a normative than empirical question, and one that is tied to whatever definition of quality we are using.
Posted by John Pfaff on April 23, 2009 at 07:04 PM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference What is Quality Empirical Work, Part 2: