Systematic Reviews and Adversarialism: Truth and Arbitrariness

In my last post, I laid out some of the major arguments for favoring adversarial debates about scientific evidence over independent experts. Now I want to start pointing out how systematic reviews retain the benefits of independent expertise while addressing the concerns raised by the defenders of adversarialism. In this post I'll talk about the relationship between systematic reviews and (1) uncovering the truth and (2) avoiding arbitary outcomes; in a subsequent post I'll take on party control and dispute resolution.

At one level, my previous posts (assuming you accept my arguments) make the question of truth-finding a trivial issue. Systematic reviews produce knowledge better than adversarial techniques, whether in a courtroom or a lab. But I want to take care to point out how systematic reviews directly confront the concerns voiced about independent experts.

1. Incentives to find relevant evidence. The rigorous search methods used in systematic reviews are designed to find every relevant study. Part of the impetus to develop EBP was the very fear that researchers often overlooked important results. But these search methods are still being developed, and analysts may still miss studies. Not a problem. Here, systematic reviews and adversarialism can work hand in hand: let the parties submit studies for consideration. Such an approach retains the information-gathering incentives of adversarialism while preserving the independence of the review itself.

2. Incentives to find the best expert. First, we have a definitional issue. The best expert for the parties is not necessarily the best expert for the factfinder. Parties have an incentive to find the expert who will advocate their positions most strongly. So if the evidence strongly favors Party A over Party B, the best expert for Party B is likely the worst expert for a neutral factfinder trying to get to the truth. So by "best" I mean best for the factfinder.

Here, we have competing effects. It is true that judges have less of an incentive to look for experts. But work by Hooper et al. indicates (not surprisingly) that scientists dislike adversarial procedures. So as the process becomes less adversarial, the pool of avaiable experts will deepen, and judges thus will not have to look so hard to find experts. Third-party assistance, such as the AAAS's CASE project, can help as well. And finally, we can once again tap into the parties' stronger incentives, such as by having the parties suggest names and then requiring both to sign off on the chosen experts.

3. Confirmation bias. Again, this was a concern that motivated the development of EBP. The ex ante guidelines are designed in part to confront this very problem--the guidelines are established before the bias has set in. Having multiple reviewers can further mitigate this risk (especially if they review the studies in different random orders). Plus, note that the sequential presentation of evidence at trial can lead to this problem even in the adversarial context.

4. Using cross-examination to reveal biases. Again, the purpose of guidelines is to highlight and reveal biases in underlying work. Guidelines can be designed in many ways to captures numerous sources of bias. Thus they can ask not only "did the study control for self-selection bias?" but "was the study funded by an industry group?" Any concern that could come up on cross-examination can be put into the guidelines.

Moreover, the party weighing the implications--the independent expert as opposed to the jury--is much more able to carry out that task. Experts are much more capable than jurors of understanding how important a particular source of bias is. As a result, outcomes will be less arbitrary, less based on guesswork by an epistemically incompetent party. Quality guidelines retain the benefits of bringing methodological, ideological, and financial problems to light, but greatly mitigate the risk of inappropriate confusion.

5. Judicial manipulation of (not-un) due deference. This is addressed by the flip side of the guidelines' goals. Guidelines are designed not just to reveal biases, but to constrain the reviewer. And by limiting reviewer biases, guidelines greatly reduce the ability judges could have to selectively choose experts.

6. Avoiding corpusculation. One concern with Daubert is the risk of what Thomas McGarity refers to as "corpusculation." the court evaluates each study on its own and finds it insufficient; each study is tossed out one by one. If viewed as a whole, the studies may have reinforced each other to provide a whole greater than the sum of its parts--this is just the crossword puzzle once more. But corpuscular review misses the forest for the trees. Clearly, systematic reviews are designed to overcome this flaw.

Despite these benefits, the use of systematic reviews is not without its challenges. If nothing else, there exists a serious issue I have glossed over: so far I've taken the existence of the guidelines as a given. In my next post I will examine how to develop guidelines in an adversarial system.