Tuesday, February 04, 2014
Judgment Calls and Reputation, Part One: Figure Skating
Thanks to Dan and the Prawfs gang for letting me hang out here again for February. Over the course of the month, I hope to explore how impressions of others influence judgments in a variety of litigation settings. With the Winter Olympics approaching, however, it seems appropriate to start outside the courtroom with a different group of high-profile decision-makers: figure skating judges.
Assessing a skater’s performance is a highly challenging cognitive activity. The judge must evaluate both the skater’s technical proficiency and artistic contribution in real time, and convert those assessments into a quantifiable score just minutes after the performance has ended. There is no video replay, no time for careful review and consideration of what was observed. Skating judges must act quickly and decisively. It seems natural that skating judges would therefore rely on mental shortcuts and other strategies to reduce their cognitive load. And indeed, one study out of the University of Ottawa found that one common and influential mental shortcut for judges was the skater’s reputation.
The Ottawa study (unfortunately, available by subscription only) concluded that when judges believed that a skater had made a positive name for him- or herself within the regional skating community, the skater received significantly higher scores than when the skater was unknown to the judges. The study concluded that a skater’s positive reputation set certain expectations for the judges about the skater’s ability, which in turn led to a more positive assessment of the skater’s performance.
Now, it is likely (though not guaranteed) that skaters with positive reputations were indeed excellent at their craft. But even if a skater’s reputation perfectly captured her average past performance, it cannot reliably capture the intricacies of any future performance. So although the judges’ reliance on reputation (consciously or not) was entirely natural, we might look to ways to reduce or eliminate the bias in the interest of obtaining the most accurate assessment possible.
One solution, proposed by the Ottawa researchers, is to ensure that judges in any given competition are unfamiliar with the individual skaters – thereby forcing them to assess the skaters on the current performance alone. This proposal might work if there was a large enough pool of qualified judges to assure that judges were always unfamiliar with the skaters before them. Still, as a long-term way of promoting better accuracy, “blind” figure skating judging seems unworkable. Sooner rather than later, the system would exhaust the number of qualified judges, or judges would share with each other what they had seen in earlier competitions. In either event, judges would eventually come into competitions with some reasonably well-defined expectation of each skater’s skill and artistic ability.
Figure skating provides a fairly straightforward introduction to the problem; the cognitive challenges that impressions and reputation pose to accurate judgments are compounded in the litigation setting. Whereas the effects of impressions in figure skating judging are essentially felt one way—even if the skaters know something about a particular judge’s reputation, there is not much they can do about it on the day of competition—the effects of impressions in litigation are multidirectional. Lawyers, judges, and parties must regularly make decisions based on their ongoing interactions and evolving perceptions of each of the other players. Other ways of promoting accurate judgments are needed and, as I will suggest in the next series of posts, the best approach may be the opposite of blind judging; that is, extensive, repeated interaction between the key players.
TrackBack URL for this entry:
Listed below are links to weblogs that reference Judgment Calls and Reputation, Part One: Figure Skating: