« More Problems With Hypothesis Testing (From a More Technical Perspective) | Main | The Beautiful Game . . . »

Tuesday, April 14, 2009

Avoiding a Biased Exam: Always Expect Students to Know the Law But Never Expect Them to Know the Facts

I remember being a law student and taking the class Women and the Law with the terrific Susan Grover when a topic came up that would (thankfully) inform the way that I draft my law school exams when I became a professor. The topic: Biased exams. One student brought up a Torts exam she had taken her first year that left her perplexed. The exam was in the fall of 2000, and it involved an XFL fact pattern with, if memory serves me, running back George W. Bush assaulting linebacker Al Gore either as part of a football play or immediately after it had concluded. The reason for the student's confusion was that she didn't follow football and therefore had difficulty answering this torts in sports fact pattern because she couldn't figure out what role the running back plays on the football field, making it tough to analyze how out-of-the-ordinary W's conduct would have been. 

I could sympathize with my fellow student's comment even though I had taken the same Torts exam and not realized the difficulties it could create at the time. I once had a Sri Lankan friend attempt to explain cricket to me (I remember this involving both explanations and diagrams), and I had about as much success in figuring out what she said as the XFL had as a sports league. I imagine that I would have struck out if given the wicked googly of an exam with a cricket fact pattern if I decided to get an LLM in England (except, wait, there is no strike out in cricket). I therefore could easily imagine a football-averse J.D. student, whether male or female, going three-and-out on an exam with a football fact pattern, and I could see the same thing happening to an LLM student from a country not as American football crazy as us Yanks (i.e., every other country in the world). 
Another student could sympathize as well. He mentioned that after his Contracts exam, several students had complained. You see, the fact pattern on the exam involved a vendor breaching a contract by supplying the buyer with off-brand clothes instead of designer duds. These students didn't understand the concept of off-brand clothes and thus the nature of the breach by the vendor. 

The discussion of the topic in class taught me an important lesson that I otherwise wouldn't have learned: Professors should always expect students to know the law on an exam, but they should never expect them to know the facts. In other words, don't throw students the curveball of a baseball fact pattern (unless you teach Sports Law) unless you expect some to fall below the Mendoza Line. Don't give students a fact pattern where you tell them that a movie finished "in the black" unless you want some exams to end up "in the red." And don't give students a fact pattern where they have to understand the difference between Berdorf Goodman and TJ Maxx and expect the Maxx for the Minimum

Of course, any of these fact patterns would be problematic, but they wouldn't be offensive. In doing research for this post, however, I came across the following passage from Angela I. Onwuachi-Willig, Note, Moving Ground, Breaking Traditions: Tasha's Chronicle, 3 Mich. J. Race & L. 255, 274 n.33 (1977):

See generally Patricia J. Williams, The Alchemy of Race and Rights (1991). In her book, Williams describes a number of race- and gender-biased exams created by White male professors. Some included:

--a tax exam that asks students to calculate the tax implications for Kunta Kinte's master when the slavecatchers cut off his foot.

--a securities-regulation exam in which the professor muses about whether white-collar defendants should go to jail, since “unlike ghetto kids” they are not equipped to fare in that environment.

--a constitutional-law exam in which students are given the lengthy text of a hate-filled polemic entitled “How To Be a Jew-N*****” and then told to use the first amendment to defend it.

--a description of the “typical criminal” as a "a young Black male with an I.Q. of 87 who is one of eight children and has always lived on welfare and who spends his time hanging out in pool halls with his best friend Slick."

--numerous criminal-law exams whose questions feature exclusively Black or Hispanic or Asian criminals and exclusively white victims.

--many questions depicting gay men as the exclusive spreaders of AIDS, asking students to find the elements of murder.

--many, many questions in which women are beaten, raped, and killed in descriptions pornographically detailed (in contrast to streamlined questions, by the same professors, that do not involve female victims) ***

The problem with such questions, as Williams argues, is that they: require Blacks, women who have been raped, gays and lesbians, to not just re-experience their oppression, but to write against their personal knowledge. They actually require the assumption of an "impersonal" (but racist/sexist/homophobic) mentality in order to do well in the grading process....[I]t requires students to suppress any sense of social conscience. It requires them to devalue their own and others' humanity for the sake of a grade....In essence, such questions disproportionately require that women and minorities move outside of their experience to perform well on exams. 

This is pretty disturbing and part of the reason I chose not to test Federal Rules of Evidence 412-415 on my Evidence exams. But these are the obvious examples. Have you come across an exam as a professor or a student with facts that seemed innocuous but which ended up being unfair to a number of students?

Posted by Evidence ProfBlogger on April 14, 2009 at 07:39 AM in Teaching Law | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Avoiding a Biased Exam: Always Expect Students to Know the Law But Never Expect Them to Know the Facts:


On my 1L contracts exam, there was a question in which the word "nursery" in a contract was critically ambiguous (child's room vs. plant room). If I remember correctly, the contract was for a babysitter who was also supposed to water the plants in the greenhouse.

I agonized over whether the ambiguity was inadvertant or whether it was the key issue that we were supposed to spot and discuss. The ambiguity seemed glaringly obvious to me, but I doubted whether it was deliberate because it seemed incredibly unfair to students (especially the foreign LLMs in the class) who might not pick up on the potential second meaning.

I ended up focussing pretty heavily on the ambiguity, and I got an A on the exam. I still wonder whether that was the answer the professor intended.

Posted by: Anonymous Associate | Apr 14, 2009 9:11:18 AM

Count me among the many who, while in law school, were subjected to a final exam in criminal law that consisted entirely of a 1-question issue-spotter with a fact pattern about rape. Furthermore, it was an acquaintance rape situation, and one of the legal questions we were forced to address was whether this was really rape at all.

Posted by: Anon | Apr 14, 2009 9:11:34 AM

What's with all the random links? Do readers of this blog really need a link to wikipedia to understand what a J.D. degree is? It's a little annoying and distracting.

Posted by: anonymous | Apr 14, 2009 9:23:41 AM

Anonymous Associate, that reminds me of the confusion many students have when reading the line "Get thee to a nunnery" in Hamlet.

Posted by: Colin Miller | Apr 14, 2009 9:29:15 AM

I think there's a difference from the types of fact patterns you initially discussed and the ones in the Williams study.

The latter reminds me of the situation where the ultra-conservative law grad refused to answer a question on his bar exam dealing with homosexuality, because it was against his moral beliefs.

Well, the reality is that in practice you will come across issues you disagree with or even find patently offensive. Many cases will disturb you. A good lawyer should be able to look past that and objectively apply their educated professional judgment to the facts.

However, in practice, the attorney will probably have time to familiarize themselves with unfamiliar facts (such as the finer points of American football).

I guess that's why I agree that time-limited exams should avoid potentially unfamiliar fact-patterns, but not potentially offensive ones.

Another point, is at what point does a question become offensive? It's easy to point out a racial situation, or a question about rape, but what about something seemingly mundane? I faced an exam in law school the day after a close relative's funeral, and the fact pattern was about a funeral home improperly disposing of bodies. Is that offensive? Disturbing? I just found it ironic and went on to get an A in the class.

Posted by: GaLaw | Apr 14, 2009 9:41:42 AM

GaLaw, I agree that there is a difference in the fact patterns. I classify the ones I initially discussed as problematic/potentially confusing and the ones from the Williams study as offensive. And I agree with you to an extent that good lawyers should be able to look past some offensive issues and objectively apply their educated professional judgement to the facts.

But is this really different from the attorney needing time to get up to speed on an unfamiliar fact-pattern? In other words, wouldn't we expect the lawyer whom had been raped to need some time before possibly being able to objectively apply her educated professional judgment to a rape case? And if we do, isn't it potentially unfair to test such issues on a time constrained exam?

Posted by: Colin Miller | Apr 14, 2009 9:54:37 AM

GaLaw is correct that Colin's situations are vastly different than those in the Williams study, many (although not all) of which are not only offensive, but *gratuitously* so. I would strongly defend the hate-speech/First Amendment one, because I think it goes to the basic need to get past the emotional reaction to speech and into the constitutional issues. This is the same dilemma that profs face in deciding whether to actually show pornography in teaching obscenity or (I imagine) crim profs face in deciding the level of detail to use in teaching rape. But I think to avoid teaching (or testing on) these essential areas is the wrong approach.

As for the need for "time," we have to distinguish emotional reaction from knowledge. That law student who has been raped is dealing with an emotional block. An exam that requires special factual knowledge (of football, fashion, whatever) is unfair because a lawyer could, with time, figure out football or high fashion.

Also, the arguments about biased exams are not about detail and specialized knowledge. Many often object to any exam that deals with a subject (sports, fashion, ballet, hip-hop) that is potentially "cultured" or "gendered," even if one could answer the question without specialized knowledge. But that seems to limit our ability to come up with anything interesting or fun for our exams.

Posted by: Howard Wasserman | Apr 14, 2009 10:22:59 AM

I find these issues critically important for my core classes - patent law and cyberlaw. Both rely on potentially technical factual issues, so providing sufficient detail without assuming knowledge is difficult. Whether I achieved success is another story altogether.

Posted by: Michael Risch | Apr 14, 2009 11:18:43 AM

Well, the reality is that in practice you will come across issues you disagree with or even find patently offensive. Many cases will disturb you. A good lawyer should be able to look past that and objectively apply their educated professional judgment to the facts.

Sure, but choosing fact patterns that are likely to be highly emotional mostly to rape victims, for instance, seems pretty unnecessary.

Posted by: anon | Apr 14, 2009 11:45:28 AM

Thanks, Michael. I think that there might be a difference between mandatory and optional classes. In other words, requiring technical factual knowldge of patents in a patent class does not seem objectionable. Requiring such knowledge in a contracts class would be. As I noted in the post, using a baseball fact pattern (or at last one that requires knowledge of the rules of baseball) would not be objectionable in a sports law class, but it would be objectionable in a torts class.

Posted by: Colin Miller | Apr 14, 2009 12:26:43 PM

Thanks, Howard. I agree that we should not avoid teaching or testing such subjects, with the question being whether they should be on the final exam. With Rule 412-415, I teach them and do hypotheticals on them in class, but I do not include them on the final exam. I'm pretty sure that I could include them on the final exam if I wrote the questions in a sensitive manner, but I have decided that the benefit of such inclusion would not outweigh the risk.

Also, you make a great point about including a certain subject matter on the exam versus requiring knowledge of the specifics of that subject matter. So, there's no problem with including a baseball fact pattern on a torts exam if the fact pattern does not require knowledge of the rules of baseball, but there is a problem if students have to know their intentional walk from their balk.

Posted by: Colin Miller | Apr 14, 2009 12:38:54 PM

As a 1L, our contracts professor had a multiple choice question on reliance where an Oakland Raider's fan was promised a ticket to a football game in "the black hole" (a specific section of the stadium for the costumed crazy fans). After getting promised the ticket she bought a costume and decked herself out in face paint and full costume, basically looking like those crazy fans you see on TV during the lead in to the ads. The promisor didn't give her the ticket and I believe the correct answer was promisor owes promisee all the money for getting all decked out due to reliance.

The vast majority of the class (me included) got it wrong, because we didn't know what the black hole was. The prof said that "as a Raider fan, the promisor should know that most Raider fans in the black hole are costumed," but of course if you didn't know what "the black hole" was you thought it was unreasonable for her to go and do that in anticipation of the ticket. My friend, a big Raider fan, was one of the few who got it right. The class was pretty pissed and the Prof was very defensive and didn't admit that was a bad question.

This Prof is not very popular I might add.

Posted by: RH | Apr 14, 2009 12:53:29 PM

Colin: that's a very good point that the emotional block in the rape case could pose the same problems as being unfamiliar with the facts.

But I wonder if it's a little different. The student will at least have an idea that rape might be on the exam, since if it's covered in class it's likely on the syllabus at the beginning of the semester.

I imagine RH's "black hole" example didn't come after that much notice.

But I agee w/ anon that choosing that particular fact pattern is simply unwise, regardless--my crim law professor actually acknowledged in class the emotional difficulty of the subject and would not cold-call anyone those two days. He didn't put it on the exam either.

Then again, part of me wants to say--you've got to move on at some point--and you knew going into it that you'd run into this subject when you enrolled in law school. I understand the desire to be sensitive, and were I in the professor's shoes I probably wouldn't test it, either, but I'm not convinced there's a compelling reason why, other than a gut feeling. Maybe the best argument against it is Colin's and Michael's distinction between mandatory and optional classes, but again that seems to apply more to issues of knowledge than emotional reaction, assuming the topic that would result in such reactions is either a) a "normal" part of the law school curriculum or b) explicitly stated on the syllabus. And surely rape would fall under one of those two categories at many law schools.

Posted by: GaLaw | Apr 14, 2009 1:10:00 PM

Count me in as another one who sees distinctions among these examples -- and, for that matter, distinctions within the examples offered by Williams. I know you don't say otherwise. But I think these are important distinctions. The tax and securities exam distract from the subject with their use of these raced examples. The discussion of the "typical criminal" is unclear to me; what subject is being taught in this exam, and why is the quoted passage relevant to answering the question? Being lazy or callous in one's exam-writing -- ie., treating all criminals as black (again, why is the defendant's race relevant? Are relevant legal issues being queued up, or not?) or offering graphic details in some cases and not others -- is obviously a problem. But surely it is OK to discuss a racist speech in a First Amendment exam, and some unsavory details may well be relevant to how one deals with the problem. If the idea is that students are thus re-experiencing racism, what about an exam question that is obviously sympathetic to the idea that the Equal Protection Clause protects against invidious racial discrimination but that describes this situation, say by detailing a pattern of residential and educational discrimination? Would Williams, or others, think this situation is equally problematic? Similarly, I do not think rape should be off-limits for a criminal law exam, notwithstanding the obvious sensitivities of some students to these issues; indeed, doubtless *some* rape victims would rather know how to use the criminal law as a tool in such situations, and about the controversial and perhaps immoral limits in rape doctrine, than to be shielded from these issues. The way to deal with these issues sensitively, it seems to me, is not to avoid them or leave them off exams, but to be sensitive in introducing them. That is, when you come to the section on rape, acknowledge that it is sensitive, that some people have been victims of rape, and that the law's treatment of rape may not satisfy them or others, but suggest that it is important to understand how this area of the law works and warn that it is a potentially testable exam topic. Prepare students for the territory; don't skirt it.

Posted by: Paul Horwitz | Apr 14, 2009 1:24:10 PM

"My friend, a big Raider fan, was one of the few who got it right. The class was pretty pissed and the Prof was very defensive and didn't admit that was a bad question."

I might add that your professor must not have been to many Raiders games, because I can tell you that people dress up outside of the Black Hole. Thus, no reliance damages - the costume is just as valuable in another part of the stadium.

Posted by: Michael Risch | Apr 14, 2009 1:30:35 PM

Paul, thanks. It definitely is important to distinguish between the two types of biased exams. The problem with the type of exam question mentioned in the beginning was that students could be unfamiliar with the subject matter of the fact pattern. Conversely, the problem with at least some of the ones cited by Williams was that students could be too familiar with the subject matter. As an example, the problem with the overly detailed rape fact pattern is that a student who had been raped might have trouble taking the exam, but the basis for the trouble would not be lack of familiarity; indeed, this students might very well have more familiarity with the way that the legal system handles rape cases the most students.

Posted by: Colin Miller | Apr 14, 2009 1:57:19 PM

I am probably less sympathetic to this issue than some, but I would recommend that all who have foreign students attend to this post carefully. (And I agree that the XFL linebacker question is simply asking for trouble.)

I have used an example that is rife with movie and cultural references, though if you didn't get them, most wouldn't matter. I realized, however, that one would. The pattern involved a cop-killer bullet that had two different marketing campaigns. One was innocuous, the "Home Protect" version. Another was called "Oink", and invited customers to purchase if they "had some bacon that needed fryin.'" The Oink Spokesman was "Jules Winnfield, Account Executive." Few can forget Samuel L. Jackson's turn in Pulp Fiction.

However, few non-Americans understand "Oink" to be a sound characteristic of pigs (which is to say, cops). I had a couple of Puerto RIcan students, a native of Russia, and a woman from Amsterdam. Thank God, I realized the problem in advance, and was able to clue them in (without giving anything away) before the exam.

Posted by: Adam Scales | Apr 14, 2009 3:19:57 PM

Huh, Adam, I'm a native English speaker and I don't think I would have gotten that reference. Now I feel dumb.

Posted by: anon | Apr 14, 2009 4:53:00 PM

The problem is that some students who were victims of, say, rape, would probably rather just avoid criminal law and evidence entirely, given that some of the most interesting, important, and testable issues in those classes involve sexual assault. But given the way law school and the bar exam are set up, avoiding these classes is difficult or impossible. That said, beyond avoiding gratuitousness, I don't see how a professor can properly teach and test these classes while avoiding these subjects.

Posted by: jd | Apr 14, 2009 6:42:18 PM

Adam Scales said:
However, few non-Americans understand "Oink" to be a sound characteristic of pigs (which is to say, cops). I had a couple of Puerto RIcan students, a native of Russia, and a woman from Amsterdam.

Hey Adam, guess what? Puerto Ricans are Americans.

Posted by: anon | Apr 14, 2009 10:16:26 PM

I took Torts in England and ran into this problem all the time. The vocabulary of every day life--the very basics that even a child would know--were a mystery to me and caused much confusion and misunderstanding. One vivid memory concerns an accident which took place when someone jumped onto "a milk float." I pictured some incident during a "dairy day" parade when in fact a "milk float" is the truck (lorry) which delivers milk door to door. (at that time milk was not only delivered but "milk bottle theft" was the most common crime in Great Britain).

Another case involved a "skip" which turns out to be a dumpster. It went on and on.

As a Torts Professor now I try to avoid this by writing very concrete exams and do not attempt humor if it is at all likely to affect understanding of the facts.

Along these lines, I often have to stop the tutors we have attached to our first year classes from developing sports-related hypotheticals that indeed turn on knowledge of things like the intense rivalry between player x and player y as well as those that depend on knowledge of the relative distance between two cities in Texas (or anywhere else-this isn't geography class). It's just as easy to rephrase these questions by including the relevant information.

Posted by: Jennifer Bard | Apr 15, 2009 3:27:45 AM

I do my best to assume no background knowledge on factual matters. I've included maps and encyclopedia entries in my exam problems so students have background information that's important. (Of course, I frame the encyclopedia entry as something like "the client comes to you upset about this issue giving you this encyclopedia entry as the source of her anger" so that it seems less blatant that I'm just feeding them background knowledge from a simple source.) I try to avoid any reference to pop culture or sports, as I think it gives an unfair advantage to those who "get it." Even if it really is irrelevant to the problem, those who don't "get it" might feel that they are missing something major by not doing so and be put in a horrible state of mind while taking the test. I don't think it's my job to do that to students (beyond the stress that always accompanies an exam).

Posted by: David S. Cohen | Apr 15, 2009 9:47:24 AM

Post a comment