« Law School Transparency Petition | Main | Responding to Rob Vischer's Review of The Agnostic Age »

Thursday, September 15, 2011

Tough Tests, Take 4: What Do You Do When a Student Misinterprets a Key Term in an Exam Question?

In my first guest post, I blogged about what I thought was a difficult question: Do you ever bump the grades of students who run out of time on your exams? Well, it turns out that it wasn't so tough. A resounding 95% of respondents answered "no." In the comments section, though, Professor Michael Froomkin (the founder of the terrific JOTWELL...seriously, check it out if you haven't already), posed a trickier question: 

A hard question for me is a student who doesn't know the meaning of somewhat but not utterly common non-legal word that I happen to use in a question, a dictionary word that is not a legal term, guesses wrong, and thus writes an essay that misses the point. This happened once, early in my teaching career (and I am sorry to say I don't even remember now what the word was, but I do remember it was not a foreign student).

I touched upon this question a bit during an earlier guest blogging stint in a post entitled, "Avoiding a Biased Exam," which was about...well, avoiding a biased exam, e.g., not using terms that could confuse certain students and not using fact patterns that could disturb certain students. But what if, despite your best efforts, a student gets confused?

Thankfully, I haven't faced this issue yet, but I might have if not for my wife. I wrote a Criminal Procedure exam question in which officers arrested a suspect in his bedroom and then searched an armoire in the suspect's bedroom and uncovered weapons, leading to a protective sweep of the house, which uncovered contraband. The call of the question asked whether the search of the armoire and/or the search of the rest of the house were constitutional.

The correct answer (in my mind) was that the search of the armoire was justified under Buie v. Maryland, 494 U.S. 325 (1990), which allows officers, incident to a lawful arrest, to "look in closets and other spaces immediately adjoining the place of arrest from which an attack could be launched." The officers' discovery of the weapons in turn allowed them to conduct the protective sweep of the rest of the house pursuant to another part of Buie, which allows searches of areas beyond the spaces adjoining the arrest room when there are "articulable facts which, taken together with the rational inferences from those facts, would warrant a reasonably prudent officer in believing that the area to be swept harbors an individual posing a danger to those on the arrest scene." 

In reaching this conclusion, I had in mind the common definitions of an armoire: a large wardrobe..., a large, ornate cabinet or wardrobe, a usually tall cupboard or wardrobe, etc. Under these common definitions, it seems clear that an armoire is a space big enough such that a person could be hiding in it ready to launch an immediate attack. But what if a student didn't know the definition of an armoire and thought it was a smaller piece of furniture? What if the student thought that an armoire was something entirely different than what it was? What if the student had in mind a smaller armoire, in which a person could not (easily) hide?

In this case, how would I grade Students A, B, and C? Student A finds that the search of the armoire and the protective sweep were warranted. She correctly applied the law to the facts and reached what I regard as the right conclusion. Student B's response is that the armoire was too small to conceal a person, so the search of the armoire was not authorized under Buie, meaning that the search of the rest of the house was illegal. She doesn't discuss the test for determining the validity of a protective sweep because she found the initial search was unconstitutional. Student C doesn't even address Buie in her answer.

Student A obviously gets full credit. But what about Student B? I think she gets full credit for her answer regarding the search of the armoire because she correctly applied the law to (her interpretation of) the facts. But does she get the same credit for her (non)discussion of the protective sweep of the house because her analysis and conclusion were correct if the search of the armoire was unconstitutional? Or does she only get partial credit because she didn't discuss the standards governing protective sweeps (but why would she?).

And what about Student C? The obvious problem here is that we don't know what Student C was or was not thinking. Maybe she envisioned an armoire as a huge dresser but blanked on Buie. On the other hand, maybe she envisioned an armoire as a tiny dresser or something else incapable of concealing a person. So, do I give Student C no credit? If there are a few students like Student B, do I recognize that Student C could have been similarly confused and decided it wasn't worth it to mention Buie because it was obviously inapplicable. And, if so, do I give her partial credit? Full credit? At some point, do I just recognize that the question was too ambiguous and throw it out? And if so, what is that point?

Luckily, my wife taught me a valuable lesson before I gave the exam, which is that clarity is key. In the question, I changed the armoire to a "large dresser," even giving the students its height, width, and depth. That said, I'm sure the day will come when I use an ambiguous term and get ambiguous results. And then, what do I do? What do you do?

-Colin Miller

Posted by Evidence ProfBlogger on September 15, 2011 at 09:43 AM | Permalink

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef01543558e92f970c

Listed below are links to weblogs that reference Tough Tests, Take 4: What Do You Do When a Student Misinterprets a Key Term in an Exam Question?:

Comments

In a variation of this problem, a fellow professor once drafted a Torts exam with a long hypo involving a "retired welder." The student read it as a "retarded welder," which led to all kinds of issues the professor did not intend to raise. The professor ended up awarding the student an A- because she wrote an excellent answer to the "retarded welder" hypo but did not give her an outright A because of her error in reading the question. I think we all balance the urge to be fair to the student who made an "honest" mistake or had an honest misunderstanding with the desire to adequately reward the student who made no such mistake or had no such misunderstanding. I really appreciate this discussion today because I am supposed to be drafting a midterm exam instead of perusing blog posts.

Posted by: Heidi Anderson | Sep 16, 2011 10:30:33 AM

I had one terrible example of this occur in one of my Sales exams years ago when I included a form document in the question that had an "/s/" at the bottom of the form above the name. A number of students said that the writing wasn't signed because the individual had just written "/s/" instead of signing. I was floored, but I had to admit that I couldn't remember advising them of the meaning of this symbol during the course and so I assumed their base of knowledge in analyzing the answer. I haven't made that mistake again, particularly now that it is easy to "sign" a form in the exam and create a PDF. I think the key is that you have to admit that your exam can create different lines of reasoning based on such (unintended, and surprising) factors, and grade accordingly. For those Sales buffs, yes I did give credit to those who recognized that signing is not necessarily a signature, but could be any symbol meant to authenticate the document.

Posted by: Jay Mootz | Sep 16, 2011 1:53:17 AM

Interesting problem.

First, it's a good reminder of how easy it is to write an ambiguous question. If you'd asked me, before reading this, to define "armoire", I'd probably have gotten it wrong - I had only a vague sense of heavy, clothing-related furniture. I've occasionally found myself feverishly editing unintended ambiguities out of exams. In one recent exam, I realized that many students did not know how to convert kilometers to miles, as would be required to complete the analysis. Since I wasn't testing them on their knowledge of the metric system, I emailed the class the formula (it was a take-home exam).

I usually tell my students to explain the reasonable basis for their assumptions about ambiguous facts, assuring them that they won't be penalized. That approach seems inadequate in two respects. First, a reasonable student might be unaware of the ambiguity (here, by entertaining a smaller notion of "armoire"), and second, I'm not entirely sure I'd give Student B the same credit as Student A here. I imagine it would be awfully tempting to give Student B (for example) a score of "Good" on her answer, while giving A - the mindreader/former antiques dealer - a score of "Excellent". This is troubling, because it implies that the question is partially, but not completely defective for ambiguity. Of course, to throw it out (making A & B equal winners) may create an unhealthful compression of grades (as there will be lots of Student As and Student Bs).

This also complicates the grading of Student C. If the question is really defective, then I suppose C should get a break - one that may conceal C's unrelated poor understanding. On this particular question, it seems strange not to mention Buie at all, so perhaps penalizing C is appropriate. But, there will be other, closer questions.

As to Tim's point, I'm pretty sure that what you describe doesn't describe how I grade exams, but it may describe other approaches. And here, it seems debatable whether all armoires could conceal someone, so addressing both sides makes some sense. However, I don't think that what you described can be completely segregated from the real world. Every brief I've read contains about 8-10 arguments that converge, however redundantly, on the conclusion that the brief writer should win. Many of these arguments are put in the alternative ("if the product wasn't defective, we win; if not defective, we still win due to comparative fault, etc.") Analysis stops, it seems, when the writer runs up against the page limits imposed by the court.

Posted by: Adam Scales | Sep 15, 2011 1:57:47 PM

I've done the picture thing when I've thought a term was potentially unknown but when I didn't want to change it (a graphic of a dirigible, for instance, in the question packet for a torts exam).

But as you note the big risk is not realizing the term isn't widely known. So another time I had a "resident" who worked for a hospital involved in a car crash on the way home. Some students told me afterwards that they didn't know that a "resident" was a physician. I suppose I've just watched too much Scrubs, but it never occured to me that this term would be confusing.

Whenever questions like this come up (before an exam), I advise students that when there is uncertainty they should state their assumptions and proceed accordingly. So a student might write, "I have no idea what a dirigible is but assuming it is a giant hair dryer the answer to the question is ..." This helps the student move themselves from being paper C in your example to B.

Posted by: Geoff | Sep 15, 2011 1:49:55 PM

Jennifer: That's a good point. It seems as if even the question I asked could have been ambiguous. I agree on Erie questions. I often find myself avoiding them on the final or at least de-emphasizing them based upon how difficult it can be to test them on an exam taken by 1Ls.

Tim S.: I appreciate that perspective. This is part of the reason that I have been gradually transforming my exams into short(er) essay questions from bigger issue spotters. If you have 1 or 2 big questions with several interconnected issues each, 1 error or misconception can doom the exam. If you have 10 questions with 1 or 2 issues each, this isn't as big of an issue.

Adam: That's true, but I see a big difference between a criminal defense lawyer missing issues over the course of a length representation of a client and a hurried law student hastily making an incorrect factual assumption on a 3-4 hour final exam.

Michael: That's a great idea about including a photo. That would certainly go a long way toward making the question less ambiguous.

Posted by: Colin Miller | Sep 15, 2011 11:47:38 AM

Josh, thanks for the comment. When I test on statutory ambiguities, I usually find cases where litigants successfully or unsuccessfully challenged statutory language. For instance, on my Criminal Law exam last semester, I had the owner of a liquor store trying to get his ABC permit renewed and allegedly bribing an Excise officer. I found a case, State v. Dugan, 793 N.E.2d 1034 (Ind. 2003), where a litigant challenged a provision that made it a crime for “a person applying for or receiving an ABC permit to offer an officer of the…Excise Police…a gratuity, commission, or profit of any kind.” I just took that language and plugged it into the exam.

Posted by: Colin Miller | Sep 15, 2011 11:36:52 AM

Colin,

Have you considered just including a photo of the "large dresser" or other relevant objects/spaces with your exams? It might help reduce some ambiguity or misunderstandings.

Josh,

As a first year professor teaching Legislation, I am already concerned about striking the right balance in the exam. Any tips would be greatly appreciated!

Posted by: Michael Teter | Sep 15, 2011 11:00:49 AM

What would a judge do with a brief filed by C? Would C's client have been well-represented? Would someone paying C have a claim against C?

I'm not a lawyer, but I've been poorly represented. So I think your question turns on how the student's mistake would hurt in the real world.

Posted by: Adam | Sep 15, 2011 10:42:04 AM

From a former law student, my perspective is that this is just one of the ways law school is different from the practice of law. In the real world, the analysis stops when you reach the clearly correct answer. In a law school exam, the analysis stops when (time permitting) you have covered all the topics covered in the class.

Thus, law school exams reward answers of the form: The result is A because (application of doctrine X). But if the answer is not A, the issue is B, with result C because (application of doctrine Z).

Just keep branching until you run out of time. What felt unfair for me as a student was when the professor expected more branches on question 3, while I thought there were more branches on questions 2 & 4. That said, I didn't really grok this branching technique until fairly late in law school, so my grades would definitely have been better earlier if I'd applied the branching technique to more of my exams.

Posted by: Tim S | Sep 15, 2011 10:23:13 AM

This is a recurring problem in civil procedure, where the underlying facts often involve some kind of torts or contracts claim, and students are sometimes thrown off by some aspect of the substantive law. Even more so when you are asking an Erie question, so there has to be some state law floating around somewhere.

FWIW, I would have understood an "armoire" to be a wardrobe, as you did. But in my mind, a "dresser" is a chest of drawers, so even a large dresser would not have space to conceal a person. (But a quick search on OED online says that a "dresser" is a sideboard! -- i.e., a table in the kitchen or dining room on which food is dressed or from which food is served.)

Posted by: Jennifer Hendricks | Sep 15, 2011 10:22:17 AM

I actually have the opposite problem. In writing my Legislation exams, I'm sometimes trying to get enough ambiguity into the question, because I want them to apply the various interpretive techniques they've learned. It can actually be kind of hard to write ambiguous statutory language (at least, it can be when you're trying to channel the ambiguity, so that you're not just writing nonsense).

On the broader point, I always tell my students that if some fact is missing, they should tell me what fact it is, why it is important, and how the various options for that fact would affect their resolution of the legal issue. That might take care of a student who knows that she doesn't know how big an armoire is, but of course it doesn't help with the student who thinks he knows how big it is but is wrong.

Posted by: Josh Chafetz | Sep 15, 2011 10:09:09 AM

Post a comment