« The Duty of Law Professors to "Change Sh*t" -- Part Two | Main | A Reversal of Fortune for the Individual Mandate? »
Wednesday, September 07, 2011
Tough Tests, Take 3: Have You Ever Given, or Would You Ever Give, a Choose Your Own Adventure (e.g., Answer 4 out of These 6 Questions) Exam?
When I was watching the "How I Met Your Mother" episode "Mind the Gap" earlier this year, it got me thinking about the exams I give. The premise of the episode is that each person, based upon her particular life experiences, has a few glaring knowledge gaps. So, Barney doesn't know how to use a screwdriver. Ted doesn't know how to pronounce the word chameleon. And Robin is adamant in her belief that there's no actual North Pole. I imagine that the same goes for law students. Maybe Student A missed my lecture on the Best Evidence Rule. Maybe, no matter how hard Student B tries, she can't make heads or tails out of fee tails. And maybe Student C is unwilfully blind of the willful blindness doctrine.
Anyway, this got me thinking about my exams, which are 4 short essay questions (for Civil Procedure, Criminal Law, and Criminal Procedure) and 10 short(er) essay questions (for Evidence). Each student must answer each question. And while I give a practice midterm and practice final in each class, these are, in the immortal words of Allen Iverson, just PRACTICE and not graded, meaning that the final exam is "one exam to rule them all" (more about this later). And what this means is if Student C has a knowledge gap with regard to the willful blindness doctrine and 1 of my 4 essay questions deals heavily with the doctrine, Student C could receive a very poor grade despite being aces on the rest of the material in my class.
So, what the HIMYM episode led me to wonder is whether I should give...I don't even know what to call them...how about Choose Your Own Adventure exams? As an example, in a 4 essay exam class, I would give students 5 or 6 essay questions, and they would choose the 4 that they wanted to answer. As you can see from what I wrote before, I haven't found any scholarship on the subject and don't even know what to call this type of exam, so my thoughts on the subject are preliminary. Such exams would seem to help students who would otherwise be sunk by knowledge gaps. I also imagine that they would increase the overall quality of exams. That said, comparisons among students could be more difficult when, for example, Student A answers questions 1, 2, 3, and 4, and Student B answers questions 1, 2, 5, and 6 (more on this later). And students might feel even more time pressured if they had to read 5 or 6 exam questions rather than 4 (although I could add another 10-20 minutes to my exams). So, I would appreciate any thoughts that readers have on the subject. You can respond by answering the following poll and/or leaving a comment. Thanks.
-Colin Miller
Posted by Evidence ProfBlogger on September 7, 2011 at 08:37 AM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef015435391663970c
Listed below are links to weblogs that reference Tough Tests, Take 3: Have You Ever Given, or Would You Ever Give, a Choose Your Own Adventure (e.g., Answer 4 out of These 6 Questions) Exam?:
Comments
Orin, the problem of comparing students who answered different questions on the exam (Student A answers questions 1, 2, 3, and 4 while Student B answers questions 1, 2, 5 and 6) on the same curve is something I will address in a future post.
Posted by: Colin Miller | Sep 8, 2011 8:48:28 AM
I can't imagine giving this sort of test in a law school doctrinal course graded on a mandatory curve. The general point of law school exams in a doctrinal course is to see how well students can apply their knowledge relative to other students. Giving the students a choice of what to answer just makes that more difficult. You can't compare answers if the students are answering different questions, so it's hard to make the curve non-arbitrary. And if there are knowledge gaps, the student with the knowledge gap should do worse than other students; it's counter-productive to allow students to hide their knowledge gaps.
Of course, this means that professors need to make sure they test lots of different subjects on the exam so that the knowledge gaps reflected on the exam reflect the students' actual knowledge gaps. And the questions have to be ones that really go to the heart of the course, not some tangential topic. That can be quite hard to do. But I think that making it hard on the professor is much fairer to students than a "choose your own answer" test.
Posted by: Orin Kerr | Sep 8, 2011 12:30:14 AM
notaprof, thanks for the offer. Could you e-mail me them at [email protected]? Thanks.
Posted by: Colin Miller | Sep 7, 2011 8:16:22 PM
Sorry for the typos in that last paragraph - looks like the formatting was screwy on my computer!
Posted by: notaprof | Sep 7, 2011 8:14:21 PM
Professor Miller,
I'm a U.S. JD holder and practicing attorney who just completed an LL.M. at Cambridge in England. This format is standard there, but is far more drastic than what you describe. Exams are three hours long and must be handwritten. You are given an exam paper with 8-12 questions and instructed to answer 3, one per hour. To my knowledge, this is fairly standard at major British universities, including Oxford, KCL, UCL, LSE, etc.
Based on my experience with the British system, I would discourage you from adopting your proposed format. The British approach is designed to require students to read broadly and deeply in 3-5 areas (of 12-16 covered), while permitting them wholly to drop other areas that they find less useful or interesting. I found it to be pedagogically ineffective: students did not gain a sufficiently thorough overview of the entire field being taught. Students who invested the time to read ALL of the course materials were unable to demonstrate the breadth of the knowledge on their exams, and thus were not rewarded. Instead, the system rewarded last minute cramming: people who read on a few topics in the last few weeks (after months of full-time socializing) received marks similar to people who had put in sustained effort throughout the course.
You are proposing something in between the traditional American system (which requires mastery of the entire syllabus and directs students to answer each question asked) and the British system described above. Your proposal seems like it would lack the benefit of the (already-problematic, in my view) British approach - which at least encourages specialization in the form of "further reading" beyond the syllabus on the narrower set of topics that the student has chosen to prepare for exams. But your proposal also would lose the benefit of the American system - that the student is expected to bring a broad, generalist understanding of the field to the final exam. I feel somewhat nonplussed by your solicitude for students who have "knowledge gaps" - isn't the very point of exams to American-style exams disincentivize and penalize knowledge gaps?
(By the way, if you think it'd be useful to look at some examples of the British exams I described above, I'm happy to email them to you.)
Posted by: notaprof | Sep 7, 2011 8:10:28 PM
I find the "choose your own adventure" format very useful in my Intellectual Property Survey course. I am very mindful of obvious gaps there -- patent-focused students may lose the nuances in copyright law and so forth. I structure my exam to include a longer essay question (something like 40%) that is mandatory, and a short answer section (something like 60%) that allows students to choose 5 out of 8-10 questions. I spread questions as evenly as I can over the four subject matters (patent, copyright, trademark and trade secret), so that even if a student desires only patent questions (and these students do exist!), he/she still has to choose some other stuff to get to the 5. I was pleasantly surprised to get a good mix of choices - i.e., not the same 5 being picked by most students. I've come to recognize that if students avoid one question altogether, it's probably because I drafted a terrible question rather than lack of understanding on the part of the entire class of that topic. Of course, it requires a little more effort to come up with the choices, but I think it works in a class like the IP survey with such clear demarcations of subject matter and legal rules. Like Hanah above, the mandatory question gives me a baseline of comparison for the entire class and the short answer questions allow students to shine as best they can on small individual topics.
Posted by: Amelia Rinehart | Sep 7, 2011 5:37:41 PM
The format you propose is not unlike the essay section of the bar exam I took years ago. There were 10 questions and we had to answer 7 (or some such). Furthermore, this was true choice, as every question was on a substantive area of the law, so you could choose not to write in an area where you felt weak. Even conceding that the bar is a different animal, this doesn't strike me as unmanageable.
Posted by: hed | Sep 7, 2011 4:11:30 PM
This format is fairly standard in the humanities and social sciences, so your students will likely have seen it in college. Indeed, I don't think I ever sat for or gave an exam in history that did not follow this format.
I call it the "illusion of choice."
The trick I found when writing these is to test the same content different ways. E.g., in your ancient Greek history class, you really want to test knowledge of the political development of Athenian democracy because you realize it took up about a quarter of the instructional time.
So you ask a straight-forward question requiring comparison on contrast of Athenian, Spartan, and Corinthian political development.
You also ask what looks like a war studies question on why the Athenians lost the Peloponnesian War--but you assigned only two readings on this, one of which offered a political explanation, and lectured on this a lot.
Then you ask a "head-scratcher" on whether Achilles would have voted to convict or acquit Socrates--but the readings and in-class exercise presented Athenian courts as essentially an extension of the democracy.
Throw in one more question on the a secondary theme also echoed in three of the questions (here, it's likely on the importance of martial values) and offer a generous "pick two of four." But, really, you've just asked about Athenian politics and heroism--you just packaged it different ways and gave students some choices about the details they'll emphasize.
Posted by: Sykes Five | Sep 7, 2011 1:53:06 PM
Students may well have deficiencies, regardless of a lawprof's efforts, and they may well prove to be randomly demonstrated on tests. I would not, however, try to institutionalize such deficiencies as an acceptable failure by offering students alternatives. No judge or client will be so kind to them in the future.
If they think they could blow a course for failure to know a particular aspect of evidence (which, I might add, should be a required course for every law student), they will study as much as possible and, hopefully, suffer as few deficiencies as possible.
If, on the other hand, they know they will have a choice, they won't be bothered if things slip through the cracks. Don't count on the bar review, subsequent training or kismet to prepare them for the day they need to object. That's what law school is for.
Posted by: shg | Sep 7, 2011 11:22:26 AM
Christie, the problem at law schools is that we use blind grading. In other words, only an exam code accompanies a law student's exam, so a law professor does not know which student wrote which exam when she is grading them. So, if Student A comes to me before the exam and explains why she is having trouble with a subject, I can't take that into account when I am grading her exam...because I don't know that it is her exam. This is one reason why I don't test the Rape Shield Rule on exams. In a future post, I will talk about what I see as the uncomfortable relationship between blind grading and the "1 final exam" model in law school.
Posted by: Colin Miller | Sep 7, 2011 10:50:26 AM
shq, that's a good question, and I think that Hanah's policy is based upon a similar concern. My hope would be that while the student didn't know the Rule on the exam, she would learn it while studying for the bar or while preparing to litigate a case that involved a Best Evidence issue.
I guess that the main point that I'm trying to make is that is follows. I think that we all agree on the following: (1) There is no way to test everything covered in a course on an exam (and most law school classes only have 1 exam); and (2) The vast majority of students have some knowledge gap. Sure, there might be a few students who know all the material cold, but from what I've seen, almost every student has at least a deficiency here or there. But maybe Student A's deficiency is a large part of my exam 1 semester while Student B's deficiency isn't on the exam at.
With just 1 exam in a class, there is already a risk of randomness. I think that a "Choose Your Own Adventure" exam might reduce that randomness, but it also possibly creates some additional problems.
Posted by: Colin Miller | Sep 7, 2011 10:42:06 AM
From a student's perspective, it feels awful to discover you don't know a subject as well as you thought you did. It feels even worse when that is the only test you'll be given that semester and so much is riding on it. I would rather have a few choices so I can show off what I have learned thoroughly than answer a question poorly and leave my professor with a bad taste in his/her mouth. Do you take into account the reason the student missed a lecture? And whether or not said student sought you out afterwards to go over the concepts presented?
Posted by: Christie | Sep 7, 2011 10:32:37 AM
TV shows are such great ways to come up with new test ideas. But what do you tell your student's client when he fails to make the "best evidence" objection at trial that would have clinched the case?
Posted by: shg | Sep 7, 2011 9:16:42 AM
I don't allow choices on the "substantive knowledge" portions of my exams, because I think that it would be wrong to give an A to someone with a knowledge gap that's relevant to the class's basic subject matter. Letting students skip their weakest areas creates that possibility -- even likelihood.
However, I do sometimes give choices on "policy" or "theory" questions on my exams. The idea with these questions is just to convince me that the student has spent some time thinking about the subject rather than just memorizing the black letter law. So I think it's totally fair to give them either a very open-ended question or a choice of more targeted questions.
My exams thus follow a format like this: Answer all of Part A, then answer one of the three questions in Part B.
Posted by: Hanah | Sep 7, 2011 8:49:56 AM
The comments to this entry are closed.