« On the Brilliance of People like Judge Sonia Sotomayor and Barack Obama | Main | Ridi Pagliaccio »

Tuesday, May 05, 2009

Bad Testing

I'm delighted to be back for another stint blogging.  I'm a little slow off the mark this month, due in part to participating in Prawfs! 5 at FSU last week (though it hasn't stopped Paul and Rob, two of my fellow participants).  But I've also been writing -- OK, rewriting and worrying about -- my final exam.  As much as we dislike the fact, testing and grading are two of the most important things we do; they matter enormously for our students, and for that reason we need to get them as right as we can.  Hence, at least for me, the constant fine-tuning (to put an optimistic spin on it).

So here's my question, for current and former students: what did you hate most about a final exam you had in law school?  (For profs, what's the biggest mistake you have made in writing an exam?)  I'd love to hear comments more specific than "the exam was too long" or "it didn't test on what the the professor stressed in class," though precise examples of those faults are welcome, too.  Sloppy drafting?  Unreasonable expectations?  Politically-biased questions?  Distracting fact patterns?  I know some of this has been aired with a few recent posts about particular exam-writing and grading issues, but I'd like to throw the topic open for a more general discussion, especially since exams are presumably on everyone's minds now.  It may be too late for comments to affect how exams are written this time around, but maybe something will stick when we sit down in November and draft our next round.

Posted by Bill Araiza on May 5, 2009 at 11:18 PM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Bad Testing:


I much appreciate the responses (and I assume that profs reading them do as well). Without trying to answer every point, I will say a few things.
1. Of course the complaints about a teaching/testing dichotomy are right. In fact this is pretty basic. Presumably we teach what we teach because we think it's important enough to spend time on. If so, then we need to grade based on whether the student has mastered that important material. It's unfortunate that such a basic lesson is apparently so unheeded.
2. I think Jeff's point about multiple testing styles is well-taken, but for reasons that go even beyond his concerns about lack of coverage via standard issue-spotting hypos (and by the way, I completely agree with the undesirability of long fact-spotters that twist and turn to pick up disparate topics). It seems to me that students need to know material at different levels: they need to know the rule and how to apply it, they need to be able to spot issues raised by a fact pattern, and they need to understand larger themes. Thus, I tend to test by asking different types of questions, including multiple choice (for coverage and simple rule knowledge reasons), standard issue-spotters and more conceptual questions. I respect Paul Horwitz's thought, in his separate posting, that you can test deeper concepts via docrinal questions, but because I think a diversity of question styles is a good thing in and of itself I tend to test that knowledge explicitly, via its own type of question: that kind of diversity picks up different kinds of knowledge and also helps ensure that a student is not penalized just because, say, he's a slow writer or a bad multiple choice test-taker.
3. More generally, I agree that law professors tend to know more about the "law" part than the "professor" part. As it applies to testing, I think most professors would be well-served by listening to an education expert talk about how to design effective tests. I've heard of a few schools doing this; I think more of it would be a good use of our time.

Posted by: Bill Araiza | May 7, 2009 3:24:58 PM

The worst exams I've had were poorly-designed multiple-choice tests. Designing a good multiple choice exam is an art, one in which not many professors are skilled. The biggest challenge is writing unambiguous questions and answers. There are few things as frustrating to me as an exam-taker as when two multiple-choice answers seem like they could be correct depending on ambiguities in the question or the answers, and I have absolutely no way to explain why I'm choosing the answer I do. Combine that with some professors' inability to refrain from using trick questions, and the result is a mess. It can be the worst form of a "read my mind" exam.

Note that I've also had a very good multiple-choice exam. The questions were unambiguous, the correct answers were clear if you really knew the material, the potential answers were selected with a plan in mind (two clearly wrong, one tempting but wrong, and one right, to separate people who didn't know, mostly knew, and really knew the material), and the professor didn't use trick questions (which is where ambiguity usually pops up). Unfortunately, that well-designed of a multiple-choice exam has been very much the exception in my experience, not the rule.

Posted by: Yet Another Anon | May 7, 2009 1:14:06 PM

Parties should have clearly distinguishable and easily abbreviatable names. I took a T&E exam with husbands named Hector and Harry, children named Juanita and Jose and nieces named Nelle and Nicole (I think there were actually more). I appreciate the alliterative humor but it was impossible to keep track of in a complicated fact pattern. Fun names and facts are fine but they shouldn't distract from the exam.

Posted by: almost finished 3L | May 6, 2009 11:32:50 PM

My CIv Pro II exam this semester was awful. It was our last exam so we were all exhausted but felt like we understood the material. The first part of the exam was an erie issue but it was impossible to figure out what federal rule we should use for our analysis. Everyone I spoke to in the class lost serious time trying to figure it out and then panicked for the rest of the exam and after a week we still have no idea which rule we should have used.

I know our prof. did not intend to screw us up that way, she isn't that type of professor but by not giving us the rule to go with the issue it took away from the important part of the exam- the analysis and she will have a class full of crappy exams.

Posted by: Civ Pro II | May 6, 2009 11:13:11 PM

And I agree with Jennifer Hendricks - the reason I hate time travel question is because they make analysis less important by confining you to a particular snapshot of history. I think that was a particular problem re: education law because most of the cases are not technically overruled, just changed over time so some of the arguments are still available and certainly still relevant.

Also really tight word limits (500 words or less in a question that gets about an hour or more of time given the overall time limits and word limits) seem to be asking for "guess what's in my head" answers. Even worse are short answer questions limited to fewer than ten words where the actual answer the professor has in mind is exactly that number of words (and is not a term of art). That's just basically a trick question. Might be fair if those kinds of short Q&A are common in the class as it is being taught, but not as a general rule.

Posted by: Anon 3L | May 6, 2009 10:26:42 PM

I agree with anon at 8:56:33 -- the people asking for "time travel" questions are missing the point of learning the history. Yes, I am looking for an additional level of analysis!

Posted by: Jennifer Hendricks | May 6, 2009 9:24:33 PM

To dmv and those who agreed, current doctrine was born out of that history and sociology and thus could still be affected by relevant arguments invoking it. This is particularly true in a law school exam where your case is likely to be reviewed by the Supreme Court! Your professors are probably looking for that additional level of analysis in your discussion of the fact pattern and doctrine.

Posted by: anon | May 6, 2009 8:56:33 PM

My greatest concern about past exams as a student stems, I think, from many law professors having core competency in the 'law' and not necessarily the 'professor'. Specifically, this arises most in exams with the more closed-end questions. Prototypically, multiple-choice tests seem to be the most perilous. Because they leave little opportunity for students to 'write around' any perceived omissions or errors in the question, a poorly-written M/C question can be deadly (to performance and to morale on the exam).

More specifically, the common problems I see are unstated assumptions or missing facts in the question; or distractor answers that also are potentially correct (in a select-one format). One professor who had previously taught in a graduate program of another discipline but was in his first law-prof year ameliorated this a bit by allowing students to handwrite any additional assumptions or confusion on the exam next to the question. This allowed post hoc fixing, to some degree. It seemed to work OK--though it's still not a great substitute for a bad question.

My hope is that more law schools will step-up new-prawf programs that focus on the aspects of teaching and pedagogy, hopefully in addition to (and not replacing) focus faculty colloquia on legal developments. Of course, old-prawfs might benefit, too...

Posted by: Aaron | May 6, 2009 5:05:02 PM

Proofread, proofread, and proofread. Nothing is as horrible as a character name change halfway through, transposed dates, etc.

In any given fact pattern, do not give two characters names that start with the same letter. If, on a property or trust and estates exam, O has two daughters Ann and Amy, and the ultimate resolution depends on which property interest either or both wound up with, you have just invited your students to make really silly mistakes, particularly if there is a time limit on the exam.

Posted by: Jonathan | May 6, 2009 3:01:25 PM

There appears to be a bit of a theme here of "test what you teach" despite Bill's request not to dwell on this theme. (In fairness, people are being fairly specific in their complaints, as he requested). I think that this theme persists because profs continue to violate this axiom.

This was one of my biggest complaints from law school test taking - there was a large disconnect - both in approach and content - between the class and the test. I can remember tests that covered *at most* 30 to 40% of what was covered in class - some were closer to 25%. This not only introduces a degree of inaccuracy into the evaluation process, but is also very frustrating for students.

Perhaps this problem is exacerbated by fact pattern/long essay testing in which it can be hard to weave the entire class into a scenario. If so, then profs should make tests more holistic, incorporating multiple question types. In similar fashion, if you are going to obsess about case minutia a la an inept Kingsfield impersonation (Petit's complaint), then your test should be geared toward such an approach (obsession over case minutia? That would just make the test as lame as the class, which isn't good either I suppose).

Posted by: Jeff Yates | May 6, 2009 2:08:45 PM

If you're going to reuse exam questions on your open book exams, make an effort not to accidentally post them on the course website in a way that makes it obvious you're likely to reuse it and thus prompts the half of the class that noticed it to prepare answers ahead of time.

Posted by: anon | May 6, 2009 11:45:24 AM

As a counterweight, I'd like to point out something that one of my 1L professors did. On the first day of class, he handed out a set of 60-odd fact sets that formed the core of the discussion for the class, and were in the same style as used for the multiple-choice portion of the exam. The real point is that these were all real-world fact sets that would remain valid for years, even as the law in that field changed... largely due to that professor's influence.

The course: Criminal Procedure
The professor: Wayne R. LaFave
The exam: The hardest exam I had in law school (and my lowest first-semester grade!)... but the exam also worked as a teaching tool, not just an evaluation tool

The contrast between this and the exam in Another Class (same semester, in fact) from a visitor was just appalling, which leads to my on-thread point: If one intends to test using short-answer and/or multiple-choice questions -- even in part -- then one's teaching style throughout the semester had bloody well better prepare students to respond in that fashion, instead of being confined to day-after-day inept imitations of Professor Kingsfield from The Paper Chase.

Posted by: C.E. Petit | May 6, 2009 11:41:41 AM

This semester I have a professor who told us flat out that his exams can't be finished during the 3 hours for the exam and he doesn't care. We just have to deal with it, and that kind of unwillingness to draft a good exam is infuriating.

Posted by: stephanie | May 6, 2009 11:06:06 AM

The most frustrating exam I took was an admin exam that only asked us to create a governance structure to help several states work together on an environmental and natural resources problem. There were not other questions and we have never once talked specifically about different governance structures in that class. Of course in an admin class some governance issues had arisen, but diffusely. We certainly had not really talked about such an applied version of admin - and we were not given much, if any, of the relevant statutes, rules, and policies in the exam itself. Most people spent a good twenty minutes reading and re-reading the questions, hoping they could find a hook of something we learned in class to start writing. It was particularly frustrating since the professor also taught a governance class that spent the entire semester essentially answering the question on our admin exam, and one student was taking both classes.

I actually really dislike the time-travel questions, a big feature of an education law final I took. While it would have been nearly impossible to teach education law without walking through the history of desegregation cases, it seemed pointless to have us apply old law that is arguably no longer good law, especially at the expense of testing the full breadth of material covered in the course (which was not a particularly wide swath of education law, it would have been easy to fit it all in).

Posted by: Anon 3L | May 6, 2009 9:14:25 AM

I'll agree with dmv also. Law professors often don't do a very good job explaining/demonstrating what the exams will be like -- mostly applying current rules to facts -- and the classes give the impression there will be much more policy and even history. I think it's especially important in first year classes to let the students know what the exams will actually test on. You can use practice tests, mid-terms, make available old finals, etc.

Don't get me wrong, I don't buy that there is some complete disconnect between "the law" on one hand and "policy and history" on the other. And I test on policy ("should this case/statute be overturned? -- give me the best arguments on both sides") in upper level classes. But students need to know what will be expected of them on exams.

Posted by: Joseph Slater | May 6, 2009 8:17:49 AM

An anti-trust exam that was 85% multiple choice/true false.

Posted by: Jones | May 6, 2009 2:00:36 AM

I agree with dmv quite a bit. My specific complaint, or perhaps it is just a suggestion, is that almost all casebooks and professors spend some time talking about the history of the subject and that is rarely tested on. Yet it would seem relatively easy to create "time travel" questions.
As an example my antitrust class, which I have not yet taken the exam for but it is on my mind, talked a good deal about the development of doctrines. It would seem to be good fodder for at least a short answer question to ask about vertical restraints in 1965 and 2009. Generally the history is not deep enough to allow an entire essay question to be asked about history, but maybe it is for some classes.
It seems to me that most classes that deal with primarily constitutional law could ask more about history as well. The danger with not asking about history is that students are likely to ignore all of it and have an attitude of "just give me the rule." Asking history questions, and telling the students that you do, would seem to me to be a good way to make students more interested in getting beyond what they could have learned in a Gilbert's.

Posted by: Ben | May 6, 2009 12:03:20 AM

I'll throw this out there for now, because it's immediately
relevant to me (as in, immediately), so it's bugging me:
the lack of consonance between teaching and testing.

Example: professor spent large amounts of time in the class
throughout the semester talking about the historical, political,
sociological, etc. background to the relevant area of law.
It was actually pretty good, insofar as it helps to see the
social and political context shaping legal development, and
vice-versa (at times). Yet, the exam is wholly doctrinal. It's
strictly fact-patterns calling for analysis. The disconnect
between what the professor focused on throughout the semester
and what is on this exam is insane.

If you're going to test doctrine and analysis, use the class to
do doctrinal exposition and analysis through hypos. If you're
going to spend the semester going over how the law has been
shaped over time, tailor your exam accordingly. I'm not saying
only teach as you test, strictly and with no deviation. But it's
absurd not to have an exam match, generally speaking, the
content and the contours of the course.


Posted by: dmv | May 5, 2009 11:54:26 PM

My Contracts exam last semester was particularly frustrating. At my school Contracts is offered as a five credit - one semester course. Unfortunately we only had a few days at the end of the semester to get through all the types of standard remedies and damages, and our professor chose to basically PowerPoint the class through this section (he never used PP the whole semester until remedies and damages). Of course, both essay questions incorporated detailed parts regarding remedies. My advice - if profs are going to speed through a topic, then either don't include it on the exam at all, or save it for one or two multiple choice or short answer questions.

Posted by: Anon Student | May 5, 2009 11:33:22 PM

The comments to this entry are closed.