« Demographics and Distrust: The Eleventh Circuit on Graduation Prayer in Adler v. Duval County | Main | Valuing Deaths Differently »

Monday, April 06, 2009

The "Open Everything" Exam: Why (or Why Not) to Choose it Over the Closed Book, Open Book, or Partial Open Book Exam

Before teaching my first class, I knew that the first question I had to answer before even choosing the casebook or deciding which topics to cover was what type of exam I should give.  And central to that question was whether that exam would be closed book, open book, or partial open book (students can only use materials that they had a hand in preparing, students can use rule books but not supplements, etc.).

The choices were many and complicated, but I felt that my answer was simple: I would give an open book exam in every sense of the word (open book, open note, open commercial outline), i.e., an "open everything" exam. The reason? Cheating. Now, I have no idea what percentage of law students cheat. (According to recent research, "seventy percent of high school and college students admit to having engaged in some form of cheating, and...forty-five percent of law students admit to having cheated."). What I do know is that at every level of school I attended starting with junior high, students were worried that other students were cheating and that they would be graded unfairly as a result.  Of course, in the curve-happy world of law school where so much depends on grades, these concerns were heightened.

An "open everything" exam doesn't eliminate these concerns, but it does greatly reduce them.  With an "open everything" exam, students can still cheat off of each other, but they can't cheat out of the casebook, an outline, a rule book, or a hornbook because everyone has access to these. Based upon feedback, this approach greatly reduces student stress about other students cheating.  It also has other things to recommend it.

First, it also reduces student stress about getting stuck in a moment during the exam and seeing their grade failing them along with their memory.  Second, I think that it improves exam performance compared to the closed book exam. After closed book exams in law school, I remember post-exam discussions focusing on what was remembered vs. forgotten instead of who best applied fact (pattern) to law.  Also, from my own experience, I remember "wasting" several minutes on closed book exams doing the brain dump (i.e., hastily jotting everything down that I might otherwise forget once the proctor told us to put our materials away). Third, I think that students in open book classes are more proactive in creating their own outlines because they know that they can use them on the exam. And from what I have heard from students, this not only helps them on the exam but also on the bar and in practice. 

Of course, there are some drawbacks to the "open everything" exam.  First, some say that it levels the playing field too much.  In other words, students who lollygag all semester can catch up to the hard workers in a way that can't occur in a closed book class. This is a legitimate concern, but it seems to me that it can be alleviated to a great extent by exam length.  I try to make it so that students who know the material and only check their materials a few times will be able to finish the exam with no time to waste. Indeed, I tell students that their materials should only be used as crutch for a forgotten point, not as a constant point of reference.

Second, some say that it creates an arms race for high priced supplements.  In my classes, however, I make sure to dig much deeper than those surface supplements and even teach opinions that are only a few weeks old. I always tell students that if they rely on supplements, they will not do very well in my classes, and as far as I can tell, that advice has rung true.

Third, some say that it is unfair because certain organizations such as law review have "outline banks," which make the "rich" richer and the "poor" poorer.  I think that this is a valid concern, but my decision was to nip it in the bud by preparing and giving students outlines throughout class and a comprehensive outline at the end of class. On the one hand, I realize that this might discourage students from making their own outlines, nullifying the third benefit listed above. On the other hand, before I gave outlines, I had several students asking me to review their outlines, and many of them were put together in a manner that didn't make (enough) sense.  My conclusion was that giving out outlines was the lesser of two evils, but based upon student feedback, my choice might not be an evil at all.  I have heard from several students that they are intimidated by the prospect of preparing outlines from scratch in other classes but that they were were able to take what I gave them and make it their own in a way that was helpful and not overly time consuming (Indeed, this is what I tell students that they should do).
Fourth, some see law school exams as bar preparation, and obviously the bar exam is closed book. I also see law school exams as bar preparation, but I just don't see law school as the time to require closed book knowledge.  I see law school as laying the groundwork for the bar exam and, ultimately, the practice of law.  With a closed book exam, it seems to me that students spend most of their exam preparation time memorizing.  With an "open everything" exam, it seems to me that students spend most of their exam preparation time working on their outlines and applying those outlines and the law to hypotheticals and practice exams (and many students have told me that they later use those outlines for bar prep).
But, of course, as I noted before, I am new to this game and still testing the waters.  What type of tester are you? Closed book? Open book? Partial open book? Open everything?  And how did you arrive at that type of test, and what do you see as its pros and cons? 

Posted by Evidence ProfBlogger on April 6, 2009 at 10:11 AM in Teaching Law | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference The "Open Everything" Exam: Why (or Why Not) to Choose it Over the Closed Book, Open Book, or Partial Open Book Exam:


I'm with Jeff almost 100%. I make my exams very difficult - information dumping is easily recognizable all of the time. I am sure there are slackers, and I make clear to my classes that if you are ready for the exam and treat it like it were in-class, you could do it in 4 hours.

As for one of the comments that the 8 hour hurts the non-traditional student - that is a concern of mine, and that's why I don't do 24 hour exams and that's why I have word limits. David Cohen makes good points. If people spend the entire 8 hours, it's because they don't know the material well enough - you can do it in much less time and in fewer words than the limit. Also, drive time to the school, getting there early, going home, etc. eats up 5 or so hours of the day - adding another 3 is not too much to ask, especially because I let them choose the day and time they want to take it - first day of finals or last, 8AM or midnight.

Finally, I did a study of my exam answers, and other than one outlier (a truly outstanding answer), once the answer hit 3000 words or so, there was no correlation between length and score. Also, many of the top scorers on my exams finish in 6 hours or so.

Posted by: Michael Risch | Apr 10, 2009 1:46:30 PM

I am a firm believer in open-everything 8-hour take-home exams with word limits for answers. As Orin Kerr said, there are very few instances in legal practice where you can't consult a source for information to help you with an issue. Being a lawyer just isn't about memorization. If we're training lawyers rather than bar takers, I just don't understand a closed book exam at all.

The 8-hour take-home is a good compromise in my mind. First, to address one comment, our computer system takes care of making sure no one spends more than 8 hours on the exam. The word limit takes care of making sure that no student spends the entire 8 hours writing rather than thinking through problems then writing. And the take-home and the length of time make sure that I get well thought out answers that are organized and not just brain dumps. I also don't want to reward speed over analysis and thoughtfulness.

If this "levels the playing field" too much, then so be it. Although, I haven't yet found it does. I still get a curve, no matter how long or short the exam is. And, if someone knows the material but is better at giving me a great answer in 8 hours rather than 3, what value would I be testing if I wanted to reward only those who can spit answers out in 3 hours? I don't get it. (I don't go longer than 8 hours for a variety of reasons, some of which were touched on above.)

Posted by: David S. Cohen | Apr 8, 2009 4:22:16 PM

As a law student, one thing I hate about exams is having to rush. I really don't see the point in distinguishing students based on how much they can say about a particular topic in a time period that is unrealistic given the depth of the questions posed. I should note that I am in the top 10% of my class and on law review at a top law school-- I do well on exams. But I think it's worth asking why students must be rushed. In other words, why should I be rewarded because I can write a better answer, though still woefully incomplete, than my classmates to a question that is impossible to fully address in the time given? I think the best test would be one that, for example, COULD be completed in 3 hours, but where the students are given 4.5 or 5 hours to do it. That seems to me to be just enough extra time to allow students to write complete answers; at the same time, the extra time is not long enough for unprepared students to completely learn the material. That's just my take on the issue.
One other unrelated point: I once had an 8 hour exam that literally required non-stop writing (everyone in the class attested to this) for the entire period. I think that's just sadistic on the professor's part.

Posted by: JBE | Apr 7, 2009 10:43:34 PM

AnonProf: Thanks for the comment. This is definitely an issue that I plan to address in my take home exam post.

Jeff: Thanks for the comment. I agree that "supplement spouters" are spottable, which is why I don't care (much) whether students use them.

Another Anonprof: Thanks for the comment. I think that the biggest drawback to an "open everything" exam is that it might discourage (some) students from making their own outlines, so I can certainly see where you are coming from.

Posted by: Colin Miller | Apr 7, 2009 3:47:20 PM

I give modified close-book exams, letting students bring on only an outline they have prepared themselves and a statutory supplement (I teach statute-heavy classes). The outline can be as long as they want, and can draw information from any source, so long as they have prepared the final outline themselves. Seems to work pretty well; students appear to think it's fair, and I believe (hope?) forcing them to prepare the outline is a good way to get them to learn the material.

Posted by: another anonprof | Apr 7, 2009 1:55:11 PM

When I started teaching several years ago, I did the "limited open book" for the reason described above - I didn't want the students who couldn't afford the supplements to feel outflanked, even though I believed, with Mike Risch, the supplements don't help much, and sometimes hurt (see below). (My rule was that you could bring in any assigned texts or statutes, your own notes, and any outlines you created or helped create.)

Our deans here at Suffolk made the very persuasive argument that any middle ground between complete closed book and unrestricted open book invariably leads to disputes over Honor Code violations (even though professors are free to give any kind of exam they want), and I got roped into one as a hangover from my year at Tulane. I decided my "sensitivity" to the students' concerns was outweighed by the administrative concerns.

After several years, I've come to do and believe the following:

1. If I make the exam long enough and hard enough, whether it's open book or not is irrelevant. If it's open book and really short and easy, it fails to do the job of giving me enough differentiation for honest grading.

2. I believe the sine qua non of good lawyering in practice is issue-spotting. If you can't sort out the facts to ask the right questions, you can never get to the right answers, whether or not they are within your particular expertise. I say that as a long time generalist. So I have no qualms about traditional issue-spotting questions.

3. I teach courses (Sec Reg, Partnership and LLC) that have statutes. Even the best prepared student may need to take a peek at the wording.

4. I can tell by reading an exam if the student didn't bother with my class and is spouting from a supplement, generally because I have no clue what he or she is talking about. I punish accordingly.

5. I don't think students really want take-home exams. In class exams are painful, but they are over relatively quickly.

6. From the first day, I stress that this is graduate school, and students are self-accountable. I don't take attendance, don't punish for failures to be prepared on "on call" days, don't restrict laptop use, and other things that strike me as throwbacks to 19th century modes of instruction (I recognize reasonable people can differ on this). I just work really hard to organize and present the material and be available. Amazingly, there is invariably a bell curve (in my casual observation) in how students take advantage of it. I suspect the people who get As combine a certain amount of inspiration with their perspiration, but I also think perspiration can get you a long way up the grading scale.

Posted by: Jeff Lipshaw | Apr 7, 2009 12:45:20 PM

I think there is another significant problem with 8-hour or 24-hour take home exams. Many students who do not have children or other significant out-of-school obligations literally spend the entire period given working on the exam. I have always been concerned that for this reason, these sorts of exams strongly favor younger, traditional students over older students.

Posted by: AnonProf | Apr 7, 2009 11:47:23 AM

Thanks, JLR. The reasons you mention are the exact reasons why I give an in-class, "open everything" exam. Do you get the sense that most students share your concerns?

Posted by: Colin Miller | Apr 7, 2009 9:50:36 AM

Thanks, Orin and Andrew. I tend to agree that "slacker" students can't "catch up" on a 3 hour "open everything" exam. I will be curious to see, however, whether you think that the same holds true for the 8 or 24 hour take home exam, so hopefully you can comment on my upcoming post on the subject. To me, this seems to be the main drawback of the take home exam, but maybe somebody can change my mind.

Posted by: Colin Miller | Apr 7, 2009 9:47:33 AM

As a law student, I prefer 'open everything' exams, mainly because I think most exam restrictions simply penalize honest students. For example, I took a 'closed book' trust & estates exam several semesters ago that focused on minutiae. But the school allows flexible exam scheduling, and there is no software on the laptops, so all 'closed book' meant that honest students would not have access to the outlines on their laptops, while dishonest students would.

Even worse are take-home three hour exams, in which students are told they are on their honor not to spend longer than three hours answering the questions; there is absolutely no way to verify how much time a student spent on their answer. These types of testing arrangements create a strong incentive not to observe the rules, and can give the perception to honest students (even if it's not accurate) that they are at a disadvantage.

Posted by: JLR | Apr 7, 2009 9:33:47 AM

"Namely, I worry that the students will wrongly conclude that they can "catch up" on anything they failed to study for during the exam."

I've heard a couple of professors say that, honestly, but I'm not sure that any of us actually think that. At least we don't at Marquette Law. Yes, there are people who slack, but I don't think they really assume they can "catch up" when you've got those 3 hour finals like we do here.

Honestly, I'm perhaps a bigger slacker than pretty much anyone I know, and even I wouldn't be silly enough to think I didn't need to memorize a lot of things prior to taking an open-book exam. All "open book" means to the vast majority of law students is that we need to know where to find the language rather than knowing what the language is. As Professor Kerr pointed out, this isn't really different than when you practice law in the real world; you're going to need to know the Terrys and Mirandas of the subject matter, but you don't need to have statutory language down cold. I really do prefer open-book exams for that reason; they let me focus on understanding the concepts rather than the verbatim words.

Posted by: Andrew Golden | Apr 6, 2009 11:32:06 PM

I always give "open everything" exams because legal practice is "open everything." I don't think I have ever been asked a legal question when it was forbidden to look through books for the answer.

I don't worry about that unfairly leveling the playing field, for two reasons: First, I doubt it makes up for much, as there's just no time on exams to learn the material from scratch. Second, if a student is really so bright that he or she can look at an outline and give an excellent exam answer with little exam prep, the student is going to be a brilliant lawyer and deserves an A.

Posted by: Orin Kerr | Apr 6, 2009 11:12:17 PM

Bruce, I tend to find that my undgraded midterm helps with the warning you mention. I tell students at the start of my classes that while my exams are "open everything," they need to know the material pretty well, with their outlines being a last resort when their memory fails them. I'm not sure how much heed they give that warning, but then the midterm comes at the middle of the class. I always tell students to take the midterm under timed conditions. That's when students actually see that they can't rely on outlines, casebooks, etc., and still be able to finish the exam within the time limit.

Posted by: Colin Miller | Apr 6, 2009 6:16:37 PM

Anon, those are some interesting points. First, I agree with you that "open everything" exams mirror the private practice experience, which is one of the reasons I give them. Of course, this factor might favor the 8 hour take-home exam that Michael mentions above, but as I will note in an upcoming post, I just can't get on board with the take-home exam.

Second, you raise an interesting point about a time pressured "open everything" exam possibly favoring the information dumper. That's certainly a concern, but it seems to me that it can be addressed in the way that you give credit on the exam. I tend to give relatively little credit for just identifying the elements of a doctrine, defense, etc. and much more credit for a student's application of that law to the facts. Doing so seems to me to favor those who know how the law works rather than those who can just regurgitate the law quickly.

Posted by: Colin Miller | Apr 6, 2009 6:12:20 PM

I typically do "open everything" exams, but my main concern is the students who read too much into that, given that I put the students under tight time constraints. Namely, I worry that the students will wrongly conclude that they can "catch up" on anything they failed to study for during the exam. So I try to stress strongly and repeatedly that students will only have time to briefly check the exact wording of rules or tests -- attempting to refresh recollection of the substance during the exam will result in *serious* time pressure.

If I ever do a closed book exam, it will be because I've concluded that my warnings aren't enough.

Posted by: Bruce Boyden | Apr 6, 2009 5:30:37 PM

I have always offered "open everything" exams. My logic was, in part, as follows: in private practice, you would have access to outside sources to answer client questions so why not have access to those sources during the exam? Do we really care about memorization in law school or after law school?

Recently, however, I have started to consider moving to a completely closed exam. As you note, allowing outside sources does level the playing field. Because of this, I tend to make my exams fairly time pressured. Part of me wonders whether this unintentionally FAVORS memorization and information dumping on the exam. If the exam were closed book it could be less time pressured and perhaps would allow students to more fully think through the problems and offer a more in-depth analysis. Thoughts?

Posted by: AnonProf | Apr 6, 2009 4:20:30 PM

Michael, here are the two main reasons the type of exam was my first choice. First, if I had decided to give a closed book or even a partial open book exam, I would have felt the need to teach fewer topics based upon the fear of overwhelming students with too much material to remeber for the final. With an "open everything" exam, I felt that I could teach more topics without the same fear (in my mind, this is another benefit of the "open everything" exam).

Second, I think that certain casebooks are suited for certain types of exams. I think that casebooks with mostly unedited opinions, more public policy, and fewer hypotheticals work better with closed book exams because, again, there is a smaller quantity of material to learn (but students learn that material very well). Meanwhile, I think that casebooks with edited opinions and many note cases and hypotheticals work better with "open everything" exams because it is more about applying law to fact (pattens) than reinforcing the learning of the law.

These are the reasons why I chose the type of exam before chosing my casebooks or syllabi. As for your use of a (1) take-home and (2) word limited exam, I think that these are both interesting topics and have posts planned on each issue, and I hope that you will comment on them as well.

Posted by: Colin Miller | Apr 6, 2009 11:59:31 AM

Perhaps I'm nitpicking, or perhaps you are just looking for an interesting intro, but you say: "Before teaching my first class, I knew that the first question I had to answer before even choosing the casebook or deciding which topics to cover was what type of exam I should give."

Why so? I didn't decide until about halfway through the semester (the syllabus said the the exam format was TBD), and I even took a poll to see what kinds of exams my students preferred.

As for your main point, I do an 8 hour take-home, word limited exam that's open everything, though I make clear that a) nothing outside of our readings/class discussion is necessary and b) if an outside source disagrees with the readings or what I said in class, use it at your own risk (and sometimes people do and totally get the wrong answer).

Posted by: Michael Risch | Apr 6, 2009 11:37:59 AM

Anon, I share some of your concerns, which is why I opted for "open everything" exams. First, as I noted, whenever you have anything short of an "open everything," exam, there is always the concern that students are using materials that they should not be using.

Second, as you note, when you have a partial open book exam, line drawing can be difficult. Do you ban just commercial outlines? Do you ban "bank" outlines? What if a student transcribes a commercial outline into a word document? What if a student mostly copies but partially changes/supplements that outline? (I know students who did this in law school). If you restrict the exam to materials that students had a "hand in preparing," how much of a hand is required? (I remember this being a big question in law school).

When I was in law school, one professor restricted students to a four page document of the Federal Rules of Evidence plus any notations that they made in the margins. So, what did some students do? Well, they scanned the document into their computer and added in most of their outlines in tiny font in the margins.

I might agree in principle that some type of partial open book exam is the best way to go. I just haven't found a fair way to do it (yet).

Posted by: Colin Miller | Apr 6, 2009 11:11:49 AM

I use partial open book - no commercial outlines, etc. I would note that I believe this allows students who spend the time preparing and reviewing material prior to the exam an appropriate "leg-up" on others because they have done the work. The downside is, as many can attest to I am sure, that law students are particularly well trained to see "line-drawing problems" and then ask every conceivable part of them. For example, are law-review-bank outlines OK? What about outlines that take substantial (and how to define that?) material from commercial outlines? To what extent is outline-sharing permitted? I know for this very reason some have abandoned partial-open-book exams, but I still think the benefits of removing commercial outlines outweigh the problem of answering multiple "line-drawing" questions before, and perhaps during, the exam.


Posted by: Anon | Apr 6, 2009 10:48:04 AM

The comments to this entry are closed.