« Public-school teaching "like Jesus Christ himself" . . . and a new paper on public-employee speech | Main | Prop 8 at the California Supreme Court »

Tuesday, September 06, 2011

Tough Tests, Take 2: If You Gave an "Answer Justification" Multiple Choice Exam, What Would You Give a Student With the Right Answer but the Wrong Justification?

I have been thinking about adding some multiple choice questions to my current all-essay exams in recognition of the fact that different types of exam questions favor different students. See, e.g., Kenney F. Hegeland, On Essay Exams, 56 J. Legal Educ. 140, 147 (2006) ("To simplify, multiple choice rewards those students who know the rules while essay rewards those who can manipulate the rules."). Indeed, some have "argue[d] that multiple choice exams are actually more sophisticated tools than essay questions for analyzing students' abilities to read facts and cases as well as their ability to apply an unfamiliar rule of law to a legal problem."  Rogelio A. Lasso, 15 Barry L. Rev. 73, 100 n.176 (2010).

But if I were to take the plunge and sprinkle in some MC, I think that it would be of the "answer justification" variety. As noted in Janet W. Fisher, Multiple-Choice: Choosing the Best Options for More Effective and Less Frustrating Law School Testing, 37 Cap. U. L. Rev. 119, 133 (2008),

Answer justification provides students with the option of converting multiple-choice items perceived by students to be “ambiguous or confusing” into short essay answers. After selecting what the student believes to be the best option, the student may write a short justification of that choice. If the student has selected the correct option, there is no reason to score the justification. If the student has selected an incorrect option, she or he may receive full credit, partial credit or no credit for the justification.

Here's the thing, though: If I were to give "answer justification" MC questions, I would be tempted to give a student only partial credit if she filled in the right bubble but gave an incorrect justification. Or maybe I would give her no credit. Or maybe I would give her full credit. I just don't know, which is why I am raising the question. Here's a poll followed by my thoughts. Any comments would be appreciated.

If You Gave an "Answer Justification" Multiple Choice Exam, What Would You Give a Student With the Right Answer but the Wrong Justification?
Full Credit
Partial Credit
No Credit
pollcode.com free polls  

So, let's start with an example. Let's say my MC question has Paul, whose domicile is in Illinois, suing Dan, who has always lived in Indiana and is going to college in Illinois but plans to return to Indiana after graduation. Paul is suing Dan an after a car accident and seeks $70,000 for medical expenses. The question asks whether there is federal subject matter jurisdiction over the action. There are 3 or 4 "yes because..." answers and one  "no" answer. Let's say that Student A fills in the bubble that corresponds with the "no" answer. Student A then provides an explanation for her choice: "I said 'no' because Paul and Dan are both residents of Illinois, so there's no diversity of citizenship and no diversity jurisdiction." Or let's say that her explanation is, "I said 'no' because while the amount in controversy requirement is satisfied, Paul and Dan are both residents of Illinois, so there's no diversity of citizenship and no diversity jurisdiction."

The first justification is wrong because Dan's domicile is in Indiana for diversity jurisdiction purposes because he never had physical presence in Illinois combined with an intention to remain there permanently. Moreover, the second justification is doubly wrong because $75,000+ is the amount in controversy requirement. So, what do I do with Student A? 

Under a classic "answer justification" model, I would give the Student A full credit. As noted in the excerpt from Professor Fisher's article above, "If the student has selected the correct option, there is no reason to score the justification." And this approach has a certain intuitive appeal. An "answer justification" MC question is still a MC question, and on MC questions, we reward the correct answer. An implicit assumption with MC questions is that a certain percentage of "correct" answers by students are guesses or the results of a blind squirrel finding the nut. Just as a broken clock is right twice a day, a student guessing on a MC question has a 20 or 25% chance of getting the answer right assuming 4-5 listed answers.

Also, while the justifications given above were clearly incorrect, how often will it be perfectly clear that the justification is wrong? If Student A's justification were, "IL & IL X," could I assume that she meant that she concluded that both Paul and Dan were domiciled in Illinois, meaning that there was no diversity jurisdiction?

On the other hand, does it make any sense to give complete credit to a student who clearly missed the boat and happened upon the right bubble through some stroke of luck? If the above MC question were an essay or short answer question, and Student A used the reasoning given in her justification to reach the conclusion that there was no federal subject matter jurisdiction, I wouldn't give her any credit for her reasoning or her conclusion. So, why should I give her full credit for filling in the right bubble based on the wrong reasoning?

And indeed, Professor Fisher notes that

A variation on the answer justification procedure is described in a study by Nield and Wintre who penalized students for “a bad explanation of a right answer.” While a penalty would likely reduce the frequency with which students wrote justifications, professors would have to read all of the justifications in order to enforce the penalty.

So, should I give partial credit to a student who fills in the right bubble but gives the wrong justification? In a way, this makes sense because I am giving students who filled in the wrong bubble but gave the right, or at least an understandable, justification partial credit. So, both students are partially right, and both students are partially wrong, so why not give them both partial credit?

Well, the reason is that I kind of think that Student A deserved no credit in the example above. Let's say that Student B in the example above fills in the bubble corresponding with, "Yes, there is diversity jurisdiction." And let's say that in her justification, she writes, "There is complete diversity (IL & IN), and I'm assuming that Paul is seeking at least $5,000+ for damage to his car." The problem for Student B is that she is assuming facts not in the question, which I tell students not to do. That said, her justification makes (at least somewhat) clear that she understands the 2 requirements for diversity jurisdiction.

Conversely, Student A in the example above makes (at least somewhat) clear that she doesn't understand at least 1 requirement for diversity jurisdiction and possibly neither of the requirements. So, does her answer of "No," while being correct, merit any points?

It is a tough question, and of course the correct answer for how much credit to give is whatever I tell students is the amount of credit a student will receive for filling in the correct answer but writing an incorrect justification. But what should be the correct answer?

-Colin Miller 

Posted by Evidence ProfBlogger on September 6, 2011 at 08:32 AM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference Tough Tests, Take 2: If You Gave an "Answer Justification" Multiple Choice Exam, What Would You Give a Student With the Right Answer but the Wrong Justification?:


I used to do MC-with-justification. I would give partial credit for a right answer/weak justification and no credit for a right answer/way-off-base justification. In the latter case, I just assumed the student guessed. I recently have gone away from that to straight short-answer, finding it better enables students to write out their answers, as well as that the MC part of it was not adding anything or saving me any time.

Posted by: Howard Wasserman | Sep 6, 2011 8:48:24 PM

anon, the Fisher article I cited is a good place to learn more about "answer justification" MC questions and why it might make sense to use them. Here is the SSRN link if you don't have Westlaw:


And, as I said, I've never given "answer justification" MC questions, but I would guess that law students provide a justification for less than 20% of questions. For undergraduate students, it is likely that the % is much lower. For instance, in her article, Professor Fisher notes that "[a] study of seventeen different multiple-choice exam administrations to undergraduate psychology students revealed that students used answer justification on only 1% of the total number of multiple-choice items."

Posted by: Colin Miller | Sep 6, 2011 8:38:45 PM

Thanks. That does make a lot of sense. I teach an undergraduate course and have been using short answer and essay but think I may switch to the mc with justification whereas you explain they only have to explain questions that they feel are ambiguous or where they are stuck. I definitely could cover more information that way.

Posted by: anon | Sep 6, 2011 8:20:59 PM

anon, I think that the distinction is this: With "answer justification" MC questions, the expectation would be that a student would not provide a justification for the (vast) majority of questions. Instead, the student would only provide a justification if (a) she thought that the question was confusing/ambiguous; or (b) she was down to two answers and wanted to explain why she couldn't decide between the 2 but ultimately picked one over the other. There is no requirement that a student provide a justification for each or any question.

With short answer questions, a student has to provide a brief answer to each question. For instance, I know one professor who has some short answer questions on his exams and limits students to three (or maybe it is is four) sentences for each answer.

The reasons I see for using answer justification MC questions instead of short answer questions are:

(1) You can have more "answer justification" MC questions. I've never given such questions, but I would guess that the average student provides a justification to less than 20% of the questions. This would allow me to ask more questions, which means that I can cover more material on the exam.

(2) MC are on the bar exam while short answer questions are not, so MC questions are better bar prep.

That said, there are certainly many reasons to prefer short answer questions, and I have considered adding them as well.

Posted by: Colin Miller | Sep 6, 2011 8:01:17 PM

Right but the bottom line is you have to read the explanation and make a decision. I agree that if a professor chooses to ignore the explanation if the mc answer is right, then a mc test makes sense. But if a professor is going to (1) require an explanation and (2) deduct points for a wrong explanation (or give points for a correct one,even if the mc is wrong), then this is no different than short answer, except more complicated to grade. Maybe I am missing something.

Posted by: anon | Sep 6, 2011 7:38:59 PM

The justification is not meant to BE the test. It's only to help the prof clarify WHY a student purposely chose a wrong answer.

Posted by: Trudi Villarreal | Sep 6, 2011 7:24:51 PM

Why bother with multiple choice with explanation? Just do short answer. Essentially multiple choice with explanation is a short answer. I don't see how you save any time on grading if you have to read the explanation. It really is short answer, right?

Posted by: anon | Sep 6, 2011 7:16:34 PM

I agree with Patrick. Your grading is your grading. However, if it were up to me, I would give partial credit for Right Answer, Wrong Justification. Reasoning being that the answer is then no longer right or wrong in the sense that they can now manipulate it. Therefore, it makes sense that grading reflects the greyness of the response. OTOH, if you PROVIDE the justification and they choose the wrong one, no credit should be given.

~A law student~

Posted by: Trudi Villarreal | Sep 6, 2011 7:10:08 PM

All three approaches make sense. As long as you tell the students up front how you're going to grade the exam, I don't think it makes a big difference which way you choose.

Posted by: Patrick | Sep 6, 2011 6:09:06 PM

The comments to this entry are closed.