Tough Tests, Take 2: If You Gave an "Answer Justification" Multiple Choice Exam, What Would You Give a Student With the Right Answer but the Wrong Justification?

I have been thinking about adding some multiple choice questions to my current all-essay exams in recognition of the fact that different types of exam questions favor different students. See, e.g., Kenney F. Hegeland, On Essay Exams, 56 J. Legal Educ. 140, 147 (2006) ("To simplify, multiple choice rewards those students who know the rules while essay rewards those who can manipulate the rules."). Indeed, some have "argue[d] that multiple choice exams are actually more sophisticated tools than essay questions for analyzing students' abilities to read facts and cases as well as their ability to apply an unfamiliar rule of law to a legal problem."  Rogelio A. Lasso, 15 Barry L. Rev. 73, 100 n.176 (2010).

But if I were to take the plunge and sprinkle in some MC, I think that it would be of the "answer justification" variety. As noted in Janet W. Fisher, Multiple-Choice: Choosing the Best Options for More Effective and Less Frustrating Law School Testing, 37 Cap. U. L. Rev. 119, 133 (2008),

Answer justification provides students with the option of converting multiple-choice items perceived by students to be “ambiguous or confusing” into short essay answers. After selecting what the student believes to be the best option, the student may write a short justification of that choice. If the student has selected the correct option, there is no reason to score the justification. If the student has selected an incorrect option, she or he may receive full credit, partial credit or no credit for the justification.

Here's the thing, though: If I were to give "answer justification" MC questions, I would be tempted to give a student only partial credit if she filled in the right bubble but gave an incorrect justification. Or maybe I would give her no credit. Or maybe I would give her full credit. I just don't know, which is why I am raising the question. Here's a poll followed by my thoughts. Any comments would be appreciated.

If You Gave an "Answer Justification" Multiple Choice Exam, What Would You Give a Student With the Right Answer but the Wrong Justification?
	Full Credit
	Partial Credit
	No Credit
pollcode.com free polls

So, let's start with an example. Let's say my MC question has Paul, whose domicile is in Illinois, suing Dan, who has always lived in Indiana and is going to college in Illinois but plans to return to Indiana after graduation. Paul is suing Dan an after a car accident and seeks $70,000 for medical expenses. The question asks whether there is federal subject matter jurisdiction over the action. There are 3 or 4 "yes because..." answers and one  "no" answer. Let's say that Student A fills in the bubble that corresponds with the "no" answer. Student A then provides an explanation for her choice: "I said 'no' because Paul and Dan are both residents of Illinois, so there's no diversity of citizenship and no diversity jurisdiction." Or let's say that her explanation is, "I said 'no' because while the amount in controversy requirement is satisfied, Paul and Dan are both residents of Illinois, so there's no diversity of citizenship and no diversity jurisdiction."

The first justification is wrong because Dan's domicile is in Indiana for diversity jurisdiction purposes because he never had physical presence in Illinois combined with an intention to remain there permanently. Moreover, the second justification is doubly wrong because $75,000+ is the amount in controversy requirement. So, what do I do with Student A? 

Under a classic "answer justification" model, I would give the Student A full credit. As noted in the excerpt from Professor Fisher's article above, "If the student has selected the correct option, there is no reason to score the justification." And this approach has a certain intuitive appeal. An "answer justification" MC question is still a MC question, and on MC questions, we reward the correct answer. An implicit assumption with MC questions is that a certain percentage of "correct" answers by students are guesses or the results of a blind squirrel finding the nut. Just as a broken clock is right twice a day, a student guessing on a MC question has a 20 or 25% chance of getting the answer right assuming 4-5 listed answers.

Also, while the justifications given above were clearly incorrect, how often will it be perfectly clear that the justification is wrong? If Student A's justification were, "IL & IL X," could I assume that she meant that she concluded that both Paul and Dan were domiciled in Illinois, meaning that there was no diversity jurisdiction?

On the other hand, does it make any sense to give complete credit to a student who clearly missed the boat and happened upon the right bubble through some stroke of luck? If the above MC question were an essay or short answer question, and Student A used the reasoning given in her justification to reach the conclusion that there was no federal subject matter jurisdiction, I wouldn't give her any credit for her reasoning or her conclusion. So, why should I give her full credit for filling in the right bubble based on the wrong reasoning?

And indeed, Professor Fisher notes that

A variation on the answer justification procedure is described in a study by Nield and Wintre who penalized students for “a bad explanation of a right answer.” While a penalty would likely reduce the frequency with which students wrote justifications, professors would have to read all of the justifications in order to enforce the penalty.

So, should I give partial credit to a student who fills in the right bubble but gives the wrong justification? In a way, this makes sense because I am giving students who filled in the wrong bubble but gave the right, or at least an understandable, justification partial credit. So, both students are partially right, and both students are partially wrong, so why not give them both partial credit?

Well, the reason is that I kind of think that Student A deserved no credit in the example above. Let's say that Student B in the example above fills in the bubble corresponding with, "Yes, there is diversity jurisdiction." And let's say that in her justification, she writes, "There is complete diversity (IL & IN), and I'm assuming that Paul is seeking at least $5,000+ for damage to his car." The problem for Student B is that she is assuming facts not in the question, which I tell students not to do. That said, her justification makes (at least somewhat) clear that she understands the 2 requirements for diversity jurisdiction.

Conversely, Student A in the example above makes (at least somewhat) clear that she doesn't understand at least 1 requirement for diversity jurisdiction and possibly neither of the requirements. So, does her answer of "No," while being correct, merit any points?

It is a tough question, and of course the correct answer for how much credit to give is whatever I tell students is the amount of credit a student will receive for filling in the correct answer but writing an incorrect justification. But what should be the correct answer?

-Colin Miller