« Ironic, crass, stark reminder, reclaiming imagery? You decide | Main | Baude on removal under the JVCA »
Wednesday, January 25, 2012
Learning from exams
I survived my first round of exam writing and grading, and now I’m meeting with students from last semester who want to go over their exams. As I do that, two questions keep coming to mind.
First, while I’m willing to accept that most of the students are just interested in knowing why they got the score they did and in seeing if they can argue their way to a higher grade (which they can’t, absent a math error on my part), I do wonder if anyone has suggested techniques for encouraging students to use these exams to further their substantive understanding of the subject.
Second, what can I do to learn from these exams? How do I assess whether it was fair? Whether it was too long? Too short? Too easy? Too hard? I’d love any suggestions people can share. Thanks!
Posted by Michael Teter on January 25, 2012 at 01:20 PM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef0163001b9692970d
Listed below are links to weblogs that reference Learning from exams:
Comments
Last semester was my first time on the other side of the exam process as well. My post-exam discussions with students have gone pretty well, so you may find my practices helpful. I believe in full disclosure to help de-mystify the process. Accordingly, after the grades were submitted I sent to the entire class:
- My grading matrix and notes for each question re: point assignment, common mistakes, and what I was looking for.
- An anonymized A exam and accompanying grade sheet.
- An anonymized B exam (right on the median) and accompanying grade sheet.
I provide every student who wants to meet with me (or just wants to see) a copy of their exam answers and grade sheet. So far I have not had anybody try to argue their grade with me. The discussions have generally centered around what students can do differently to do better on future exams from me and, to the extent that I notice systemic issues, do better on exams generally. I think it is helpful to provide a model "pretty good" answer as well as a model "near perfect" answer so the students can assess where their answers stand on the continuum.
I definitely made the exam too hard as there were more incomplete and rushed answers than I wanted to see. I think the problem was that I was aiming to make the exam interesting/tricky for me to take, which is a little unfair. However, the exam still produced a workable curve with clear stratification at the grade cutoffs, so it worked out ok. After all, it doesn't really matter to the students if they got a raw score of 70 or a raw score of 90, if the letter grade is the same. I think I will tone down future exams a little bit though.
Posted by: Jake Kreutzer | Jan 29, 2012 3:02:55 PM
Aprof, here is an example from a crim law exam I once took: "We are told A throws something at B while trying to escape," on the model answer there was a medium-sized discussion of whether it mattered what the something thrown was, I think the example the prof used was "a bowl of grapes." When, on my own exam (a take-home) I saw a similar fact pattern, I went back to the model answer and used the exact same reasoning the Prof had done.
Now if the Prof has 15 years worth of model exams chances are many of the issues are going to recur in.
Now again, I am not expressing a view as to whether a student who used this model answers in this way was doing something wrong or something we as professors would not want: there is an argument that what I did on my own exam is one of the skills of very successful lawyers, recognizing a pattern and where to find the answer when someone else has done a lot of the work. All I wanted to suggest is that this is a non-trivial effect of releasing model answers. I agree the problem is worse for take-homes than in-class
Posted by: I. Glenn Cohen | Jan 28, 2012 1:37:34 PM
I don't understand how this last problem can arise if you are using issue spotter-type questions - the facts are always different and the key is applying the facts to the law, so how could a student use a pre-fab paragraph or two in there in any beneficial way? In my classes, I do not permit the students to use any electronic materials. I guess they could bring model answers into the exam and type them in, but it seems to me this would gain them no advantage.
As for the questions above, I think when you've written a good exam you get almost everyone finishing every question, and you see a spread in quality of answers. I think at least some of the students should produce good, solid answers where they spot every issue. If you have very few or no good exams then there is a problem either with the exam or the teaching. If you have a lot of people missing an issue, it may be that the call of the question was too vague or the issue too buried or not consistent with how the material was presented in class. I am torn in exam writing between the focus on issue spotting and good analysis as I find when the question is geared towards issue spotting some don't spot it and then don't have an opportunity to show whether they could analyze the question.
Posted by: AProf | Jan 26, 2012 1:08:26 PM
I too follow a similar pattern to those who weighed in above, but I do want to sound one note of caution about releasing your own model answer or grade matrix. Even though we write a new exam every year, there are often sub-issues and analyses that recur from time to time. If your model answer is out there, you run a significant risk of having students merely copy and paste your analysis back to you. I know because on occasion as a student I did exactly this in a class where the Professor had made available his past model answers to exams for the past 10 years. Perhaps you don't think that is a problem -- that this is a form of studying or learning that is just as valuable as others -- but it is a predictable result of releasing matrices or model answers, that I thought I should flag. My own solution is to allow students to read model materials in my assistant's office before they go over their exams with me, but not take them with them or copy from them.
Posted by: I. Glenn Cohen | Jan 26, 2012 9:59:03 AM
I mostly follow the practices of what is outlined here, detailed memo and for first-year students I try to find a specific time (in class if I still have them or one evening if I do not) to go over the exam. I don't encourage students to review their exams with me, in part because I really can't tell them how to do better on other professors' exams but also because my experience has been that somewhere between 50-75% of the students who want to go over their exam do so with the express purpose of getting me to change their grade and they usually relent once they realize I won't. (We have faculty who do change grades for persistent students so I can't blame the students for trying.) As for evaluating the substance of the exam, ideally those who seemed most engaged in class, those who came regularly, spent time on the material would fare best, and those who did not show up or did not do the reading would be at the bottom, and that is one way, but an imperfect one, to assess whether the exam was measuring skills/knowledge dervied from the class. Depending on the nature of the exam, you can also see if there was consistency across answers; I typically include at least one very difficult question that few students will get but it often proves to be the indicator of the very best exams, and often those are from what I consider the very best students. When I was in law school, I always hated the exams/students/professors where someone who had done none of the reading or rarely been to class did perfectly well, and I have always worked to avoid that, generally successfully.
Posted by: MS | Jan 26, 2012 9:17:51 AM
Thanks for the suggestions, everyone. If folks have ideas about ways to assess the exam from an exam-writing standpoint, I'd appreciate those, too.
Student Exam Reviewer -- I also wrote out answers to the exam in "exam conditions," but I wasn't sure that this was particularly good indicator of how long the exam would take students who, after all, are studying for several subjects and didn't get the benefit of having written the exam!
One other quick point: I'm not entering meetings with preconceived notions about why they're there. I actually ask each student what it is they're hoping to get out of the meeting at the start so I can tailor the meeting to that purpose. Several of them said directly that they thought they should have gotten more points here and there. That said, I still do think that they're also interested in a more complete picture of how they did, what issues they missed, and how they can improve their exam-taking for the future. And I'm happy to help with that as best I can.
Posted by: Michael Teter | Jan 25, 2012 8:30:47 PM
I have never had a student come to me to argue about a grade. Maybe it is because I give off signals that the grade (excepting a clerical error which I've never seen) is not a matter for negotiation.
For 1L contracts essay questions, I always write a complete model answer. I translate the model answer into a scoring matrix roughly based on an IRAC format, and assign points to the issues (the matrix can be as much as 8 pages). Before I start grading, I make copies of the matrix, and use one for each exam. I assign points on the matrix. So when a student comes in, I can give him or her a copy of the exam and a copy of the matrix, and it will be absolutely clear which issues he/she missed, and which he/she saw but didn't answer fully or correctly.
Also, I do a timed practice exam about 2/3 of the way through the semester in which I have the students exchange and grade each other's exams based on the same kind of point-assigned matrix. So they know beforehand exactly what to expect in terms of what I'll be doing when I grade.
Finally, I post the best student answers on TWEN.
Posted by: Jeff Lipshaw | Jan 25, 2012 8:09:07 PM
P.S. one of my favorite professors to review an exam with had a good way to decide whether his exam was too long/short. He would write the exam and then he himself would write out answers to the questions under the exams time constraint. He would then put a "suggested amount of time to answer" next to each question/fact pattern.
Posted by: Student Exam Reviewer Alumnus | Jan 25, 2012 6:19:38 PM
First of all, I am a little disappointed that you hold the preconceived notion that your students are coming to you to practice there oral advocacy skills and argue for a better grade. When I took time to review my exams, I always went in knowing the only way that my grade may be changed was due to "Lawyer math". As a student, I reviewed the majority of my exams when I did sub-par, and even also when I aced it. My reasons for reviewing the exams were: (1) I wanted to learn from what I did wrong in regards to formulating an answer and/or whether I got the law wrong so to minimize the chances that these shortcomings would continue in both my academic and professional life; (2) I wanted to see what my strengths were (if any), in order to reenforce them for future exams; (3) if it was a professor I enjoyed and would take again (even if he gave me a grade, which I was dissatisfied) and/or was a professor who would be teaching a course I was interested in pursuing, I wanted to make sure I knew exactly what the most important parts/criteria in grading his/her exam (i.e. issue spotting v. analysis v. "public policy" arguments) so that I could be sure to know what to focus on in their future exam; and lastly and probably least importantly (disclaimer---def. not a gunner), (4) I wanted to start/maintain a connection with someone who I either respected, or thought was worthwhile in knowing. For ex, even though I did not do amazingly well on one professor's first year exams,the professor still hired me to serve as one of his research assistants in both of my remaining years. I think this exam review meeting showed him my willingness to learn, my motivation, and my genuine intelligence (which an exam does not always do a good job in evincing), which gave him faith in my abilities to serve him well as a research assistant.
Posted by: Student Exam Reviewer Alumnus | Jan 25, 2012 6:14:15 PM
I do something very similar to FYP's "post-exam memo", but in truncated, email form. I give my general observations as to overall exam answers and also give students a sense of "those issues which a full-credit exam included". I couple this email with an anonymized version of the best exam in the class (along with the exam questions) left with my assistant for students to "check out" and review in the faculty office area.
In terms of whether the exam is too short, or too long, I expect students to use the full allotted time for their exam responses. So, if everyone manages to get to every question, then I know it's not too long; if on the other hand, every answer is so thorough as to make a fair "B curve" impossible, then I know the exam was too short. I usually can't assess the relatively length of the exam until I'm grading it (and hating myself for writing such a long exam!).
Posted by: SecondYearProf | Jan 25, 2012 2:06:27 PM
Michael, one of my goals has been to turn my exams from purely summative to summative/ formative. To do this, I create a "post-exam memo," posted on the course and library website, which includes: (1) a copy of the exam; (2) my brief remarks regarding observations both of common errors and strong responses; (3) a copy of the strongest student's essays (edited a bit and anonymized); and (4) a self-assessment rubric.
These materials are available to all my students, and I encourage them to use them. If students ask to meet with me about the exam, I require them to read their exam and all of the post-exam memo docs, fill out the self-assessment rubric, and turn in their self-assessment rubric and essays to me a day before our meeting. In so doing, I provide them with an opportunity to self-regulate their own learning, and I think I dissuade the "grade grubbers" from wasting my time and theirs. Moreover, these materials are available for next year's students, so they can use them as practice exams before the real one.
Posted by: FifthYearProf | Jan 25, 2012 1:38:40 PM
The comments to this entry are closed.