« Something new under the sun: Actors for hire to read depositions | Main | Judges: Ask not what legal academics can do for you, but what you can do for legal academia! »
Tuesday, July 12, 2011
The Sesame Street Effect
I have experimented in recent years with giving feedback to my students in my large, first-year class during the academic year. I have done this primarily by giving them the opportunity, two or three times during the semester, to do a practice exam question. They can hand this in to me and I mark it up with my comments and corrections, but I do not give any sort of grade. I also agree to meet with any student who seeks me out to discuss it. My thinking is that it would be helpful for students not only to get a sense of how they are doing in terms of the substance of the course, but also to have an opportunity to practice and get feedback on their exam-writing skills. I make it voluntary, however, on the assumptions that 1) it would be very onerous to review in detail 80 such answers at two or three points in the semester, especially when 2) not all students would have the time or the inclination to put any real effort into the exercise if it were mandatory but ungraded, and 3) some students don't really need the practice anyway.
I have found, however, that this strategy creates what I call a "Sesame Street effect." It is my understanding that the idea behind Sesame Street at its inception was to create an opportunity for children who did not have access to a preschool program to still get some of the benefits and to learn some early preschool material through watching TV at home, preferably with a parent/caregiver. As it turns out, I gather, the children who benefitted most from Sesame Street were affluent children who were already going to preschool, and then getting further enrichment by also watching the program (often with a parent or caregiver). The children who didn't have access to preschool often also did not watch the program, or they didn't get as much benefit from it as they could because they weren't able to watch it with a parent or caregiver. In other words, metaphorically and no doubt literally as well, the rich got richer and the poor got poorer.
I don't intend to compare law students to preschoolers, but I do find that this effect holds to some extent when I offer voluntary feedback opportunities during the semester. The students who end up performing very well in the class often take advantage of these opportunities, whereas I rarely receive optional papers from the students who performed the worst. One possibility, of course, is that the feedback I give them is particularly helpful, and that's why those students perform well, but I doubt that very much - the high performers usually submit very good papers for the practice exercise. Instead, I suspect that the students who ultimately perform worse are overwhelmed, pressed for time, feeling like they don't know the material well enough to answer a practice question, and/or not prioritizing this particular class, and that's why they don't take the opportunity that is offered to them, and that they probably know would benefit them.
Does anyone have any thoughts on how to minimize the Sesame Street effect, without making these exercises mandatory (and graded)?
Posted by Jessie Hill on July 12, 2011 at 10:48 AM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef015433a99a98970c
Listed below are links to weblogs that reference The Sesame Street Effect:
Comments
"I don't intend to compare law students to preschoolers"
Why not? Lawyers routinely show as much claimed inability to understand what other people mean to say and as much incivility and unreasonableness as preschoolers. Surely, the adults coming into the institution aren't like that. So, the job of law school is to turn law students who used to be adults into people who behave like preschoolers.
Posted by: ohwilleke | Jul 14, 2011 3:45:22 PM
Oh, and in my previous comment, the "grades" are never recorded, not part of the student's real academic evaluation. They are just a feedback device within the series of assignments.
Posted by: Zack | Jul 13, 2011 8:09:43 PM
Make the first assignment mandatory. Tell the class to have it done by the following week.
Following week, begin class by handing out a model "good" answer and model "bad" answer. Collect and re-distribute papers so each student is holding someone else's, labelled only by exam number.
Walk the students through evaluating the paper they are holding, by discussing the model answers. Have each student grade the one they are holding: A, B, C, etc., or A, A-, B+, etc. Collect the graded papers and put them on the table at the end of class, so each student can retrieve their own paper by exam number.
Subsequently, declare to the class that those students who received a "good" grade (set an arbitrary cut-off, like B+ or higher) are done with the assignment. The rest of the students are "strongly recommended" (or required) to participate in a repeat.
Later, assign another practice test. Maybe don't use class time with the grading exercise. Distribute model answers, and tell the students to grade them themselves (either grade their own papers, or trade with a friend). Tell the students who got As the first time around that they are presumptively excluded from this round of the exercise. (Bend the rules on a case-by-case basis, if you want.)
A few weeks later, email out a third "practice test" and declare it to be mandatory for those students who have yet to receive a "good" grade (A, A-, etc.) on one of these. Collect them if you want, or suggest to the students that they self-grade again.
Repeat as necessary.
This may sound complex, but if you look through it, you'll see that very little work actually falls on your shoulders. Not much more than other approaches which have been discussed.
You can police participation or make steps more/less mandatory as necessary.
Posted by: Zack | Jul 13, 2011 8:06:25 PM
WPB, I have been chair of this committee for about 3 years now, and so my experience is probably not wide enough to say anything that has any statistical significance. But our rule is that students who get below a 2.33 cumulative GPA after their first year are dismissed but can petition for re-admission for one more probationary semester; if they don't get into good standing by the end of that semester, they are permanently dismissed with no appeal. This rule is partly based on data indicating that students who have a cumulative GPA under 2.33 are statistically very unlikely to pass the bar exam on their first try. So far, my sense is that the only students who manage to get into good standing quickly after re-admission are those who came very close to a 2.33 in their first year or who had a very compelling reason for their performance (eg, medical issues, family problems, etc. - not poor study skills or the like). But that's just anecdotal at this point. As both a short-term and a long-term proposition, I think the cards are stacked against anyone who doesn't fall into that category, at least in my experience.
Posted by: Jessie Hill | Jul 13, 2011 4:13:12 PM
I'd be curious to hear how effective you think the readmission process is. In your experience, how many of the students who are readmitted are better off going back to law school as opposed to abandoning law for a different pursuit?
Posted by: WPB | Jul 13, 2011 2:50:48 PM
Thanks to all of you. These are really great comments. Jen Kreder and 4thyrLawProf have given me a lot to think about in terms of a long-term solution; and Positroll and David, those are very ingenious immediate short-term fixes that would satisfy a lot of my concerns.
I'm also really interested by a lot of the comments that suggest that I shouldn't worry about it so much, or that this isn't really my job. Personally, I do think it's my job to be concerned about those students who aren't really getting the message in my classes, because I feel that's part of my role as an effective teacher. If some students just aren't getting it, it's at least possible that it's at least partly my fault, right? At the same time, some of you have pointed out an important aspect of this problem - that law school classes are curved, and some people have to be at the bottom. That's not going to change no matter how good a teacher I am. But I experience this fact as particularly agonizing, not as liberating. Perhaps this is because I also chair the committee that decides whether to re-admit students who have been dismissed after receiving poor first-year grades, and I have seen firsthand the often heartbreaking results of some students just not getting it. I'm not sure that means that I shouldn't try to reach those students. But it's true, in any case, that we simply can't make sure all of the students are above average!
Posted by: Jessie Hill | Jul 13, 2011 2:40:50 PM
Have you considered making the assignments mandatory but staggering them so that you don't have to review them all at once? I have graded assignments that are a part of all my classes (big and small) but make them staggered so that I'm never crushed at once with grading. (Instead, I'm lightly smothered all semester long....)
Posted by: David S. Cohen | Jul 13, 2011 2:18:45 PM
Comparing law students to preschoolers is very accurate. :)
But on the whole, the gist of your post seems to be implicitly assuming that it's right to actually treat law students like preschoolers.
In any given class, about 50% of the people are going to do worse than the median. There are probably some students who come into a class knowing that it'll be them at the bottom, and they're fine with that. (Maybe they're strong students who are just weak in a required subject, but it's in a field that they don't think they'll use that much. Maybe they're weak students who have decided to focus their efforts on a different class.)
If you start out with the assumption that your students are somewhat rational, then it may well be that some of them have rationally decided to not put extra effort in in this class, in order to save it for other classes. If that's the case, I'm not sure there's value in trying to arbitrarily alter the priorities they've chosen. You don't know as much about their motivations as they do.
For students in a graduate or professional school, grade school should have been about learning how to learn, college should have been about learning, and grad/prof school should be about learning how to do what they want to do for the rest of their lives. If they haven't figured out how to learn, or how to allocate their time and efforts, it's surprising that they made it to law school.
Being there for your students is great. But recognize that they're adults, and their failure to take advantage of opportunities is mostly due to them.
(I will add one caveat: there's definitely some complicated psychology behind students' decisions to turn in non-final writing. If the student thinks it's substandard, they're torn between (a) wanting the help, (b) being worried that they'll get a reputation with the professor as a bad writer, (c) demonstrate improvement later that will help their grade. Anything you can do to assure students that this assignment won't alter your evaluation of them later would probably help alleviate that anxiety.)
Posted by: Andrew MacKie-Mason | Jul 12, 2011 9:49:01 PM
The underlying issue is that, in your metaphor, your class, and in life, motivation is as much or more of a predictor of performance as raw ability. As someone who regularly provides students with voluntary opportunities to improve, I would say that the optional pre-test program you offer probably ends up being as much of an assessment of motivation as an educational tool. So how do you motivate the others to take advantage of the learning experience (and just engage more generally)? 4thyrLawProf's comments offer several great examples, including signaling that it is worthwhile, important to faculty, and normative for law students to do. That said, there will always be some students are motivation resistant.
Posted by: Erik | Jul 12, 2011 5:26:43 PM
Hi, Jessie. I think 4thyrLawProf's suggestions are dead-on. Even if those suggestions won't fit 100% given your school's culture, many of the suggestions can be modified. I have started to do some of those things in modified form, and I found that the overall performance of the class improved. I also use a combo of group drafting projects and individual mid-term exams, with the next entire class devoted to going over the mid-term and all of the feedback I provide (model answer, grading sheet, feedback memo). Some of these things are more popular than others among students in given years, but the exam results have improved. Good luck!
Posted by: Jen Kreder | Jul 12, 2011 2:48:15 PM
Jessie,
We've managed to achieve some of the results you're seeking vis-à-vis a rigorous academic support program. Importantly, this program is a bit different from standard ASPs in that it is not remedial and is not directed (at least not in relevant respects) towards underperforming students. Following the recommendations of the Carnegie Report and Best Practices, we've implemented this program as a proactive measure to make teaching and learning objectives more explicit. Here's how it works....
The ASP has weekly class sessions. In Week 8, let's say, students have one hour to respond to a Con Law essay. After that class, they then hand in their answer to their ASP professor (via TWEN). In Week 9, the ASP professor and the Con Law professor co-teach a review of the essay and hand out a model "good" answer and a model "not so good" answer. The review explicitly describes the positives of the "good" model answer and explains the flaws in the "not so good" answer. Students may see either the ASP prof or the doctrinal prof for individualized feedback, but we've found that the review largely satisfies many students. Each spring semester, we cover each of the doctrinal courses in this way.
But, your post focuses on the fact that only the top of the class takes advantage of feedback exercises like this. To that point, we have managed to get about 80% participation. How? I think there are three causes. First, the doctrinal professors have bought into the program, and this communicates to students that it's worth their time. If you are an isolated member of the faculty offering feedback exercises (or you are one of only a handful), students may read this as the law school as a whole (or the faculty) not finding these activities worthy. Thus, I think a concerted and pervasive effort is important ... as recognized by Carnegie and Best Practices.
Second, the program uses powerful formative assessment methods. We've found that the technique of giving out a good model answer and a "not so good" model answer is very effective. Students see both what TO do and what NOT to do. As such, they get a lot out of the exercise. Importantly, I think that having students hand the essay in to the ASP prof takes a lot of risk out of the exercise; if students must hand in their work to the doctrinal prof (even voluntarily), they might feel that doing so might "out" them to their doctrinal prof in terms of their weak abilities or their lack of diligence.
Third ... peer pressure. We've developed things to the point that we can tout some fairly persuasive statistics indicating that participation in these sessions leads to better performance, and the students (2Ls and 3Ls) are carrying that message forward. At some point, we reached a critical mass where the social norm became such that not attending the co-taught sessions was the exception, not the rule.
I think that an important note here is the deconstruction of some counterproductive, but long-held presumptions: (1) that ASP must be for struggling students and decontextualized from doctrine; (2) that formative assessment can only be individualized and only emanate from the doctrinal prof; and (3) that doctrinal faculty can't seek support from other teachers in the faculty to achieve pedagogical objectives.
I hope these ideas generate some discussion. (And sorry for the long comment).
Posted by: 4thyrLawProf | Jul 12, 2011 12:12:53 PM
I don't so much mind the analogy between law students (particularly 1Ls) and preschoolers. What I don't understand is why a law professor's role is analogous to that of the Children's Television Workshop.
I would suggest that you disclose any historical statistics you have on mock essay participants attaining higher final grades, as well as lay out the reasons it is a good idea (practice essay writing, gain insight into my standards, etc.). One of my law professors did this and still had a less than 50% participation rate.
Look, students who are able to figure out that writing the mock essay will help them are just going to better in law school and in life than their classmates. You can only do so much to change that--and what I wonder is why you seem to want to.
Posted by: Abe Delnore | Jul 12, 2011 11:59:31 AM
I'm about to start law school. This is interesting and motivational. I want to be in the have group by the middle of the semester, not the have nots!
I'm sure you'd be surprised how many students start the midterm answer, but don't turn it in because they don't feel like it's good enough. If you think that's the biggest obstacle, you could do things to make it a less formal assignment. Maybe another option could be for them to come to your office and talk through their approach at attacking the question, prior to writing it. Maybe allow students to email you an outline of what they'd want to cover in their paper.
Hell, what about this: offer the opportunity for them to come take the "practice exam question" in a simulated test environment - maybe it's easier for the less organized students to show up to a place with their notes and laptop than it is for them to plan and complete a project. For those brave souls who complete the practice exam, offer the opportunity to then join you for a happy hour beer or fructose beverage of their choice. Bribery works on preschoolers, after all.
One benefit for you of this last idea is that exams written in haste will probably be shorter than the treatises I'm sure you get currently, and less time to grade.
Posted by: Justin | Jul 12, 2011 11:47:18 AM
"Does anyone have any thoughts on how to minimize the Sesame Street effect, without making these exercises mandatory (and graded)?"
Well, kinda. How about
- offering the "test" at 3 points during the semester
- making participation in one of the first two test "mandatory" in the sense that successful completion improves their oral grade (or, if you don't give oral grades, counts as a "plus" toward the final exam).
- no student may participate in test no 3 unless you marked in no 1 or no 2 that he should (because he would have failed a real test); in that case only his second test (test no 3) counts for the oral grade / as a plus towards the exam.
So one not-really-mandatory* test for everybody and those who need it can turn in a second piece to show they learned from it.
* if the student is sure to receive an A anyways he doesn't have to participate ... ;-)
Posted by: Positroll | Jul 12, 2011 11:24:02 AM
Can you create law school gunners from undergrad slackers? Can the less intelligent law school classmates become equalized through taking an exam and getting some feedback where the exam question is materially different the final exam?
I think you're forcing an issue here. The people I knew in law school who did well were either 1) super studiers treating law school like a biglaw job or 2) inherently smarter and better writers then the median student. Likely those who took practice exams were those who studied the law to complete the exam, and it's just a subclass of 1) above.
Posted by: Marc R | Jul 12, 2011 10:59:34 AM
The comments to this entry are closed.