Tuesday, December 18, 2012
Student Class Evals, Transparency, & Self-Selection Bias
When catching up with one of my former students/RAs yesterday, he brought up the topic of student evaluations. More specifically, he asked what influence, if any, they have on law professors’ careers. Given that I am still on the junior side of the tenure divide and am still learning about how my university’s promotion procedures operate, I didn’t have a great answer for him. My impression is that glowing evaluations are helpful in the tenure-track professor’s quest for permanence, but that they have less institutional importance past that point. Whether student evaluations should play a larger role in the tenure-track/promotion/retention process is an interesting question, but not one that I do not feel particularly qualified to comment on. My student’s question did, however, lead me to consider a pair of related issues.
Accessibility to Students: My impression is that most law schools (like undergraduate programs) do not make class evaluation data available to students. This has always struck me as a bit peculiar—why shouldn’t students have access to information that they help create and that will help them make intelligent scheduling decisions? Making professors’ aggregate scores available online would cost nothing and require little additional administrative work. While having such information publically accessible might be uncomfortable for those who have received low scores, I am not sure whether this harm is actually a bad thing (it could incentivize those individuals to put greater effort into class prep or seek outside assistance) or unique (student bodies have institutional memories, websites exist where students can publically rate and comment on their professors). What, if anything, am I missing here?
Data Collection Methods: From talking to fellow junior profs and my lawyer friends who attended different law schools, it seems that most (if not all) schools have transitioned to online-based evaluations and that the majority of schools have not made filling out a class eval mandatory. The former makes sense to me, but the latter does not—by making evaluation optional, schools invite self-selection bias and drastically decrease the meaningfulness of the entire endeavor. One way of instituting mandatory reviews (used by my alma mater) is to make a student’s completion of a course’s online survey a prerequisite for viewing their grade. I imagine that other institutions have tackled this issue in other ways—anybody care to share their institution’s approach?
Posted by Max Helveston on December 18, 2012 at 10:45 AM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference Student Class Evals, Transparency, & Self-Selection Bias:
At Harvard our student eval summaries with aggregated data are all available online to our students when making course selections.
On the question of data collection, here is a good empirical methods article on the subject Daniel Ho and Timothy H. Shapiro, Evaluating Course Evaluations: An Empirical Analysis of a Quasi-Experiment at the Stanford Law School, 2000-2007, 58 J. LEGAL EDUC. 388 (2009).
Posted by: I. Glenn Cohen | Dec 18, 2012 10:57:26 AM
Georgetown also has aggregated data available to the students.
Posted by: Adam | Dec 18, 2012 11:14:57 AM
Suffolk makes all the numerical data available online for upper level classes. I don't know whether a student could get access to the comments by going to the registrar's office.
I don't think the evaluations for first year courses are published, but the students don't have any choice in that regard.
Posted by: Jeff Lipshaw | Dec 18, 2012 11:17:26 AM
All the law schools at which I have taught (UCI, GW, UVA, and NYU) make teaching evaluations available to students, both the numerical and, I believe, the narrative portions of the evaluations.
The Ho and Shapiro article is good but is also, I think, somewhat out of date, or at least limited due to its focus on one particular law school. For example, while filling out evaluations is not mandatory at any of the schools at which I've taught, at UCI and GW (at least), and I think at UVA and NYU too (though I'm not 100% sure about them), a portion of classtime is set aside for filling out the evaluations and the response rate is therefore generally relatively high.
Posted by: Sarah L. | Dec 18, 2012 11:40:06 AM
Columbia requires professors to set aside 15 minutes at the end of one class for student evaluations. Students may view course evaluations at any time. Some semesters, students who completed evaluations for all their courses would be entered into a raffle for an Amazon git card or some other equivalent prize. It was a factor in whether I filled out the evals, but I'm not sure whether my participation could be called "meaningful."
The evaluations are two sets of questions. One set are multiple choice questions such as "how many times did you attend class," "what is the professor's style of in-class discussion," "how helpful were the course materials." The second are fill-in-the-blank open-ended questions- "share an anecdote and the class," "what could this professor improve upon." There were more responses to MC questions than the short answer sections.
One thing to remember in terms of evaluations is that, like law school exams, they tend to be "curved." If you ask me how I liked the casebook, I will compare the casebook to other casebooks. If you ask me whether I thought the casebook was worth $160 versus materials provided by the professor free of charge, use of a hornbook/supplement, or other course materials, you might get a very different answer.
Posted by: BoredJD | Dec 18, 2012 1:40:35 PM
One thing you may have told your student is that evaluations do seem to play a large role in faculty hiring. When I went on the market recently, every school wanted to see copies of teaching evaluations, and I was asked about particular outliers. Now, I know that these are usually aggregated in to a memo to the faculty, but I can imagine that relatively poor or relatively excellent evaluations can make a difference in hiring at many schools.
Posted by: anonandoff | Dec 18, 2012 2:07:51 PM
anonandoff - i would say your impression is generally wrong. I am a second year tenure-track prof and while i had a similar experience to you while on the market (every callback school wanted to see evals) I have little sense (given various anecdotes) that they played any role whatsoever. at my own school now I see what a very small and insignificant role they play in our hiring. for reference, I teach at a top 100 school that actually places a terrific emphasis on teaching.
Posted by: anon 2nd year law prof | Dec 18, 2012 11:37:11 PM
UChicago (undergrad, not sure about the law school) makes full evaluation data (multiple-choice scores and comments) available online to anyone who's either (a) on the campus network or (b) has a school login. Those restrictions cut down somewhat, I think, on concerns about the data being too public.
Students are required to either fill out an evaluation or affirmatively opt-out in order to see their grades. Professors can only see the evaluations once they've submitted their final grades.
Posted by: Andrew MacKie-Mason | Dec 19, 2012 2:11:04 PM