« Happy New Academic Year & Shana Tova | Main | Fashionably Late ... »

Thursday, September 09, 2010

It's All in the Numbers: How Can We Increase the Percentage of Students Completing Student Evaluations?

Student evaluations. Some people love them. Some people hate them. Some claim that they are reliable and reasonably valid. See, e.g., Arthur Best, Student Evaluations of Law Teaching Work Well: Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree, 38 SW. L. REV. 1 (2008). Others claim that they are unreliable and invalid. See e.g., Roger W. Reinsch et al., Evidentiary and Constitutional Due Process Constraints on the Uses by Colleges and Universities of Student Evaluations, 32 J.C. & U.L. 75 (2005). Meanwhile, others agree with the idea of student evaluations but believe that law schools must fundamentally change the way that we conduct them. See, e.g., Deborah J. Merritt, Bias, the Brain, and Student Evaluations of Teaching, 82 ST. JOHN'S L. REV. 235 (2008).

Some claim that attractive professors fare better on student evaluations than their less attractive counterparts. See, e.g., Todd C. Riniolo et al., Hot or Not: Do Professors Perceived as Physically Attractive Receive Higher Student Evaluations?, 133 J. Gen. Psychol. 19, 30 (2006). Others claim that male professors fare better than their female colleagues. See, e.g., Susan A. Basow & Nancy T. Silberg, Student Evaluations of College Professors: Are Female and Male Professors Rated Differently?, 79 J. Educ. Psychol. 308 (1987). Meanwhile, others claim that Caucasian professors fare better than their minority counterparts. Kathryn Pourmand Nordick, Essay, A Critical Look at Student Resistance to Non-Traditional Law School Professors, 27 W. New Eng. L. Rev. 173, 188, 191 (2005).

In this post, my intent is neither to praise student evaluations nor to bury them. Instead, I seek to raise a new issue with students evaluations and solicit opinions on how to correct it (or whether it needs correcting). That issue? The decrease in the number of students filling out student evaluations as law schools move from paper to online evaluations.

Student evaluations are actually a pretty new feature of the American education system. Apparently, "[t]eaching evaluations were initiated on many U.S. campuses in the 1960s, as students clamored for more of a say in their education." These evaluations thereafter didn't exactly catch on like wildfire, with "[o]nly about 30 percent of colleges and universities ask[ing] students to evaluate professors in 1973." Instead, the adoption of evaluations was more of a slow burn, with more and more schools deciding to use them to the point where it is now shocking if a school does not have student evaluations.

Initially, these evaluations were strictly a paper and pencil affair, with the first quantum leap being the addition of Scantron forms on which students could now bubble in whether their professor deserved a 1, 2, 3, 4, or 5 for organization, clarity, overall quality, and a host of other measures.  Recently, these traditional forms of evaluations have gone the way of the pager, with schools increasingly creating online student evaluation systems with the goals of saving money and increasing efficiency.

Many schools also thought that this conversion would increase the percentage of students completing student evaluations, but anecdotal evidence suggests that the conversion has had the opposite effect. For instance, at Northeastern, the percentage of students completing evaluations dropped from 80% to 54% after the school converted to online evaluations. See Getting students to make their mark.  I know that at my law school, we have seen a similar dip.

And I think that regardless of your view of the reliability and validity of student evaluations in general, we would all agree that student evaluations become more reliable and more valid as the percentage of students completing them increases. Moreover, I have heard that many colleagues ignore the overall numbers on their student evaluations and simply focus upon student comments. Obviously, if only 50% of students fill out evaluations, such professors have a lot less feedback to use than if 80% of students completed them.

So, what can and/or should schools and individual professors do? Our school has decided to award prizes randomly to students who fill out all of their evaluations, but it hasn't really worked. "Harvard, which moved to online evaluations in 2005, began allowing undergraduates who fill out the surveys to view their grades a few weeks early. As a result, participation rose to 96 percent, compared with 65 percent a few years ago." People at my school have floated the idea of students not being able to take their final exam until they fill out their evaluation or docking the grades of students who do not fill out evaluations. Obviously, there would be many negative consequences to such an action, which is why this remains just an idea. Schools could also go back to paper and pencil, but would they be willing to eat the cost and inefficiency (and would today's students even know how to use pencil and paper j/k)?

So, do any readers have any ideas? Is there anything that your school has done or that you have done as a professor to increase the percentage of your students completing student evaluations? And is the drop in the percentage of students completing student evaluations a problem we need to address or something that we should just accept?  

-Colin Miller

Posted by Evidence ProfBlogger on September 9, 2010 at 11:11 AM | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference It's All in the Numbers: How Can We Increase the Percentage of Students Completing Student Evaluations?:


I've been lucky enough to attend a law school where they use the online system and I think it works about as well as it could. Were they of the pencil/paper variety I don't think I'd ever get around to it. However a nice motivator would be to withold student marks until the evaluations have been completed. Maybe a little extreme but I feel it would do well to encourage their timely completion...

Posted by: undergraduate law | Oct 20, 2010 7:12:54 AM

My law school switched to an online system my last year. There was a bit of a brouhaha over the anonymity question.

My school also tried to entice more evaluation completions through pizza parties and an iPod drawing. This had no bearing on my completing them; nor did it have any bearing on many of my classmates. (Interestingly, the 1Ls seemed most motivated by this at the time.)

Of the above comments, I think the constant spam reminder until final exams might have been the best motivator for me. And, sure, getting the exam ID would have worked to -- except my school gave us one for our entire three years.

The biggest problem with evaluations that I saw was folks not taking it seriously. It was a hassle to write comments on the forms. It was even more of a hassle to log in to do the online evaluations. Couple this with it often being done prior to or in-lieu of a class and folks just made stuff up on the multiple choice and didn't write comments.

Posted by: John Nelson | Sep 13, 2010 9:36:04 AM

Just a thought. Are the students 100% confident that their evaluations will remain anonymous? Is that confidence level in those who have it reasonable?

I remember not filling out an "anonymous survey" in college, only later to be called and told I still needed to fill out my survey!! Yep, sure enough, on the back of the page, in invisible ink, (but quite visible and legible under a Wood's light) was my name!

Posted by: Uno Hu | Sep 10, 2010 6:19:23 PM

Studies show that evaluations are unreliable if they go below a 66% completion rate. William E. Cashin 1999. Therefore, we need to get the completion rate up.

Posted by: Mark | Sep 10, 2010 12:08:38 PM

Penn has a system similar to the one from Pace, mentioned by Emily. To take exams, students need an exam ID number. You can't get your number until you fill out your evaluations on-line (and you can't do that until the reading period.) Students don't need to actually fill them out, though- they can just click through them all w/o actually filling them out and get the number. But, once there, it is easy to fill them out, so I suspect most do. (Also, when I've taught, I've always told the students that I take the evaluations seriously and that I especially appreciate the written remarks [and I think I've improved my teaching because of those, or at least have tried do], and that I would appreciate it if they took them seriously, too. I've always had a good response rate, though I can't say if this is caused by my specifically asking them to do the evaluations or not.)

Posted by: Matt Lister | Sep 9, 2010 4:10:52 PM

At Pace, students don't get their exam ID numbers (which they need in order to take their finals) until they submit their course evaluations.

Posted by: Emily Gold Waldman | Sep 9, 2010 3:30:39 PM

At one school I'm familiar with, the library entices students to fill out surveys about library services by entering all the participants in a drawing for a prize, like an iPod or gift card. I don't know how effective it is, but at least it provides some incentives for the students.

Posted by: Hanah | Sep 9, 2010 1:03:44 PM

At the school I attend, we do online evaluations. About three weeks before the end of the term, an automatic e-mail is generated from the registrar's office reminding us to fill out evaluations. There's one e-mail per class, and the e-mails come every three or so days. The only way to make the e-mails stop is to fill out the evaluation. (Or wait until the start of the final exam period, 5 or so weeks later.) It's pretty effective for me. I always fill out the evaluations right after I get the first e-mail, just to stop the spam.

Posted by: Anon 3L | Sep 9, 2010 11:39:17 AM

Yes, the quote refers to the college, not the law school.

Posted by: Colin Miller | Sep 9, 2010 11:34:01 AM

Colin, I'll just clarify as to your quote regarding Harvard (presumably the college?) that the law school still uses paper and pencil evaluations...

Posted by: I. Glenn Cohen | Sep 9, 2010 11:23:48 AM

The comments to this entry are closed.