« Can I Say "Hullabalooza?" | Main | An interesting church-state anniversary »

Wednesday, April 22, 2009

The "Seminar Exception": Is It Defensible To Remove Smaller Law School Classes From The Grading Curve?

Here's something that gets the goat of many law students: uncurved law school classes. At most law schools with grading curves, students taking classes with roughly more than 30 students will be graded according to the grading curve. Students taking classes with about 30 students or less will not be graded according to the grading curve. So, what happens in these smaller classes (usually seminars)? Well, according to Robert M. Lloyd, Hard Law Firms and Soft Law Schools, 83 N.C. L. Rev. 667, 678 n.62 (2005):

At my institution, a report showing the average grade in each course is circulated to the faculty at the end of each semester. These show the grade disparity very clearly: in Spring 2003, 14 of 18 upperclass exam-graded courses had a mean grade below 3.3, while only one of 16 paper-graded classes had a mean grade below 3.3....All of the anecdotal evidence from other law schools confirms that higher average grades in Softer courses is a universal (or near universal) phenomenon in American law schools. For example, one U.S. News top-fifty law school provided me with a semester grade report listing two seminars in which all students received A's....While some of this grade disparity is due to schools giving more grading discretion in smaller classes and non-exam-based classes, many schools have actually mandated higher mean grade ranges in "seminars." (emphasis added).

Now, this gets the goat of many law students because some 2Ls and 3Ls load up on these smaller classes, "artificially" inflate their GPAs, and end up with higher class ranks than students who outranked them after 1L year and then took mostly curved classes in their last two years. The question is whether uncurved classes are defensible. My (tentative) answer is that they are not.

It seems to me that, as Barbara Glesner Fines noted in Competition and the Curve, 65 UMKC L. Rev.879, 893-94 (1997) there are three main grounds upon which these "seminar exceptions" might be defended:

These “seminar exceptions” may be based on two rather divergent rationales:one relies on the randomness of ability, the other relies on the educational effectiveness of these classes. The first rationale exempts small classes from a curve because these classes do not provide a statistically significant sample--there is less likely to be a random distribution of ability in these classes, so we cannot expect a curve to result. Even if there is a random distribution of ability, many seminar classes allow such a degree of flexibility in the products by which students are measured (e.g. seminar papers) that we do not trust our ability to adequately differentiate student ability in these settings. The operative assumption here is that the purpose of law school is to test the ability of students to perform rather than to teach the students to perform ably.

However, given that these seminar exceptions usually do not completely eliminate the requirement of a curve, but rather increase the required means or broaden the distribution requirements, another rationale seems equally likely. This rationale accepts the legitimacy of higher grades when instructional effectiveness is higher. The rationale is well supported by educational literature. Students do truly learn better in smaller rather than larger classes. Students do learn better when they have more formative evaluation (such as rough drafts, practice arguments, etc.). Students do learn better when they have the ability to choose their topic of study (as is often the case in seminars).

In other words, smaller classes should not be curved because (1) they might not contain a random distribution of students, (2) it is difficult to distinguish students in such classes given the flexibility of grading measures; and (3) instructional effectiveness is higher.
 
There is some merit to each of these arguments, but I think that there are also some problems. With regard to argument 1, here are my thoughts. First, aren't the current policies at most law schools underinclusive? I have two examples from when I was in law school: (a) I took a moot court class in law school with the other 31 students who had made the Moot Court team, and (b) I took Federal Courts along with many other members of Law Review, Moot Court, and the top 10% of the class. Both of these classes were curved. Neither had anything close to what I would call a random distribution of students, and I doubt that they are the only examples. If law schools are worried about certain classes not having random distributions of students, shouldn't they just review the GPAs/class ranks of students in particular classes and see whether the law school curve needs to be altered or removed from that class?
 
Second, how did most law schools reach (about) 30 as the magic number where there is no longer a statistically significant sample of students? 168 students graduated in my law school class, and classes with fewer than 30 students were uncurved. Here at John Marshall, where I now teach, classes with 30 or less students are also uncurved, but the overall class size is much larger. From what I have heard, most schools cut off their curves at about the same number, regardless of their overall class sizes. How can 30 be the magic number when overall class sizes can vary by hundreds of students?
 
Third, accepting the argument that there might be random distributions of students in smaller classes, wouldn't we expect a non-insignificant number of those classes to result in mean grades below the law school mean? And yet, based upon the data compiled by Lloyd (and my own personal observations), noncurved classes almost always end up with mean grades significantly above the law school mean. It thus seems to me that the remedy of taking smaller classes off of the curve is worse than the ailment it is intended to cure.   
 
With regard to argument 2, here are my thoughts. Initially, doesn't the flexibility of grading measures help (albeit in a different way) to counterbalance the statistical significance problem mentioned above? Most "larger" law school classes contain one point of grading: the final exam. Most "smaller" law classes (at least the ones I took) had at least three points of grading (class participation, some type of presentation/oral argument, and a paper). I always feel a bit uncomfortable giving students grades on a curve based just upon the final exam. Conversely, I would imagine that I would feel much more comfortable distinguishing between students if I had several points of grading (I haven't yet taught a seminar).
 
Also a related argument to argument 2 is the argument raised by Jeffrey Evans Stake in Making the Grade: Some Principle of Comparative Grading, 52 J. Legal Educ. 583, 601 (2002):
For a class of thirty students,...forcing the scores onto a curve will be troublesome because some of the intervals will contain very few students. For classes of the size that are common in law schools and for grading scales with more than a few intervals, forcing a specific percentage of students into each interval is riskier than mandating a standard deviation because each of the intervals involves only a portion of the class, whereas the standard deviation relates to the whole class. With the whole class, small variations will average out in a way that is not likely when an interval includes only a few students.
In other words, if a professor needs to give a certain percentage of Bs (or Bs, B+s and B-s), it can be arbitrary to force the professor to make fine distinctions where intervals might contain only 1, 2, or 3 students. This seems to me to be a valid criticism, but it seems to me that the remedy is removing intervals and not removing the law school mean. In other words, if curved classes at a school require a 3.0 mean and 10% of students getting a B-, a professor in a class with 20 students shouldn't have to adhere to the 10% B- requirement, but it seems that the professor should have to adhere to the 3.0 mean requirement.
 
Finally, with regard to argument 3 here are my thoughts. First, aren't the current policies at most law schools underinclusive? I give an ungraded midterm in each of my classes. On most of my midterms, the mean grade is in the low to mid 60s (out of 100). In my Criminal Procedure I class this semester, my students blew that mean away, beating the previous high mean by over 10 points. But even if these students repeat their performance on the final, they are still stuck on the curve despite one conclusion being that instructional effectiveness was higher than (my) usual in this class (of course, another argument is that I didn't have a random distribution of students;).
 
Second, aren't the current policies at most law schools overinclusive? Sure, many "smaller" classses are popular seminars taught by terrific professors that have less than 30 students based upon an artificial cap. But, let's face it. There are also some smaller classes that have less than 30 students because, well, the professors in those classes just aren't that great. For instance, I took a 12 student criminal law seminar in law school with an AUSA who cancelled class half of the time and based students' entire grades on a 10-12 page paper (on which I believe everyone got an A or an A-).

So, what are your thoughts/experiences? Does your law school remove smaller classes from the curve, and is (about) 30 the magic number? Has your experience been that these uncurved classes typically produce mean GPAs well above the law school mean? Do you think that making these classes uncurved is defensible? Is there a middle ground that can and should be reached?

Posted by Evidence ProfBlogger on April 22, 2009 at 08:58 AM in Teaching Law | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef011570323392970b

Listed below are links to weblogs that reference The "Seminar Exception": Is It Defensible To Remove Smaller Law School Classes From The Grading Curve?:

Comments

How big a problem is this? The answer has to depend on how many classes of each size a law school offers, and on how the opportunity to take smaller classes is distributed among students. Precisely because small classes are small, they don't affect as many students. It's like the jerk who parks next to the fire hydrant. Sure it's unfair to the dozens of people circling the block, but it's only one parking space. I'd love to see some numbers.

NYLS sets 20 as the cut-off number, by the way. In my limited experience, I've found that the quality of participation and the quality of exams in my genuinely small classes has been such that trying to grade proportionately to the curve that makes sense in the larger classes would have underrated the performance of the students in the smaller classes.

Perhaps one middle ground option would be to have a curve that kicks in fully at, say, 25, but to have the curve fill in from both ends as the class size goes from 0 to 25. That is, for a class size of 15, the professor can give out grades no higher than the top 15 grades that would be allowed for a class size of 25--and no lower than the bottom 15 grades that would be allowed for a class size of 25. This would give more flexibility in small classes, but it wouldn't create a single arbitrary cut-off line.

Posted by: James Grimmelmann | Apr 22, 2009 11:12:00 AM

Great post. GW has a strict grade distribution requirement for classes above a certain size, and a less specific requirement for smaller classes (you're supposed to try to adhere to the stricter grade distributions, but even if you don't, under no circumstances may more than half the grades be in the A-range.)

Though I do generally agree with this approach to grading, I struggled with it last semester when I taught an 8-student section of Partnership Tax. This was not a seminar; Partnership Tax is just so extremely hard that not many students registered. The students who did choose to take the class were eight extremely smart, motivated, hard-working students. Their participation was, to a person, outstanding. Their exams were uniformly excellent. These were the students who had been at the top of the class in both basic and corporate tax. Indeed, had more people registered for Partnership Tax, I feel confident that all of these 8 students would have been at the top of the class in Partnership Tax as well. But I was stuck giving half of them B-range grades.

Posted by: Sarah L. | Apr 22, 2009 11:23:43 AM

Would it work to curve upper level classes around the average 1L GPA of the students in the class? This might create more fairness across the board, because I agree that classes like fed courts are an issue.

Posted by: anon | Apr 22, 2009 11:39:47 AM

Thanks, James. All I have is anecdotal evidence from the law school I attended (William & Mary) and the law school where I teach (John Marshall). From what I have seen at both of these schools, there are enough novcurved classes at both for 2Ls and 3Ls to fill their entire "elective" schedules with noncurved classes.

You also say, however, that the answer depends in part "on how the opportunity to take smaller classes is distributed among students." I'm not sure that I agree on this. Assume that Law School has a B curve and that

Student A chooses to take mostly curved classes and gets mostly As and A-s, like most other students in those classes, and

Student B, who chooses to take mostly curved classes and gets mostly B+s.

Is it "fair" that Student A has a higher 2L/3L GPA than Student B because Student B had the option to take mostly noncurved classes? In my opinion, the answer is "no."

The defense in your second paragraph seems to be the "institutional effectiveness" defense. And I agree with that defense, but, as I noted, I think that it should apply equally to "larger" classes where students overperform (like the Criminal Procedure class mentioned in my post). And I would be surprised if students in seminars always overperform (at least that wasn't my experience in law school), and yet it seems that seminar grades are (almost) always (significantly) better than the law school mean.

Finally, your "graduated curve" option seems interesting. I could see that being a great alternative to the status quo.

Posted by: Colin Miller | Apr 22, 2009 11:52:18 AM

Thanks Sarah L. and anon. It seems to me that the best way to address classes such as Federal Courts and Partnership Tax is, as I noted in my post, and as anon notes in his comment, to "review the GPAs/class ranks of students in particular classes and see whether the law school curve needs to be altered or removed from that class." As anon notes, it might make even more sense just to look at 1L GPAs/class ranks. I would much prefer this approach to the "seminar exception" approach.

Posted by: Colin Miller | Apr 22, 2009 11:57:30 AM

I'm always amused and frustrated when people ask the "why 30?" question. It's the math behind the statistics, combined with more math on confidence in the result of the statistical tests. In summary, a sample of 30 of a larger population -- if if is a truly random sample of that larger population -- should replicate the actual characteristic of that larger population. That is, my sample of 30 should have the same distribution of the characteristic at issue as the entire law-school class of 225. Conversely, the fewer data points in my sample, the less confidence I can have that the sample's characteristics accurately predict the population's characteristics.

Needless to say, this sort of reasoning is actually inapplicable. For one thing, it depends upon a random sample from the larger population... and, as the word "elective" makes clear, the samples are anything but random! For another, it actually asks a completely different question: If the class had been the entire population, and not a sample of it, what would the actual performance have looked like on the grading instrument? However, since teaching methods, grading methods, testing instruments, and so on will necessarily for that larger group, it's an impossible hypothetical.

All of that said, there's another issue lurking in this discussion: Are the grades assigned in upper-level classes as a whole, and in particular in seminars, meaningfully correlated to student achievement in the first place? Establishing a grading curve assumes the answer to that question is "yes"... but it's very much like trying to correlate actual bar-exam scores with success in practice.

In short, grading curves of this nature don't answer the questions they purportedly answer. Given my druthers, I'd mandate that all seminars be graded on a pass/fail basis, with failure being rare; the individual students' urge to excel should be enough incentive, particularly if the instructor is at all engaged. I'd even say the same for larger classes, but one step at a time!

Posted by: C.E. Petit | Apr 22, 2009 12:08:39 PM

A course-wide statistic that's very useful for looking at grading inconsistencies within a law school's program of study is the average GPA of the students in a course, not counting that course. It can help distinguish cases in which students who do well take a particular course because it lets them really push themselves (e.g. Sarah L's Partnersnip Tax) from cases in which students take a particular course because it offers them an easy good grade.

I mentioned the "opportunity" to take smaller classes because at some law schools, the limited-enrollment course rules and rules about things like writing requirements can serve the role of rationing access to smaller courses. That's not a good thing in its own right, but it can counteract some of the grading-unfairness issues, at least.

Posted by: James Grimmelmann | Apr 22, 2009 12:57:21 PM

Thanks, C.E. Petit. Is your comment at the end regarding pass/fail grading directed at every law school class or just 2L and 3L classes? If it is just directed at 2L and 3L classes, the problem I see is that it places even more emphasis on 1L grades. If it directed at law school in general, it would be very tough (especially at lower ranked schools) for students to distinguish themselves.

Posted by: Colin Miller | Apr 22, 2009 1:26:44 PM

Thanks, James. I am liking this idea of law schools looking at the GPAs/class ranks of students in a particular class and modifying/eliminating the curve for certain classes more and more.

Posted by: Colin Miller | Apr 22, 2009 1:28:50 PM

The excerpts from my article appear to suggest that I approve of curved grading and am making an exception for seminar classes only. Just to be clear, the article argues that systems of mandatory curved grading, and ranking systems especially, serve primarily to sort students for the distribution of wealth and privileges values of competition and control over values of learning, equality and professionalism. Were it not for the need to sort students for the Big Law marketplace, I would not use norm-based summative evaluation at all, but would provide much richer formative assessment and would adopt a grading system much as the pass-fail system C.E. Petit proposes.
Peace,
Barb Glesner Fines

Posted by: Barbara Glesner Fines | Apr 22, 2009 2:10:51 PM

At Florida State, one grading option for classes with fewer than 30/35 students was a "Class Profile" rule for grade normalization. I forget exactly who it worked, but basically the class mean had to be within a range (it may have been +/-2 points on a 100-point scale) of the class mean GPA entering the course. This allowed for a certain amount of grade inflation in those smaller classes that draw the better students (such as Federal Courts).

Posted by: Howard Wasserman | Apr 22, 2009 2:11:25 PM

Thanks, Barb. I didn't mean to set you up as the defender of the "seminar grading." While researching the post, I came across your article and thought that the reasons you listed for not curving smaller classes matched with the defenses I generally heard, so I included them in the post. And I agree that were it not for the need for sorting, I (think that I) would engage in some type of pass/fail grading.

Posted by: Colin Miller | Apr 22, 2009 2:29:17 PM

First, I teach at the same law school—the University of Tennessee--as Professor Lloyd, the author of the first-quoted article above. Since the focus of this post is curved versus un-curved, pleases note that there is no mandatory or suggested curve for any class at our law school. There is what I would call a customary curve in first-year classes, but faculty are free to assign grades without regard to that custom. While Professor Lloyd’s anecdotal experience may be relevant to a discussion of grading outcomes in large classes versus small classes, mandatory curves are not a factor here.

To the extent that one believes there is a problem with average grades in small classes being higher (I'm not convinced that it is a problem, for some of the reasons discussed above), curving the class based on the prior GPAs of the enrolled students is a terrible response. It presumes that first-year law school grades are some sort of supreme assessment of students’ abilities, even though the typical first-year curriculum is geared to particular types of students with particular learning styles and backgrounds. (Not to mention the increased stress such a system would put on 1Ls and the havoc it would wreak on course registration and add/drop.) Really, why not just assign class rank according to LSAT score and skip the bother of grading all those exams?

Posted by: Jennifer Hendricks | Apr 22, 2009 4:27:35 PM

Thanks, Jennifer.

First, I'm interested in what you have seen at Tennessee. Given discretion and a "customary curve," do you get the sense that many or most 1L professors follow the customary curve pretty closely or that many or most (significantly) deviate? I like the idea of a customary curve combined with discretion, but if certain professors exercise that discretion too much, it could be unfair to students in other classes (or students in their classes if they give grades lower than the customary mean).

Second, I think that you have levied some compelling criticisms against using the prior and/or 1L GPA/class rank of students in a particular class to change/remove the curve. As you note, the skills tested in most 1L classes are different from the skills tested in most seminars. And, as I noted above, I agree that it is problematic to placing undue emphasis on 1L grades.

At the same time, I just hate that there are seminars where, semester in and semester out, certain professor s give every student an A or A- while classes such as Federal Courts are almost always packed with "top" students and yet subject to the curve. Maybe Tennessee's approach is the best way to address these issues.

Posted by: Colin Miller | Apr 22, 2009 5:04:48 PM

I always thought that the curve exemption for seminars was due to the paper-writing requirements. At my school, seminars with papers required are limited to 15 students (well below the 20 student cut off for non-graded classes). I can't imagine grading substantive papers through a curve!

Posted by: Laura | Apr 22, 2009 5:28:09 PM

Thanks, Laura. If the curve exception is based upon the difficulty in grading papers according to a curve, from what I have seen it is overinclusive. I am aware of several exam classes with less than 30 students that were not subject to the curve.

Assuming, though, that the basis for the curve exception is the difficulty in grading substantive papers on a curve, I wonder whether that basis is legitimate. In other words, is it more difficult to curve papers than it is to curve exams? I can't speak to that issue because I haven't yet taught a "paper" class, but I don't think that I would have more trouble curving papers than I have curving exams (after all, isn't this partially what many law professors did when they were selecting/rejecting articles while members of law review?). Moreover, as I noted in my post, in a "paper" class, professors often don't grade students based solely on their papers, but also based upon their class participation, oral presentations, etc. I would think that these multiple points of grading would make the curving process easier.

But, as I said, I haven't taught a "paper" class yet. Does anyone who has want to speak to this issue?

Posted by: Colin Miller | Apr 22, 2009 7:21:28 PM

I found papers easy to curve when they were on the same or even similar topics. Where the students were allowed to pick their own topic and develop it, though, I found it nearly impossible. The top couple - the students who had really gone above and beyond - were clear, as were the bottom couple. But in the middle were students who had gone in vastly different directions and had vastly different strengths and weaknesses and I simply couldn't find what seemed like a fair way to rank them.

This was different than what I did in selecting law review articles, where basically you are culling out the top few that are clearly superior instead of trying to distinguish between the large category of rejects. Anyway, everyone accepts that many good, even great, articles won't be accepted by a particular journal, whereas grades are supposed to be an objective measure of quality.

Posted by: anon | Apr 22, 2009 7:43:33 PM

I think it's worth considering that the grading curve itself is only defensible if you believe that curved exams meaningfully differentiate between students. That is, they clearly and fairly measure "merit" (something relating to subject-matter knowledge and intellectual ability) -- they're an objective measure of quality.

(As a 3L who has gotten good grades, I'm skeptical of this. A 4.0 student who consistently gets As is probably better than a 3.0 student who consistently gets Bs, but does anyone really believe that the difference between an A- and a B+ on a given evidence exam [or, on a given exam, quite possibly even a broader range than that] really meant as much in actual performance as it does with respect to the student's GPA?)

Posted by: joe. | Apr 22, 2009 10:43:01 PM

When I was in law school (about a decade ago), my top-twenty law school made class-by-class curves available so you could see, for example, how many people in each class got As, Bs, etc. (I don't think they do this any more.) I put this data into a spreadsheet and generated a graph of the percent of students who got As as a function of class size. Smaller classes got significantly more As (as a percentage) than larger classes did. I wasn't very surprised by this.

Posted by: anon | Apr 23, 2009 12:13:58 AM

Thanks, anon (7:43:33). I think it makes sense that it would be easier to curve papers on the same subject than it would be to curve papers where students are allowed to pick their own topics. My school gives professors the option of giving a different exam to students who have an exam conflict and thus have to take the exam earlier or later than the exam date. I never choose this option because I would have no idea how to put students taking different exams on the same curve.

Posted by: Colin Miller | Apr 23, 2009 9:34:17 AM

Thanks, joe. I think most professors realize that the one exam/one curve format of law school grading is used more as a necessary sorting technique than something that we should expect to be a completely fair and objective measure of quality (at least with regard to the "middle" of the class).

Posted by: Colin Miller | Apr 23, 2009 9:37:34 AM

Thanks, anon (12:133:58). My law school did the same thing until it changed the grade curve (from a B to a B= curve). I did the same thing as well and reached the same conclusion.

Posted by: Colin Miller | Apr 23, 2009 9:38:46 AM

Hi Colin,
Most 1L classes end up pretty close to the customary curve, with deviations mostly when a particular section happens to do especially well or especially poorly in class. The same is probably true for many of the large upper level classes. I think there are several legitimate reasons why grades in small classes might be higher, but also some worrisome reasons, and I'm pretty sure that both kinds are in play.

Posted by: Jennifer Hendricks | Apr 23, 2009 10:43:05 AM

But Colin, why is it a "necessary" sorting measure if it's a bad measure? And if it's a bad measure, why is it problematic to throw in uncurved seminars? I realize you're suggesting that it may only be a poor measure with respect to the middle of the class (I think that would surely be a large middle, however), but isn't it that group of students (and not the few all-stars at the very top of the class) that will be impacted by uncurved seminars?

Posted by: joe. | Apr 23, 2009 11:58:44 AM

Thanks, Jennifer. It seems as if this soft curve approach could be a good middle ground.

Posted by: Colin Miller | Apr 23, 2009 1:06:38 PM

Thanks, joe. My point is that, while the law school curve is not perfect, at least it should be consistent. In other words, the curve might not do a good job of sorting the "middle" of the class, but at least that same curve would be applied to all classes if we removed the "seminar exception." Under the current approach, I don't think that seminar/small class non-curved grading is perfect either. And from I've seen, it results in grades significantly higher than the law school mean.

Posted by: Colin Miller | Apr 23, 2009 1:10:24 PM

I have never heard of this practice prior to this post. Is it common for students to take more "uncurved" classes to bump up their GPA? If so, does it matter? By acknowledging this exception to the general rule (that students take classes for various reasons but most likely because a) they are interested and b) the class "fits" w/ in their schedule) aren't we perpetuating the already ridiculously competition among law students?

At both the law school I went to and the law school I now teach at, I have never heard of a student employing this strategy. Perhaps I live in a bubble or perhaps I never thought to employ this strategy because I was too busy focusing on my own work to worry about competing with others.

Not to take this completely in another direction but I think that the hyper-competitiveness of law students is ridiculous and "creating" issues like this and having discussions about them only fuels the fire. Plus, as more top schools play the US News game and stop giving grades (i.e. Yale), will this discussion even matter anymore?

Posted by: anon anon | Apr 24, 2009 8:33:22 AM

There are definitely students who take more seminars because they think it will help their GPAs. Though I've heard multiple explanations: papers instead of exams as well as higher grades to begin with.

Posted by: anon | Apr 25, 2009 5:56:42 PM

Post a comment