The Educational Quality of Law Schools: What's the Relevant Data?

Yesterday, I laid out conceptually how we might think about comparing the "value added" for students across law schools in order to fill out the U.S. News survey. A critical component of this, of course, will be relative educational quality, and one obvious question is how exactly to get at that.

In talking about the rankings as part of a 2000 symposium on law schools and the legal profession, Stanford's Deborah Rhode, a leading ethics and gender-and-the-law scholar as well as a former president of the AALS, answered the question this way: "Prospective students need more comparative data, and schools need more incentives to compete, across a broader range of characteristics than current rating systems address. So, for example, applicants might benefit from approaches adapted from undergraduate education that evaluate schools by reference to "best practices" in teaching. Such approaches can provide comparative data on students' experiences on matters such as faculty contact, effective feedback, skills instruction, and collaborative projects."

With that and other scholarship on teaching and learning in legal education in mind, below is a first stab at the kind of information that one might want in an "Voters' Guide" to the U.S. News survey of law professors, lawyers and judges.

And before you say, "Wait a second, how can this possibly be objective?" remember, U.S. News is not asking law professors, lawyers and judges for objective information -- they've got that already. They’re asking for our expert opinions (rate from 1 to 5) as to the quality of different law schools' programs – and they’re not asking about reputation. We make evaluative judgments about quality all the time in the absence of precise quantitative formulas -- we compare the quality of doctors, restaurants, books, and movies. Most law professors and many lawyers are fortunate enough to be in the income bracket where we can ask: "Is the local public school good enough for our kids? Or should we spend the money to send them to private school? If so, which private school is the best?" Are you really telling me that we can assess the relative educational quality of elementary schools, but not law schools?

Let me also be as transparent as I can about the assumptions relied upon here. First, the overall concept: to a certain extent, Korobkin is right. In today's legal education marketplace, it is difficult to distinguish among the legal education that one gets at one law school versus another. Most law schools have a similar curriculum, with the new exceptions of Northwestern and Washington and Lee, and existing innovators like Georgetown's first year, the University of Dayton and others. One could say that all law schools have a mix of teachers that use techniques that engage students more or less, etc.

So the idea here is to focus on the areas of likely distinction. We have two recent, comprehensive reports on legal education – and in many ways, as these reports acknowledge, their assessment of the relative strengths and weaknesses of legal education are those that have been made for years, going back to the Carnegie Foundation’s 1921 study by Alfred Reed, Jerome Frank's call in the 1930s and 40s for a lawyer school, etc.

We know what the weaknesses are. The question is: which schools are currently doing the most to address the long-identified weaknesses of legal education? Those schools are doing a better job to prepare their students for the practice of law that those who are not. That's the theory. The other meta-point reflected below is to listen to students and recent graduates -- yes, surveys of student satisfaction, student evaluations, are imperfect. But students have a good sense of when they are more or less engaged, when they are getting excellent or subpar help from student and career services, etc. And in the aggregate, this can help us compare schools on these measures.

The specific assumptions below are drawn mostly from the recent landmark studies of legal education done by the Carnegie Foundation ("Educating Lawyers") and Roy Stuckey and others ("Best Practices For Legal Education"). Obviously, the assumptions are open to challenge; but I don't think you can challenge the people who spent years digging into learning theory, professional education, and the connection between the two without a real theory or research of your own. I.e. "I've been a law professor for a while, I've been a law student, I don't think the case method is overused" doesn't cut it.

So the conclusions from the Carnegie Report, Best Practices, and elsewhere are:
(1) Active learning leads to better outcomes than passive learning.
(2) The Socratic case method, focusing on the in-class dissection of appellate cases, is overused in the second and third years of law school, leading to student disengagement.
(3) Student-faculty contact helps increase analytic ability. (LSSSE 2006)
(4) Law students are generally undertrained in skills like interviewing, counseling, factual investigation, and negotiation. (this goes back to the MacCrate Report, at a minimum)
(5) Just as we would be shocked to allow doctors to practice medicine without first having seen a patient, we should expect law students to have experience dealing with clients before they do so "for real." Relatedly, law students are too infrequently put in the "role" of lawyers.
(6) Feedback is critical to learning. Law schools tend to be terrible at providing feedback.
(7) Coursework that integrates doctrine, skills, and issues of professionalism is better than coursework that deals with these on an isolated basis.
(8) The opportunity to specialize in a particular area of law is important, and leaves the student who specializes in a significantly better position than someone who does not and goes into that area of law.
(9) The work that most lawyers do can be broken down into three categories: litigation, transactional, and regulatory (Rubin 2007). Law schools tend to do a decent job at preparing students for litigation, but a bad job at preparing students to do transactional and regulatory work.

So here are some pieces of data we might use to evaluate relative educational quality -- where there is an existing data source, I indicate it. The ABA data is all on the Web here (can search by school); one question is how exactly one could use The Princeton Review’s Best 170 Law Schools data.

I. Education
A. Teaching Quality
-- Princeton Review ratings: Profs Interesting/Accessible
-- mean or median “contact hours” per professor
-- learned-ness -- Leiter rankings on scholarly impact

B. Classroom Experience
-- participation in LSSSE, and use of the data to increase student engagement (LSSSE)
-- average class size in 1L yr (ABA)
-- small section in 1L year (ABA)
-- number of upper-level courses with enrollment under 50, excluding seminars (ABA)
-- study hrs/day (Princeton Review)
-- # of feedback opportunities per course in the first year

C. Curriculum
Required:
-- what are courses required (school websites)
-- is there a statutory requirement? (school websites)
-- strength of 1L research and writing program (U.S. News, Princeton Review data), including # of credits and broader “lawyering” skills (school websites)

Upper-level Curriculum:
-- simulation-class slots per 100 students (ABA)
-- clinic slots per 100 students (ABA)
-- drafting or advanced-writing (but not seminar) course slots per 100 students (school websites)
-- transactional-course slots per 100 students (school websites)
-- course slots per 100 students in classes that use small-group work
-- opportunities to specialize -- is there a business law or criminal justice track, for example? (AALS Curriculum Committee Survey on Innovations)
-- skills curriculum – factual investigation, interviewing, counseling, negotiation (school websites)
-- indicators of chronic problems with insufficient course offerings (look to Princeton Review, law student blogs, etc.)

D. Bar Prep
-- Actual bar passage rate +/- that predicted based on mean LSAT score (either compare within jurisdiction, or use MBE score instead of passage as dependent variable)

II. Extra-and Co-curricular Activities
-- journal slots per 100 students (ABA)
-- moot court/mock trial opportunities per 100 students (ABA)
-- externships/field placements (ABA/school websites)
-- pro bono requirement (school websites)

III. Career Advising and Assistance
-- career services staff per 100 students (school websites)
-- student satisfaction with career services (Princeton Review)
-- loan repayment for public-interest (school websites)
-- starting salary +/- that predicted based on median LSAT score (I’m nervous about this one because discourages schools from encouraging students to go into public interest)
-- number of OCI interviewers

IV. Alumni network, present and future (associative good)
-- alumni giving participation rates
-- concentration in particular geographic region (assumption is that more concentration = stronger alumni network)
-- alumni career-advising network – yes/no (school websites)
-- Alumni Network National Reach (done by MoneyLaw guestblogger and entering Michigan 1L Michael Shaffer)

I would very much welcome thoughts on this, as it is really a first cut, but also keeping in mind that "it takes an alternative" to improve upon a suggestion. Thanks.