What Would Leiter Do?

I've been considering here and here how we ought to fill out the US news rankings survey that asks us to rate the "quality" of the "program" of each school. It seems like there are three basic possibilities: rate the school's scholarship program, educational program, or some combination. And given that the purpose of the rankings is to coordinate between students and employers, education seems like the answer. But maybe I'm wrong, so I thought I'd look to perhaps our leading critic of U.S. News and ask: what would Leiter do?

The answer might seem to be obvious -- Professor Leiter ranks schools by faculty scholarship, therefore we should use his rankings for 100% of our rankings when we fill out the survey -- but it's not clear this is his view.

In his recent "open letter" to Bob Morse, U.S. News's methodology czar, who is to be commended for opening a dialogue on such issues on his blog, Leiter says:

"I suggest you switch to an on-line survey system with academics (your response rate from academics is already quite high, and I imagine that for an on-line survey it will be even higher), in which evaluators are presented with concrete information about each school, rather than simply a school name: e.g., a current faculty roster, numerical credentials of the student body, a list of distinguished alumni (let the school provide a list, limited to 50 names, say), and so on. Ask academics to evaluate the scholarly and professional excellence of the school, not simply the "reputation" they associate with a name."

OK, first, unless what U.S. News currently has on its website is wrong, they don't ask faculty to assess the "reputation" of the schools; they ask faculty to assess the quality of the "program." So that piece of Leiter's criticism appears to be off-base -- and if faculty are treating it as a "reputation" survey, that's our fault, not U.S. News's.

Leiter says academics should evaluate the "scholarly and professional excellence of the school" and use the composition of the faculty, "numerical credentials" of the student body, and an alumni list to make such an evaluation. Okay, so he wants scholarly excellence to be a part of it, and that's what the faculty roster is for -- that's no surprise, given the focus of his own rankings. Then he wants students' "numerical credentials" to be used by faculty as well -- why? Sure, stronger colleagues likely means a stronger education, but LSAT scores and GPA are already part of the U.S. News rankings -- it doesn't seem to make sense to double-count it this way. If you think LSAT scores should count more, then just say that and do it directly.

What about the list of 50 distinguished alumni? That's an interesting thought, but what is it trying to get at? It could be trying to give prospective employers and students a look at past outcomes to predict future outcomes: if there is a current University of Georgia alum, for example, serving as the General Counsel of the FBI (Valerie Caproni), then future UGA students might reach such levels. Alternatively, it could be a signal of the strength of the alumni network, something that prospective students might want to consider in assessing the value added that one school might provide versus another.

In his contribution to the Indiana symposium on rankings a few years ago, Leiter said this about how schools ought to be ranked: "rankings of academic institutions should emphasize and reward academic values: scholarly excellence, pedagogical skill, and student ability and achievement." And shortly before saying that, he indicates this about "pedagogical skill":

"I am still attracted to the old-fashioned view that those are who smarter and more learned can provide higher-quality instruction.... [T]his is not to say that the best scholars are the best teachers: that plainly is not true, since there are a variety of pedagogical skills that are unrelated to intellectual acumen. But it is to say that no set of pedagogical skills can compensate for lack of intellectual depth in one's subject-matter, and I am reasonably confident, based on experience on both sides of the podium, that this is true.... [I]f the old-fashioned view is correct, then it will affect educational outcomes. With all that in mind, I think an assessment of academic institutions ought to weigh heavily the intellectual and scholarly caliber of the faculty, not to the exclusion of other factors, but as a way of putting education at the center of any evaluation of institutions in the business of educating."

This passage, I think, helps clarify the likely answer to the question: what would Leiter do? I think he would have us fill out the survey based on the quality of the "knowledge production" program -- that is, faculty scholarship -- because it both measures scholarly excellence, and serves as a proxy for the quality of the "education" program, thereby (one assumes) providing students and prospective employers with the information they're looking for.

I disagree. The question, I think, is how to assess for students and prospective employers the quality of the educational program of a particular school versus another, or as Bill Henderson put it in a slightly different context in this fantastic post, to assess on a relative basis how much the school "adds value to the personal and professional lives of its current and former students." If we borrow Henderson's formulation, we might include more than just assessments of the education itself, but also things like assessments of a school's career services function, academic advising, and the strength of the alumni network.

And, assuming this "value added" is what we should be after in filling out the survey, either in whole (as I tend to think) or in part (imputed-to-Leiter), then I'm quite skeptical that assessments of faculty scholarship serve as a decent proxy. The principal mistake Leiter makes here, in my view, is focusing too much on, well, the individual talents of law professors. Curriculum matters to the quality of the education. Student culture matters to the quality of the education. And pedagogical skill certainly matters a lot as well.

Nonetheless, even if we are to assess pedagogical skill, faculty scholarship is a poor proxy. Interestingly, Leiter used to include "teaching quality" in his "educational quality ranking" based on the student surveys done by The Princeton Review, which includes ratings by school for "professors interesting" and "professors accessible," but he stopped using this data -- I'm not sure why.

I'll talk more next week about how we might actually assess and compare the "value added" or the quality of the educational "program" across schools, but for now: am I right about what Leiter would do? And is he right about what we should do?