« Deterring Adolescence: A Friday Farewell Post | Main | Getting law right on "BCS" »

Friday, April 03, 2015

Contemplating Academic Analytics for Law Schools

There is a recent trend in higher education to standardize assessment of faculty’s academic achievement across disciplines.  For example, a company called Academic Analytics markets itself as providing university administrators “with objective data that administrators can use . . . as a method for benchmarking in comparison to other institutions.”  As its website explains, it measures productivity and excellence by quantifying:

  1. the publication of scholarly work as books and journal articles
  2. citations to published journal articles
  3. research funding by federal agencies
  4. honorific awards bestowed upon faculty members

Because it is seeking to assess academics generally, the metrics that Academic Analytics uses are not necessarily well suited to assessing law faculty.  For example, the number of faculty members with a grant and grant dollars per faculty member (two data points that the company uses to quantify research funding) are not particularly good measures for law faculty because many law schools do not encourage their faculty to obtain grants.  Similarly, the number of faculty members who have published a book may not be as good of a metric for law faculties as in other disciplines.

Looking at the academic analytic metrics, I’m contemplating how it is that one might attempt to construct an instrument that would assess law faculty productivity and excellence.

Brian Leiter’s study of citations (which some faculty at St. Thomas have reproduced in recent years) is one possible example.  That study gives some sense of the impact of a law school faculty’s scholarship by counting the number of citations each faculty member receives in other legal scholarship.

As Leiter himself notes, this study has real limitations.  And I imagine that there will be serious limitations on any attempt to quantify academic achievement. That said, if university administrators are seeking to engage in these sorts of analyses, it strikes me that law schools ought to give serious consideration to developing better metrics at assessing their productivity and excellence.

I’ll share some preliminary thoughts about what those metrics might look like in a future post.

Posted by Carissa Byrne Hessick on April 3, 2015 at 10:51 PM in Life of Law Schools | Permalink

Comments

Michael Risch: Quantifying institutional prestige, apart from raw page counts, is an important first step in assessing productivity. As between two scholars with identical page counts, you'd rather have the one with higher impact-factor placements and higher citation counts. Whether a scholar with 500 annual SSRN downloads is pulling her or his weight depends on institutional affiliation. So think of it as the academic equivalent of ballpark effects in sabermetrics. Perhaps they aren't actual performance metrics. But impact factors and download counts get you off to a good start on Carissa's quest.

Another point I tried to communicate in my paper was the notion that academic analytics should be based on models whose underlying mathematical assumptions and results lend themselves to clear, physically coherent interpretations. The ordinary exponential distribution as a special case of the stretched exponential (Kohlrausch) function certainly qualifies. We should strive for a comparable degree of rigor in all of our analytical tools.

Posted by: Jim Chen | Apr 10, 2015 4:10:55 PM

Jim Chen - your model is quite elegant, but does it really measure productivity and not prestige, for example?

Posted by: Michael Risch | Apr 9, 2015 9:36:42 AM

I propose two methods of measuring academic achievement -- law review citations and SSRN downloads -- in http://ssrn.com/abstract=905316. The ordinary exponential distribution provides a nearly perfect fit for both of these measures.

Posted by: Jim Chen | Apr 8, 2015 5:24:18 PM

I'm sure it was contemplated, but your comment quoted a single section "grant dollars and grant dollars per faculty member" and tried to make it seem like one of us had to be wrong. I suppose you can use endowed funds as analytics (and maybe even should), but it's not what the post meant by grant dollars (nor is it very transparent). The private money I'm talking about isn't often in the form of open grants either, and thus not really public (much to the chagrin of many in the academy).

Posted by: Michael Risch | Apr 7, 2015 4:01:07 PM

Michael,

I thought the post was contemplating any type of quantifiable data that could be used to assess scholarly quality, not merely the four specific things Academic Analytics used.

Posted by: Derek Tokaz | Apr 6, 2015 3:46:35 PM

Derek - None of my three examples are the traditional types of grants that this post contemplates.

Posted by: Michael Risch | Apr 6, 2015 12:13:11 PM

"For example, the number of faculty members with a grant and grant dollars per faculty member (two data points that the company uses to quantify research funding) are not particularly good measures for law faculty because many law schools do not encourage their faculty to obtain grants."

But wait, just a month ago Michael Risch wrote on TFL that law faculty were getting significant grant awards: "I think that people underestimate the amount of scholarship that is funded by outside sources. First, many summer grants are endowed. Second, many endowed professorships come with research money. Third, outside money is growing, though much of it is private."

Posted by: Derek Tokaz | Apr 6, 2015 7:44:04 AM

What do the university administrators do with these metrics? Without knowing what administrators do with the metrics, it's hard to assess the pros and cons of different ones.

Posted by: Orin Kerr | Apr 6, 2015 1:43:32 AM

Then let the battle begin. I'm sure others will be interested also.

Posted by: Sydney | Apr 4, 2015 7:38:42 PM

Thank you for starting a discussion on this. Quantification of legal scholarship as it is currently done and likely to be done in the future is an important issue. I look forward to your future posts.

Fair warning, you should expect some push back from me on the idea of using a Leiter-type metric, at least as that type of metric is currently used. See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2506633

Posted by: Patrick Woods | Apr 4, 2015 9:21:04 AM

Post a comment