Law Review Rankings

Maybe it’s the hundred-degree heat talking, but I think law review rankings are a little bit useful.
As a reader and researcher, I do make some use of an article’s placement as a screen for how close of an initial read to devote to it. When I look at the c.v.’s of two scholars whose work I’ve never read, I’m probably inclined to look more attentively at the work of the one with the fancy cites.  Yeah, I said it. Put away the pitchforks, dear readers: I don’t think I’m alone. Satisficing is not going away.  And, by the way, perceived prestige is an important motivator for the nonprofit labor force.

It would be nice, then, if there were reliable guides to the signaling value of a given journal placement.
U.S. News gives us a decent if limited signal; since most authors agree that at the pinnacle its rankings are roughly meaningful, we get scarcity.  So we can assume that journals at the top are more selective than others.  Whether they make good decisions when picking the few from the many we don't know.  And in the end, using selectivity as a measure of quality leads us, um, to this.  Is there a better way to rank journals?  

An under-appreciated problem here is that this throws us back into the problem of defining what is good legal scholarship.  Given that journal editors are likely to respond to the incentives of an explicit ranking system, some care has to go into constructing it.  An approximation of a value-neutral approach might be to simply rank publications based on the use others scholars make of them.  (For a thoughtful review of why that method works and what its problems are, see Russell Korobkin, 26 FSU L. Rev. 851, and Ronen Perry.)  Korobkin argues that, basically, citation counts create the least bad set of incentives; usefulness to others seems like a decent result even if it's somewhat distorting of the real scholarly mission (which, of course, is to be completely useless).

Well, the Washington & Lee Law Library, as many readers will know, offers a ranking of law journals based on total citations and “impact factor,” or IF. IF in the larger scholarly world is a widely-used metric of the quality of journal editors’ judgment; it represents the mean number of citations per article per year for the journal. It’s not actually a great measure, since it tells us nothing about the quality of the citing articles, and reputation probably produces IF as much as the other way around.

As weak as IF is in general, W&L’s implementation is particularly problematic.  

If you probe the W&L description of their methods closely, you find that they aren’t really calculating IF. What they’re doing instead is counting how many times each journal is cited at least once in a given article. That method tends to shrink the distance between top journals and others (and, probably, to underweight specialty journals), because it gives journals no credit for being cited more than once per article. A real IF would count citations separately for each published article, add them up, and then divide by the number of articles published per year.

Also, there’s gaming, as some have noted around here recently. Thomson Reuters, which compiles IF rankings for non-law subjects and sells the results to journals for their advertising purposes, reports self-citations for each journal. Users can then make up their minds whether they care.

Finally, to be parochial, W&L only uses Westlaw to generate its citation counts, and Westlaw doesn’t include Tax Notes, a major publication for us tax types. (This is also our gripe with Leiter). So tax articles are (sniff) even more under-appreciated.

None of this is to pick on W&L. It’s wonderful that someone has taken on the task of generating information that’s useful to all of us. But hopefully they are open to reform. Another path forward is Thomson to enter the law market.

Either way, what I'd particularly like to see is some kind of quality-weighted influence measure, along the lines of google pageview, as described here.