« Unanticipated Side Benefits of Pro Bono Amicus Work | Main | In Memoriam, Steve Gey »

Friday, June 10, 2011

57 is the new 40

In an earlier blog post, I inquired about the perceived strength of law reviews and specialty journals. I received a lot of helpful feedback. I've recently tumbled onto an attempt to crowdsource law journal rankings. The poll rated 170 journals, in pairwise comparisons. Journals are ranked on a 0-100 scale, where the number is the estimated chance that it will win against a randomly chosen idea. Stanford boasts the highest score of 90, while the Kentucky Journal of Equine, Agriculture & Natural Resources Law scored lowest with a 12. All 170 journals ranked are among the top 300 journals according to Washington & Lee's rankings.

I have heard repeatedly that a given specialty journal is roughly equivalent to a main journal at a law review ranked 40 steps lower by USNWR. I thought the crowdsourced rankings might bear that out, so I sorted the journals according to the crowdsourced rankings, and compared the 3-year average USNWR ranking of each specialty journal's home institution with the 3-year average  ranking of the nearest law review. So what's the average difference between specialty journal and its nearest law review neighbor?

57.36, with a median difference between specialty and law review of 53.16. This suggests that, on the whole, the +40 heuristic may overestimate the relative strength of specialty journals.

What I like about the crowdsourced rankings is that they hold out promise of disclosing an overall indication of how other academics rank specialty journals. Unfortunately, votes are entirely volunteer basis, by those who found the survey, so it is unlikely that the votes are representative of the legal academy as a whole.

There are also at least two potential problems with my calculations. First, as of today, there were only 14,271 votes cast, enough to compare each journal to 84 others, or roughly half of the total journals included. Stanford and Chicago were 1 and 2, and more votes might change the relative placement. Thus, the crowdsourced ranking strikes me as a tentative projection, assuming that we can rely on the anonymous individuals doing the voting. (The poll is still open, so we could get more votes, and I would be happy to recalculate if there was reason to do so).

Second, some journals ranked in W&L's top 300 come from instutitions that are unranked by USNWR, either because they are in the new "Tier 2," or because they are not hosted by a U.S. law school, so there are awkward gaps. In addition, for some schools, I don't have a three-year rank, because they've just emerged from behind the alphabetical anonymity of the USNWR rankings, so I used this years ranking. Unfortunately, I have rough proxies all the way down.

Finally, you may wonder why we should care. Like other authors, I want the best home for my articles. I write in intellectual property, and there is some indication that law review editorial boards tend not to accept more than one IP article per cycle. Placement is important, and I still hold out hope that I can find something more than "feel" to guide me here. You may feel much the same, and I hope this at least opens up new questions to ask. You can download the spreadsheet used to make these calculations. Feel free to play around with it and see what conclusions, tentative or otherwise, you can draw. Download Rankings

With that, I need to get back to writing that next article. Thanks to Dan for the platform, and to all the gracious commentors for engaging with my lines of inquiry this time through.

 

Posted by Jake Linford on June 10, 2011 at 03:16 PM | Permalink

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c6a7953ef014e890c6c0e970d

Listed below are links to weblogs that reference 57 is the new 40:

Comments

I tend to think that IP specialty journals are generally better regarded than other specialties.

Posted by: anon | Jun 10, 2011 3:49:59 PM

Anon, I tended to think the same thing, but at least with these numbers, the IP specialty journals do a bit worse as a group: the median USNWR distance between the IP specialty journal and its nearest crowdsourced neighbor is 64.24, and the median distance is 65.

Posted by: Jake Linford | Jun 10, 2011 4:02:40 PM

I'm not sure (I'm not even close to an expert on these things) but my understanding is that for a poll like this, w/ this many elements, you'd need many, many more votes to get valid results (especially if you discount multiple votes by people trying to skew things.) Brian Leiter did several of these polls, and I think this was a moral he tried to impress on others doing them- that they don't work well on this scale unless you can get a really huge number of votes- much more than used here, I think. If that's right (and it might not be- see my disclaimer above again) than no conclusions should be drawn from the results here at all, not even weak ones.

Posted by: Matt | Jun 10, 2011 5:54:55 PM

Jake,

That's interesting. In the end, though, I suppose all that matters is what faculty know. If they see Northwestern's IP journal, for example, they probably aren't likely to know that it's ranked (and regarded) significantly lower than say, Fordham or Cardozo's IP journals. So, in such a case I think the rankings (W&L or otherwise) then to help other faculty members (if they take the time to look). What that means for these rankings isn't clear.

Anon

Posted by: anon | Jun 10, 2011 6:03:59 PM

Perhaps law schools should offer a special 1-credit course for law review editors, to teach them about the different law review ranking systems and their proper interpretation?

Posted by: andy | Jun 10, 2011 7:25:57 PM

Part of the problem here is the attempt to generalize. That's what has always struck me as downright silly about the "plus 40" rule of thumb. What should I take away from the fact that the median difference according to these votes is 57? How does that guide my decision making in any particular placement decision? There is no one size fits all algorithm here. Nor should there be.

In the IP field, for example, the reputation of say Berkeley or Harvard's specialty journals may be near or better than the plus 40 mantra. But for schools like Yale or Virginia, the disparity is likely much larger. Placement is already a bad proxy for scholarly quality. I don't see why we should compound that problem by relying on generalizations about the quality of placements, whatever that means.

Posted by: Aaron Perzanowski | Jun 11, 2011 12:02:58 AM

Damn. My 57th birthday is in five days. I liked the title.

Posted by: Jeff Lipshaw | Jun 11, 2011 7:24:35 AM

I think this is interesting, even if not reliable - and about what I personally think as to assessing specialties versus flagship law reviews (i.e., the 57 differential). Of course, some specialties will tend to best this average (Harvard & Yale come to mind), but I think in general you have to discount a ton to arrive at an appropriate assessment of most specialties. IP, International, civil rights - heck, even law & policy specialties - simply don't have near the amount of submissions that flagship journals receive. And this naturally decreases the prestige with publishing at these journals.

Posted by: vap | Jun 11, 2011 2:59:53 PM

A more accurate (albeit also "general") rule may be something like - a top 3 specialty journal in your area (as ranked by W&L) is approximately worth a general law review placement at the very bottom of the first tier, or the top of the second tier. And #4-#10 in a specialty area are somewhere around the bottom of the second tier. Do people agree with this - generally?

My proposed rule of thumb is also very general, but at least it takes into account the fact that the strength of the specialty journal is not always as close to the USNRW ranking as the general journals tend to be. And it takes into account that the strongest specialty journals may be tied to different school.

For example, in the corporate world, the Delaware Journal of Corporate Law is a top 3 specialty journal, even though it is published by a 4th tier school. Stanford's corporate law journal, on the other hand, is currently ranked WAY below the DJCL. With the +40 or +57 rule, you would choose Stanford's business law journal, which most corporate law profs would consider an absurd choice (unless you were merely aiming to impress the uninformed). With the "top-3 specialty journals roughly equals the very bottom of the first tier" rule, you would choose the DJCL over general law reviews ranked in the 60s or below (but not the 40s or above), which I think is very reasonable.

Posted by: Juli | Jun 12, 2011 4:43:18 PM

Juli, I think talking about specialties as, e.g., a "top 3" specialty only begs the question of how we assess what is 'top'. Is W&L really a better metric than USNWR (i.e., the home institution of the specialty). Generally, I would say no. Most people are not super-familiar with W&L's rankings but most people ARE super-familiar with USNWR's rankings. There is simply no reliable way to talk about a specialty being "top _" - except where maybe there is an informal consensus. Perhaps there is with the Delaware Journal of Corporate Law - I don't write in that area. Certainly, HCRCLLR is the best journal in the area of civil rights. But in most areas, I would venture it isn't clear. Which means we can only go by the "majority" advice we get on close calls or perhaps adopt some sort of general +40, +57, etc. metric for sorting out the answers to these questions. I have really enjoyed this thread, as this dilemma (specialty vs. flagship offer) seems to come up all the time and you get such conflicting advice.

Posted by: 1styearprof | Jun 12, 2011 7:32:40 PM

1styearprof. I am also a relatively new prof, so I may not know best, but at least in the corporate world, I think academics have a pretty good idea which specialty journals are at the top. People may disagree on which journal is at the very top, but most people's top three would look similar. And the specialty journals at some very top schools are not currently as highly regarded as the DGCL or the Journal of Corporate Law - both of which are put out by relatively low ranked schools.

Posted by: Juli | Jun 13, 2011 8:36:55 AM

Post a comment