« Should I Say Write a Response to a Law Review Article (or Allow the Law Review to Solicit Responses to Mine)? (Jr. Law Prawfs FAQ) | Main | 2016 Texas Legal Scholars Workshop »

Monday, April 04, 2016

2016 Meta-Ranking of Flagship US Law Reviews

This post was written by Bryce Clayton Newell of Tilburg University.

I want to thank Howard for posting this to the blog for me. I have been an avid reader of the blog for a number of years now, and it is nice to have the chance to try something out with you all and get some feedback on an idea for a new way to rank law reviews.

(I realize I may have just scared some of you off :) If not, I appreciate your continued attention.)

I am a long-time Angsting Thread lurker and sometimes commenter/spreadsheet contributor (when I have something to add). This submission cycle, during my “free” time between repeatedly hitting “refresh” on the Spring 2016 Angsting Thread (to read updated comments) while waiting for two articles to get accepted, I put together a meta-ranking of general/flagship US law journals. Law journal rankings show up periodically on PrawfsBlawg (as well as on other popular law blogs), but the semi-annual Angsting Thread continuously includes comments and questions about how to compare offers, whether to use US News rankings (either the Overall Ranking or the Peer Reputation ranking), and how (or whether) to consider alternative, impact-based, metrics like the W&L Combined Ranking or Google Scholar Metrics.

As a junior academic and aspiring prof., I’ve also asked these questions of numerous mentors and former law professors of mine. The advice I’ve gotten generally mirrors the advice I’ve seen in the comments: generally use the US News Overall Ranking (or maybe the peer reputation ranking, although this is less frequently mentioned), and if you can find it, consider the average ranking of a school over the last X years rather than only the most recent annual ranking; the alternative rankings can be useful to e.g., compare a specialized journal with a flagship one or as a way to distinguish between offers from two closely ranked (by US News) general journals, but they should not replace the general consensus that (one of ) the US News rankings is the best gauge of journal prestige.

However, as an interdisciplinary scholar who publishes work in both peer-reviewed social science journals and law reviews, it strikes me as odd that we would discount measures of journal impact completely when choosing where to submit and which offers to prioritize. To be sure, the W&L ranking has some flaws (some described earlier on PrawfsBlawg here) and other citation-based impact factor rankings commonly used in other disciplines (like JCR/ISI) also have their limitations (including poor coverage of law journals). Google Scholar Metrics represents an interesting alternative way to measure impact (Google’s metrics description is here), but also doesn’t have full coverage of law journals and comes with a different set of concerns. Regardless, though, it appears important that some measures of citation or impact are taken into account, as direct correlations between US News rankings of law schools and law journal importance seem a bit weak as the primary (or only) measure to evaluate.

So, to get to the point, I decided to create a meta-ranking of the possible contenders for gauging the relative importance of journals and offers: US News Overall Ranking (averaged from 2010-2017), US News Peer Reputation Ranking (also averaged from 2010-2017), W&L Combined Ranking (at default weighting; 2007-2014), and Google Scholar Metrics law journal rankings (averaging the h-index and h-median of each journal, as proposed here by Robert Anderson). I've ranked each journal within each ranking system, averaged these four ranks using a 25% weighting for each, and computed and ranked the final scores. I think this approach benefits from incorporating a couple different forms of impact evaluation (W&L + Google) while not disregarding the general sentiment that law school “prestige” (USN combined rank + peer reputation rank, each averaged over an 8-year period) is an important factor in law review placement decisions.

I would love to get feedback about whether you think there is any usefulness to doing this in this way, whether you would suggest alternative weightings, different combinations of rankings, or if I have overlooked something (entirely possible, as I was paying more attention to your comments on the Angsting Thread than anything else when I put this together), etc. If it seems that folks are interested and that this might be useful, I can also post full ranking (I’ve ranked 194 journals). I am also working on an attempt to evaluate equivalencies between specialty journals and general ones, and I’m happy to take suggestions or share my initial thoughts on doing that if you’d like to get in touch.

The big movers here (in this ranking versus the average US News Overall Rank from 2010-2017) seem to be (but there are quite a few others who moved around):

  • New York Law School moved up a whopping 38 places (to #99);
  • Vermont moved up 31 places (to #91);
  • UC Irvine dropped 30 places (to #59);
  • Akron moved up 28 places (to #99);
  • Albany moved up 27 places (to #96).

Journals like Fordham (#26, up 10 places), Hastings (#36, up 12 places), Cardozo (#42, up 18 places), American (#46, up 11 places), and Lewis and Clark (#53, up 23 places) that have been frequently referred to in Angsting Thread comments as “hitting above their weight” all also improved at least 10 places (as did Missouri, Connecticut, Denver, Brooklyn, Chicago-Kent, Seattle, Oregon, Buffalo, Santa Clara, Indy, DePaul, South Carolina, St. Louis, Hofstra, Marquette, and Howard). Other journals dropping 10 or more places include: Arkansas-Fay., Kentucky, Georgia State, Temple, SMU, Arizona State, Georgia, and Alabama.

Other sizable moves in the top 20:

  • Chicago (#12) drops 7 places (Google’s ranking moderated the even more drastic difference between Chicago’s US News rank and W&L rank);
  • Iowa moved up 5 places (to #20);
  • Northwestern dropped 4 places (to #16);
  • Michigan (#6), Georgetown (#10), Texas (#11), and Notre Dame (#19) all moved up 4 places.

Finally, here are the top 100 ranking journals in the 2016 Meta-Ranking:

MetaRank

Journal

Change from USN Rank

MetaScore

Avg. USN Peer Rank

Avg. USN Overall Rank

W&L Rank

Google Rank

1

Harvard Law Review

1

1.5

1

2

2

1

2

The Yale Law Journal

-1

1.75

1

1

3

2

3

Stanford Law Review

0

2.75

3

3

1

4

4

Columbia Law Review

0

3.75

4

4

4

3

5

University of Pennsylvania Law Review

2

6.5

9

7

5

5

6

Michigan Law Review

4

8

8

10

8

6

7

California Law Review

1

9

7

8

12

9

8

New York University Law Review

-2

9.25

6

6

14

11

8

Virginia Law Review

1

9.25

9

9

9

10

10

The Georgetown Law Journal

4

9.75

13

14

6

6

11

Texas Law Review

4

12

15

15

10

8

12

University of Chicago L. Rev.

-7

12.75

5

5

25

16

12

Duke Law Journal

-1

12.75

11

11

16

13

14

Cornell Law Review

-1

13.25

12

13

15

13

15

UCLA Law Review

1

13.5

16

16

7

15

16

Northwestern University Law Review

-4

15.25

14

12

13

22

17

Minnesota Law Review

3

15.75

20

20

11

12

18

Vanderbilt Law Review

-1

17.5

17

17

20

16

19

Notre Dame Law Review

4

21.75

27

23

19

18

20

Iowa Law Review

5

22.5

27

25

18

20

21

Boston University Law Review

3

24.25

25

24

22

26

22

William and Mary Law Review

8

25.5

32

30

21

19

23

The George Washington L. Rev.

-2

26

23

21

29

31

23

North Carolina Law Review

11

26

21

34

28

21

25

Southern California Law Review

-7

26.5

19

18

32

37

26

Boston College Law Review

5

27.25

29

31

23

26

26

Fordham Law Review

10

27.25

35

36

16

22

26

Indiana Law Journal

0

27.25

30

26

27

26

26

Washington University Law Review

-7

27.25

18

19

37

35

30

Emory Law Journal

-8

27.5

22

22

36

30

31

Wisconsin Law Review

4

30.25

24

35

40

22

32

University of Illinois Law Rev.

6

31.25

34

38

24

29

33

U.C. Davis Law Review

-1

33

26

32

31

43

34

Florida Law Review

16

36

38

50

34

22

34

Washington Law Review

-6

36

37

28

30

49

36

Hastings Law Journal

12

37.25

36

48

33

32

37

Ohio State Law Journal

3

39

31

40

42

43

37

Washington and Lee Law Review

0

39

33

37

39

47

39

Arizona Law Review

4

39.25

41

43

38

35

40

Alabama Law Review

-13

40.75

42

27

45

49

41

Wake Forest Law Review

0

41.5

44

41

43

38

42

Cardozo Law Review

18

43

53

60

26

33

43

Georgia Law Review

-10

43.75

40

33

47

55

44

Connecticut Law Review

12

45.75

52

56

35

40

45

Colorado Law Review

0

46.25

43

45

50

47

46

American University Law Review

11

47

48

57

43

40

47

George Mason Law Review

-3

48.5

55

44

46

49

48

Brigham Young University Law Review

-6

49.5

50

42

54

52

49

Maryland Law Review

-2

50

47

47

61

45

50

Tulane Law Review

1

52.25

45

51

49

64

51

Utah Law Review

-5

56.5

51

46

57

72

52

Florida State University Law Review

0

56.75

49

52

58

68

53

Houston Law Review

1

57.25

66

54

51

58

53

Lewis & Clark Law Review

23

57.25

79

76

41

33

55

Pepperdine Law Review

-2

58.5

70

53

59

52

56

Arizona State L. Journal

-18

62.5

46

38

73

93

56

Loyola of Los Angeles Law Review

9

62.5

69

65

70

46

58

Missouri Law Review

22

65

65

80

60

55

59*

UC Irvine Law Review

-30

65.5

38

29

111

84

60

University of Cincinnati Law Review

7

66

74

67

55

68

60

University of Miami Law Review

6

66

54

66

67

77

62

Denver University Law Review

11

68

63

73

72

64

63

Brooklyn Law Review

16

69

67

79

53

77

64

Chicago-Kent Law Review

10

69.5

68

74

78

58

65

Seton Hall Law Review

2

70.25

84

67

78

52

66

SMU Law Review

-17

70.5

64

49

74

95

67

Tennessee Law Review

-4

70.75

62

63

65

93

68

The University of Kansas Law Review

7

71.25

61

75

81

68

68

Seattle University Law Review

24

71.25

89

92

66

38

70

Case Western Reserve Law Review

-9

71.5

59

61

85

81

70***

Penn State Law Review

0

71.5

91

70

67

58

72

Oregon Law Review

15

72.25

56

87

74

72

73

University of Richmond Law Review

-9

72.5

82

64

67

77

73

San Diego Law Review

-4

72.5

57

69

78

86

75

Buffalo Law Review

16

73.75

94

91

52

58

76

Temple Law Review

-18

74.25

60

58

84

95

77

Loyola University Chicago Law Journal

1

77.5

77

78

74

81

78

Santa Clara Law Review

22

78

74

100

74

64

79

Georgia State University Law Review

-21

80

76

58

114

72

80

Indiana Law Review

10

81

73

90

89

72

81

DePaul Law Review

23

83

99

104

48

81

82

South Carolina Law Review

19

83.5

94

101

71

68

83***

Rutgers University Law Review

3

83.75

72

86

91

86

84

Nevada Law Journal

-7

84.5

93

77

82

86

85

Kentucky Law Journal

-23

86

71

62

93

118

86

Louisiana Law Review

-3

86.5

101

83

98

64

87**

University of Pittsburgh Law Review

-6

87.75

58

81

106

 

88

Villanova Law Review

-3

88.25

86

85

96

86

89

Saint Louis University Law Journal

10

89.25

99

99

64

95

90

Nebraska Law Review

-8

90.25

78

82

88

113

91

Vermont Law Review

31

91

108

122

94

40

92

Hofstra Law Review

13

91.75

96

105

62

104

93

Marquette Law Review

10

92

87

103

83

95

94

Michigan State Law Review

2

92.75

102

96

55

118

95

Howard Law Journal

23

93

98

118

101

55

96

Albany Law Review

27

98

124

123

87

58

97

Catholic University Law Review

-2

98.75

90

95

97

113

98

Arkansas Law Review

-14

99.25

97

84

139

77

99

Akron Law Review

28

101.25

143

127

63

72

99

New York Law School Law Review

38

101.25

118

137

92

58

I have also calculated the Spearman's Rank Correlation Coefficient for the MetaRank versus each of the four underlying rankings (for schools ranked in the top 100 in the table above) and have charted the correlation (for all journals ranked 1-150 in the MetaRank).

 

USN

USNpr

Google

WLU

MetaRank

0.913

0.930

0.867

0.902

Notes about method: Google Scholar Metrics are currently based on the most recent Google index (updated to June 2015). Some journals that began after 2007 are effected by W&L’s ranking formula described here. Because the US News Ranking has changed how it reports 3rd and 4th Tier schools over the relevant period, I have done the following: for the 2010 and 2011 editions of the ranking, I gave all “Tier 3” schools a value of 115, and all “Tier 4” schools a 150; for 2012-2017, I assigned all unranked schools (those not ranked 1-149) to 150.

* As some journals were not included in the US News Rankings for all 8 years (e.g. UC Irvine was only in the 2016 and 2017 rankings), I have averaged the ranks over just the years they appear, rather than by 8.

** University of Pittsburgh Law Review is the only top 100 journal not included in Google Scholar, as such, I have used W&L’s rank twice to keep the impact portion of the ranking at 50%.

*** Rutgers recently consolidated two law journals into one and Penn State now also has two ranked law schools but only one flagship law review. For each of these schools, I have used the rank in each category (W&L/US News, etc.) that is the highest. Similarly (although not appearing in the top 100 reported here), Widener split and was ranked separately in 2017 by US News (each school taking a separate flagship journal). As such, I have used the separate 2017 ranks from US News but shared the single Widener school ranking reported in US News in its 2010-2016 editions of the rankings.

Last note: I cannot locate the 2015 US News Peer-Reputation rank for Loyola-New Orleans (it is missing from Paul Caron’s annual posting at TaxProf, and I don’t have access otherwise). If someone has that information (or even the underlying scores themselves) and could pass it along, I would appreciate it.

 

Posted by Howard Wasserman on April 4, 2016 at 09:31 AM in Howard Wasserman, Teaching Law | Permalink

Comments

I've posted the graphical representation of the correlation between my "meta-ranking" (top 149 journals) and these four other ranking systems (as mentioned in the above post) at http://bcnewell.com/my-2016-meta-ranking-of-flagship-us-law-reviews-prawfsblawg/.

Posted by: Bryce Newell | Apr 4, 2016 10:11:00 AM

Following US News is absolute NONSENSE. If anything, follow the academic ranking, but the other measures of US News have absolutely nothing to do with the value of scholarship the journal publishes.

Posted by: tony smith | Apr 4, 2016 7:06:19 PM

No, it's not nonsense, and here's why:

1) Most people are never going to read your scholarship; instead, they'll merely look to see where it placed to determine if you're doing "good" work.
2) Those people are then going to ask themselves is the school at which it placed a "good" school.
3) The answer to that question is overwhelmingly going to be based on where that school is currently ranked in US News.
4) NOBODY checks the W/L rankings except authors trying to decide between competing publication offers (i.e., nobody is going to look at your pubs and then turn to W/L and see how "good" they are).

In other words, US News is very important to take into account simply because others in the academy will judge your publication record largely on that basis. To advice otherwise is what is actually nonsense.

Posted by: Anonprof | Apr 5, 2016 9:16:07 AM

I included US News metrics because almost everyone I've asked has said it's the primary metric to use (although some disagree about whether to use the overall or reputation-based ranking). The purpose of this meta ranking is to keep that fact in mind while also to trying to account some for journal impact. The strong correlation between these new rankings and the US News scores indicates some continuity, while allowing particularly strong (or weak) impact performance to have some place in how we evaluate journals (e.g. when trying to decide where to submit, which offers to prioritize, or which journals to include in an expedite request, etc.).

Posted by: Bryce Newell | Apr 5, 2016 11:13:08 AM

Annonprof: Still NONSENSE. The people who care about my scholarship will read it. The people who just look at US News are not in the position of evaluating me. I am highly paid and get nice raises each year, largely because my scholarship gets cited (and I am a good institutional citizen, etc.). I have never had a law dean or appointments committee tell me I need to place higher. Instead, they have focused largely on my over 1000 citations.

Posted by: tony smith | Apr 5, 2016 11:36:11 AM

"Almost everyone" you asked said it "is the primary metric to use." That's not everyone, however. Nor does it even indicate a popular opinion among law professors. This is an embarrassment to the legal academy. I understand that you included the data because people informed you that it was important. But if you ask the same people whether they like US News or whether it has any substantive value, they would overwhelmingly say "no." Pull off the mask.

Posted by: tony smith | Apr 5, 2016 11:39:44 AM

Tony, I too am a successful scholar and I network regularly with other successful scholars and almost all of us make publication decisions largely on the basis of a school's average USNews ranking (I generally go by the school's five year average). Glad whatever system you've used has worked for you, but you are in the minority. Oh and congrats on your "over 1000 citations" -- LOL.

Posted by: Anonprof | Apr 5, 2016 11:58:57 AM

Tony baloney. We all rely on USN since as pointed out others do consider it important. Tell us Tony phoney, when you were younger did you borrow a Honda civic or a BMW Z4 for that hot date
? Oh of course, since she doesn't rank you by your car so you drove around on your dodge dart and had 1000s of dates...

Posted by: whiteguy | Apr 5, 2016 12:42:57 PM

For an awful moment I thought you might have overlooked the Spearman's Rank Correlation Coefficient, but there it is at the end. What a relief.

Posted by: Lois Turner | Apr 5, 2016 2:03:58 PM

Well, I'm a little disappointed that things have gotten a bit uncivil here. But I do want respond to a couple of comments briefly. @Tony: I am very aware that I didn’t conduct a census of US law profs before drawing conclusions about what the general consensus re using US News rankings was, but I did try to be transparent about my choice. @Lois, I am sorry you found the inclusion of a statistic worth joking about – I only included it to provide a little more information about the rankings (and it relates to a graph that wasn’t reproduced in the blog post). In the end, of course, this ranking needn’t be taken too seriously. I offered up an example of a different way to rank law journals in the hopes of starting a civil discussion about whether there might be any good reason to think about developing a different way to measure law journal prestige (for those who care about such things) besides the oft-cited US News rankings or even just W&L or Google, etc. I would still appreciate any more comments along those lines, even if the consensus is that “what we have works fine, so why try to fix it?” Cheers!

Posted by: Bryce Newell | Apr 5, 2016 2:44:09 PM

Bryce, am I right in guessing that you'd also see something in the .9 range running google on usn/peer? (Actually, would you mind sharing an entire correlation matrix? Or just a CSV of the data?

Posted by: Paul Gowder | Apr 5, 2016 3:31:26 PM

Buffolo's fish wrapper is 75? Recalculate!

Posted by: Bflo | Aug 12, 2016 5:39:00 PM

Post a comment