Wine, Soda Pop, and Law Schools - More on "Law Review Lift (Drag)"

Some time this month I will get to a relatively more serious topic, like textual opportunism, but for right now I'm still fiddling around with Al Brophy's ranking system.  

So  that I don't bury the lead, let me say up front that I have played some simple-minded statistical games with Al's data.  What I come up with is that, among academics, "brand," as with soda pop, means a lot, and it is relatively sticky and independent of what is going on with the students.

I also think it's pretty obvious that there is a relationship between the "brand" and student data (i.e. high correlations between any ranking system and LSAT scores, for example). What got me interested, however, as I noted a few days ago, was the differential when Al included or didn't include a different and interesting stat: how often the school's main law review (not its faculty) got cited. My intuition is that what other profs think about placing articles in a school's review (based on my own experience) is a lot like the peer reputation score, except that it does measure a revealed preference (i.e., when you rank "peer reputation" as a participant in USNWR, it doesn't cash out to anything; placing an article does!)

The problem with all of these systems, in which we are "ranking" something with many complex factors (like wine) is that the judgment is qualitative, even if it looks quantitative. Often it's qualitative simply because it's qualitative (e.g., "peer reputation"), but even when it's fully quantitative it's qualitative because of the judgments one makes in weighting the quantitative factors.  I was once a partner in a big law firm. Our partnership agreement called for compensation to be determined by a committee, which in turn used a list of factors like "billable hours," "service to the firm," "client responsibility," etc. Every two years the committee turned out a ranking that set your compensation relative to all the other partners. Similarly, if you aren't a hermit during early March of each year, you hear about a double ultra secret committee in Indianapolis deciding which of the "bubble teams" gets into the NCAA basketball tournament. Same thing.  Recent results? Body of work? Bad losses? Good wins?

In any event, I played with Al's data and made some scatter plots and regressions in Excel, all of which follow the break.

I should note that I ran my little exercise by one of the toughest critics of empirical work I know, not for an endorsement, but to see if it was okay to "bin" the data into that 10+, 20+, and 30+ differentials between Al's 2 variable and 3 variable results. My interlocutor (who will remain nameless to protect the innocent) said that binning was okay if there was some theory behind it, but his or her very, very fulsome and thoughtful reply to my question reaffirmed my belief that data without judgment is blind (and judgment without data is empty, to be fair, in each case paraphrasing Kant). The big issue is whether just a few outliers are responsible for the outcomes (which you can see by eyeballing the scatter plots). That may be true here. So with that disclaimer, and recognizing this is a blog post, for God's sake, and not a peer reviewed research paper, here's what I came up with.

If you plot law review "lift (drag)" of 10+, you come up with a positive correlation to law review volume number (.339).  See chart above the break.

If you do the same for "lift (drag)" of 20+ and 30+, you come up with even higher correlations, .42 and .55, respectively.  (See above left and right.)

What do I conclude? Probably nothing more than common sense would tell me: "brand" makes a difference; it takes a long time to develop one; and once you have it established, it sticks around enough to bias other data.