Tuesday, August 26, 2008
U.S. News and Part-Time Program "Gaming"
The front page of today's WSJ has an article about a possible change in the U.S. News law school ranking formula. The change, which has already been much discussed, might require law schools to count the LSAT scores and GPA of part-time students in schools' overall LSAT and GPA numbers.
I'm sure others will have more to say about this than me. I just want to draw your attention to one curious sentence in the article: "Mr. Morse of U.S. News says the magazine will run tests of how the change would play out in rankings, and then decide in January." Now, the sentence does not precisely say that the U.S. News formula decision will depend on "how the change would play out in rankings," but it sure seems to imply it. Otherwise, why are they running the tests?
Presumably, U.S. News wants to be sure that such a change in methodology would not make the rankings seem less legitimate. But if there is some gold standard that U.S. News uses to see how well its tests come out, then just show us the gold standard law school rankings! Perhaps there is some other explanation. Perhaps one can test the "stability" of the rankings or some other set of criteria that are neutral with respect to the rankings of particular schools. But, it will not be shocking to suggest that the primary impetus for the testing may have more to do with the business goals of U.S. News rather than the development of an unbiased, careful approach to measuring law school quality.
Posted by Adam Kolber on August 26, 2008 at 08:49 AM | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference U.S. News and Part-Time Program "Gaming":
It's possible to perform tests on a the new method of ranking without having a "gold standard" to test them against. There are at least two reasons.
First, it's possible to have a testing system that isn't itself generative. This is common with regression testing; U.S. News may be checking to make sure that the change doesn't fundamentally break their rankings -- e.g., we could probably all agree that the exact opposite of their 2008 ranking would be wrong.
Second, the tests could be exploratory. People who work with large datasets run new kinds of queries precisely because they don't know what will happen and they're curious. If we make this change, what happens? Hmm, why is it that we see a correlation between X and Y when we do? This calls for further investigation. In the process of seeing what happens when they change the formula in this way, they might discover other patterns that could call for other adjustments.
Posted by: James Grimmelmann | Aug 26, 2008 10:08:34 AM
About time. Schools have been gaming the system like this for too long. They need to be punished for accepting people with sub-165 LSATs at all. A good law school is about intellectual purity.
Posted by: S.c.cutus | Aug 26, 2008 12:18:40 PM
I'm sure they just want to see if this change will lead the rankings to change drastically. If it does, that casts serious doubt on the validity of the rankings to begin with. If this change leads to a number of small changes, it seems like "fine-tuning;" but if it knocks half the second tier into the third, or whatever, consumers should wonder why they are trusting these rankings at all.
Posted by: Heloise | Aug 26, 2008 9:06:39 PM
Wha? Purity? S.c.cutus, please tell me you're joking and not some kind of plainclothes facist with a license to practice.
Posted by: Kevin | Aug 27, 2008 12:26:36 AM
Necessary reading: "The Empirics and Ethics of USNWR Gaming" by Henderson and Lipshaw over at the Legal Profession Blog http://lawprofessors.typepad.com/legal_profession/2008/08/posted-by-jeff.html#comments
Posted by: Patrick S. O'Donnell | Aug 27, 2008 9:34:42 AM