« Coronvirus Humor - Part 3 | Main | Justice Thomas speaks »

Sunday, May 03, 2020

LSAT-Flex, test adjustments, and predictive validity: At least some intermediate scrutiny is warranted, no?

The new LSAT-Flex being unfurled by LSAC will have one fewer section, Logical Reasoning being reduced from two sections to one.  Perhaps this has something to do with the administration of the test as it is being delivered in this new format because of the COVID-19 crisis.  Not expressing a view on this change, which we can assume was arrived at after careful deliberation.  But is it not odd that the GRE's use continues to be strictly scrutinized by the ABA under Rule 503 for predictive validity while this change is presumably immune from such scrutiny?

I will leave it to others with more sophisticated empirical chops to weigh in on whether this change could affect predictive validity in any meaningful way.  Again, credit what the LSAC says in its FAQ section here:  These questions to be used have gone through the crucible of careful review.  And they say, explicitly, that the test will be neither easier or harder. But this doesn't really answer the question of whether changing the number of questions and the number of sections from four to three can have an impact.  The question I raise is principally a process one:  The GRE has shown it itself to be have strong predictive validity in myriad different settings (though, to be sure, not all or even most law schools have done their own individual predictive validity studies).  Yet schools which would use it, and perhaps with greater urgency because of the current crisis, have significant hoops to jump through.  Meanwhile, the LSAC makes its tweaks, on its own initiative, without (as best I understand it; correct me if I am wrong) undergoing the same ABA review that would apply to ETS and its GRE.  And if this change is viewed as de minimus, what kind of change in the LSAT would trigger a validity review?

The LSAT-Flex initiative, provided in a remote form, is a positive step forward.  What would also be especially positive is if the ABA treats both LSAC and ETS fairly and transparently.  I hope it does; the ABA could easily confirm that this is so.

[As I have written on this blog before, I disclose that I have been consulting with ETS on the use of the GRE. Readers will impose whatever discount rate they think warranted because of this fact]

 

Posted by Dan Rodriguez on May 3, 2020 at 06:34 PM in Daniel Rodriguez | Permalink

Comments

No, Paul B, the gold-standard question is not the correlation between the GRE and the LSAT. Instead, we want to know if the GRE predicts law school performance itself. And that study has been done. The answer is yes. https://onlinelibrary.wiley.com/doi/full/10.1002/ets2.12213

Posted by: ctr | Jun 2, 2020 5:17:51 PM

2 cents: LSAC has enough of a history at schools to predict first-year law-school GPA with the LSAT. Although the GRE is highly correlated with the LSAT, ETS's published work indicates a considerable range of LSAT scores into which GRE scores would translate. That means that predictions of law school performance using ETS's GRE-to-LSAT converter are likely to be considerably more variant than those based directly on the LSAT. That doesn't mean they can't be used, only that they should be used with caution.

Of course, with enough GRE-taking students, any individual school could generate that information for itself.

An additional peril in using GREs is that LSATs were used to select students, leading to greater score compression at any individual law school than would likely exist with the GREs. This might also deflate LSAT performance in a way that the GRE has not yet suffered, but would if it were to be used generally to select law students.

Posted by: Greg Sergienko | May 6, 2020 3:21:45 PM

Exactly right. Why indeed?

Posted by: Daniel Rodriguez | May 5, 2020 10:41:55 AM

Dan, a very simple test of the interchangeability of the GRE and LSAT would be to have a statistically valid number of students take both tests and look at the correlation coefficient of the results. My hypothesis would be that the correlation would in fact be extremely high. If true, the logical result would be to end the LSAT as a separate test. If the physics and history departments can use the same standardized test for their PhD programs, why can't law schools rely on that test as well?

Posted by: PaulB | May 3, 2020 11:46:39 PM

Post a comment