PrawfsBlawg

The Significant Decline in Null Hypothesis Significance Testing?

(Cross-posted at Co-Op.)

Prompted by Dan Kahan, I've been thinking a great deal about whether null hypothesis significance testing (NHST, marked by p values) is a misleading approach to many empirical problems.  The basic argument against p-values (and in favor of robust descriptive statistics, including effect sizes and/or   Bayesian data analysis) is fairly intuitive, and can be found here and here and here and here.  In a working paper on situation sense, judging, and motivated cognition, Dan, I, and other co-authors explain a competing Bayesian approach:

In Bayesian hypothesis testing . . .  the probability of obtaining the the effect observed in the experiment is calculated for two or more competing hypotheses. The relative magnitude of those probabilities is the equivalent of a Bayesian “likelihood ratio.” For example, one might say that it would be 5—or 500 or 0.2 or 0.002, etc.—times as likely that one would observe the results generated by the experiment if one hypothesis is true than if a rival one actually one is.

Under Bayes’ Theorem, the likelihood ratio is not the “probability” of a hypothesis being true but rather he factor by which one should update one’s prior assessment of the probability of the truth of a hypothesis or proposition. In an experimental stetting, it can be treated as an index of the weight with which the evidence supports one hypotheses in relation to the another.

Under Bayes’ Theorem, the strength of new evidence (the likelihood ratio) is, of course, analytically independent of one’s prior assessment of the probability of the hypothesis in question. Because neither the validity nor the weight of our study results depends on holding any particular prior about the [question of interest] we report only the indicated likelihood ratios and leave it to readers to adjust their own beliefs accordingly.

To be frank, I've been resisting Dan's hectoring entreaties arguments to abandon NHST.  One obvious reason is fear: I understand the virtues and vices of significance testing well.  It has provided me a convenient heuristic to know when I've "finished" the experimental part of my research, and am ready to write the over-promising introduction and under-delivering normative sections of the paper.  Moreover, p-values are widely used by courts (as Jason Bent is exploring).  Or to put it differently, I'm well aware that the least positive thing one can say about a legal argument is that it is novel.  Who wants to jump first into deep(er) waters?  

At this year's CELS, I didn't see a single paper without p-values. So even if NHST is in decline, the barbarians are far from the capital.  But, given what's happening in cognate disciplines, it might be time for law professors to get comfortable with a new way of evaluating empirical work.

Breaching a Law Review Contract?

I'm one of Temple Law Review's advisors.  Given my views on student-run journals, this is a  bit ironic. But the experience so far has taught me  how much student editors care about getting it right, and how invested they can be in their journal's success.  Or to put it differently, though in theory a goofy academic could generate a hundred more useful ways to spend students hours than law review, it's not at all obvious that any of those alternatives would generate equivalent passion and commitment from students. 

The advising process has also recently given me a new perspective on an old problem. Very often, in the insane & dispiriting process that we call the submissions cycle, you hear of professors getting a great (read: higher prestige journal) offer just after they've accepted at a less great (read: lower prestige journal) placement.  Counterfactual reasoning sets in -- "if only I'd pushed back against those meddling kids!" - and everyone who hears the story feels a punch in the gut, excepting those who refuse to play the game. Inevitably the question is entertained: what, exactly, is stopping the professor from backing out of the deal with mediocre law review A to accept the offer of awesome law review B? After all, the process is crooked, everyone is just reading expedites, and reliance arguments are weak.  Law reviews aren't going to sue for breach of contract -- even if one exists, which might be doubtful.  If they did , this is the clearest case of efficient breach possible. 

But then norms of professional courtesy typically set in. And, though I've been teaching for over a decade, and heard literally dozens of stories like this, I'd never actually heard of anyone backing out of a law review acceptance until this cycle.  Temple just had someone back out.  Because that person is junior - and no doubt listening to a more senior mentor's advice - I'm not going to provide more details.  I will say that the acceptance/rejection cycle was very dispiriting to the students involved, and it rightly might make them quite cynical. And it did make me wonder whether  publication decommitments are  more widespread than I'd thought, and whether journals could (or should) do anything to stop them. 

Have I just been naive? Is law review conscious decoupling common? Is that behavior, in fact, righteous?

Random Updates on Punitive Damages, Eighth Amendment and Family/Criminal Law Work

Hope everyone had a good weekend following the Wall Street crisis and the Noles game. I spent most of my time the last few days revising my draft of Retributive Damages, which I've just uploaded to SSRN over here. As I've mentioned before, this paper is the first in a series of three papers proposing a new way to think about punitive damages. The first one is coming out this January in Cornell Law Review, and the draft on SSRN reflects some of the edits I've received from the first round of editing there.  The sequel to that paper, How Should Punitive Damages Work?, is coming out later in the spring in the Penn L. Review. I'll be presenting the sequel paper this Friday afternoon at the Canadian Law and Economics Association in Toronto (and I hope local Prawfs readers will get in touch with me there; I'll be hanging out with Dave Hoffman and Erik Knutsen). With some luck, there will be some discussion of that sequel paper in Penn's excellent online companion, Pennumbra. (Please let me know if you're interested in participating.) In the next few months, I am also hoping to get the third part of the trilogy ready for the spring submission cycle, tentatively titled, Punitive Damages and Complex Litigation, and a book proposal unifying these ideas in a project called Fixing Punitive Damages.

I've also put up a lightly revised draft of Executing Retributivism, a paper I mentioned a few weeks ago about the implications of the Supreme Court's decision in Panetti v. Quarterman for Eighth Amendment review of cases involving both capital and non-capital punishments. I'm pleased to say that the piece will come out as an article in Northwestern U. Law Review in the late spring. In the next few months, I plan on growing the arguments in Part IV.A of the paper (about the implications of "negative retributivism" for non-capital punishment), which right now, are largely allusive in nature. I'll be presenting a version of this paper at Amherst College at an Austin Sarat-fest in a couple weeks and then revising it in light of comments around early to mid October. As a result, I'll be especially grateful for any comments that roll in prior to October 12. Also, Northwestern has an outstanding online companion, Colloquy, which sometimes publishes in both the print volume and online those responses to articles in the print volume. So if you're interested in participating in a Panetti-inspired forum, let me know.

Last, Ethan, Jennifer Collins and I are in the home stretch of getting our book manuscript together for Oxford University Press. The book is called Privilege or Punish: Criminal Justice and the Challenge of Family Ties. If you have the time and inclination to read it in draft, we'd be very grateful for your comments; we'll probably have a draft to circulate in the next week or so. The book is scheduled to appear in the early spring of 2009, and draws on but revises much of the work we've done here (our Illinois piece) and here (our BU piece forthcoming this December). The BU piece, Punishing Family Status, is the subject of a mini-symposium with wonderful and provocative responses by Rick Hills and Michael O'Hear. Michael's response is available in draft here. Our draft reply, Voluntarism, Vulnerability and Criminal Law, is available here. Come 2009, keep your eyes open for a Privilege or Punish Roundtable coming to you. We'll have a great group of people to talk about the book and its themes at Law and Society in Denver (tentatively Naomi Cahn from GW, Elizabeth Scott from Columbia, and Melissa Murray from Boalt) and at SEALS in August (tentatively Alafair Burke from Hofstra, Jack Chin from Arizona, Alice Ristroph from Seton Hall, and Bob Weisberg from Stanford).

Migrating to the Co-Op.

Starting today, I'll be posting on my new blog-home, Concurring Opinions, a blog started by Prawfs alums.  At Dan and Ethan's kind invitation, I'll still be posting around here occasionally over the next month as well. 

One of the many unique and great things about Prawfs that I will miss is its active network of commentators.   I hope that at least a few of you make a bit more time in your daily internet surfing to follow the goings on at the new digs, where Kaimi, Dan and Nate have already attracted a fair bit of attention (some of which has nothing to do with toys).

Hello Again

Thanks to Dan & Ethan for allowing me to rejoin the blog.  I've had a good summer "off" (consisting of blogging at the Conglomerate, writing, and visiting scattered family units).

My first time here, as our loyal readers may recall, I posted about a variety of random loosely connected topics, including: Nikolai Chernyshevsky, Internet lawlessness, juries, corporate law and the Lord of the Rings, a possible SSRN/Google deathmatch, faking conservatism, Kelo, and then more Kelo.  It was fun; I got  hate mail!  I even made it onto Wikipedia.

This time around, I'll try to be more focused.  To do so, I've developed a list of rules that may help me to comply with Kaimi's on-topic/off-topic guidelines: 

1. Only blog about Kelo when traffic is down;
2. Never blog about privacy except when Dan Solove is on vacation;
3. Try hard to resist self-referential posts that contain numbered lists;
4. Don't criticize other bloggers, unless they aren't lawyers or legal academics, in which case, savage them;
5. Don't blog about the first amendment unless Eugene Volokh is on vacation on a desert island without WiFi access;
6. Avoid linking to my SSRN page (its been done before!) even though it is now clear that SSRN hits are quite valuable;
7. Do mention Howard "the blawg daddy" Bashman often;
8. Don't blog about law school rankings, ever.  He who shall remain unnamed has got it covered;
9. Don't try to preempt other academic writing on a subject by merely by posting a short blog entry -- the rule is clear that preemption requires at least two major or five minor trackback entries;
10. Always remember the key to success:  more cowbell.