Corpus Linguistics Re-Redux

Since my last post on Corpus Linguistics two weeks ago, several things have happened in the corpus linguistics world that I’d like to discuss:  Stephen Mouritsen posted a significant and substantive response to several of my questions.  (His response can be found in this thread, and it is dated September 20).  Neal Goldfarb wrote two lengthy and important posts on his blog. And a new corpus linguistics paper about the likely meaning of the word “emolument” as it is used in the Constitution was posted on SSRN.  All three of these things have helped me further refine my views on corpus linguistics.  But I remain deeply concerned about using corpus linguistics as a methodology to interpret criminal statutes.

First, let me begin by saying that many law professors have not yet made up their minds about corpus linguistics.  They haven’t made up their minds because it is not clear what corpus linguistics and the law aims to do.  A number of people—especially those who consider themselves textualists—tell me that they see corpus frequency analysis as potentially useful for identifying possible or permissible meanings of an otherwise unclear statutory term.  But that is not what those who are advocating for corpus linguistics in the law say.  They tell us that, while dictionaries can help us identify permissible meanings, corpus linguistics can do more.  Specifically, they say it can help judges identify the ordinary or plain meaning of the statute.  In fact, they (at least occasionally) tell us that the frequency with which a word is used a particular way is information that *must* be taken into account in determining the ordinary meaning of a statutory term.

It is this aspect of corpus linguistics that has led me to characterize it as a method of statutory interpretation.  In his comment on my last post, Stephen Mouritsen offered some thoughts on whether he thinks that corpus linguistics is a methodology/theory of statutory interpretation.  While I recommend that you read his entire comment, I want to respond to a few of the things that Mouritsen said on this question.  Among other things, he said the following:

I disagree with your description of the approach that I (and my co-author) have described. I think corpus linguistics can be a very useful tool for providing information that can be relevant to some of the linguistically oriented questions that the law sometimes asks. I don’t think that it should be used to foreclose consideration of other evidence of meaning. While there may be methodological differences among the very few papers that apply a corpus approach, I disagree with the suggestion that what Justice Lee and I have advocated treats corpus linguistics as more than a tool that can provide useful information when the law asks certain linguistically oriented questions. In short, I don’t believe, and haven’t advocated, corpus linguistics as a “new twist on textualism.”

. . .

I think the dichotomy between, on the one hand, corpus linguistics as an interpretive methodology on par with textualism and purposivism (or even as you characterized it as “a new twist on textualism”), or, on the other hand, corpus linguistics as a source of “marginally” useful information, may not be an accurate way to frame the utility of linguistic corpora for interpretive problems. I think corpus linguistics can be a source of “very” useful information that can provide helpful insight in many cases in which the question that the law poses is a linguistically oriented question. Applications of the ordinary meaning canon are one example. And I don’t think that they are a necessarily “textualist” example. I can’t imagine that there is any jurisdiction in the United States that doesn’t use some variety of the plain or ordinary meaning canon of statutory interpretation, regardless of whether the judges who invoke it are textualists, purposivists, Dworkinian moralists, Sunsteinian minimalists, or fans of Nickelback.

I can’t be sure, but these passages suggest that Mouritsen would like corpus linguistics to be something more than an interpretive tool, but something less than a fully formed theory of interpretation.  But in saying that corpus linguistics ought to be considered “a source of ‘very’ useful information” for *any* method of statutory interpretation, including “purposivists, Dworkinian moralists, Sunsteinian minimalists, or fans of Nickelback,” I find it difficult to understand how Mouritsen sees corpus linguistics as something *less than* a theory of statutory interpretation.  A methodology that ought to be used no matter what your theory of interpretation is bigger than a “tool.” 

To try and get at what, precisely Lee and Mouritsen are trying to accomplish, I asked a question in my last post about whether they think that a frequency analysis is required in statutory interpretation.  Specifically, I asked whether Mouritsen was “backing away from a statement that he and Lee made last month on the Volokh Conspiracy that ‘a complete theory of ordinary meaning requires us to take into account the comparative frequency of different senses of words’”

Mouritsen responded:

[T]hat is not, in fact, what we said. We said: “[A] complete theory of ordinary meaning requires us to take into account the comparative frequency of different senses of words, the (syntactic, semantic and pragmatic) context of an utterance, its historical usage and the speech community in which it was uttered.” When you take into account each of these factors you may often find that a highly specialized, infrequent sense of a word is the most likely candidate for ordinary meaning. You may find that the context clearly eliminates some or all but one of the senses of the word as possibilities. You may also find that the word or phrase in question has many potential uses in that same context, in which case it is difficult to make the case that any of the potential senses is the ordinary meaning. So the characterization of the approach in your prior post that “if the database says a term is more often used as X than Y, then corpus linguistics tells us that is the ‘ordinary meaning,’” to me doesn’t capture the approach we have outlined in our paper. Yes, I think that the frequency with which a word gets used in a particular way, in a particular syntactic, semantic and (sometimes) pragmatic context that is similar to the legal text in question, in the speech community or linguistic register that is similar to the one of those subject to the statute, and from a similar time frame is a relevant consideration when determining the “ordinary meaning” of the words of a statute. As I noted before, “[t]o the extent that you find that a given sense of a word is overwhelmingly more common in a particular context similar to that of the statute, in a relevant speech community or register, and from a similar timeframe, I don’t think it is an extraordinary leap to conclude that the people subject to that statute would have understood the word in a way that is consistent with its most common meaning in those circumstances.” To the extent you have such evidence, I think it would be useful in resolving the question of ordinary meaning.

Mouritsen is, of course, correct that the sentence he and Lee wrote for the Volokh Conspiracy included more than just frequency as a requirement for ordinary meaning.  I thought I’d sufficiently acknowledged that by putting the period to end my sentence outside of the quotation marks.  But in retrospect I should have indicated the additional text with an “. . .”  

But I’m still left with my original question:  Do Lee and Mouritsen think that a determination of ordinary meaning should *always* include a corpus frequency analysis?  After all, Lee and Mourtisen presented the list of factors with the word “and”—that is to say, suggesting that *each* of the factors is necessary for what they called a “complete theory of ordinary meaning.”  But in his response, Mouritsen talks about frequency as “a relevant consideration” and “useful in resolving the question of ordinary meaning.”  So I’m afraid that I still don’t know whether Mouritsen thinks corpus linguistics is something that judges can consider if they feel like it, or if it is something judges ought to consider in every case (even if it isn’t dispositive).

This is the question I am trying to get at in my post (and to some extent in my paper)—What role do corpus linguists see frequency playing in statutory interpretation? I understand that corpus linguistics do not think that frequency will always tell us the meaning of a statutory term—though as I explain in the paper, if that frequency analysis is used by textualists, it may keep the courts from using other tools I think are better.  But in order to understand corpus linguistics as a legal methodology rather than as a linguistic methodology, it is important to know the precise role that Lee, Mourtisen, and others think that frequency data should play.  Is corpus linguistics a method for determining permissible/possible meanings, simply one tool that could be used to help determine ordinary meaning, or something that judges should always consider when choosing between various possible meanings?  After Mouritsen’s comments on my previous posts, I’m just not sure. I don’t see him saying that corpus linguistics should be used merely to determine possible or permissible meanings.  He thinks it can do more.  But how much more is unclear. 

Next, I want to return to the claim that corpus linguistics is a superior tool for discerning meaning because it is objective and scientific in nature.  In my last post, I took issue with Mouritsen’s statements that corpus linguistics “findings are replicable and falsifiable.”  Mouritsen responds:

You disagree with the characterization that the corpus linguistics approach to ordinary meaning is scientific. You acknowledge that the search that gathers the language data is replicable, but argue that “it is the analysis that matters” and different parties can draw (and have drawn) different conclusions from the same data. I simply don’t agree that the idea that people drawing different conclusions from the same data, or subjectively interpreting objective data, undermines the scientific nature of a corpus-based the inquiry. If we are going to classify as “unscientific” any form of inquiry where two people interpret the same data differently (and even subjectively), then we would have to condemn economics and every social science as “unscientific.” We may, in fact, have to shut down many upper level, theoretical courses in the “hard sciences” where subjective interpretation of data may lead different people to different theoretical conclusions. Disagreement about the interpretation of data is part of what Thomas Kuhn called the “route to normal science.” As I noted in my prior comment, corpus data won't “tell you what to do with th[e] information or whether that information is helpful for resolving certain types of questions. Shared standards, practices, and methods emerge when people in the relevant field start using the tool and start debating where it is useful and where it is not useful (or even harmful).” The fact that two parties look at the same data and interpret the data differently doesn’t render the enterprise unscientific.

I am afraid that I have to object to how my argument is being characterized. My complaint is not that corpus linguistics is unscientific, but rather that the use of the terms “replicable and falsifiable” suggests that the method connotes the same objectivity as an experiment in the hard sciences.  My post said:

I want to push back on the scientific/objective language that Mouritsen uses in his comments (and elsewhere) to advocate for corpus linguistics.  He tells us that “one of the chief benefits of the corpus approach is transparency. When corpus linguists are wrong about ordinary meaning, they are transparently wrong, because their approach and their findings are replicable and falsifiable.”  I see the rhetorical value of this language, but I have a very hard time understanding the analytical work that it does. 

In the hard sciences, we place great stock in findings that are replicable and falsifiable.  If, for example, someone conducts the same physics or chemistry experiment that I conducted, and their measurements are the same as mine, then we can be reasonably confident that my experiment was conducted appropriately and that its findings tell us something about the physical world.  But that hard sciences model does not map onto corpus linguistics—at least not corpus linguistics as a method of statutory interpretation.  That is because a corpus *search* is replicable, but the search itself doesn’t tell us anything about usage/meaning.  It is the *analysis* that matters.  To say that someone could conduct the same corpus search and obtain the same results is no different than saying someone could consult the same dictionary that I consult and find the same entries.  But just as I might draw different conclusions from those dictionary entries, so too are people likely to draw different conclusions based on their corpus analyses.

I don’t think that it is fair to characterize this objection to corpus linguistics as a claim that an enterprise is “unscientific” if “two parties look at the same data and interpret the data differently.”  Instead, my criticism is that while a corpus search is objective and replicable, the frequency analysis itself is highly subjective.  And the subjective nature of the analysis undercuts the promise of corpus linguistics as an objective method for determining ordinary meaning—something that Mouritsen has claimed in the past. 

Let me give an example of how non-objective a frequency analysis can be using the new corpus linguistics paper on the meaning of the word “emoluments” in the Constitution.  In their methodological section, the two authors explicitly note that the analysis of corpus search results is subjective, and thus they realized that they were likely to code results differently. In order to facilitate consistent coding, they practiced: that is to say, they independently coded material and then met to discuss why they had arrived at particular decisions.  After multiple rounds of these practice coding sessions, they were only agreeing on how to code a particular result 70% of the time.  This 30% disagreement rate after several rounds of practice, in my mind, severely undercuts the claim that corpus linguistics frequency determinations are “replicable and falsifiable” findings.

To be fair to Mouritsen, he backs away from the idea that corpus linguistics can tell us when a judge is “transparently wrong.” But he sticks to his guns about the benefits of corpus linguistics.  He says

the corpus data gives some content to these differing conceptions of ordinary meaning. It forces the judges to show their work and gives a transparent account of what each opinion appears to mean by “ordinary meaning.” This is in contrast to what happens when a court fails to explain what they mean by ordinary meaning and merely invokes the canon and it is not what happens when courts simply cite a dictionary (given that dictionaries don’t typically provide information about ordinary meaning and are typically cited, as James Brudney and Lawrence Baum pointed out in their article “Oasis or Mirage,” merely as window dressing).

I am unconvinced that we need corpus linguistics in order to ensure that judges engage with each other and transparently explain why they think a particular term ought to be read to have a particular meaning.  We already see a significant amount of back and forth between judges when they disagree about meaning—far more than we saw in the mid-twentieth century.  So I don’t think that we need corpus results in order for judges to explain their interpretive decisions; we just need a culture of reasoned explanation in judicial opinions. 

But even if Mouritsen is right that corpus linguistics will cause judges to engage with each other more, that does not make corpus linguistics more attractive to me. That is because I don’t think that a dispute over how to interpret frequency results of a corpus search would improve the quality of an interpretive dispute.  For one thing, as I explained at length in my essay , I don’t think that frequency is a good measure or method for determining the ordinary meaning of criminal laws.  For another, (also explained at length in the essay) I think that judges have a significant constitutional role to play in the interpretation of law—at least the criminal law—and that role is likely to be neglected if judges get caught up in how to appropriately categorize and count database results.  And given that corpus linguistics analysis is so subjective, I don’t see what we add to the enterprise by having judges fight about how to analyze the results of a corpus search.  I’d rather they engage directly with the statute and how the scope of the statute affects individual rights.

Although this post is already long enough, let me address some of the comments that Neal Goldfarb made in his response to my paper.  I (of course) agree with Neal that usage is not distinct from meaning, but rather meaning arises from usage.  But while usage is what determines permissible or possible meanings, those who advocate for corpus linguistics in the law go a step further: They tell us that frequency of usage can help us determine the meaning of an otherwise unclear term.  Here is how Neal explains the argument in his blog post on meaning:

It seems reasonable to expect that the higher the frequency with which a particular sense is associated with a particular type of context, the more likely it is that when the word is used in similar contexts in the future will be understood to have the same meaning. In this way of looking at the issue, the corpus data is seen as a rough representation of the input—what people hear and read—that shapes people’s understanding of word meanings.

It is known that the frequencies with which specific patterns and constructions occur has an effect on how language is learned and mentally processed. It doesn’t strike me as a big leap from that to the conclusion that the strength of the association between the use of a word in a particular type of context will depend at least in part on how frequently one encounters that word being used, in that type of context, to express that meaning. Or at least that seems reasonable if the idea of “types of context” can be adequately defined.

I will get to that, but first I need to note that I’m extrapolating from what I know of the literature, not reporting on what it says; I can’t point to any work that has been done on this specific question. Nevertheless, the assumption I’m discussing is, as far as I know, consistent with the fact that frequency effects are so widespread.

Importantly, Neal tells us that the linguistics literature does not speak to the fundamental premise of corpus linguistics and the law—namely, whether the frequency with which a particular word is used a particular way will affect how a listener is likely to understand the term when it appears in a statute.  Neal admits that he is making a leap from the literature and relying on his own common sense, and I sincerely appreciate Neal’s transparency about the assumptions that he is making.  But I will add that the assumptions that Neal makes do not address my concerns about notice.  Simply put, why would we think that frequency data would capture an average citizen’s intuitions about a statute’s meaning, but not a judge’s?

I can guess that Neal would say that my concerns about notice fail to appreciate the difference between interpretation and comprehension—the subject of his first response to my paper.    As he explains in that post “nobody is suggesting that corpus linguistics be used for investigating those kinds of intuitions. The whole purpose of using corpus linguistics is to learn things that aren’t accessible to intuition.”  But here is why I base my critique of corpus linguistics in the criminal law:  The criminal law does care about an average person’s ability to read a statute and articulate what it allows and what it prohibits.  If corpus linguistics is going to give us a different answer to the question “what does this criminal statute prohibit” than the answer we expect, then there are real problems under the Due Process Clause.  That is because one of the fundamental limitations on criminal statutes in modern America is that they must give people sufficient notice about what a particular law covers and what falls outside of it.  If corpus linguistics is valuable precisely because it gives us different answers than our intuition about statutory meaning, then I think it poses a significant threat to the need for notice in the interpretation of criminal laws.

Now maybe Neal and other corpus linguists would like to argue about what the Due Process Clause requires for the interpretation of criminal laws. Maybe they do not think that judges ought to ask how an ordinary citizen is likely to interpret the scope of a criminal statute.  I’d be quite happy to have that discussion.  Frankly, if lawyers, judges, and law professors are going to say that corpus linguistics ought to play a role in statutory interpretation, I think that they need to have these discussions.  We aren’t linguists. Our inquiry is not simply how might we better understand how people use words.  Our inquiry is how should we interpret statutes.  In advocating for the use of corpus linguistics in the law, that is what we need to be most focused on.  We can’t push that question down the road and say that “best practices” will develop over time. 

While I find much to disagree with in what Stephen and Neal have written, I can’t stress enough how much I appreciate them taking the time and effort to respond to my paper and my blog posts.  The engagement has allowed me to better understand and articulate the problems that I see with corpus linguistics and the criminal law. And it represents the best type of scholarly interaction—fair, dispassionate, and substantive.  And because of their helpful and thoughtful comments, I hope to have another post soon about context and corpus linguistics.