« Insights on Online Teaching | Main | Yet evading review »
Wednesday, September 12, 2018
Tacit Citation Cartel Between U.S. Law Reviews: Considering the Evidence
In my previous posts, which draw on my co-authored paper ‘The Network of Law Reviews: Citation Cartels, Scientific Communities, and Journal Rankings’ (Modern Law Review) (with Judit Bar-Ilan, Reuven Cohen and Nir Schreiber) I described how the metrics tide is penetrating the legal domain and also described the findings of our analysis of the Web of Science Journal Citation Reports of law reviews. We studied a sample of 90 journals, 45 U.S. student-edited (SE) and 45 peer-reviewed (PR) journals and found that SE generalist journals, direct and receive most of their citations to and from SE journals. We argued that this citation pattern is a product of tacit citation cartel between U.S. SE law reviews. Most of the comments focused on the following valid point: how can we distinguish between a tacit citation cartel and epistemically-driven scientific community (generated by common scientific interests). We argue, generally, that in tacit citation cartels, the clustering observed should extend beyond what can be explained by epistemic considerations, reflecting some deep-seated cultural and institutional biases.
In the paper we provide several arguments (both quantitative and qualitative) in support of our tacit cartel thesis. While none of them is conclusive in itself we think that jointly they provide a robust support for our thesis. First, we considered whether the clustering of U.S. SE journals could be explained by geographic proximity. Our sample included 57 U.S. journals consisting of all 45 SE journals and 12 PR ones. Statistical analysis reveals however that US PR journals do not receive more citations than non U.S. ones. Second, we also analyzed separately the sub-sample of generalist (PR & SE) journals but the citation pattern remained the same. Third, we considered the hypothesis that U.S. SE journals constitute a separate epistemic field – maybe due to their emphasis on U.S. law. We rejected this explanation on qualitative grounds, primarily because U.S. SE journals have become increasingly more theoretical and interdisciplinary over the past few years (Harry T. Edwards, ‘Another Look at Professor Rodell's "Goodbye to Law Reviews’; George L. Priest, ‘The Growth of Interdisciplinary Research and the Industrial Structure of the Production of Legal Ideas). This trend should make PR journals very relevant to U.S. legal scholarship. Fourth, one may try to explain the citation pattern by assuming a deep difference in the quality of the papers published in the two journal groups. We do not think this argument stands up to scrutiny. First, the selection practices of SE journals were subject to strong critique (e.g., Richard A Posner, ‘The Future of the Student-Edited Law Review’ (1995)). This critique casts doubts on the thesis that there is a strong and systemic difference in quality of papers published in the two categories. We also examined this claim empirically by looking into the citations received by the 10 top-cited articles published in PR journals in our dataset. We found that even these highly cited papers received only a small percentage of their citations from SE journals.
Finally, we also considered the accessibility of PR journals in Lexis, Westlaw and Hein. We found indeed that these databases only offer access to approximately half of the PR journals (See Table F, technical appendix.) However, we do not think that this fact provides a convincing explanation to the phenomenon we observed. We believe that most U.S. law schools have access to digital depositories that allow access to the PR journals in our sample. A quick search in 3 US libraries demonstrates that (https://www.law.pitt.edu/research-scholarly-journals; https://library.columbia.edu/find/eresources.html ; http://moritzlaw.osu.libguides.com/legalresearchdatabases ). Rather than providing an explanation to the citation pattern we found, this claim constitutes a manifestation of the institutional culture that facilitates the citation bias we identify. The comment we received from an AnonymousLawLibrarian (suggesting that U.S. legal academics, unlike equivalent scholars in the social science disciplines, only use Westlaw/Lexis/Hein or in-discipline journal research) seems to support our interpretation.
We think that this citation pattern is epistemically problematic because it hinders the flow of ideas. Further (and independently of the question of whether or not we are right in describing it as a tacit cartel) it can also influence the journals’ ranking. I will discuss this latter question in my next post.
Posted by Oren Perez on September 12, 2018 at 02:10 PM in Article Spotlight, Howard Wasserman, Law Review Review, Legal Theory | Permalink
Comments
I am glad the comment was useful.
To Paul's query, the metrics are currently tied purely to measuring cites without regard to peer-review or not peer-review. So while there are levels of evaluation which used to take peer-reviewed or not into consideration at the faculty level, a feature of growing central administration power is that university-level committees are rarely rubber stamps for both hiring and promotion. And these committees often have administrators and faculty members from other disciplines (including the natural sciences). So they look for "objective" metrics like citation counts (and again very few even now about US reviews are student run). This feeds back into faculty-level evaluations as they realize candidates need to conform to the grounds of these higher level reviews.
Some countries have had federal or university-system level initiatives that tried to rank journals (like A-F or 1-4) to evaluate publications, but even here many student-run reviews are rated highly in part because that it where the builders are now publishing and there is no investigation into whether they are peer reviewed or not. For an Australian example from 2010: lamp.infosys.deakin.edu.au/era/?page=fordet10&selfor=1801. Noticeably, after 2010 the Australian project stopped creating these gradations as the same issue arose that generally drives the resort to quantitative metrics - accusations of public choice style bias, politicization or other sources of discord over the shared basis of qualitative evaluation.
It is an awkward situation as if someone has published in the US (and there is a lot of great scholarship published in US international law journals by foreign academics), then even raising the issue would invalidate past publications. And foreign academics have also realized the great advantage of speed and article length that US reviews provide because we have the resources to host multiple reviews at each school, these resources present an attractive audience for the databases which the citation metrics pull from to include many of those reviews, and convince students it is worthwhile to work for "free" on them. It creates a situation akin to that in the US where people know the system is highly flawed - and no one would be able to transition their own system in the the student-run direction - but there really isn't any incentive to make a fuss about it because no one is empowered to organize a systemic change in the whole US system. Moreover, those that figure out the system best enjoy a competitive advantage over there less savvy colleagues. But, as the work done by the blog authors shows, you can develop other critiques and perhaps try to to raise awareness in the US or make an argument at home that there should be some coefficient applied to discount US reviews in the production of citations metrics.
For an similar sort of conversation, the next time you have chat up a foreign colleague ask them about the role of competitive grants in promotion and hiring, and public funding. And then be very thankful this near universal global phenomenon has not yet come to the US.
Posted by: USForeignProf | Sep 14, 2018 12:34:54 AM
I'm with Orin in finding the USForeignProf comment interesting and helpful. I certainly am aware in general terms of UK practices and the degree to which one's time is now taken up with preparing annual reports and so on, but it's one thing to read about it and another to live it. One thing surprises me and one thing is interesting about your comment:
a) I'm surprised that this has created rather than diminshed an incentive to publish in a US journal. I would have thought that the metric turn, especially insofar as it may be built by those used to peer review, would lead to US law journals being rejected as an unserious, ineligible, or not-worth-measuring site to publish.
b) I wouldn't be surprised if you're entirely right about the editor-pressuring or walk-the-paper-over-to-the editors phenomenon, and if there are denials that this occurs, obviously they are silly. (Although I don't have any data on how often it happens.) So I don't find it surprising. But I do find it depressing. As with novelty claims, overclaiming abstracts, and so on, I find it depressing both that junior scholars have incentives to do the wrong thing (from a scholarly/ethical/systemic perspective), that they may well be taught and encouraged to do this by mentors or by US acquaintances, and that this early training reinforces these practices and is likely to be passed on to the next generation of scholars and entrenched in the practices of these scholars as they get hired and eventually become tenured and established. People often complain about navel-gazing when legal scholars talk about scholarly processes and so on, and I get that; there is a big world outside law review publishing. But the tactics and practices are important for their systemic effects and their (in my view) distorting and harmful effects on what, in theory, ought to be a decent system with some scholarly seriousness and integrity. And it's important to talk about them because, as the commenter notes, alongside the standard complaints and criticism there is a good deal of happy-talk at the retail level from US law profs about the system, even (or especially) in cases in which those engaging in that talk behave entirely differently from the way in which they describe the system; we need more honesty and transparency about what actually happens and the kind of advice that junior scholars actually get from their fellows in US law schools or elsewhere. It's especially discouraging to think of junior foreign scholars, many of whom come from departments and systems and interdisciplinary backgrounds that may be more serious than the US law school system, coming under the influence of and even contributing, in the long run, to the problems it faces and for which we are responsible. Thanks again for your comment.
Posted by: Paul Horwitz | Sep 13, 2018 8:37:57 AM
USForeignProf, that was an extremely informative comment. Suddenly a lot of this makes sense. Thanks for explaining.
Posted by: Orin Kerr | Sep 13, 2018 2:38:54 AM
The prior three comments perfectly encapsulate my views on this (echoing the point that I didn't read the paper). Even so, I'll summarize:
1. I think your normative takeaway is clear as noted by anonprof.
2. I am perplexed by your statement that the use of Hein supports your thesis. As you say: "We believe that most U.S. law schools have access to digital depositories that allow access to the PR journals in our sample." You can lead a horse to water, but you can't make it drink. I use peer reviewed journals a lot, but I have to explicitly tell my RAs to look outside (and do so myself).
3. Thus, I agree with Orin - what you are describing is shoddy research practice, not a cartel. The journals aren't driving this. The professors aren't driving it (not by concerted action, anyway). My own suspicion is that the cause is fragmentation in both the journal consolidation market (Hein v. other databases), fragmentation in library funding (our law school has Hein, but I have to go to the main library for other journals - same was true at my last school), different views of prestige, and well-settled habits. I don't think any of this has to do with citation metric cartels. I'd love to hear why your tests disprove my theory (or why my theory is really a cartel).
Posted by: Michael Risch | Sep 12, 2018 9:45:00 PM
As someone who started their academic career in the US and then moved abroad, I think there is are unstated issue that might help clarify the motivation here. While some in the US may feel the sting of perceptions/rankings of law reviews for promotion, scholars in other countries are increasingly formally subjected to metric based discipline and sanction. While US universities have shifted much governance power to administrators, and state legislatures use public funding/tenure as a political weapon, US law schools still enjoy a comparatively high level of faculty control and insulation from these developments (such as the RAF in the UK).
Because of this use of citation metrics, US law review publications are weighted highly because of the citation practices discussed in these blogs. Honestly, as much as many US law profs have no idea how foreign law schools operate (as any discussion of US legal education reform implicitly reveals), many abroad had not idea until recently that high status journals in the US were student run (some still don't). So it was unclear for foreign academics (outside of the Israeli academy) whether publishing in the US would even get recognized for promotion/tenure as peer review was/is a universal global requirement outside the US. But the move to citation metrics muddied these waters, and as power shifted to central university and government administrators for whom the numbers only mattered.
So foreign journals have had their citation metrics diluted, and there is pressure to publish in US journals. The other lynchpin factor is that the core use of proxies by law students to select articles makes it near impossible for foreign academics to get published in higher ranked US journals, especially flagships, and even harder still as comparative scholarship is very common abroad and again is much harder to get 2L's to engage with. Many JSDs students who get prestigious publications with their advisors early on are shocked when their articles don't get offers once they submit from abroad on their own. And those that do quickly learn that asking their friends to pressure/advise student editors to look at their pieces is a normal practice (the anecdotal denial of this as a major explanatory factor in many US placements is understandable for insiders but is borne out by surveys of student editors).
Summarily, whatever the merits of the US system its inherent parochialism has merged with these disciplinary metrics to make life even more difficult for academic around the world. As in many other contexts, insularity may seem a negligible concern to those at a center of global power/prestige but it finds its way to impacting those abroad whether intended or not.
Posted by: USForeignProf | Sep 12, 2018 7:20:48 PM
Based on my reading of the blog posts and skimming the paper, I wonder if it might help to stopping using the word "cartel." That word conjures up a group acting quietly in perceived shared self-interest to achieve some improper goal. You don't have any evidence of the law reviews engaging in a cartel under that definition, it seems to me. Rather, you have evidence that U.S. law professors and U.S. law reviews are less integrated than they should be into the environment of peer-edited journals. In my view, using the word "cartel" to describe that is really distracting.
Posted by: Orin Kerr | Sep 12, 2018 4:08:58 PM
The nature of blogging is that you'll get comments without full reads of your paper. But based on these blog posts, I basically read you as making a normative claim that U.S. law professors ought to be more interdisciplinary, more international, and altogether more interested in the kind of articles that are generally published in peer-reviewed journals as opposed to student-edited journals. (Having published in both, many of your law prof readers have some idea of the difference.) Your empirical observation is that this is not happening as much as you think it ought to. Fair?
Posted by: anonlawprof | Sep 12, 2018 3:49:23 PM
The comments to this entry are closed.