Revising the Web of Science JCR ranking of law reviews

I want to conclude my discussion of the Web of Science JCR ranking of law reviews by offering several proposals for revising this ranking, which  draw on my co-authored paper ‘The Network of Law Reviews: Citation Cartels, Scientific Communities, and Journal Rankings’ (Modern Law Review) (with Judit Bar-Ilan, Reuven Cohen and Nir Schreiber). I want to emphasize that our proposals are tentative because I don’t think there is a single right answer as to how to devise such a ranking. They also do not cover the whole range of problems associated with such rankings. One of the main lessons of our analysis is that the choices underlying any ranking should be made explicit and that anyone using them should make sure that these methodological choices fit his needs. A further important note concerns the purpose of our project. We do not call for the use of metrics in evaluating research. As I noted in my first post, our project is based on the observation that there is currently an increasing global pressure to use metrics in order to evaluate research (both at the individual and the institutional levels). This trend makes it worthwhile to critically examine the methodology and structure of such metrics.

Our proposal draws on our finding that that PR and SE journals form two separated communities (see the citation graph here); however, this inward tendency is more pronounced in SE journals, especially generalist ones. We found that SE generalist journals, direct and receive most of their citations to and from SE journals. This finding reflects, we argued, a tacit cartelistic behavior, which is a product of deeply entrenched institutional practices (for a defense of this argument see my previous post). Because the mean number of references in SE journals is about 2.5 times greater than the mean number of references in PR journals lumping the two categories can generate a distorted image of the ranking of law reviews (see my post for a demonstration of this effect).   

We believe that there are two main paths for revising the WOS ranking. The first path is to create two separate rankings, one for student-edited (SE) (non-peer-reviewed) journals, and another for peer-reviewed (PR) journals. This approach reflects the different writing and citation styles of the two categories and their strikingly different article selection practices. Creating two different rankings would also cancel out the advantage that U.S. SE journals have in a combined ranking structure. While this approach does have some logic, it is also problematic because the two journal categories, despite their differences, still belong to the same scientific domain, explore similar questions and have over-lapping audiences. It is also inconsistent with the current practice of all the existing global law reviews rankings.  A second strategy would continue the current practice of lumping the two journal categories in a single ranking, but would offer a way to counter some of the distortive effects of the current structure of the WOS ranking. A basic component of this strategy would be to adjust the value of citations received from SE journals. As I demonstrated in a previous post, adjusting the citations of SE journals by a factor of 0.4 significantly changes the relative ranking of PR journals. Using an adjusted impact factor would not amount to a satisfactory solution by itself. We think that a better strategy would be to combine an adjusted impact factor with an algorithm that takes into account the prestige of the citing journal drawing on some variant of the page-rank algorithm. The idea is to calculate the prestige of a journal through an iterative process that computes the “prestige” gained by the journal through the transfer of prestige from all the other journals included in the network through citations. JCR already offer a ranking based on such algorithm although it is not widely used. 

Another problem concerns the composition of the ranking sample. The WOS includes a relatively small sample of law reviews (147 out of more than 1600 law reviews based on our recent counting of the Scopus, WOS and Washington and Lee datasets). This reflects the WOS philosophy that only well established and high quality journals should be included in the list. While this approach has some merit the current list leaves out many good journals which should have been included (both SE and PR). Another problem concerns the inclusion of interdisciplinary journals (especially PR) such as the Journal of Law & Economics, Law and Human Behaviour and International Environmental Agreements-Politics Law and Economics. These are high-quality publications, which publish articles that are very related to law, but are dominated by economists, psychologists and political scientists that study law-related questions (although law profs do publish in these venues occasionally). Should these publications be included in the same list as more classical law journals? I believe that they should because they provide a high-quality venue for interdisciplinary work that discusses legal problems, but I can see good arguments for both sides.