« The Power of SSRN | Main | More merits and standing overlaps »
Wednesday, August 30, 2023
The Peer Review Follies
Law professors all know about the problems with law reviews, but peer reviews in other fields have their own problems, one of which is interminable delay and ridiculous amounts of make-work for reviewers.
Because I have published interdisciplinary work in history, sociology, and medical ethics, I am sometimes asked by social science, humanities, and medical journals to peer review submissions. I got one such request from a flagship humanities journal in early autumn 2021. I read the draft and recommended publication, with a few suggestions for making it better, in November 2021. Other than a thank-you note, I heard nothing more from the journal until about a week ago.
On August 21, 2023, I got an email asking me to review a revision of the article. Other reviewers had evidently been less enthusiastic about the article – apparently have recommended a “revise and resubmit,” or R&R – although it seemed to me that 21 months was an exceptionally long time between my report and the requested revision review.
Now, it is possible that the author was responsible for some of the lengthy delay, although it seems more likely that at least one of the other reviewers (there are usually three) was tardy in submitting their report, followed by more delay from the journal editor. But that was the smallest part of the problem.
I was willing to read the revision, but having recommended publication of the original – which of course I hadn’t read for nearly two years – I asked to see the R&R recommendations. Otherwise, it would be impossible for me to assess whether the changes resolved the other reviewers’ reservations.
As I explained in my reply,
I already recommended publication of this article. I cannot review the revisions unless I understand why they were required. Was this an R&R? If so, was there another review besides mine? If so, I will need to see it in order to evaluate the new version. If there was no other review, what am I supposed to be evaluating?
I received this message in response:
We appreciated very much your report on the first submission of this manuscript. At that time, we received a number of reports and, as you surmised, asked the author to revise and resubmit their manuscript. We understand that you already thought it deserved publication from the strength of the first version but would be interested in knowing whether you thought this version remained as strong.
However, we do not distribute the reports of other readers when we ask people to consider whether a revised piece deserves publication. We completely understand if you feel this prevents you from being able to evaluate the revised piece.
The revised article is 42 pages long. The editors evidently wanted me to reaffirm my positive review – which would no doubt take two days or so – without knowing what perceived shortcomings were supposed to be remedied.
And to what end? My original recommendation was evidently insufficient to outweigh the R&R recommendations, so renewing it obviously would not matter if the other reviewers remain unsatisfied. If my evaluation wasn’t good enough then (and I am not saying it should have been), why would it be good enough now?
And that is why I declined the review. It was (1) impossible to do competently without seen the other recommendations, and (2) pointless under the circumstances.
Meanwhile, the author – who might well be an assistant professor who needs the publication for promotion – is stuck waiting over two years for an answer. And they cannot submit it anywhere else until then, because non-law journals do not permit multiple submissions.
Comments are open and will be monitored for relevance and civility.
Posted by Steve Lubet on August 30, 2023 at 06:27 AM | Permalink
Comments
P.S.
It is important to note:
“Implementation
It is unclear whether Section 3 is self-executing, which, if it is not, would leave federal and state courts or election authorities without power to determine the eligibility of candidates unless Congress enacts legislation to permit it.”
Not to mention Due Process Of Law- which I suppose would mean to say Section 3 is self executing is to ignore Section 1 and 5.
https://crsreports.congress.gov/product/pdf/lsb/lsb10569
Posted by: N.D. | Sep 10, 2023 4:43:16 PM
F.Y.I.
Useful information on how to not Folly by following original intent with the understanding that The Spirit Of The Law serves to complement The Letter Of The Law:
“originalism
ə-rĭj′ə-nə-lĭz″əm
noun
The theory that the US Constitution should be interpreted based on the intent of its authors, as determined by examining evidence of their understanding of the meaning of its wording in its historical context.
The view that a text should be interpreted according to the intent of its original authors.
The belief that the United States Constitution should be interpreted in the way the authors originally intended it.”
And:
https://legalknowledgebase.com/has-the-14th-amendment-section-3-ever-been-used
Posted by: N.D. | Sep 10, 2023 4:01:23 PM
“whole process
collocation in English
MEANINGS of whole and process
a series of actions that you take in order to achieve ...”
In closing, “How did American legal education become unmoored from the classical tradition”, which one could argue resulted in many a Folly?
Perhaps it was due to “mereology”, that moment in time when American Legal Education no longer recognized the self-evident Truth that God, The Most Holy And Undivided (Blessed )Trinity ( see Treaty Of Paris that ended The Revolutionary War), Is The Author Of Love, Of Life, And Of Marriage, and thus The Author Of Our Inherent Unalienable Right to Life, to Liberty, and To The Pursuit Of Happiness, the purpose of which can only be what God intended, and began to render onto Caesar what belongs to God, separating our inherent Unalienable Rights from The Author And The True God Who Endows us with our inherent Unalienable Rights from the moment we were created and brought into being, equal in Dignity, while being Complementary as a beloved son or daughter, at conception, and thus we are wholly equal before the Law.
Posted by: N.D. | Sep 9, 2023 4:20:52 PM
Two thoughts about Follies in general or how to avoid Follies in general:
https://plato.stanford.edu/entries/mereology/
When one considers The Fourteenth Amendment one must consider the various sections and how they relate to the whole as in, for example how section 3 relates to section 5, or section 1 relates to both citizens and non citizens in regards to not depriving any person of Life, Liberty, or Property without due process of Law, as every son or daughter of a human person from the moment of conception can only be, in essence, a wholly human person.
“Fourteenth Amendment Explained”
Section 1
“All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States and of the State wherein they reside. No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.”
Section 2
“Representatives shall be apportioned among the several States according to their respective numbers, counting the whole number of persons in each State, excluding Indians not taxed. But when the right to vote at any election for the choice of electors for President and Vice-President of the United States, Representatives in Congress, the Executive and Judicial officers of a State, or the members of the Legislature thereof, is denied to any of the male inhabitants of such State, being twenty-one years of age, and citizens of the United States, or in any way abridged, except for participation in rebellion, or other crime, the basis of representation therein shall be reduced in the proportion which the number of such male citizens shall bear to the whole number of male citizens twenty-one years of age in such State.”
Section 3
“No person shall be a Senator or Representative in Congress, or elector of President and Vice-President, or hold any office, civil or military, under the United States, or under any State, who, having previously taken an oath, as a member of Congress, or as an officer of the United States, or as a member of any State legislature, or as an executive or judicial officer of any State, to support the Constitution of the United States, shall have engaged in insurrection or rebellion against the same, or given aid or comfort to the enemies thereof. But Congress may by a vote of two-thirds of each House, remove such disability.”
Section 4
“The validity of the public debt of the United States, authorized by law, including debts incurred for payment of pensions and bounties for services in suppressing insurrection or rebellion, shall not be questioned. But neither the United States nor any State shall assume or pay any debt or obligation incurred in aid of insurrection or rebellion against the United States, or any claim for the loss or emancipation of any slave; but all such debts, obligations and claims shall be held illegal and void.”
Section 5
“The Congress shall have the power to enforce, by appropriate legislation, the provisions of this article.”
Posted by: N.D. | Sep 8, 2023 11:22:44 AM
Folly is as Folly does ; I wonder how often a Folly is due to sheer intellectual laziness in regards to using one’s critical thinking skills or just a lack of developing those skills due to a lack of practice?
Posted by: N.D. | Sep 1, 2023 11:49:18 AM
The law review peer review system is completely broken. I was recently a reviewer for a law review that purports to have strong peer review and recommended against publication because the empirics were not robust, spending about 3 hours on a review. I never even received a response from the journal and I see now that the article was published.
Many law reviews only give 24-48 hours notice as well which is plainly insufficient. For empirical or indisciplinary articles there needs to be more robust peer review. I cannot tell you how many empirical articles in law review are just plain out wrong - mistakes that a first year grad student in the social sciences or statistics would know. It's all the more troubling when empirical works in law reviews are cited by courts or newspapers to advance policy agendas. Indeed, it's dangerous that much empirical work in law reviews are not done very well, and then claims based on dubious statistics are then advanced to make policy points or - even scarier - by judges to make law. Something needs to change as even prestigious law reviews who purport to do peer review often do not do a meaningful job of evaluating pieces.
At the very least, it needs to change for empirical articles.
Posted by: Anon | Sep 1, 2023 11:31:20 AM
The norm in social science I think is that almost no articles are automatically accepted on the first pass. This is not a norm in law so I could see why the author of this post would be confused. Especially when the work is empirical, it can always be improved so in much of economics or political science, journals just are not use to any reviewer hitting the accept button on the first pass. I think I read the statistics for a top social science journal at some point and it was like under 5% were accepted without an R&R.
If you think the paper should be published, the norm at many journals is to give it an R&R. Scholars celebrate when they get an R&R, because, for the most part, if they make the revisions, it will probably be accepted. So many journals are used to having the reviewers review it again to make sure the revisions were made. It's weird here that the journals did not provide the other reviews. I have mostly received the other reviews so I can then assess whether changes were made, in addition to the changes I recommended. There really is no "privacy" reason not to provide the other review. In any event, because the editor probably forgot or didn't know that the blog post author didn't have any comments to the paper.
The purpose of peer review is to provide comments to the paper, not just to say accept it. Perhaps that is a norm that should be changed that would speed things up, but for the most part, if you are doing peer review, and you like the paper for most journals you say R&R, give some comments, and then when you get it back a few months later, check to see whether the changes were made. Often, the author of the article will do a memo responding to the reviewers. The blog post author should have been able to see this memo as well as the other reviews.
Posted by: Anon | Sep 1, 2023 11:24:50 AM
I also agree with Jason. You probably should have recommended publication again.
Posted by: Scott | Aug 31, 2023 11:01:27 PM
My comments are not unique, as others have posted similar comments already, but I wanted to offer some response based on my experience. My experience as a reviewer has varied across journals and disciplines. Generally, however, I am offered at least the author's "response to reviewers" document if not the text of the other reviews at the second review stage. This journal's lack of providing any context for the added review seems odd to me, and unhelpful to reviewers who most commonly volunteer their time for this process. I agree viewing the response from the author to others' comments can be helpful, especially when the other reviewers' perspectives may come from different fields than my own (which can cause me to reflect on ideas I had not considered in my own initial review, potentially helping me make an even more thoughtful recommendation the second time around). But I also agree with earlier comments that a quick scan through could have confirmed your initial recommendation, and thus provided at least one positive review on the paper's merits. And, a period of delay between reviews of almost 2 years seems very long. Unless the author was the cause of much of the delay, it seems there are some structural problems at play (not least of which may be that finding reviewers seems to have gotten much harder in the past few years).
Posted by: Bryce Newell | Aug 31, 2023 3:56:44 PM
My reaction is the same as Jason's. If you thought it was worthy of publication to begin with, you should just be able to do a quick pass-through to confirm that nothing has happened to the paper that renders it now unpublishable. Whether the other reviewers will change their minds upon resubmission is up to them.
Posted by: AB | Aug 31, 2023 11:30:09 AM
It does strike me as a valid point, albeit some level of dissatisfaction with a process may require or make reasonable a refusal to cooperate notwithstanding the effect on the author. The suggested short note also strikes me as a reasonable response. It also strikes me that even if Steve's choice was mistaken, it was reasonably mistaken, and that such choices are hardly uniquely the province of cranky old men. His choice was not itself especially cranky in any event, although one is welcome to think the post-event commentary was. (I take no strong view on that, other than that Steve has written and thought about academic practices a good deal, so it should be read in that context; it's not as if it's some one-off display of distemper.) But even if it were, crankiness obeys no age limits. (The larger point--that our own practices are out of step with the rest of the academy and even with the legal academy outside the United States, and are an embarrassment to our "discipline"--is both true and a drum many of us have been beating for a long time.)
Posted by: Paul Horwitz | Aug 31, 2023 11:18:59 AM
Jason raises good points. Perhaps I was wrong to decline the second review, though I continue to think (1) that a blandly renewed approval would have been irrelevant if the negative reviewers were still dissatisfied, (2) it would have been unnecessary if they were satisfied, and (3) the only way to write an effective review required access to the other reports and revision memo.
In the future, however, I will delete comments that include ad hominem insults, especially of the ageist variety.
Posted by: Steve L. | Aug 31, 2023 10:35:41 AM
I think that Steve's response is inappropriate. He screwed over the author, who now presumably has to wait for the journal to find another reviewer, and that new reviewer may reject the article for issues that none of the other reviewers raised or thought were significant. Journals often ask reviewers if they are willing to review a revised submission when they submit their first revision, but even if the journal here didn't specifically ask Steve that question, I think there is an understood norm, especially for reviewers of flagship journals, to of course agree to review a revised article. And I question how difficult it would have been for Steve to simply read the article quickly, and, presuming that nothing stood out as especially or, best he could remember, newly problematic, to send a short response to the editor: "I've read the revised version of the article and I continue to recommend publication." To refuse to do that strikes me as cranky-old-man behavior.
Posted by: Jason Yackee | Aug 31, 2023 10:13:24 AM
“However, we do not distribute the reports of other readers when we ask people to consider whether a revised piece deserves publication. We completely understand if you feel this prevents you from being able to evaluate the revised piece.”
The fact that the revised piece may have been revised due to a clarification of previous information that may have been erroneous or incomplete, and yet you have not been allowed to review this information before evaluating the revised piece is troublesome.
No doubt, when Time is of the essence, we can Trust, but should still verify, because in the whole scheme of things, while Truth begets Truth, error begets error.
If there is one thing that proved to be lacking in the days of Covid 19, it is our critical thinking skills.
We can know through both Faith and reason rightly ordered to Truth, that the battle against the atheist materialistic, over population alarmist globalist, is not a cultural battle, it is a spiritual battle, “against the rulers of darkness”, so “put on the full armor of God”, which includes your God given critical thinking skills, “the sword of The Spirit”, and never cease Praying for Wisdom, “to judge the things of the world in light of the highest end of man-The Contemplation of God”, The fullness of Life-affirming and Life-sustaining Perfect Love.
Godspeed!
Posted by: N.D. | Aug 31, 2023 7:35:04 AM
Anon is spot on. Long past time for law to join the rest of the academy.
Posted by: Anon2 | Aug 31, 2023 1:07:40 AM
I hope this doesn't come across as contrarian -- I'm a practicing lawyer with no direct experience of the peer-review process and limited experience with publication, so am genuinely curious . . .
My (perhaps erroneous) impression is that peer review is intended to identify quality scholarship that contributes to the field or is otherwise worth publishing on its merits. If that's even close to correct, then I'm struggling to understand why it would be particularly valuable, much less necessary, to review the revision history under these circumstances.
I can understand the benefits in terms of efficiency -- you took the time to review the previous version and concluded that it was worthy of publication, so it would save time to look only at the changes to assess whether those alter your conclusion. The passage of time and faded memory of that previous version would seem to diminish that rationale, but even if it were in full force, the introduction of some inefficiency doesn't seem to be your primary concern.
The idea that it would be necessary to see how the author addressed another reviewer's concerns also seems a bit odd to me. The objective of peer review does not seem to be to assess the author's growth, but rather the merits of the article. Perhaps the interest in the comments of other reviewers and the author's reactions to those are based on a sense of collaboration and value placed on the insights of another reviewer? I can see some logic in such a position, but only some. You were willing to participate in the process at the initial stage without visibility into other reviewers' comments, so it's not clear why a second round would be treated as collaborative or bust.
All of this is to wonder: if the point of peer review is to assess the merits of an article, why couldn't you make such an assessment of the updated article in the same way you made the assessment of the original, which was likewise without regard to its revision history?
Posted by: Curious reader | Aug 30, 2023 9:32:01 PM
In sociology, FYI, reviewers always get to see the other reviews - even if the article is rejected or accepted as is. And when there is a revision we get to see the author's revision memo. The practice you describe here is stupid and terrible.
Posted by: Philip N. Cohen | Aug 30, 2023 5:06:36 PM
Are you leaving a comment to draw attention to yourself, or to add to the conversation? As with most everything else in social media, blog comments work best as a way to Indirectly promote yourself. Write a comment that others find value in, and that encourages others to check out your blog, follow you on Twitter, etc.
Posted by: East Sussex | Aug 30, 2023 2:02:39 PM
Thanks, Jeff. A redlined version was only possible if the author had submitted one. I didn't ask and the editors didn't offer, so my supposition is that the revisions were indeed structural.
A colleague of mine outside the law school tells me that social science journals regularly provide all of the reviews when asking for second round reads. That has been my experience, too, when peer reviewing for social science and humanities journals, so this journal -- one of the top five or six in all humanities -- seems to be an outlier.
Posted by: Steve L. | Aug 30, 2023 11:50:51 AM
If the revisions were not structural, perhaps the editors could have given you a redlined version with the changes indicated? Then at least you would know what changed and could fairly quickly figure out whether to amend your original view.
FWIW, my experience (in terms of turnaround and quality of response) with peer-reviewed "law &" journals has been good. Weeks at most, and never anything like what you described.
Posted by: Jeff Lipshaw | Aug 30, 2023 11:42:50 AM
As you probably know, a good peer-reviewed journal will require an author who submits a R&R to additionally provide a "response to reviewers" document. Therein, the author addresses both the reviewer's/reviewers' original concerns and how the author has sought to address some/all of them (and if not some of them, why not).
I've never received the other reviewers' reports, and don't think an anonymous reviewer should either. However, with a "response to the reviewers" document (which, as far as I can recall, I've always received when dealing with someone's R&R) you should be able to learn at least some of the other reviewer's/reviewers' issues, and so be able to decide for yourself whether they are valid, important, spurious, etc.
You also don't know why this process took two years. (Economics journals are-- apparently--notorious beasts, though.) The turnaround time for the top journals in my areas at least (in the social sciences and humanities) are usually quite good.
Your annoyance with the process is valid, but still pales in comparison to the lack of professionalism and scholarly rigor that is the American law review system. No other academic discipline, and no other country, would ever (want to) emulate it--and for very good reasons.
Posted by: Anon | Aug 30, 2023 9:06:31 AM
The comments to this entry are closed.