Research funding, monograph production, and journal pages are finite resources. Quality in selection turns on excellent appraisal. And peer review is the engine of excellent appraisal. Constructive criticism in the form of peer reviews lies at the heart of scholarship because it both facilitates better decisions in the allocation of limited resources and improves the quality of scholarly work. Peer review is not merely a filter; it is a crowbar that leverages the intelligence of anonymous experts in the service of the refinement and growth of knowledge.
Much has been written about the merits and limitations of the peer review process. Journal peer review, at least in our lifetimes, has been a process that typically happens behind closed doors. Editors are entrusted with seeking reviewers, assessing the quality of reviews, and making editorial decisions about the appropriateness of manuscripts for publication based on those reviews. While scholars hope that editorial oversight of peer review is a fair process, and that there are adequate controls to foster objectivity (such as sharing reviews and editorial decision letters with other reviewers as we do at Religion, Brain & Behavior), anyone who has participated in this process – author, reviewer, or editor – will be aware that biases and personal factors may creep in at times.
The peer reviewing landscape has undergone significant changes recently. For example, reviews in many journals, including top journals such as Science and Nature, are no longer double-blind: the identity of the authors is revealed to reviewers. Though RBB editors have discussed the pros and cons of this development, the review process at RBB remains double-blind.
Another recent development has been the emergence of online communities that publicize reviewer activity. We recently received an email inviting us to join such a community, which is aimed at opening up the peer review process to greater transparency and acknowledging the efforts of scholars who contribute to the review process. In such communities, with the permission of journal editors, reviews are posted online. The invitation we received described the goal of this particular service as follows:
Our mission is to improve the peer review process by giving reviewers a verified record of the work they do for journals. We feel that peer review constitutes an important contribution to science and want to help academics demonstrate their experience with the activity.
We would make this point stronger. Peer review is not merely an “important contribution to science”; it is a defining feature of excellent scholarship. For this reason, journal editors have a core responsibility to strive for ever fairer, more efficient, and more effective practices of peer review. Yet we (the editors of RBB) hesitated to accept this invitation because we did not agree that the proposed mechanism would improve scholarly outcomes. Consider our reasons.
First, while we appreciate the value of open access publications, online reviewer communities operate without clear evidence of expert evaluation of the quality of the posted reviews. For this reason, the on-line postings may be indistinguishable from “comments” sections in online newspapers, magazines, and blogs. After all, the posted reviews themselves are not reviewed for quality.
Second, and related to the first point, reviews vary in quality. In our experience, reviews can occasionally miss factual or thematic points in a paper and unfairly criticize or endorse claims based on those mistakes. Moreover, editors must frequently sift through opposing recommendations from reviewers. Knowing all this, we hesitate to agree to post reviews online.
Third, the process of publishing reviews online invites ever-spiraling problems of interpretation. Some scholars have suggested that all submitted versions of the manuscript, all reviews, and all editorial decision letters should be publicly accessible. It seems likely, however, that such complete transparency would bias the pool of potential reviewers. Reviewers who do not wish their reviews to be publicly available will simply opt out of the reviewing process. Furthermore, allowing reviewers to decide whether or not their reviews are posted would raise difficulties in deciphering editorial decisions: how will readers interpret editorial decisions when they only have access to part of the information available to editors?
Fourth, the motivational consequences of posting reviews online are murky, even for those of us who have spent our academic careers examining human motivations. One of the services these online communities offer is assistance in building an official record that can be highlighted on one's curriculum vitae. On the surface this seems laudable, but we are concerned that the shift to an explicitly self-serving motivation for reviewing will have unintended deleterious consequences. There are always better ways to beef up one's curriculum vitae than peer reviewing and once this is appreciated there will be little motivation for productive scholars to serve as peer reviewers.
Typically, we think, reviewers are motivated by a desire to contribute to something larger than themselves, such as the refinement and growth of science or knowledge in general. We are grateful for our reviewers' commitments to these values. To shift reviewer motivation away from these principled sensibilities toward adding gold stars to one's curriculum vitae seems detrimental to the stated aims of peer review. Admittedly, we may be excessively idealistic about scholarly motivations. One of us, at least, can trace his idealism to his own doctoral advisor. Shortly after graduate school, he received an extensive and critical peer review of one of his journal submissions, and the review came from none other than his graduate advisor, who chose not to remain anonymous. Initially startled by the inherent conflict of interest, on reflection it was obvious that the journal editor trusted this scholarly mentor to hold the values of peer review above his personal biases – in this case, the natural desire of a mentor to see a former student succeed. Supported by this evidence from personal experience, we believe that, no matter the status and promotional rewards pervading academic life, ultimately the life of scholarship is driven by our collective quest for understanding rather than personal career advancement.
In sum, we appreciate the efforts of these online communities and admire their goals, but we are not convinced that their methods will further the values inherent in peer review. We await better solutions.
We are left with a puzzle that all journal editors face: how to motivate scholars to review manuscripts and subsequently acknowledge their efforts while maintaining reviewer anonymity. A rule of thumb one of us learned in graduate school is that scholars should minimally put back into the system what they take out of it. That is, we should provide as many reviews per year as we have required through our own publishing efforts. Without such a balance of contributions the peer review system will collapse, or at least be severely compromised. Such a rule of thumb should also give authors pause before resubmitting rejected articles without seriously addressing revisions, since the cost of another submission will be two or three additional reviews. We do not know if such a norm is prevalent in academia, or in our small corner of it, but we do know that at RBB we have been very fortunate with the contributions of our reviewers. We are frequently thanked for the high quality reviews RBB submissions receive and, remarkably, this appreciation often comes from authors whose submissions we have rejected. We are genuine when we describe this situation as “fortunate.” We simply invite reviewers and, like our authors, we are continually impressed by the quality of reviews submitted.
We have also been impressed by the generosity of our reviewers in accepting our invitations. Lately, however, finding reviewers has been increasingly difficult. We are not sure why, and we do not aim here to make anyone feel guilty about declining an invitation to review a manuscript. We are hopeful that the challenge we are currently facing is an aberrant trend but, if it continues, we will indeed need to consider alternative strategies. Nonetheless, while we make an effort to thank all of our reviewers individually through private correspondence, we take this opportunity to acknowledge your contributions publicly. We know that Religion, Brain & Behavior could not exist without your generous efforts and we are truly grateful.