Publishing review reports to reveal and preserve the quality and fairness of the peer review process

Peer review centrality in science

Peer review has several known limitations, but it is still an important and central institution in modern science (Seeber 2022): findings that have not faced the test of peer review are seen as unreliable, while works that have passed the scrutiny of peer review assume a sort ‘truth’ status (Baldwin 2018). Peer review is therefore adopted by academic outlets as the legitimate way to select scientific contributions (Lamont 2009), and to guarantee the quality of the research published and the credibility of scientific claims (Warren 2003; Bornmann 2008; Kalleberg 2012).

It is commonly assumed that the establishment of peer review occurred with the first example of peer review, in the Philosophical Transactions of the Royal Society in 1665 (National Academy of Science 1995). However, despite a common belief that peer review has always been a central, immutable institution of modern science, the institutionalization of peer review is a much more recent phenomenon (Fyfe et al. 2020). Only in the second half of the twentieth century, did editors begin to rely systematically on external referees to cope with the increasing size and specialization of scientific production (Burnham 1990; Baldwin 2018). Over time, peer review adopted different forms to respond to priorities and necessities of the scientific community. For instance, peer review was initially single blind, meaning that the authors do not know the identity of the reviewers, while reviewers know the identity of the authors. In the 70s, sociology journals started to adopt a double-blind peer review, whereby authors and reviewers do not know each other’s identity. This choice was driven by the universalism norm of Science (Merton 1973), which prescribes that scientific claims should be subjected to the same ‘pre-established impersonal criteria’, regardless of their source and not affected by the reputation and prestige of its authors or their institutions of affiliation. In computer science, the prime channel of communication is conferences, rather than journals, and reviewers are not selected by an editor: instead, conference chair members act as reviewers and choose which paper they wish to review (Seeber and Bacchelli 2017).

Current threats to research ethics in scientific publishing

In recent decades, major challenges for science’s integrity have derived from scientific malpractices, such as so-called paper mills and scientific plagiarism, as well as by the emergence of various degrees and types of predatory publishing.

Predatory and ‘grey’ publishers (i.e. publishers adopting borderline and contested practices; Siler 2020) are ‘corrupting the open access’ publishing model which started in early 2000s with the aim for ‘the better, lowered costs and expanded worldwide access to the latest research’ (Beall 2012, 179). These publishers exploit the Article Processing Charge (APC) business model of scientific publishing, which was introduced by the open-access movement and in which journal revenues, and thus sustainability of a journal, depend on the number of articles published. APC fits well Christensen’s definition of disruptive innovations, as those innovations that make it much simpler and affordable to own and use a product for people who, historically, didn’t have the money or skills to be in the market (Christensen and Euchner 2011). In the traditional, subscription-based business model, the revenues of scientific publishers are derived from subscription fees. The clients are readers – who want high quality and relevant articles, leading to an incentive for journals to perform at best their curation function, and select the most innovative and valid contributions by relying on rigorous peer review. This makes getting published difficult for authors as not all authors have academic competences to meet the quality standards. On the contrary, in the APC model the publisher`s revenues come from the fees paid by authors to publish an article, and the authors become the main clients. This transactional relationship between authors and publishers created a market for predatory and ‘grey’ publishers, seeking to publish as many articles as possible at a fee charged to authors and without rigorous peer review or conducting merely a superficial – or ‘façade’ – peer review. Predatory publishing makes publication much simpler for a whole population of people who historically did not have the academic competences – but do have the money – to publish in a scientific journal.

Disruptive innovations typically start in marginal market segments and incumbent organizations need time before being able to realize its importance. The open-access scientific publishing using APC model has increased tremendously in recent years (Zhang et al. 2022), especially after many research funding bodies attached a requirement to publish open access any research funded by them. Many subscription-based journals have added open-access publishing option for authors and some scientific publishers have established open-access journals. However, the legitimate journals which offer an open-access publishing option for any article that has been accepted for publication after rigorous peer review, are undermined by the predatory and ‘grey’ journals with façade peer review and subordination of academic functions to business interests (e.g. Beall 2012; De Vrieze 2018; Enserink 2015; Wals 2018). This is partly due to the fact that defining and distinguishing predatory from non-predatory journals is not always clear-cut, because of the scarcity of information and data availability for research and the difficulty of proving its quality and rigour (Squazzoni et al. 2020), and because most of the ‘grey’ and predatory journals appear or claim to conduct some sort of peer review (Siler 2020).

However difficult it is to demarcate legitimate from illegitimate scientific outlets, this is of foremost importance because it affects the legitimacy of scientific claims, and professional credit for hiring, tenure and promotion. Yet, how can legitimate scientific outlets be distinguished from journals adopting a façade peer review process?

One proposed measure to demonstrate the quality of peer review and expose poor scientific and peer review practices is the publication of peer review reports (Siler et al. 2021). The practice of publishing peer review reports, or ‘transparent’ peer review, is being piloted by an increasing number of journals, with mostly positive receptions (Nature 2020; Waltman, Larivière, and Milojević 2022). Publishing peer review reports can help to distinguish legitimate journals conducting rigorous peer review from predatory or ‘grey’ journals. In short, the legitimate publishing outlets can demonstrate rigorous peer review by publishing high quality peer review reports along with the published articles, whereas the predatory and ‘gray’ outlets cannot not. This can help expose paper mills and façade peer review (e.g. Day 2022).

Publishing peer review reports also makes standards of peer review across journals more transparent. It can expose potential malpractices in peer review, such as editors’ or reviewers’ coercion to self-citations (Chorus and Waltman 2016), or more lenient peer review of publications by scholars connected to the journal (Sarigöl et al. 2017; Tutuncu, Yucedogru, and Sarisoy 2022). It can also be informative to younger researchers who are only just beginning to publish. Published peer review reports offer examples to younger scholars of the learning trajectory by the author from the submission of the research manuscript to its publication, and the value of scholarly input through peer review in improving the quality of the manuscript. Moreover, publishing peer review reports enables rigorous research on the peer review process (Squazzoni et al. 2020), including the important questions on how scholars pass judgment on academic quality through evaluations of research articles (Lamont 2009), how scholars help advance knowledge and curate body of literature in their field, and how through formative comments in peer review they guide their colleagues to what they deem more quality academic publishing. The scarcity of data available on peer review is a major limitation to the advancement of knowledge in this area (Squazzoni et al. 2020). Such research is immensely important in itself, especially in times of fast proliferation of scientific publishing.

Publishing peer review reports at the EJHE

The European Journal of Higher Education seeks to pioneer the policy of ‘transparent peer review’ among higher education journals by publishing anonymous peer review reports to demonstrate the rigour of its peer review process. Starting in April 2023, the European Journal of Higher Education will start a pilot policy to publish the peer review report with the published article. Hence, any submission received after the launch of the policy and accepted for publication will at the time of publication include a link to an open access online peer review report including anonymous peer reviews from all rounds of review, while not including the responses of the authors. Both authors and reviewers will be alerted of the policy and asked to affirm their awareness of the policy before they can proceed with the submission or the review. This is a pilot of one year, initiated by the editorial team, approved by our Editorial Board, and enthusiastically supported by our publisher. We are grateful to Katrina Hulme-Cross, the Portfolio Manager for Education journals at Routledge (Taylor & Francis Group) for working with the legal, submission management and production teams to pave the way for this pilot to happen. This is not only the first such pilot for higher education journals, but also for journals within the Taylor & Francis Group.

The European Journal of Higher Education guarantees high and homogeneous standards of peer review, which involves two to three referees. Publishing the peer review reports will showcase that the peer review process is rigorous, high quality and fair. Performing a rigorous peer review process requires time and effort. We are extremely grateful to the members of our academic community for their invaluable contribution to our journal by submitting high quality peer reviews. Our journal prides itself with high standards of peer review processes which result in high quality publications. This is how all scientific publishing should be. This aim of upholding high standards of peer review can be reinforced through transparent peer review publishing.

Publishing peer review reports at the European Journal of Higher Education is conceived as a micro-level, bottom-up initiative in the context of a broader movement to preserve the high standards of evaluation of academic quality in scientific publishing. Ultimately, this is to preserve credible knowledge production, diffusion, and reputational signals, in a period in which these are severely challenged. It is paramount that we, as scientific community, collectively adopt solutions to avoid the diffusion of scientific and peer review malpractices and uphold high standards for scientific publications.

To promote the pilot in a transparent way, we will provide information to authors and reviewers at several stages before and during the peer review process. The pilot will last one year, which will give us the opportunity to understand whether this practice is welcome in our academic community. No innovation comes without risks. While some peer reviewers may be more willing to accept to peer review for a journal adopting a transparent approach, others may be hesitant to have their anonymous peer reviews available to the public. Some (young) authors can perceive an added value in showing that their work went through a rigorous peer review process and their capability to address challenging comments, others may be hesitant to submit to a journal which features peer review reports along with the published articles. Knowing that their comments will be published, peer reviewers can be less frank in their comments to the authors. A recent study monitored the effects of publishing peer review reports on the reviewers’ willingness to review, the type of recommendations, review turn-around times, and the tone of the report and found no difference, except for a slightly more positive tone (Bravo et al. 2019). We will closely monitor effects of the policy on submissions and peer review. Before the end of the pilot, we will canvass input from the academic community. This will be the basis for a decision whether to maintain the policy or to end it.

Authors’ and reviewers’ willingness to participate in this pilot is crucial to determine its long-term adoption, and for this practice to inform other academic journals in our field. We invite authors to continue to submit manuscripts for consideration for publication in our journal and reviewers to continue accepting our invitation to review. We invite all members of our academic community to share with us your thoughts on and experiences with publishing peer review reports and any other measures to uphold the high standards of academic publishing.