Science in peril: the crumbling pillar of peer review

Science relies upon the assiduous and critical observations of peers to ensure the validity and quality of refereed publications. And although the process of peer review is itself the subject of much debate and contention,¹ it currently threatens to be hounded into progressive disarray. Regardless of any on-going definitional and procedural disputes, the dissolution of peer review risks the viability and value of all sciences when science is needed, perhaps more so than at any other time.² This assessment is founded upon numerous, assembled accounts of contemporary experiences in review and editing for a number of Journals as well as discussions with a group of Journal Editors each facing these growing challenges. My personal experiences here are founded most especially in the discipline of psychology³ and its allied and satellite disciplines such as my own area of Human Factors and Ergonomics.⁴

In brief, the validity checking process of peer-review has traditionally been accomplished as primarily a pro bono activity. Predominantly, it has been undertaken by academics and research scientists, sometimes as part of their formal appointment profile, but most often as an adjunct and unremunerated service activity intrinsic to their professional career. A sense of obligation to this particular service function, alongside some supposed and purported traditional advantages of early exposure to new findings, served to create an ethos of review collegiality.⁵ In the same way that one’s own work was reviewed, so that putative debt was discharged through one’s own reviewing of the work of others. Institutions, in general, understood, condoned and to a greater or less degree ‘supported’ this crucial scientific function. In light of pre-print releases and immediate on-line access, the long-supposed advantage of surveying coming work has now largely evaporated.

In terms of the context of peer-review, times have changed.⁶ Most especially in university life, the ‘spare’ time to conduct this activity has almost completely evaporated. Increased administrative loads upon faculty, alongside an ever-burgeoning set of obligatory tasks have eaten in to any such discretionary pursuits.⁷ And the absolute burden of review has, of course, grown in proportion to the proliferation of the number of Journals, whether they be legitimate or otherwise.⁸ Thus, like comparable developments in multiple domains,⁹ the demand for what was once essentially a ‘free’ resource is itself growing at a non-sustainable rate.

It has, of course, not escaped the attention of various profit-driven constituencies that reviewing efforts, especially those undertaken by highly qualified individuals, represents a ‘cost-free’ resource, ripe for exploitation by those so inclined to do so in both the publishing world and beyond. Most especially, this has not been over-looked by the purveyors of ‘scam’ journals for whom the imprimatur of review by a high-profile scientist can be a fillip to their profit-generating, illegitimate activities.¹⁰ We are witnessing a system under stress that is endeavoring to adapt accordingly.¹¹ Harried Editors and their Associates search fervently for qualified reviewers to answer the call. For established and highly prestigious Journals, that call may well elicit positive response. But for Journals in unenvied lower quartiles, the task of securing informed and insightful peer-reviewers can, and is, becoming ever more arduous.

Adaptations to these increasing demands include the recruitment of younger scientists, including graduate and even undergraduate students, to fulfill this function. The quality of the reviews produced are often commensurate with the experience of their sources. Unfortunately, it is now not uncommon to find that blind review has transformed rather to uninformed review in which part of the author’s publishing process now requires the ‘education’ of one or more of their nominal ‘peer’ reviewers in terms of both content and methodology. Obviously, this is also a function of the ages and experience of the author and the reviewer respectively. Stressed Editors may well transmit these sub-standard assessments as they themselves are fixed upon the wheel of increasing professional demand. A second adaptation can involve a reduction in the number of reviews both elicited and utilized. The range in the number of reviews for any single paper can be surprisingly large. And with an almost necessary increase in the rate of reviewer disagreements as that number grows, it is hard to establish consensus in such cases. Thus, the attraction of using fewer reviewers grows commensurately. And all the while the pressure to return reviews in ever shorter cycle times is publicly acknowledged. Most especially this is because average time to first review completion is frequently expressed as one of the outward metrics of supposed quality by which journals are now judged. In consequence, the use of only one reviewer is certainly not unknown. Of course, this raises the issue as to whether any such singular assessment actually constitutes peer review at all.¹²

Such challenges and shortfalls imply a need to decide whether peer review is aimed at the highest common factor for publication, or the lowest common denominator for acceptance. Hence, there is a known and continuing blurring of quality differentiation across new and traditional scientific outlets.¹³ Not to mention the pay to play issues associated with open access articles.¹⁴ A third form of adaptation, of perhaps longer standing, is the screening of submission via Editorial or Board assessments. As one Journal recently observed: “[Our journal only] uses reviewers when the editor has decided that the manuscript is competitive” (parentheses mine). These ‘pre-screens’ can apparently be pragmatically justified, especially in ultra-high rejection rate outlets.¹⁵ However, in reality such strategies can prove to be, albeit perhaps unintentionally, pernicious. In a utilitarian fashion they permit the smooth functioning of the publication and elevate the coveted metric of ‘rejection rate.’ But at heart they are at best only an imperfect shadow of true peer-review and can be, if exercised by uncomprehending individuals, a screening system founded on perceived popularity and not on scientific merit as true peer-review requires. Like much of the rest of media and publishing industry, what is ‘hot’ can then often, and is, featured over what is substantive.¹⁶ It is, however, acknowledged, that the two attributes are, of course, not necessarily exclusive one of the other. Yet more and more Journals are forced toward these strategic exigencies as ‘turn-around’ times are touted on websites, alongside the aforementioned rejection rates and impact factors, as the source of the cybernetic feedback cues¹⁷ which mediate a Journal’s nominal quality.

If indeed peer review is truly in a parlous state, are there any potential solutions to offer? One that is especially relevant to the psychological sciences, is represented by the now almost ubiquitous problem solution method de jour, artificial intelligence.¹⁸ While application of this potentiality is not without its own perils,¹⁹ at least part of the ‘reviewing’ burden can be assumed by algorithmic inspection. Since many rejections in psychology, and presumably beyond, originate with flaws and failures in structuring and following appropriate methodological procedures, along with the appropriate and allied statistical analyses, then at least these elements could be checked for reviewers in the same, pattern-matching fashion that plagiarism is currently assessed in numerous academic contexts.²⁰ The caveat here being the tendency of such pattern-motivated technologies to reject the truly new and innovative, the assessment of which is a task still best left to human eyes.²¹ If technology is truly the best hope of avoiding an existential crisis in science, as well as all of the other technical innovations that are predicated upon scientific advances, then surely it is critical to find ways to buttress the process of peer-review, one major pillar of that scientific enterprise? In this way we help the scientific research process through innovations made possible by the prior benefits of science itself. The goal being not the automation of peer-review but the human-centered augmentation of a critical component of science which is, worryingly, threatening to get away from us. Like other aspirational visions, this prospective is evidently open to ready abuse. The potential for AI-written articles reviewed by AI-mediated systems and published in an ever-burgeoning panoply of unregulated, unmonitored, and unread Journals is a daunting prospect indeed. However, the status-quo in peer review appears untenable and the whole of the academic community needs to shore up this crucial crumbling pillar of science lest the best avenue to resolve the imminent existential crises that face us, is itself devalued.²²

Relevance to human factors/ergonomics theory

Science lies at the heart of Human Factors and Ergonomics and this work considers one of the most proximal threats to all of science.

Notes

1 Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99 (4), 178-182. And see: Benos, D.J., Bashari, E., Chaves, J.M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, D., McGowan, S., Polter, A., Qadri, Y., Sarfare, S., Schultz, K., Splittberger, R., Stephenson, J., Tower, C., Walton, G., & Zotov, A. (2007). The ups and downs of peer review. Advances in Physiology Education, 31, 145-152. Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: the author’s perspective. Scientometrics, 113 (1), 633-650.

2 See e.g., Hancock, P.A. (2019). In praise of civicide. Sustainable Earth, 2 (8), 1-6. Hancock, P.A. (2019). The humane use of human beings. Applied Ergonomics, 79, 91-97. Hancock, P.A. (2020). The humanity of humanless systems. Ergonomics in Design, 28 (3) 4-6. De Winter, J., & Hancock, P.A. (2021). Why human factors science is demonstrably necessary: Historical and evolutionary foundations. Ergonomics, 64 (9), 1115-1131.

3 And see e.g.,: Moreno-Fernández, M. M., Ramos-Álvarez, M. M., Valdés-Conroy, B., & Catena, A. (2008). Criteria of the peer review process for publication of experimental and quasi-experimental research in Psychology: A guide for creating research papers. International Journal of Clinical and Health Psychology, 8 (3), 751-764.

4 e.g., Dempsey, P.G., Wogalter, M.S., & Hancock, P.A. (2000). What’s in a name? Using terms from definitions to examine the fundamental foundation of Human Factors and Ergonomics science. Theoretical Issues in Ergonomic Science, 1 (1), 3-10. Hancock, P.A. (2022). Machining the mind to mind the machine. Theoretical Issues in Ergonomic Science, in press. Hancock, P.A. (2022). How Human Factors and Ergonomics saves lives. Applied Ergonomics, 98, 103585.

5 Kassirer, J.P., & Campion, E.W. (1994). Peer review: Crude and understudied, but indispensable. Journal of the American Medical Association, 272 (2), 96-97. Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002). Effects of editorial peer review: a systematic review. Journal of the American Medical Association, 287 (21), 2784-2786.

6 see Hancock, P.A., & Hoffman, R.R. (2015). Keeping up with intelligent technology. IEEE Intelligent Systems, 30 (1), 62-65.

7 Tight, M. (2010). Are academic workloads increasing? The post‐war survey evidence in the UK. Higher Education Quarterly, 64 (2), 200-215, cf., Soliman, I., & Soliman, H. (1997). Academic workload and quality. Assessment & Evaluation in Higher Education, 22 (2), 135-157. It is by no means the claim here that Faculty are unique in respect of the never-ending pursuit of greater effort for lesser remuneration by a system specifically oriented to that purpose. Rather, the proximal point is that non-obligatory tasks are readily ‘shed’ as obligatory requirements occupy more and more of the time available.

8 Daruka, I. (2014). Confined to grow?-Publication dynamics and the proliferation of scientific journals. Europhysics News, 45 (1), 19-22. And: Goel, R.K., & Faria, J.R. (2007). Proliferation of academic journals: effects on research quantity and quality. Metroeconomica, 58 (4), 536-549.

9 Dekker, S.W.A., Hancock, P.A., & Wilkin, P. (2013). Ergonomics and the humanities: Ethically engineering sustainable systems. Ergonomics, 56 (3), 357-364.

10 Beall, J. (2016). Essential information about predatory publishers and journals. International Higher Education, (86), 2-3.

11 Hancock, P.A., & Warm, J.S. (1989). A dynamic model of stress and sustained attention. Human Factors, 31, 519-537.

12 It can obviously be argued that the Editor constitutes at least one other set of eyes on the submission. However, the pressure on the latter individuals may induce them to ‘go with the Reviewer.’ After all, an Editor, even of a medium size Journal cannot diligently review every article that is submitted.

13 Ewing, J. (2006). Measuring journals. Notices of the AMS, 53 (9), 1049-1053. This can be an especially challenge when trying to make assessments of the quality and impact of an individual who publishes in a wide range of journals, some of which are relatively new and difficult to assess.

14 See e.g., Parker, M. (2013). The ethics of open access publishing. BioMed Central: Medical Ethics, 14 (16), 1-4.

15 The present work has itself fallen prey to this very strategy. The following response was received: “Thank you for submitting your manuscript "Science in Peril: The Crumbling Pillar of Peer Review." I am sorry to inform you that we cannot recommend your manuscript for (publication). Space in our Perspectives section is extremely limited, and most articles are invited, leaving very little room for uninvited contributions. In the context of other articles under consideration we did not find your inquiry to be competitive, our recommendation is that you seek publication in a more specialized journal.” Since the submission is manifestly about the whole of science, it is hard to see why the recommendation to a ‘specialized journal’ is offered, except as a standard pabulum for a journal which, in actuality, cannot now claim itself to be fully peer-reviewed! And see: Jefferson, T., Wager, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. Journal of the American Medical Association, 287 (21), 2786-2790.

16 The propensity is then to create self-isolated ‘echo chambers’ in which the permeation of the chamber walls becomes ever less achievable. An example from global warming is given in: Walter, S., Bruggeman, M., & Engesser, S. (2018). Echo chambers of denial: Explaining user comments on climate change. Environmental Communication, 12 (2), 204-217.

17 Hancock, P.A. (2019). The humane use of human beings. Applied Ergonomics, 79, 91-97.

18 But see: Kaplan, A.D., Kessler, T.T., Brill, J.C., & Hancock, P.A. (2022). Trust in Artificial Intelligence: Meta-analytic findings. Human Factors, in press. Hancock, P.A. (2022). Advisory adumbrations about autonomy’s acceptability. Human-Computer Interaction, 37 (3), 263-280. Hancock, P.A. (2014). Automation: How much is too much? Ergonomics, 57 (3), 449-454. Salmon, P.M., Carden, T., & Hancock, P.A. (2021). Putting humanity into inhuman systems: How Human Factors can be used to manage the risks associated artificial general intelligence. Human Factors Ergonomics in Manufacturing and Service Industries, 31 (2), 223-236.

19 Hancock, P.A. (2017). Imposing limits on autonomous systems. Ergonomics, 60 (2), 284-291. And see: Hancock, P.A. (2022). Avoiding autonomous agents’ adverse actions. Human-Computer Interaction, 37 (3), 211-236.

20 Li, Y. (2013). Text-based plagiarism in scientific publishing: issues, developments, and education. Science and Engineering Ethics, 19 (3), 1241-1254. And see: Suls, J., & Martin, R. (2009). The air we breathe: A critical look at practices and alternatives in the peer-review process. Perspectives on Psychological Science, 4 (1), 40-50. Of course, there remains a wide gap between a flawed design and analysis and an intentional attempt at plagiarism. The point here is that pattern-matching (widely writ) can accomplish both of the tasks to hand.

21 Braga, A., & Logan, R.K. (2017). The emperor of strong AI has no clothes: limits to artificial intelligence. Information, 8 (4), 156. And see: Hancock, P.A. (2022). Advisory adumbrations about autonomy’s acceptability. Human-Computer Interaction, 37 (3), 263-280.

22 It may be of more than passing interest to know that the present work was ‘desk rejected’ by one of the leading journals in science as not of general interest to the readership.