Requiem for impact factors and high publication charges

ABSTRACT

Journal impact factors, publication charges and assessment of quality and accuracy of scientific research are critical for researchers, managers, funders, policy makers, and society. Editors and publishers compete for impact factor rankings, to demonstrate how important their journals are, and researchers strive to publish in perceived top journals, despite high publication and access charges. This raises questions of how top journals are identified, whether assessments of impacts are accurate and whether high publication charges borne by the research community are justified, bearing in mind that they also collectively provide free peer-review to the publishers. Although traditional journals accelerated peer review and publication during the COVID-19 pandemic, preprint servers made a greater impact with over 30,000 open access articles becoming available and accelerating a trend already seen in other fields of research. We review and comment on the advantages and disadvantages of a range of assessment methods and the way in which they are used by researchers, managers, employers and publishers. We argue that new approaches to assessment are required to provide a realistic and comprehensive measure of the value of research and journals and we support open access publishing at a modest, affordable price to benefit research producers and consumers.

KEYWORDS:

• Metrics
• journal impact factor
• open access
• publication costs

1. Introduction

COAlition S, a consortium of research funders, launched “Plan S” in 2018 with the goal of providing a pathway to open access (OA) for research publications that were supported by state-funded grants with the cost of publication to be borne not by the researchers but by the funders or the institutions. 2021 will see the start of Plan S and it has the support of many European granting agencies, the World Health Organization, Wellcome Trust in the UK, and the Bill & Melinda Gates Foundation in the USA (Van Noorden 2020). The value of OA is unquestionable and we have previously argued very strongly for the removal of paywalls. Unfortunately, however, OA can come with a high price tag that is usually paid from the public purse (Triggle and Triggle 2017). These paywalls discriminate against those researchers and institutions that lack the financial resources to cover the cost of OA (Triggle and Triggle 2017).

The phrase “Publish or perish” appeared in a number of texts in the late 1920s and in the 1932 book “Archibald Cary Coolidge: Life and Letters” (Coolidge and Lord 1932). Credit should be given to Eugene Garfield (Garfield 1996a) for researching its current use in academia and tracing it back to Logan Wilson (Wilson 1942) who stated: “The prevailing pragmatism forced upon the academic group is that one must write something and get it into print. Situational imperatives dictate a ‘publish or perish’ credo within the ranks.” This is particularly important in the twenty-first century with increasing competition for jobs, career advancement, and research funding.

“Publish or perish” has become “Pay to Publish or Perish”, but does it have to be this way? In this article we review the role, value, and shortcomings of Journal Impact Factors (JIFs) as a metric for the assessment of the quality of research output, discuss the advantages versus disadvantages of other research quality assessment approaches, and whether the increasing financial burden of publishing in high impact factor journals is damaging access to and exchange of scientific knowledge.

2. The increasing number of papers, journals and preprint servers

Casadevall and Fang (2014) stated that there were more than 25,000 journals and over one million new articles published each year and the latest (2018) International Association of Scientific, Technical and Medical Publishers report lists over 30,000 English-language journals and over 3 million articles published annually (Johnson, Watkinson, and Mabe 2018). In this competitive environment some successful journals have introduced additional “stable partner” journals to compete with new journals established de novo and, to quote a question raised by the Editor of Acta Physiologica, Pontus Perrson: “Soon more journals than authors?” (Persson 2016). PubMed provides access to over 20 million published articles and as of 2010 it was estimated that 50 million articles had been published (Jinha 2010). With the exception of a pilot project launched by the NIH in mid-2020 to list COVID-19 preprints reporting NIH-supported research (see https://nlmdirector.nlm.nih.gov/2020/06/09/the-nih-preprint-pilot-a-new-experiment-for-a-new-era/), preprint server articles do not appear in PubMed and this could be perceived as a disadvantage, but they do provide an avenue for the progression from preprint server to a peer-reviewed journal.

The large increase in the numbers of manuscripts submitted to journals for publication has placed a greater workload on editors and reviewers of traditional journals and increased the importance of evaluation. The increase in research output has increased the demand for publications outlets, however, and we have seen the emergence of unscrupulous so-called deceptive, or predatory, publishers and journals that have, for financial profit, taken advantage of mostly inexperienced scientists who are looking to publish their works and, quite understandably, seek career advancement (Beall, 2016; Beall 2017; Cobey et al. 2019; Triggle and Triggle 2017).

The act of publishing in such journals may lead the authors into a “Publish & Perish” trap where careers might be jeopardized if it is assumed that they are boosting their curriculum vitae by bypassing a more vigorous peer-review process. Sydney Brenner, the Nobel Prize winner for Physiology or Medicine, 2002, expressed concerns over several issues related to the evaluation and funding of science and stated: “ – it’s not publish or perish, it’s publish in the okay places [or perish]” (Dzeng 2014): a view reemphasized by Dean Tantin in The Scientist (Tantin 2016).

The level of scrutiny that a manuscript receives prior to publication can vary dramatically, ranging from two to four reviewers for many established journals to less rigorous or no substantive peer review for predatory journals (Cobey et al. 2019). This is particularly relevant today with the explosion of open access and free preprint services that have been used extensively during the COVID-19 pandemic. In preprint publication mode, authors first publish (upload) their work to a server, and it is then reviewed, sometimes anonymously and sometimes openly. These comments and suggestions can be taken into account by authors. This “publish first, curate second” approach contrasts with the traditional approach. It is rapid, transparent and can have wider peer-review involvement and beneficial dialogue at a potentially lower cost to authors, their institutions, and funders (Stern and O’Shea 2019). Well-known examples of pre-print servers include arXiv (https://arxiv.org/) for the physical sciences, bioRxiv (https://www.biorxiv.org/) medRxiv (https://www.medrxiv.org/) for the biomedical sciences, SSRN (https://www.ssrn.com/index.cfm/en/) for the social sciences that includes a wide range of disciplines from accounting to women’s and gender studies, and Research Square (https://www.researchsquare.com/legal/editorial) for a variety of scientific areas. These services provide forums where preprints can be uploaded and their findings made immediately accessible for feedback before submission, sometimes to “traditional” established journals with formal peer-review.

COVID-19 has dramatically changed the way biomedical research is being disseminated. As of 15 December 2020, approximately 11,500 preprints related to COVID-19 were posted just on the bioRxiv and medRxiv servers. Else reported in late December 2020 that the number of COVID-19 related preprints uploaded to all preprint servers exceeded 30,000 and more than two-thirds of all the pre-prints posted on medRxiv since its launch in June 2019 were related to COVID-19 (Else 2020a). Although traditional journals also saw dramatic increases in submissions related to COVID-19, and, indeed, reduced the time for review as discussed by Fraser et al. (2019) in a bioRxiv preprint, bioRxiv-deposited journal articles were more favorably cited. This has been highlighted in the high-pressure publishing environment created by the COVID-19 outbreak where preprints became part of the scientific and public discourse, with dissemination of preliminary data that is not peer-reviewed and potentially contains errors (Majumder and Mandl 2020). The potential dangers are obvious, as became evident in the USA with COVID-19, where there was a clash between the needs of politicians to make informed decisions swiftly and the need for scientific evidence before the introduction of potentially dangerous treatments affecting millions of patients. The rush to publish results has clear and obvious dangers but is not necessarily associated with preprints as evident by retractions from high profile journals such as The New England Journal of Medicine and The Lancet (Soltani and Patini 2020). There are, of course, also significant advantages to publishing data via preprint servers, since providing the information to the global community may accelerate scientific progress and, in the case of COVID-19, speed up drug discovery, development and treatment.

Polka et al. (2021) asked the question: “Does the information shared in preprints typically withstand the scrutiny of peer review, or are conclusions likely to change in the version of record?”. Their study compared abstracts from bioRxiv and medRxiv preprints with the abstracts from the same studies after publication in a peer-reviewed journal. The answer is interesting as the changes from preprint to peer-reviewed resulted in only modest changes in 6% for non-COVID-19 studies and 15% for COVID-19-related abstracts, suggesting that the impact of peer-review was not dramatic. Once the COVID-19 pandemic is under control it is unlikely that the use of preprint servers will decrease. The value of OA and preprints has been recognized by F1000Research and PeerJ that provide OA and open peer-review. Originally launched in 2002 as F1000 biology F1000Research now covers the life and clinical sciences and charges up to 1350 USD per article. Similarly, PeerJ, launched in 2012, charges 1,195.00 USD per article but with a life membership fee of ~$400.00 USD an author can publish one article/year for free. eLife is also now offering an interesting hybrid model that recognizes the value of the preprint and was announced on 1 December 2020 by the OA journal eLife (Eisen et al. 2020). Commencing July 2021, eLife will only review and publish papers that have previously been posted on a pre-print server such as arXiv, bioXiv, or medXiv, and will adopt a “publish, then review” policy, with the peer-review reports to be made publicly available via the Sciety platform (https://sciety.org/) (Eisen et al. 2020; Kwon 2020). Eisen et al. (2020) summarize as follows:

All of these changes, if widely adopted, will create a version of the current publishing system that is more efficient, effective and transparent. But the real opportunities – and challenges – will come from the more radical and dramatic changes to science publishing that will be possible once we finally break free of the ‘one paper, one journal, one publication model’that still dominates the field

Thus, in the life sciences publishing models are changing with an increasing use of preprint servers to match what has been the practice in the physical sciences for 30 years as evident from the success of arXiv that was launched in 1991.

3. Price versus cost of publishing

The scientific and publishing industry is a hugely profitable 10 USD billion industry with the scholarly journal at its heart and earning the major publishers double digit profits (“Global 10B Scientific & Technical Publishing Industry Report, 2019-2023” 2020). In an article entitled “ Current market rates for scholarly publishing services”, Grossmann and Brembs (Grossmann and Brembs 2021) calculated that a scholarly article should not cost the tax payer more than a few hundred dollars or euros, yet the price charged for publication in some journals is at least 10 times higher and funders and researchers should ask what justifies the cost and what they get for the money.

Ideally OA should be readily available to all at low cost; after all why pay a high price to publish your paper when the research has most likely been supported from the public coffers and yet the profit goes to a third party – is this money for nothing? In November 2020 the publisher Springer Nature, announced that in 2021 it will provide immediate OA to articles published in Nature after authors or their institutions pay an APC of €9500 (~$11,560.00USD) thus providing the “gold route” to OA where there are no subscription or charge-barriers to access the published article. Whereas this is good news for researchers wishing to access the published paper, the amount to be charged exceeds that of other prestige journals and is substantially higher than the mean for OA (approximately 2000.00 USD) (Else 2020b; Brainard 2021). Interestingly, Cell Press journals followed Nature and announced that commencing in 2021 the APC for full OA to its flagship journal, Cell, would cost 9900.00. USD Removing a paywall and providing access to the scientific literature is a positive step, but the cost for many researchers and academic institutions is very high, especially for researchers in poorer countries and less well-funded labs. Khoo (2019), however, pointed out that when authors can choose between venues with different costs, they elect to publish in the higher-cost venue.

Golden Open-access (Gold OA) is defined as providing immediate access to the publication subject to meeting the APC requirements of the journal publisher. Publishing in Gold OA journals makes research more accessible, which probably increases citations, but the publication fees of journals perceived as high quality are often very high. The Research Coucils UK (RCUK) have stated that Gold OA is the preferred mechanism for OA publishing of outputs that they have funded and award grants to eligible institutions to help ahieve this, and have also indicated a positive correlation between APCs and JIFs (Andrew ,T 2013). Indeed, the high cost of APCs and OA fees correlates well with JIF metrics such as IF, Eigen Factor (EF), citability, h-index, are correlated with APC charges and for publishers from high-income countries (Gray 2020). Perhaps one perceived benefit of publishing in Nature is that it includes the bragging rights of publishing in not only a journal with a high IF but also the journal with the most expensive OA price tag? As stated by Brainard (2021): “If paying for open-access publication becomes the default route for scientists, and publishers hike prices as expected, many analysts worry publishing will become a luxury that only better funded researchers can afford. That could create a self-reinforcing cycle in which well-funded researchers publish more, potentially attracting more attention – and more funding”.

4. Evaluation of published research

In an age of rapid worldwide communication, widespread use and abuse of social media, and frequent examples of “fake news” and misinformation, it is of particular importance for scientists to be able to assure colleagues, policy makers and their wider audience of the accuracy of the information that they publish. As Hippocrates is often quoted as saying:

There are in fact two things, science and opinion; the former begets knowledge, the latter ignorance.

We should ask: “Has this increase in quantity of both paper and journals reduced the quality of the published products?” We will argue that the answer is, “Yes”. According to Charles Jennings, the former editor with Nature and founder of Nature Neuroscience: “To succeed in science; one must climb this pyramid: in academia at least; publication in one or more prestigious journals is the key to professional advancement” (Jennings 2006). This raises the question: How are prestigious journals identified? In the current environment there is a tendency for this to be done by IF and a competition to publish in journals with the highest JIF. JIF has become the most commonly used metric for assessing scientific journals and also the most fiercely debated and criticized thus raising serious concerns about the validity of JIF as a measure of quality. However, as we discussed above, will a high OA fee now become the new gold standard for prestige? Based on data presented by Khoo (2019) the answer to the latter question is yes.

There are several mechanisms used to assess and evaluate the reliability of articles published in the scientific literature, including peer-review, calculation of citations, and impact factors, discussed below. Seglen (1997), however, stated the problem clearly: “Evaluating scientific quality is a notoriously difficult problem which has no standard solution”. Furthermore, as in the formulation of Goodhart’s law (“any observed statistical regularity will tend to break down when pressure is applied to it for control purposes”) or the similar Campbell’s Law (“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”) and paraphrased by Strathern (1997): “When a measure becomes a target, it ceases to be a good measure”, evaluation can be frequently distorted by “gaming the metrics” (Biagioli and Lippman 2020).

Top journals trumpet their high IFs. Editors and their boards compete for ranking positions in league tables in order to demonstrate how important their journals are. This helps to attract submissions from top authors and continues the flow of popular papers. IF is easy to calculate and generates a number that can be used in league tables. It is often argued that IF is related to the importance of not only the journal but, incorrectly, also all manuscripts published in that journal. Indeed, subject to the assumptions made, statistical arguments can be made to justify IF as a valid metric, but variables such as a small number of articles published annually in a journal can greatly distort citations and JIFs (Waltman and Traag 2021). Of course, a strong argument can be made for a vigorous peer-review process for all journals, but the outcomes of peer-review are not measured directly nor can they be quantified. We can ask: does high IF guarantee high quality reviewing and is a publication in a high impact journal “better” than one published in a lower impact journal? The answer is there is no convincing evidence that a publication in a journal with a high JIF is of greater value than one from a journal with a lower JIF. There is no simple mathematical formula that can be applied to quantify each contribution to the published literature (Figueredo 2006). It is also difficult to rank the contributions of individual authors to a multi-authored, particularly when the number of authors is in the double or even treble digits, as we will discuss again later.

Despite these criticisms and the availability of alternative routes for the dissemination of research data, JIF is stll used as an important metric for assessing the quality of both the journal and the peer-reviewed science that it publishes. So, what is JIF, how is it calculated and what does it measure?

4.1. The impact of JIF on scientific publishing

The original purpose behind JIF, introduced by Scientific Information in 1955, was to provide libraries with a metric to aid in prioritizing journal subscriptions. It was never intended that JIF would be used to judge the quality of an individual publication as it should not be assumed that citations of articles within the same journal show a narrow quality distribution (Garfield 1994, 1996b). JIF, as defined by Eugene Garfield, is derived by dividing the total number of citations of all of the articles for that journal published in the previous 2 years by the total number of articles published in the same 2-year period, using citation data from the Web of Science database (Garfield 1996b).

Several national governments have implemented policies to assess the value of research work published by investigators, and to encourage publication in international journals with high JIFs (Franzoni, Scellato, and Stephan 2011). Such initiatives, including the UK Research Assessment Exercise, which began in 1986 and occurs regularly, were implemented to facilitate decisions about the allocation of research funding to academic departments and are indirectly related to career progression. In some countries, such as Spain and Germany, salaries and promotion have been linked to research performance, and the JIF of outputs are widely used to assess achievements and suitability for appointing new academic staff, career advancement, and award of competitive research funding. A reward system provided as a cash bonus for publishing in high IF journals such as Nature and Science was introduced by Korea and China in 2006, and other countries have followed, although for China, cash rewards had started locally in some institutions much earlier (Ding 2001; Fuyuno and Cyranoski 2006; Quan, Chen, and Shu 2017; Zhang 2006). The concept of “pay-for-performance” has long existed in the private sector with benefits in terms of productivity and loyalty (Frey, Homberg, and Osterloh 2013). Arguably, the concept of the motivation inherent in the “self-interested homo oeconomicus” should also apply to academia and there are clearly short-term benefits to such cash incentives. The long-term benefits, however, can be questioned, as such incentives may encourage a scientist to pursue projects with a quick return on results rather than pioneer much more difficult, but ultimately more innovative, long-term research (Quan, Chen, and Shu 2017).

The financial rewards for publishing in the ‘right” journals can be very generous. In China, publication in Science and Nature can benefit the authors >$40,000 USD, but just ~$3000 for PNAS, despite its high status, giving rise to a “publish or impoverish” scenario (Quan, Chen, and Shu 2017). However, even as China is making advances in heightening awareness of research ethics and integrity issues (Yang 2013), concerns are being expressed that too much emphasis is being placed on the number of published papers and career advancement (a universal problem for young scientists) resulting in incidents of buying authorships being reported (Hvistendahl 2013). There are now numerous cases where multiple papers published by highly regarded authors, or groups of scientists, in high impact factor journals have been retracted (Brainard 2018). The concern for scientific integrity is further heightened when very high-profile scientists such as the immunologist, Professor Cao Xuetao, President of Nankai University in Tianjin, and between 2011 and 2015 also President of the Chinese Academy of Medical Sciences, have been accused of data manipulation in a number of publications. With respect to Professor Cao the specifics as to who was responsible and whether it was in error rather than fraud remain unclear and investigations continue (Silver 2019). On a positive note China has announced major reforms in how science is assessed that include a “Farewell to “SCI worship and a move away from Web of Science as a standard for assessment” (Zhang and Sivertsen 2020).

Elsewhere, cash bonuses are also being considered and in India used as an incentive for PhD students to publish in high IF journals (Vaidyanathan 2019). Although financial incentives can provide generous rewards to the authors and have enhanced submissions to high IF journals (Abritis and McCook 2017; Franzoni, Scellato, and Stephan 2011), the obvious danger is that availability of such incentives and the pressure to publish increases the risk of both unintentional errors and fraud. Concerns over the misinterpretation and misuse of metrics as a means of assessing the impact of research and productivity of researchers have been well documented in a series of articles published in “Gaming the Metrics” (Biagioli and Lippman 2020) and many of the concerns expressed in their review are detailed with examples by the authors. Attempting to gain an advantage over competitors for career advancement, salary bonuses, and fame, by focusing on the manipulation of performance measures, such as increasing the number of publications in prestigious journals, rather than producing better research – in other words “gaming the metrics” – is just one example of incentive gaming endemic in many aspects of life (see Frey, Homberg, and Osterloh 2013 for numerous examples).

The use of incentives such as performance-related pay (PRP) to enhance productivity and benefit both employees and managers is widespread, but not always ethical, and does not necessarily have the desired result (Frey, Homberg, and Osterloh 2013). For example, in order to reduce the long patient wait time, the Veterans Administration (VA) in the USA initiated a bonus system for administrators designed to limit waiting times for doctor appointments to 14 days or less. In one instance, patients were placed on a pre-wait list for up to 6-weeks and only allowed to phone for an appointment when they reached the top of the waiting list, with the wait time recorded from the date of the phone call, excluding the prior waiting list time (Pierce 2019). When in 2014 the UK replaced pay progression based on length of service with PRP, this was challenged by the UK National Education Union as unfair and not a motivating factor likely to enhance student performance (National Education Union 2019). Indeed, a prior review by the Organization for Economic Co-operation and Development (OECD) had revealed the lack of relationship between average student performance in a country and the use of PRP schemes (OECD 2012) and simply concluded that higher pay (and status in society) for teachers results in higher student performance.

4.2.. JIF is not a proxy for quality

In 1997 Per Seglen (1997) summarized in four points why JIFs should not be used for the evaluation of research:

1. “Use of journal impact factors conceals the difference in article citation rates (articles in the most cited half of articles in a journal are cited 10 times as often as the least cited half).

2. Journals’ impact factors are determined by technicalities unrelated to the scientific quality

of their articles.

3. Journal impact factors depend on the research field: high impact factors are likely in

journals covering large areas of basic research with a rapidly expanding but short lived

literature that uses many references per article.

4. Article citation rates determine the journal impact factor, not vice versa.”

It is therefore surprising that publishing in journals with a high JIF is still widely used as a proxy for quality and performance. It is important to detail that beneath the seemingly simple calculation that determines JIF there are a number of underlying complications. The editors of PLoS Medicine (PLoS Medicine Editors 2006) have noted that JIF can vary dramatically depending on which articles in the journal are considered as citable. Thus, for PLoS Medicine the 2005 JIF was 11 when only research articles were considered but less than 3 when all articles within the journal were included, hence the opportunity for impact factor gaming, whereby journals can boost their JIF by publishing more highly-cited kinds of articles but not necessarily “better” ones. In fact, it is not necessary for journals actually to publish more highly-cited papers, as many editors have made a practice of negotiating down the number of “citable items” in the JIF calculation while keeping the number of citations to the journal high and thus manipulating JIF (Rossner, Van Epps, and Hill 2007; Brembs, Button, and Munafo 2013) and have done so since at least the mid-1990s (Moed and Van Leeuwen, 1995). Stephen Royle has clearly illustrated that this makes reproducing the data upon which the JIF is calculated “far from easy” and “reverse engineering the JIF” (Royle 2016).

Another fundamental problem is the highly skewed distribution of the number of citations received by each article within a journal (Lariviere and Sugimoto 2018). In most cases, a comparatively small percentage of articles are highly cited, while the remainder have a much lower citation rate (Larivière et al. 2016; Seglen 1997). Such highly skewed data should persuade every scientist that a simple average like JIF is not, from a statistical point of view, an accurate indicator of scientific excellence. The lack of any clear correlation between the journal JIF and the popularity or impact of any individual paper is reflected in the disparity of the citation rate for different papers published in the same journal, It was on this basis that Larivière et al. (2016) made the following recommendations:

“We encourage journal editors and publishers that advertise or display JIFs to publish their own distributions using the above method, ideally alongside statements of support for the view that JIFs have little value in the assessment of individuals or individual pieces of work.”

For JIF it can be argued that despite the considerable criticisms discussed by many and emphasized in this article, its widespread use persists not because of its true assessment of quality but because it is so easy to calculate, and perhaps also because retaining it is in the interests of certain publishers and also some researchers. JIF not only ignores the origins of the citations that support its score, but also neglects the fact, discussed above, that the impact of an individual published paper may be quite different from the JIF for the journal as a whole.

There is an extensive literature that provides arguments as to why we should not use JIF as the metric for determining the quality of the science, with evidence that many successful scientists have credible and even exceptional CVs without publishing in the so-called high-impact journals (“San Francisco Declaration on Research Assessment” 2013; Archambault and Larivière 2009; Brembs, Button, and Munafo 2013; Brembs 2019; Callaway 2016; Casadevall and Fang 2014; Chawla 2018; Garfield 2006; Gingras 2016; Larivière et al. 2016; Leydesdorff, Wouters, and Bornmann 2016; Seglen 1992, Seglen 1997; Triggle 2015; Triggle and Triggle 2007; Vanclay 2012).

Papers in journals with high JIFs can be very readable and give notice of major advances and new discoveries. One might expect, therefore, that a high JIF factor indicates a higher standard of interest, accuracy and reliability of papers published therein. This is sometimes true but unfortunately is certainly not always the case (Brembs 2018, 2019). Thus, Björn Brembs (2019) concluded: “There is a growing body of evidence against our subjective notion of more prestigious journals publishing ‘better’ science. In fact, the most prestigious journals may be publishing the least reliable science.” It has been argued by Ioannidis (2005) that based on misuse of statistical analysis a large percentage of the conclusions based on published research data are invalid. Based on computer modeling it has been argued that approximately half of studies published will make erroneous conclusions (Higginson and Munafò 2016). A similar conclusion was reached by Smaldino and McElreath (2016) in an article with the title: “The natural selection of bad science” in which the authors detail the contribution of powerful career advancement incentives to the perpetuation of bad study design and analysis that results in false-positive conclusions. Smaldino and McElreath also raise the potential contribution of junior apprentice scientists learning bad research habits from their mentors and peers akin to natural selection, thus perpetuating the bad habits. These concerns as well as those documented about the inconsistencies of the peer-review process (see Triggle and Triggle 2007), the amount of often wasted time writing grants, the low percentage of grants funded, has prompted the controversial argument that a contest model (partial lottery system) would be a fairer way of awarding grant funds from the larger pool of those applications identified as fundable (Fang and Casadevall 2016; Gross, 2019).

The increase in the number of retractions is a major concern. These have risen dramatically from <100 in 2000 to >1000 in 2014 and although this only represents ~0.025 to 0.04% of published papers (Brainard 2018; Campos-Varela, Villaverde-Castañeda, and Ruano-Raviña 2020) retractions are a major concern for influential journals. Retraction Watch reported 1433 retractions in 2019 (Oransky and Marcus 2019) and it is notable and concerning that there have been a high number of retractions published in high JIF journals in the last couple of decades, with detailed post publication analysis identifying evidence of either extreme carelessness or data manipulation. One estimate credits the highest impact journals with ~30% of the retractions and the lowest impact journals with just 10%, although misconduct was higher for low impact journals (73 versus 61%) but with differences between the disciplines. The highest retraction rates were for biochemistry, molecular biology and pathology journals (Campos-Varela, Villaverde-Castañeda, and Ruano-Raviña 2020; Conroy 2019). It is possible, of course, that higher retraction rates for high JIF journals are a consequence of the presumed greater scrutiny papers in high JIF journals are subjected to. Authors are to be congratulated for retracting unsound papers but the growing number of incidences is a concern, which increases the burden on editors and reviewers to ensure the highest standards of reviewing and assessment. The availability of electronic methods for detecting plagiarism and dodgy data presentation can now assist the reviewing process although plagiarism remains a key contributor (Bik, Casadevall, and Fang 2016; Conroy 2019).

In 2013 the San Francisco Declaration on Research Assessment (DORA) San Francisco Declaration on Research Assessment 2013) essentially rejected the use of JIF as a measure of scientific excellence and made the general recommendation:

“Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”.

Subsequently, several organizations and institutions have endorsed the recommendations of DORA (Schmid 2017). For instance, both the Wellcome Trust (2020) and Research Councils UK, now part of UK Research and Innovation (Research Councils UK. 2018) for England, state that JIF should not be used for the evaluation of funding, appointments, and promotion decisions (Gibney 2013). In addition, a number of granting agencies and scientific organizations are following this lead, including the Australian Research Council (2020), Canadian Institutes of Health Research (2020), and European Molecular Biology Organization (2016) – see Larivière et al. (2016). Furthermore, the American Society of Microbiology (ASM) made the following announcement: “The editors-in-chief of ASM journals and ASM leadership have decided to no longer advertise the impact factors of ASM journals on the journals’ websites. This decision was made in order to avoid contributing to a distorted value system that inappropriately emphasizes high IFs. High-IF journals limit the number of accepted articles to create a perception of exclusivity, and individuals receive disproportionate rewards for articles in high IF journals, while science as a whole suffers from a distorted values system and delayed communication of research” (Casadevall et al. 2016).

Such views provide an opportunity to add a quote attributed in various sources to both the Librarian of the US Congress, Daniel J Boorstin, and the physicist, Stephen Hawking:

‘The greatest enemy of knowledge is not ignorance, but rather the illusion of knowledge.”

Unfortunately, despite DORA and a substantial literature that documents the limitations of JIF and the removal of the mention of impact factors from a number of journal websites, IF and JIF are still being used to not only rank journals but also to evaluate authors and their contributions. Publishing in high impact journals such as Nature and Science has been described as joining “The Golden Club” and the pathway to a successful career (Reich 2013). Indeed, maintaining the perceived importance of JIF may even be preferred by some scientists (Abbott et al. 2010; Callaway 2016; Verma 2015). A survey examining the factors that influence the choice of publication venue among researchers in Canadian and US universities (Niles et al. 2020) found that respondents’ listed intended readership, overall prestige, and whether their peers read the journal as their top choices. In contrast, they considered that their peers were more concerned with overall prestige, JIF, and readership. Niles et al. (2020) also found that those in research-intensive institutions placed higher importance on JIF than those in more teaching-focused institutions. They suggest that “any shift away from JIF, journal names or citation measures may be challenged not by faculty’s own values, but by the perception they have of their peer’s publication decisions.” This may partly explain why, despite the many criticisms of JIF and recommendations to discontinue their use for assessment, they are still widely used.

A report in 2019 provided an analysis of criteria used for the review of promotion and tenure collected from academic units within Canadian and United States universities and concluded that JIF still featured prominently in the evaluation process, albeit perhaps less than in the past (McKiernan et al. 2019). How many of us have received the review comment back from a grant competition: “Does not publish in high impact journals”? How many career advancements have been halted by similar negative comments? That said, “How will you judge me if not by impact factor?” was the title of a viewpoint in Nature by John Tregoning (Tregoning 2018). The problem with JIF, as pointed out by Tregoning, is that as a metric “it is so easy” to use.

Interestingly, however, the 2013 Nobel Prize winner for Physiology or Medicine and also Editor-in-Chief of eLife, Randy Schekman, has referred to journals such as Cell, Nature and Science as “luxury journals” (GlamMagz) and indicated that he would no longer submit papers to such journals (Schekman 2013; Schekman and Patterson 2013).

5. Alternatives to JIF?

“Nothing is more dangerous than an idea, when you have only one idea” (“Alain” -Emile-Auguste Chartier).

Interest in JIF and its relevance to current scientific publishing is reflected in the growth in the number of publications discussing the subject since the early 2000s: recent years have seen in excess of 500 publication per year appearing in Scopus, although interest may be leveling off (Figure 1). While publications about JIF only ever constitute a small proportion of all publications found in Scopus, the topic has attracted more attention in some fields than others. Ranking Scopus subject areas by the percentage of publications that mention “journal impact factor” (Figure 2) indicates the Medical and Life Sciences subject areas include a greater proportion of JIF-related publications than is found on average for all Subject Areas combined. In contrast, most subject areas in the Social Sciences and Physical Sciences include lower percentages of publications relating to JIF.

Figure 1. Number of Scopus records (1983–2019) that include the phrase “journal impact factor” in the title, abstract, or keywords.

Figure 2. Percentage of records in each Scopus Subject Area that include the phrase “journal impact factor” in the title, abstract, or keywords. Subject Areas are grouped as follows by Scopus: Health Sciences (red); Life Sciences (green); Physical Sciences (orange); Social Sciences (purple). The percentage for all Subject Areas combined is shown in blue.

Is there an alternative to JIF? Is there really a quick fix? Ignoring the dangers of simply repeating what others have tried, we will attempt to provide a path forward, while being mindful of the utility and applicability of the Bellman’s blank map in Lewis Carroll’s “The Hunting of the Snark” (Carroll 1876): are we being too brave to assume we can find the elusive answer when we may also lose our way?

Other maps are such shapes, with their islands and capes!

But we’ve got our brave Captain to thank

(So the crew would protest) “that he’s bought us the best—

A perfect and absolute blank!.

A number of alternative metrics to JIF have been developed (Table 1). All of these are based on citation counts for individual papers but vary in how the numbers are used to assess impact. As discussed later, the accuracy of data based on citation counts is highly questionable. CiteScore calculates a citations/published items score conceptually similar to JIF but using Scopus data to count four years of citations and four years of published items. The Source Normalized Impact Factor also uses Scopus data to take a citation/published items score and normalizes it against the average number of citations/citing document. The Eigenfactor (EF) and Scimago Journal Rank work in a manner analogous to Google’s PageRank algorithm, employing iterative calculations with data from Journal Citation Reports and Scopus respectively to derive scores based on the weighted valuations of citing documents. Finally, h-indexes attempt to balance the number of papers published by an author or journal against the distribution of citation counts for those papers. This metric is frequently used and is discussed in more detail in a following section.

Table 1. Journal impact factor and (some of) its alternatives

Metric	Definition	Detail	Citation source(s)
Journal Impact Factor (JIF)	“The average number of times articles from a journal published in the past two years have been cited in the Journal Citations Report (JCR) year.”^1	JIF 2019 = Number of citations in 2019 to Articles published in 2017 + 2018/Number of articles published in 2017 + 2018.	Journal Citations Reports
CiteScore	“The number of citations to documents (articles, reviews, conference papers, book chapters, and data papers) by a journal over four years, divided by the number of the same document types indexed in Scopus and published in those same four year”2	CiteScore 2019 = Number of citations in 2016–2019/Number of articles published in 2016–2019.	Scopus
Source Normalized Impact per Paper (SNIP)	“The ratio of a source’s average citation count per paper and the citation potential of its subject field. The citation potential of a source’s subject field is the average number of references per document citing that source.”3	SNIP 2019 = (Number of citations in 2019 to articles published in 2016–2018/Number of articles published in 2016–2018)/Average number of references per citing document.	Scopus
Eigenfactor	A measure of a journal’s influence in a citation network, based on citations to that journal, iteratively weighted by the importance of the citing journals.4	Uses citations from 2019 to articles from 2014–2018, excluding self-citations.	Journal Citations Reports
ScImago Journal Rank (SJR)	A modified Eigenfactor.4	Uses citations from 2019 to articles published in 2016–2018, limiting self-citations to no more than one third of all citations.	Scopus
h-index	For a given body of articles, h is the largest value for which at least h articles have received at least h citations.^a	Usually applied to authors or journals.	Web of Science; Scopus; Google Scholar
h5-index	A modified h-index.^b	Limited to articles published in the last five calendar years.	Google Scholar

1 https://incites.help.clarivate.com/Content/Indicators-Handbook/ih-journal-impact-factor.htm

2 https://service.elsevier.com/app/answers/detail/a_id/14880/supporthub/scopus/

3 https://service.elsevier.com/app/answers/list/c/10547/supporthub/scopus/

4 https://scholarlykitchen.sspnet.org/2017/05/15/citation-performance-indicators-short-introduction/

^a(Hirsch 2005)

^bhttps://scholar.google.com/intl/en/scholar/metrics.html#metrics

6. Open access and downloads

Since an increasing number of journals and publications are now available via “open access” perhaps the number of times the article is downloaded could be used as an “Index of Interest”? Unfortunately, however, a download may reflect interest or availability of the subject matter in the title but not necessarily the quality or impact of the science. Furthermore, not all journals, including some high JIF journals, provide free access and thus “downloads” as a metric will be biased. Thus, for 2019, 312 journals were listed as having a JIF >10 but only 26 were listed as completely OA. Thus, because of the limitations of access, using the number of downloads as a statistic of even interest, let alone quality, is highly questionable.

Furthermore, OA for papers in many journals can be very expensive and, as previously mentioned, commencing January 2021, the price to be paid for publishing an OA paper in Nature has been set at €9500 and lower, €4790, for other journals in the Nature group such as Nature Genetics (Brainard 2020). Such high price tags will clearly limit access to those who have sizable research or institutional budgets and bias against those with limited resources. This carries the potential disadvantage that top journals may only make scientists aware of the work of the rich, rather than the best output from the entire scientific community.

7. Why not measure impact based on citations per author?

Based on how JIF is calculated, it has been argued that a better descriptor is “CAPCI factor”, standing for Citation Average Per Citable Item (Diamandis 2017). The argument is that removing the word “impact” and adopting CAPCI as the metric de-emphasizes the false interpretation that every paper published in a journal has the same “impact”, but would it? The answer is “no” because of the variability inherent in determining citation scores. Although assessing citation metrics became comparatively easy in 2002 when Thomson Reuters made the Web of Science database available and access became even easier when Elsevier launched Scopus and Google Scholar in 2004, the number of citations per given publication may vary quite dramatically dependent on which database you search. An additional problem is what should be considered a citation. For instance, should citations appearing in Wikipedia be included? Are we simply chasing numbers without understanding the meaning of the numbers? Essentially, the answer to these questions is that using citation numbers alone as the primary assessment tool is not going to provide an accurate measure of scholarly ranking. The literature concerning the use of bibliometric indicators has been extensively reviewed and the reader can reference the following for more detailed comparisons and discussion than provided in this article: Rowlands (2018); Waltman (2016); Wildgaard (2015).

So, what are some of the key problems? As pointed out by Price (1976) citations may simply reflect popularity:

Success seems to breed success. A paper which has been cited many times is more likely to be cited again than one which has been little cited. An author of many papers is more likely to publish again than one who has been less prolific. A journal which has been frequently consulted for some purpose is more likely to be turned to again than one of previously infrequent use.

In most instances the use of the number of citations as a metric will favor the more senior investigators, as they have simply had longer to publish more work and accumulate more citations (the h-index, discussed below, seeks to take advantage of this accumulation when comparing authors). In addition, the question of seniority in determining the order of authorship as well as the contribution of individual authors to the publications are factors that can greatly complicate the interpretation of the true role of an individual author in a collaborative project (Macfarlane 2017). This is particularly problematic for multi-authored manuscripts as it is likely that some authors may only be familiar with one aspect of the publication and, although most journals provide guidelines on how to credit individual authors, the accuracy of the statements is not subject to review (Eggert 2011) and there is no agreed methodology for assigning sub-scores to individual coauthors. In a 2015 publication 5,154 authors shared ownership on a paper that appeared in Physical Review Letters. Listing all of their names occupied 24 of the 33 pages of the article. Such an example raises questions as to how rebuttals to reviews and revisions are addressed in a timely manner when the input from so many authors would, or should, be collected and addressed (Aad et al. 2015; Castelvecchi 2015). The National Information Standards Organization (NISO) introduced CrediT (http://crdit.niso.org/) that provides a taxonomy template for clarification of the contribution of each author and if unversally accepted this could valuable information. In China, the authorship issue is simplified by giving the major credit for authorship to the first author and the corresponding author(s) only. Perhaps as a consequence, however, joint corresponding authorships and first authorships are now becoming common.

Self-citations are another problem that may result in inflation of citation statistics for an individual author. Whereas it may be appropriate to cite your own work, it should not be done at the expense of more appropriate citations and/or prior publications pertaining to the original finding. When there are clear incentives in using such metrics, cheating always remains a risk (Biagioli 2016). Reports from Italy (Baccini, De Nicolao, and Petrovich 2019; Baccini, Petrovich, and De Nicolao 2019) emphasized this problem and related it to the emergence in 2011 of the use of bibliometric indicators for the national research assessment exercise in Italy. This, unfortunately, did not exclude self-citations, hence promoting further examples of “citation gaming”, or “gaming the metrics”, and a reduction in the level of international research collaborations. Interestingly, in their conclusions the authors state: “-our results show that the mere presence of bibliometric indicators in the evaluative procedures is enough to structurally affect the behaviour of the scientists, fostering opportunistic strategies”(Baccini, De Nicolao, and Petrovich 2019). Another study, which was based on a data set of approximately 100,000 researchers, reported that although the median self-citation rate was 12.7%, at least 250 scientists amassed >50% of their citations from themselves or their coauthors (Ioannidis et al. 2019; Van Noorden and Chawla 2019). These examples support the conclusion, discussed in section 3, how the process of evaluation can distort and devalue mesurement.

Citations can, nevertheless, provide a useful guide when evaluated from the standpoint of who is citing whom. Thus, if prominent scientists in the field are citing a publication then perhaps that does provide a notable measure of impact. Evaluating citation practice can also provide useful information, especially in identifying unbiased reviewers for grants (Wallace, Larivière, and Gingras 2012). The policy of some journals, however, of restricting the number of references authors may cite encourages the use of review citations, which can have the unfortunate effect of reducing the visibility of often seminal original papers. While review articles can be very valuable, especially if they contain critical analysis of work in a field, publishing more reviews can also be seen as a strategy for journals to increase citations and their JIF by publishing an even greater percentage of review articles.

8. Simply counting citations can be mis-leading

As already discussed, it is very easy to assess and compare the number of citations credited to individuals in the same discipline but what does that really mean? Whilst it is possible to make qualitative comparisons there are several confounders that suggest that counting citations alone can result in unfair comparisons. Thus, reasons for citations could include:

• An important suggestion

• An important discovery

• A critical discovery that laid the foundation for a whole field

• A useful method

• An incorrect assumption or conclusion

• A widely criticized paper

• My contribution to the field

• My colleague’s contribution to the field.

Clearly, a paper that details a major breakthrough in a field of science would be expected to have a higher impact than a paper that describes a useful methodology that is widely used, but the citations for these papers may be quite similar. Similarly, a paper that is cited frequently, gains greater visibility by virtue of such frequent citations, and therefore is also likely to be cited by others. Innovative ways of providing more information relating to citations, such as citation typing ontology (http://purl.org/spar/cito) have been available for a number of years but, as found for a number of other technologies, they have not been adopted by publishers.

Furthermore, some institutions, journals, and authors use a number of approaches to inflate their status by encouraging or promoting the use of citations from specific research areas or publications in the same journal, generating what has been referred to as a “Citation Cartel” (Poppema 2020). An additional problem with citations is the effect of the “Elite Researcher” who dominates the ranking table. This problem has been clearly detailed by Reardon (2021), based on an analysis of 26 million papers published during 2000–2015, which found that the top 1% most cited authors accounted for a disproportionate 21% of the total number of citations, with the greatest disparity in the fields of astronomy and physics.

9. The Hirsch h-index

Despite reservations that relying solely on citations can be misleading, the widely used h-index is based on the number of citations accredited to an individual. Proposed by Jorge Hirsch in 2005 (Hirsch 2005), the h-index, which can be obtained from Google Scholar, ignores JIF and simply provides a metric based on how many publications an individual has and how many times (s)he has been cited (see Table 1 for a strict definition). Thus, an individual with an h-index of 50 has 50 published papers that have been cited 50 or more times. In most instances the h-index benefits the more senior investigator, as it is expected to increase over time for a given researcher as citations accumulate. The h-index is also very much discipline-dependent, making comparisons between different subject areas both very difficult and potentially unfair. Interestingly, it was not Hirsch’s intention that the h-index would be used outside of his field of theoretical physics (Conroy 2020; Hirsch 2020). The h-index can be subject to the same misinterpretations discussed above as to the role and contribution of each author in each paper, the issues of self-citations, and citing of former coauthors. Furthermore, the role of collaborators and other inconsistencies also need to be addressed (Waltman and Van Eck 2012).

As noted by Conroy (2020), Hirsch himself has been quoted as saying: “One has to look at the nature of the work,” says Hirsch. “If you make decisions just based on someone’s H-index, you can end up hiring the wrong person or denying a grant to someone who is much more likely to do something important. It has to be used carefully.” Having a high h-index can also be deployed as bragging rights, “My h-index is bigger than yours”, but the significance of the number varies from field to field, emphasizing the need for caution in using this metric alone for assessment purposes. Thus, these concerns indicate that the sole use of the h-index is inappropriate as a tool to determine eligibility for appointment, career advancement and funding, despite the criteria that have been indicated by certain national funding agencies.

10. Beyond Hirsch?

From the preceding discussion we can conclude that citations alone do not necessarily reflect the quality of a scientific publication, although arguably they may be superior to JIF. It is not therefore surprising that an extensive literature has been generated within the field of bibliometrics that has examined ways in which citation scores can be normalized to provide an indicator that fairly compares impact across different fields of sciences. Ideally a metric is needed that identifies the role of each author in the publication, i.e. first or last author versus middle author(s) and also takes into account not only differences within the fields of science (i.e. physical versus biomedical sciences) but also the impact of a publication within that scientific field. Many authors have proposed alternative methods for normalizing and quantifying contributions to the impact of scientific publications.

11. Normalizing the citation score – The crown indicator

The concept of analyzing bibliometric data as a measure of impact and generating a “Crown Indicator” was first introduced by the University of Leiden in The Netherlands as result of a change from the allocation of resources based on student numbers to an allocation system based on scientific quality (Moed et al. 1985; Waltman et al. 2011). The Leiden approach attempts to enable comparisons across fields by normalizing the citations of any published paper to the mean citation rate for a paper in the same field for that year. Thus, the citation count for the paper is divided by the mean citation rate of other papers of the same type (i.e. research paper, letter etc.) published in the same field (this being a measure of “expected citations”) for that year, and this provides the “Crown Indicator”, or Mean Normalized Citation rate (MNCS). Variations and percentile scores can also be generated to reflect whether the paper under consideration belongs to a % range of the most cited papers in the field (i.e. top 1%, 10%, 25%, etc.) Arguments that the Leiden approach biases against fields where citation rates are low have been made and counter-argued (Opthof and Leydesdorff 2010; Smolinsky 2016; Van Raan et al. 2010). Other approaches, including the use of post-publication review data from InCites and F1000 that provide peer ratings of published papers as a measure of quality, have also been proposed (Bornmann and Leydesdorff 2013).

12. Can citation ratio bibliometrics provide the answer?

The Office for Portfolio Analysis at the NIH developed a Relative Citation Ratio (RCR) and demonstrated its utility for determining whether NIH awardees maintain high or low levels of influence in their individual fields of research (Hutchins et al. 2016). Their RCR is based on determining the Article Citation Rate (ACR) and the Expected Citation Rate (ECR). The ACR is determined by the total citations divided by the number of years but excluding the calendar year of publication. The ECR is determined by the number of citations from a representative comparison, benchmark group. The RCR is then determined by the ACR/ECR ratio. In addition, recognizing differences between fields and the most likely journals where the publications occur, a Field Citation Rate (FCR) is determined from the average Journal Citation Rate (JCR = JIF) and then the ACR/FCR provides a field-independent metric that can then be compared to the ACR/FCR ratio for the co-citation network. The RCR gained interest from a number of granting agencies outside of the USA, including the Wellcome Trust in the UK and the Fondazione Telethon charity in Italy (Naik 2016), but has also been criticized as “not better than other indicators” by Bornmann of Germany’s Max Planck Society. In addition, concerns over a seemingly complicated normalization process, definition of the field of research, and derivation of the citation rate, the JCR, from a 2-year period have been expressed (Bornmann and Haunschild 2017; Janssens et al. 2017).

If one assumes that a paper that is never cited has no value (an argument that we do not necessarily agree with – see also (Larivière, Gingras, and Archambault 2009; Van Noorden 2017) then using the citation ratio analysis could be considered as a significant metric. However, as discussed above, if a paper has been cited once it is also more likely to be cited again. This raises the question as to what extent the failure to cite a published paper represents laziness on the part of the (non-) citing author and an over reliance on citing papers that have previously been cited in the authors literature database? As pointed out by Patience et al. (2017) the number of times a paper is cited does not necessarily correlate with quality and originality and varies between disciplines. Furthermore, retracted papers may also be included in citation scores. Nicolaisen and Frandsen (2019) assessed the “uncitedness ratios” of items in seven subject areas in Scopus. Overall, rates of uncitedness ranged from 0.19 (i.e. 19%) in Physics and Astronomy to 0.38 in Arts and humanities. There were also variations according to item type, e.g. articles versus reviews, in each subject. It is far from clear that uncited or seldom-cited papers have no value: Van Noorden (2017) describes a paper by Egger et al. (2010) that had never been cited but had been viewed over 1500 times and downloaded nearly 500 times. By January 2021, this paper had still only been cited 3 times (by PLoS One’s count) but had received over 2,850 views and been downloaded 648 times. Clearly, this paper is valuable but its value is not reflected in its citation count.

13. Conclusions

Is there a simple solution?

As we have described, there are many arguments as to why JIF is unsatisfactory as an authoritative measure of research quality and impact, yet many alternatives, such as the RCR, also have been criticized and may not be applicable or acceptable to all disciplines. Could this be a “catch-22 moment” when there is no good, or simple solution? If so, then to exit a “catch-22” situation we have to change the status quo. If we assume that the arguments presented against the use of JIF, number of citations, the h-index, or variations thereof for the purpose of assessment are convincing, then we need to offer an alternative that is both transparent and easy to use. Is there a simple easy-to-use process that is fair, non-biased, and recognizes an individual’s contribution to their discipline? It has been argued that the Relative Citation Ratio (RCR) proposed by Hutchins et al. (2016) offers an improvement and comes closest as in applying “field normalization” it provides a measure of where an author ranks relative to others in the same field of research (Surkis and Spore 2018). This reduces the difficulties in comparing productivity and impact between different disciplines, but it lacks the simplicity of using JIFs as the sole metric and significant criticisms have been raised as previously discussed. Thus, to exit the catch-22 trap it is necessary to completely eliminate the use of the flawed JIF in terms of evaluation, or, at the very least, add several other measures to be included alongside JIF, in order to provide a more effective and useful evaluation. Furthermore, we have to promote OA, but it has to be affordable. Can this all be done? If so, and recognizing that any evaluation based on a single criterion alone can be criticized, what are the criteria we should consider in order to devise a more effective system for recognition and assessment of accomplishments which also supports an equitable publishing process that is not hidden behind expensive paywalls and OA fees? The following are all metrics that could be collectively looked at to aid in assessment although as we have discussed, if used alone, all have their limitations:

1. Contribution of an author to the paper including preprints, i.e. first author, last author, conducted experiments, analyzed data, contributed to the writing, other?

2. Number of years active in research field and productivity

3. Number of publications in journals where others in the same field also publish

4. Views and downloads

5. Number of citations as first, last, or corresponding author.

Considering the wider publishing landscape, Brembs (2019) has argued for replacing the legacy journals completely, with peer-review and publishing infrastructure governed by scholars themselves. While some will argue that such a scenario is still a long way off, however, as a first step it should be possible to build on the success of pre-print servers and provide a new model where more OA journals emerge that operate under the same principles of preprint servers with an open review process conducted on-line and at minimal cost. Via this route there would be no reason to publish in traditional journals. Recommendations as to how a change could be implemented have been detailed (Bilder, Lin, and Neylon 2015). Such an initiative certainly has its attractions; however, many questions remain such as how a new publishing model would be managed and by whom, and how it would be sustained. Also, how rapidly and how universally would such changes be accepted and granting agencies, universities and industry adjust their practices and their methods of assessment? As noted by Niles et al. (2020) resistance from some authors can also be expected. Nevertheless, it is definitely time for a change and, hopefully in 2021, 57 years after the release of Bob Dylan’s “The Times They Are a Changin’”, there will be changes that facilitate a fully OA and online affordable publication platform as suggested above.

Contribution of authors

CRT conceived of and initiated the first draft and finalized revisions.

RM: Conducted literature research on scientometrics and prepared figures and table

DJT: Contributed to the research and writing and revisions.

DG: Contributed to the finalization of the first draft and initiated revisions and preparation of revised manuscript following peer preview.

Acknowledgments

Weill Cornell Medicine - Qatar, Medical Education Open Access Fund #87061606

Any opinions expressed in this document are those of the authors and do not necessarily reflect the views of the organizations and institutions they are affiliated to. We gratefully acknowledge the reviewers of the manuscript for their valuable and constructive input.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

No funding source was associated with supporting this publication.