Expert bias in peer review

Keywords::

• Bias
• Conflicts of interest
• Editorial
• Expert
• Peer review

Introduction

Peer review before publication is a pivotal process that can facilitate the selection of appropriate manuscripts for publication in scientific journals. This system may involve a number of procedures that serve to judge the suitability of the material being submitted in relation to the remit of the journal, as well as a process by which the scientific content undergoes critical appraisal by appropriately qualified experts in that field. Editors and scientists alike agree that scientific scrutiny is key; however, the system is not perfect and accusations have been made that manuscripts may be assessed more on the basis of acceptability rather than validity1.

The concept of publication bias has been well described, particularly with respect to the reporting of a significant outcome, leading to the term ‘bias against the null hypotheses’2. However, a bias that arises from an expert’s predilection for one opinion rather than another has received far less attention. This editorial will attempt to shine a light on expert bias in peer review and offer solutions for overcoming the problem.

Khun’s ‘paradigm shift’

In the 1960s Thomas Kuhn described how scientists favour working within a framework of their own preconceived ideas3. He outlined how over time evidence contrary to these ideas is dismissed until a time comes when overwhelming evidence in support of new ideas can no longer be ignored. Kuhn described this critical point in time as a ‘paradigm shift’. These paradigm shifts often involve the development of radically new ways of seeing a problem and are essential in advancing our understanding of science. Many current mainstream theories, such as relativity, quantum theory, and the theory of plate tectonics are examples of paradigm shifts that evolved after lengthy periods of time where they were dismissed in the face of evolving evidence.

In the 21st century there remains much to be discovered and thinking ‘outside the box’ is what is required to avoid a prolonged period of scientific stagnancy. To address the balance scientific journals have created sections in their journals for ‘hypothesis articles’, where articles describing a substantial jump in current thinking can be published. These types of articles have the potential to influence a unique perspective and apart from promoting debate, if well written, may inspire these very ‘paradigm shifts’ that are so important in pushing forward the frontiers of science.

As a ‘scientist’ having worked in the UK and abroad I have encountered opposition toward proposing ideas that do not harmonise with traditional dogma. Having discussed this informally with many prominent colleagues in my personal fields of interest, I have sadly discovered that my experiences are far from unique. Furthermore, examples to support this behaviour have been documented in the literature4. The history of science as a whole is fraught with stories documenting how the publication and dissemination of important discoveries were resisted or ignored by the contemporary scientific community of the time5. Campanario details the instances in which 19 future Nobel Laureates encountered resistance on the part of the scientific community towards their discoveries, and instances in which 24 future Nobel Laureates encountered resistance on the part of scientific journal editors or referees to manuscripts that dealt with discoveries that later would earn them the Nobel Prize6. In response to these claims the journal Nature admits that “… there are unarguable faux pas in our history. These include the rejection of Cerenkov radiation, Hideki Yukawa’s meson, work on photosynthesis by Johann Deisenhofer, Robert Huber and Hartmut Michel, and the initial rejection (but eventual acceptance) of Stephen Hawking’s black-hole radiation’7. Other more fundamental examples are also worthy of note. In the 1500s Nicolaus Copernicus demonstrated a great deal of courage when he proposed that the Earth orbited the Sun rather than the other way around8. When Roentgen announced his discovery of X-rays, Lord Kelvin denounced his discovery to be an elaborate hoax9. In my own field of Otolaryngology, when John Epley proposed his manoeuvre for Benign Paroxysmal Positional Vertigo he faced a lengthy battle against his adversarial colleagues that at one point resulted in him being notified by the Oregon Board of Medical Examiners that he was under investigation for alleged unprofessional conduct10.

What is expert bias?

From a peer review perspective, conflict of interest may be considered to be an author only bias. However, in the modern age of publication and dissemination, journal editors and reviewers play a role in potentially introducing a bias of their own. Larger, broad interest journals rely on ‘experts’ to assess manuscripts; however, these experts may be biased for a variety of reasons that may prevent them from accurately and objectively reviewing these manuscripts. The current system of peer review lends itself to a situation whereby reviewers may bias their appraisal if the manuscript contents contradict either their own or mainstream thinking6,11. This form of bias has been substantiated experimentally by cognitive psychologists12. In a study by Mahoney, 75 journal reviewers were asked to review manuscripts that described identical introduction sections and experimental procedures but the data reported were either consistent or inconsistent with the reviewer’s presumed theoretical perspective. Mahoney’s study demonstrated that reviewers were “strongly biased against manuscripts that reported results contrary to their theoretical perspective.” This form of behaviour has been labelled as ‘confirmatory bias’ and has been defined as an erroneous cognitive process whereby there is a “tendency of human behaviour to seek out, attend to, and sometimes embellish experiences that support or confirm their beliefs”, whereas “disconfirmatory experiences are often ignored, discredited, or treated with obvious defensiveness”12. This type of undesirable behaviour has been identified in research on the decisions from peer reviewers by considering the influence of reviewer identity on the acceptance rate for submitted manuscripts when authors were given the opportunity to suggest or exclude particular reviewers from reviewing their manuscripts13. It has been reported that nominating reviewers to be excluded from the peer review process increased the odds of acceptance almost two-fold, whereas nominating reviewers to review a manuscript did not have a significant effect on manuscript acceptance13. A number of international committees exist to promote good practice in the publication of medical journals; these include the Committee on Publication Ethics (COPE), the Council of Science Editors (CSE), the International Society for Medical Publication Professionals (ISMPP), the International Committee of Medical Journal Editors (ICMJE) and the World Association of Medical Editors (WAME). Wager et al.’s international survey of science journal editors reported that many editors were unfamiliar with available guidelines but would welcome more guidance and training14. However, all codes of practice amongst medical journal editors are entirely voluntary15 and some have suggested that these codes of practice are ineffective and lack sufficient endorsement outside the largest medical journals16.

How does expert bias arise?

There may be a number of conscious, as well as subconscious, explanations for why expert bias takes place. In 1620, Sir Francis Bacon wrote that “The human understanding when it has once adopted an opinion draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusion may remain inviolate”17. A number of motives may act in promoting expert bias. Studies have explored how the evaluation of evidence is highly biased by existing beliefs and prior theories18. The biased appraisal of submitted research may occur due to a variety of ‘private interests’ as well as financial motives, these may include professional affinities and rivalries, nepotism, scientific or technological competition, or religious beliefs19. To substantiate this claim, it has been reported by COPE that many cases of undeclared and problematic competing interests go beyond financial motives19. Notably, one study in the field of psychotherapy has demonstrated how allegiance to a given ideology exerted a bias on study design and outcomes comparable to that which has been documented for financial interests20.

When proponents of a belief are strongly committed, contrary arguments that challenge these firm beliefs can occasionally cause proponents to become even more convinced that they are correct. This phenomenon is known as the ‘boomerang effect’21. Scientific opinion regarding certain matters may become even more polarized where a number of specialists overlap in expertise. This is particularly significant in my field of otolaryngology where neurotological conditions are dealt with by both otolaryngologists and neurologists. For instance, neurologists may favour neurological aetiologies for conditions that cause vertigo, whereas otolaryngologists would favour aetiologies arising from the inner ear. In such cases a reviewer’s standpoint would need to be taken into consideration as this may significantly bias their review. Examples of interdisciplinary prejudice have been reported in other fields4.

Experts arise due to experience in their chosen field and are often involved in a number of ongoing research projects. Their livelihoods and reputations depend on this work, not to mention, in some cases, the funding of their labs. With this in mind, providing credence to manuscripts that support an alternative theory, treatment or association may divert precious funding to work based in other disciplines and consequentially receive a biased review. This may be particularly pertinent in the realm of drug trials where advocating the efficacy of a rival drug may be devastating to the future development of the reviewer’s currently studied agent; however, biased appraisal may not be limited to the drugs industry. Osmond quotes Lewis Carroll’s Alice’s Adventures in Wonderland, where he humorously compares Alice’s experiences with the challenges involved with addressing the peer review process for research funding11. He aptly remarks that “In any valid contest the judges are on the sidelines – not in the race itself”. In the current global economic climate funding budgets may very well shrink for many researchers and the role of the expert reviewer will become more critical in shaping the future of who, where and how research is carried out.

Experts may bias collectively as well as biasing as individuals, with many experts practicing as members of professional advisory committees. Groups are often composed of like-minded people, and members are often invited to join on this basis. Whereas one would expect that each member of the group would drift to a mid-position, it has been demonstrated in psychological experiments that if a bias exists in one direction, simple group interactions result in greater bias in the same direction22. It has been proposed that this is because group members may try to obtain approval of their co-members by expressing extreme attitudes in the favoured direction23. Furthermore, when compiling professional committees it is unlikely that prospective members who express opposing views will be selected23. Professional advisory committees are often called upon to provide recommendations for the distribution of funds for large scientific research projects. There is opportunity for a ‘herd mentality’ to be adopted whereby the opinions of the majority are represented in order to maintain stability within the group. This would favour the funding of pedestrian research rather than the more unconventional revolutionary research performed in the past that have resulted in the paradigm shifts of Kuhn.

Is expert bias a common problem?

It is very difficult to quantify the extent of this problem for two possible reasons; firstly due to a possible lack of realization that a problem exists on the part of journal editorial teams, and secondly due to possible barriers placed to prevent external review of journal editorial practice.

Bias introduced by reviewers seems to gain less emphasis than it deserves4. A study that considered ethical practice in the accounting publishing process has demonstrated a significant difference between the perceptions of authors and editors on the ethics and frequency of specific author, editor and reviewer practices, with the majority of authors supporting codes of publishing ethics, and editors not in agreement24. An international survey of science journal editors concluded that most editors seemed not to be very concerned about publication ethics and that they believed that misconduct occurred only rarely in their journals14. This finding is particularly interesting when placed in the context of a survey by Geggie that found that 56% of newly appointed UK consultant doctors had witnessed some form of research misconduct25.

Luty et al. have attempted to consider the extent of preferential publication of manuscripts submitted by members of a journal’s editorial team26. Their study was limited as they were denied access to information related to rejected articles26. In response to this Luty et al. challenge journals to demonstrate that they are keeping a fair and orderly house26.

So what’s the solution?

1. It is impossible to completely eliminate bias from the peer review process27. Chalmers suggests that reviewers are not all uniformly conscientious about avoiding biased behaviour and that “the reviewer would not be aware of a deeply rooted bias, by definition”4. However a good initial step would be to raise awareness of this problem through the edification of the scientific community.

2. Although editor membership of organizations such as COPE, CSE and WAME is voluntary, membership should be encouraged and supported by medical journal accreditation in a similar way that products receive a CE mark, or doctors are provided a license to practice.

3. Potential reviewer conflicts of interest should be identified in a similar manner to the way conflicts of interest are declared by authors. Some scientific journals, including CMRO and BioMed Central series journals, seek out reviewer conflicts of interest that may be due to a financial, commercial or other form of conflicting relationship; however, this is not regular practice amongst the publication policies of most scientific journals. Taking matters a step further, a complete declaration of reviewer conflicts of interest could extend to a position whereby, for example, competitors in certain fields of drug research would have to declare that they are striving towards similar goals and proponents of certain aetiologies for certain conditions would have to make these opinions known. Many organizations that represent journal editors have suggested fuller disclosure; however, not all journals are compelled to abide with their various codes of conduct15. The ICMJE encourages the disclosure of a number of non-financial interests, such as personal relationships, academic competition and intellectual passion, however, other interests should be declared to envelop other ‘private interests’ expanding the list of biases to embrace biases of personal, political, academic, ideological or religious persuasion19.

4. The EMBO journal has recently adopted a policy of publishing referees’ reports, author’s responses and editor’s comments alongside accepted manuscripts. The injection of transparency into the peer review process is certainly a step in the right direction; however, it falls short in that it does not release reports on rejected manuscripts. This is a crucial limitation particularly as it has been considered that these are often the most interesting papers28. One solution would be to offer the authors of these articles abstract publication together with an explanation for rejection. Many journals reject far more manuscripts than they accept for publication, so an online source of public transparency may be far more efficient than providing a hefty reject section at the back of a paper journal. Public transparency may also resolve another particularly pernicious by-product of expert review – plagiarism. Any expert reviewer may be susceptible to the desire to use rejected material to form the basis of research proposals in the future. This is facilitated by the great detail that is provided in research manuscripts, although this would be more significant for fund applications than the publication of finished research. Despite being almost impossible to provide a definitive indication of how often this practice takes place, a survey of accounting authors and editors reveal that 20% of the authors believed that reviewers ‘borrowed ideas’ whilst only 5% of the editors believed this to be a frequent occurrence24.

5. To completely remove experts who pose any potential conflict of interest from the review and publication process poses the further risk of restricting the academic pool of potential reviewers but editorial judgement needs to balance the likelihood of a biased unobjective review with a review by fewer experts. The ultimate decision to accept or reject a manuscript on the recommendation of a self confessed biased reviewer lies in the hands of the journal editor who would then consider any disclosure information to ensure that new ideas, or data that conflict with classical thinking, are reviewed with an open mind29. Seeking the opinions from a diverse range of experts where possible may also help.

There are many positive aspects of the current peer review system and a world without an expert’s opinion would be wholly inappropriate. Despite the prevalence of expert bias, it is imperative that the methodologically invalid and scientifically implausible be denied an audience. Reassuringly, it has been evident from history that truth prevails in the face of adversity. Unfortunately, to the detriment of mankind, the dissemination of much essential knowledge has in some cases been substantially delayed. Hopefully, as the process of publication and dissemination evolves, the opportunity for expert bias will diminish without jeopardizing those future breakthroughs in science that are so essential in pushing forward the frontiers of our understanding.

Transparency

Declaration of funding

This paper was not sponsored.

Declaration of financial/other interests

J.S.P. has disclosed that he has no actual or potential financial, personal, political, academic, ideological or religious conflicts of interest to declare.

CMRO peer reviewer #1 has disclosed being a volunteer chairperson of COPE; CMRO peer reviewer #2 has disclosed being a J&J shareholder, and an employee of a medical communications company.