Retraction of peer-reviewed articles, a difficult but crucial choice: our experience from The European Journal of Contraception & Reproductive Health Care

Introduction

During the last decade, the scientific world has been increasingly affected by research integrity issues due to problems with scientific fraud, questionable research practices, and the unreliability of scientific results. This problem has now also reached medical journals resulting in serious implications for people’s health.

Although aware of its existence, the editors of the EJCRHC were not confronted with this issue until July 2021. Then we were surprised by mail from a reader expressing concerns about logistically and biologically implausible findings in several peer-reviewed articles previously published in the EJCRHC. These were mainly RCTs in which the reader reported concerns including unusually large effects of the reported treatments, peculiar dropout rates among participants resulting in perfectly equal study groups, the same data in different research reports, statistical improbabilities, and inexplicable differences in comparable study populations from the same area during the same period.

After an initial reaction of doubt and disbelief, moral considerations did us decide to start a serious investigation into the flagged studies, realising that the results could have a major impact on the reputation of the journal.

January 2022, as it turned out that the Publishing Ethics and Integrity Team within Taylor & Francis and its Editorial Team were involved in the same investigations, we agreed to cooperate to ensure consistency in the approach to these investigations. The relevant data we already collected were shared. Future correspondence with the authors and institutions would be done by the publisher’s team.

Methods

Initially, we had no experience with how to handle this problem. We started with studying the Committee on Publication Ethics (COPE) guidelines on what to do when concerns are raised about data integrity in a published manuscript [1]. Our first action was to ask an independent reviewer for a second opinion. In the meantime, the corresponding author was informed about the concerns raised and asked to respond to the concerns and if possible, to share the protocol, the ethical approval and informed consent of the study, and the original anonymised data of the study participants.

In case of no response or unsatisfactory answers we attempted to contact all other authors. In case of repeated non-response or unsatisfactory answers, we informed all authors and in some cases where appropriate, we contacted the author’s institution requesting an investigation.

During their investigation, the publisher’s team shared any new findings with the editors who assessed and thoroughly discussed these findings before making a decision. In case of a satisfactory explanation, the authors were thanked for their cooperation and the case was closed. Based on the seriousness of the unsatisfactory response in cases where significant concerns remained unresolved, editors and the publisher’s team together decided to whether an “Expression of Concern” or “Retraction” of the article in accordance with the journal and publisher’s editorial policies. The decision resulted in publication of an expression of concern, or a retraction notice in the EJCRHC. A retracted article remained online to maintain the scholarly record but was digitally watermarked on each page as “Retracted”.

Results

From 19 suspected articles published between 2012 and 2021, 18 were retracted for serious integrity issues and one case was closed due to insufficient compelling evidence. Reasons for retraction included concerns about patterns in the data inconsistent with authentic data, serious statistical problems or even impossibilities, lack of approval from ethics committees, ambiguity about written informed consent by participants, and identical studies from the same population with the same condition, the same institute and the same study period but with different basic characteristics.

As is standard in cases like this, authors were asked to share their data for assessment. Request to provide original data was refused by most authors for various reasons, such as that keeping the data (whether digital or non-digital) after 2 years of the research publication was not the policy of their university due to the limited archiving ability. Other authors commented that all participants in the study signed a consent form with the knowledge that their information would be confidential and would not be released to a third party. They judged it as unacceptable to get a new consent from patients to release their data after 5 years of the recruitment and publication. In some instances, authors did not take responsibility for the statistical analysis in their papers or did not understand that they are responsible for all aspects of a submission to the journal, even after publication. In one case the explanation was that the statistician who kept the original data had recently passed away due to COVID. Requests for data or further information from co-authors or authors’ institutions were also generally unsuccessful, but some authors were able to submit their protocols and ethics approval document.

Discussion

More than 2 ½ years after the first alarming mail, we have published 18 retraction notices in our journal from a total of 19 suspected articles. Several reasons are responsible for this relatively long period of investigation. First, an incoming letter from a reader with serious concerns about a published article is shocking news for the editorial team usually leading to feelings of insecurity and disbelief. Initially there will be denial or doubt as the article had been peer-reviewed before publication. But soon we realised that the review process of incoming articles in itself can be insufficient to prevent publication of studies with serious integrity problems, mainly because reviewers generally do not have access to the underlying raw individual participant data and there is some basic trust in the colleagues who work in this important field of women’s health.

Secondly, there might be fear resulting in a wait-and-see attitude. Fear from legal prosecution by the authors is one factor. But also fear for a negative assessment of the journal’s reputation by readers in case the concerns turn out to be correct, which might result in bad publicity and a snowball effect on the submission of new articles.

Other emotions are anger or feelings of incompetence, as checking the concerns means a lot of extra work for which there is generally insufficient time, money or qualified professionals.

Having gone through these emotions, we decided to check the concerns according to the COPE guidelines as we were convinced that scientific investigations and their published results must be reliable. After all, they can be used in systematic reviews and (inter)national guidelines for clinical decision making in the treatment of patients. Moreover, unreliable data could misinform future research, as investments made in new large clinical research projects are usually based on meta-analyses of existing evidence.

We were and are still upset about the unwillingness of some authors to share their original data, as this could have been very helpful in resolving problems regarding the authenticity of the data, statistical problems and many other open questions. We want to emphasise that we have made every effort to make our investigations objective and independent and that its results were seriously discussed by the editors before coming to a conclusion.

We can only speculate about the motives of authors to send in articles with non-authentic data. Well-known pressures on researchers globally are publication output, scientific status, money or dependence on the pharmaceutical industry. We realise that retraction of an article may have a great impact on the reputation of the author(s) and their institutions. But we also realise that not retracting an unreliable article, regardless of whether due to fabrication or poor research methods, may harm our readers, our patients, and the integrity of our journal.

For the future we think that it will be very helpful for peer reviewers if researchers when submitting their manuscript, share the protocol, the ethical approval and informed consent of the study, and the original data of the study participants. We as editors are determined to continue our activities to ensure that research published in our journal for the benefit of women’s health is trustworthy. We hope that editors of other medical journals will join us on this difficult but important venture.

Acknowledgements

The Authors want to thank Julia Gunn for the valuable help in evaluation of the problematic manuscripts and the draft of this Editorial.

Disclosure statement

No potential conflict of interest was reported by the author(s).