Retraction of COVID-19 Pharmacoepidemiology Research Could Have Been Avoided by Effective Use of Reporting Guidelines

Abstract

Introduction

Two recent high-profile publications (and subsequent retractions) of pharmacoepidemiology studies reporting the effectiveness and risk of hydroxychloroquine in COVID-19 patients received international media attention. Transparent and complete reporting of these studies could have provided peer reviewers and editors with sufficient information to question the methods used and the validity of results. Since these studies used routinely collected health data, the guidelines for the REporting of studies Conducted using Observational Routinely collected health Data (RECORD) should have been applied to ensure complete reporting of the research.

Methods

We evaluated the two retracted articles for completeness of reporting using the RECORD for Pharmacoepidemiology (RECORD-PE) checklist, which includes the checklists for the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) and RECORD. We compared the proportion of STROBE, RECORD and RECORD-PE items adequately reported using Chi-squared statistics.

Results

In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%) and 9.5 of 15 RECORD-PE items (63.3%)(χ² = 14.839, P <0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%) (χ² = 11.668, P = 0.003). Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, validation of the coding or algorithms, a description of the underlying database population and the accuracy of data linkage methods.

Discussion

While STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Reporting guidelines should be effectively implemented in order for transparency and completeness of research manuscripts, allowing for adequate evaluation by editors and peer reviewers.

Keywords:

• COVID-19
• journalology
• peer review
• pharmacoepidemiology
• reporting guidelines
• routinely collected health data

Introduction

Routinely collected health data, such as registries, health administrative data, and electronic health record data, are increasingly being used for pharmacoepidemiology, clinical and health services research. As the COVID-19 pandemic rages in the world, we observed with great concern that two studies were published and subsequently retracted by The Lancet¹ and The New England Journal of Medicine.² These two pharmacoepidemiology studies used routinely collected health data in the form of the Surgical Outcomes Collaborative (Surgisphere), an international registry derived from data automatically extracted from electronic health records, health and financial administrative data, and a point-of-care data entry process for procedures. Both studies made a significant impact in COVID-19 research, public health, and health policy spheres, and received international attention from the media, as evidenced by Altmetric scores of 24,615 and 3829, respectively, at the time of this writing. The conduct of research that is subsequently retracted represents “research waste”, as initially defined in The Lancet.³ Publication of these manuscripts resulted in the temporary suspension of important COVID-19 clinical trials of hydroxychloroquine by the World Health Organization and others. More importantly, the publication and subsequent retraction of research erodes the public trust in science. While publication of these studies may have been symptomatic of the known limitations of the peer review and editorial process, lack of transparent and complete reporting may have also contributed to this failure. Both retracted studies would fall under the purview of the guidelines for the REporting of studies Conducted using Observational Routinely collected Data (RECORD),⁴ and more specifically the extension of RECORD for Pharmacoepidemiology (RECORD-PE).⁵ RECORD and RECORD-PE are extensions to the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines for non-interventional research,^6,7 the overall study design for the retracted studies. RECORD and RECORD-PE more directly address observational research using routinely collected health data, expanding on parts of the STROBE checklist to ensure readers, peer-reviewers and editors recognize the strengths and weaknesses of research using health data. We believe that, had these checklists been submitted for assessment by peer reviewers, inherent weaknesses in the manuscripts would have become apparent, potentially resulting in rejection by the journals. Unfortunately, journals and editors still do not routinely ask peer reviewers to use reporting guidelines as part of their assessment.⁸ In fact, the pressures from both the scientific community and journal publishing industry resulted in the strong drive to accept COVID-19 articles early in the pandemic, which may have compounded the problem of inadequate peer review and the need for retraction of published articles.⁹

Most research using routinely collected health data is observational in design. While NEJM endorses CONSORT for reporting of clinical trials, STROBE for non-interventional research is not suggested in the instruction for authors.¹⁰ The Lancet endorses use of STROBE,¹¹ but endorsement is insufficient to ensure the appropriate use of reporting guidelines by authors and peer reviewers. In addition, STROBE does not fully convey all intricacies in the methodology of research using routinely collected health data. We previously demonstrated that the reporting of research using routinely collected health data was gravely deficient, even in journals that endorsed STROBE.¹² Therefore, RECORD and RECORD-PE were created as extensions to STROBE for observational studies involving such data.

Methods

We evaluated both articles in question using the RECORD-PE checklist (Tables 1 and 2), and determined the proportion of adequately reported items on the STROBE, RECORD, and RECORD-PE checklists. We reported the proportion of items on each checklist that were adequately reported, allowing for half-points when an item was partially reported. Chi-square statistic was used to determine differences in the proportion of adequately reported items for each checklist.

Table 1 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)⁵ Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, The Lancet, 2020¹. Red Font Indicates Areas That Were Not Adequately Reported

Item No	STROBE Items	RECORD Items	RECORD-PE Items	Location in Manuscript (Page Number and Description of Reporting)
Title and abstract
1	(a) Indicate the study’s design with a commonly used term in the title or the abstract. (b) Provide in the abstract an informative and balanced summary of what was done and what was found.	1.1: The type of data used should be specified in the title or abstract. When possible, the name of the databases used should be included. 1.2: If applicable, the geographical region and timeframe within which the study took place should be reported in the title or abstract. 1.3: If linkage between databases was conducted for the study, this should be clearly stated in the title or abstract.	–	STROBE ITEMS: (a) Study design not reported in the Title or Abstract. (b) P. 1 - Summary provided in Abstract. RECORD ITEMS: 1.1: P. 1 – Abstract reported use of registry data was used, but not that health administrative data was used. 1.2: P. 1 – Multinational registry data were used and time frame is noted in the Abstract. 1.3: P. 1 – Data linkage is implied by the Abstract, but not explicitly stated.
Introduction
Background rationale
2	Explain the scientific background and rationale for the investigation being reported.	–	–	STROBE ITEM: P. 1-2 – Study rationale and scientific background was explained adequately in the Background and Research in Context sections.
Objectives
3	State specific objectives, including any prespecified hypotheses.	–	–	STROBE ITEM: P. 2 – The final paragraph of the Background section adequately describes the study objectives and prespecified hypotheses.
Methods
Study design
4	Present key elements of study design early in the paper.	–	4.a: Include details of the specific study design (and its features) and report the use of multiple designs if used. 4.b: The use of a diagram(s) is recommended to illustrate key aspects of the study design(s), including exposure, washout, lag and observation periods, and covariate definitions as relevant.	STROBE ITEM: P. 3 – Overall design of the study was described in the Methods, Study Design section, although the study design (historical cohort study) was not named. RECORD-PE ITEMS: 4.a.: P. 3 – Study design details was described. 4.b: P. 3 – Study flow diagram and Study Design section described exposure, lag and exclusion criteria. Washout period not relevant, since medication not used for non-COVID-19 purposes. Since events occurred while hospitalized, length of stay represents observation period. However, admission time was not aligned with start of treatment. How patients who died between COVID-19 diagnosis and receipt of treatment was addressed in the study was unclear, resulting in the risk of immortal time bias.
Setting
5	Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection.	–	–	STROBE ITEM: P. 2–3, appendix p 3 – Setting and locations of data location described.
Participants
6	(a) Cohort study—give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up. Case-control study—give the eligibility criteria, and the sources and methods of case ascertainment and control selection. Give the rationale for the choice of cases and controls. Cross sectional study—give the eligibility criteria, and the sources and methods of selection of participants. (b) Cohort study—for matched studies, give matching criteria and number of exposed and unexposed. Case-control study—for matched studies, give matching criteria and the number of controls per case.	6.1: The methods of study population selection (such as codes or algorithms used to identify participants) should be listed in detail. If this is not possible, an explanation should be provided. 6.2: Any validation studies of the codes or algorithms used to select the population should be referenced. If validation was conducted for this study and not published elsewhere, detailed methods and results should be provided. 6.3: If the study involved linkage of databases, consider use of a flow diagram or other graphical display to demonstrate the data linkage process, including the number of individuals with linked data at each stage.	6.1.a: Describe the study entry criteria and the order in which these criteria were applied to identify the study population. Specify whether only users with a specific indication were included and whether patients were allowed to enter the study population once or if multiple entries were permitted. See explanatory document for guidance related to matched designs.	STROBE ITEMS: (a): P. 3 – Methods, Study Design reported eligibility criteria. P. 2-3 – Methods, Registry Features and Data Acquisition reported methods of participant selection. (b): Matching criteria not applicable. RECORD ITEMS: 6.1: Methods of study population selection (such as codes or algorithms used to identify participants) were not listed. 6.2: While ISO procedures and quality control were reported (P. 2-3), validation studies were not referenced, and success or failure of quality control procedures were not discussed. 6.3: While linkage of databases occurred (including multiple health administrative databases, electronic health record data, and manually-entered data from 671 participating hospitals), no details on the linkage process were provided. RECORD-PE ITEM: 6.1.a: P. 3-4 – Methods, Study Design adequately reported study entry criteria.
Variables
7	Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable.	7.1: A complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers should be provided. If these cannot be reported, an explanation should be provided.	7.1.a: Describe how the drug exposure definition was developed. 7.1.b: Specify the data sources from which drug exposure information for individuals was obtained. 7.1.c: Describe the time window(s) during which an individual is considered exposed to the drug(s). The rationale for selecting a particular time window should be provided. The extent of potential left truncation or left censoring should be specified. 7.1.d: Justify how events are attributed to current, prior, ever, or cumulative drug exposure. 7.1.e: When examining drug dose and risk attribution, describe how current, historical or time on therapy are considered. 7.1.f: Use of any comparator groups should be outlined and justified. 7.1.g: Outline the approach used to handle individuals with more than one relevant drug exposure during the study period.	STROBE ITEM: Outcomes: P.4 – Methods, Outcomes described outcome measures, including clinical definition. Exposures: P. 3-4 – Methods, Study Design and Data Collection described clinical definitions of COVID-19 diagnosis, as well as criteria for drug exposures. Effect modifiers: P. 3-4 – Methods, Data Collection described effect modifiers and other data collected. RECORD ITEM: 7.1: List of codes or algorithms for outcomes, exposures, confounders, and effect modifiers was not provided. RECORD-PE ITEMS: 7.1.a: P. 3-4 – Methods, Study Design and Data Collection describe how exposure to drugs were defined. 7.1.b: Precise data sources for drug exposures were not provided. It was not clear whether these came from electronic health record data, health administrative data, or other database (only “We collected data on ... ” was mentioned). 7.1.c: Methods, Study Design discusses the time window of the study, which was the same time window for drug exposure. 7.1.d: Information on prior or ever exposure to study drugs was not provided. 7.1.e: P. 3 – Methods, Study Design describes the definition of current exposure. Historical or time on therapy considerations were not reported. 7.1.f: P. 3 – Methods, Study Design describes the comparator groups (“four distinct treatment groups”). 7.1.g: Not applicable.
Data sources/measurement
8	For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group.	–	8.a: Describe the healthcare system and mechanisms for generating the drug exposure records. Specify the care setting in which the drug(s) of interest was prescribed.	STROBE ITEM: Data sources and details on measurement were not adequately reported. Comparability of assessment methods for various data sources was not reported. RECORD-PE ITEM: 8.a: Not applicable since many hospitals and countries contributed to the data.
Bias
9	Describe any efforts to address potential sources of bias.	–	–	STROBE ITEM: P. 2-3 - Methods, Registry Features and Data Acquisition section describes quality control efforts used to reduce the risk of bias due to poor data collection. P. 4-5 - Methods, Statistical Analysis describes the use of multivariable regression and propensity score analyses to control for or reduce the effect of confounding factors.
Study size
10	Explain how the study size was arrived at.	–	–	STROBE ITEM: Not applicable, since registry is fixed and all available data were used.
Quantitative variables
11	Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen, and why.	–	–	STROBE ITEM: P. 4-5 - Methods, Statistical Analysis reported on use of quantitative variables.
Statistical methods
12	(a) Describe all statistical methods, including those used to control for confounding. (b) Describe any methods used to examine subgroups and interactions. (c) Explain how missing data were addressed. (d) Cohort study—if applicable, explain how loss to follow-up was addressed. Case-control study—if applicable, explain how matching of cases and controls was addressed. Cross sectional study—if applicable, describe analytical methods taking account of sampling strategy. (e) Describe any sensitivity analyses.	–	12.1.a: Describe the methods used to evaluate whether the assumptions have been met. 12.1.b: Describe and justify the use of multiple designs, design features, or analytical approaches.	STROBE ITEMS: (a): P. 4-6 - Methods, Statistical Analysis. (b): P. 4-6 - Methods, Statistical Analysis reported on subgroup and sensitivity analyses. Tests for collinearity, correlation and interaction of independent variables were not reported. (c): P. 4 - Methods, Statistical Analysis reported “Multiple imputation for missing values was not possible ... ” (d): Ways in which loss to follow-up was addressed were not reported. RECORD-PE ITEMS: 12.1.a: Assumptions were not reported. For example, the study did not address immortal time bias or misclassification bias, or how these were addressed. 12.1.b: P. 4-6 – Methods, Statistical Analysis describes all analytic approaches.
Data access and cleaning methods
12	–	12.1: Authors should describe the extent to which the investigators had access to the database population used to create the study population. 12.2: Authors should provide information on the data cleaning methods used in the study.	–	RECORD ITEMS: 12.1: P. 6 - Methods, Role of the Funding Source: “The corresponding author and co-author ANP had full access to all the data in the study ... ”. 12.2: P. 6 - Methods, Role of the Funding Source reports that two authors had full access to all data, but not what data cleaning methods were used in the study.
Linkage
12	–	12.3: State whether the study included person level, institutional level, or other data linkage across two or more databases. The methods of linkage and methods of linkage quality evaluation should be provided.	–	RECORD ITEM: 12.3: While linkage between data sources occurred (as evidenced by various data sources used), the methods of data linkage and quality evaluation were not reported.
Results
Participants
13	(a) Report the numbers of individuals at each stage of the study (eg, numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed). (b) Give reasons for non-participation at each stage. (c) Consider use of a flow diagram.	13.1: Describe in detail the selection of the individuals included in the study (that is, study population selection) including filtering based on data quality, data availability, and linkage. The selection of included individuals can be described in the text or by means of the study flow diagram.	–	STROBE ITEMS: (a): Figure 1 and Table 1 and Table 2. (b): Figure 1 reports reasons for study exclusion. (c): Figure 1. RECORD ITEMS: 13.1: Filtering based on data quality, data availability and linkage were not reported.
Descriptive data
14	(a) Give characteristics of study participants (eg, demographic, clinical, social) and information on exposures and potential confounders. (b) Indicate the number of participants with missing data for each variable of interest. (c) Cohort study—summarise follow-up time (eg, average and total amount).	–	–	STROBE ITEMS: (a): Table 1 and Table 2 (b): P. 4 - Methods, Statistical Analysis reported “ ... there were no codes to indicate that data were missing; if the patient’s electronic health record did not include information on a clinical characteristic, it was assumed that the characteristic was not present.” (c): Table 1 and Table 2, length of stay variables.
Outcome data
15	Cohort study—report numbers of outcome events or summary measures over time. Case-control study—report numbers in each exposure category, or summary measures of exposure. Cross sectional study—report numbers of outcome events or summary measures.	–	–	STROBE ITEM: Table 1 and Table 2.
Main results
16	(a) Give unadjusted estimates and, if applicable, confounder adjusted estimates and their precision (eg, 95% confidence intervals). Make clear which confounders were adjusted for and why they were included. (b) Report category boundaries when continuous variables are categorised. (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period.	–	–	STROBE ITEMS: (a): P. 6–7 (Results), Figure 1 and Figure 2. Listing of confounders discussed on P. 4 (Methods, Statistical Analysis). (b): Not applicable. (c): P. 7 - Results reported on proportion of outcomes in each group in addition to hazard ratios.
Other analyses
17	Report other analyses done—eg, analyses of subgroups and interactions, and sensitivity analyses.	–	–	STROBE ITEM: P. 7 - Results and appendices reported sensitivity analysis results and tipping point analysis results.
Discussion
Key results
18	Summarise key results with reference to study objectives.	–	–	STROBE ITEM: P. 7 - first paragraph of the Discussion summarizes key results.
Limitations
19	Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias.	19.1: Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported.	19.1.a: Describe the degree to which the chosen database(s) adequately captures the drug exposure(s) of interest.	STROBE ITEM: P. 8-9 - Discussion reports study limitations. RECORD ITEM: 19.1: P.8-9 - Discussion mentions unmeasured confounding, but does not report limitations due to missing data, changing eligibility over time or misclassification bias. RECORD-PE ITEM: 19.1.a: Not reported.
Interpretation
20	Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence.	–	20.a: Discuss the potential for confounding by indication, contraindication or disease severity or selection bias (healthy adherer/sick stopper) as alternative explanations for the study findings when relevant.	STROBE ITEM: P. 9 - Discussion gives overall interpretation. RECORD-PE ITEM: 20.a. Not reported.
Generalisability
21	Discuss the generalisability (external validity) of the study results.	–	–	STROBE ITEM: P. 3 - Methods, Registry Features and Data Acquisition discusses generalisability to the larger population.
Other information
Funding
22	Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based.	–	–	STROBE ITEM: P. 6 - Methods, Role of the Funding Source P. 9 - Acknowledgements
Accessibility of protocol, raw data, and programming code
22	–	22.1: Authors should provide information on how to access any supplemental information such as the study protocol, raw data, or programming code.	–	RECORD ITEM: 22.1: Not reported.

Notes: Langan SM, Schmidt SA, Wing K, et al. The REporting of studies Conducted using Observational Routinely collected health Data for Pharmacoepidemiology statement (RECORD-PE). BMJ. 2018;363:k3532. Creative Commons Attribution (CC BY 4.0) license.⁵ Mehra MR, Ruschitzka F, Patel AN. Retraction–Hydroxychloroquineor chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020.¹

Abbreviations: RECORD, REporting of studies Conducted using Observational Routinely collected health Data; RECORD-PE, REporting of studies Conducted using Observational Routinely collected health Data for Pharmacoepidemiology; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology.

Table 2 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)⁵ Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, N Engl J Med, 2020². Red Font Indicates Areas That Were Not Adequately Reported

Item No	STROBE Items	RECORD Items	RECORD-PE Items	Location in Manuscript (Page Number and Description of Reporting)
Title and abstract
1	(a) Indicate the study’s design with a commonly used term in the title or the abstract. (b) Provide in the abstract an informative and balanced summary of what was done and what was found.	1.1: The type of data used should be specified in the title or abstract. When possible, the name of the databases used should be included. 1.2: If applicable, the geographical region and timeframe within which the study took place should be reported in the title or abstract. 1.3: If linkage between databases was conducted for the study, this should be clearly stated in the title or abstract.	–	STROBE ITEMS: (a) Study design not reported in the Title or Abstract. (b) P. 1 - Summary provided in Abstract. RECORD ITEMS: 1.1: P. 1 – Abstract reported use of an “observational database from 169 hospitals” was used, indicating that these were routinely collected health data. 1.2: P. 1 – Multinational registry data were used and time frame is noted in the Abstract. 1.3: P. 1 – Data linkage is implied by the Abstract, but not explicitly stated.
Introduction
Background rationale
2	Explain the scientific background and rationale for the investigation being reported.	–	–	STROBE ITEM: P. 2 – Study rationale and scientific background was explained adequately in the Background.
Objectives
3	State specific objectives, including any prespecified hypotheses.	–	–	STROBE ITEM: P. 2 – The final paragraph of the Background section adequately describes the study objectives and prespecified hypotheses.
Methods
Study design
4	Present key elements of study design early in the paper.	–	4.a: Include details of the specific study design (and its features) and report the use of multiple designs if used. 4.b: The use of a diagram(s) is recommended to illustrate key aspects of the study design(s), including exposure, washout, lag and observation periods, and covariate definitions as relevant.	STROBE ITEM: P. 3-4 – Overall design of the study was described in the Methods, Data Collection section, although the study design (historical cohort study) was not named. RECORD-PE ITEMS: 4.a.: P. 3-4 – Study design details was described. 4.b: Not reported.
Setting
5	Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection.	–	–	STROBE ITEM: P. 2, Table S1 – Setting and locations of data location described.
Participants
6	(a) Cohort study—give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up. Case-control study—give the eligibility criteria, and the sources and methods of case ascertainment and control selection. Give the rationale for the choice of cases and controls. Cross sectional study—give the eligibility criteria, and the sources and methods of selection of participants. (b) Cohort study—for matched studies, give matching criteria and number of exposed and unexposed. Case-control study—for matched studies, give matching criteria and the number of controls per case.	6.1: The methods of study population selection (such as codes or algorithms used to identify participants) should be listed in detail. If this is not possible, an explanation should be provided. 6.2: Any validation studies of the codes or algorithms used to select the population should be referenced. If validation was conducted for this study and not published elsewhere, detailed methods and results should be provided. 6.3: If the study involved linkage of databases, consider use of a flow diagram or other graphical display to demonstrate the data linkage process, including the number of individuals with linked data at each stage.	6.1.a: Describe the study entry criteria and the order in which these criteria were applied to identify the study population. Specify whether only users with a specific indication were included and whether patients were allowed to enter the study population once or if multiple entries were permitted. See explanatory document for guidance related to matched designs.	STROBE ITEMS: (a): P. 2-3 – Methods, Data Collection reported eligibility criteria and methods of participant selection. (b): Matching criteria not applicable. RECORD ITEMS: 6.1: Methods of study population selection (such as codes or algorithms used to identify participants) were not listed. 6.2: While ISO procedures and quality control were used in data collection were reported (P. 2 – Methods, Data Source), and validation is implied (“The collected data sample from each health care entity is validated against financial records and external databases.”). However, no reference to a validation study was provided, and the success or failure of quality control procedures were not discussed. 6.3: While linkage of databases occurred (including multiple health administrative databases, electronic health record data, and manually-entered data from 169 participating hospitals), details on linkage type (deterministic, probabilistic), process, completeness, validity were provided. RECORD-PE ITEM: 6.1.a: P. 2-3 – Methods, Data Collection adequately reported study entry criteria.
Variables
7	Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable.	7.1: A complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers should be provided. If these cannot be reported, an explanation should be provided.	7.1.a: Describe how the drug exposure definition was developed. 7.1.b: Specify the data sources from which drug exposure information for individuals was obtained. 7.1.c: Describe the time window(s) during which an individual is considered exposed to the drug(s). The rationale for selecting a particular time window should be provided. The extent of potential left truncation or left censoring should be specified. 7.1.d: Justify how events are attributed to current, prior, ever, or cumulative drug exposure. 7.1.e: When examining drug dose and risk attribution, describe how current, historical or time on therapy are considered. 7.1.f: Use of any comparator groups should be outlined and justified. 7.1.g: Outline the approach used to handle individuals with more than one relevant drug exposure during the study period.	STROBE ITEM: Outcomes: P.3 – Methods, Statistical Analysis reported “ ... with the end point of in-hospital death”. Exposures: P. 3 – Methods, Data Collection described clinical definitions of COVID-19 diagnosis, as well as criteria for drug exposures. Effect modifiers: P. 3 – Methods, Data Collection and Statistical Analysis described effect modifiers and other data collected. RECORD ITEM: 7.1: List of codes or algorithms for outcomes, exposures, confounders and effect modifiers were not provided. RECORD-PE ITEMS: 7.1.a: P. 3 – Methods, Data Collection describes how exposure to drugs were defined. 7.1.b: Precise data sources for drug exposures were not provided. It was not clear whether these came from electronic health record data, health administrative data, or other database (only reported “Cardiovascular drug therapy recorded at the time of hospital admission was also included ... ”). Thus, it was unclear whether the drug exposures were measured by prescriptions, dispensings, or administrations. 7.1.c: P. 3 - Methods, Data Collection discusses the time window of the study, which was the same time window for drug exposure. 7.1.d: Information on prior or ever exposure to study drugs was not provided. 7.1.e: P. 3 – Methods, Data Collection describes the definition of current exposure. Historical or time on therapy considerations were not reported. 7.1.f: P. 3 – Methods, Statistical Analysis describes the comparisons made. 7.1.g: How multiple drug exposures was handled was not reported.
Data sources/measurement
8	For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group.	–	8.a: Describe the healthcare system and mechanisms for generating the drug exposure records. Specify the care setting in which the drug(s) of interest was prescribed.	STROBE ITEM: Data sources and details on measurement were not adequately reported. Comparability of assessment methods for various data sources was not reported. RECORD-PE ITEM: 8.a: Not applicable, since many hospitals and countries contributed to the data.
Bias
9	Describe any efforts to address potential sources of bias.	–	–	STROBE ITEM: P. 2 - Methods, Data Source section describes quality control efforts used to reduce the risk of bias due to poor data collection. P. 3 - Methods, Statistical Analysis describes the use of multivariable regression to control for or reduce the effect of confounding factors.
Study size
10	Explain how the study size was arrived at.	–	–	STROBE ITEM: Not applicable, since registry is fixed and all available data were used.
Quantitative variables
11	Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen, and why.	–	–	STROBE ITEM: P. 3 - Methods, Statistical Analysis reported on use of quantitative variables.
Statistical methods
12	(a) Describe all statistical methods, including those used to control for confounding. (b) Describe any methods used to examine subgroups and interactions. (c) Explain how missing data were addressed. (d) Cohort study—if applicable, explain how loss to follow-up was addressed. Case-control study—if applicable, explain how matching of cases and controls was addressed. Cross sectional study—if applicable, describe analytical methods taking account of sampling strategy. (e) Describe any sensitivity analyses.	–	12.1.a: Describe the methods used to evaluate whether the assumptions have been met. 12.1.b: Describe and justify the use of multiple designs, design features, or analytical approaches.	STROBE ITEMS: (a): P. 3 - Methods, Statistical Analysis. (b): P. 3-4 - Methods, Statistical Analysis reported on subgroup and sensitivity analyses. Tests for collinearity, correlation and interaction of independent variables were not reported. (c): P. 3 - Methods, Statistical Analysis reported “Multiple imputation for missing values was not possible ... ” (d): Ways in which loss to follow-up was addressed were not reported. RECORD-PE ITEMS: 12.1.a: Assumptions were not reported. For example, the study did not address immortal time bias or misclassification bias, or how these were addressed. 12.1.b: P. 3-4 – Methods, Statistical Analysis describes all analytic approaches.
Data access and cleaning methods
12	–	12.1: Authors should describe the extent to which the investigators had access to the database population used to create the study population. 12.2: Authors should provide information on the data cleaning methods used in the study.	–	RECORD ITEMS: 12.1: Data access was not reported. 12.2: Data cleaning methods were not reported.
Linkage
12	–	12.3: State whether the study included person level, institutional level, or other data linkage across two or more databases. The methods of linkage and methods of linkage quality evaluation should be provided.	–	RECORD ITEM: 12.3: While linkage between data sources occurred (as evidenced by various data sources used), the methods of data linkage and quality evaluation were not reported.
Results
Participants
13	(a) Report the numbers of individuals at each stage of the study (eg, numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed). (b) Give reasons for non-participation at each stage. (c) Consider use of a flow diagram.	13.1: Describe in detail the selection of the individuals included in the study (that is, study population selection) including filtering based on data quality, data availability, and linkage. The selection of included individuals can be described in the text or by means of the study flow diagram.	–	STROBE ITEMS: (a): P. 4, Table 1, Table S2 - The overall study numbers were reported, however, the numbers of individuals eligible and excluded were not reported. (b): The numbers of individuals excluded were not reported. (c): A study flow diagram was not provided. RECORD ITEMS: 13.1: Filtering based on data quality, data availability and linkage were not reported.
Descriptive data
14	(a) Give characteristics of study participants (eg, demographic, clinical, social) and information on exposures and potential confounders. (b) Indicate the number of participants with missing data for each variable of interest. (c) Cohort study—summarise follow-up time (eg, average and total amount).	–	–	STROBE ITEMS: (a): Table 1, Table S2, Table S3. (b): P. 3 - Methods, Statistical Analysis reported “ ... there were no codes to indicate that data were missing; if the patient’s electronic health record did not include information on a clinical characteristic, ..., it was assumed that the characteristic was not present.” (c): Length of hospitalization was not reported.
Outcome data
15	Cohort study—report numbers of outcome events or summary measures over time. Case-control study–report numbers in each exposure category, or summary measures of exposure. Cross sectional study—report numbers of outcome events or summary measures.	–	–	STROBE ITEM: P. 4, Table 1 and Table 2.
Main results
16	(a) Give unadjusted estimates and, if applicable, confounder adjusted estimates and their precision (eg, 95% confidence intervals). Make clear which confounders were adjusted for and why they were included. (b) Report category boundaries when continuous variables are categorised. (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period.	–	–	STROBE ITEMS: (a): P. 4-5 – Results, Multivariable Logistic-Regression Analysis, Figure 1, Figure S1, Table S4, Table S5, Table S6 provide estimates of multivariable regression analyses. (b): Not applicable. (c): Not applicable.
Other analyses
17	Report other analyses done—eg, analyses of subgroups and interactions, and sensitivity analyses.	–	–	STROBE ITEM: P. 4-6 - Results and Supplementary Appendix reported sensitivity analysis results and tipping point analysis results.
Discussion
Key results
18	Summarise key results with reference to study objectives.	–	–	STROBE ITEM: P. 6 - first paragraph of the Discussion summarizes key results.
Limitations
19	Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias.	19.1: Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported.	19.1.a: Describe the degree to which the chosen database(s) adequately captures the drug exposure(s) of interest.	STROBE ITEM: P. 7 - Discussion reports study limitations. RECORD ITEM: 19.1: Not adequately reported. Discussion mentions confounding, but does not report limitations due to missing data, changing eligibility over time or misclassification or selection bias. RECORD-PE ITEM: 19.1.a: Not reported.
Interpretation
20	Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence.	–	20.a: Discuss the potential for confounding by indication, contraindication or disease severity or selection bias (healthy adherer/sick stopper) as alternative explanations for the study findings when relevant.	STROBE ITEM: P. 7 - Discussion gives overall interpretation. RECORD-PE ITEM: 20.a. Not reported.
Generalisability
21	Discuss the generalisability (external validity) of the study results.	–	–	STROBE ITEM: Generalisability is not discussed.
Other information
Funding
22	Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based.	–	–	STROBE ITEM: P. 7 – The funding sources are provided, however the role of the funders is not discussed.
Accessibility of protocol, raw data, and programming code
22	–	22.1: Authors should provide information on how to access any supplemental information such as the study protocol, raw data, or programming code.	–	RECORD ITEM: 22.1: Not reported.

Notes: Langan SM, Schmidt SA, Wing K, et al. The REporting of studies Conducted using Observational Routinely collected health Data for Pharmacoepidemiology statement (RECORD-PE). BMJ. 2018;363:k3532. Creative Commons Attribution (CC BY 4.0) license.⁵ Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction:cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med. 2020;382(26):2582.²

Abbreviations: RECORD, REporting of studies Conducted using Observational Routinely collected health Data; RECORD-PE, REporting of studies Conducted using Observational Routinely collected health Data for Pharmacoepidemiology; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology.

Results

In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%), and 9.5 of 15 RECORD-PE items (63.3%) (χ² = 14.839, P <0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%)(χ² = 11.668, P = 0.003). These results indicate that while STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, and validation of the coding or algorithms. RECORD also requires a detailed description of the database population (ie, how data were collected in the population from which the study database is created), and the description and accuracy of linkage of databases. The lack of description of the underlying Surgisphere database should have raised concerns amongst peer reviewers and editors.

Discussion

Non-transparent and incomplete reporting may have prevented peer reviewers from recognizing significant weaknesses in the research methods, especially if peer reviewers were unfamiliar with research using large health datasets. We found significant differences in the proportion of adequately reported items on the STROBE, RECORD and RECORD-PE checklists, with STROBE items being better reported than the items on the checklists designed for studies using routinely collected health data.

The reason given by three of the authors for the retraction of The Lancet manuscript was the inability of independent auditors to obtain the original data to verify the analyses.^1,2 While we understand that legal circumstances may be in place that prevent the data holder from sharing the original data, this runs counter to the movement toward open science and data sharing being adopted by many journals, especially those that endorsed the Wellcome COVID-19 statement on Gold Open Access and reusability,¹³ indicating underlying support for data availability. The availability of data and programming code allows for the replication of research and verification of results, paramount to the process of scientific discovery.¹⁴ The RECORD reporting guidelines include an item (RECORD 22.1) that requests authors “provide information on how to access any supplemental information such as the study protocol, raw data, or programming code”, and the availability of these data was not adequately reported in the two retracted manuscripts. As detailed in the RECORD explanatory document,⁴ we recognize that some circumstances may restrict the free sharing of raw data, such as privacy laws or data sharing agreements. The RECORD statement is a reporting guideline and does not mandate the open sharing of data. However, RECORD mandates reporting of whether data are available for sharing, and how to access the data, programming code, and study protocol. Had this information been reported in the two manuscripts, the editors and peer reviewers could have made informed decisions about whether to accept the manuscripts for publication.

Reporting guidelines improve the quality of research publications. There is both direct evidence and indirect evidence to support this statement. A randomised controlled trial (RCT) demonstrated a small benefit of including CONSORT and STROBE in the journal peer review process.¹⁵ Similarly, a systematic review and meta-analysis of reports assessing the completeness of RCTs reported that journal endorsement of the CONSORT guidelines were associated with improved completeness of reporting.¹⁶ Compared to manuscripts based on RCTs, observational research may be more at risk for hidden sources of bias. Therefore, it is essential to encourage transparent and complete reporting of research using routinely collected health data. Journals play a key role in raising awareness amongst authors and peer reviewers about the importance of reporting guidelines to aid with transparent reporting. However, journal editors may be concerned with the rigidity posed by reporting guidelines, and the risk that authors may bypass submission to journals requiring them as part of the submission process.¹⁷ Nevertheless, researchers have a generally positive attitude toward reporting guidelines.¹⁸ The Equator Network’s catalogue of reporting guidelines by main study type (https://www.equator-network.org/) and interactive walk-through of choice of reporting guideline (https://www.goodreports.org/) has made their use easier for both authors and peer-reviewers.

The design, conduct, and submission of research that is not transparent, open, and adequately reported is a danger to the public trust in science and is wasteful. We have demonstrated that the two retracted studies were inadequately reported, as defined by RECORD and RECORD-PE. We have written this article as an appeal to authors, publishers, editors, and peer reviewers to endorse and effectively implement the correct reporting guidelines in their submission and evaluation of manuscripts. Scientists, journals and the public can no longer accept research that is not transparent and open, especially in the time of a public health emergency such as the COVID-19 pandemic.

Author Contributions

All authors made substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; took part in drafting the article or revising it critically for important intellectual content; agreed to submit to the current journal; gave final approval of the version to be published; and agree to be accountable for all aspects of the work.

Acknowledgment

Eric Benchimol and Vera Ehrenstein serve on the editorial board of Clinical Epidemiology.

Disclosure

EIB and SML are co-Chairs of the RECORD Steering Committee. DM serves on the RECORD Steering Committee, serves on the EQUATOR Network Steering Group, and is Director of the Canadian EQUATOR Centre. SML reports grants from Wellcome Trust, during the conduct of the study. The authors report no other potential conflicts of interest for this work.

Additional information

Funding

SML is supported by a Wellcome Senior Clinical fellowship in Science (205039/Z/16/Z).