The design and analysis of non-randomized studies: a case study of off-label use of hydroxychloroquine in the COVID-19 pandemic

ABSTRACT

Introduction: The COVID-19 pandemic has prompted researchers to conduct non-randomized studies in an effort to find an off-label drug that can effectively combat the virus and its effects. While these studies can expedite the drug approval process, researchers must carefully design and analyze such studies in order to perform rigorous science that is reproducible and credible. This article focuses on several key design and analysis considerations that can improve the scientific rigor of non-randomized studies of off-label drugs.

Areas covered: The aim of this article is to provide an overview of best approaches that should be considered for non-randomized studies on off-label drugs. We discuss these approaches in detail and use a non-randomized study by Rivera et al. in Cancer Discovery as an example of methods that have been undertaken for COVID-19.

Expert opinion: While non-randomized studies are inherently biased, they may be unavoidable in situations such as the COVID-19 pandemic, where researchers need to find an effective treatment quickly. We believe that a well-formed experimental design, high-quality data collection, and a well-thought-out statistical and data analysis plan are sufficient to produce rigorous and credible results for making an optimal decision.

KEYWORDS:

• Clinical trials
• COVID-19
• non-randomized study
• off-label drug
• hydroxychloroquine
• pandemic
• SARS-CoV-2

Article highlights

•Non-randomized studies are not a substitute for randomized clinical trials; however, they are unavoidable in situations, such as the COVID-19 pandemic, when scientists need to expedite the process of evaluating off-label drugs.

•Before conducting any experiment on off-label drugs, investigators should first develop a transparent experimental design outlining the set of procedures for addressing the research question.

•While bias is an inherent problem, non-randomized studies with a well-formulated study design, including an appropriate selection of the control group, a high-quality data collection, and a rigorous analysis plan, can provide dependable results.

•In non-randomized studies, the data analysis plan should include how to handle missing data, adjusting for imbalance in the baseline variables, variable selection, model building, sensitivity analysis, and incorporating causality.

•Careful variable selection needs to be considered separately for the propensity score model and the main models that are addressing the scientific questions.

•Data dredging or data fishing continues to plague the scientific community, and a system that requires a record of the study design and the statistical analysis plan, similar to the U.S. National Library of Medicine’s ClinicalTrials.gov Protocol Registration and Results System, should be developed for non-randomized or real-world studies to address this issue.

This box summarizes the key points contained in the article.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Additional information

Funding

This paper was not funded.