Response to “Why all randomized controlled trials produce biased results”

In April 2018, the Annals of Medicine published a provocative piece titled “Why all randomized controlled trials produce biased results” [1]. While the strengths and limitations of RCTs remain a fertile discussion topic, the piece in question leaves much to be desired. In many places, the piece ignores prior literature that explains and/or refutes several key points. Absent a formal response, this piece may create confusion to understanding of RCTs in the research community.

The piece begins by stating that trials are perceived to be exempt from strong theoretical assumptions, methodological biases, and the influence of researchers. Indeed, many researchers and consumers have limited understanding of the complexities in trial design and analyses. However, considering the thousands of peer-reviewed manuscripts, dozens of textbooks, semester-long courses and even professional societies dedicated to various aspects of trial design, conduct, and analysis, this statement also trivializes the existence of an entire subfield of methodological research and innovation in clinical trials. Furthermore, trials that have been published for long enough collect large numbers of citations also would have occurred before modern innovations in trial design and reporting, making the “10 most cited trials of all time” a poor choice to comprehensively assess quality of trial design, conduct, reporting, and the degree of bias present in trial results.

The piece states that the simple-treatment-at-the-individual-level limitation is a constraint of RCTs not yet thoroughly discussed and notes that randomization is infeasible for many scientific questions. This, however, is not relevant to the claim that all RCTs produce biased results; it merely suggests that we should not use randomized controlled trials for questions where they are not applicable. Furthermore, the piece states that randomized trials cannot generally be conducted in cases with multiple and complex treatments or outcomes simultaneously that often reflect the reality of medical situations. This statement ignores a great deal of innovation in trial designs, including some very agile and adaptable designs capable of evaluating multiple complex treatments and/or outcomes across variable populations.

Some of these innovations include Bayesian statistical approaches which allow continuous monitoring, adaptive-enrichment designs that enable trial populations to evolve based on early feedback, pragmatic trials, platform trials, and designs which permit analysis of multiple dynamic or complex endpoints [2–7]. Consider the I-SPY 2 trial [8], which allows treatment combinations to be added, dropped, and combined with others while being tested simultaneously across different target populations (defined by biomarkers), graduating to the next stage when the Bayesian predictive probability of success exceeds the prespecified threshold. The STAMPEDE trial [9–14] in prostate cancer is another example of a multiarm, multistage trial evaluating multifactorial treatment combinations in prostate cancer. For an even more ambitious proposal that would fuze several of these principles into a single trial, consider the JAMA article by Angus [15] proposing the randomized, embedded, multifactorial adaptive platform (REMAP) trial. These topics have been discussed in such widely read journals as JAMA, Lancet and the New England Journal of Medicine [16] as well as the more topical Trials, Clinical Trial, and Contemporary Clinical Trials.

Despite the assertion that the achieving-good-randomization assumption is a foundational and strong assumption of RCTs, there is no requirement for baseline balance in all covariates to have a valid statistical inference from the trial. This myth has been discussed by accomplished statisticians such as Altman [17], Senn [18–21], Zhao and Berger [22] repeatedly over several decades. Allowing this statement to stand unchallenged merely perpetuates the myth of balance as a necessary condition for valid inference in RCTs.

Paraphrasing the Zhao and Berger paper, under proper random treatment assignment, distributions of all baseline covariates among treatment groups are random. Therefore, random baseline covariate imbalances and random treatment assignment must be accepted or rejected together. It is important to remember that the fundamental goal of randomization in clinical trials is preventing selection bias; this should not be undermined by excessively forced balancing of baseline covariates among treatment groups. As Senn has discussed previously, the standard probability calculations applied to clinical trial results already make an allowance for the fact that the treatment groups will almost certainly be imbalanced.

The piece pursues the balance question further with an untenable suggestion that researchers should create multiple randomization schedules and, if the first randomization does not yield a balanced allocation, simply randomize again to achieve better balance. Even if we suspend the debate over balance as a required condition for valid inference in RCTs, this proposal is not practical for the majority of RCTs in medicine, in which most trials require a treatment decision at a specific point in the disease course (e.g. patients admitted to the hospital having a stroke or heart attack, patients that are being treated for acute dehydration, patients with active cancer that need to begin therapy to retard progression of disease, pregnant women). Several trials mentioned in the original piece illustrate this clearly. The trial of acute ischemic stroke patients [23] began enrolling in January 1991 and completed enrollment over three years later; sadly, acute stroke is not a condition that can wait three years to give out the randomization assignments. The trial of intensive insulin therapy in critically ill patients [24] began enrolling in February 2000 and completed enrollment in January 2001 after an interim analysis suggested early stopping for efficacy, no less – another subject of which the piece fails to acknowledge and for which the get the full patient cohort and randomize them all before doing anything approach has no solution. The trial of colorectal cancer [25] enrolled from September 2000 to May 2002, stopped at an interim analysis that stopped enrollment in one of the treatment arms for safety reasons). In none of those settings would it have been possible to enroll the entire cohort, check for balance, and then re-randomize the patients.

Finally, the piece failed to acknowledge much prior discussion of strategies to achieve balance, including several proposals that would be feasible in the clinical situations discussed above. Techniques such as stratified randomization [26], minimization [27,28] and covariate-adaptive randomization [29–39] have been published and discussed for many years as options to enforce some degree of balance on selected covariates if/when that is considered essential. In particular, consideration of the literature on covariate-adaptive randomization is essential for any discussion of covariate balance and its implication in RCTs.

The piece appears to suggest that the solution to selection bias in RCTs is just making sure that everyone involved with the trial is sufficiently blinded. As previously noted in Senn’s 2013 article, full blinding is achieved with the help of randomization; it is not clear how blinding everyone involved with the trial is even possible without randomly assigning the treatments. The piece incorrectly says that small trials are more likely to be biased than large trials. The first confusing element is the focus on total enrollment; as any trialist knows, total enrollment is not the only design factor that determines whether a trial is sufficiently large for precise estimates of treatment effect. The distribution of primary outcome and effect size are also critical (and arguably more so) in determining whether a trial is large enough to answer its primary question. Some trials may be adequately sized with 200 patients, whereas other may be inadequately sized even with 2,000 patients.

Moreover, the problem is not actually a problem of bias but rather a problem of precision. Large trials can provide a more precise estimate of the main effect(s) being tested; smaller trials run the risk of being underpowered and thus unable to make a precise statement about whether the main effect size(s) of interest are statistically or clinically significant. Here, a statement about trial results merits clarification: “An example is that the stroke trial with 624 participants in total reports that at three months after the stroke, 54 treated patients died compared to 64 placebo patients – with the main outcome thus being just a difference of 10 deaths.” While technically true – 54 is 10 less than 64 – it is a poor representation of the purpose of statistical analyses performed on the results of RCTs. The very reason for performing probability calculations with RCT results is to determine how likely the observed difference(s) were under an assumed null hypothesis that there was no difference between the treatments. The difference in the absolute number of events in each arm is only relevant insofar as it informs this calculation. It is very possible for a small absolute difference in the number of events to be relatively strong evidence for the superiority of one treatment. The probability is the relevant statistic, not the absolute number of events.

The piece discusses the unique time period assessment bias to mean that most researchers only collect baseline and endline data points and assess one average outcome instead of another. This is, in many cases, incorrect, since there are many different types of trial outcomes: one may analyze yes/no survived to hospital discharge outcomes, time-to-event outcomes for longer-term survival or freedom from some event, and continuous variables that are often collected as repeated-measures such as changes in blood pressure, cholesterol, or functional capacity. One obvious example of trials that do this are Phase 2 trials of lipid-lowering medications, which typically collect far more than just baseline and endline data points, but rather collect lab measurements repeatedly over the study period to get an accurate picture of the change in lipid profile over time [40].

The piece criticizes trials having unequal follow-up time for participants, but there are three important considerations that were not acknowledged: (a) in medical trials this is often a necessary condition to carry out the trial – otherwise, there would be no way to assess time-to-event or survival outcomes; (b) there are statistical approaches designed to account for this (survival analyses and time offsets being the most obvious examples) and (c) there is no reason to believe this impedes valid inference as long as an appropriate statistical approach is used. For example, in survival analyses, neither Kaplan-Meier curves nor Cox models are disturbed by unequal follow-up time across treatment arms.

In discussion about the background-traits-remain-constant assumption the piece ignores the extensive literature and historical discussion of mediation analyses [41–47] in RCTs; statistical modeling may be applied to estimate the direct effect of treatment versus other indirect effects acting through intermediate variables measured post-randomization. Whether this is advisable remains a subject of debate among statisticians.

The average treatment effects limitation is another much-discussed issue. Concerns about this are typically overblown as estimates of relative risk, odds, or hazard are generally transportable across subgroups (within the same basic treatment population, that is; we are not referring to nonsense scenarios like performing a heart transplant to improve survival in a patient that needs a hip replacement). Furthermore, as discussed earlier in this article, modern innovations in trial design (such as Bayesian adaptive-enrichment designs) allow the trialist to use ongoing data to determine which patients appear to benefit the most and/or those in whom therapy is futile or even harmful if there is significant evidence of a heterogeneous treatment effect.

We are not intended to suggest that RCTs are unimpeachable. Quite the opposite: RCTs must be planned with careful consideration of the requisite assumptions, monitored with extreme rigor, and analyzed properly to ensure valid statistical inference from the results. We also acknowledge that when RCTs are impractical or unavailable, we must utilize non-RCT evidence to support decisions and draw conclusions about the world around us. We simply prefer that discussion of RCTs be adequately informed and concentrate on legitimate questions and solutions. We hope that this leads to better-informed discussion of true strengths and weaknesses of RCT’s in practice.

Disclosure statement

No potential conflict of interest was reported by the authors.