There is no single gold standard study design (RCTs are not the gold standard)

KEYWORDS:

• Epidemiologic research design
• biomedical research/*standards
• drug-related side effects and adverse reactions
• evidence-based medicine/*standards

1. Introduction

Randomized controlled trials (RCTs) are often referred to as the gold standard of biomedical research [1]. However, there is no single gold standard of study design, especially when it comes to studies of drug safety. The gold standard design is whatever method provides the information you need in the most reliable way in view of what is currently known or can be obtained [1].

2. Body of text

2.1. Oversimplified hierarchy

As an epidemiologic expert witness who addresses drug safety information in federal court cases, I have on multiple occasions been confronted with a misleading narrative that promotes a hierarchy of evidence based on study type. This hierarchy places RCTs at its top, followed in sequence by cohort studies, case-control studies, and case reports [2]. The reasoning behind this scheme is that the ‘better’ designs have lower risks of bias than the ‘lesser’ designs and therefore provide more reliable evidence. However, this rationalization boils down the complexities of drug safety research into a simplistic hierarchy based on only four study design elements while ignoring many other important study features. The remainder of this essay addresses why use of this or any other rigid evidence hierarchy is ill-advised.

2.2. Randomization is no panacea

Randomization is an important tool for mitigating baseline confounding in RCTs. However, it does not free a study of all other sources of error, such as those that might occur from post-randomization confounding and selection bias [3]. Other forms of error that may adversely affect trials include inaccurate case ascertainment, inadequate blinding, differential loss to follow-up, noncompliance, inadequate follow-up, failure to adequately consider time-dependencies, failure to consider all contributing factors, and, of course, random error. In the words of pioneering biostatistician Bernard Cornfield, ‘randomization by itself is insufficient. We must indicate the specific variables we wish to control and must devise the specific experimental procedures to control them’ [4].

If randomized study were truly perfect, we would expect consistent results from RCTs addressing the same question. However, contradictory results from RCTs are not uncommon. Ioannides found that 14 (36%) of 39 highly cited RCTs were either contradicted or shown to have a weaker effect in RCTs that attempted to replicate their results [5]. This is not to suggest that the problem of irreproducibility is unique to experimental studies; nonexperimental studies fail this type of replication test at an even greater frequency [5]. The point here is that contradictory results from even highly cited RCTs are not unusual.

I recently published a review of an RCT that was initially designed to determine whether a particular anticoagulant (rivaroxaban) prevented major cardiovascular events in high-risk patients, and whether a particular proton pump inhibitor (PPI) decreased the risk of major bleeding in the anticoagulant users [6]. The trial initially designed to test these questions was sound for its initial purposes. However, data were repurposed to assess whether the PPI randomly assigned in this study was associated with previously unanticipated adverse effects, including whether it caused chronic kidney disease [7]. With respect to chronic kidney disease, the study failed on several accounts: it lacked a case definition for chronic kidney disease, kidney function was unmonitored during the trial, prevalent cases were not excluded before the trial was begun, prior PPI use (before the trial) and over-the-counter PPI use (during the trial) were unconsidered, discontinuation of the study drug during follow-up was common, and duration of follow-up was limited (median follow-up 3 years) [8]. These limitations reduced the utility of the data for testing whether PPIs cause chronic kidney disease, even though PPI use had been randomized at the start of the trial.

As noted earlier, even high impact RCTs are not immune to error. An insightful article with the provocative title Why All Randomised Controlled Trials Produce Biased Results identified multiple unrecognized biases in the 10 most cited RCTs in the medical literature [9]. Among these limitations were inadequate randomization, initial sample selection bias, failure to collect data on all relevant variables, neglect of contributing factors during follow-up, incomplete blinding, unauthorized use of the study drug in the control group, too small a sample, limitations in quantitative measurements, and failure to verify all study assumptions.

In addition, RCTs regularly include design features that bolster internal validity while limiting generalizability. One such limitation is found with intention-to-treat (ITT) analysis. While ITT provides reassurances to prevent selective follow-up, it also minimizes differences in observed effects, biasing results toward the null. Moreover, RCTs are conducted under highly controlled conditions in specialty clinics within narrowly defined patient populations with relatively simple disease. That is, trials usually have admissibility criteria that exclude patients with specific comorbidities that may accompany the condition in real clinical situations. While RCT admissibility criteria may increase the rigor and internal validity of the study, they can also limit external validity [10].

Of course, we must also consider that for many questions, RCTs are not feasible, practical, or timely. Such is the case when the adverse event in question is serious, rare, and occurs only after an extended induction period. And, of course, there are many situations for which it would be unethical to randomly expose participants to potential harms. RCTs can only be ethically pursued when there is balanced doubts (‘equipoise’) as to risks and benefits. Thus, many questions of drug safety are just beyond the reach of RCTs. Under these circumstances, retrospective cohort studies and case-control studies may provide viable alternatives to RCTs.

2.3. Studies of intended effects (efficacy studies) vs. studies of unintended effects (safety studies)

The extent to which randomization is required in a study depends on many factors. One such factor is whether the study addresses intended effects (efficacy) or unintended effects (safety). This issue was clarified in a 1983 landmark paper by Miettinen in which he coined the terms ‘confounding by indication’ and ‘confounding by contraindication’ [11]. Confounding by indication/contraindication occurs when therapeutic choices are closely linked to prognostic factors for the outcome. When adverse effects are not widely anticipated, this link does not exist and confounding by indication/contraindication is unlikely [12].

Even when there are suspected links between the study drug and an adverse effect, the effects of confounding by indication/contraindication can be weakened if not entirely precluded through observational study methods such as the careful use of active controls, prudent use of study base exclusion criteria, closely matching subjects based on propensity scores, and analytic regression techniques. While RCTs are uniquely superior in evaluating the anticipated benefits of a drug, these advantages are not as extreme when studying adverse reactions. In fact, observational studies may provide more accurate estimates for the incidence of adverse events in actual practice [12].

2.4. RCTs vs. observational studies

Observational studies (cohort and case-control studies) have contributed greatly to our understanding of adverse drug effects. Properly conducted observational studies can even produce quantifications of efficacy that are, on average, similar those of RCTs [13]. Contrary to popular opinion, well-designed cohort and case-control studies do not systematically overestimate the magnitude of associations reported by RCTs [13,14].

But what about situations in which results from observational studies differ from those of RCTs? Can we assume that the RCTs were correct while the observational studies were biased? Evidence suggests otherwise. Consider, for example, results from the randomized Women’s Health Initiative trial on postmenopausal combined hormone use and the observational Nurses’ Health Study on the same topic [15,16]. The former (the trial) found a similar or slightly higher incidence of coronary disease in the treatment group than in the control group. In contrast, the latter (the observational study) found a slightly reduced risk associated with treatment. While it is facile to conclude this difference was due to confounding in the observational study, Hernan and colleagues demonstrated that the discrepancies were largely explained by differences in (a) the distributions of the times since menopause among the participants in the two studies, (b) differences in lengths of follow-up, and (c) the use of ITT in the trial but not the observational study [17]. Thus, the discrepant results were due to differences not directly related to randomization.

2.5. Cohort vs. case-control studies

The standard evidence hierarchy also assumes that cohort studies are superior to case-control studies [2]. This, too, is an oversimplification. Some of this oversimplification can be traced to the non-standardized use of ‘retrospective/prospective’ terminology. I believe this misunderstanding is also do to what Miettinen referred to as the ‘trohoc fallacy’ [18].

A contemporary usage of the term ‘prospective’ refers to studies in which the exposure measurement comes before the disease ascertainment, so that knowledge of disease status cannot influence classification of the exposure [19]. Accordingly, a study can be prospective even when events occurred in the chronologic past, using historical data for instance. Thus, case-control studies can be either prospective or retrospective.

The ‘trohoc fallacy’ assumes that case-control studies are inherently ‘backward looking’ by comparing prior exposures in cases and controls [18]. This is false. A more meaningful conceptualization of the function of the control series in case-control studies is as a stochastic estimate of exposed-to-nonexposed person-time in the underlying study population (incidence density sampling). This permits a meaningful statistical estimate of the rate ratio in the source population – no rare disease assumption required [20]. It also provides a scientific basis for selecting controls in case-control studies. In effect, properly conducted case-control studies provide the same estimates of risk as properly conducted cohort studies.

2.6. Case reports

What of the lowly case report? Because premarketing RCTs are conducted in a limited number of volunteers who are followed for relatively short periods of time, it is not possible to know about all potential adverse effects of a drug before it is marketed. Therefore, many serious adverse drug reactions are discovered only after a drug is released for general consumption. An essential element of discovering previously unrecognized adverse reactions postmarketing is the ‘lowly’ case report. In fact, case reports have provided the primary evidence for approximately two-thirds (66.1%) of withdrawals of medicinal products from the market launched since 1950 [21]. The remainder of withdrawals have relied on systematic reviews (2.1%), randomized studies (8.7%), non-randomized studies (10.5%), and mechanism-based reasoning (12.6%) [21]. These historical percentages may no longer fully reflect current practices because they include information from the 1950s. Note that the milestone RCT on streptomycin for the treatment of pulmonary tuberculosis was not completed until 1948 [22]. Nonetheless, case reports remain an important and necessary element of adverse drug reaction discovery and explanation.

3. Conclusion

Rescher defines oversimplification as ‘simplifying matters beyond the warrant of the functional requirements of the particular situation at hand’ [23]. Throughout this essay, I have tried to make the point that the ‘RCT gold standard’ and its associated hierarchy are oversimplifications that do not serve the functional requirements of drug safety research. This is not to say that there aren’t times when oversimplifications might be appropriate and even necessary. For example, oversimplification may be required in a scientific discipline when it has not yet developed the theories or sophistication to better reflect the particular situation [23]. However, in my opinion, this is not the case in today’s epidemiology, where more sophisticated means of assessing bias exist [24].

An additional example in which oversimplification may be necessary is in teaching situations where it may function as ‘a heuristic way station to more adequate subsequent treatment’ [23]. However, it is my experience that the RCT gold standard paradigm is used beyond the entry-level classroom in venues such as medical journals, the formulation of medical protocols, and in courtrooms. In these practice venues, it is best to ditch the oversimplifying cliché of evidence hierarchies and instead judge each study on its own merits. To do otherwise does a disservice to discourses about drug safety and, ultimately, to patients.

Abbreviations

ITT	=	Intention-to-treat;
RCT	=	randomized controlled trial;
PPI	=	proton pump inhibitor

Declaration of interests

BB Gerstman serves as an expert witness on behalf of the plaintiffs in Proton-Pump Inhibitor Multi-District Litigation 2789, United States District Court, District of New Jersey. He has received no fees for writing this commentary and has not shared this manuscript before its publication. The author has no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Additional information

Funding

This paper was not funded.