Assessing the applicability of obstetrical randomized controlled trials in real-world practices

Abstract

This article examines the applicability of obstetrical randomized controlled trials (RCTs) in the real-world and proposes a classification of the value of these trials based on their potential for achieving sustainable practices. In the context of this discussion, real-world results pertain to the potential impact of the RCT on sustainable interventions and practices, and its implications for healthcare practice or policy, in the country (or countries) that was conducted. While RCTs are generally regarded as the gold standard of medical evidence, their effectiveness in producing meaningful real-world results depends, among various other factors, on the clarity and specificity of the trial definitions used for diagnosis (characteristics of the study group or enrollment criteria) and treatment (intervention). The definitions used for diagnosis and treatment, especially in pragmatic trials, can influence the likelihood for real-world implementation. By analyzing notable obstetrical RCTs, the authors find that trials with well-defined diagnoses and treatments that can be implemented without specialized expertise are more likely to generate results that are relevant to general practice, indicating higher value. In contrast, RCTs with ambiguous or undefined diagnoses and treatments often lead to variations in practice and produce unreliable real-world outcomes and practices suggesting lower value. Recognizing this variability can offer valuable guidance for the design and evaluation of RCTs in obstetrics.

Keywords:

• Randomized controlled trials
• real-world practices
• classification
• value-based
• taxonomy

Introduction

Randomized controlled trials (RCTs) are widely regarded as the gold standard of medical evidence. They offer the advantage of minimizing confounders between randomized groups, assuming an effective randomization process. To ensure the quality and validity of RCTs, certain critical elements need to be considered, including registration prior to patient enrollment, proper recruitment, implementation, appropriate sample size determination, statistical methods, assessment of limitations, and correct interpretation. Additionally, trials may incorporate proper concealment (blinding process) to minimize bias. For an RCT to influence practice, the nature of outcome(s) used is very important with respect to the clarity and clinical relevance. RCTs with large sample sizes and outcomes that align with clinical relevance and patient needs are more likely to drive practice changes. However, clear definitions of diagnoses (characteristics of the study group or enrollment criteria) and intended treatments (interventions) are also crucial to maximize internal validity, reliability, and potential implementation in clinical practice. Unfortunately, ethical, practical, and economic considerations often complicate human trials, leading to the use of incomplete or unspecified definitions for diagnosis, or treatments requiring special expertise, which can interfere with proper real-world application. Another issue to consider is also the possible occurrence of failed randomization which can lead to an imbalance in confounders at baseline, which can further complicate the interpretation and applicability of the RCT findings [1]. The potential for clinical applicability of failed RCTs may not be significantly different from that of observational studies which can also be affected by imbalances in baseline confounders requiring statistical adjustments.

This Clinical Opinion aims to examine several landmark obstetrical RCTS which have used outcomes of clinical relevance and highlight the lessons learned with respect to their sustained clinical applicability in real-world practices. These RCTs were selected due to the sufficient elapsed time since their publication, allowing for a more comprehensive assessment of their applicability in obstetric practice. Thus, we aim to introduce the concept that not all RCTs are equal in their potential for applicability and propose different levels of likelihood for their results to be applied in real-world practices, reflecting their value (value-based taxonomy). While adherence to registration and the CONSORT statement are essential, the practical value of an RCT should also be evaluated based on the likelihood of its findings translating into real-world practices. The gold standard for achieving real-world results should be the widespread acceptance and sustainable implementation of the RCT findings in practice. Our hypothesis is that in obstetrical RCTs with outcomes of significant clinical relevance, the likelihood of achieving sustainable real-world practice results depends on how the diagnosis and treatment elements are defined in the RCT.

Value-based taxonomy of RCTs

Diagnoses (characteristics of the study group or enrollment criteria) and treatments (interventions) in an RCT can be categorized as (i) requiring no special skills, (ii) requiring special skills, or (iii) not defined. The hypothesis suggests that variations in practice may compromise the applicability of an RCT’s findings when it comes to real-world scenarios. Diagnoses or treatments that do not require special skills are considered of the highest value, as they can be easily applied and minimize variations in practice. Diagnoses or treatments requiring specialized expertise may lead to some practice variations due to the limited availability of such expertise. RCTs with diagnoses or treatments that are “not defined” have the highest potential for practice variations and thus lower value in terms of real-world practice expectations. This categorization introduces a tool for evaluating the value of RCTs by estimating the likelihood of achieving real-world outcomes that resemble those observed in the trial.

Table 1 presents ten possible combinations of RCT values based on the definitions used for diagnosis and treatment. The ranking of each combination, from highest to lowest, reflects the likelihood of achieving real-world results and, thus the value of the RCT. The highest likelihood (level 1 value) occurs when both the diagnosis and treatment do not require any special skills, minimizing practice variations and closely mirroring real-world scenarios. Most randomized controlled drug trials are in this category. Drug administration, regardless of the route, is not considered as a special skill. In contrast, RCTs with undefined diagnoses and treatments that are “left up to the clinician” are vulnerable to significant practice variations and unexpected results. RCTs that define certain aspects of the diagnosis or treatment but not in their entirety should be assessed to determine if the primary outcome could have been influenced by variations resulting from the non-defined parts. The lowest value ranking (level 10) pertains to RCTs with failed randomization, characterized by atypical placebo groups, which may result from chance or sampling bias errors, rendering the sample non-representative. Although failed randomization may be technically viewed as a separate category because it is not linked to definitions of diagnosis or treatment, we believe that it deserves a (last) place in the taxonomy, from the value-based point of view, since statistical adjustments for baseline differences are necessary, much like those needed for an observational study. Failed RCTs, are given the lowest value in our proposed scoring system which could be used as a check and balance system in the future to prevent premature adoption of low-value RCT science into practice.

Table 1. Value-based ranking of randomized controlled trials (in descending order) based on the likelihood to be applied to general practice.

Level	Diagnosis	Treatment/intervention
1	No special skills required	No special skills required
2	No special skills required	Special skills required
3	No special skills required	Not defined
4	Special skills required	No special skills required
5	Special skills required	Special skills required
6	Special skills required	Not defined
7	Not defined	No special skills required
8	Not defined	Special skills required
9	Not defined	Not defined
10	Failed randomization process	Not Applicable

Value of prominent obstetrical RCTs

The proposed value-based RCT classification (Table 1) can provide a reasonable explanation for the varying success of obstetrical trials in real-world applications. Here, we describe the classification and associated real-world results of prominent obstetrical RCTs which have used outcomes of significant clinical relevance. These RCTs have focused on antenatal corticosteroid administration, premature rupture of membranes, ultrasonography, electronic fetal heart rate monitoring (EFM), and prevention of preterm birth.

One successful obstetrical practice supported by RCT results is the prenatal administration of antenatal corticosteroids to enhance fetal lung maturity [2] although it took a significant amount of time for this practice to be widely adopted in routine obstetrical practice. These trials, which have led to decreased neonatal morbidity and mortality in preterm infants, demonstrate a high likelihood of producing real-world results (value-based level 1). The methodology used in these trials is easily applicable in practice, as the diagnosis (e.g. gestational age estimation) and treatment (injections) do not require special skills although the fact that the diagnosis of true preterm labor has its own challenges. Another example of value-based level 1 value is the RCTs involving the use of antibiotics in patients with preterm rupture of membranes [3] that had clear diagnoses (gestational age estimation and preterm premature rupture of membranes) and treatments (antibiotic administration) without the need for special skills. The results of these RCTs and their meta-analysis [3] continue to be the standard of care over the last 20 years. A more recent example of value-based Level 1 RCT is the Chronic Hypertension and Pregnancy (CHAP) trial [4] which used clear blood pressure criteria for enrollment (i.e. diagnosis) and treatment (antihypertensive medication) both of which do not require special expertise; the results of this RCT were adopted quickly into guidelines [5].

In the case of vaginal progesterone for patients with short mid-trimester cervical length, an RCT demonstrated a reduction in preterm birth and improved neonatal outcomes [6]. While the diagnosis of a short cervix required special ultrasound skills, the treatment of self-administered vaginal progesterone gel did not. This RCT (value-based level 4) produced results that have become the standard of care for the past 11 years. Similarly, the cerclage RCT for high-risk patients with a short cervix required special skills for diagnosis (transvaginal ultrasound) and treatment (cerclage) [7]. This RCT (value-based level 5) showed a reduction in pre-viable births and perinatal mortality, and prevention of preterm births <35 weeks when cervical length was <15 mm, leading to the adoption of this practice for the past 14 years [7]. In the aforementioned examples, while special skills such as transvaginal ultrasound and measuring cervical length were required during the conduct of the RCTs, they have become more widely available with subsequent training.

Conversely, there have been prominent obstetrical RCTs in ultrasonography and intrapartum EFM whose conclusions did not align with subsequent real-world practices. Despite negative findings, the diagnostic tools used in these trials became widely used. The RADIUS trial, conducted over 30 years ago, assessed the benefits of ultrasound screening [8,9]. Although the trial suggested that screening ultrasonography did not improve perinatal outcomes, the trial was conducted at a time when ultrasound expertise and image quality were limited. Furthermore, the treatment decisions based on ultrasound findings were left up to the clinical judgment of physicians [8,9], which further contributed to the disparity between the trial results and real-world practices (value-based level 6). The challenges regarding the interpretation and application of the RADIUS findings in real-life were eloquently addressed in an editorial by Dr. Romero [10] the same year that the trial was published. The outcome was that the RADIUS negative results were dismissed in the real-world. Presently, even low-risk patients routinely undergo multiple ultrasound exams, including in the first trimester [11,12], as obstetrical ultrasound benefits have become widely recognized [13].

Intrapartum EFM trials also exhibited discrepancies between their negative findings and subsequent real-world practices [12–19]. In addition to small sample sizes [14–18,20,21], the diagnosis of fetal intolerance to labor was not consistent in these trials, the interpretation of the fetal heart rate tracings were left up to experts or “senior consultant obstetricians” [14,16,18–21] and treatment decisions were left to individual clinicians [14,15,17–19]. The value-based levels of these trials ranged from 7 to 9. Today, the use of intrapartum EFM is considered the routine standard of care in both the United States [22] and Europe [23], even among low-risk patients [24].

Lastly, two RCTs involving the use of weekly intramuscular injections of 17-alpha-hydroxyprogesterone (17-OHP) for the prevention of recurrent preterm birth produced opposite results [25,26]. The Meis trial suffered from failed randomization, resulting in an imbalance of baseline risk factors between the comparison groups, thus requiring statistical adjustments for baseline differences [25]. As a result, the trial’s findings were not reflective of sustainable real-world outcomes, leading to a value-based level of 10. Unfortunately, despite the failed randomization, this intervention was adopted into practice for a few years due to multiple pressures. In contrast, the PROLONG trial [26], which demonstrated no benefit of weekly 17-OHP injections, had balanced comparison groups and well-defined diagnosis and treatment, resulting in a high likelihood of real-world results (value-based level 1). Following the PROLONG trial, the manufacturer and FDA removed from the market 17-OHP for the prevention of recurrent preterm birth [27].

Factors impacting real-world clinical practices beyond RCT diagnosis and treatment

Applying the proposed value-based RCT classification to prominent obstetrical trials may help explain the variation in their real-world applications. However, it is essential to emphasize that factors other than RCT diagnosis and treatment may influence the likelihood for real-world practices. These factors include the choice of outcome measures, challenges with implementation, the potential for harm, as well as issues related to “pragmatic” designs.

The choice of outcome measures used in an RCT can have a significant impact on the likelihood of the results becoming common practice. The use of actual patient-centered outcomes such as reduced maternal or perinatal mortality, reduced hospital readmissions, improved quality of life, are more likely to lead to changes in clinical practice. On the contrary, outcomes that are less directly related to patient well-being or are considered surrogate endpoints (e.g. low Apgar scores or neonatal resuscitation without considering the longer-term health or developmental outcomes) or process measures (i.e. number of prenatal visits) may require additional evidence or validation to influence practice. A common characteristic of all the obstetrical RCTs that we examined was the use of outcomes with significant clinical relevance. Outcomes such as intrapartum stillbirths, neonatal deaths, perinatal mortality, substantial neonatal morbidity (inclusive of neurological symptoms, and neonatal infection) and significant maternal morbidity [2–9,14–21] or prevention of preterm birth and its related consequences [6,7,25,26] are of great clinical relevance. Therefore, it seems that our taxonomy, which, relied on RCT diagnosis and treatments, was not confounded by variations in outcome significance in the RCTs that we reviewed. Some obstetrical RCTs may have used unclear outcomes, such as composite outcomes, which however may be necessary due to the rarity of certain obstetrical complications; in these cases, although it may be challenging to interpret the overall impact of the intervention, there is no evidence to suggest that this issue has compromised the likelihood of adopting the results of the obstetrical RCTs that we reviewed.

Another issue to consider is the process of implementing the RCT results into practice. Some implementation challenges may include ability to produce and disseminate practice guideline documents, resistance to change, lack of awareness, issues related to patient access and equity, resource constraints, organizational or regulatory/legal barriers, patient preferences and the possibility of harm associated with the interventions tested in RCTs; examples of RCTs with possible harm may include the performance of cordocentesis in fetal alloimmune thrombocytopenia cases [28,29] which carries a significant risk for fetal loss due to exsanguination [30]. Another example of possible harm is the open hysterotomy, used in the MOMS trial [31], with its associated significant risk for maternal complications [32]. Nevertheless, there is no evidence to indicate that any of the implementation issues mentioned above had a negative effect on the real-world adoption of the results from the obstetrical RCT trials we reviewed.

Our proposed taxonomy may have some implications for RCTs with “pragmatic” designs. Pragmatic trials are intended to assess the effectiveness of interventions in real-world settings, but distinguishing between an explanatory and a pragmatic trial can be challenging, as many trial designs seem to exist on a continuum [33]. There is a balance between the degree of liberalization of the criteria used for diagnosis and treatment in a pragmatic trial and the likelihood for real world application of the RCT results. In our view, allowing practitioners to exercise their own judgment or protocols in diagnosis and/or treatment aligns theoretically with the requirements of a “pragmatic” study design. However, this “pragmatic” approach is applicable only to the specific population under study and may not apply to the broader diversity of practices found in the real world. This holds especially true for RCTs conducted in academic centers, which, by definition, are likely to involve different populations of patients and practitioners, as compared to the real-world. In our taxonomy of likelihood to become general practice, obstetrical pragmatic RCTs with treatments left up to the practitioners have value-based levels 3, 6 or 9; and those RCTs with diagnoses are left up to the practitioners have value-based levels 7, 8 or 9.

We found that obstetrical trials with well-defined diagnoses and treatments that do not require special skills tend to have a higher likelihood of producing sustainable real-world results. On the other hand, trials with failed randomization or undefined diagnoses and treatments may not accurately reflect real-world practices and should be approached with caution.

Conclusions

Based on our review of prominent obstetrical RCTs, it is evident that not all RCTs are equal in terms of their likelihood of producing real-world practices. The value-based taxonomy we propose, which categorizes RCTs based on expected real-world outcomes, may provide valuable insights into the potential success of these trials for real-world application. The concept of ranking RCTs aligns with the principles of pragmatism that suggest a spectrum of real-world applicability in routine practices rather than a strict binary distinction, as has been pointed out by previous investigators [33]. According to our proposed taxonomy (Table 1), the reviewed obstetrical trials with value-based levels 1–5 consistently yielded results that aligned with real-world practices, while trials with value-based levels 6–10 produced conclusions not consistent with subsequent real-world practices. Therefore, in the design of future RCTs and in evaluating the value of existing ones, it is crucial to consider the likelihood of achieving real-world results. This assessment can be made by examining the taxonomy of diagnosis and treatment used in the RCTs. For future RCTs, the criteria, if special skills are required for diagnosis or treatment/intervention, may be determined by investigator consensus before finalizing the study design; these criteria may vary depending on the scope of the randomized controlled trial (RCT) in relation to the specific patient and provider characteristics. It is our hope that further research determines the extent to which our proposed RCT value-based taxonomy in this Clinical Opinion correlates with sustainable real-world practice results.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Dr. Ananth is supported, in part, by the National Heart, Lung, and Blood Institute (grant R01-HL150065) and the National Institute of Environmental Health Sciences (grant R01-ES033190).