Improving the Efficiency of Randomized Trials: The DYNAGITO Example

Abstract

A traditional principle of the randomized controlled trial is that the p-value comparing treatment groups should be computed from the simple crude data. This assumes that the size of the study is sufficiently large for an accurate and precise estimation of the difference between groups. There are situations where the accuracy and precision of the study can be improved by statistical adjustment for baseline covariates, including when the outcome is continuous and available at baseline. The European Medicines Agency (EMA) and the Food and Drug Administration (FDA) recommend covariate adjustment in trials as a tool to improve the efficiency of the analysis and to avoid conditional bias from chance covariate imbalance. The recent DYNAGITO randomized controlled trial compared the effectiveness of the single inhaler tiotropium-olodaterol combination for the treatment of COPD with tiotropium alone in reducing the rate of exacerbations over 52 weeks. The pre-specified crude analysis produced a rate ratio of 0.93 (p-value > 0.01, not significant) comparing the tiotropium-olodaterol combination with tiotropium alone. However, a sensitivity analysis adjusted for the baseline rate of exacerbations and other factors produced a rate ratio of 0.89 (p-value 0.001, significant). The DYNAGITO trial is an example where statistical adjustment was justified but not used. Future trials of COPD therapy could be designed more efficiently, be made less costly, while improving the accuracy and precision of their data and the resulting conclusions.

Keywords:

• COPD treatment
• data analysis
• randomized controlled trials
• research methods
• study design

Introduction

The DYNAGITO randomized controlled trial assessed whether the combination of the long-acting muscarinic agent (LAMA) tiotropium and the long-acting beta-agonist (LABA) olodaterol in a single inhaler for the treatment of chronic obstructive pulmonary disease (COPD) reduces the annual rate of exacerbations more than tiotropium alone over 52 weeks [1]. The trial enrolled 7880 patients with COPD and a history of exacerbations who were randomly assigned to receive tiotropium-olodaterol (5 μg–5 μg) or tiotropium (5 μg) once daily. The study was powered to detect a 12% reduction in the rate of exacerbations with tiotropium–olodaterol compared with tiotropium. This power calculation was based on a two-sided significance level of 0.01 which was selected for regulatory purposes because, for results to be included in a product label, regulators require stronger proof of an effect than the usual 0.05 level if these results are based on one rather than two trials. The primary endpoint, namely the annual rate of exacerbations, was analyzed using a negative binomial model that included only the effect of treatment, with exposure time as an offset.

The rate of moderate or severe exacerbations with tiotropium-olodaterol was 0.90 per patient-year compared with 0.97 with tiotropium during the 52-week treatment period. The resulting rate ratio was 0.93 (95% CI 0.87–1.00; p-value 0.0498) for tiotropium-olodaterol compared with tiotropium. The targeted p-value of 0.01 was thus not met and the authors concluded that “combining tiotropium and olodaterol did not reduce exacerbation rate as much as expected compared with tiotropium alone” [1].

The methodological problem

The DYNAGITO trial specified that the primary data analysis was crude, with the negative binomial model including only one factor in the model, the allocated treatment. As noted by the authors in the Appendix, other similar trials in COPD all based their evidence on the adjusted, not the crude, analyses [2–4]. Indeed, with the rate of exacerbation as the primary outcome in most recent trials of COPD therapy, the main data analysis is typically based on models that include several pre-specified baseline factors strongly associated with exacerbations, with key ones such as symptom score, lung function and particularly exacerbation history.

The authors of the DYNAGITO trial performed several sensitivity analyses to adjust for different baseline covariates, mimicking the primary analysis presented in several previous trials. For example, the analysis resembling that of the FLAME trial involved adjusting for baseline data on smoking status, inhaled corticosteroid (ICS) use, GOLD stage, region, symptom score and exacerbations treated with antibiotics or steroids [2]. This resulted in an adjusted rate ratio of moderate or severe exacerbations of 0.89 (p-value 0.0010) with tiotropium–olodaterol compared with tiotropium, in contrast with the crude rate ratio of 0.93 (p-value 0.0498). The adjusted analysis would have resulted in a p-value < 0.01, the pre-set significance level selected for regulatory purposes, and thus a “significant” finding.

The methodological recommendations

A traditional principle of the randomized controlled trial has been that the p-value comparing the two groups should be computed from the simple crude data arising from the two groups. This is the approach taken by the authors of the DYNAGITO trial. This principle assumes that the size of the study is sufficiently large that the precision of the study will permit to detect the desired differences between the two groups. However, there are situations where the precision of the study can be improved not only by increasing the size of the study but by improving the technique of data analysis by statistical adjustment for baseline covariates. The International Conference on Harmonization (ICH) guidelines recommend that “pre-trial deliberations should identify those covariates and factors expected to have an important influence on the primary variable(s), and should consider how to account for these in the analysis in order to improve precision and to compensate for any lack of balance between treatment groups” [5]. The European Medicines Agency (EMA) also addresses covariate adjustment in trials, as a tool to improve the efficiency of the analysis and to avoid conditional bias from chance covariate imbalance [6]. For trials of COPD treatment, such as DYNAGITO, two recommendations are particularly relevant:

1. Variables known a priori to be strongly, or at least moderately, associated with the primary outcome and/or variables for which there is a strong clinical rationale for such an association should also be considered as covariates in the primary analysis. The variables selected on this basis should be pre-specified in the protocol.

2. If a baseline value of a continuous primary outcome measure is available, then this should usually be included as a covariate. This applies whether the primary outcome variable is defined as the ‘raw outcome’ or as the ‘change from baseline’.

In the DYNAGITO trial, the primary data analysis was crude, namely a simple comparison of the two rates of exacerbations by a negative binomial distribution. This outcome measure, however, should be an excellent candidate for covariate adjustment as per the EMA guidelines. Indeed, as per the first recommendation, the rate of exacerbations is known to be highly associated with the presence of previous exacerbations and other factors, including age, sex and FEV₁% predicted [7–10]. Moreover, as per the second recommendation, the rate of all moderate and severe exacerbations occurring over time can be considered as a quasi-continuous outcome. Moreover, a measure of this outcome is generally also available during the baseline period, though it is typically analyzed in COPD trials directly as a raw exacerbation rate rather than as the change in the exacerbation rate from baseline. In DYNAGITO, while all patients had at least one exacerbation in the baseline year, 45% had 2 or more (or at least one severe) exacerbations. These are frequent and likely highly correlated with the subsequent rate of exacerbations during follow-up. Thus, even modest imbalances in such strong risk factors after randomization can introduce small biases in the estimated rate ratio, and affect the precision of its estimate. Simply adjusting for this single factor would have likely improved the accuracy and precision of the rate ratio and reduced the corresponding p-value.

Illustration

As noted in the Appendix of the DYNAGITO paper, other similar trials all based their evidence on the adjusted, not the crude, analyses [1]. These trials adjusted for several pre-specified baseline factors including key ones such as symptom score, lung function, as well as exacerbations in prior year. The DYNAGITO authors used three different adjustment models to estimate adjusted rate ratios of exacerbations and p-value.

To illustrate the impact of statistical covariate adjustment, we converted the reported p-values for the three different adjusted rate ratios of exacerbations into 99% confidence limits using log transformations. Figure 1 displays the forest plot of the rate ratio of exacerbations comparing tiotropium-olodaterol with tiotropium along with 99% confidence limits (corresponding to the pre-specified 1% significance level) for the crude analysis and for three adjusted analyses according to different covariate choices used in previous trials. Of note is the reduction of the rate ratio from 0.93 to 0.89 in the two models that included the baseline measure of the outcome, namely “prior exacerbations,” as a covariate. In addition, with adjustment, the standard error of the logarithm of the rate ratio using the crude analysis was 0.0344, which was reduced to 0.0315 with the first adjustment model. This reduction of over 9% in the standard error translates to an approximate increase of 19% in the sample size necessary to reach this lower standard error. Thus, the DYNAGITO sample size of 7,880 with an adjusted analysis would need to be increased to around 9,377 to attain the same precision with a crude analysis.

Figure 1. Crude and adjusted rate ratios of exacerbations comparing tiotropium–olodaterol with tiotropium along with 99% confidence limits from the DYNAGITO trial, using various adjusted analyses according to different baseline covariate choices used in previous trials: ¹ Adjusted for smoking status, ICS, GOLD stage, region, CAT score, treated exacerbations in baseline year. ² Adjusted for age, sex, smoking status, LABA/ICS use, region, % predicted FEV₁. ³ Smoking status, airflow limitation, region, treated exacerbations in baseline year.

Conclusion

The randomized trial is the fundamental study design to evaluate drug effectiveness and receive approval from regulatory agencies. Beyond randomization, several methodological decisions remain to improve the validity, accuracy and precision of the results on which the conclusions will be based. We described how the DYNAGITO trial, comparing the dual tiotropium-olodaterol long-acting bronchodilator with tiotropium alone, did not use the most efficient method of data analysis for its primary outcome involving the number of exacerbations during follow-up. By not adjusting for baseline covariates, the resulting rate ratio of 0.93 did not reach the pre-specified significance level, but would have with statistical adjustment. The sample size of a trial would need to be increased by almost 20% to attain the same precision with a crude analysis as an adjusted one with lower sample size. Moreover, the 7% reduction in the rate of exacerbations with the dual long-acting bronchodilator would have improved to 11% with statistical adjustment. These effects of statistical adjustment with equally affect the accuracy and precision of the rate difference, if this is the measure of interest.

The unusual decision of the DYNAGITO trial authors to base the primary analysis on the crude rate ratio was particularly unexpected since the authors of most of the recent trials of combined inhalers used adjustment in their primary analysis involving the number of exacerbations as a primary outcome. As specified in both the EMA and FDA guidelines, adjustment is recommended in this context. Future trials of COPD therapy could be designed more efficiently, be made less costly, while improving the accuracy and precision of their data and the resulting conclusions.

Conflict of interest

Pr. Suissa and has participated in advisory board meetings, as speaker or received research funding from Boehringer-Ingelheim and Novartis.

Acknowledgments

Pr. Suissa is the recipient of the Distinguished James McGill Professorship award.