Observational Studies in COPD: Summary of Guidance for Authors

Keywords:

• COPD
• observational study
• pharmacoepidemiology

Observational studies have played an important role in identifying risk factors associated with chronic obstructive pulmonary disease (COPD), as well as its natural history, and prognosis (1–3). These days, such study designs are used to expand this role by evaluating the effectiveness and safety of pharmacological therapeutic options for COPD, particularly in the context of real world clinical practice (4–8).

Traditionally, the quality of observational studies has been deemed poor in the pyramid of the quality of scientific evidence when compared with experimental studies, particularly because of their potential low internal validity (9). On the other hand, their external validity can be higher following their less restrictive selection criteria. Moreover, observational studies can provide important information on exposures potentially associated with adverse health effects as well as accurate measures of effectiveness (10).

Over the last two decades, important developments in observational research methods have happened in the field of epidemiology, with many of these advancements around the notion of “causal inference” (11). Causal inference focuses on the effects of an exposure on an outcome. It addresses questions of both etiology and prognosis of disease, such as whether exposure to biomass fuel use in women increases the incidence of COPD (12), whether the use of statins lowers mortality in patients with COPD (13), or whether inhaled corticosteroids (ICS) increase the incidence of community-acquired pneumonia in patients with COPD (14–16).

Because of the proliferation of observational studies with a causal inference perspective published in medical journals, and the variable methodological quality of many of these studies, the Editors of respiratory, sleep, and critical care Journals produced a document to “offer guidance to authors, peer reviewers, and researchers on the design and reporting of observational causal inference studies” (17). In the spirit of causal inference, this guidance focusses exclusively on the control of confounding bias, not on issues related to the other sources of bias in epidemiology, namely selection and information biases.

The document offers three key principles to guide authors in the design, analysis, and reporting of causal inference studies. The first relates to the control of confounding, while the other two relate to the interpretation of the results and the presentation of data. In this editorial, we briefly summarize the three key principles presented in the guidance document and add some discussion of issues relevant to selection and information biases, of particular importance to many observational studies in COPD.

Key principle #1: Carefully consider confounding

The paper explains how to define and select confounders in observational studies. A confounder is a “third variable” that is associated with the exposure under study and is associated with the outcome of interest, but does not reside in the causal pathway between the exposure and outcome. Randomized trials will inherently eliminate confounders since strict selection criteria and randomization remove the association between the confounder and the exposure. For observational studies, where there is no random allocation of exposure, the paper recommends a “historical approach” to selecting potential confounding variables whereby these variables are selected based on prior knowledge. The paper does not recommend selecting variables associated with the exposure or outcome in the study database. The paper also recommends the use of visuals, so-called directed acyclic graphs (DAGs), in which arrows represent known causal effects based on prior knowledge. The paper details this approach extensively.

As a general rule, the paper argues against selecting confounding variables based on statistical hypothesis testing or model fit, as they rely only on the available data. Using p-values or changes in beta-coefficients are also not recommended to identify confounders. Nonetheless, we think it is important to present a previously unknown or unsuspected factor that fully fits the confounder definition identified in the study database at the data analysis stage, and report its effect in some analyses. Finally, the paper recommends that the results should only present effect estimates for the exposures of interest, not for all effects from the statistical model designed to test a single causal association. This so-called “Table 2 fallacy” urges to avoid also presenting effects of the confounding factors, for example the relative risks of age, sex, co-morbidity, etc…, but rather only the effects under study.

Key principle #2: Do not rely on p-values

For the effects of interest, the paper recommends against the use of the p-value in causal inference, as these are frequently misinterpreted and misused. Because of the sometimes very large study sizes (or very small), the p-value provide no information about the magnitude, direction, or clinical importance of an association. It is recommended to present it only rarely in isolation, such as for tests of interaction. Instead, one should present effect estimates and measures of precision such as confidence intervals in addition to, or in lieu of, p-values.

In essence, it is important to interpret both the magnitude of the effect estimate and its variability when making conclusions about causal associations. The paper uses the example of a rate ratio of 2.1 with 95% confidence interval (0.97–4.2) and a corresponding p-value of 0.10. It argues that this effect estimate should not be reported as “no association” or “not significant” since a rate ratio as large as 4.2 cannot be plausibly ruled out. The paper recommends against using the labels “significant” and “non-significant” which tend to blur the distinction between statistical significance and clinical significance.

Key principle #3: Present data transparently

The paper recommends that authors adhere to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement when reporting results of observational studies that test causal associations. In particular, tabular presentation of results from cohort studies should include the number of events, person-time, incidence rates, and unadjusted and adjusted incidence rate ratios for each exposure level. The same with cross-sectional studies that estimate instead prevalence and prevalence ratios. Case-control studies should include the frequency and percent exposed and not exposed for cases and controls separately, and unadjusted and adjusted odds ratios for each case group.

Other issues: Selection and information biases

The guidance paper focused only on the control of confounding bias because causal inference essentially assumes that the study design is free of common biases due to selection and information. However, while controlling confounding is important to make exposure groups comparable, thereby emulating a randomized trial, this does not much matter if the selection mechanism skewed the study population and the exposure measures are inaccurate.

Selection and information biases have affected many observational studies in COPD, with a greater impact on skewing the results than from confounding bias. The most frequent and impactful biases have been time-related, including immortal time and immeasurable time biases. Immortal time refers to a period of cohort study follow-up during which the outcome under study cannot occur (18). Misclassifying exposure or excluding this “immortal time” will introduce immortal time bias (19). On the other hand, immeasurable time refers to a period during which the exposure of interest could not be measured, such as when the patient is hospitalized and inpatient exposure is not recorded (20). Using only the available outpatient exposure data will introduce immeasurable time and can affect both cohort and case-control studies (20).

For example, an observational study of the effect of ICS on mortality in patients with COPD, claiming to be “free of immortal time bias” in the title (21), was, in fact, affected by both immortal and immeasurable time biases (22, 23). It reported that ICS reduced mortality by 31% (hazard ratio 0.69: 95% CI: 0.52–0.93), but a re-analysis correcting for immortal time bias showed that the hazard ratio should have been 1.48 (22). These forms of selection and information bias plagued many studies on the effect of ICS on major COPD outcomes (23). Similarly, the effects of statins in reducing mortality and other major COPD outcomes suggested by several observational studies (24, 25) were explained by these time-related biases (26). The STATCOPE 3-year trial of simvastatin in COPD also refuted these claims (27, 28).

Conclusions

With the important recent developments in observational research methods in the field of epidemiology, we now have tools to properly design and conduct observational studies in COPD. These can play important roles in identifying factors associated with COPD occurrence, its prognosis, as well as in the assessment of the effectiveness and safety of pharmacological treatments for COPD in the context of real world clinical practice. The recent guidance document produced by the Editors of respiratory, sleep, and critical care Journals will provide some help to authors, peer reviewers, and researchers on the design and reporting of observational causal inference studies, especially regarding confounding bias, data interpretation, and reporting.

Beyond issues of confounding bias, one should pay proper attention to avoiding information and selection biases in observational causal inference studies. In particular, the design and analysis of observational studies should circumvent time-related biases, which tend to suggest falsely significant benefits of a treatment. Indeed, while confounding bias receives much attention, as the guidance document does, we often overlook time-related biases that resulted in reporting remarkable and exaggerated benefits for ICS, statins, and beta-blockers in COPD.

Disclosure statement

S. Suissa has participated in advisory boards, as speaker, or received funding from AstraZeneca, Boehringer Ingelheim and Novartis. G. Sotgiu has no conflict of interest to declare. V. Brusasco received research funds from Novartis and fees for speaking at meetings from Novartis, Menarini, AstraZeneca, and Lusofarmaco.