Interpreting Results from Clinical Trials: Understanding Minimal Clinically Important Differences in COPD Outcomes

Abstract

This issue of the COPD: Journal of Chronic Obstructive Pulmonary Disease presents the papers from the “Workshop on Minimal Clinically Important Differences in COPD” held in Bal Harbour, Florida, on January 11–13, 2004. The goals of this meeting were to: discuss the significance of and methodology for determining minimal clinically important differences (MCID) in health outcomes of importance in chronic obstructive pulmonary disease (COPD); review the current knowledge about MCID in outcomes of importance, including dyspnea, quality of life, exercise performance, and pulmonary function, in the management of patients with COPD; apply methodologies to determine MCID using existing data for outcomes of importance in COPD; and develop recommendations for further studies to determine MCID in health outcomes of importance in COPD.

Introduction

Clinicians are bombarded with results of clinical trials published in peer-reviewed medical journals. The growing emphasis on evidence-based medicine requires that clinicians and other decision makers throughout the health care system assess the significance of published investigations and base therapeutic, regulatory, and reimbursement decisions on results from carefully controlled trials. Because these decisions have implications for patient care, there is a great deal of interest in going beyond the statistical significance of clinical studies to understand and quantify the clinical benefit offered by the treatment under investigation. This symposium was convened to discuss methods for determining the clinical significance of treatment outcomes of importance to the care of patients with chronic obstructive pulmonary disease (COPD).

Which outcomes are potentially important in clinical trials in patients with COPD and how are these outcomes clinically interpreted? Investigations in COPD now routinely assess not only physiologic outcomes but also patient-centered outcomes including respiratory symptoms, function, and health-related quality of life. Table 1 offers a framework for identifying the types of outcomes of importance in COPD. In organizing this workshop, we brought together methodologists with experience in the development of interpretive guidelines for new outcome measures and recognized experts with an understanding of the current state of knowledge of the clinical interpretation of key outcome measures in COPD to discuss the state of the science in this disease and make recommendations for further research. Stephen Rennard’s paper offers a view of the problem of interpretation from the clinician’s perspective [[1]]. The importance of the concept of minimal clinically important difference to patients and health care providers is further highlighted in papers by James Kiley et al. of the National Heart, Lung, and Blood Institute [[3]] and Robert Meyer of the Food and Drug Administration [[4]]. The current state of the science in interpreting key outcomes in COPD, including quality of life, symptoms, pulmonary function, exercise, exacerbations, body mass, and muscle strength is discussed by experts in each of these areas.

Table 1.  Outcomes of Importance in COPD.

•FUNCTION	•DISEASE
Pulmonary physiology	Pathophysiology
Functional capacity (exercise)	Pathology
Neuropsychological function	Radiology
•PATIENT EFFECTS	•OTHER
Respiratory symptoms	Medications/therapies
Quality of life	Costs
	Adverse effects
•CONSEQUENCES	Caregiver burden
Exacerbations	Healthy behaviors
Morbidity (hospitalizations)	Assistive technology need
Mortality	Satisfaction with care

In this editorial, we provide an overview of issues and methods for determining numeric values indicative of clinically meaningful treatment effects, highlighting insights offered by the contributing authors in this issue of the Journal of COPD.

Statistical Vs. Clinical Methods of Interpretation

The traditional gold standard method for interpreting results of clinical trials is a determination of whether or not results are statistically significant, e.g., whether an observed difference could have occurred by chance alone. A probability (p) value of less than 0.05 (meaning that the difference in results would have occurred by chance less than 5% of the time), is widely accepted as the line of demarcation for a “statistically significant” result, with values < 0.01 often used in studies involving high-risk treatment. This statistical hypothesis testing approach leads to a dichotomous decision point: either accept null hypothesis (p > 0.05, the result was not statistically significant, and a conclusion is made that there is no treatment effect); or reject the null hypothesis” (p < 0.05, the result is statistically significant, and a treatment effect was detected).

It is important to note that in health care, statistically significant effect does not necessarily equate with clinically meaningful benefit. Clinicians and others involved in health care policy clinical decision making must consider more than the fact that the criterion of statistical significance was met when interpreting treatment effects seen in clinical trials and deciding on the extent to which a given therapy is likely to have a true therapeutic benefit. The phrase “minimal clinically important difference,” often abbreviated “MCID,” incorporates the concept of clinically meaningful benefit to the patient and was originally defined as “the smallest difference in score … which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management” [[2]]. Although many investigations involve therapies that do not have significant adverse effects and costs, the article by Make and Sutherland on exercise outcomes of lung volume reduction surgery highlights the importance of taking potential adverse effects, complications, and costs of a treatment involving major surgery into account in interpreting a given outcome [[5]]. Our view is that an estimation of clinical benefit and decisions concerning the adoption of new treatments for any given outcome need to take into consideration not only the magnitude of the effect, but the nature of the intervention and the specific population of patients under study (for example, patients with differing degrees of disease severity or differing severity of symptoms). MCID values for a given outcome may be different for a treatment that carries a high degree of risk in a mild population than MCID values for a treatment with no risk employed in a severe patient population. Thus, a certain level of “tolerance” is necessary when dealing with MCID estimation; i.e., there should be an allowed amount of variation in the MCID rather than exact conformity to a specific MCID in all clinical settings. In fact, we recommend that clinical outcomes be interpreted with ranges for MCID, rather than universally applying a single MCID cut off value. Developed and used appropriately, interpretive guidelines that incorporate carefully constructed values for the clinical interpretation of treatment effects and allow for tolerance levels can be used by physicians, patients, health care regulators, insurers, and society to determine the relative desirability of management options.

In addition to the nearly universally accepted set points (p < 0.05; p < 0.01) and the dichotomous decision point (accept or reject the null hypothesis) characteristic of statistical significance levels, these empirical decision tools are also characterized by their generic applicability. They are applied across all endpoints. This is not the case for clinical interpretation; there is currently no single MCID or set of MCIDs upon which to base clinical interpretation. Interpretive guidelines and MCID values are expressed in outcome-specific units—clinically significant changes in expiratory flow (FEV1) are expressed in liters; improvements in 6-minute walk distance (6-MWD) in feet or meters; and changes in St. George’s Respiratory Questionnaire (SGRQ) are interpreted in terms of “units.” This seems both logical and desirable. After all, determining clinically meaningful change or differences is a clinical question requiring a clinical answer—in the outcome unit of concern. On the other hand, what if there was a simple, easy to apply method for estimating the MCID—one that could be easily applied by clinicians as they review a published study to get at least a general sense of the possible clinical benefit of treatment? In a paper in this issue, Sloan discusses a pragmatic, unified approach to estimating the MCID originally proposed by Norman et al. [[6]], that suggests that one-half standard deviation of the outcome variable’s baseline value provides a simple estimate of the MCID in the absence of further information [[7]]. Although the MCID unit will be outcome specific, the method for estimation would be universal. Intriguing? Yes. However, in our view there is insufficient data to support the link between this simple mathematical computation and indications of the clinical benefit of treatment.

Methods for Estimating MCID

There are a number of methods for estimating MCID, many of which are used in the papers included in this issue. As you read these papers, you will notice some variance in terminology and application, an accurate reflection of the state of the science in the field. In general, it is useful to think about these methods within the context of three general categories (mathematical, opinion, and external measurement):

Mathematical or Statistical Methods

These methods for estimating the MCID are often referred to as “distribution-based” approaches [[8]]. These techniques are based on the distributional properties of a given instrument in an untreated population or sample and consider the signal and noise associated with measurement. For example, if the values of an outcome measure in an untreated population fall within a narrow range (small standard error of measurement with repeated measurements), then a small change may represent a clinically important difference. Alternatively, it is logical to assume that if the range of measurements is wide, then a larger change would be necessary to connote a clinically significant treatment effect. Mathematical or statistical methods for estimating the MCID include standard error of measurement (SEM), standard deviation (SD), standard response mean (SRM), and the responsiveness statistic (RS).

Some investigators suggest that specific distribution-based approaches yield values remarkably close to other methods of assessing the MCID (4, Hays, 2005 #85). In this issue, Norman highlights differences between MCID in individual patients and group effects; he posits that effect size (which can only be determined in the context of a treatment rather than from an untreated population) is more important than the MCID [[8]]. Several authors discuss the utility of the half standard deviation estimation approach (6, Hays, 2005 #85, Sloan, 2005 #1557). The absence of a clinical component to mathematical approaches to estimating MCID is noteworthy. As Hays et al. point out, these methods express observed change in a standardized metric, but do not provide information about what size of change is minimally important [[9]]. To make certain interpretive guidelines or MCID values are truly clinically meaningful, a clinical dimension or link should be evident. A generally shared consensus at the workshop was that, although the distribution-based approach is useful in providing an estimate of the MCID for purposes of study design, this approach alone is insufficient to interpret the results of clinical trials.

Opinion

This approach uses an assortment of qualitative and quantitative methods with physicians and/or patients to arrive at group consensus regarding MCID values for a given outcome [[10]]. Although expert-based consensus on MCID values can provide important insights into outcomes, they have not been widely employed in COPD. In our view, opinions should also be sought and incorporated from a wide range of experts including nurses and behavioral scientists and, most importantly, must incorporate the end-user of the intervention, the patient with COPD.

External Measurement Methods

These approaches for estimating the MCID use an external criterion for classifying magnitude of change in the underlying patient population and apply this classification scheme to the outcome under investigation to arrive at an MCID for the new variable. The most common external measurement approach, often referred to as “anchor-based” [[11]], is a physician or patient retrospective global rating of change. Briefly, patients or physicians participating in a treatment program are asked to rate the extent to which the outcome variable of interest changed. Data for the subgroup of patients experiencing little or minimal change on the outcome of interest are then used to estimate the MCID on this variable. This approach originated, and is used widely, in health-related quality of life (HRQL) research, where the patients themselves serve as important “anchors” of change in HRQL. Alternatively, the anchor could conceivably be another outcome measure; this approach is only useful when there is a universally accepted “gold-standard” outcome measure that can be compared to the outcome under investigation. Since there is no such “gold-standard” outcome in COPD, the outcome variable of interest might be compared to a variety of other outcomes, such as using HQOL and exercise capacity to assist in the interpretation of changes in pulmonary function.

The anchor-based approach using retrospective assessments of perceived change was one of the first methods for estimating MCID for health-related quality of life (HRQL) measures. This technique has been used as part of the MCID estimation programs for two commonly used disease-specific HRQL measures for COPD: the Chronic Respiratory Questionnaire (CRQ) and the St. George’s Respiratory Questionnaire (SGRQ). In this issue, Schünemann and colleagues [[12]] describe how anchor-based methods, specifically retrospective global assessments, have been applied to the CRQ, later complemented by distribution-based estimates for the MCID. They conclude that the use of the measure under a variety of circumstances and subjected to a variety of estimation methods leads one to confidently conclude that the minimal important difference for the 7-point domain scores of the CRQ is 0.5. Jones discusses the use of mortality or readmission to hospital as an anchor for estimating MCID values on the SGRQ [[13]]. Jones also applies expert consensus and distribution-based approaches to arrive at a conclusion that changes greater than 4 units on this 100-point scale may be judged “equivalent to a clinically significant effect.”

Identifying appropriate, clinically meaningful external measures or anchors for other important clinical outcomes in COPD, such as FEV1, exercise tests, walk of tests, dyspnea, exacerbations, or body weight, may be more difficult. Papersby Donohue [[14]]. Casaburi [[15]], Sutherland and Make [[5]], Wise and Brown [[16]], Ries [[17]], Mahler and Witek [[18]], Leidy and Wyrwich [[19]], Calverley [[20]], and Wouters [[21]] offer an overview of current thinking on the MCID for each of these outcomes, including, in some cases, data on external anchors for interpreting change. Using HRQL as an anchor for interpreting these outcomes may be a useful way to associate these clinical efficacy parameters to improvements in patient health and well-being. In fact, Kaplan proposes that quality adjusted life years (QALY), a form of generic HRQL, be used to define the MCIDs of other clinical endpoints [[22]]. HRQL is only part of the clinical story, however. Examining change and difference scores in light of other clinically relevant variables adds strength to the conclusion that a given value or range of values should be considered clinically meaningful from various perspectives. Existing large databases from the pharmaceutical industry and from other large, multi-center clinical trials such as the National Emphysema Treatment Trial would be useful for developing MCID guidelines for many outcomes of importance in COPD [[5]].

It is unlikely that any one approach or even a series of tests within a single method category, such as a series of mathematical methods, will yield values that accurately reflect clinically meaningful change or differences. It is useful, therefore, to apply several different methods in order to gain confidence that the recommended value or range of values is clinically valid. In this issue, Leidy and Wyrwich propose the use of triangulation, a systematic approach in which multiple tests from each of the three method categories are used to arrive at clinically useful interpretive guidelines for a given outcome [[19]].

Interpreting Outcomes in COPD

As shown in the articles in this issue, there is significant variability in the information available about the MCID of clinical variables commonly used to evaluate treatment outcomes in COPD trials. There is a significant body of information supporting an MCID for some outcome measures; limited or new and untested information to suggest an MCID for others; and questionable MCID values for still others, due to methodologic limitations or sheer volume of work performed to date. Thus the suggested MCIDs discussed in this workshop and summarized in Table 2 should be interpreted in light of the available information contained in the articles in this issue and with the caveat that these are point estimates with tolerance levels, rather than strict cutoff values. There are relatively few outcome measures for which MCID has been assessed by using all three methodologies outlined above. Through this series of papers, we hope to raise awareness of the importance of MCID and spur further investigations to more clearly define the MCID for outcomes in COPD.

Table 2.  Suggested MCID of Commonly Used Outcome Measures in COPD.

Outcome measure	Suggested MCID
Forced expired volume in one second [[14]]	100 ml.
Maximal exercise test [[4]]	10 watts**
Submaximal exercise endurance test [[15]]	≥ 1.75 minutes
Six-minute walk test [[16]]	37–71 meters
Transition dyspnea index [[17]]	1 unit
UCSD dyspnea questionnaire [[18]]	5–7 units
Borg scale of perceived dyspnea [[18]]	2 units
Visual analogue scale of dyspnea [[18]]	10–20
St. George's respiratory questionnaire [[12]]	4 units
Chronic respiratory disease questionnaire [[11]]	0.5 units
Quality of well-being scale [[19]]	0.03 units
Exacerbations [[20]]	22% change; 1 exacerbation/year
Body weight	Not available

These values are provided for informational purposes and should be considered estimates with tolerance levels, rather than definite cut-off values. The development of ranges for these estimates is recommended.

*Suggested MCID is based upon the articles in this issue of COPD: Journal of Chronic Obstructive Pulmonary Disease (COPD, 2005, Vol. 2, No. 1).

**In context of lung volume reduction surgery as an intervention.

We are pleased to able to edit this Workshop on Minimal Clinically Important Differences of Outcomes in Chronic Obstructive Pulmonary Disease, and thank the Journal of COPD for publishing the manuscripts. We hope the articles in this issue will provide guidance to investigators as they present and analyze the results of clinical trials in patients with COPD.

Acknowledgments

We gratefully acknowledge the assistance of the following MEDTAP scientists who reviewed and commented on the manuscripts appearing in this issue of Journal of COPD: Kathleen Beusterien, M.P.H., Julia Fox-Rushby, Ph.D., Lori Frank, Ph.D., Gale Harding, M.A., Miriam Kimel, Ph.D., Andrew Lloyd, D.Phil., Kimberly Niebauer, M.P.P., Anne Rentz, M.S.P.H., R.D.H., Charles Ruetsch, Ph.D., and Rich Shikiar, Ph.D.