Patient-reported Outcome (PRO) Measures for Clinical Trials of COPD: The EXACT and E-RS

Abstract

The precise assessment of treatment efficacy in clinical trials requires scientific instruments that are not only relevant to the target population and treatment, but have been shown to be reliable, valid, and sensitive to change within the intended context of use. This paper describes the background, procedures, and current status of 2 patient-reported outcome (PRO) instruments developed for use in clinical trials of chronic obstructive pulmonary disease (COPD). The first measure, the EXAcerbations of Chronic pulmonary disease Tool (EXACT), was developed under the EXACT-PRO Initiative, a multi-year, multi-sponsor project involving experts in pulmonary medicine, instrument development, and drug development regulatory issues, dedicated to the development of a single, standardized instrument for evaluating the effects of treatment on acute exacerbations of COPD. The second measure, the EXACT-Respiratory Symptoms (E-RS) scale, is a derivative instrument comprising a subset of EXACT items to test the effect of treatment on the severity of respiratory symptoms in stable COPD. The EXACT-PRO Initiative was the first PRO instrument development consortia, and the EXACT and E-RS are the first PRO measures to undergo qualification review by the United States Food and Drug Administration (FDA).

Keywords: :

• endpoints
• measurement
• qualification
• regulatory
• drug development
• drug development tools
• treatment efficacy

Background

Understanding disease and effectively testing new treatments depend on the development and standardization of scientific instruments. In clinical research, these instruments are tools of discovery, facilitating the conduct of basic research, the acquisition of new knowledge, and the evolution of novel questions based on new understanding. In clinical trials evaluating the efficacy of new medicines, precise measurement is needed to identify targeted or high-risk patients and/or evaluate treatment outcomes. Toward this end, regulatory agencies in the United States (US) and Europe have published guidelines for clinical trial design (1–3), the use of outcome measures (4, 5), and procedures for qualifying drug development tools (DDTs) for a specific context of use (6, 7). Concurrently, and not unrelated, is the evolution of consortia in which multiple pharmaceutical sponsors share resources to understand the need, identify the measurement challenges, and develop and validate new DDTs. The CBQ-C (8), C-Path PRO and biomarker consortia, and the Innovative Medicines Initiative (IMI) (9–11) have been dedicated to this purpose, with smaller partnerships and consortia underway within and across private companies and academic institutions. The intent is to build the science and facilitate the efficient and effective evaluation of new medical treatments through precise, standardized measurement.

The EXAcerbations of Chronic pulmonary disease Tool (EXACT) –Patient-Reported Outcome (EXACT-PRO) Initiative was the first consortia convened to develop a PRO instrument for use in medical product development trials and is the first such instrument, along with a derivative measure, the EXACT-Respiratory Symptoms scale (E-RS), to undergo qualification review by the US Food and Drug Administration (FDA). These instruments are also under qualification review by the European Medicines Agency (EMA). Although use in drug development trials is one objective of these measures, the intent is to advance the science of chronic obstructive pulmonary disease (COPD) through improved measurement methods. This article describes the process used to develop the EXACT and E-RS and their current status as outcome measures in clinical studies of COPD.

The Exacerbations of Chronic Pulmonary Disease Tool (EXACT)

Measuring exacerbations of COPD

Exacerbations are events characterized by an acute, sustained worsening in the patient's COPD beyond normal day-to-day variability, including an increase in respiratory symptoms such as dyspnea, cough, and sputum production (12). The EXACT is a 14-item daily diary designed to standardize the assessment of the patient's condition in order to capture this dynamic process. Figure 1a shows a schematic representation of the dimensions of exacerbation that can be measured in prevention trials, while Figure 1b depicts exacerbation outcomes in acute treatment trials (13).

Figure 1a. Exacerbation outcomes in prevention trials.

Figure 1b. Exacerbation outcomes in acute treatment trials.

The EXACT provides a direct measure of patient-reported symptoms of exacerbation, complementing and extending information provided by traditional health care resource utilization (HCRU) data by capturing unreported, symptom-defined events, and standardizing the evaluation of symptoms around medically treated events, including magnitude of change around events seen in the emergency room or clinic and before and after hospitalization. Advantages to including a standardized, validated diary-based symptom assessment in exacerbation studies include uniform metrics, reduced recall bias, and the ability to fully characterize exacerbations of COPD, including the estimated 50 to 70% of events that are unreported (14–16).

EXACT development and validation

Development and validation procedures for the EXACT were consistent with guidelines for PRO instrument development proposed by the FDA and EMA (4, 5) and well-known psychometric procedures. The process involved over 500 people, including 493 patients with COPD and 18 professionals from the US and Europe with expertise in pulmonary medicine, clinical research, instrument development, and drug development regulations. A comprehensive review of the published literature was performed as a first step in instrument development to identify and evaluate existing PRO instruments used in clinical trials of exacerbations of COPD. This informed the development of protocols and interview guides used in the qualitative research that formed the foundation of the tool.

To assure and document content validity, 83 men and women with a diagnosis of COPD and history of exacerbations participated in focus groups or interviews to elicit concepts for instrument development (13). An iterative analytical process was used to identify themes and concepts in the qualitative data to inform instrument content and structure. A draft instrument comprising 23 items was subjected to cognitive interviewing methodology in patients with COPD. During 2 advisory panel meetings, experts critiqued the research methods and results and assisted in the design of a prospective study to reduce the number of items and evaluate the performance characteristics of the instrument in patients with acute and stable COPD.

The next phase of instrument development and validation involved a prospective 2-group observational validation study of 410 patients with COPD (17, 18). Acute patients with a clinician-confirmed exacerbation (n = 222) completed the draft item pool and additional items via an electronic diary each evening before bedtime on Days 1–28 of their exacerbation and again on Days 60—67, with clinical assessments performed on Days 1, 10 (±2), 29 (±2), 60 (±7), and 68 (±7). The second group, 188 patients considered clinically stable, completed the diary over seven days.

Item analyses and Rasch item-response theory (IRT) were used to derive the final 14-item EXACT (17). The EXACT Total score is computed by using logit values and a simple look-up table to yield interval-level scores ranging from 0 to 100 where higher scores indicate a more severe condition. The total score is used to determine the patient's stable baseline, onset of an event (threshold of sustained worsening) and recovery (threshold of sustained improvement).

Tests of the reliability, validity, and responsiveness of EXACT scores were performed using an a priori statistical analysis plan (SAP) and follow-up secondary analyses. Scores exhibited high levels of internal consistency and reproducibility. Evidence of construct validity included the ability to discriminate acute and stable groups and sensitivity to change over time in the acute group (18). Results of the validation study were presented and critiqued during a third expert panel meeting; the outcome of additional secondary analyses and proposed interpretation guidelines were discussed during a fourth meeting.

As of April 2013, the EXACT has been translated into 50 languages, with cognitive interviews performed in patients with COPD in the target countries. Results of this work support the conceptual equivalence of the items and use of the instrument in international trials. To facilitate standardized use, an EXACT-PRO User Manual was created, describing the instrument's development, intended use, administration procedures, and scoring, and an eDiary certification program was developed to enhance consistency across eDiary vendors and platforms.

The EXACT-Respiratory Symptoms (E-RS) instrument

Measuring respiratory symptoms of COPD

Upon completion of EXACT development and initial validation, the E-RS was developed to address the need for a standardized instrument to assess respiratory symptom severity in stable COPD. The intent was to reduce respondent burden by using a single diary to capture acute exacerbation events (14-item diary and total score) and respiratory symptom severity (subset of 11 respiratory items contained in the diary). E-RS scores were designed to serve as a primary, secondary, or exploratory efficacy endpoint in clinical trials evaluating interventions to reduce the severity of respiratory symptoms of stable COPD.

E-RS development and validation

The 11 respiratory symptom items comprising the E-RS were selected from the 14-item EXACT diary, with content validity supported through the literature, secondary analyses of qualitative data gathered during the development of the EXACT, and results of an additional prospective qualitative study of patients with stable COPD and no history of exacerbation for at least 12 months (19). The naming convention was adopted to recognize the origin of the items. The E-RS is not intended to be completed in isolation, but rather is self-administered by study participants as part of the 14-item daily diary.

Tests of the reliability, validity, and responsiveness of E-RS scores were performed with validation data from the parent instrument (EXACT) using an E-RS specific a priori SAP and follow-up post hoc analyses (19). Exploratory factor analysis (EFA) of the 11 E-RS respiratory items in patients with stable COPD showed the presence of 3 factors that can serve as symptom subscales: RS-Breathlessness (5 items), RS-Cough and Sputum (3 items), and RS-Chest symptoms (3 items). The RS-Total score, reflecting overall respiratory symptom severity, is computed by taking the sum of items comprising the instrument. Evidence of E-RS score internal consistency, reproducibility, responsiveness, and construct validity were also provided (19).

Translations of the EXACT and the eDiary certification program apply to the E-RS. An E-RS User Manual was developed to facilitate standardized use of this measure, outlining the instrument's development, intended use, administration, and scoring.

Current status of the EXACT and E-RS

Further tests of reliability, validity, and responsiveness

The first validation study provided strong evidence of the reliability, validity, and responsiveness of EXACT and E-RS scores. However, new instruments need to be subjected to multiple tests of reliability and validity in the intended context of use. In the case of the EXACT, prospective data were needed to assess the performance of EXACT scores from stable through acute states and test the sensitivity of the onset and recovery rules. Data were also needed to evaluate E-RS score performance in patients with stable COPD over longer periods of time. For regulatory review purposes, the instruments also needed testing within the context of a randomized, controlled clinical trial in the target population.

To address these needs, the performance properties of EXACT and E-RS scores were tested using data from 3 Phase II, multicenter, randomized, double-blind, placebo-controlled clinical trials provided by 2 pharmaceutical companies. One was a 6-month trial conducted in the US (N = 235) and 2 were 12-week multi-national trials (N = 749; N = 597). Participants had a medical diagnosis of COPD with a forced expiratory volume in 1 second (FEV₁ % predicted) ≤ 80% and at least 1 medically treated exacerbation the prior year. SAPs for the validation analyses were completed a priori with methods and results presented in reports suitable for submission to regulatory agencies as part of the qualification process. Results were also presented at the American Thoracic Society (ATS) annual meeting (20, 21). With the submission package completed and under qualification review by the FDA and EMA, manuscripts describing the methods and results are in development.

Results of these studies support the validity and reliability of the instruments in the intended context of use. To serve as an outcome measure in clinical trials, an instrument must also show sensitivity to treatment effects. In a non-pharmaceutical setting, Halpin and colleagues’ (22) 4-month randomized trial of the effect of health risk winter alert calls on exacerbation rate found that patients receiving calls had fewer symptom-defined (EXACT) events (34% vs. 53%) and that these events were shorter (8.2 ± 2.0 vs. 10.1 ± 1.9 days) and less severe (area under the curve (AUC) 65 ± 21 vs. 115 ± 22) than events in patients receiving no calls. Although not statistically significant due to sample size limitations, the large effect sizes were consistent with the EXACT's sensitivity to treatment effects, with results providing insight into the effect of weather and early intervention on exacerbations of COPD with implications for further research.

In the drug development setting, significant treatment effects were found in the ATTAIN study, a 6-month international Phase III randomized, controlled clinical trial testing the efficacy of aclidinium for the maintenance treatment of COPD (N = 828) (23, 24). This study showed a significant difference in exacerbation rate between each active treatment group and placebo for both medically-treated events (200 ug: 29% (rate ratio 0.72, 95% CI (0.52, 0.99), p < 0.05); 400 ug: 33% (rate ratio 0.67, 95% CI (0.48, 0.94), p < 0.05)) and symptom-defined (EXACT) events (200 ug: 28% (rate ratio 0.72, 95% CI (0.55, 0.94), p < 0.05); 400 ug: 29% (rate ratio 0.71, 95% CI (0.54, 0.93), p < 0.05)) (23, 24). Significant differences in respiratory symptom improvement were also observed (RS-Total score, p < 0.001, both doses), with subscale analyses showing effects for RS-Breathlessness, (p < 0.001, both doses), RS-Cough and Sputum (400 ug: p < 0.001; 200 ug: p < 0.05), and RS-Chest Symptoms (400 ug: p < 0.001; 200 ug: p < 0.01) (23).

Challenges and Opportunities

Standardizing the symptomatic assessment of exacerbations will facilitate research into the nature of these events, including an exploration of exacerbation prodrome, acuity, recovery, persistent worsening of COPD post exacerbation, and relapse. Additional areas of research include patient self-care and treatment-seeking behaviors, physician decision making related to diagnosis and treatment, and the effect of early detection and intervention. The similarities and differences between COPD exacerbations identified by the “traditional” HCRU definition and symptom-defined events identified by the EXACT are also of interest. The EXACT detects more events than HCRU counts, while some HCRU events do not reach the symptom threshold set by the EXACT. Whether these types of events are biologically similar, have the same impact on disease and quality of life, or have independent or cumulative effects on morbidity and mortality are yet to be determined.

Although the EXACT was designed for use in clinical research, the widespread use of mobile phones and applications may make it suitable for clinical practice as well. Recent successes in research settings, using the EXACT to identify threshold-based changes in patient symptoms and trigger communication with a clinical site for further assessment and possible intervention (25), suggest it may be possible to use the measure similarly in clinical practice. The feasibility of this use and the performance of the instrument in varied practice settings and COPD populations are unknown at this time.

The E-RS provides a tool for quantifying respiratory symptoms in stable COPD to study day-to-day severity and variability of the cardinal symptoms of this disease: breathlessness, cough and sputum, and chest symptoms. Standardizing symptom assessment across studies will facilitate meta-analyses and the accumulation of knowledge related to symptom severity and relief, which are so important to patients with this disease.

Conclusions

The EXACT and its derivative instrument, the E-RS, were designed to advance our understanding of COPD and its treatment by standardizing symptomatic assessments in clinical research and medical product development trials. The instrument development procedures included extensive qualitative work and the involvement of clinical and measurement experts to assure content validity, with validation evidence from a dedicated prospective validation study and 3 controlled trials involving a total of nearly 2000 patients from the target population. Evidence of sensitivity to treatment effects has been shown in 2 clinical studies to date, including a Phase III pharmaceutical trial. Reports detailing the development and validation of the EXACT and E-RS for use in medical product development trials of COPD are currently under qualification review by the FDA and EMA. In the meantime, the instruments are being used in multiple studies, with the hope that results will provide further insight into exacerbations and respiratory symptoms of COPD and the efficacy of interventions to reduce these debilitating aspects of the disease and improve patient quality of life.

Acknowledgments

The authors thank Kathryn Miller for her editorial comments and formatting this paper for publication, Dr. Stephen Rennard for his review and comments on the paper, and the following individuals and companies for their contributions to EXACT and E-RS development:

EXACT-PRO Expert Panelists

Table

Senior Clinical Research & Consultants Co-Investigators:	Affiliation:
Paul W Jones, PhD, FRCP	St. George's, University of London
Sanjay Sethi, MD	University at Buffalo
Expert Panelists:	Affiliation:
James Donohue, MD	University of North Carolina, Chapel Hill
Sonya Eremenco, MA	CORE Translation Services
Pennifer Erickson, PhD	OLGA & Pennsylvania State University
Fernando Martinez, MD	University of Michigan
Donald Patrick, PhD	University of Washington
John Powers, MD	George Washington University
Stephen Rennard, MD	University of Nebraska
Roberto Rodriguez-Roisin, MD, PhD	University of Barcelona
Holger Schünemann, MD, PhD	University at Buffalo
Jadwiga Wedzicha, MD, FRCP	Royal Free & U College Medical School

and staff from the United States Food and Drug Administration (FDA) participating in the expert panel meetings and qualification review process.

Declaration of Interest Statement

Sponsors

The following companies provided unrestricted funds for one or more phases of the EXACT-PRO Initiative: Adams Respiratory; Almirall; Altana (Nycomed); AstraZeneca*; Bayer; Boehringer Ingelheim*; DEY; Forest Laboratories; GlaxoSmithKline; Johnson & Johnson; Merck*; Mpex (now Aptalis)*; Novartis; Ortho McNeil; Pfizer; Sepracor (now Sunovian); and Schering-Plough (now Merck). Clinical trial data for validation analyses were provided by Mpex (6-month trial) and AstraZeneca (2 3-month trials). *Also sponsors of E-RS development and validation work.

Scientific Staff

Wen-Hung Chen, Asha Hareendran, Christy Houle, Kelly Howard, Ray Hsieh, Sherilyn Notte, Jennifer Petrillo, Laurie Roberts, Chris Sexton, Laurie Smith, Marilyn Stolar, Chris Thompson, Ingela Wiklund, Teresa Wilcox, Randall Winnette, Ren Yu

The authors also gratefully acknowledge the participation of patients, sites and site coordinators in the qualitative and quantitative development work.

The authors alone are responsible for the content and writing of the paper.