Disregarding clinical trial-based patient-reported outcomes is unwarranted: Five advances to substantiate the scientific stringency of quality-of-life measurement

Abstract

Background. The clinical impact of trial-based quality of life (QL) outcomes is frequently underestimated due, in part, to prejudice and lack of knowledge by the medical community. The objectives of this paper are to show that QL assessments build upon an empirically based and stringent approach to measurement and QL outcomes should not be viewed nor handled differently than any other parameter in medical research. Material and methods. Literature overview. Results. The objectives are substantiated with empirical evidence showing that: (1) existing QL measures are as reliable as most other clinical outcomes; (2) available guidelines improve the quality of trial-based QL data; (3) QL data have strong prognostic value for survival; (4) clinical significance of QL data can be established; and (5) accounting for response-shift effects in QL data over time is feasible. Finally, the investigation of the genetic disposition of QL is described as an emerging area of research. Discussion. It is a waste of effort and money and also unethical when collected trial-based QL data are not used to their full power. QL and other patient-reported outcomes deserve to be included in more trials, with full disclosure of all results, and standardized interpretation. Only the combined use of patient-reported and clinical outcomes will enable the examination of the extent to which cancer patients live a qualitatively good life as long as possible.

Since the nineties of the past century, there is a general recognition of the need to expand the traditional end points in comparative cancer clinical trials, such as survival and/or time to disease progression, to include the impact of cancer and its treatment on patients’ health-related quality-of-life (QL) [1–4]. Following the World Health Organization's triad of physical, mental, and social health, it is generally accepted that, health-related QL also entails at least three domains: physical, psychological, and social functioning. Additionally, there is consensus that it encompasses an overall judgment of health and/or QL and that these overall as well as more specific judgments are affected by one's disease and/or treatment [5,6]. Recently, the Food and Drug Administration (FDA) introduced the umbrella term patient-reported outcomes (PRO). A PRO is any report coming directly from patients, without interpretation by physicians or others about how they function or feel in relation to their health condition and its therapy [7]. Thus, it is a broad term not only encompassing health-related QL, but also, for example, perceptions about treatment, satisfaction with care received, and satisfaction with professional communication [8]. Since the focus of this paper is on QL, we will refer to patient-reported (instead of health-related) QL in the remainder of the paper.

The past two decades have witnessed an increasing number of cancer clinical trials that included QL as secondary and sometimes as primary outcome. This increased use led the Journal of Clinical Oncology to devote its entire November 2007 issue to the incorporation of PROs in cancer clinical trials. It is generally agreed that QL may assume primary importance in phase III trials when no or limited effects on survival and tumor response are expected; when survival is gained, however, at the expense of major toxicity; and when survival is gained, however in both treatment arms equally. In all three instances, QL may be a decisive outcome [9]. The key question is whether QL measurements in cancer trials to date have affected clinical decision-making. The answer is disappointing.

A systematic review of randomized clinical trials in breast cancer indicated that in only five of the 46 randomized trials of biomedical interventions QL outcomes influenced clinical decisions. These five studies examined the primary management of breast cancer and compared medical treatments whose medical outcomes were found to be equivalent [10]. A review of 33 surgical oncology randomized trials indicated that in one-third (11) of the trials QL outcomes did not influence treatment decisions or did not provide comprehensive data about the impact of treatment on QL. Interestingly five of the 12 trials with methodologically robust QL data, did not base their treatment recommendations on these outcomes because the authors did not consider QL outcomes to be sufficiently important [11]. This underuse of collected QL data is not restricted to cancer but is characteristic of trials in general. A systematic review was conducted on randomized trials in different disease sites that included outcomes based on the most widely used generic QL questionnaire, the Medical Outcome Study Short Form Health Survey, SF-36 in short. This review aimed to determine how often QL evaluations reached different conclusions from those of primary efficacy outcomes [12]. Of the 66 comparisons, 21 discordant results were found between SF-36 results and primary efficacy outcomes. However, the interpretation of the study findings was modified in only two of these discordant cases. There was also a general tendency to belittle rather than to pronounce discordant results, not to discuss the SF-36 findings at all, and not to report all tested SF-36 outcomes. These three reviews indicate that the clinical impact of trial-based QL outcomes is frequently underestimated or even disregarded.

Apparently, QL outcomes still elicit a sceptical view from the medical community, which is, in part, based on prejudice (e.g., “You cannot trust the patients to provide reliable data”) and lack of knowledge (e.g., “There is no science behind it”). This situation is unfortunate and unwarranted. QL results deserve a more systematic use to enhance their ability to improve clinical decision-making. The objectives of this paper are to show that QL assessments build upon a long, empirically-based, and stringent approach to measurement and that patient reported QL outcomes should not be viewed nor handled differently than any other parameter in medical research. These claims will be supported by evidence showing that: (1) Existing QL measures are as reliable as most other clinical outcomes and QL assessment is further advanced by the use of sophisticated statistical techniques, such as Item Response Theory; (2) Available guidelines for stringent and standardized patient-reported research improve the quality of trial-based QL data; (3) QL data have strong prognostic value for survival; (4) Clinical significance of patient-reported QL data can be established; and (5) Accounting for response-shift effects in QL data over time is feasible. Whereas the advances in these areas will be highlighted, the challenges and need for further research will be described as well. Finally, the investigation of the genetic disposition of QL will be described as an intriguing emerging area of research, which may have implications for cancer treatment and the design of cancer clinical trials.

Existing QL measures are as reliable as most other clinical outcomes

A wealth of standardized QL questionnaires are currently available, stored in electronically accessible databases, that provide adequate coverage of the basic QL domains and that yield adequate levels of reliability and validity. For example, the PROQOLID database maintained by Mapi Institute (http://www. goqlid.org/) includes over 650 of such questionnaires. The most common instruments in QL research are (a) generic and are intended for use across a wide range of chronic disease populations (e.g., the previously mentioned SF-36 [13], the EQ-5D [14]), (b) cancer-specific (e.g., the Functional Assessment of Cancer Therapy-General (FACT-G) [15], the EORTC Core Quality of Life Questionnaire (EORTC QLQ-C30) [16], and (c) domain-specific addressing one specific aspect of QL in greater detail (e.g., the McGill Pain Questionnaire [17], the Hospital Anxiety and Depression Scale (HADS) [18]).

In a recent study, Hahn and colleagues [19] conducted a literature review to compare the reliability or measurement precision of QL questionnaires with that of common medical measurements in a wide range of disease areas, including oncology. They pre-specified cut-off points for different reliability measures (e.g., weighted kappa, Intra-Class Correlation Coefficients, Cronbach's alpha) to indicate high, moderate, and low levels of reliability and applied these equally to QL and medical measures wherever appropriate. The findings were similar across the disease areas: all measurements are associated with error, whether from examination, laboratory findings, or self-report. To elucidate for oncology, the medical measures ranged from highly reliable (e.g., survival), moderately reliable (e.g., tumor response classification using chest radiograph) to poorly reliable (e.g., tumor mass diameter, tumor size measurements over time, and time to tumor progression). Whereas tumor measurements were long held to be objective, it is clear that the degree of misclassification places them in the low reliability ranges. Similarly, the reliability of QL measures also varied across questionnaires and across subscales within questionnaires. For example, the subscales of the SF-36 were found to be highly reliable (e.g., physical functioning), moderately reliable (e.g., pain, vitality), and poorly reliable (e.g., role-physical, role-emotional). The authors concluded that if one were to relax the requirements of patient-reported QL measures for use in clinical practice to a level comparable to that of medical measures, the trustworthiness of QL assessments would compare favorably. It should be noted that QL questionnaires need to adhere to higher reliability requirements if they are to be applied to individual patients in clinical practice (i.e., Cronbach's alpha> 0.90). If used in the context of medical research, a moderate level of reliability suffices (i.e., Cronbach's alpha > 0.70).

The field of QL assessment has started to embrace advanced statistical techniques that were developed in educational psychology, such as Item Response Theory. This technique entails a set of statistical models used to analyze multiple categorical variables that measure the same concept. Hence, its relevance for analyzing the reliability of questionnaire items in a scale. Item Response Theory has a number of advantages over classical test theory or clinimetry, such as the ability to more realistically assess measurement precision, allowing linking and calibrating scales from different questionnaires, assessing translation equivalence across different language versions, and enabling computerized adaptive testing. This latter technique makes use of the computer to select the most appropriate items and the optimal test length to match each respondent's level, while scoring the responses on a scale that allows comparison with people answering other items [20]. Computer adaptive testing requires large and well-studied pools, or “banks” of items for specific QL domains, such as fatigue, pain, depression, mobility, and social function. The support of the National Institutes of Health (NIH) through the NIH Roadmap Initiative known as PROMIS (Patient- Reported Outcomes Measurement Information System; http://www.nihpromis.org), allowed the accelerated construction of those item banks. With time, we will not only be able to use fewer items, but the scores that emerge from these items will also have a common metric and range, a shared meaning and understanding across users [21].

Available guidelines improve the quality of trial-based QL data

Efforts have been mounted to improve the quality of clinical trial-based QL studies by providing guidelines [22–24] and checklists [25]. The guidelines that have been forwarded during the past two decades have paid off: the quality of patient-based QL reporting in cancer clinical trials has improved over time. For example, between 1990 and 2004 159 randomized clinical trials were reported in the top four major tumor sites. Of the studies that were published prior to 2000, 39% was evaluated as methodologically robust whereas this percentage increased to 64% for the studies published later. This indicates a remarkable progress in a relatively short period of time [26]. However, the latter percentage also shows that there is still room for improvement.

Sometimes, clinical-trial based PROs are used for labeling and promotional claims. Such claims need to be approved by the FDA in the United States and the European Medicines Agency (EMEA) in Europe. These agencies have therefore also released institutional guidelines to standardize the stringent incorporation of PRO measures, which in turn will facilitate a systematic review process. The FDA draft guidelines that were released in February 2006, invoked a debate with the PRO community, which was documented in a series of articles published in Supplement 2, 2007 of Value in Health. Despite some concerns, primarily directed at their prescriptive nature, these guidelines have been found valuable and have further stimulated the debate and the acceptance of PRO measures in oncology [27].

QL data have strong prognostic value for survival

Perhaps the most provocative finding in QL research is that QL is often superior to clinical and biomedical assessments for predicting survival duration in general populations [28] as well as in a range of chronically ill patients [29–31], including those with cancer [32]. A simple question as “How, in general, would you rate your health: excellent, very good, good, fair, or poor?”, is independently predictive of longevity. A critical review of 39 randomized clinical cancer trials revealed that global QL and physical functioning were particularly predictive of survival, as were the symptoms appetite loss, fatigue and pain [32]. Notably, these PROs retained their predictive power after accounting for well-established and powerful medical predictors, such as disease stage, tumour size, and physician-rated performance status.

A range of possible explanations for this ubiquitous finding has been suggested. First, QL may be a more inclusive and accurate measure of health status and/or disease experience than any other measure. QL may thus highlight an early perception by the patient of subtle signs of disease progression not recorded by the conventional prognostic factors. Second, according to “the trajectory hypothesis” [33] QL can be seen as a dynamic evaluation reflecting changes in health, lifestyle and life circumstances and not only current level of health. Third, QL is indicative of patients’ adherence to treatment and follow-up schedules or other health practices, and is thus indirectly related to the disease process. Fourth, QL might influence how people perceive their bodies. Finally, QL may induce changes in physiological processes. For example, positive feelings of well-being may influence survival by enhancing the immune system response via endogeneous opioids, autonomic nervous and hypothalamic-pituitary-adrenal activation, and buffering the system from the negative effects of stress [34].

Whatever the cause of the relationship between QL and survival, this finding has practical importance. For example, QL can help signal those patients who are in need of medical attention, and can be an early warning useful for clinical decision-making. In clinical trial-based research, QL can be used as a stratification variable. Finally, suggestions have been made to conduct interventions to improve QL to potentially increase survival. This implication is intriguing and widely discussed but rarely tested. It would require insight into the relationship between QL and survival. That relationship might be more complex than originally thought and may be different for early and advanced stages, and extreme versus moderate levels of QL [34]. For example, improving patients’ QL might be detrimental if it obscures early warning signs. Clearly, more research is needed to understand the role of patient-reported QL in predicting survival.

Clinical significance of QL data can be established

Oncologists are familiar with pathology reports, laboratory chemistry values and even the qualitative aspects of a medical history. However, they are not routinely taught how to use patient-reported QL outcomes. For example, when the mean score of a QL questionnaire increases 7 points on a 0-100 scale in the experimental group with no change in the control group, it is unclear whether this is a large or a small effect, or somewhere in between.

There are basically two families of approaches to establish clinical meaningfulness in QL [35]. First, distribution-based methods relate the results to some measure of variability, such as effect size. For example, Cohen [36] defined effect sizes as the mean change score divided by the standard deviation of stable subjects, e.g., as obtained at baseline. His guidelines to interpret effect sizes in the range of 0.2 standard deviation units as small, 0.5 as moderate, and 0.8 as large, have been widely adopted. Second, anchor-based approaches incorporate a meaningful, external measure that is more clearly understood than QL scores themselves. For example, a frequently used approach is the minimal clinically important change [37, 38] that correlates change over time with patients’ overall evaluations regarding the extent to which they improved, remained stable or deteriorated.

Whereas these approaches are helpful, they do not answer the pressing question what guidelines to adopt to interpret QL data and what benchmark to use for determining clinical significance in the absence of relevant information. This and other questions inspired the establishment of the Clinical Significance Consensus Meeting Group to provide a series of state-of-the art articles [39]. Enthused by this first effort, group members continued investigating empirically the relationship between the different approaches. They showed convergence in two instances. First, distribution- and anchor-based approaches yielded similar results. Norman and colleagues [40] conducted a systematic review of studies that computed minimally important change and that contained sufficient information to compute an effect size. Results indicated that in most circumstances, the threshold of discrimination for clinically meaningful changes in QL appears to be approximately half a standard deviation. This equals a change or difference of approximately 8–10 scores on a 0–100 scale. Whereas this relatively straightforward and simple conclusion elicited criticism [41–43], the guideline of 0.5 effect size has been found to be robust and could in some circumstances even be relaxed to 0.25 to 0.35 effect size [44]. Second, convergence was found among the distribution-based responsiveness measures. Based on a mathematical analysis of the variance components underlying these different change scores, Norman and colleagues [45] showed that the measures were mathematically equivalent. They recommended that future analysis of responsiveness be restricted to Cohen's effect size because this measure is less vulnerable to extreme values, is more readily interpretable, and has already accepted standards of magnitude.

The resulting and currently generally adopted guidelines for assessing clinical significance include a number of steps: (1) define clinical significance a priori and incorporate it into the methodology of the study at the design phase; (2) use a combination of anchor-and distribution-based approaches (e.g., minimal important difference and effect size); (3) use results of previous studies wherever possible; (4) in the absence of data use a 0.5 effect size or 10 points on a 0-100 scale as benchmark; and (5) take relevant issues into account (e.g., psychometric properties of the instrument, patients’ initial score, and direction of change [44, 46]). Given the increasing familiarity with QL measures, clinicians are currently developing evidence-based estimates of QL, just as they already have for clinical measures.

Accounting for response-shift effects in QL data over time is feasible

When cancer patients have the time to adapt to their disease, they can experience a remarkably good QL, despite lasting physical limitations. Such findings might be interpreted as resulting from changes in patients’ internal standards, values and/or the conceptualisation of QL over the course of the disease trajectory. These changes, that are inherent to the process of accommodating to the illness, are referred to as ‘response shift’ [47, 48]. Since patients may view their QL differently over time, QL is also referred to as a “moving target”.

Response shift poses methodological challenges as it may not only render assessments completed over time incomparable but may also affect cross-group comparisons. For example, in the context of clinical trials differences in QL across treatment arms may be jeopardised when response shift affects the treatment groups differentially [49, 50]. Random allocation to treatment arms does not solve this problem since treatments may still result in different levels and types of side effects and symptoms. Assessing response shift may therefore be needed to obtain a valid and sensitive assessment of change over time.

Since the late nineties of the last century when response shift was introduced to QL research [47, 48] increasing efforts have been mounted to develop and test new and existing measures that can be integrated into research designs, so that response-shift effects can be explicitly measured and taken into account. Examples include the so-called ‘then-test’ approach, individualized methods, direct questioning, qualitative questioning, vignettes ratings, questionnaires on response shift, and statistical methods, such as factor analysis, growth curve analyses, residual analyses, and Structural Equation Modeling [51]. To date, most response shift research has employed the then-test approach, which proved to provide valuable information in a number of clinical studies in oncology [49, 52–58].

The question arises how meaningful such response-shift effects are. The magnitude and clinical significance of response-shift effects were examined in a meta-analysis using 19 published QL studies [59]. The effect sizes of response shift were found to be relatively small according to Cohen's criteria. The largest effect size was detected for fatigue (0.32), followed by global QL (0.30), physical limitations (0.24), psychological well-being (0.12), and pain (0.08). It should be noted that even a small response shift may result in an under- or overestimation of the true QL change, dependent on its direction. One may thus erroneously conclude that the effect is small when it is in fact moderate, or that it is moderate when it is large.

Whereas the stringency of response-shift research needs to be improved, the empirical evidence of response shift is sufficiently compelling: response shift cannot be ignored. A note of caution is in order.

Sceptics of PROs might view response shift as a fatal complication. However, if clinical measures, such as tumour response, are found to be unresponsive, no one would discard them but rather try to improve them. Similarly, QL is highly relevant, but its measurement requires further development to integrate response-shift effects. A growing number of researchers further this blooming field as is illustrated by a special section on response shift in the November issue 2009 of the Journal of Clinical Epidemiology. The key point is that response shift is vital to better capture changes in QL over time and that assessing response shift is feasible.

Advances in QL research: Searching for the genetic disposition of patient-reported QL

PROs are not only affected by disease and treatment. Recent data provided preliminary evidence that the genetic disposition of patients may impact their QL. For example, research on twins has provided ample empirical evidence that emotional states are to a substantial degree heritable. Positive states, such as subjective well-being and life satisfaction, are slightly more heritable (40 to 50%) [60–62] than negative states, such as depression and anxiety (30 to 40%) [63, 64]. The remaining variances can be attributed to environmental influences unique to the individual, in which the intrauterine period may play an important role. Genetic influences have also been reported for self-rated health [65–68], indicating somewhat lower heritability estimates that range between 20 and 40%.

An abundance of studies have focused on the delineation of the biological pathways of negative emotional states, such that one can speak of the hypothalamo-pituitary-adrenal (HPA) axis [69] as the “final common pathway” of depressive symptoms. By contrast, biological and genetic research into positive emotional states is scarce. Multiple genes are involved in emotional states, each exerting a small effect. While the rate of progress is dazzling, particularly for negative affect, the biological complexities do not allow definitive answers yet. Additionally, biological and genetic research into symptom experience, e.g., pain and fatigue, is also expanding rapidly [70].

Sloan & Zhao [71] were the first to examine the direct link between polymorphisms and cancer patients’ QL, using a large randomized North Central Cancer Treatment Group clinical trial. More than triple the number of relationships between genetic variables and PROs were observed than would be expected by chance alone. They found evidence for relationships between overall QL, symptom distress, and fatigue with variant genotypes of three enzymes involved in folate metabolisms. Recently, Yang et al. [72] evaluated the role of glutathione-related genotypes on QL in advanced non-small cell lung cancer patients who participated in a clinical trial. Patients carrying the glutathione peroxidase 1 (GPX1-CC) genotype had a clinically significant decline in overall QL, physical, functional, and emotional well-being.

The findings from the few studies performed so far are sufficiently compelling to justify further exploration of the relationships between genetic variants and patient-reported endpoints. An international and interdisciplinary consortium, the GENEQOL Consortium, has been established in 2009 to translate and plan clinically relevant research to identify and investigate potential genes and genetic variants involved in QL [70]. Genetically informative studies are needed as they may provide insights into a wide variety of complex questions that traditional QL studies cannot deliver. For example, how can we predict which patients will suffer from mood disturbances or fatigue when taking a specific chemotherapeutic regimen? Why do some pharmacotherapeutic treatments not work in all patients with the ‘same’ level of distress or fatigue? Insight into the genetic versus environmental components of PROs will ultimately allow us to explore new pathways for improving patient care. If we can identify patients who are susceptible to poor QL, we will be able to better target specific support, such as psychological and/or pharmacological treatment, and clinical management interventions. The genetic information can be used to tailor individualized treatments for QL in the same manner as individualized treatments for the underlying disease itself [71]. To cite Sloan and Zhao [71]: “Doctors will eventually use genetic patterns for several tasks: to tell whether a cancer will spread, to predict how various therapies such as specific drugs or radiation will work, and perhaps even to see how someone's QL will be affected.” Increased understanding of the genetic underpinning of QL will also have clear effects on the design of randomized controlled trials. In the future, genetic information may, for example, be included as an eligibility criterion or as a stratification variable prior to randomization.

Epilogue

The reviewed research testifies that QL assessments build upon a long, empirically based, and stringent approach to measurement. QL data can be handled and viewed like any other clinical outcome in clinical research and daily clinical practice. Since QL assessments provide a different view on patient outcomes, they may impact treatment recommendations and thus enhance our ability to improve clinical decision-making and cancer care. It is a waste of effort and money and also unethical when collected trial-based QL data are not used to their full power. QL and other PROs deserve to be included in more trials, with full disclosure of all results, and standardized interpretation.

The role of QL and other PROs in clinical research as well as in clinical practice is expected to increase. First, we will become older, but unfortunately not healthier. The American Medical Association estimates that half of the population will have a chronic illness in 2020 [73]. Second, patients’ political awareness and empowerment will increase, both at the individual and at the societal level. Patients are increasingly organized and take part in advisory and administrative committees at all levels and in all areas of the health-care system. The patient perspective will therefore be increasingly included in the design of care. Third, the limits of medical care have been reached in a number of areas. Treatments are increasingly equivalent regarding survival gain, e.g., surgical treatments in oncology [11], but may differ in side effects, functional limitations, and long-term QL. Fourth, the objectives of medical treatment are therefore expanding. For example, the aim is no longer whether a surgical procedure is successful, but whether the patient can function well and can participate in society with a good QL, particularly in the long run. Finally, the need to reduce health care costs will only become more pressing. This need will require evidence-based healthcare based, in part, on patient-reported data, at the level of individual patient care as well as at the level of policy and governmental decisions.

Additionally, the conceptualization of health and QL of both individuals and populations are increasingly broadening to encompass not only symptoms and limitations, but also positive aspects of human existence, such as happiness. For example, the World Health Organization has started to emphasize happiness as a component of health [74]. Whereas the inclusion of happiness as one of cancer treatment goals may currently seem outlandish, the close relationship between health and happiness warrants attention. There is a growing empirical body of research showing that happiness has a salutary impact on health, particularly regarding the immune system response and pain tolerance [34, 75]. In other words, happy people are in general healthier. Interestingly, a recent article in the British Medical Journal was focused on the dynamic spread of happiness in a large social network [76]. This study showed that happiness is a collective phenomenon where an individual's happiness depends on the happiness of others with whom he/she is associated. This finding has implications for public health. As authors indicated: “To the extent that clinical or policy manoeuvres increase the happiness of one person, they might have cascade effects on others, thereby enhancing the efficacy and cost-effectiveness of the intervention.’ [76]. Hence, providing better care for cancer patients, may not only improve their happiness but also the happiness of those who care for them and numerous others. Perhaps in the future positive aspects of patients’ QL, such as happiness, will be systematically included in cancer clinical trials. The use of such all encompassing PROs in combination with clinical parameters will enable the examination of the extent to which cancer patients live a qualitatively good life as long as possible.

Acknowledgements

I am indebted to Pythia Nieuwkerk, Elsbeth Bloem, and Marion de Boer for astute and useful comments to earlier drafts of this manuscript. This paper is based on an invited oral presentation held at the conference “State of Science in Cancer Care” on August 19, 2009 in Stockholm, Sweden.

Declaration of interest: The author report no conflicts of interest. The author alone are responsible for the content and writing of the paper.