Is it time to move beyond the QALY in vision research?

Economic evaluation of health programs is the science of weighing the benefit of a health intervention against its cost. The metric used to evaluate this relationship is the incremental cost-effectiveness ratio (ICER), which represents the cost of each unit of effectiveness “purchased” through the health program.1 For instance, if effectiveness of a medical treatment for diabetic retinopathy was measured in letters correct on an Early Treatment Diabetic Retinopathy Study (ETDRS) chart, an economic evaluation would compare the new intervention to the existing standard practice, estimating the difference in letters correct and the difference in cost. The cost-effectiveness of the intervention would be measured by dividing the difference in cost by the difference in letters correct, resulting in the ICER. The decision as to whether the intervention is “cost-effective” is based on comparing the ICER to the “willingness to pay” for that unit of effectiveness. If the ICER is less than the willingness to pay, the program would be adopted.2

While this process is straightforward in theory, the problem in practice is that there are few payors who have established the willingness to pay for any unit of effectiveness – be it letters correct, lives saved, or cases screened. The notable exceptions to this are payors in Canada, the United Kingdom, and the European Union who established a standard willingness to pay for a quality-adjusted life year (QALY) gained through a health program.3–5 The QALY is a composite measure in which a year lived is weighed by a value representing the quality of life enjoyed during that year. The metric used to estimate the quality of life associated with that year is a preference-based measure referred to as a utility. It is measured on a scale ranging from perfect health (1.0) to death (zero)6 which, in theory, encompasses all possible health states. Therefore, a health-related utility elicited in this manner provides a common measure of effectiveness by which all health-related interventions might be judged and a standard willingness to pay for a QALY might be established.

Estimating utilities in this fashion is a time consuming and cognitively difficult task for most study participants requiring that the respondent engage in a series of trade-offs in which they balance their desire for perfect health against a risk of death or reduced life span.7 In order to reduce this cognitive burden, improve the stability of the utility metric and speed the elicitation process in clinical trials, investigators have developed multi-attribute instruments that elicit health-related utilities using questions concerning daily function. The most popular of these are the EQ-5D (also referred to as the Euro-QoL)8 and the Health Utility Index (HUI).9 The EQ-5D is the preferred source for utilities in the UK and Europe, and the HUI is the preferred instrument in Canada.

The National Eye Institute Visual Functioning Questionnaire (NEI VFQ) has been the standard for quality of life assessment in people with eye disease for over a decade.10 Among practitioners of economic evaluation of treatment of eye disease, the creation of a multi-attribute instrument that permits estimation of vision-related utilities using the NEI VFQ has long been considered the field’s “holy grail.” In this issue of Ophthalmic Epidemiology, Kay and Ferreira present an excellent example towards that goal.11 The authors have relied on the results of RESTORE, a diabetic retinopathy treatment trial in which participants completed both the NEI VFQ and the EQ-5D. They employed very sophisticated and rigorous methods to map NEI VFQ responses to EQ-5D results, and used appropriate methods to conduct a rigorous cross-validation. In the end however, the authors found that even their best models failed to predict the respondents actual EQ-5D scores, leading them to note in their discussion that, “…none of the NEI VFQ-25 scales strongly influence the EQ-5D utility values.”11

As Kay and Ferreira themselves point out in their excellent discussion, their difficulties in mapping the NEI VFQ to the EQ-5D had far less to do with the method they employed than the two instruments that they were seeking to correlate. The EQ-5D has been shown in a number of studies to be unresponsive to changes in vision status, and Kay and Ferreira cite several examples. It is not surprising that the EQ-5D would be unresponsive to vision changes in vision status as it measures quality of life across five domains: mobility, self-care, usual activities, self-care and anxiety/depression – each using 3 or 5 levels of measurement. The impact of changes in vision status would only be measured to the extent of its interaction with the domain was measured; and even then, given the limited resolution of measurement, the impact of modest changes in vision status are not likely to be identified. Kay and Ferreira in evaluating the results of the RESTORE trial are only the most recent investigators to demonstrate that the EQ-5D is problematic for estimation of effectiveness in economic studies.

Kay and Ferreira point out in their paper that they were attempting to map the NEI VFQ to the EQ-5D because the EQ-5D is the preferred utility measure for payor authorities in the UK and Europe. However, the EQ-5D itself is an indirect measure of utility, as is the HUI. The developers of both of these instruments performed direct elicitation of utility from community members using time tradeoff before they mapped utility scores to the attributes used in their instruments to measure quality of life.12,13 A team of health state valuation researchers recently took a similar approach in mapping items from the NEI-VFQ to utilities elicited in a time tradeoff study from a community-based sample.14 While evaluation of the strength of this instrument is beyond the scope of this editorial, we note that the investigators did not have the same difficulties with face validity and model fit that Kay and Ferreira found in mapping the NEI VFQ to the EQ-5D.

Kay and Ferreira are not the first investigators to find that there are problems with utility elicitation for vision-related problems. Other investigators have found that the results of utility elicitation exercises in vision are highly dependent on the method or instrument used.15–17 Notably, Espallargues and colleagues had the most illustrative finding as they tested the EQ-5D, HUI, SF-6D and time tradeoff in a sample of 209 patients with age-related macular degeneration with visual acuity ranging from a logarithm of the minimum angle of resolution (logMAR) of 0.3 to worse than 2.0.15 The range that they found across this spectrum of vision loss varied from 7 points for the SF-6D to 40 points for the HUI (the directly elicited time tradeoff had a range of 26 points).15 Such poor consistency in instruments should make all policy makers (as well as thoughtful practitioners of economic evaluation) cautious in accepting the results of any cost-utility study conducted in vision research.

Vision research investigators are not alone in their disillusionment with health-related utilities as a method of evaluation for health programs. A number of economists and experts in quality-of-life assessment have raised issue with the theoretical underpinnings of the QALY and the practical aspects of utility elicitation.18 Vision researchers do however face the additional challenge that the process of utility elicitation is based on a construct of “perfect health” that some have suggested excludes the concept of perfect vision.6 Whether this is the case or not, there are enough problematic aspects of utility measurement in vision that it is clear we would benefit from thoughtful consideration of alternative methods of health state valuation. In response to their frustration with the QALY, investigators considering non-vision diseases have advocated for direct estimation of the patients willingness to pay for improvement of health states using discrete choice experiments.19 Ironically, given the prevalence of out-of-pocket payment for LASIK surgery in the US and elsewhere, visual improvement is one area where direct measurement of the willingness to pay for change in vision status is highly practical.

Kay and Ferreira are to be congratulated for demonstrating a thoughtful approach to utility mapping, but the reader is cautioned from taking the results of this study and employing them in economic evaluation. The result of their study is but one more that has demonstrated that the EQ-5D is a poor tool for evaluating the result of treatment of vision problems. It might also be one more example that leads us to ask if it is time to move beyond the QALY as a measure of effectiveness in economic evaluation.

Declaration of interest

The author reports no conflicts of interest. The author alone is responsible for the content and writing of the paper.