What outcomes should US policy makers compare in comparative effectiveness research?

The last year has been a watershed in the advancement of comparative effectiveness research (CER) in the USA. Not only did the American Recovery and Reinvestment Act appropriate US$1.1 billion for CER, but also, a national institute for such research was created as part of healthcare reform. Despite the hype, little is known about how policy makers will use CER in funding and reimbursement decisions. Already in the USA, history shows a plethora of ways that agencies and payers have attempted to include evidence-based medicine in their coverage decisions [1]. However, because of divisions over whose perspective should take precedence – patients, clinicians, payers, manufacturers and so on – the USA will likely follow a path pursuing evidence-based comparisons of treatments without using cost. Hence, it is likely that the USA will join a growing a list of countries that envision a world of health technology assessment without using cost–effectiveness analysis [2,3].

Comparative effectiveness research has already drawn proponents and detractors in the USA. According to these views, CER will either solve the two scourges of the US healthcare system – high cost and variation in quality – or it will establish ‘death panels’ that will be set on ‘killing granny’. It is clear that some policy makers and purchasers are in favor of comparative effectiveness. Peter Orszag, while he was director of the Congressional Budget Office (Orszag is now director of the White House Office of Management and Budget, Washington DC, USA), said that one key to reducing costs will be to start incentivizing providers to use treatments that have been demonstrated to be more effective with this type of research [4]. However, as a whole, US law makers refused to acknowledge the explicit use of cost concerns when they created a council to oversee CER as part of the American Recovery and Reinvestment Act last year. The language in the bill prohibits the Council from mandating ‘coverage, reimbursement, or other policies for any public or private payer’ [101].

In the absence of cost, CER must differentiate treatments on the basis of measurable outcomes, but to date, there has been very little discussion as to what outcomes will be compared by US policy makers [5]. This is of vital importance given that medical technologies differ vastly in terms of clinical outcomes. There are also nonhealth factors related to treatments that should be considered: quality of life, productivity, functioning, satisfaction and other patient-reported outcomes. Given the complexities of outcomes across patients, policy makers could take the easy way out and choose to focus on a ‘common denominator’ outcomes measure, such as the quality-adjusted life year (QALY).

The current reference point for comparative effectiveness in the USA has been the synthesis of a definition and a list of research priorities from the Institute of Medicine (Washington, DC, USA) [102]. The Institute of Medicine defines CER as: “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition or to improve the delivery of care”. It goes on to say that CER is designed to “assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve healthcare at both the individual and population levels” [102].

Comparative effectiveness is not a new phenomenon in the USA. The country previously experimented with some types of evidence-based coverage decisions; for example, when the Centers for Medicare and Medicaid covered the off-label use of colorectal cancer drugs only if the patient enrolled in an information-gathering study [1]. However, the USA, as of yet, does not have a centralized authority to make coverage recommendations similar to that of the UK, Australia and Canada [6,7]. Those systems mainly use the QALY as a way to measure outcomes across conditions – they are taking a societal view of allocating scarce resources for treatments to create the greatest improvements in health, often for the lowest cost. In one number, QALYs capture a trade-off between morbidity and mortality. One of the major benefits of the QALY is its simplicity – a scale that rates all possible health states from best (full health or 1 QALY) to worst (death or 0 QALYs). Its intuitive appeal is that it can be used as a measure of societal welfare – how many extra QALYs will be gained with a new intervention over the existing one?

The societal point of view used in other countries is not the sole perspective needed for selecting an outcome for a US system of comparative effectiveness. The director of the NIH, Francis Collins, recently highlighted the doubt some US policymakers have when using the QALY to compare two treatments:

“Models that attempt to capture this cost–benefit balance in disability-adjusted life years, QALYs, value of investment approaches or other metrics are only partially successful in providing the kind of information that policy makers need.”[8]

Given this doubt, we find four reasons why QALYs are fundamentally a poor outcomes measurement: average population weights of health states differ from individual appraisals of their own health states, QALYs are inconsistent over time and vary widely depending on the way they are measured, using QALYs as an outcome misses other aspects of a patient’s interaction with the health system and using a population-based comparator like the QALY drowns out voices of small numbers of patients who may need access to high-cost technologies.

Population versus the patient

One can imagine countless real-world examples that may be sufficiently complex to challenge the usefulness of utilities alone. Aspects of personality, disease subtleties and the usual demographic contextual situations such as age, race, gender and socioeconomic status make an overall risk–benefit ratio inadequate for assessing real decision making at the patient level. The use of QALYs for policy decisions remains controversial and inadequate in many areas of healthcare, particularly in mental health where patients may not be empowered to make the same choices as those in full health [9]. QALYs also emphasize living in perfect health, when a patient may prefer to live feeling better, but not necessarily in perfect health [10].

Time inconsistencies

Quality-adjusted life years do not provide consistent outcomes measurements over time for all populations. Utility measurements like the QALY are based on estimates of average life expectancies and probabilities of full health, which are not shared equally across all populations [11]. QALYs cannot distinguish between health states over time. A patient who experiences living in full health for 6 months and then dies will have a QALY of 0.5. This is the same QALY value as someone who experiences partial health for an entire year. Both have different policy implications and neither take into account how individuals coping with the disease in question would prefer to live. People adjust to their health condiditons and may value health states differently over time. Inherently, older lives are valued less [12]. Willingness to trade some health for longer life can also change depending on how the question is phrased [13].

Fixed domains

Many articles in this journal and others have focused on the inability of health-related quality of life measurements to accurately take into account individual patient’s valuations [14,15]. QALYs are centered on global valuations of health states. Since QALYs are often assessed with standard surveys such as the EuroQol 5-Dimensions (EQ-5D™ [EuroQol Group, Rotterdam, The Netherlands]) or the Health Utility Index, they are only able to provide information on the predetermined outcomes measured in the surveys such as mobility, emotion or pain. There are other, more accurate tools available. Conjoint is just one of such methods that has shown to better measure preferences of patients. Conjoint analyses allow individuals to rank certain scenarios that can be tailored to any intervention. Eliciting preferences in this fashion allows researchers to measure not only how patients value trade-offs between certain successful health outcomes, but also other nonhealth characteristics such as the friendliness of staff [16].

Access/innovation

The implications of choosing proper outcomes measurements go beyond just the treatment preferences of patients. Valuing technologies with these metrics can keep valuable new technologies from consumers and are not able to accurately reflect the opportunity costs of allocating resources among the spectrum of available options [2]. While valuable from a payer’s perspective, determining those treatments that are more effective for their price creates a ‘fourth hurdle’ of regulatory approval for makers of new drugs or medical technologies [2]. It currently takes over a decade for a new drug to reach the market in the USA. Institutionalizing the economic evaluation process adds further delay, which could impact drug companies’ decisions to bring a new product to the market. Furthermore, patients in need of a new, innovative medicine may be unable to access it if an agency such as NICE in the UK determines that it is not as effective as an existing treatment.

One way to avoid neglecting the end user of a treatment but still satisfy the efficiency demands of payers may be to establish a middle-ground where both preferences and patient-relevant end points are systematically measured alongside each other. This would be more similar to the model developed in Germany, under the Institut für Qualität und Wirtschaftlichkeit im Gesund- Heitswesen (IQWiG). The agency compares different interventions in the same therapeutic class. If a new treatment is found to have more benefits, the agency sets a ceiling price for how much it is willing to reimburse for the technology for the added benefits [3]. This does not preclude patients from accessing any treatment, although they may be required to pay the difference between the ceiling price and the treatment’s actual cost.

As the USA begins its discussion over comparative effectiveness, it would be wise to learn from international experiences with cost–effectiveness. Researchers need to develop outcomes measurements that reflect the primacy the USA places on individuals, while giving payers the ability to effectively manage finances. The choice of appropriate outcomes for CER is complicated given that it should inform various players in the healthcare system, ranging from patients to clinicians and from manufacturers to payers. It is unclear how outcomes can be chosen to meet the often competing interests of these interest groups. Clinicians, purchasers and policymakers have adequate means and resources for advancing their interests. It is questionable whether the patient’s voice will be adequately represented in this process. Despite the many patient organizations representing specific diseases, only a few are active in research. Pharmaceutical companies fund many of those groups, which could possibly be viewed as corrupting their perspectives. The USA has long been an advocate of the power of the consumer, so comparing any treatment should start with the patients’ perspective. The tide has turned and comparative effectiveness is here to stay, but what form it will take and what outcomes it will measure remain up for debate.

Financial & competing interests disclosure

John Bridges acknowledges funding support form The Institute for Health Technology Studies (InHealth) and Christine Buttorff is supported by a National Research Service Award (T32) Training Fellowship in Health Services Research from the Agency for Healthcare Research and Quality. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.