Limitations of traditional health technology assessment methods and implications for the evaluation of novel therapies

Introduction

Traditional health technology assessment (HTA) draws upon a data modeling and evidence synthesis framework to arrive at conclusions about the clinical efficacy, safety, cost-effectiveness, and budget impact of treatments. HTA is used to guide policy-makers in making regulatory, funding, and/or access decisions. HTA is undertaken in an effort to maximize health benefits given limited resources, and it has many advantages. First, policy-makers are able to combine estimates of benefit and harms along with preferences for those things in an attempt to yield coherent (i.e. non-contradictory) decisions. Typically, HTA uses results from published studies and benefits from formalized, validated, and transparent methods that draw from prior careful research to estimate treatment effects, such as meta-analysis. Graphical models also allow for consensus on clinical and disease processes and serve as a way to easily explain to clinicians the reasons behind the results. In addition, assumptions are often explicit so they can be evaluated for validity and revised if necessary. This last benefit means that, when new treatments or understanding of disease become available, sometimes a critique of assumptions, revisions, or new solutions are needed, and recommendations (e.g. to use more comprehensive measures of value) are made1^,2.

However, HTAs also often include informal methods/assumptions used on an ad hoc basis. Recent changes in drug technology and development, growth in the use of biologics, and other advances in treatment highlight the nagging issues with these less formal and transparent HTAs methods/assumptions and their implementation3^,4. In this paper, we provide guidance for addressing these ongoing challenges to strengthen assessments of novel therapies and technologies. We also discuss how new technology allocates health benefits across patients and how this may require modifications and updates to formal HTA methods to improve the concordance between policy decisions and societal preferences. Many of the points we bring up cannot be handled with better data, but require fundamental changes to the studies that inform HTAs and formal HTA methodology.

We focus on five aspects of HTA where formal HTA methodology is not uniformly applied. These different factors affect the objectives and characteristics of real persons in society either paying for or receiving new technologies. They are:

1. the effect of treatment history of the effectiveness of a new therapy;

2. risks for different patient sub-groups;

3. patient preferences for health technology options themselves;

4. societal preferences for the distribution of health among individuals; and

5. lack of long-run evidence.

In what follows, we discuss how formal HTA methodology can be improved such that drug makers are held accountable for demonstrating real value to patients and society.

The effect of treatment history on the effectiveness of a new therapy

Novel therapies may offer options for patients with limited treatment choices and unmet needs, e.g. those who cannot tolerate or have failed to reach targets with status quo or conventional therapy. Cost-effectiveness analyses, a key component of HTA evaluations, needs to account for the unique risks, costs, and benefits in these patient sub-populations. It is important to make sure that the event risk and benefit profile mirrors that of the real-world population.

Cost-effectiveness analysis (CEA) treats therapy decisions like chance experiments that are similar to rolling a die or drawing a colored marble from an urn. In the case of therapy, these “experiments” involve drugs or treatments that are administered to patients. Like rolling a die or drawing a colored marble, a new drug administration has its own set of possible outcomes pre-intervention. Problems arise when we apply this logic to patients for whom some chance experiments have already been run. Belief in drawing a colored marble changes after a therapy has been tried (post-intervention; Figure 1). Yet, to the extent that a CEA result reports average effects over success and failure cases, the results may not apply directly to patients with one of these known outcomes (Figure 1). A newer drug, if pre-intervention is not cost-effective, may actually be cost-effective post-intervention, when the conventional therapy has failed. It is sometimes the case that such secondary analyses are never evaluated.

Figure 1. Expected benefit of a therapy trial before and after trial.

Conventional therapies provide inadequate or no relief to many patients. For these patients, novel therapies fill an “unmet therapeutic need”—i.e. they offer improvements in health outcomes to patients for whom conventional therapies alone represent an inappropriate level of care. Distinguishing analyses by line of therapy can effectively address this issue; however, some who conduct HTAs do not fully account for the value of these molecules to patients with “unmet need”. Failing to properly account for differences in the probability and magnitude of patient benefit between the novel treatment—which in some cases is used in combination with the conventional therapy—and conventional therapies will understate the relative health improvement for the novel therapy, and over-estimate cost in the incremental cost-effectiveness ratio (ICER).

Comparisons across treatments should account for differences in the likelihood and magnitude of patient improvement between the novel treatment and conventional therapies by, e.g., developing separate HTA analyses for patients who have failed traditional therapy. This will both encourage the use of conventional therapies and also more likely demonstrate cost-effectiveness for persons who have failed certain therapies.

Risks for different patient sub-groups

Drug makers make strategic decisions on drug positioning, and this has downstream effects on HTA. Market access authorities could suggest reimbursement alternatives and improve population health if this process involved formalized methods for understanding baseline risk.

Differences in baseline risk of morbidity and mortality

Some patient populations face an elevated baseline risk of complications, morbidity, and mortality that should be accounted for in HTAs, but the methods used to address this issue are often not applied consistently, or different methods are used to address this issue by the different institutions, agencies, and countries conducting HTAs. Many HTAs attempt to address the issue using a variety of methods and to varying degrees of success. Among patients with an elevated baseline risk of morbidity and mortality the same relative treatment benefit translates to a greater reduction in absolute risk, implying a greater costs-effectiveness in the high-risk patient population than that in the lower-risk or general patient population. Examples include recommendations for proprotein convertase subtilisin/kexin type 9 inhibitors (PCSK9is) in patients with familial hypercholesterolemia, or evidence of clinical atherosclerotic cardiovascular disease (ASCVD)5–9, biologic disease modifying anti-rheumatic drugs (DMARDs) for RA patients with moderate- or high-activity disease who have inadequate response to conventional DMARD (cDMARD) therapy10, and newer therapies when metformin monotherapy fails to provide adequate glycemic control to patients with type 2 diabetes after ∼3 months11^,12. One recent HTA of biologic DMARDs for RA did not properly account for the relationship between HAQ scores and mortality risk, or account for patient heterogeneity13^,14.

To draw a general conclusion, HTAs may rely on approaches that exercise a simplification of risk that does not represent the empirical detail observed in the real-world setting. For example, patients eligible for PCSK9i treatments are generally those for whom statin treatment is not sufficient due to higher baseline cardiovascular risk or statin intolerance15. The higher baseline risk of cardiovascular events among populations eligible for PCSK9i treatment means that the same relative treatment benefits should lead to greater absolute risk reductions and, thus, higher cost-effectiveness of treatment compared with that in a lower baseline risk population. Cost-effectiveness analyses that compare PCSK9i treatments with statins for first-line treatment of hyperlipidemia in the general population would not capture the unique risks among the candidates for PCSK9i treatment. Differences in baseline risk can have a significant impact on cost effectiveness assessment outcomes, as evidenced by recent estimates of ICERs for PCSK9is ranging from $120,000–$350,000 per QALY16–21. Those studies that utilized real-world data to establish baseline cardiovascular event risk for PCSK9i eligible patients had higher cardiovascular event risk and lower cost-effectiveness ratios17^,18^,20.

Differences in patient characteristics by baseline risk

Similarly, clinical guidelines recommend that newly-diagnosed diabetes patients with marked symptoms and/or elevated blood glucose or hemoglobin A1c (≥9%) consider initiating treatment with insulin (rather than metformin)22. The higher baseline risk of diabetes complications (e.g. retinopathy and diabetic kidney disease) among newly diagnosed diabetes patients eligible for insulin treatment means that the same relative treatment benefits may lead to greater absolute risk reductions and, thus, higher cost-effectiveness of treatment compared to a lower baseline risk population if the cost of therapy remains unchanged. Cost-effectiveness analyses that compare insulin (other non-metformin drugs) with metformin for first-line treatment of diabetes in the general population would not capture the unique risks among the high blood glucose/A1c (metformin intolerant/contraindicated) population. Similarly, novel diabetes therapies are recommended when patients fail to achieve glycemic control with conventional therapy (usually metformin). Poor glycemic control is associated with significantly greater risks of complications and mortality23^,24.

By way of an example, we conducted descriptive analyses to demonstrate the differences in baseline patient characteristics, risk, and costs. Using nationally representative data from the National Health and Nutrition Examination and Medical Expenditure Panel Surveys for years 2011–2014 we compared patient characteristics and healthcare expenditures between newly diagnosed (≤12 months) type 2 diabetes patients with hemoglobin A1c ≥9% and (i) newly diagnosed (≤12 months) type 2 diabetes patients with hemoglobin A1c < 9% or (ii) non-recently diagnosed patients. Newly diagnosed patients with A1c ≥9% were significantly (p < .01) younger, less likely to be Medicare beneficiaries, more likely to be obese (body mass index [BMI] ≥ 30), less likely to have renal impairment, high blood pressure, high blood cholesterol, or a history of stroke, and had higher total annual healthcare expenditures ($23,894 vs $5,311, p < .01). Similarly, the American College of Rheumatology Guidelines recommends that RA patients with moderate or high activity disease fail cDMARD therapy before switching to a biologic DMARD10. Patients who use cDMARDs only are different from those who use bDMARDs ± cDMARD/MTX. Our analysis showed that cDMARD users are significantly (p < .01) younger, more likely to have private health insurance coverage, were more recently diagnosed, have higher total annual healthcare expenditures ($21,139 vs $5,286, p < .01), and have lower out-of-pocket total healthcare and pharmacy expenditures. Differences in baseline characteristics likely translate to differences in treatment costs and cost-offsets, which would in turn affect ICERs and, therefore, should routinely calculate ICERs by baseline risk. Compared with CEAs that utilize risk and efficacy estimates based on real world data (RWD), those based on clinical trial evidence often generate higher ICERs25^,26. Several agencies have recommended using RWD to assess baseline risk and efficacy in CEA2^,27–29.

Differences in baseline risk and time-critical decisions

Difference in baseline risk of morbidity and/or mortality may necessitate swift medical intervention, making treatment choice a time-critical decision. If a clinician can justify ignoring a recommendation for a particular patient, then it suggests that the patient’s clinical case is out of guideline scope. Take, for example, time-critical clinical decision-making, which is often neglected in CEA. Formulary decisions to include a particular drug for patients with disease that requires time-critical care should take into account only the drugs that are effective within that time window, given the urgency of intervention and rapid risk reduction. Utility models for time-critical decisions take a different form than those used to measure QALYs, and have not been applied to CEA30, although they represent a clinical reality for clinicians and patients.

For instance, consider patients with osteoporosis at high near-term risk of fracture who require time-critical decisions to prevent future fracture events. In this case, bisphosphonates—which prevent further bone loss rather than stimulating bone growth—may not be an appropriate comparator to bone-forming agents in osteoporosis patients with a high near-term risk of fracture, a problematic comparison recently proposed by one HTA agency31. Likewise, consider the decision to use a rapid anticoagulant reversal agent to treat a life threatening bleeding complication among individuals using vitamin K antagonist anti-thrombotics. Studies have shown that treatment with 4-factor prothrombin complex concentrate (4F-PCC) is superior to fresh frozen plasma (FFP) at reducing international normalized ratios to <1.3 within 30 min of infusion initiation32–34. FFP also requires additional time to match blood type, thaw, and infuse. FFP also comes with a risk of fluid overload due to the large volumes needed35, which can lead to hospitalization complications, increasing the cost of using FFP36.

Alternatively, consider the treatment options for mild-to-moderate vs moderate-to-severe Alzheimer’s, a progressive disease with limited treatment options, especially at the advanced stages when currently available therapies may be ineffective. The limited number of available therapies only have clinically relevant benefits when used at a specific stage of the disease37.

Acetylcholinesterase (AChE) drugs work mainly on mild cognitive impairment, and the decision to use them is time-critical in a progressive disease like Alzheimer’s. The first drug chosen may prohibit successful trials of other drugs as the disease progresses. Often HTA simplifies the treatment decision, assuming that all drugs showing similar efficacy will work the same. However, individual differences in response might dictate that one drug will work best for a patient, while others will not. In the UK, the National Institute on Cost Effectiveness (NICE) recommends picking the cheapest AChE inhibitor because studies show no difference in efficacy between the three AChE inhibitors38. However, what is really needed is an understanding of which treatments work for whom, and which treatment can be delivered to the right patient during the potentially brief time period when they have the mild form of the disease. This may mean that, among patients facing time critical treatment decisions, valid conclusions about the best alternative require a clinical understanding of the patient’s condition.

When assessing the cost-effectiveness of treatments, comparisons across treatments should only be made within a specific, relevant patient population—e.g. high-risk sub-populations—and be based on the specific costs and benefits achieved among these sub-populations. Formalizing methods to account for and assess differences in baseline risk may increase access to treatments among sub-populations of patients for whom the probability of benefit is greatest.

Patient preferences for health technology options themselves

Adherence

Many newer therapies such as biologics offer different delivery mechanisms than traditional small molecule drugs. Sometimes these delivery mechanisms can overlap considerably with health quality. For example, dialysis interferes with social role functions. Yet, even in situations where delivery does not interfere with functions, patients may have strong opinions about treatments that are delivered by different means (e.g. pill vs injection), and may be more adherent when they receive the treatment through the mode of administration that they prefer and/or are committed to and/or that they are required to take less frequently.

For example, despite the often debilitating effects of a migraine39, adherence and persistence to commonly prescribed prophylactic oral medications for episodic and chronic migraine are poor (<80%) and long-run (12–26 week) discontinuation rates are high (∼ 45%) in both randomized control trials (RCT) and real-world/community settings40. In two large RCTs, discontinuation rates for less frequently (every 12 weeks) administered injected onabotulinumtoxin A at 24 weeks was relatively low (10.4%)41^,42. Despite this evidence, and requests for these differences to be taken into account in the Institute for Clinical and Economic Review’s 2014 assessment of migraine treatments43, the subsequent evaluation did not adequately account for differences in medication adherence and persistence44.

Similarly, persistence with injectable bone-forming treatments for fracture prevention in osteoporosis have significantly—nearly double in one study—better adherence and persistence rates than daily oral bisphosphonates/anti-resorptive therapies45–50. Newer injectable bone-forming agents that require less frequent (monthly) administration may have even higher persistence rates; however, these potential differences in treatment adherence have been overlooked in at least one CEA of bone forming agents31. Overstating patient medication adherence will overstate treatment benefits (e.g. QALYs gained) and bias comparisons among treatments. Alternatives that may be developed in the future could offer clinically comparable effects, but be preferred to their traditional counterparts based on route of administration, dosing frequency, or both, making patients more likely to adhere to them.

While some HTAs have accounted for changes and differences in adherence well, others have not. To overcome this issue, HTA should consistently and uniformly adjust for differences in real-world medication discontinuation and adherence rates among therapies. Although real-world adherence data may not be available for the drugs being evaluated at the time of the evaluation, data on real-world adherence to related drugs or drugs with similar characteristics are likely available, and could be utilized to forecast adherence to the drug being evaluated.

Choice

Participation in treatment begins with choice of treatment. In evaluating cost-effectiveness, though, in general the HTA framework does not consistently take into account the causal effects of having choice, nor does it consider whether having choices in and of itself may lead to lower economic costs or improved patient outcomes. Specifically, HTA focuses almost exclusively on assigning to treatments a funding recommendations or rejection. Choice is not part of this assignment procedure. Furthermore, the traditional drug regulatory framework is not well equipped to handle comparisons between different treatment modalities, e.g. drugs vs surgery or a medical device. Patients consider many factors when making treatment decisions, e.g. convenience, that are omitted from traditional HTA cost-effectiveness models. The cost-effectiveness component of HTAs may omit patient perspectives/experiences, which may significantly affect the assessment of treatments’ clinical and economic/cost-effectiveness and result in different agencies issuing opposite recommendations for the same therapy51–53. When novel oral anticoagulants (NOACs) are compared to warfarin, HTAs often consider lower bleeding risk and greater stroke risk reduction, but may under-value the reduced burden of monitoring burden of NOACs relative to warfarin. Patients are willing to pay significantly more for the increased stroke risk reduction and decreased risk of major bleeding events offered by NOACs54. In addition, the increased convenience of NOACs may result in greater medication adherence55. Not accounting for the effect of choice may dilute ICERs and fails to capitalize on the direct cost-effectiveness benefits choice may have to offer.

When data are available, one solution is to incorporate doubly randomized preference trials into HTA evaluations to capture the benefits—e.g. QALY gains or direct medical cost off-sets—associated with patients having the ability to choose between therapies56. While there may be a moral hazard associated with the choice of treatment for both patients and providers, as well as supplier induced demand through marketing, when patients are fully informed, randomized preference studies in health and mental health research have demonstrated that giving patients a choice of therapy is sometimes cost-effective57–61. When choice is cost-effective, such doubly randomized trials suggest that drug formularies should include both the status quo and the comparator treatment and let patients choose between them. Currently, HTA thus far has not been concerned with the issue of choice and how it may affect outcome. Real value to patients may be squandered when there are substantial comparator differences that could affect choice and in turn outcome (e.g. drug vs device, pill vs injectable, or behavioral treatment vs pharmaceutical treatment).

Societal preferences for the distribution of QALYs among individuals

In addition to concerns that directly affect the patient are social concerns for how healthcare benefits are allocated and to whom. In traditional CEAs, which inform HTA, the QALY gains of all patients are weighted equally, ignoring the distribution of lifetime QALYs across a heterogeneous patient population, which may differ. General population stated preferences suggest that society may prefer to consider severity of illness or lifetime QALY allocations when distributing health gains. Generally people prefer to devote some resources to relatively sicker patients at the expense of efficiency (i.e. total QALYs produced)62–65. Empirical studies with representative samples show public support for this62^,65^,66. Such descriptive findings are not just of academic interest, but actually influence the public’s perception of HTA and their support or rejection of it67. For example, the uproar among politicians and policy-makers after it was shown that Viagra™ was more cost-effective than a kidney transplant can be explained by preferences for helping those persons with more severe disease68. If we want the public not only to support HTA, but also benefit from it, then we must accurately describe their preferences for health allocations. This has led to special exceptions in HTA policy-making, but ones that do not include explicit decision criteria for equity acceptance that comport with societal stated preferences for equity. Technical approaches that evaluate not only quality-adjusted life years, but also how policies distribute them, come closer to this goal62. Yet, these approaches are not typically part of HTAs.

Healthy and ill individuals value healthcare and medical innovation for different reasons. For healthy people, access to healthcare/novel treatments reduces the financial risk of falling ill, but for sick people it reduces the physical risks of being ill. The more likely and/or more severe the disease, the more valuable the treatment to healthy people69. Fairness or equity concerns arise when members of society do not believe that health benefit is being allocated to persons who most deserve to enjoy benefits. Often, when persons have experienced considerable health over their lifetime, there is the preference to give some health to those who have been worse off. In such cases, there is a tradeoff between efficiency and equity, with a willingness to sacrifice some of the former for the latter. While it is often recognized in HTA that equity concerns are important, there has been a lack of uptake of formally accepted methods for addressing equity, which reduces transparency65^,69.

Consider a society with four individuals. As illustrated in Figure 2, Policy A and Policy B both generate a health benefit of 190 QALYs, and are, therefore, equivalent by traditional HTA standards, even though Policy A distributes a greater proportion of the total QALY gain to the sickest individuals than Policy B. Society likely prefers Policy A to Policy B. Moreover, suppose Policy A was comprised of a set of funded treatments to beneficiaries (e.g. Medicare, single-payer, etc.), then persons in society may find it distasteful if policy-makers switched to another set of funded treatments, Policy B, that took QALYs from the sickest and gave them to the those wealthiest in QALYs. Some treatments that are expensive by traditional HTA standards may have a more favorable ICER when consideration is given to who receives the QALYs from the treatment. Just as progressive taxes are often implemented to mitigate the societal problems associated with higher income inequality, inequalities in health may need some mitigation as well.

Figure 2. Distribution of QALYs across individuals by policy. Note, both policy options offer the same total increase in QALYs, and, therefore, are equivalent by traditional HTA standards. However, society may prefer Policy A to Policy B because A distributes a larger share of the QALY gains to relatively sicker individuals (Persons 3 and 4).

One solution to this issue is to adjust cost-effectiveness ratios for societal preferences to more equitably allocate QALYs, e.g. placing greater weight on QALY gains to those with relatively lower baseline QALYs65. In doing so, an intervention that improved outcomes for relatively sicker patients would be more desirable/valuable from a societal perspective than one that improved outcomes by the same absolute amount for healthier patients. Modeling decisions about societal utility weights should be a formalized part of HTA and, therefore, explicitly discussed and fully transparent.

Lack of long-run evidence

In making drug approval decisions, the costs and benefits of granting earlier approval based on surrogate end-points must be weighed and, often (e.g. in many cancer types and type 2 diabetes), the expected benefits outweigh the costs15^,70. HTAs often use information on adverse events observed in clinical trials to infer real-world and long-run side-effects. However, there may be long-run outcomes and or/side effects of use that clinical trials are unable to detect because they do not follow patients long enough or cannot follow them long enough to observe the ideal end-point because of cost, feasibility, ethics considerations, and/or other constraints. The level of competition between drug developers and the prevalence of the disease both affect the pool of potential trial participants and trial duration. The more competition and/or less prevalent the disease, the smaller the trial participant pool, and the shorter and/or smaller the trial71.

Yet, just because HTA can be applied and a policy decision can be made does not necessarily mean that it should. It is possible that the cost of making the wrong decision now based on a limited evidence base outweighs the benefit of making an earlier decision. For instance, RCTs studying opioids for chronic (>6 months) non-malignant pain ran for ∼12 weeks72–74, and were too short to detect the issues with addiction and overdose observed in long-term users of these medications. Also, in clinical practice doses are typically titrated over time, leading to variation in adverse event profiles72.

Standardized HTA methodology should include confidence intervals that reflect the underlying uncertainty in outcomes when long-run outcomes and/or side-effect evidence is limited, and the potential costs of making an inappropriate policy decision are high, and there should be a consensus on the thresholds.

Discussion

Most criticisms of HTA focus on conflicts of evidence, such as the appropriate risk estimate to use or most accurate preference estimate for a health outcome. This paper focuses on the conflict between methods of studies that HTA relies upon and how they relate to the objectives and characteristics of real persons in society either paying for or utilizing novel therapies/technologies. Such objectives need to be expressed and managed appropriately in any formal assessment that will inform policy. Health technology assessment is valuable if it leads to treatment decisions that result in better patient outcomes. As drug development evolves and advances, HTA methods and standards must also evolve to address new issues and situations.

In this study, we have presented five diverse issues with inconsistently applying informal HTA methods to novel therapies. Differences in response to treatment and morbidity and mortality risk within the patient population of interest can be addressed by adjusting for differences in treatment efficacy and baseline patient characteristics, including morbidity and mortality risk, in the comparison of treatment efficacy and cost. Adjusting for differences in real-world medication discontinuation and adherence rates among therapies—and the resulting differences in efficacy—as well as taking the effects of having choice in treatment into account via evidence from doubly randomized preference studies when possible is also advised. Assessments should also consider placing greater weight on health gains for relatively sicker/less healthy patients to be consistent with empirical evidence on societal preferences for distributing health benefits. Lastly, HTAs should utilize long-run evidence whenever possible, and should be revised when new information on long-run evidence becomes available. HTA should not be applied in cases where decisions based on weak and/or short-run evidence could lead to inappropriate and costly decisions. HTA is a valuable exercise that has come a long way over time, but further refinements are needed to adapt the framework to an ever-evolving drug innovation environment.

Transparency

Declaration of funding

This commentary was funded by Amgen.

Declaration of financial/other relationships

JD is a consultant/advisor for Precision Health Economics & Amgen. JPM is an employee of Precision Health Economics, a research consulting firm owned by Precision Medicine Group and compensated by Amgen to conduct the study. Peer reviewers on this manuscript have received an honorarium from CMRO for their review work, but have no other relevant financial relationships to disclose.

Acknowledgments

The authors would like to thank the three anonymous reviewers for their thoughtful feedback and input on the manuscript.