Does meta-analysis of economic evaluations have the potential to play a role in healthcare decision-making in the United States?

Abstract

The current use of economic evidence in the decision-making process in the US is increasing. Meta-analysis of economic evaluations (MAEE) has gained recognition in recent years and can support decision-making at the global level or for countries with resource constraints. The focus of this article is to demonstrate how MAEE may contribute to the decision-making process in the US healthcare system. We demonstrated that MAEE can provide an efficient mechanism to quantitatively summarize cost-effectiveness findings based on all existing studies from the US answering the same question across different assumptions. This sort of evidence is important for US policymakers to support policy decision-making. MAEE methods can streamline the process of reviewing complex economic models and their findings, which has been previously reported by stakeholders as a barrier to the use of economic evidence in decision-making in the US. However, the currently proposed method may not fully address the issue of heterogeneity observed among MAEEs. There is a critical need to explore sources of heterogeneity and develop a standardized approach to handle it to improve the efficiency and acceptability of future MAEEs.

Keywords:

• Meta-analysis
• economic evaluations
• cost-effectiveness analysis
• United States

JEL classification codes:

• C18
• C1
• C
• C54
• C5

Introduction

Since the 1990s, the number of economic evaluations (EEs), such as cost-effectiveness or cost-utility analyses (CEA/CUA), has grown considerably, involving a wide range of diseases and interventions¹. Although the evidence on the cost-effectiveness of health interventions is increasingly being used in healthcare decision-making globally, the extent to which this evidence influences decision varies by country. Many public and private organizations in developed countries have formally adopted economic evidence to inform healthcare policy and reimbursement decisions¹. Meanwhile, considering economic evidence in healthcare decisions has long been viewed as unattractive in the United States (US). One exception is the use of economic evidence by the Centers for Disease Control and Prevention’s (CDC) Advisory Committee on Immunization Practices (ACIP) to inform national recommendations on immunization policy².

Recently, a growing concern about rising costs and inefficient health care spending has led to greater use of economic evidence in public and nonprofit organizations’ health care decisions in the US. For instance, the Institute for Clinical and Economic Review (ICER) Group, a US-based independent, nonprofit organization, generates drug value assessment reports to provide a reasonable benchmark with which to assess drug prices set by manufacturers and assess the success of public and private strategies designed to place downward pressure on those prices³. A drug value assessment report is a systematic review of a drug’s comparative clinical effectiveness in comparison to other drug treatment options, combined with an analysis of its long-term cost-effectiveness versus those other options³. Recently, such reports are increasingly being used by state Medicaid programs, many of which have deployed them for prior authorization elements of coverage decisions³. The US Department of Veterans Affairs is making greater use of them in price negotiations with manufacturers³. Private insurers and pharmacy benefit managers have started using them in their coverage processes as a guide to fair pricing and fair access^3,4. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) special task force on the US value assessment frameworks proposed greater use of economic evidence in payer’s decision-making process in order to better serve the interest of the plan members and patients who they represent⁵. Recent surveys of US payers, including decision-makers in managed care, pharmacy benefit management organizations, hospitals, and others, revealed that 59% of respondents or their organizations had used ICER's assessments in formulary decisions⁴. Moreover, medical professional societies and other organizations also initiated to develop practice guidelines that take economic evidence into account (e.g. American College of Cardiology and the American Heart Association (ACC–AHA), the American Society of Clinical Oncology (ASCO), etc.)¹. It is clear from the evidence gathered to date that economic evidence can be a valuable supplemental tool in the decision-making process in the US. The current use of economic evidence is increasing, and stakeholders especially payers are using them in a number of different ways^4,6. However, stakeholders observed several challenges such as reviewing complex economic models and their findings, the risk of potential bias, and other concerns that have been discussed in greater detail elsewhere^4,5,7,8, while employing economic evidence in decision-making in the US.

Why meta-analysis of economic evaluations?

The use of EEs to inform decision-making has rapidly increased in recent decades across the world. Decision-makers are confronted with often unmanageable amounts of information and conflicting findings from such evaluations. Systematic reviews of economic evaluations (SREEs) can provide an efficient mechanism to synthesize existing cost-effectiveness evidence to answer a specific question⁹; the number of such evaluations is steadily increasing. However, SREEs provided only a descriptive and qualitative summary of evidence. The Cochrane handbook for Systematic Reviews of Interventions discussed that there are no agreed methods for pooling estimates of cost-effectiveness but did not expand on the issues¹⁰. Later, Crespo et al. 2014 suggested statistical methods for performing meta-analysis of economic evaluation studies (MAEE) using comparative efficiency research (COMER)¹¹. The incremental net benefit (INB) is used as the outcome of interest in this approach, rather than the more commonly used incremental cost-effectiveness ratio as seen in CEAs. Our research teams have recently further modified their methods and devised a step-by-step data harmonization to ease in performing MAEE effectively¹². A detailed methodology for performing MAEE that covers all aspects has been published elsewhere¹². The Immunization and Vaccine-related Implementation Research Advisory Committee (IVIR-AC)-World Health Organization (March 2021) agrees that the quantitative evidence generated from MAEEs was useful to support clear policy recommendations and can facilitate decision-making in resource-strained settings where context-specific EEs are not available¹³. Instead of using MAEEs to support decision-making at the global level or for countries with similar resource constraints, the rest of this article uses examples of MAEEs to describe the potential application of MAEEs in the US healthcare setting.

The potential application of MAEE in the US healthcare setting

The first example demonstrated the findings from a MAEE comparing the cost-effectiveness of Recombinant Zoster Vaccine (RZV) vs. no vaccination (NoV) using four studies^14–17 conducted from a societal perspective in the US (Figure 1). All studies were CEAs reporting cost per quality-adjusted life-year (QALY). Previously, CDC conducted a CEA comparing RZV, Zoster Vaccine Live (ZVL), or NoV from a societal perspective, using time horizon of lifetime¹⁴. It estimated that vaccination with RZV, compared with NoV, was cost-effective for immunocompetent adults aged ≥50 years. Currently, the ACIP Herpes Zoster Vaccines Work Group is recommending RZV for immunocompetent adults aged ≥50 years for the prevention of Herpes Zoster². We included three more CEAs in this MAEE^15–17, one of which was a manufacturer-sponsored analysis¹⁶. The primary outcome of interest was INB. A positive INB favors the intervention as cost-effective, whereas a negative INB favors the comparator with the intervention being not cost-effective. In this meta-analysis, quantitative synthesis of economic evidence indicated that RZV is significantly cost-effective compared to NoV (pooled INB, $248.72 (95% CI, $112.42, $385.01), I² = 45.3%), which is consistent with ACIP's recommendation which supports the use of RZV². MAEE methodology uses a systematic review process to identify comprehensively all studies for a specific focused question, appraise the model and assumptions of the included studies, and scrutinize their overall quality¹². This streamlines the process of reviewing complex economic models and their findings, which has been previously cited by payers^4,8 and other stakeholders⁷ as a barrier to using economic evidence in decision-making in the US. In addition, this process may also enable to generate a collection of study strata with similar characteristics, for example, subgroup of population, perspective, assumptions of EE model and clinical settings, study quality, funding, etc., allowing further analyses such as sensitivity or subgroup analyses to observe how findings varied across different conditions.

Figure 1. MAEE comparing Recombinant Zoster Vaccine (RZV) vs. no vaccination (NoV) in the US.

Generally, CEAs that address the same question are likely to generate varying or conflicting findings, which might be due to the substantial variability in assumptions (such as willingness-to-pay (WTP) threshold, perspective, model assumptions, etc.) employed in their analysis¹⁸. As shown in this example, four studies that addressed the same question generated somewhat varied results (Figure 1 and Table 1). Two studies (Curran, 2018; Carpenter, 2019) that used a willingness-to-pay (WTP) threshold of $100,000/QALY failed to attain statistical significance, but one study (Prosser, 2019) with a threshold of $100,000/QALY and another (Le, 2018) with a threshold of $50,000/QALY did. MAEE can provide an efficient mechanism to quantitatively summarize cost-effectiveness findings based on all existing studies from the US answering the same question across different assumptions. This sort of evidence might be important for US policymakers who want to endorse a clear policy recommendation. Furthermore, as shown in Table 1 additional analyses can be performed by applying a single WTP threshold value across individual studies, thus minimizing the variability in WTP threshold across studies and deriving pooled INB from MAEE when all studies used the same WTP threshold. WTP threshold is used as a cut-off for determining whether an intervention is cost-effective in CEAs. Even though there was no specific threshold explicitly stated, a range of $50,000–150,000 has been used in the context of the US healthcare system¹. In this example, additional analyses illustrate that RZV is cost-effective in all acceptable threshold ranges in the US, including $50,000, with no heterogeneity (Table 1), demonstrating the value of this intervention. Identifying the cutoff threshold value at which an intervention would be cost-effective is a piece of valuable information that may support decision-making, and MAEE can offer such evidence using this approach.

Table 1. The calculated INB values and MAEE results when WTP threshold cut-offs were varied and applied uniformly across studies for cost-effectiveness of Recombinant Zoster Vaccine (RZV) vs. no vaccination (NoV) of four studies conducted from a societal perspective in the US.

WTP Threshold (WTPT)	INB [95% CI]				Pooled INB [95% CI]	I^2
	Le, 2018^18	Curran, 2018^17	Carpenter, 2019^16	Prosser, 2019^15
Main analysis
WTPT ($50,000–$100,000)*	150.88	200.15	261.78	406.07	248.72	45.3%
	[30.02, 268.75]	[–85.55, 485.84]	[–33.72, 557.28]	[225.91, 586.23]	[112.42, 385.01]
Additional analyses [using a single WTPT]
$50,000	150.88	85.60	100.71	149.91	127.33	0%
	[30.02, 268.75]	[–139.39, 310.59]	[–76.77, 278.19]	[59.69, 240.14]	[71.19, 183.48]
$100,000	395.03	200.15	261.78	406.07	335.74	0%
	[174.03, 616.03]	[–85.55, 485.84]	[–33.72, 557.28]	[225.91, 586.23]	[228.39, 443.09]
$150,000	656.19	314.70	422.84	662.24	532.71	0%
	[342.78, 987.59]	[–36.86, 666.26]	[9.29, 836.39]	[392.07, 932.39]	[376.96, 688.45]

*WTP Thresholds used in the main analysis (Figure 1) WTP = $50,000/QALY [Le, 2018]; WTP = $100,000/QALY [Curran, 2018; Carpenter, 2019; Prosser, 2019].

Abbreviations. INB, Incremental net benefit; MAEE, meta-analysis of economic evaluations; WTPT, willingness-to-pay threshold.

Heterogeneity: a potential challenge in MAEE

Although we discussed several possible MAEE applications in the US healthcare setting, one of the major methodological challenges, i.e. the large heterogeneity observed in MAEEs¹², has yet to be fully addressed. There are many sources that might contribute to heterogeneity while performing MAEE, including study characteristics (such as setting, WTP threshold, perspective, funding source) and methodological characteristics (such as time horizon, data source, model type, input parameters, and assumptions)¹⁸. As an example, EEs that utilized the same meta-analysis of clinical effectiveness as an input parameter in the model are more likely to have similar cost-effectiveness findings, allowing the results to be pooled with minimal heterogeneity. It is important to emphasize that because we are pooling findings across those EEs, this is not directly related to the issue of double counting as seen in traditional meta-analysis of clinical effectiveness. Instead, this is a part that would explain potential differences in findings across studies. The larger the differences in input parameters and assumptions used across EEs, the more likely substantial heterogeneity in MAEE would be observed. As shown in the previous example, we observed a high level of heterogeneity of 45.3% in the main analysis (Figure 1). However, when we used a single WTP threshold value across individual studies while performing MAEE, heterogeneity is reduced to zero in all scenarios as shown in Table 1. It suggests that variation in the WTP threshold observed in individual studies might be one source of heterogeneity. Table 2 provided findings on heterogeneity from MAEEs comparing Direct Oral Anticoagulants (DOACs) (i.e. dabigatran, rivaroxaban, and apixaban) versus warfarin in patients with atrial fibrillation in the US setting¹⁹. With the exception of the comparison between rivaroxaban vs warfarin, we found significant heterogeneity in all other comparisons. As shown in Table 2, when we standardized the WTP threshold across individual studies into a single value, we found only a minimal change from heterogeneity observed in the main analysis. As an example, MAEE comparing Dabigatran vs. warfarin, demonstrated substantial level of heterogeneity (I² = 89.28%) in the primary analysis (Table 2). Among 9 studies included in this MAEE, three were manufacturer’s sponsored economic analyses¹⁹. It is expected that findings in cost-effectiveness analyses may vary by funding source and there is likelihood of observing high level of heterogeneity when we pool these studies. The potential underlying reason for this phenomenon is the assumptions applied to the economic evaluations that vary by funding source. According to meta-regression, funding source partially explains heterogeneity observed in this MAEE¹⁹. This clearly demonstrates that, in many instances, other factors, in addition to WTP, might contribute to heterogeneity. This indicates a strong need to explore other drivers that may be a potential cause of heterogeneity.

Table 2. The changes in the magnitude of heterogeneity (I²) compared to the main analysis when WTP threshold cut-offs were varied and applied uniformly across studies for the cost-effectiveness of Direct Oral Anticoagulants in the US.

Comparisons (No. of studies)	I^2 (%) in main analysis WTPT-50,000 to 100,000	I^2 (%) in MAEEs using single WTPT [changes of the magnitude of I^2]
		$50,000	$75,000	$100,000	$125,000	$150,000
Apixaban vs. Warfarin (n = 7)	63.99	50.93	69.23	77.05	81.02	83.29
		[–13.06]	[5.24]	[13.06]	[17.03]	[19.3]
Dabigatran vs. warfarin (n = 9)	89.28	91.74	90.28	88.97	87.43	86.24
		[2.46]	[1.00]	[0.31]	[1.85]	[3.04]
Rivaroxaban vs. warfarin (n = 6)	0	3.22	0	0	0	0
		[3.22]	[0]	[0]	[0]	[0]

Abbreviations. WTPT, willingness-to-pay threshold.

Conclusions

The current use of economic evidence in the decision-making process in the US is increasing. We believe that MAEE makes SREEs more meaningful and can provide an efficient mechanism to quantitatively summarize cost-effectiveness findings based on all existing studies from the US answering the same question across different assumptions. There is a critical need to explore sources of heterogeneity seen among MAEEs and to develop a standardized approach to handle it.

Transparency

Declaration of funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Declaration of financial/other relationship

The authors declare that they have no known conflicts of interest that are directly relevant to the content of this article.

A reviewer on this manuscript has disclosed that they are an employee of PRECISIONheor, a health economics consulting firm that provides services to pharmaceutical companies. I also hold equity in our parent company, Precision Medicine Group. The other peer reviewers on this manuscript have no other relevant financial relationships or otherwise to disclose.

Author contributions

SKV, AT, and NC designed and drafted the article.

SKV and SS performed data preparation and analysis.

Code availability

The data analysis file is available from the corresponding author upon request.

Consent for publication

All authors approved the final version of the manuscript for submission.

Acknowledgements

None stated.

Data availability statement

All data are available from the corresponding author upon request.