Comparative analyses of published cost effectiveness models highlight critical considerations which are useful to inform development of new models

Abstract

Background: Comparative analyses of published cost effectiveness models provide useful insights into critical issues to inform the development of new cost effectiveness models in the same disease area.

Objective: The purpose of this study was to describe a comparative analysis of cost-effectiveness models and highlight the importance of such work in informing development of new models. This research uses genotypic antiretroviral resistance testing after first line treatment failure for Human Immunodeficiency Virus (HIV) as an example.

Method: A literature search was performed, and published cost effectiveness models were selected according to predetermined eligibility criteria. A comprehensive comparative analysis was undertaken for all aspects of the models.

Results: Five published models were compared, and several critical issues were identified for consideration when developing a new model. These include the comparator, time horizon and scope of the model. In addition, the composite effect of drug resistance prevalence, antiretroviral therapy efficacy, test performance and the proportion of patients switching to second-line ART potentially have a measurable effect on model results. When considering CD4 count and viral load, dichotomizing patients according to higher cost and lower quality of life (AIDS) versus lower cost and higher quality of life (non-AIDS) status will potentially capture differences between resistance testing and other strategies, which could be confirmed by cross-validation/convergent validation. A quality adjusted life year is an essential outcome which should be explicitly explored in probabilistic sensitivity analysis, where possible.

Conclusions: Using an example of GART for HIV, this study demonstrates comparative analysis of previously published cost effectiveness models yields critical information which can be used to inform the structure and specifications of new models.

Keywords:

• Comparative analysis
• economic evaluation
• cost effectiveness modeling
• health economics methodology

JEL CLASSIFICATION CODE:

• I10

Introduction

Comparisons of cost effectiveness models are frequently undertaken and published^1–7 to assess quality, bias⁸ and transferability of results^9,10. When developing a new cost-effectiveness model, it is important to investigate and compare previous models to gain insight into analytical approaches, model assumptions and natural history of disease, which remain relevant over time^11,12. Such comparisons are less frequently published. Those which have been published use methods such as exchanging models^1,13, “experimental” models² and sensitivity analyses to compare data and assumptions¹⁴. However, due to resource constraints and the proprietary nature of models, it is often necessary to perform comparisons based on publications alone, which is time consuming and challenging, in part due to inconsistency in outcome selection, discrepancies in terminology, and reporting^4–6,11,13.

This research expands on the limited published literature in this area by undertaking a comprehensive comparison of published models on a specific topic (genotypic antiretroviral resistance testing), to gain insight into critical issues which can be used to inform development of new cost-effectiveness models.

Methods

A systematic search identified models evaluating GART after first line (1 L) antiretroviral (ART) failure. Selection was restricted to manuscripts reporting results of full economic evaluations which measured both cost and consequences¹⁵ (full search criteria in Supplementary Appendix). A comprehensive, qualitative comparative analysis was undertaken to describe all aspects of the models using publicly-available primary publications and Supplementary Material. No direct contact was made with the authors regarding the published models. Uncertainty in interpretation of the models was resolved by economic and clinical study team members. Accepted principles of good practice in modeling were applied as a frame of reference for the comparative analysis^16–19 (full methodology described in the Supplementary Appendix).

Published model results were compared to understand how the factors modeled related to the overall results. Reported results were compounded to present day values according to the method by Welte⁹. Reported costs were converted to original currency according to historic exchange rates. Values were adjusted for inflation according to country-specific GDP deflators for source year and latest year available²⁰. Values were converted to USD according to price level ratio of Purchasing Price Parity (PPP) conversion factor (GDP) to market exchange²⁰. Health benefits were assumed constant with no discounting of health outcomes (full methodology described in the Supplementary Appendix).

Results

Five published cost-effectiveness models met eligibility criteria for the comparative analysis (hereafter named according to first author)^21–25. The main results of the comparative analysis are presented below with the full comparative analysis provided as Supplementary Material.

Model comparator, time horizon and scope

Key differences were identified in comparator, time horizon and scope used across models. Weinstein, Corzillius, Sendi and Levison compared GART to no GART, whereas Phillips used “no monitoring and no second line (2 L) treatment”. For time horizon, Weinstein, Corzillius, Sendi and Levison use a lifetime model and Phillips reports a time horizon of ten years (2015–2025).

With respect to scope, some models were more comprehensive than others. Sendi and Levison modeled patients from time of 1 L initiation to 2 L treatment failure only, whereas Weinstein modeled treatment-naïve patients up to and including 2 L failure, including disease progression and death. Corzillius modeled patients at diagnosis through four treatment regimens and failures to death. By comparison, Phillip’s original HIV synthesis model begins with HIV incidence, progressing from treatment-naïve to 2 L failure, disease progression and death, accounting for adherence and transmission^26–30. The paper included in this comparison (Phillips 2014) focuses on 1 L failure and follows patients for ten years, including disease progression and death.

Model assumptions of prevalence of drug resistance, ART “efficacy”, GART “efficacy” and whether to switch therapy to 2 L for GART positives

Both Weinstein and Levison used a base case prevalence of 20% for wild type virus and primary DR respectively. Corzillius, Sendi and Phillips did not explicitly report primary or secondary prevalence of DR.

Model assumptions for ART “efficacy” differed widely between studies. Weinstein assumed 40% of patients failed IL at 24 weeks and 30% at 48 weeks, with all patients assumed to fail 1 L at 24 months, based on a matrix derived from clinical study data. The 18% probability of 1 L failure in Corzillius was based on prior published data³¹ and Sendi varied their probability according to levels of HIV drug resistance (DR) defined by Haupts³². Levison modeled an “initialization cohort” of which 25% fail 1 L, and at the start of the ten-year period Phillips assumed that 13% of patients have failed 1 L.

For GART “efficacy”, Weinstein considered the proportion of patients on suppressive ART as 34% after GART, versus 22% without GART. Corzillius reported a relative risk of 0.79 for the protective effect of GART, and the probability of virologic failure was reduced by 21%. Sendi assumed a relative risk of 1.27 for treatment failure without GART versus with use of GART. Assumptions about the effect of GART on virologic suppression were not explicitly stated in Levison and Phillips.

There was a wide range of assumptions about whether patients stay on 1 L or switch to 2 L after GART/no GART. Sendi assumed 67% of patients would change to 2 L after GART while 33% continued on 1 L, and for patients without GART, 97% would change to 2 L while 3% stayed on 1 L³². Levison assumed a proportion of patients without DR would achieve viral load suppression (VLS) by continuing 1 L, and the proportion with DR switching to 2 L after GART was not reported. Phillips’s probability of switching to 2 L was assumed as 0.3 every 3 months, irrespective of the GART strategy used. The proportion of patients changing regimens after GART/no GART in the Weinstein and Corzillius models was not explicitly stated.

Modeling disease status (CD4, VL) and disease progression (AIDS, OI)

A wide range of approaches for modeling disease status and disease progression were used across studies. Weinstein used CD4 count and VL as surrogate markers for disease progression. Both CD4 count (cells/μL) and VL (copies/mL) were divided into six strata and associated with states of VLS and virologic failure. In those with VLS, CD4 increased by 0.083 (GART) or 0.060 cells/μL (no GART) per month and VL decreased to <500 copies/mL. In virologic failure, VL increased one stratum/month to viral set-point (not specified) and CD4 decreased at a monthly rate governed by VL. Weinstein then modeled disease progression based on CD4 count and risk of developing opportunistic infections (OI).

Corzillius had a starting CD4 count of 360 cells/μL, which increased by a mean of 150 cells/μL after every 6-month cycle. In virologic failure, CD4 count decreased 1 year after failure (two 6-month cycles). Corzillius used VL only as a surrogate for disease progression based on prior data³³. Baseline VL was 10,000–30,000 copies/mL³⁴.

Sendi used a combined assessment of CD4 count, VL, and presence (or absence) of AIDS-defining conditions to determine disease progression. After year three, Sendi stratified AIDS-free patients according to CD4 strata 0–200, 201–500 and >500 cells/μL and VL <1,000, ≥1,000 copies/mL.

Levison set a mean starting CD4 count of 173 cells/μL. The paper describes a "CD4 count increase at 24 weeks (all strategies/cohorts)" of 148 cells/μL, which presumably impacted clinical risk, although it was unclear precisely how this was applied in the model. In 1 L failure, a mean monthly CD4 decrease was applied according to VL strata (0–500; 501–3,000; 3,001–10,000; 10,001–30,000; >30,000 copies/mL). VL decreased in VLS and the rate of virologic failure was not specified. Levison appeared to determine risk of developing an OI based on CD4 count.

Phillips included changes in CD4 and VL as a function of adherence, number of active drugs, time on the current regimen and VL (Phillips supplementary 1)²⁶. The original publication supplement describes VL as a mean change from VL maximum at <3 months, 3-6 months and >6 months (Phillips supplementary 1)²⁶. It also describes a patient-specific factor built into the model, which captured the variable tendency for CD4 count to rise on ART for any given level of VLS. The model considered (i) the rate of increase of CD4 count reduces over time in treated patients, (ii) this rate of increase is slower in patients who have been on ART >2 years, (iii) CD4 count decreases faster if patients are not on a PI based regimen, (iv) there is person-specific variability in the rate of CD4 count change, and (v) that CD4 count and VL changes will be different when ART is interrupted (Philips 2007 supplement)²⁶. How changes in VL were modeled was not fully evident from the primary paper and related publications^28–30,35. Disease progression depended on CD4 count, age and OI prophylaxis. In Phillips, the risk of clinical disease (and death) reported in the 2007 Supplementary Methods link CD4 strata to WHO disease stage 3 and 4 (categories of AIDS-related OIs), which consequently inform risk of AIDS. OIs are not mentioned in the 2014 paper.

Modeling quality of life

Two of the five models include quality of life (QoL) as an outcome. Weinstein used utility derived from a previous study³⁵, based on the single global health status question, “How would you rate your current state of health?” rated as excellent, very good, good, fair and poor, and data were transformed to utility weights according to Torrance³⁶. Sendi used utility derived from a previous study by Zinkernagel, who collected QoL data from the HIV Medical Outcome Study^35–37 among 41 asymptomatic patients with a CD4 count >400³⁷. Data were transformed to utilities using methods described by Mrus³⁸. The values used in Weinstein and Sendi are shown in Table 1.

Table 1. Weinstein and Sendi’s utility values according to CD4 count +/− AIDS status and VL.

Parameters	Utility	Model
CD4 Stratum (cells/µL)
0–200 (no AIDS)	0.739	Sendi
50–100	0.81	Weinstein
100–200	0.87	Weinstein
201–500 (no AIDS)	0.78	Sendi^a
>200	0.94	Weinstein
≥500 (no AIDS)	0.801	Sendi^a
Viral Load
Undetectable viral load (first 2 years)	0.79	Sendi^a
Detectable viral load (<2 years)	0.755	Sendi^a

^aSendi also included the disutility for a detectable VL (>2 years)=0.035 [37] and disutility of an AIDS indicator disease = 0.233.

Standardized base case results

This comparison provides insight into the impact of specifications on model results (full methodology described in the Supplementary Appendix). Weinstein, Corzillius, Sendi and Levison showed GART after 1 L failure was cost effective (CE) from the US, German, European and South African perspective, respectively. Phillips reported GART is not CE from a Zimbabwean perspective.

Of all the model results reported, the only instance where GART exceeded a willingness to pay (WTP) threshold of ≥$50,000 was Sendi’s societal perspective, which discounted at 2% for cost and outcomes and gave an incremental cost effectiveness ratio (ICER) of $54,960 USD 2018 per Life Year Gained (LYG) ($54,960), which was 9.92% over the WTP threshold. All other studies reported GART in routine practice after 1 L failure as a potentially cost-effective intervention, ranging from $21,178 to $49,721 (2018 USD). Although Phillips concluded GART was not CE, their results were reported as cost per Disability Adjusted Life Years (DALYs) averted preventing direct comparisons to the other models (cost/QALY).

The global comparison results are illustrated in Figure 1.

Figure 1. Standardized incremental cost effectiveness ratios (2018 USD). Author, study (if relevant), perspective, cost discount rate, outcome discount rate, output measure. HP, healthcare payer (public); MS, modified societal; S, societal. Phillips results not directly comparable because results are reported as total (2015–2025) incremental cost of $191.1m per 139,589 Disability Adjusted Life Years (DALYs) averted over ten years compared to a no monitoring strategy (discounted at 3.5%) (Phillips 2014 referencing Figure 2)²⁵.

Sensitivity analysis

Comparing the sensitivity analysis of the models provides insight into the influence of model specifications on the direction and magnitude of potential uncertainty on model results. Weinstein described many one-way sensitivity analysis (SA) and used a sample size of 1 million to minimize sampling error. It was not clear if multivariate SA was performed, and Probabilistic Sensitivity Analysis (PSA) was not reported. Corzillius performed one-way SA for plausible ranges of all model parameters, multivariate scenario analysis and PSA. Sendi reported no univariate or multivariate SA, however, PSA was reported. Levison performed univariate and multivariate SA. Phillips conducted several one-way SA and undertook enough runs to minimize stochastic effects. No PSA was reported, although this may not have been possible for the individual patient simulation model.

Corzillius varied all model parameters simultaneously in PSA yielding a median ICER of 22,200 €/life year (LY) (95% CI: 16,900–34,600 €/LY). In 98.96% of simulated cases, ICER was below 50,000 €/LY. Sendi showed that, from a healthcare perspective, at a WTP threshold of ≥$35,000 GART had a higher probability of being CE than no GART.

Levison’s delay in switching to ART after GART overturned the CE of GART. From baseline 3 months delay, GART remained CE up to <5 months and greater than this threshold no GART was the preferred strategy.

GART remained CE under variation of GART test cost (Weinstein, Corzillius, Levison, Phillips); GART “efficacy” (Weinstein, Corzillius); ART “efficacy” (Levison); health related QoL (Weinstein); ART 2 L efficacy (Weinstein, Levison, Phillips); ratio of costs to charges (Weinstein), discount rate (Weinstein, Levison); prevalence of wild type virus (Levison), cost of 2 L care (Levison), time horizon (Phillips); CD4 count at 1 L (Levison, Phillips); monthly probability of late ART failure (Levison) and ART adherence (Phillips).

In summary, Weinstein, Corzillius, Sendi and Levison agreed that addition of GART to HIV care at 1 L failure is CE, and standardized results confirm it is highly likely this currently holds true. Both models with PSAs (Sendi and Corzillius) showed high probability of CE. Note that a key driver of the Corzillius model results was the costs of care in AIDS versus non-AIDS patients. For Levison the only factor overturning the results was a delay in receiving GART results before switching treatment. In contrast, Phillips concludes that GART is not cost-effective.

Discussion

This comparative analysis identified the following critical considerations which can be applied to improve the structure, specifications and validity of future cost effectiveness models.

The first critical consideration is the choice of comparator. Four of the models compared GART to a no GART scenario where 2 L treatment was available (Weinstein, Corzillius, Sendi, Levison), which appropriately represents current standard of care³⁹. This specification resulted in no potential for benefit from differential switching of patients to 2 L under a GART versus a no GART strategy, and results in the comparator strategy (no GART) being comparatively low-cost in all scenarios.

A second critical consideration is the time horizon used for the model. Four of the models used a lifetime horizon (Weinstein, Corzillius, Sendi, Levison). Phillips reported the results of a ten-year time horizon only, distributing the economic impact over a shorter duration.

A third critical consideration is the model scope. Phillips used the broadest scope which included adherence and HIV transmission and considers the widest range of costs/outcomes. Corzillius also used a relatively broad scope by including third- and fourth-line treatment failure (although CE results were still favorable for GART, in contrast to Phillips). Weinstein, Sendi and Levison used a comparatively narrower scope. In future models, the scope should sufficiently capture the differences between costs and outcomes of patients from time of 1 L failure to the time when all potential outcomes are realized after GART (death).

It is worth noting that Phillip's combination of these three considerations (absence of 2 L comparator, comparatively short time horizon and relatively broad model scope) likely contributed to their unfavorable CE results for a GART strategy.

The fourth critical consideration is the modeling of CD4 count, VL and disease progression. A wide range of starting values and approaches to modeling changes have been used across the models. The critical issue appears to be capturing differences in high cost/low QoL disease states versus low cost/higher QoL disease states, as these are likely to drive the CE results. Indeed, the SA by Corzillius confirmed a key driver in their model results was the difference in costs between patients with AIDS versus without AIDS. In future modeling, it is likely that a simple approach dichotomizing patients according to higher cost/lower QoL (AIDS) versus lower cost/higher QoL (non-AIDS) status will adequately capture potential costs and health outcomes.

The fifth critical consideration is the health outcome measure used for the model. Two models (Weinstein, Sendi) included QoL, and some variation was observed between the values used in the two models. Weinstein’s utility of “sicker” patients resulted in notably higher values than Sendi’s (Table 1). Weinstein reported varying QoL weights in the SA, with no notable impact on model results. It was presumed that utility was included in Sendi’s PSA, although this was not explicitly stated. The lack of influence on model results may be due to small changes in utility between categories of CD4 count strata (0.94–0.81 = 0.13 for Weinstein; 0.801–0.739 = 0.062 for Sendi). Future models will benefit from more recent data on utility of HIV patients with and without AIDS, using preference-based measures and explicitly exploring in a variety of SA, including PSA.

The final critical consideration is the compound effect of four sequential assumptions which can be likened to model “filters” (which direct patients in a cohort model through the different pathways), namely prevalence of DR (primary and secondary), ART efficacy, GART efficacy and which patients are switched to 2 L. Each “filter” potentially impacts the cost effectiveness results.

The first filter is prevalence of DR, including primary (transmitted, pre-treatment) resistance or secondary (acquired) DR. Note that prevalence of wild type HIV (which implies no DR) varies between countries and changes over time³⁹. Only two of the models considered primary prevalence and this is likely to impact on model results. Corzillius acknowledged that the CE of GART inherently depends on the prevalence of DR.

The second filter is ART efficacy, defined as the proportion of patients who achieve VLS on 1 L. The models use varying assumptions including 13% (Phillips), 18% (Corzillius), 25% (Levison) and 30–40% (Weinstein). This is likely to drive results because the potential to benefit from GART is applied to patients with 1 L VLF.

The third filter is the test performance characteristics (diagnostic test accuracy) of GART which was not explicitly described in any of the models. Implicit in this is that GART is a ‘perfect’ and binary test. Although such an assumption is reasonable for the sake of simplicity, it arguably overlooks factors such as invalid test results, low levels and archived resistance, proper clinical management of resistance results, and the ability of some HIV drug resistance to be overcome with high adherence⁴⁰.

The fourth filter is the clinical decision of when to switch patients to 2 L in the GART and no GART strategies, which is the probability of switching to 2 L after GART/no GART. Importantly, the models appear to be based on a wide range of assumptions relating to this, as clinical outcomes of switching patients after GART have not been well described in the published literature. Levison assumed that a proportion of patients without DR in the GART arm remain on 1 L and achieve VLS (although CE results remain favorable). Phillips appears to apply a constant probability of switching to 2 L (0.3/3 months) irrespective of whether GART has been used. This means that, of the 30% of patients switching to 2 L, some were assumed to have DR and some were non-adherent or had wild type virus. The VL outcomes for these two subgroups are likely to be different as the presence of drug resistance has been strongly associated with future rates of re-suppression⁴¹. By comparison, Sendi assumed 67% of GARTs were switched to 2 L and 97% of no GARTs switched to 2 L, therefore potential cost savings may arise from GART patients without DR who stay on 1 L. There is likely to be a better evidence base for these values in future modeling when the results of the REVAMP study become available, as they compare the efficacy of GART to VL monitoring and intensified adherence counseling alone in people failing 1 L in Uganda and South Africa⁴².

Based on the four filters above, it seems that GART has the potential to be CE if it impacts favorably on clinical outcome (VLS), QoL and/or cost of care of 1 L VLF patients over their lifetime. GART can achieve this if it directs patients with higher cost/lower quality of life (1 L VLF) to lower cost/better quality of life (VLS) states. However, it is less likely to prove CE if it either does not impact regimen choice or assumes that VLS will be achieved (post GART), in patients with DR by staying on 1 L therapy. For example, if the results of GART do not influence switch to 2 L (as in the Phillips study) or delay change of regimen (as in the SA in the Levison study), then it will have little chance of improving value of HIV care. To increase accuracy of future modeling, primary data on the impact of GART on treatment decisions, the impact of using both 1 L and 2 L among people with and without DR, and the impact of HIV disease status on progression and quality of life should be incorporated.

Conclusions

Using an example of GART testing in HIV, this research demonstrates that undertaking comparative analysis of published cost-effectiveness models can provide insight into critical issues which should be considered when developing a new cost-effectiveness model for a specific intervention.

Transparency

Declaration of funding

This study is funded by the National Institute of Allergy and Infectious Diseases with support from the President’s Emergency Plan for AIDS Relief (NIH R01 AI124718). MJS receives additional support from the National Institutes of Mental Health (K23 MH099916). VCM and RTG receive additional support from NIH/NIAID.

Declaration of financial/other interests

The authors have no relevant financial or other relationships to disclose.

JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

Conception and design: all; analysis and interpretation of the data: TR, GG, MJS; drafting of the article: TR; critical revision of the article for important intellectual content: all; obtaining of funding: MJS.

Previous presentations

None.

Acknowledgements

Thank you to Debra Kiss for review of the manuscript. Thank you to Ute Zerwes who performed the literature searches, Maggie Feng and Rebecca Fain Gilbert who sourced the literature and Michael Strauss who provided expertise in standardizing the results.