Prioritizing HIV comparative effectiveness trials based on value of information: generic versus brand-name ART in the US

Background:

Value of Information (VOI) analysis examines whether to acquire information before making a decision. We introduced VOI to the HIV audience, using the example of generic antiretroviral therapy (ART) in the US.

Methods and Findings:We used a mathematical model and probabilistic sensitivity analysis (PSA) to generate probability distributions of survival (in quality-adjusted life years, QALYs) and cost for three potential first-line ART regimens: three-pill generic, two-pill generic, and single-pill branded. These served as input for a comparison of two hypothetical two-arm trials: three-pill generic versus single-pill branded; and two-pill generic versus single-pill branded. We modeled pre-trial uncertainty by defining probability distributions around key inputs, including 24-week HIV-RNA suppression and subsequent ART failure. We assumed that, without a trial, patients received the single-pill branded strategy. Post-trial, we assumed that patients received the most cost-effective strategy. For both trials, we quantified the probability of changing to a generic-based regimen upon trial completion and the expected VOI in terms of improved health outcomes and costs. Assuming a willingness to pay (WTP) threshold of $100 000/QALY, the three-pill trial led to more treatment changes (84%) than the two-pill trial (78%). Estimated VOI was $48 000 (three-pill trial) and $35 700 (two-pill trial) per future patient initiating ART.

Conclusions:A three-pill trial of generic ART is more likely to lead to post-trial treatment changes and to provide more value than a two-pill trial if policy decisions are based on cost-effectiveness. Value of Information analysis can identify trials likely to confer the greatest impact and value for HIV care.

Keywords:

• Value of information
• HIV
• Generic drugs
• Cost-effectiveness analysis
• Decision modeling

Introduction

Value of Information (VOI) analysis is a decision-analytic method that evaluates the potential clinical and economic impact of knowledge gained from a clinical trial or other source of new data compared to current knowledge.^1,2 The value of the information obtained from a trial is equivalent to the expected improvement in outcomes owing to the ability to base future clinical decisions on the information obtained from the trial.

An important goal of a clinical trial is to improve clinical practice by reducing uncertainty around clinical decisions. Without new information from clinical trials, suboptimal care results in fewer clinical benefits and inefficient allocation of limited health care resources. All possible trials cannot be conducted, however, due to limited financial and non-financial resources. Value of Information analysis can be used before a trial is conducted to compare patients outcomes based on the standard of care, informed by the uncertain evidence as it exists now, with expected patient outcomes based on the clinical decisions that would be informed by the possible results of the trial, incorporating uncertainty about what those results might be. In doing so, it anticipates the clinical policy impact of anticipated trial results to assess whether the trial is “worth” doing. Because HIV clinical trials are costly and draw resources from limited research funds, these methods can help to prioritize whether resources would better be spent on alternative HIV trial designs or research questions.^3–7

We sought to illustrate the usefulness of VOI analysis in HIV clinical trials by applying it to an anticipated dilemma in US HIV care – whether or not to encourage the broader use of generic antiretroviral therapy (ART). With the anticipated approval of generic efavirenz (EFV),⁸ US policy makers and payers will soon face the decision of whether to recommend a switch from the current standard of care for first-line ART – most commonly a branded, single-pill, co-formulated regimen – to a multi-pill, generic-based regimen option.⁹ A large-scale, head-to-head comparative effectiveness trial would compare the viral suppression rates of these two options; the results would be influenced by adherence to multi-pill regimens and, in turn, efficacy. The selection of the multi-pill regimen would, therefore, be important in designing such a trial.

Methods

Analytic overview

We considered two different, potential trials among patients initiating ART: a “three-pill versus single-pill trial” (three-pill trial hereafter) with generic EFV, generic lamivudine (3TC), and branded tenofovir (TDF) (three-pill generic) versus branded TDF/emtricitabine (FTC)/EFV (single-pill branded), and a “two-pill versus single-pill trial” (two-pill trial hereafter) with generic EFV and branded TDF/FTC (two-pill generic) versus single-pill branded.²

Because fully generic regimens are not currently available in the US and their adoption in the absence of a trial is unknown, we assumed that without any new trial data, the default standard of care would be single-pill branded. This is a small but important change from the standard VOI approach. Generally, VOI is based on a pre-trial default strategy deemed most optimal before the trial is conducted. In this case, because generic EFV is not yet available in the US, even though it may well be cost-effective, using generic EFV as the current standard of care would not be logical. We therefore chose single-pill branded as the current standard of care. Following standard VOI methods to evaluate potential benefits to society, we further assumed that after the results of a trial are known, patients would be prescribed the most cost-effective regimen from among those evaluated in the trial. We followed standard practice and identified the most cost-effective strategy as the one with the highest incremental cost-effectiveness ratio below a pre-specified threshold value of willingness to pay (WTP) for extending a patient's quality-adjusted life expectancy by an extra year. In VOI analysis, determining the most cost-effective strategy is achieved using a measure called “net monetary benefit” (NMB) per patient, which is defined as the average number of quality-adjusted life years QALYs conferred by the strategy multiplied by the WTP, minus the lifetime cost of the strategy (i.e. NMB = QALY*WTP – Cost, denominated in USD). We measured VOI as the gain in lifetime NMB associated with treatment changes due to information obtained from the trial. That is, how likely is it that the trial results will indicate that treatment should be changed from the current standard to some new standard, and what is the value of that change? For the base case, we considered a WTP of $100 000/QALY (a frequently cited WTP threshold in the US^10–13). In addition, because there is no definite value of WTP for health care, we varied the threshold from $50 000 to $200 000/QALY. Details on the principles of cost-effectiveness analysis and VOI-related concepts are available in Appendices A.1 and B. A technical derivation of VOI as used in this analysis is available in Appendix C.

Probabilistic sensitivity analysis

Using the Cost-Effectiveness of Preventing AIDS Complications (CEPAC)-US model,¹⁴ a previously published Monte Carlo microsimulation of HIV disease and treatment, we modeled lifetime clinical outcomes, e.g. QALY, and economic costs of a cohort of HIV-infected patients initiating ART in the US. Details on the CEPAC model are available in Appendix D. For evaluating VOI – capturing the impact of reducing uncertainty in trial endpoints – we conducted probabilistic sensitivity analysis (PSA). A single PSA simulation selects values from the distributions representing the pre-trial uncertainty around each key efficacy parameter (Appendix A.2). We then run the PSA simulations multiple times – with many draws from these parameter distributions – to produce the clinical outcomes (QALYs), lifetime costs (USD), and NMB (USD) for our strategies of interest.² A total of 360 000 simulations were considered per trial (180 000 simulations for each trial arm). In addition, we used the PSA results to estimate the probabilities that different treatment decisions would be deemed optimal once information gained from the new trials was taken into account.

We refer to the efficacy parameters for which information would be gained from a trial as target parameters. Target parameters included the probabilities of HIV RNA suppression at 48 and 96 weeks. Each suppression result was transformed to its corresponding: (1) probability of initial suppression (24-week HIV RNA suppression), and (2) monthly probability of failure after 24 weeks by assuming a constant rate of decline after 24 weeks (Appendix E).

We also considered parameters that would not be informed by the trial, referred to as complementary parameters. Complementary parameters included: (1) mean initial CD4 count of HIV-infected patients initiating ART, (2) frequency of loss to follow up and return to care, and (3) future cost of generic drugs. All complementary parameters were assumed to be independent of trial outcome but ultimately influential in implementation of trial results.

Outcomes of interest

Table 1 summarizes all model outcomes and their respective concepts, definitions, and references. A more detailed description of the calculations underlying the model outcomes and a table of 10 sample simulations are included in Appendix B and Table S1.

Table 1 Model outcomes of interest.

Outcome of interest	Definition/methodology
		Result	Supplementary materials
Pre-trial outcomes
Pre-trial expected QALY and expected cost for all strategy options	Average QALY/cost across the entire set of simulation runs performed per strategy	Table 3	Appendix A
Post-trial outcomes
Distribution of post-trial optimal strategies	The proportion of trial simulations for which a strategy has the highest NMB	Fig. 1	Appendix A and B
Post-trial expected QALY and expected cost	Average QALY/cost conditional on the trial result (assuming the post-trial optimal strategy is selected for each trial result)	Table 4	Appendix A and B
VOI outcomes
Probability of changing treatment decision with new trial information	The proportion of PSA runs of which the pre-trial default strategy (i.e. single-pill branded) does not have the highest NMB post-trial (after ascertaining perfect information of the target parameters from the trial)	Fig. 1, Table 4	Appendix B
Expected change in QALYs and cost associated with these treatment decision changes	Difference between post-trial optimal and pre-trial default (i.e. single-pill branded) QALYs and cost	Table 4	Appendix B
Value of information per person associated with these treatment decision changes	Expected change in QALYs (due to treatment changes) multiplied by WTP minus expected change in cost (due to treatment changes)	Fig. 2, Table 4	Appendix B and C
Optimal trial design	The trial design which highest VOI per person for a given WTP	Fig. 2	Appendix C

NMB: net monetary benefit; PSA: probabilistic sensitivity analysis; QALY: quality-adjusted life year; VOI: value of information; WTP: willingness to pay.

Pre-trial outcomes

For each trial design, we used the CEPAC model and pre-trial uncertainty around each target and complementary parameter to estimate the pre-trial expected outcomes (QALY and lifetime cost) of the two treatment arms in the absence of new trial information. Both QALYs and costs were discounted to present value at an annual rate of 3%.

Post-trial outcomes

We ran a large number of simulations, using parameter values drawn from their pre-trial distributions, to obtain the range of outcomes that would be observed in each arm of the two trial designs. For every simulated outcome within a given trial design, we determined the costs and QALYs of each trial arm. For each simulation, we selected the optimal post-trial strategy (and its associated cost and QALYs) for a given trial outcome and WTP. Next, we tallied the fraction of the simulations in which each strategy was deemed optimal, using a particular WTP threshold. We used a base case WTP threshold of $100 000/QALY and repeated this analysis for the entire range of interest for the WTP threshold ($50 000–$200 000/QALY). We then averaged the costs and QALYs associated with the trial outcome and WTP-specific optimal strategies.

VOI outcomes

The main outcome of interest answers the question: did the trial results provide an evidence basis for changing clinical care from the current clinical norm to some new standard? That is, did the trial results show that the multi-pill strategy (two- or three-pill generic) would be more cost-effective than the standard of care (single-pill branded)? For each trial simulation, we estimated the expected change in per person QALYs (post-trial expected QALY minus expected QALY of single-pill branded) and the expected change in per person lifetime cost (post-trial expected cost minus expected cost of single-pill branded) associated with the change in post-trial decisions. We then calculated the per person VOI for each proposed trial, which was defined as the per person gain in NMB associated with the expected post-trial decision changes:

Per patient VOI = WTP*(Post–trial Expected

QALY–Expected QALY of current

SOC[Single–pill Branded])

– (Post–trial Expected Cost

– Expected Cost of current

SOC[Single–pill Branded])

For a specified WTP threshold, the optimal trial (two-pill vs three-pill) was the one that had the higher VOI per patient.

Model inputs

ART efficacy (target parameters)

To determine pre-trial evidence-based input parameters for ART efficacy, we first conducted a systematic search of possible relevant trials. We included trials that evaluated the efficacy of TDF, FTC, and EFV as single-pill, two-pill, or three-pill formulations. We excluded studies that included only portions of the combinations or additional components (see Appendix E.1). First, for each type of regimen (three-pill generic, two-pill generic, and single-pill branded), we derived two efficacy input parameters: the first relates to the initial suppression rate (percent suppressed at 24 weeks), and the second relates to the monthly probability of failure subsequent to initial suppression after 24 weeks. Next, we derived distributions around these point estimates that reflect both between-study and within-study uncertainty. We used reported ranges of efficacy across studies as upper and lower bounds for a distribution that reflects between-study uncertainty, and we used reported confidence intervals to reflect within-study uncertainty. These distributions are as important as the individual point estimates because: (i) uncertainty around a point estimate increases when only older studies are available; and (ii) efficacy point estimates and the uncertainty around them are key determinants of the VOI estimates.¹⁵ As defined in the analysis, highly effective treatment would involve high rates of suppression, low rates of later failure and narrow distributions around each parameter.

Some of the efficacy estimates, especially related to multi-pill regimens (Table 2, Target Parameters) were based on older studies with twice-daily ART dosing and pill burdens that included placebos. In these cases, we relied on the literature to create wide distributions of possible efficacy.^16–18 Further details on the calculation of each regimen's efficacy input (both percent suppressed and subsequent failure) as well as the actual values drawn for simulation are in Appendices E.2 and E.3.

Table 2 Model inputs

Target parameters (inputs with information to be gained from the trial)
	Distribution	Mean	SD	Reference
Patients with HIV RNA suppression after 24 weeks on first line ART
Three-pill generic (generic EFV+generic 3TC+TDF)	Hyper-beta^¥	Uniform (78%, 85%)	2.40%	^Citation16,Citation19
Two-Pill generic (generic EFV+ TDF/FTC)	Hyper-beta^¥	Uniform (84%, 85%)	2.18%	^Citation17,Citation19
Single-pill branded (TDF/FTC/EFV)	Beta	85%	1.90%	^Citation19
Monthly probability of failure after 24 weeks on first line ART^Δ
Three-pill generic (generic EFV+generic 3TC+TDF)	Hyper-adjusted log-normal (bounded above by 1)^¥	Uniform (0.21%, 0.45%)	0.44%	^Citation16,Citation19
Two-pill generic (Generic EFV+TDF/FTC)	Hyper-adjusted log-normal (bounded above by 1)^¥	Uniform (0.21%, 0.43%)	0.43%	^Citation17,Citation19
Single-pill branded (TDF/FTC/EFV)	Adjusted log-normal (bounded above by 1)	0.21%	0.27%	^Citation19
Complementary parameters (other inputs to be varied in PSA)
Cohort's initial mean CD4	Lognormal	400	80	^Citation20,Citation21
Annual cost of generic EFV+3TC (part of three-pill generic first line ART)	Lognormal	$6330	$3160	^Citation22
Annual cost of generic EFV (part of two-pill generic first line ART)	Lognormal	$2360	$1180	^Citation22
Monthly probability of loss to follow-up	Beta	0.62%	0.05%	^a
Monthly probability of returning to care	Beta	1.38%	0.11%	^a
Other baseline inputs
		Mean	SD¶	Reference
Cohort characteristics
Age, years		43	12	^Citation20
Male		84%	–	^Citation20
Distribution of Initial HVL
HVL 100 000+copies/mL		25%	–	^Citation23
HVL 30 001–100 000 copies/mL		42%	–
HVL 10 001–30 000 copies/mL		21%	–
HVL 3001–10 000 copies/mL		6%	–
HVL 501–3000 copies/mL		6%	–
HVL 0–500 copies/mL		0%	–
Other non-PSA efficacy parameters of ART
Patients with HIV RNA suppression after 24 weeks on second line ART (ATV/RTV+TDF/FTC)		73%	–	^Citation24,Citation25
Monthly probability of failure after 24 weeks on second line ART (ATV/RTV+TDF/FTC)^Δ		1.71%	–
CD4 increase at 48 week, × 10^9 cells/L‡
First line 3-pill generic (generic EFV+generic 3TC+TDF)		0.206	0.045	^Citation19
First line two-pill generic (Generic EFV+TDF/FTC)		0.206	0.045
First line single-pill branded (TDF/FTC/EFV)		0.206	0.045
Second line (ATV/RTV+TDF/FTC)		0.110	0.028	^Citation24,Citation25
Annual costs, 2013 U.S. $
First line, branded TDF (part of 3-pill generic ART)		$8950	–	^Citation22
First Line, branded FTC/TDF (part of 2-pill generic ART)		$13 560	–
First line, single-pill branded EFV/TDF/FTC		$20 830	–
Second line, two-pill branded ART, ATV/RTV+TDF/FTC		$28 490	–

3TC: lamivudine; ART: antiretroviral therapy; ATV: atazanavir; CD4: cluster of differentiation 4; EFV: efavirenz; FTC: emtricitabine; HVL: human immunodeficiency virus viral load; mL: milliliters; PSA: probabilistic sensitivity analysis; QALY: quality-adjusted life year; RTV: ritonavir; SD: standard deviation; TDF: tenofovir

a Source: Personal communication with Dr. John Fleishman and the HIV Research Network on 20 June 2014

¥ Hyper distribution refers to a hyper prior of a uniform distribution for the mean of the outcome. Thus, a hyper-beta is equivalent to a distribution of beta distributions; where the mean is uniformly distributed and the standard deviation of the beta distributions is kept constant. Similarly, hyper-lognormal distribution is equivalent to a distribution of log-normal distributions; the mean of the log-normal distributions is uniformly distributed and the standard deviation of the log-normal distributions is kept constant. Details of the derivation of the actual inputs can be found in Appendix D

¶ For variables not varied in the PSA and for which a SD was included (e.g. age and CD4 increase), its SD refer to the uncertainty within the population, i.e. the variation across individuals, which is different from the uncertainty around the input itself

Δ The monthly probability of failure after 24 weeks was estimated by transforming the probabilities of HIV RNA suppression at 48 and 96-weeks, assuming HIV RNA suppression would achieve its maximum at 24 weeks and that its rate of decline would be constant post 24 weeks

‡ We assumed independence between CD4 increase (among patients who were virologically suppressed) and the probability of HIV RNA suppression.²⁶

Characteristics of trial-applicable population

We assumed the entering cohort in the model was representative of HIV-infected patients initiating ART in the US at the point of time the study would be completed (Dr. K Althoff, personal communication, May 2013).²⁷ We included mean CD4 count at ART initiation (expected mean at trial completion of 400 cells/mm³) as a complementary parameter (Table 2, Complementary Parameters; details in Appendix F.) Mean age of the cohort was 43 years (SD 12 years) and 84% were male (Table 2, Other Major Baseline Inputs).

ART costs

Given the uncertainty around the cost of generic drugs, we included the cost of the generic components of the three- and two-pill generic regimens as complementary parameters and kept the cost of the branded components constant. We considered a mean 75% price reduction from average wholesale price (AWP) for the generic components of the 3- and two-pill generic regimens, resulting in an annual cost of $6330 for three-pill generic (comprising generic EFV and generic 3TC components) and $2360 for two-pill generic (generic EFV only, Table 2, Complementary Parameters).²² The annual costs for the branded components of three-pill generic, two-pill generic, and single-pill branded regimens were $8 950 (TDF), $13 560 (TDF/FTC), and $20 830 (TDF/FTC/EFV), which are 77% of the published AWP for standard dosing (Table 2, Other Major Baseline Inputs).²² All costs were in 2013 US dollars.

Methodological simplification

We made a simplification by assuming that the trials would reveal the “true” values of the key endpoints (perfect information). That is, we assumed trials were sufficiently large such that sampling error in the trial results would not affect the results of the VOI analysis. To do otherwise would require further evaluation of the uncertainty around outcomes with alternative trial sample sizes (N = 100 vs N = 1000 vs N = 10 000), which would add additional complexity to this introduction to VOI methods.²⁸ We therefore estimate the computationally and methodologically simpler expected value of perfect information (EVPI), a commonly reported measure that provides an upper bound on the value of new data acquisition. If a VOI assessment suggests that there is insufficient value in conducting a trial that would yield perfect information, it follows that there would be insufficient value in conducting a trial that would yield less-than-perfect information.²⁸

Results

Pre-trial results

Based on current information, without any new trials, the pre-trial per-person discounted QALYs were 13.43 QALYs (undiscounted, 20.79 QALYs) for three-pill generic; 13.51 QALYs (undiscounted, 20.95 QALYs) for two-pill generic; and 13.58 QALYs (undiscounted, 21.13 QALYs) for single-pill branded. Pre-trial per-person discounted lifetime costs were $318 600 for three-pill generic, $340 800 for two-pill generic, and $376 100 for single-pill branded (Table 3).

Table 3 Pre-trial outcomes: Pre-trial results from probabilistic sensitivity analysis (PSA) of the treatment comparisons included in alternative trials

	Expected QALYs
	Discounted	Undiscounted	Expected cost
Three-pill generic vs single-pill branded
Three-pill generic (generic EFV+Generic 3TC+TDF)	13.43	20.79	$318600
Single-pill branded (TDF/FTC/EFV)	13.58	21.13	$376100
Two-pill generic vs single-pill branded
Two-pill generic (generic EFV+TDF/FTC)	13.51	20.95	$340800
Single-pill branded (TDF/FTC/EFV)	13.58	21.13	$376100

3TC: lamivudine; EFV: efavirenz; FTC: emtricitabine; QALY: quality-adjusted life year; TDF: tenofovir.

Post-trial results

With a base case WTP threshold of $100 000/QALY, after ascertaining trial-based efficacy in the three-pill generic trial, there was an 84% probability of selecting three-pill generic as the post-trial optimal strategy. For a two-pill trial, there was a 78% chance of selecting two-pill generic as the post-trial optimal strategy (Table 4). The three-pill generic trial, therefore, would be more likely to change the standard of care from single-pill branded to a generic alternative. While the multi-pill generic regimen was selected more frequently for both trials for the entire WTP range of interest ($50 000– $200 000/QALY), the chance of selecting single-pill branded increased with WTP (Fig. 1A and 1B).

Table 4 Value of Information of alternative trials: comparing “three-pill generic versus single-pill branded regimens” to “two-pill generic versus single-pill branded regimens”

	Three-pill generic vs single-pill branded	Two-Pill generic vs single-pill branded
WTP threshold ($/QALY gained)	$50 000	$100 000	$150 000	$200 000	$50 000	$100 000	$150 000	$200 000
Post-trial outcomes
Chance of being selected as most CE strategy post-trial
3-Pill generic (generic EFV+generic 3TC+TDF)	90%	84%	77%	71%	–	–	–	–
2-Pill generic (generic EFV+TDF/FTC)	–	–	–	–	83%	78%	73%	69%*
Single-pill branded* (TDF/FTC/EFV)	10%	16%	23%	29%	17%	22%	27%	31%
Post-trial expected QALY	13.48	13.51	13.53	13.55	13.58	13.59	13.60	13.61
Post-trial expected lifetime cost	$319 100	$321 000	$323 500	$326 300	$340 400	$341 300	$342 500	$343 800
VOI outcomes
Chance of changing treatment decisions post-trial	90%	84%	77%	71%	83%	78%	73%	69%
Expected change in QALY associated with treatment decision changes	0.097	0.071	0.051	0.035	0.003	0.009	0.018	0.026
Expected change in lifetime cost associated with treatment decision changes	$57 000	$55 100	$52 600	$49 800	$35 700	$34 900	$33 600	$32 280
VOI from trial ($ per person)	$52 100	$48 000	$45 000	$42 900	$35 600	$35,700	$36 400	$37 500

3TC: lamivudine; CE: cost-effective; EFV: efavirenz; FTC: emtricitabine; QALY: quality-adjusted life year; TDF: tenofovir; VOI: value of information; WTP: willingness to pay.

Numbers in brackets are negative

* Equivalent to the probability of re-selecting the pre-trial default strategy (i.e. single-pill branded) after ascertaining trial-based efficacy.

Figure 1 Distribution of post-trial optimal strategies of alternative trials. (A) depicts the distribution of post-trial optimal strategies of the head-to-head trial of three-pill generic ART versus single-pill branded ART. For a given willingness to pay (WTP) threshold (horizontal axis), the proportion of PSA runs (vertical axis) of which three-pill generic ART strategy and single-pill branded ART strategy has the highest net monetary benefit (NMB) after ascertaining perfect information on ART efficacy for both treatment arms in the trial are represented by the red and green curves, respectively. Net monetary benefit was defined as QALY times WTP, minus cost. The dotted vertical line represents the WTP threshold at baseline ($100 000/QALY). (B) depicts the distribution of post-trial optimal strategies of the head-to-head trial of two-pill generic ART versus single-pill branded ART. For a given WTP threshold (horizontal axis), the proportion of PSA runs (vertical axis) of which two-pill generic ART strategy and single-pill branded ART strategy has the highest NMB after ascertaining perfect information on ART efficacy for both treatment arms in the trial are represented by the blue and green curves, respectively. NMB was defined as QALY times WTP, minus cost. The dotted vertical line represents the WTP threshold at baseline ($100 000/QALY).

When we selected the post-trial optimal strategy (Fig. 1A and B) for each simulated trial result for both trials, at a WTP threshold of $100 000/QALY, the post-trial per-person discounted QALYs were 13.51 and 13.59, and the post-trial per-person discounted lifetime costs were $321 000 and $341 300 for the three-pill and two-pill trials, respectively (Table 4). At higher WTP, post-trial per-person discounted QALYs and lifetime costs both increased.

VOI results

With a WTP of $100 000/QALY, the three-pill trial led to a higher probability of treatment decision changes (84%) than the two-pill trial (78%) (Table 4). These percentages are the probabilities that the trial will demonstrate that the current standard of care is not cost-effective compared to either the three-pill or the two-pill regimen. These decision changes – based on the information from the trials – translated to an expected decrease of 0.07 QALYs and an expected increase of 0.009 QALYs per person compared to the status quo (i.e. continuing to use single-pill branded) for the three-pill and two-pill trials, respectively. The decision changes led to expected lifetime cost decreases of $55 100 (three-pill trial) and $34 900 (two-pill trial) versus the status quo (Table 4). In terms of NMB, the VOI was $48 000 and $35 700 per future person starting ART in the US for the three- and two-pill trials, respectively (Table 4). For the range of WTP thresholds, the VOI per person for the three-pill trial decreased with increasing WTP (Fig. 2, red solid line); whereas for the two-pill trial, VOI per person increased with WTP (Fig. 2, blue solid line).

Figure 2 Direct comparison of the value of information (VOI) of alternative trials in relation to generic drug price discount and willingness to pay (WTP) per QALY. The value of ascertaining perfect information per person of ART efficacy from the three-pill trial (for both three-pill generic and single-pill branded treatment arms) and the two-pill trial (for both two-pill generic and single-pill branded treatment arms) are represented by the red and blue solid curves, respectively. The VOI per person was estimated as the difference in net monetary benefit (NMB) between single-pill branded (the pre-trial initial strategy of choice) and the post-trial optimal first line ART strategies. Net monetary benefit was defined as QALYs times WTP, minus cost. Further, the graph shows the changes in the VOI per person of a three-pill trial (red curves) and a two-pill trial (blue curves) while the generic drug discount from the AWP is varied from 60% (evenly dotted) to 90% (unevenly dotted), in comparison to the base case where generic drug discount was 75% (solid).

Based on the point estimates of per person VOI, for the entire range of WTP thresholds of interest ($50 000–$200 000/QALY), conducting a three-pill trial provided higher VOI than a two-pill trial (Fig. 2, solid lines).

Due to uncertainty about future generic drug prices, which may be influenced by trial results, we examined the impact of these prices on the VOI estimates by varying the price reduction from AWP for all generic components of the generic-based strategies (Fig. 2). We found that the VOI increased for both trials at greater price reductions for generic drugs (Fig. 2, unevenly dashed lines). This is because less expensive multi-pill generic regimens were even more cost-effective, which in turn led to more treatment decision changes. The per person VOI of a three-pill trial was consistently higher than that of a two-pill trial for the entire price reduction range considered (60–90% price reduction from AWP for all generic components of the generic-based strategies, Fig. 2).

Discussion

In HIV disease, as in other areas of medicine, resources to conduct clinical trials are limited. We demonstrated the utility of VOI methods to assess the value that trials may be anticipated to provide by comparing the impact and economic value of two hypothetical comparative effectiveness trials that examine generic versus brand-name first-line ART in the US. First, we illustrated how VOI methods assess the potential clinical and economic impact of new knowledge gained from a trial compared with the current state of knowledge about treatment. Second, by comparing VOI estimates for alternative trials, we showed how VOI can be used to prioritize trials. In our illustrative example, we used these methods to quantify and compare the potential value of a hypothetical three-pill generic trial versus a two-pill generic trial. The three-pill trial was more likely to lead to changes in treatment decisions, and offered greater value than the two-pill trial over the full range of evaluated WTP thresholds.

The VOI approach is well-described in the recent decision science literature,^28,29 and has been applied to cardiac and cancer screening and treatment interventions.³⁰ One recent analysis did a post hoc VOI analysis of a trial costing $260 million that resulted in reduction of combined hormone replacement therapy in post-menopausal women. The post-trial VOI analysis projected 4.3 million fewer hormone replacement users and fewer breast cancer and cardiovascular disease cases, resulting in $35.2 billion expenditure savings.³¹ Conducting similar analyses is increasingly being called for as a mechanism for guiding research priorities and for deciding when more evidence is needed to inform clinical care decisions. For example, the 2012 recommendations commissioned by the US Patient-Centered Outcomes Research Institute (PCORI) noted “we believe VOI has significant potential as a tool for research prioritization by PCORI.”³² In HIV disease, modeling analyses have already been used to pre-evaluate clinical trial data³³ and to assist in clinical trial design by forecasting critical clinical outcome measures.³⁴ We introduce the VOI concepts here to promote their use as a state-of-the-art method for HIV clinical trials prioritization and to guide HIV clinical trial design.

Importantly, VOI analysis is not limited to publicly funded studies and does not specifically address who should sponsor a trial. Private payers and industry could similarly gain knowledge about trial design and prioritization by conducting VOI analyses. Those analyses would not necessarily reflect societal benefits because the VOI would be determined based on the value to the payer or a pharmaceutical company.

The case of generic EFV⁸ offers an illustrative study of importance to decision makers. Generic HIV drugs will save money but their impact on health outcomes remains uncertain because they will likely require switching to multi-pill once-daily regimens. Given the uncertainty regarding the relationship between ART efficacy and pill burden for once-daily regimens,³⁵ we considered two hypothetical comparative effectiveness trials (three-pill or two-pill) of “alternative” first-line generic-based ART compared with single-pill branded ART in the US. Our study results continue to have implications in the current treatment era, where EFV-based regimens are no longer recommended as first-line therapy. Specifically, there is a branded single-pill treatment containing dolutegravir (DTG), abacavir (ABC), and 3TC; both ABC and 3TC are available as generics, with the potential to significantly lower costs if given as separate components.

While single-pill regimens are the standard of care in the US, they may be less widely used in other countries. The VOI framework we have illustrated can readily be adapted to consider alternative standards of care, new treatment strategies, costs, and WTP benchmarks in other settings.

Our example deviates slightly from the standard approach to VOI. Generally, VOI analysis assumes that the pre-trial default decision is optimal according to pre-trial evidence and the specified WTP. We believe our suggested approach for estimating VOI captures the underlying value of conducting a trial more realistically, because it reflects the current standard of care as the baseline policy. As a consequence, our VOI estimate includes two components: (i) the value of converting current decisions from the single-pill branded strategy to the generic-based strategy, which our model indicates is cost-effective based on current (not RCT-based) evidence, and (ii) the standard VOI estimate of the incremental benefit assuming pre-trial treatment decisions are already the most cost-effective based on pre-trial evidence. Including the first of these components in our VOI result, which reflects that the current standard of care is probably not cost-effective, explains why the per-patient VOI is higher than might otherwise be expected.

There are several limitations to this analysis. First, we assumed that post-trial treatment decisions would be based on cost-effectiveness – patients would receive the most cost-effective treatment after new information is gained from the trial.^36–38 We acknowledge that HIV prescribing decisions in the US have not generally been affected by cost issues, and that it is unlikely that cost-effectiveness considerations and the results of a single study alone would trigger an immediate change in treatment policy and prescribing behavior.^{12,13,39–42} The VOI methodology is flexible enough to consider decision-making based on criteria other than cost-effectiveness, although these approaches require more complex analyses. The estimates presented here, therefore, serve as upper bounds on the VOI resulting from the clinical trials considered.

Second, we assumed treatment changes would be implemented soon after trial results become available. The field of HIV has responded rapidly in revising treatment guidelines based on new evidence – important examples include revisions associated with the ACTG 5202, HPTN 052, and SINGLE trials.^9,43–53 While the dissemination of guideline changes might take longer than the guideline changes themselves, the ordering of the VOI estimates would not be affected if the rate of uptake of results is similar among the trials of interest.

Third, adherence levels observed in an RCT may be higher than observed in other settings (e.g. in observational cohorts and clinical practice), which in turn may create a bias in our results in favor of multi-pill regimens. We are using this example to illustrate the value of a trial to inform guidelines and practice for first-line ART: all currently recommended first-line regimens in the DHHS guidelines carry “AI” level evidence – that is, they are informed by one or more randomized clinical trials. Hence, we assume that a guideline change of this stature would require the same level of evidence, even if it includes the recognized bias of improved adherence in a trial setting.

Lastly, our analysis does not address the scope, impact, or magnitude of uncertainty associated with the structure of the simulation model. While the CEPAC model has been widely published and externally validated,^15,54 the results of any model-based analysis depend on the underlying structure of that model.

In the hypothetical case of conducting a comparative effectiveness trial of generic ART in the US, we used VOI to demonstrate that greater value would be obtained by comparing single-pill branded ART to a three-pill generic option than to a two-pill generic option. VOI analysis is a state-of-the-art analytic method that can be used to prioritize clinical trials most likely to have the greatest impact on HIV care and value for society.

Acknowledgements

The authors thank Kelly Gebo, MD, MPH, and the HIV Research Network for supplying helpful data; and Margo Jacobsen and Marion Robine for providing administrative support to the preparation of this manuscript.

Disclaimer statements

Contributors All contributors are co-authors of the paper.

Funding Research reported in this publication was supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health under Award Numbers R01 AI093269 and R37 AI42006. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflicts of interest Dr. Hughes reports personal fees from Boehringer Ingelheim, Pfizer, and Tibotec. Dr Pei reports personal fees from United BioSource Corporation. Dr. Sax reports personal fees from AbbVie, BMS, Gilead, GSK, Janssen, Merck, ViiV and grants from BMS, Gilead, and GSK. Dr. Weinstein reports consulting for OptumInsight.

Ethics approval The research in this paper was approved by the Partners Human Subjects Protocols 2011P001481 and 2000P001927.