A review and validation of overall survival extrapolation in health technology assessments of cancer immunotherapy by the National Institute for Health and Care Excellence: how did the initial best estimate compare to trial data subsequently made available?

Abstract

Background: Validation of overall survival (OS) extrapolations of immune-checkpoint inhibitors (ICIs) during the National Institute for Health and Care Excellence (NICE) Single Technology Assessment (STA) process is limited due to data still maturing at the time of submission. Inaccurate extrapolation may lead to inappropriate decision-making. The availability of more mature trial data facilitates a retrospective analysis of the plausibility and validity of initial extrapolations. This study compares these extrapolations to subsequently available longer-term data.

Methods: A systematic search of completed NICE appraisals of ICIs from March 2000 to December 2017 was performed. A targeted search was also undertaken to procure published OS data from the pivotal clinical trials for each identified STA made available post-submission to NICE. Initial Kaplan-Meier curves and associated extrapolations from NICE documentation were extracted to compare the accuracy of OS projections versus the most mature data.

Results: The review identified 11 STAs, of which 10 provided OS data upon submission to NICE. The extrapolations undertaken considered parametric or piecewise survival models. Additional data cut-offs provided a mean of 18 months of OS beyond the end of the original data. Initial extrapolations typically under-estimated OS from the most mature data cut-off by 0.4–2.7%, depending on the choice of assessment method and use of the manufacturer- or ERG-preferred extrapolation.

Conclusion: Long-term extrapolation of OS is required for NICE STAs based on initial immature OS data. The results of this study demonstrate that the initial OS extrapolations employed by manufacturers and ERGs generally predicted OS reasonably well when compared to more mature data (when available), although on average they appeared to underestimate OS. This review and validation shows that, while the choice of OS extrapolation is uncertain, the methods adopted are generally aligned with later-published follow-up data and appear appropriate for informing HTA decisions.

Keywords:

• Overall survival extrapolation
• health technology assessment
• validation
• immune-checkpoint inhibitors

JEL classification codes:

• I10
• I15

Introduction

Extrapolation of overall survival (OS) data for patients treated with immune-checkpoint inhibitors (ICIs, a type of “cancer immunotherapy”) is associated with several methodological challenges, owing to their expected clinical outcomes (i.e. improved OS)^1–6. When submitting to Health Technology Assessment (HTA) agencies, such as the National Institute for Health and Care Excellence (NICE), OS may be considered immature, as emerging evidence in the field of ICIs demonstrates long-term survival outcomes, previously unattainable with conventional therapies^5,7,8.

As part of the NICE Single Technology Appraisal (STA) process, the manufacturer must submit a cost-effectiveness analysis (CEA), which typically incorporates an extrapolation of the observed OS data from the pivotal clinical trial to project patients’ survival beyond the trial period, and typically over their lifetime⁹. The independent Evidence Review Group (ERG) then critique the CEA and the associated extrapolations, and provide the committee with comments, suggestions, or alternative analyses.

The primary goal of treatment with ICI is to target and kill cancer cells, ideally leading to the eradication of the cancer, and thus improving OS^10,11. OS is often a key driver in measures of both clinical- and cost-effectiveness of ICIs. Thus, if OS data are immature, the committee may reach a negative recommendation. A draft negative recommendation may lead to the manufacturer providing a confidential discount for the health technology being assessed (via a patient access scheme, for example), or the health technology may simply not be recommended for routine National Health Service (NHS) commissioning indefinitely.

Where there is a high degree of clinical uncertainty associated with the trial data and, thus, a low chance of recommendation by a committee under normal circumstances, the Cancer Drugs Fund (CDF) may offer intermediate access to new cancer medicines pending collection of further data (typically as part of further follow-up from the pivotal clinical trial)¹². Following a period of data collection, the technology will be re-assessed by NICE in consideration of these newly-available data. Several ICIs have been recommended for intermediate funding through the CDF to date^13–17.

Whether an ICI is recommended (for routine commissioning or within the CDF) or not, it is of clear interest to ascertain whether the estimated longer-term OS was aligned with data from the pivotal trial made available post-recommendation. An over-estimation of OS in the intervention arm may have led to positive recommendation of a technology that does not demonstrate true value for money, and an under-estimation of OS may have misleadingly led to an incorrect negative recommendation. However, there is no explicit mechanism wherein this is routinely assessed via the NICE STA process.

Within this study, the following research questions were addressed:

1. Where reported, did the manufacturer and ERG-preferred extrapolations over-estimate or under-estimate OS for the intervention?

2. To what extent were the projections of OS for the intervention aligned with the more mature Kaplan-Meier (KM) curves of OS from the pivotal clinical trial (i.e. what is the difference between the originally projected OS and KM curve from a later data cut with longer follow-up)?

In addition, a review of the discussion—either criticism or acceptance—around the manufacturer-preferred OS extrapolations by both the committee and ERG was also undertaken. This study was undertaken to highlight the importance of appropriate survival extrapolation methods, by quantifying differences between the OS initially projected by survival extrapolation methods (based on shorter follow ups) and the KM curve derived from longer follow-ups. This information is valuable to reduce uncertainty in decision-making in the presence of immature survival data.

Methods

Identification of relevant NICE STAs and additional data

A systematic search of the NICE website (https://www.nice.org.uk/guidance/published) was conducted to identify relevant STAs of ICIs. The search was limited to completed STAs of ICIs published between March 1, 2000 (date of first published technology appraisal guidance) and December 31, 2017¹⁸. The search strategy was constructed by considering the US National Cancer Institute definition of an ICI, in order to identify all relevant (and licensed) technologies by the European Medicines Agency and US Food and Drug Administration¹⁹. This covers all licensed technologies targeting the known immune checkpoints PD-1, PD-L1, and CTLA-4⁸. An overview of the technologies searched for is provided in Table 1.

Table 1. Known licensed immune-checkpoint inhibitors.

Generic name	Brand name	Manufacturer	Target	Licensed
Ipilimumab	Yervoy	Bristol-Myers Squibb	CTLA-4	2011
Pembrolizumab	Keytruda	Merck, Sharpe and Dohme	PD-1	2014
Nivolumab	Opdivo	Bristol-Myers Squibb	PD-1	2014
Atezolizumab	Tecentriq	Roche	PD-L1	2015
Durvalumab	Imfinzi	AstraZeneca	PD-L1	2017
Avelumab	Bavencio	Merck KGaA/Pfizer/Eli Lilly	PD-L1	2017

CTLA-4, cytotoxic T-lymphocyte associated protein 4; PD-1, programmed-death 1; PD-L1, programmed-death ligand-1.

Note: The immune-checkpoint inhibitors (ICIs) presented in this table are all ICIs with a license issued by the European Medicines Agency (EMA) before December 31, 2017.

After identifying the relevant NICE STAs, the manufacturer’s submission dossier was searched to obtain details of the pivotal clinical trial used to determine estimates of OS within the submitted CEA. Given the intention of this study was to compare survival extrapolations considered within the manufacturer’s submission to longer-term OS data from the pivotal clinical trial, a targeted search of the literature was undertaken to identify later data cuts. The sources searched included the clinicaltrials.gov website, conference websites for the American and European Societies for Clinical and Medical Oncology (ASCO and ESMO, respectively), as well as a Google® search. Advice was also sought to validate the identified data cuts by clinical experts in ICIs.

The following exclusion criteria were applied for the selection of NICE STAs:

1. A clinical indication wherein an alternative potentially curative non-ICI intervention (such as stem cell transplant) may be considered in a subsequent line of therapy (as this could impact the estimation of OS).

2. No later data cut than that presented as part of the manufacturer’s submission to NICE could be identified (as a lack of longer-term data prohibits comparison, as per this study’s aim).

The following data were extracted from the relevant NICE STAs:

1. The available OS data at the time of submission, provided in the form of a KM curve;

2. The manufacturer’s extrapolation of OS used to inform the submitted CEA; and

3. Where reported, the ERG-preferred extrapolation method prior to the first Appraisal Committee Meeting (ACM).

OS data in the form of KM curves were extracted based on the aforementioned targeted search to identify later OS data cuts in published literature. Based on the KM curves extracted from the literature and NICE STA documentation, the difference in OS was determined between the initial and the most mature KM curves. This was calculated at the maximum follow-up time based on the initial KM curve.

Measurements of difference in observed vs projected OS

For each of the relevant STAs, the extrapolation methods employed by the manufacturer were extracted and the KM curves and OS extrapolations were digitized using the GetData Graph Digitizer digitization software^20,21. The KM curves based on trials with different cut-offs and the OS extrapolation (manufacturer- or ERG-preferred) were overlaid for visual inspection and assessment.

To quantify the difference between initial OS extrapolation and most mature trial data, the proportion of patients alive at two time points were considered:

1. The maximum follow-up based on the initial KM curve. This comparison is more robust as it is expected at-risk patient numbers are highest for the most mature KM curve at the maximum follow-up for the initial KM curve. However, the difference calculated is expected to be small and less informative for addressing the research questions (as OS extrapolations will likely align with the initial KM curve within the trial period).

2. The mid-point between the maximum follow-ups of the initial and most mature KM curve. This choice provides a pragmatic trade-off between robustness of the most mature KM curve and the relevance of addressing the research questions (i.e. how initial OS extrapolation predicts future, more mature OS data).

The time point for the maximum follow-up based on the most mature KM curve was not assessed, because the KM curve towards the end of the duration of trial follow-up tends to be highly uncertain due to the small number of at-risk patients²².

To illustrate the comparisons made, TA319 may be considered as an example. The original KM curve (as included in TA319) had a maximum follow-up of 47 months, and the most mature KM curve had a maximum follow-up of 72 months. The difference in OS at 47 months for both KM curves was first calculated. Then, the differences between the manufacturer- and ERG-preferred OS extrapolations and the most mature KM curve were calculated at 47 and 60 months.

Results

Identified NICE STAs

A total of 362 completed STAs were identified as completed between March 1, 2000 and December 31, 2017 within the search limits, of which 348 did not assess ICIs and were excluded. Hence, 14 completed STAs of ICIs were identified, spanning seven distinct clinical indications^{13–17,23–31}. Of these, one (TA462) incorporated the use of allogeneic stem cell transplantation in classical Hodgkin’s lymphoma; this STA was excluded because stem cell transplantation can be considered subsequent treatment with curative intent³¹.

The targeted search of the literature to identify relevant data cuts from each of the pivotal clinical trials yielded 22 distinct sources of OS data for the pivotal trials of the 13 remaining STAs^{1–4,32–49}. Two STAs (TA268 and TA492) considered a pivotal trial for which no longer-term OS data beyond those considered at the time of submission were identified, and so these STAs were excluded^17,23. Therefore, a total of 11 STAs form the evidence base for this study. An overview of the STA identification process is provided in the form of a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram in Figure 1.

Figure 1. PRISMA diagram for inclusion of relevant NICE STAs. ASCT, autologous stem cell transplantation; CHL, classical Hodgkin’s lymphoma; ICI, immune-checkpoint inhibitor; NICE, National Institute for Health and Care Excellence; NSCLC, non-small-cell lung cancer; RCC, renal cell carcinoma; SCCHN, squamous cell carcinoma of the head and neck; STA, Single Technology Appraisal.

Included NICE STAs

The identified STAs covered the following advanced/metastatic cancer indications: melanoma (n = 5), non-small-cell lung cancer (NSCLC, n = 4), renal cell carcinoma (RCC, n = 1), and squamous cell carcinoma of the head and neck (SCCHN, n = 1). The ICIs assessed were: nivolumab (n = 6), pembrolizumab (n = 4), and ipilimumab (n = 2). STAs for the other three identified ICIs (atezolizumab, avelumab, and durvalumab) were not completed by December 31, 2017.

The preferred method employed by the manufacturer to extrapolate OS in each STA is presented in Table 2. Seven of the 11 STAs adopted a piecewise approach—five of which^24–28 considered external data^5,50 to adjust long-term OS estimates. The remaining four STAs applied standard parametric survival models capable of accounting for non-monotonic hazard functions (i.e. the log-logistic, log-normal, and generalized gamma distributions).

Table 2. Extrapolation methods utilized in each relevant NICE STA.

STA identifier, ICI, and indication	Trial	Manufacturer extrapolation method
TA319, ipilimumab, melanoma	CA184-024	Piecewise: KM to 2 yrs, log-normal to 5 yrs, AJCC data
TA357, pembrolizumab, melanoma	KN-002	Piecewise: KM to 1 yr, ipilimumab data to 7 (TA366) or 10 yrs (TA357), AJCC data
TA366, pembrolizumab, melanoma	KN-006
TA384, nivolumab, melanoma	CM-066	Piecewise: TTP/PrePS/PostPS until 3 years, ipilimumab data
TA400, nivolumab + ipilimumab, melanoma	CM-067
TA417, nivolumab, RCC	CM-025	Generalized gamma
TA428, pembrolizumab, NSCLC	KN-010	Piecewise: KM to 1 yr, exponential
TA447, pembrolizumab, NSCLC	KN-024	Piecewise: KM to 22 wks, exponential
TA483, nivolumab, squamous NSCLC	CM-017	Log-logistic
TA484, nivolumab, non-squamous NSCLC	CM-057	Generalized gamma
TA490, nivolumab, SCCHN	CM-141	Log-normal

AJCC, American Joint Committee on Cancer; CM, CHECKMATE; ICI, immune-checkpoint inhibitor; KM, Kaplan–Meier; KN, Keynote; NICE, National Institute for Health and Care Excellence; NSCLC, non-small cell lung cancer; PostPS, post-progression survival; PrePS, pre-progression survival; RCC, renal cell carcinoma; SCCHN, squamous-cell carcinoma of the head and neck; STA, Single Technology Appraisal; TA, technology appraisal; TTP, time to progression; wk(s), week(s); yr(s), year(s).

Notes: Ipilimumab data reported by Schadendorf et al.⁵. AJCC data reported by Balch et al.⁵⁰.

Evidence Review Group extrapolation methods not reported, as this information was unclear in six of the STAs.

The 11 included STAs were critiqued by six ERGs. The methods preferred by each of the ERGs were not systematically reported. Each of the four STA committees assessed at least one of the 11 included STAs. Committee A conducted the five melanoma STAs included. Committees B, C, and D conducted the STAs related to NSCLC, RCC, and SCCHN. The choice of base-case extrapolation method was amended and/or rejected by the ERG in all but one of the STAs (TA447)¹³. The committee only accepted one manufacturer-submitted extrapolation in preference to the ERG-preferred extrapolation (TA384)²⁷.

Figure 2 provides an overview of the OS extrapolations alongside available trial data. Quantitative results regarding the difference between initial OS extrapolation and most mature OS data are presented in Tables 3 and 4.

Figure 2. Observed and extrapolated OS.

Table 3. Overview of results: individual STAs.

Comparison	STA number
	319^24	357^25	366^26	384^27	400^28	417^29	428^30	447^13	483^14	484^15	490^16
Maximum reported OS time (months)
Original KM	47.4	16.5	17.2	16.7	NR	30.6	22.2	18.7	22.7	25.3	16.6
Most mature KM	72.0	35.2	26.1	27.8	45.9	54.0	35.0	33.0	51.9	51.9	24.4
Absolute change in OS % at end of original KM for the original KM vs*
Most mature KM	+9.6	+0.0	−0.5	−4.5	NR	+2.6	−4.0	−7.2	−5.1	+6.7	−6.8
Absolute change in OS % from most mature KM at end of original KM vs^†
Manufacturer-preferred extrapolation	+0.5	−5.0	−8.1	−3.6	NR	−0.2	+1.8	+0.4	−0.3	+2.4	+4.2
ERG-preferred extrapolation	NR	−1.8	−3.1	NR	NR	−0.2	NR	+0.4	NR	+2.5	NR
Absolute change in OS % from most mature KM at midpoint between original and most mature KMs vs^†
Manufacturer-preferred extrapolation	−0.6	−5.6	−9.2	−8.2	NR	−2.5	+3.5	−2.5	−0.0	+1.5	+4.6
ERG-preferred extrapolation	NR	−2.8	−2.9	NR	NR	−2.5	NR	−2.5	NR	−2.7	NR

ERG, Evidence Review Group; KM, Kaplan–Meier; NR, not reported; OS, overall survival; STA, single technology appraisal.

*A negative value here indicates that the most mature (updated) KM provided a lower estimate of OS compared to the original KM, and vice versa.

^†A negative value here indicates that the extrapolation under-estimated OS, whereas a positive value here shows the extrapolation over-estimated OS.

Table 4. Overview of results: summary.

Comparison	Outcome
	Number of STAs	Mean	Median	Lower bound	Upper bound
Change in OS % at end of original KM for the original KM vs*
Most mature KM	10	−0.9	−2.2	−7.2	+9.6
Difference in OS % from most mature KM at end of original KM vs^†
Manufacturer-preferred extrapolation
All STAs	10	−0.8	+0.1	−8.1	+4.2
STAs which under-estimate OS	5	−3.4	−3.6	−8.1	−0.2
STAs which over-estimate OS	5	+1.9	+1.8	+0.4	+4.2
ERG-preferred extrapolation
All STAs	5	−0.4	−0.2	−3.1	+2.5
STAs which under-estimate OS	3	−1.7	−1.8	−3.1	−0.2
STAs which over-estimate OS	2	+1.4	+1.4	+0.4	+2.5
Difference in OS % from most mature KM at midpoint between original and most mature KMs vs^†
Manufacturer-preferred extrapolation
All STAs	10	−1.9	−1.5	−9.2	+4.6
STAs which under-estimate OS	7	−4.1	−2.5	−9.2	−0.0
STAs which over-estimate OS	3	+3.2	+3.5	+1.5	+4.6
ERG-preferred extrapolation
All STAs	5	−2.7	−2.7	−2.9	−2.5
STAs which under-estimate OS	5	−2.7	−2.7	−2.9	−2.5
STAs which over-estimate OS	0	N/A	N/A	N/A	N/A

ERG, Evidence Review Group; KM, Kaplan–Meier; N/A, not applicable; OS, overall survival; STA, single technology appraisal.

*A negative value here indicates that the most mature (updated) KM provided a lower estimate of OS compared to the original KM, and vice versa.

^†A negative value here indicates that the extrapolation under-estimated OS, whereas a positive value here shows the extrapolation over-estimated OS.

Overview of additional OS data

Of the 11 included STAs, 10 included OS data to inform the manufacturer’s submission to NICE. For one of the STAs (TA400) OS data were not available at the time of submission, but later longer-term OS data were identified in published literature^3,28. Therefore, while a comparison to the OS data available at the time of submission for TA400 was not possible, a comparison of the OS extrapolation and subsequently published data was undertaken.

Following the manufacturer’s initial submission for each of the 10 STAs that included OS data, follow-up data published post-NICE submission provided a mean additional 18 months of OS data to the end of the KM curve (median =16; range =8–29). The mean difference in OS at the end of the original KM curve and the OS at the same time point using the most mature published KM curve was −0.9% (median = −2.2%; range = −7.2–9.6%).*

Comparison of OS extrapolation and most mature OS at the end of originally submitted OS data

When comparing the original extrapolation preferred by the manufacturer versus the most mature KM curve at the last available time point from the original KM curve, the mean difference was −0.8% (median =0.1%; range = −8.1–4.2%), which shows on average the original OS extrapolations slightly under-estimated OS when more mature data became available. Five of the 10 included STAs for which data were originally submitted were shown to under-estimate OS by a mean of 3.4% (median =3.6%; range =0.2–8.1%). Conversely, five STAs were shown to over-estimate OS by an average of 1.9% (median =1.8%; range =0.4–4.2%).

Of the five STAs for which the ERG-preferred extrapolation was presented (and with a long enough extrapolation to facilitate comparison with the latest available data), the mean difference was −0.4% (median = −0.2%; range = −3.1–2.5%). On average, this equates to a slight under-estimate of OS compared to the latest KM curve. Three ERG-preferred extrapolations under-estimated OS by a mean of 1.7% (median =1.8%; range =0.2–3.1%). Two STAs over-estimated OS by 0.4% and 2.5%.

Comparison of OS extrapolation and most mature OS beyond the end of originally submitted KM curve

For each of the 10 included STAs for which OS data were originally submitted, the manufacturer-preferred extrapolation and the most mature KM curve at the midway time point between the end of the originally submitted and the most mature KM curves were compared. On average, this midway time point was 9 months after the end of the original KM curve (median =8; range =4–15). The mean difference in OS at this time was −1.9% (median = −1.5%, range = −9.2–4.6%), with seven STAs under-estimating and three STAs over-estimating OS.

Of the seven extrapolations shown to under-estimate OS, the average difference was 4.1% (median =2.5%; range =0.0–9.2%). The remaining three STAs were shown to over-estimate OS by an average of 3.2% (median =3.5%; range =1.5–4.6%). All five STAs for which the ERG-preferred extrapolation was presented were shown to under-estimate longer-term OS by an average of 2.7% (median =2.7%; range =2.5–2.9%).

Discussion

The choice of OS extrapolation method and base-case selection among multiple parametric functions has the potential to affect estimates of the clinical- and cost-effectiveness of ICIs. Within the STA process the manufacturer and ERG may each prefer different extrapolations of OS, and the committee must come to a decision about which method it believes provides a more plausible basis for decision-making. This study attempts to compare OS extrapolations preferred by the manufacturer and ERG at the time of submission to later published data cuts from the pivotal clinical trials.

Extrapolation methods

The methods presented in completed STAs of ICIs largely revolved around two approaches: standard parametric models that consider non-monotonic hazard functions, or piecewise models that combine the KM curve with parametric models (and may also consider external data). Standard parametric models are limited in terms of their flexibility in quantifying the underlying hazard function⁶. Piecewise models require specification of the number of cut-offs and/or the time points at which they are applied, which may be subject to debate⁵¹. Since publication of these STAs, alternative extrapolation methods have become more prominent in the literature within the context of modeling OS for ICIs, such as spline-based and parametric mixture-cure models, which may avoid some of the aforementioned issues^52–55.

Many of the extrapolation methods relied upon data from external data sources^5,50. However, these data may not always be suitable to provide insight into the likely longer-term OS if deemed incomparable (e.g. in terms of patient population or treatment[s] received). Despite the limitations of naïvely applying external data, they may provide a useful benchmark for the longer-term OS of patients within a given clinical indication (e.g. American Joint Committee on Cancer [AJCC] data in melanoma), where used appropriately, and may consequently be used in decision-making.

The extrapolation method preferred by the manufacturer for the ICI intervention was either rejected or subject to amendment by the ERG in nearly all of the identified STAs. For the only exception (TA447), the ERG instead focused on amending the extrapolation for the comparator arm. The ERG cited external registry data that suggested 5-year OS within the manufacturer-preferred extrapolation for the comparator arm was too low, resulting in potential bias in favor of the intervention¹³. Therefore, the choice of manufacturer-preferred extrapolation method for at least one of the modeled health technologies was criticized by the ERG in each of the included STAs.

The ERG-preferred method was only clearly reported in five STAs for a time period long enough to facilitate comparison to the OS data. Furthermore, the maximum time point for which the manufacturer-preferred OS extrapolation was presented across all 11 included STAs was 5 years, with the only exception being TA319, which reported up to a maximum of 40 years²⁴. Given the importance of OS extrapolation, the preferred OS extrapolations should be made clearer in the reports (for example, by providing plots over the entire modeled time horizon, and presenting model coefficients) produced by both manufacturers and ERGs.

Availability of OS data

The OS data with the longest follow-up for any treatment in any indication came from the pivotal CA184-024 trial of ipilimumab in melanoma (considered in TA319)²⁴. The manufacturer-preferred OS extrapolation was shown to match the KM curve from this trial well, up until and including 6 years of follow-up data. While there is often strong justification for the lack of follow-up in ICI trials (which may be un-blinded at interim analysis following recommendations from independent safety monitoring boards due to the success of efficacy outcomes), there is a clear need to continue data collection to validate OS extrapolations.

Uncertainty in the tail of the KM curve is a known limitation in the field of survival extrapolation²². The results of this analysis show that the difference between the point estimates of OS between the initial and the most mature KM curves at the end of the initial KM curve ranges between 9.6% and −7.2%, demonstrating that the far tails of KM curves may be unstable. Therefore, it is likely beneficial for manufacturers to present, and for ERGs and committees to acknowledge, a broad range of candidate OS extrapolations, which may provide OS estimates higher or lower than the end of the initially available KM curve. These differences in estimated OS in the short-term may lead to larger differences in longer-term OS, which could considerably affect the estimates of overall clinical effectiveness.

As expected, the STA for which the manufacturer’s extrapolation differed most from the observed data was TA400, as there were no reported OS data at the time of submission. For the other 10 STAs, for which OS data were available, initial extrapolations appear to provide reasonable estimates of OS when compared to the mature OS data. This shows the importance of presenting and using OS data in the STA, as TA400 demonstrates that the OS benefits of the ICI may have been under-estimated due to lack of OS data in the initial submission.

Comparison of predicted and observed OS

The results of this study suggest that, while the manufacturer-preferred OS extrapolation may over- or under-estimate longer-term OS, on average OS was under-estimated. This comparison was only possible to make when more mature data became available. In one case, OS was under-estimated by as much as 9.2% (TA366, midpoint estimate), whereas, for another case, OS was over-estimated by 4.6% (TA490, midpoint estimate)^16,26. In general, the extent of OS over-estimation was less pronounced vs OS under-estimation, by both the manufacturer and ERG.

Two measures are proposed in this study to assess the prediction error between initial OS extrapolations and more mature (previously unavailable) KM curves. The first uses the time at the end of the original KM curve and places a greater weight on the robustness of the more mature KM curve. However, it is less informative regarding the assessment of the predictability of the original OS extrapolation. The second uses the midpoint between the end of the original and more mature KM curve. This method was chosen pragmatically to place more importance on the assessment of predictability of the original OS extrapolation, but to some extent sacrifices the robustness of the KM curve in doing so. While each method is associated with advantages and disadvantages, the latter approach seems to provide a more suitable approach for this study.

To the authors’ knowledge, this study reports the first comparison of OS extrapolations presented by manufacturers and ERGs in published NICE guidance to subsequently published OS data unavailable at the time of submission. Systematic methods were adopted to identify relevant STAs, and objective and quantifiable measures were used to perform the comparison.

A clear limitation of this study is the reliance on sufficient presentation of manufacturer- and ERG-preferred OS extrapolations. The manufacturer-preferred extrapolation was typically clearly presented within the manufacturer’s submission, but was often truncated at a seemingly arbitrary time point—for example, in TA447 the manufacturer’s plot of extrapolated OS was cut at 5 years, at which point ∼20% of patients were still estimated to be alive¹³. The ERG-preferred extrapolation was often unreported, and so the results of the comparison of ERG extrapolations was limited to those for which data were available.

The accuracy of the survival extrapolations presented in this study are expected to be influenced by a multitude of factors, including the cancer type, baseline patient characteristics, and the time point at which survival accuracy was assessed. It was not possible to adjust for these factors due to the reporting of only aggregate-level data and published KM curves at specific points in time for specific groups of patients. Recent initiatives such as Project Data Sphere and Clinical Study Data Request have been established to encourage the sharing of individual-level data^56,57, which may permit in the future a more in-depth comparison of population-adjusted survival extrapolations.

In addition to the reporting of data, a further limitation of the analysis is the need to digitize KM curves and survival extrapolations to facilitate comparison. Digitized data are produced via manually tracing images; therefore, the results of this study are subject to human measurement error. However, as the digitized data are used only to facilitate a comparison of survival times with no formal statistical testing, we do not consider this limitation constitutes a high risk of introducing bias into the results of the analysis.

This study considers published STA guidance by NICE up until December 31, 2017. Since this time, NICE has published guidance for another two STAs (TA517, avelumab for metastatic Merkel cell carcinoma; and TA519, pembrolizumab in metastatic urothelial carcinoma)^58,59. Longer-term OS data may be published over time for each of the pivotal trials considered in these STAs, as well as even longer-term data for those already considered within this study. An assessment of each of the extrapolation methods used within these STAs may be undertaken in the future.

A total of 11 STA’s met the inclusion criteria to be considered within the study. Consequently, it was not possible to identify any clear patterns regarding the extent to which different methods may be prone to over- or under-estimate OS due to the small sample size. However, as NICE perform a larger number of ICI STA’s, it may be possible to assess extrapolation method performance in the future.

Conclusions

Long-term extrapolation of OS is required for STAs of ICIs and other OS-extending treatments, and are often based on maturing OS data available at the time of submission. The choice of OS extrapolation method is of clear importance in order to accurately ascertain the expected clinical and cost-effectiveness of new health technologies such as ICIs. The results of this study demonstrate initial OS extrapolation employed by manufacturers and ERGs generally predicts OS reasonably well when more mature data become available, though on average appeared to under-estimate OS.

This review and validation shows that, while the choice of OS extrapolation at the time of submission may not be perfect, the methods adopted are generally aligned with later-published follow-up data. Differences in predicted and observed OS were shown to be limited, and no clear pattern of over- or under-estimation was identified. However, further data collection and research is required to confirm the findings of this study and to improve the accuracy of survival extrapolation in general. Nonetheless, our study illustrates that the extrapolation methods used to inform STA’s of ICI’s appear appropriate for HTA decision-making, and constitute a reasonable basis from which extrapolation methods for future appraisals may be further developed.

Transparency

Declaration of funding

This study and manuscript were supported by funding from Merck, Sharp & Dohme Limited, Hoddesdon, UK.

Declaration of financial/other relationships

The co-authors RAI, TH, and CO are employees of Merck, Sharp & Dohme Limited (UK). Peer reviewers on this manuscript have received an honorarium from JME for their review work, but have no other relevant financial relationships to disclose.

Acknowledgments

We thank Miss Noni Bryson for the administration and editorial support. We also thank the reviewers for their helpful comments that greatly improved the manuscript.

Notes

* A negative value here indicates that, on average, the most mature (updated) KM provided a lower estimate (by 0.9%) of OS compared to the original KM.