Use of real-world evidence in meta-analyses and cost-effectiveness models

Abstract

Real-world evidence (RWE) provides external validity, supplementing and enhancing the randomized controlled trial data with valuable information on patient behaviors and outcomes, turning efficacy and safety results into real-world effectiveness and risks, but the use of RWE is associated with challenges. The objectives of this communication were to (1) summarize all guidance on how to conduct an RWE meta-analysis (MA) and how to develop an RWE cost-effectiveness model, (2) to describe our experience, challenges faced and solutions identified, (3) to provide recommendations on how to conduct such analyses. No formal guidelines on how to conduct an RWE MA or to develop an RWE cost-effectiveness model were identified. Using the context of non-vitamin K antagonist oral anticoagulants in stroke prevention in atrial fibrillation, we conducted an RWE MA, after having identified sources of uncertainty. We then implemented the results in an RWE cost-effectiveness model, defined as a model where all inputs come from RWE, including all parameters related to treatment effect. Based on challenges faced, our first recommendation relates to the necessity of conducting sensitivity analyses, either based on clinical or methodological considerations. Our second recommendation is the need for extensive collaboration with a wide range of experts, during the development of the analyses protocols, the implementation of the analyses and the interpretation of the results. RWE may address a number of gaps related to the treatment effect, and RWE economic evaluations for the treatment effect can provide extremely valuable insights into real-world economic value of interventions. As the increased recognition of the value of RWE could influence health technology assessment decision, and current practices, this communication supports the urgent need of more formal guidelines.

Keywords:

• Anticoagulants
• atrial fibrillation
• meta-analysis
• cost-effectiveness
• real-world evidence
• stroke prevention

JEL CLASSIFICATION CODES:

• I19
• I20

Introduction

Randomized controlled trials (RCTs), considered the gold standard for therapeutic evidence, are required by health technology assessment (HTA) agencies, as having the benefit of internal validity. Real-world evidence (RWE), defined as clinical evidence derived from analysis of real-world data (RWD), provides external validity, turning efficacy and safety results into real-world effectiveness and risks, thus supplements and enhances the RCT data with valuable information on actual patient behaviors and outcomes, and is more likely to be generalizable to real-life clinical practice^1,2.

RWD comprises longitudinal, cross-sectional, retrospective, or prospective data obtained from databases, population surveys, patient chart reviews, cohort studies, or registries. As such, inherent differences between RCT data and RWD should be noted, such as the purpose of data collection, the collection design, the data selection process, the duration of follow-up, the types of endpoints available, etc. However, one can benefit from the other: while RCT data may demonstrate causality, the use of RWE may improve the understanding of the patient population who will receive a treatment strategy, and of relative effectiveness.

To address concerns regarding escalating costs and appropriate reimbursement, HTA agencies are increasingly asking sponsors to confirm the safety and effectiveness for drugs that are being re-assessed, as well as the cost-effectiveness in real-life settings. This leads sponsors to increasingly consider generating RWD, paying attention to the role of RWE to introduce effectiveness data into decision-analytic models to conduct RWE cost-effectiveness analyses, as opposed to trial-based cost-effectiveness analyses^3–5.

However, the use of RWE is associated with several challenges. One of the main challenges in these types of studies is the selection bias; because the treatment assignment is non-random, the estimated effects of treatment on outcomes are subject to bias in attributing causality. Also challenging is the lack of accurate data on drug exposure and outcomes, missing data, and errors during the record-keeping process^6–9. Therefore, standard methodologies for the conduct of meta-analyses may not be appropriate, and implementation in cost-effectiveness models may not be straightforward.

In 2007, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Real-World Data Task Force issued a statement officially supporting the use of RWE for coverage and reimbursement decision-making³. However, a lack of consensus regarding how best to integrate RWD into economic models, and to what extent, persists^3,10.

Recently, two systematic literature reviews (SLRs) on the existing recommendations for working with RWE in an MA¹¹ and in RWE cost-effectiveness models¹² were published. In light of the existing recommendations identified in these SLRs, we conducted an RWE meta-analysis (MA)¹³ and developed an RWE cost-effectiveness model¹⁴, on the use of non-vitamin K antagonist oral anticoagulants (NOACs) in the context of stroke prevention in atrial fibrillation (AF).

This therapeutic area was considered as an illustrative example as per the substantial burden of AF, whether in terms of morbidity, mortality, or associated economic burden on the healthcare system¹⁵. Until the European Medicines Agency approval of the first NOAC (dabigatran) in 2008, warfarin, a vitamin K antagonist (VKA) was the mainstay drug for stroke prevention in patients with non-valvular AF^16,17. Since then, three additional NOACs have been approved (apixaban, edoxaban, and rivaroxaban). RCTs have shown NOACs at least as efficacious as VKA for the prevention of strokes or systemic embolism in patients with non-valvular AF^18–21, and numerous regulatory bodies and guidelines worldwide have endorsed the use of NOACs¹⁷.

This communication has a three-step objective: first, to summarize the guidance we found in the literature on how to conduct an RWE MA and how to develop an RWE model; second, to describe our experience, highlighting the methodology developed, the challenges faced and the solutions identified; and finally, to provide recommendations on how to conduct such analyses.

RWE meta-analysis

Although guidelines are available on how to conduct an MA using RCTs²², the published SLR revealed that there are no formal guidelines on how to conduct an RWE MA¹¹, i.e. an MA with real-world studies, for the purpose of effectiveness/safety demonstration. However, individual initiatives provided recommendations, and confirmed that researchers should take into consideration the specific challenges associated with these studies to avoid producing results that are misleading^23,24. The two main recommendations identified from the literature refer first to the need for quality assessment of the real-world studies and second to the need to conduct sensitivity analyses to better understand the uncertainty around RWE MA results.

Assessing the quality of RWE may be helpful to understand the extent of bias affecting the data. Some useful tools that can be used for this purpose include the one from Downs and Black²⁵, the GRACE²⁶ or the GATE²⁷ checklists²⁸. No consensus was reached on the best tool to use for quality assessment or on how to properly correct for bias, and no consensus was found on the way the assessment should be measured¹¹. Very different results are obtained depending on the tool considered²⁹ mainly because some tools are scales, with the possibility to derive a single score, whereas some other tools are checklists, only assessing components, or are even only qualitative. Finally, there was no consensus on whether to and how to consider both the risk of bias of individual studies, and the quality of the evidence.

The second common recommendation was the need for sensitivity analyses; however, it was not clear what these sensitivity analyses should consider. Several examples were identified, such as the need for different analyses based on different study designs, or subgroup analyses based on different population definitions or outcomes definitions. Exploration and consideration of the risk of bias in the MA was also reported as of interest. It would be possible to record the biases that affect each study, to discuss them qualitatively, to exclude bad-quality studies, to perform quality-based subgroup analyses, or to consider credibility ceiling corrections³⁰. Another option would be to weigh studies by overall quality in quantitative synthesis³¹. However, none of these options has been validated empirically and there is no consensus on the best option so far. An additional important limitation is that the elicited biases are necessarily subjective, as assessors might not be consistent in how they judge the same bias on different occasions.

Taking into consideration these recommendations, an MA of RWE was performed on the use of NOACs vs. VKAs in non-valvular AF¹³. The first step was to conduct an SLR of the available real-world studies³², on the guidance by the Centre for Reviews and Dissemination from the University of York²² and the Cochrane Handbook for Systematic Reviews of Interventions³³. Next, the different sources of uncertainty were identified. From a clinical perspective, patient population definition, the outcomes definition, the dosage, and the follow-up duration were found relevant. From a methodological perspective, the issue of sample overlap, types of reference used (full text or abstracts), types of adjustment used in the statistical analysis, the type of model used, and quality assessment were also important.

Having identified these sources of uncertainty, a base case for the MA was agreed, and a list of scenario analyses was performed, all considering random effect models as per the studies heterogeneity. For the base case, no evidence was excluded, i.e. all full texts, as well as abstracts, and regardless of adjustment type were included. No particular patient populations were excluded, i.e. both incident and prevalent populations were included, and different doses were pooled together. The analysis was restricted to avoid including the same patients several times, by selecting only the study with the highest level of precision whenever there was more than one study using the same database. Finally, regarding follow-up, only the study with the longest follow-up was used.

Different alternative scenarios to explore the impact of methodological choices on the MA results³⁴ were then investigated. In the first scenario, which involved the patient population, only incident cases were considered. The second scenario included only studies with properly adjusted hazard ratios and discarded those that presented only crude values or inadequate adjustments. The third scenario included all studies, without concern for sample overlap. In the fourth scenario, different doses were considered separately instead of being pooled together like in the base case. Finally, since the quality of RWE was found important from existing recommendations, the fifth scenario involved performing a quality assessment. The method presented by Doi et al.³¹ was used to weigh the studies based on their quality assessment using the Downs & Black checklist²⁵.

In the development of this MA, the useful recommendations identified in literature were followed. Nevertheless, there is a great need for guidance regarding the use of RWE in an MA. Figure 1 presents a summary of our recommendations. First, before formal guidelines are developed, researchers should enlist the help of relevant experts, including clinical, SLR, MA, and economic experts. Essential to this work was the extensive support from experts during each step of the process. During the identification of studies to include in the MA, experts validated the search strategy, the inclusion/exclusion criteria, and the extraction grid. They were also instrumental in the discussion around the sources of uncertainty and in the validation of the best scenario for the base case. Finally, their expertise was key to the interpretation of the base case results and scenarios. Our second recommendation is to assess uncertainty by running different scenarios. Five alternative scenarios were conducted, including one on quality of evidence, to assess the uncertainty regarding our methodological decisions. However, despite running all these analyses, guidance on which additional scenarios to run would have benefited this analysis.

Figure 1. Recommendations for the development of an real-world evidence meta-analysis protocol.

RWE cost-effectiveness modeling

There is no consensus on what exactly constitutes an RWE cost-effectiveness model. In this study, it is proposed that it is one where all inputs come from RWE, including all parameters related to treatment effect. Although use of RWE in cost-effectiveness analyses is very common in the literature, it is mainly used to populate costs, utilities, or epidemiological parameters, or to support external validation. To populate the parameters related to treatment effect, RCT data are very often used.

The use of RWE for treatment effect has several limitations, which might create challenges for incorporation in a model, particularly when an RWE MA is used. A recent SLR on the existing guidelines or recommendations revealed that guidelines on how to deal with challenges associated with the development of an RWE model are non-existent¹². Most elements found in the literature confirmed that RWE may be included or requested by many EU HTA agencies, and the potential biases associated with non-RCT data were acknowledged^35,36, but most of the time this was not related to treatment effect. Furthermore, although RWE is already used in submission dossiers, different HTA agencies have different policies regarding its use¹⁰.

As an illustrative example, the RWE cost-effectiveness of rivaroxaban compared with VKAs was assessed, using a French national healthcare insurance perspective¹⁴. The French perspective was chosen because NOACs are part of the national stroke plan in France, and their cost is, therefore, the object of intense examination.

RCT-based models were reviewed, with the aim of identifying key features of model structures and reviewing the critique from the appraisal of these models. In particular, a review of the UK National Institute for Health and Care Excellence dossiers for the four available NOACs, and of their associated appraisals, revealed that all four reviewed NOACs demonstrated a treatment benefit on stroke reduction and safety, including bleeding outcomes, but that several elements would benefit from RWE^37–40. First, key cost-effectiveness drivers of the appraised NOAC economic models included several factors that could potentially vary in real world. These included the discontinuation rate, cost of monitoring visits, and patient baseline age. Second, treatment effectiveness in the VKA arm is based on international normalised ratio control. Additionally, considerable variation between different centers and settings, depending on the patient group, were also identified. More generally, the generalizability of clinical evidence to patients diagnosed with AF in the national population for modeling was a key element of uncertainty.

For this analysis, a previously published Markov model⁴¹ was populated with inputs from RWE, including patient characteristics, the clinical event rates of VKAs, the persistence rates for VKAs, the treatment effect for rivaroxaban, the switch rates, utility, all-cause mortality, event-specific mortality, and costs, as shown in Table 1.

Table 1. RWE used as a key source of inputs.

Inputs	Sources
Patients characteristics	French real-world study^42
Clinical event rates for VKA	Real-world studies^13
Persistence rates for VKA	Real-world studies^43
Treatment effect for rivaroxaban	HRs from the RWE MA (incident and prevalent)^13
All-cause mortality	French life tables
Event-specific mortality	Real-world studies^44–47
Utility	European real-world studies^48–50
Costs	French real-world studies^44^,^51^,^52

Abbreviations. HR, Hazard ratio; MA, Meta-analysis; RWE, Real-world evidence; VKA, Vitamin K antagonist.

In addition, adjustments were required to update the model as per recently published guidelines (such as different monitoring), to incorporate new specificities (such as different treatment patterns), and finally to account for the critiques reported by HTA agencies.

Throughout the whole process, the close cooperation with clinical and economic experts was indispensable, given the lack of formal guidance on how to develop RWE cost-effectiveness models. Figure 2 presents a summary of our recommendations.

Figure 2. Recommendations for the development of an RWE cost-effectiveness model. Abbreviations. HTA, Health technology agency; RCT, Randomized controlled trial; RWD, Real-world data; RWE, Real-world evidence.

There is a clear need for methodological guidance regarding the use of RWE for cost-effectiveness analyses. Individual initiatives suggested that different types of evidence (e.g. pragmatic trials, registries, administrative databases, health surveys) have different benefits and limitations, which must be considered and weighted when seeking to integrate them to inform decision-making, but no clear recommendation was given⁵. There are several aspects that require further research, including the need to adjust endpoints to RWD reliability/consistency, and to develop relevant tools to evaluate how RWE cost-effectiveness models are developed.

A checklist to critically appraise statistical methods to address selection bias in cost-effectiveness analyses that use RWD was identified⁵³, in which five questions are considered:

1. Whether the study assessed the “no unobserved confounding” assumption and the validity of the instrument variable?

2. Whether the study assessed if the distributions of the baseline covariates overlapped between the treatment groups?

3. Whether the study assessed the specification of the regression model for health outcomes and costs?

4. Whether the covariate was balance assessed after applying a matching method?

5. Whether the study considered structural uncertainty arising from the choice or specification of the statistical method for addressing selection bias?

However, the use of this checklist in practice is limited, because it does not support the choice of any statistical method that should be preferably used, and only addresses the selection bias.

More generally, multiple other tools exist to determine how much confidence to place in the results, and to estimate if they are sufficiently credible and relevant to inform the decision^54,55. These tools mostly evaluate two dimensions:

1. Relevance of the structuring choices, i.e. how closely does the problem apply to the setting of interest to the decision maker (in particular, whether any underlying theories and assumptions specified relating to the structure of the model is clear, with a justification given for any choices made);

2. Credibility of the analysis, in particular in terms of design, data, validation, reporting, and interpretation

A recent review of these tools⁵⁴ identified the 36-item Drummond and Jefferson checklist⁵⁶ as the most common (30% of cases), and the Consensus on Health Economic Criteria list⁵⁷ as the second most common (18% of cases), out of 18 tools identified in total, since 2010. These tools, as most of the others identified, request the reviewer to subjectively evaluate appropriateness of model specifications, mostly with yes/no questions (Table 2), but do not consider the issues and challenges RWE cost-effectiveness models should address to ensure appropriate correction of bias.

Table 2. Items related to relative treatment effect in existing tools to evaluate cost-effectiveness analyses.

Adams et al. (1992)^58	• Were direct incremental benefits measured correctly? • Were indirect incremental benefits measured correctly? • Were the incremental benefits adjusted appropriately for differential timing?
Sacristan et al. (1993)^59	• Are the data on the effectiveness of alternatives adequately established?
Drummond and Jefferson (1996)^56	• Are the sources of effectiveness estimates used stated? • Are details of the design and results of effectiveness study given (if based on a single study)? • Are details of the method of synthesis or meta-analysis of estimates given (if based on an overview of a number of effectiveness studies)?
Sculpher et al. (2000)^60 and Philips et al. (2006)^61	• If relative treatment effects have been derived from trial data, have they been synthesized using appropriate techniques? • Have alternative assumptions been explored through sensitivity analysis?
Ungar et al. (2003)^62	• Is effectiveness, rather than efficacy, assessed? • What approach is used to assess the effectiveness/efficacy? • Are the details of the design of the effectiveness/efficacy study(s) provided? • Are the results of the efficacy/effectiveness of alternatives reported?
Chiou et al. (2003)^63	• Were the health outcomes measures/scales valid and reliable? If previously tested valid and reliable measures were not available, was justification given for the measures/scales used?
Evers et al. (2005)^57	• Are all outcomes measured appropriately? • Are outcomes valued appropriately?
Drummond et al. (2005)^64	• Where data from different sources are pooled, is this done in a way that the uncertainty relating to their precision and possible heterogeneity is adequately reflected? • If data from other settings are used, have these been assessed for relevance in the settings of interest?
Husereau et al. (2013)^65	• If synthesis-based estimates, is there a full description of the methods used for identification of included studies and synthesis of clinical effectiveness data?
Caro et al. (2014)^66	• Are the data used in populating the model suitable for the decision problem? • Were the analyses performed using the model adequate to inform your decision problem?

As for common cost-effectiveness analyses conducted using RCT information, updated or newly generated tools would enable identifying strengths and weaknesses of RWE cost-effectiveness models, and allow any user to rate the relevance and the credibility to inform decision-making.

Discussion

Historically, cost-effectiveness analyses were limited to cost-efficacy analyses for practical reasons; indeed, effectiveness was not available at the time of the HTA assessment. But this is less applicable for HTA re-assessment. In a context where RWE is more and more accessible, and its value more acknowledged, it is anticipated that the use of RWD in MA and cost-effectiveness models will be increasingly requested by HTA agencies, as this will provide decision makers with necessary and valuable information.

Unfortunately, formal guidelines are non-existent, and as of today the current guidance only relies on transparency, researcher judgment, and assessment of uncertainty. We need formal guidelines to avoid considering only recommendations from individual initiatives, and to clearly define what constitutes an RWE cost-effectiveness model. In our work, we decided to fully populate the model using RWD, but the question of how much RWD a cost-effectiveness model needs to have to be considered an RWE cost-effectiveness model should be discussed and answered. Perhaps an RWE model could also be one in which only treatment effect is sourced from the real world. Even in this case, though, methodological guidance is needed.

With clearer policies and guidance, the conduct of rigorous real-world studies and the appropriate incorporation of their findings into MAs and cost-effectiveness models may lead to analyses that more accurately represent the value of existing and emerging drugs. Efforts at providing guidance are currently under way (IMI GetReaL, the International Society for Pharmacoeconomics and Outcomes Research and International Society for Pharmacoepidemiology joint taskforce or the Center for Medical Technology Policy and Green Park Collaborative RWE Decoder) but none are yet fully available. We advocate for a discussion around at least the following aspects in these guidelines:

1. Real-world studies designs. Studies may present results at different follow-up times, and assumptions will be required alongside a discussion on which timeframe should be used (e.g. whether the longest is the most suitable). Additionally, studies may present results for different dosages, and sensitivity analyses may be required. Any study design characteristics should be investigated for potential impact on the results

2. Heterogeneity of population. Numerous assumptions are often required to define the population of interest in real-world studies. Therefore, patient populations are very likely to differ between included studies, and this heterogeneity should be considered. Restricted analysis to subgroups may be relevant, as well as identification of any correlations between the initial treatment choice and the characteristics of patients, as this could potentially reveal patient factors that may influence treatment decisions

3. Database overlap. Including studies conducted on the same database and analyzing similar outcomes could lead to the same patients being repeatedly included in the analysis, which could bias the results. Assumptions will be required to avoid pooled confidence intervals to tighten, which could lead to the detection of additional statistically significant outcomes

4. Outcome definition. The definition of outcomes may vary substantially in real-world studies, leading to difficulties in interpreting the results. Differences should be identified and discussed, and sensitivity analyses should be carried out. Development of guidance for researchers on defining events captured in real-world studies may be required

5. Types of adjustment used in the statistical analysis. Varying adjustment of hazard ratios may have an impact on the results. A discussion on the methods identified in the real-world studies should be expected, as it is likely that results obtained in the studies presenting crude comparisons would differ from those with very complex adjustments

6. Quality of evidence. Exploration and consideration of the risk of bias is crucial, as in general real-world studies are associated with a higher risk of bias than RCTs, implying a heightened need to evaluate their quality, and to consider scenarios

7. Types of reference. Conference abstracts (not published in full-text) may provide relevant evidence, although these are usually not very detailed, and may not contain most of the considerations listed below

Our recommendations include first the necessity of sensitivity analyses. Several scenarios were found relevant, either based on clinical or methodological considerations. While the latter considerations are likely to be transferable to other indications, the question of how the discussion around uncertainty of these results is transferable to other indications, first in cardiology, but more generally to other therapeutic areas, is of importance. Similar discussions were presented by Blommestein et al.² in the area of oncology. Authors concluded that outcomes of economic evaluations based on real-world studies are to be assessed differently than economic evaluations based on RCT data. First, endpoints in registries and RCTs may differ. For example, primary endpoints of RCTs in cancer are most often response, time to progression, and progression-free survival; in real-world studies, however, data on response and progression may be biased as it is not accurately captured in patient files. More generally, challenges associated with the use of RWD may differ depending on the disease area considered. In some disease areas, hard endpoints, or final outcomes are needed, such as in cardiovascular areas, where for instance myocardial infarctions or strokes are often implemented in models. These may be available in real-world databases, but definitions may be different as those in RCT. In other areas, only surrogate endpoints might be of interest. In such context, it might not be straightforward to collect RWD from claims databases, and specific registries or other observational studies may be more of interest; other types of challenges are to be expected, though.

Our second recommendation relates to the need for extensive collaboration with a wide range of experts that was essential in our experience, when using RWE for MAs and cost-effectiveness models. Our experts supervised the development of our protocols, the implementation of the analyses, and the interpretation of the results. Multidisciplinary expertise was considered important to ensure diversity of knowledge and perspectives.

We encourage other researchers to share their experience relative to RWE generation, synthesis or implementation in cost-effectiveness models, as well as acceptability of such evidence to key stakeholders, including HTA agencies or for the monitoring of post-launch real-world cost-effectiveness.

Conclusions

The integration of RWE into cost-effectiveness analyses of NOACs is likely to provide an additional level of information that can be used to evaluate the clinical and economic value of these medicines, with more generalizable data on patient population characteristics, by extension, on associated subsequent outcomes. RWE may address a number of gaps related to the treatment effect, and the increased recognition of the value of RWE could influence HTA decision, and current practices. Most existing cost-effectiveness models already consider RWE, either from international sources or focusing on a specific country, for resource use, or epidemiological data. But economic evaluations based on RWE for the treatment effect can provide extremely valuable insights into real-world economic value of interventions and will be increasingly required by decision makers in the coming years. Nevertheless, no consensus currently exists on how to use RWE for treatment effect in cost-effectiveness analyses, and more generally on what an RWE cost-effectiveness model is. Although key stakeholders from several European countries are working to establish an RWE roadmap to facilitate the generation of credible data that enhance the use of RWE in decision-making, this communication supports the urgent need of more formal guidelines. Research in this area is ongoing and what will be published will add to what has been proposed in this communication.

Transparency

Declaration of funding

This work was supported by Bayer AG.

Declaration of financial/other relationships

KB and JBB were employees of Bayer AG at the time of the study. AM is an employee of Creativ-Ceutical, which received funding from Bayer AG. MT is a consultant for Creativ-Ceutical. PL received consulting fees for his critical inputs on study design and results interpretation.

JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

All authors were involved in the design of the study and the interpretation of the results in addition to the drafting of the paper and the critical revision of the content.

Acknowledgements

The authors would like to acknowledge Chameleon Communications International, who provided editorial support with funding from Bayer AG.