Building trust in real world evidence (RWE): moving transparency in RWE towards the randomized controlled trial standard

Abstract

The increasing use of RWE in regulatory and reimbursement decision-making indicates the significant progress that has been made in building trust in RWE through greater transparency. This review of the published literature and key online sources was conducted to provide an update on progress towards greater transparency in RWE, based on four key barriers to trust identified in a 2016 paper and applying learnings from transparency initiatives established for RCTs, such as the US FDA Amendments Act (FDAAA) 2007 Final Rule. Multiple initiatives from the US FDA, EMA and organizations such as the ISPOR-ISPE Joint Task Force have yielded new guidance documents and tools that enable greater transparency in RWE study methodology (STaRT-RWE and HARPER templates), data source selection and quality (SPIFD2 framework, REQueST tool), strategy (the Center for Open Science RWE Study Registry), and will therefore improve transparency in RWE study reporting. Programs such as the REPEAT Initiative and RWE DUPLICATE are investigating reproducibility of RWE studies and improving understanding of the circumstances when valid inference on treatment effects can be obtained from RWE studies. Further work is needed to embed and to implement the new tools and guidance that are available, and thus raise standards for RWE transparency towards the levels expected for RCTs.

Keywords:

• Observational
• real world data
• regulatory
• reimbursement

Introduction

Real world evidence (RWE) is defined by the US Food & Drug Administration (FDA) as ‘clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of real world data (RWD)’, with RWD in turn defined as ‘data relating to patient health status and/or the delivery of healthcare routinely collected from a variety of sources’.¹ RWE has been used for decades in evaluating the epidemiology and burden of diseases, mapping treatment patterns and assessing the safety of therapeutic interventions in the post-marketing setting. Further to the US 21st Century Cures Act of 2016, the FDA established a program to evaluate the potential use of RWE to support a new indication for a drug approved under section 505(c) and to support post-approval study requirements; a draft framework and multiple guidance documents have followed,² with final guidance published in August 2023 on the use of RWE for Drug and Biological Products.³ The European Medicines Agency (EMA) has similarly put forward its vision, anchored in the European medicines regulatory network (EMRN) strategy, that by 2025 use of RWE will have been enabled and that value will have been established across a spectrum of regulatory use cases.⁴ The Pharmaceuticals and Medical Devices Agency in Japan has also published its position on RWE and efforts to promote appropriate utilization of RWD/RWE in a regulatory setting.⁵ At a national reimbursement level, decision-makers such as the Canadian Agency for Drugs and Technologies in Health (CADTH) and the UK National Institute for Health and Care Excellence (NICE) have published their own frameworks for use of RWE in health technology assessment (HTA) submissions.^6,7

As a result of these legislative and regulatory initiatives, recent years have seen an explosion of interest in the application of RWE to regulatory and reimbursement decision-making,⁸ to the extent that in July 2023 the FDA approved an indication expansion based solely on RWE from routine clinical practice.⁹ The product concerned was an approved medical device and the RWE was from a retrospective, comparative effectiveness cohort study using electronic health records from two US hospitals.¹⁰ So, what’s the problem? The answer is that challenges remain regarding trust in RWE as a reliable source of evidence for decision-making. Several high-profile retractions of RWE study publications (for example, on potential treatments for COVID-19) have shown that it is still too easy to publish RWE studies that have been inadequately designed and/or conducted.¹¹ A 2016 paper identified four key barriers to trust in RWE (see Table 1) and provided recommendations for those conducting such studies to increase transparency and to build trust in RWE data.¹² These recommendations applied learnings from transparency initiatives established for RCTs, such as the FDA Amendments Act (FDAAA) of 2007 Final Rule. The present article provides an update on the progress that has been made in addressing these four barriers to trust in RWE, and summarizes current best practices for transparency in RWE study planning, design, conduct and reporting.

Table 1. Issues for trust in RWE and how transparency can address them.

Barrier	Challenge for trust	Addressing the challenge
Lack of randomization and risk of bias	You haven’t randomized, so you’re not comparing apples with apples	Transparency in methodology
Appropriateness of data sources	Researchers design studies to give the results they want to see	Transparency in data source selection and quality
Multiplicity of studies	Industry does lots of studies but publishes only those with the best results	Transparency in strategy
Contradiction of studies	RWE studies aren’t consistent with RCTs, or even with other RWE studies	Transparency in reporting

Adapted from White.¹²

RCT, randomized controlled trial; RWE, real world evidence.

Barrier 1 – lack of randomization and risk of bias

Randomization in RCTs matches treatment arms for characteristics that may bias outcomes (i.e. potential confounders). By contrast, selection bias is almost inevitable in RWE because routine clinical practice treatment decisions are not random. Although it is essentially impossible to eliminate all bias in RWE, it can be minimized by applying methods to adjust for potential confounders (e.g. propensity score matching), and considerable progress is being made in improving these methods further. Templates such as the structured template and reporting tool for RWE (STaRT-RWE)¹³ and HARmonized Protocol Template to Enhance Reproducibility (HARPER)¹⁴ now provide researchers with a framework to inform choices in RWE study planning, design, conduct and reporting. Methodological transparency is key in showing that an RWE study has been well designed and builds trust that results will be robust.

Beyond selecting and reporting clearly on appropriate methods to adjust for confounding in RWE studies, ‘hybrid’ methodologies are increasingly being applied that retain core principles of RCTs but integrate RWE elements to streamline or augment the trial process. Examples of the complementary use of RWE for regulatory and reimbursement decision-making within a more traditional RCT framework include screening RWD for determining eligibility, using RWD in synthetic control arms/external comparators, and RWE sub-studies or extension studies.¹⁵

Barrier 2 – appropriateness of data sources

Data source selection transparency is important to build trust that the most appropriate data sources for the research question have been chosen for an RWE study (rather than the most readily available, cheapest or those generating preferable results, etc.). The FDA has provided guidance on what should be considered and communicated around selection of fit-for-purpose data sources.² Specifically in oncology, the FDA Oncology Center of Excellence Real World Evidence (OCE RWE) Program launched guidance in June 2023 from its Quality, Characterization and Assessment of Real-World Data (QCARD) Initiative describing key design and data source elements for oncology RWE studies, to improve the quality of initial study proposals.¹⁶

The European Network for Health Technology Assessment (EUNetHTA) has released the Registry Evaluation and Quality Standards Tool (REQueST) to help registry owners to maximize data quality and to ensure that registry-based RWE can be used for HTA and regulatory purposes,¹⁷ and the EMA has provided guidance on data source selection through its RWE Methodology Working Party.⁴ Another initiative from the EMA to improve the quality of RWE, established as part of the European Health Data Space, is the Data Analysis and Real World Interrogation Network (DARWIN EU) – a pan-European, federated data network that will enable the exchange of RWD for use in healthcare delivery, policy-making and research across Europe.¹⁸

Beyond these regulatory initiatives, detailed guidance on data source selection is now available at a practical level. The Structured Process to Identify Fit-For-Purpose Data (SPIFD) and updated SPIFD2 frameworks are data-feasibility assessments providing step-by-step decision tools and templates to facilitate identification, selection and rationalization of fit-for-purpose RWD sources.^19,20 Availability of effective tools does not necessarily lead to their adoption, so it is notable that transparency on the choice of data source is increasingly expected for RWE studies being used as supporting evidence in HTA submissions. This is exemplified by NICE (UK) introducing the Data Suitability Assessment Tool (DataSAT) as part of their wider RWE framework.²¹

Barrier 3 – multiplicity of studies

A major driver of the FDAAA of 2007 Final Rule, and the mandate for RCT registration and reporting, was the concern that RCTs yielding neutral/negative results were less likely than those with positive findings to be published in peer-reviewed journals. RWE faces not only this barrier, but also the concern over ‘data dredging’ (i.e. conducting multiple analyses until one gives the hoped-for result).²² Such practices, and consequent distortion of the published evidence base, are almost impossible to identify when there is no mandate for registering or reporting RWE studies.

A significant step forward for transparency in strategy for RWE came with the publication of guidance from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) Special Task Force that differentiated between simple exploratory RWE studies and pre-planned hypothesis-evaluating treatment effectiveness (HETE) studies, which test a specific hypothesis in a specific population. The Task Force published clear recommendations that conduct of HETE studies should involve commitments similar to those required for RCTs, including prior study protocol registration and publication of results.²² The RWE Transparency Initiative has continued that work in developing a plan to make registration of HETE studies routine and creating relevant tools to foster broad stakeholder acceptance of such commitments.²³ The most important practical barrier to RWE protocol registration – lack of a study registry specifically tailored to the information required for RWE study protocols – has been removed following launch of the Real-World Evidence Study Registry by the Center for Open Science (https://osf.io/registries/rwe/discover).

These initiatives provide foundations for RWE studies to be subject to similar transparency requirements to those that the FDAAA 2007 Final Rule established for RCTs. Indeed, a 2022 International Coalition of Medicines Regulatory Authorities (ICMRA) statement on international collaboration on RWE identified the future need for systematic registration of prespecified protocols and study results in publicly available registries, and publication of results in open-source, peer-reviewed journals.²⁴

Barrier 4 – contradiction of studies

Reproducibility is a key determinant of trust in any research, and a challenge to using RWE in decision-making, particularly when decisions concern treatment effectiveness, is that RWE studies do not always reproduce findings of RCTs (or other RWE studies) addressing similar research questions. In many ways this is an unfair challenge, because contradictory results are inherent in all research (with RCTs on the use of aspirin in primary prevention providing an excellent example²⁵). Indeed, a growing body of literature demonstrates that examples of RWE failing to reproduce RCT results are not reflective of a general inadequacy of RWE, but rather indicate specific limitations in reporting transparency, framing of the research question and understanding of potential confounders.

Can RWE studies reproduce findings of other published RWE studies? The Reproducible Evidence: Practices to Enhance and Achieve Transparency (REPEAT) initiative (www.repeatinitiative.org) is evaluating the reproducibility of published RWE studies using different RWD sources. REPEAT initiative projects have shown that, if reporting of details of study implementation are sufficiently transparent, healthcare database studies can be reproduced with great accuracy.²⁶ However, the considerable variability in the degree to which published healthcare database studies are reproducible demonstrates the need for improved transparency in reporting. Checklists for RWE studies, such as the REporting of studies Conducted using Observational Routinely collected health data statement for pharmacoepidemiology (RECORD-PE),²⁷ the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement²⁸ and the Good Research for Comparative Effectiveness (GRACE) checklist,²⁹ in addition to the Risk Of Bias In Non-Randomized Studies – of Interventions (ROBINS-I) bias assessment tool,³⁰ have been available for some time and should be completed routinely to ensure that an RWE study publication contains all relevant information. Improved guidance on study design and protocol templates, such as STaRT-RWE and HARPER, will enhance transparency in reporting. The focus is now on encouraging adoption and routine utilization of these tools by the various stakeholders involved with RWE studies.

Can RWE studies reproduce findings of RCTs? This is being addressed by the Randomized Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology (RCT DUPLICATE) initiative, a collaboration between Brigham and Women’s Hospital, Boston, MA, USA, the FDA and Aetion, Inc. (www.rctduplicate.org), and other projects such as Observational Patient Evidence for Regulatory Approval Science and Understanding Disease (OPERAND)³¹ and the Aetion Coalition in Oncology.³² An RCT DUPLICATE project applied a predefined, transparent process to try to emulate the treatment effects observed in a selected sample of 32 RCTs using RWE (new-user cohort studies with propensity score matching in three US claims databases).³³ When selected RCTs could be closely emulated by RWE in terms of study design and analysis, which was possible in about half of the RCTs selected, the RWE studies yielded similar treatment effects to the RCTs. Results were divergent when emulation was not possible (e.g. because of unavoidable differences in endpoints or residual confounding). Ongoing RCT DUPLICATE projects are investigating the ability of prospective RWE studies to replicate RCTs across several therapy areas and will thus improve understanding of the circumstances when valid inference on treatment effects can be obtained from RWE studies, and when it is not feasible.

Summary

The increasing use of RWE in regulatory and reimbursement decision-making indicates the significant progress that has been made in building trust in RWE through greater transparency. Further work is needed on the part of researchers (in academia and industry alike) to embed and to implement the new tools and guidance that are available. Peer-reviewed journals also have an important role to play in ensuring that publications of RWE studies are held to the standards of study planning, design, conduct and reporting that they expect for RCTs. It is encouraging that a ‘road map’ article written on behalf of ISPE and the International Network for Epidemiology in Policy (INEP) now provides clear guidance for journal editors to improve the quality of peer review and editorial processes for RWE studies.¹¹ It remains true that, with access to potentially life-enhancing new treatments as our goal, all stakeholders owe it to patients to act together in building trust in RWE through greater transparency.

Transparency

Declaration of financial/other relationships

RW is an employee and shareholder of Oxford PharmaGenesis Ltd and declares no other potentially competing financial interests.

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

RW takes responsibility for the conception and drafting of the paper, final approval of the version to be published and declares accountability for all aspects of the work.

Acknowledgements

No assistance in the preparation of this article is to be declared.

Additional information

Funding

No sponsorship or funding was received.