Development of a definition for real-world evidence using the Jandhyala method for observing consensus opinion among medical affairs pharmaceutical physicians

Figures & data

Table 1. Regulatory policy, frameworks and definitions of RWD/RWE.

Regulatory body	Policy on the use of RWD/RWE	Framework for use of RWD/RWE	Definition of RWD/RWE
Food and Drug Administration (USA)	Framework for FDA’s Real World Evidence Program (2018)^10	Three pillars: Fitness of RWD for use Study design Whether the study meets FDA regulatory requirements^10	“Real-World Data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. Real-World Evidence (RWE) is the clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD.”^10
European Medicines Agency (EU)	EMA Regulatory Science to 2025: Strategic Reflection (2020)^11	OPTIMAL framework: Criteria for acceptability of RWE; Operational, Technical, and Methodological challenges and solutions^2	“Real-world data is routinely collected data relating to a patient’s health status or the delivery of health care from a variety of sources other than traditional clinical trials. Real-world evidence (RWE) is then defined as the information derived from analysis of RWD.”^2
Medicines and Healthcare Products Regulatory Agency (UK)	MHRA draft guidance on randomised controlled trials generating real-world evidence to support regulatory decisions (2021)^12 MHRA guidance on the use of real-world data in clinical studies to support regulatory decisions (2021)^13	MHRA guideline on randomised controlled trials using real-world data to support regulatory decisions^14	“RWD are defined as data relating to patient health status or delivery of health care collected outside of a clinical study. Sources of RWD include electronic healthcare records (EHR) defined as structured, digital collections of patient level medical data, primary and secondary care records, disease registries, and administrative data on births and deaths. Other sources of RWD include patient reported outcomes (PRO) data and data which are collected outside of a clinical study setting such as through wearable devices, specialised/secure websites, or tablets… When such data is analysed to make inferences about the effects of different treatments, the information produced is similarly called real-world evidence (RWE).”^13
Therapeutic Goods Administration (Australia)	Real world evidence and patient outcomes in the regulatory context (2021)^15	Guidance on data quality and existing frameworks that may be adopted and updated in 2021/2022^15	Considering the adoption of the following, based on US definition: “Clinical evidence regarding the use and potential benefits or risks of a medical product derived from analysis of Real World Data, usually collected outside of the clinical trial (for therapeutics) or investigational testing (for medical devices) setting.”^15
National Institute for Health and Care Excellence (UK)	NICE real-world evidence framework; Corporate document [ECD9]	Specific guidance in NICE real-world evidence framework; Corporate document [ECD9]	“We define real-world evidence as evidence generated from the analysis of real-world data. It can cover a large array of evidence types including disease epidemiology, health service research or causal estimation (see uses of real-world data in NICE guidance). It can be generated from a large range of study designs and analytical methods (including quantitative and qualitative methods) depending on the research question or use case. A real-world evidence study may use routinely collected data, bespoke data collection, or a combination of the two. We consider single-arm trials that use real-world data sources to create an external control to be real-world evidence studies.”

Figure 1. Data saturation. Bars represent the percentage of unique items generated by participants in order of entry into the study; line represents the cumulative percentage of unique items, which was achieved by three participants in total.

Table 2. Definition items generated by participants.

Number	Item	A	C
1	Scientific knowledge [generated via a scientific method]	3	1
2	Answering a research question [the scientific knowledge constitutes an “answer” to a specific question]	3	1
3	Supported by the results of [results support the answer and constitute “evidence” for this answer. The “answer” is separate to the “results”]	3	1
4	An observational cohort clinical study [this is a method to generate results supporting the answer to a research question outside the clinical trial definition]	3	2
5	Regulators [a stakeholder to whom scientific answers to research questions are relevant]	2	1
6	Payors [a stakeholder to whom scientific answers to research questions are relevant]	3	1
7	Prescribers [a stakeholder to whom scientific answers to research questions are relevant]	2	1
8	Patients [a stakeholder to whom scientific answers to research questions are relevant]	2	2
9	Local disease management practices for investigations and treatments [this means avoiding exposing patients to potentially experimental procedures, crucially this includes cohorts receiving a specific treatment, or not, as long as the decision to prescribe is part of local management]	2	1
10	Primary data collection [data collected by researchers for the first time directly from the subjects they are interested in, specifically designed for the study]	2	1
11	Secondary data source [data already collected by other parties with no direct contact with the subjects by the researchers]	1	1
12	Effectiveness [a constituent of scientific knowledge – part of expanded definition]	4	1
13	Safety [a constituent of scientific knowledge – part of expanded definition]	3	1
14	Quality of life [a constituent of scientific knowledge – part of expanded definition]	4	1
15	Healthcare utilisation [a constituent of scientific knowledge – part of expanded definition]	3	1
16	Unselected [a population consisting of all appropriate participants in line with local management practices]	2	1
17	Representative of disease population [a further qualifier for “unselected” above]	2	1
18	Outside a randomised controlled trial [a further qualifier for “unselected” and “representative” emphasising contrast and exclusion from an RCT population]	2	1

Awareness scores (A) are presented as follows: (1) 100% complete awareness, (2) > 50% – < 100% awareness, (3) > [f/f_min × 100] and ≤ 50%, (4) f/f_min × 100, (5) 0% awareness; Consensus scores (C) are presented as follows: (1) 100%, (2) > 50% – < 100%, (3) > 0 – < 50%, (4) 0%. Definitions: f is defined as the frequency of item reported by participants in responses to the Round 1 question, f_min is the number of times the least frequently mention item was mentioned in response to the Round 1 question.

Table 3. Conceptual maturity of the generated definition of real-world evidence.

Criteria for conceptual maturity	Items	Definition
Consensual definition across stakeholders	Regulators (5), Payors (6), Prescribers (7), Patients (8)	…relevant to regulators, payors, prescribers, and patients
Clearly described characteristics	Scientific knowledge (), Answering a research question (), Supported by the results of (), Observational cohort clinical study ()	Real-World Evidence is scientific knowledge that answers a research question… supported by the results of an observational cohort clinical study
Specified pre-conditions and outcomes
Preconditions	Primary data collection (), secondary data source (), Local disease management practices (), Unselected (), Representative of disease population ()	…which has used local disease management practices for investigations and treatments and employed primary or secondary data collection from an unselected and representative disease population
Outcomes	Effectiveness (), Safety (), Quality of life (), Healthcare utilization ()	[expanded definition: including stakeholder-specific endpoints, such as effectiveness, safety, quality of life and healthcare utilization]
Delineated boundaries	Outside a randomized controlled trial ()	… outside a randomized controlled trial.

Consultation document: MHRA draft guidance on randomised controlled trials generating real-world evidence to support regulatory decisions. GOV.UK. 2022; [cited 2022 Nov 21]. Available from: https://www.gov.uk/government/consultations/mhra-draft-guidance-on-randomised-controlled-trials-generating-real-world-evidence-to-support-regulatory-decisions/consultation-document-mhra-draft-guidance-on-randomised-controlled-trials-generating-real-world-evidence-to-support-regulatory-decisions.

 Google Scholar

MHRA guidance on the use of real-world data in clinical studies to support regulatory decisions. GOV.UK. 2022; [cited 2022 Nov 21]. Available from: https://www.gov.uk/government/publications/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions.

 Google Scholar

Lokugamage AU, Simpson FK, Chew-Graham CA. Patient commentary: How power imbalances in the narratives, research, and publications around long Covid can harm patients. BMJ. 2021;373:n1579.

 PubMedGoogle Scholar

MHRA guideline on randomised controlled trials using real-world data to support regulatory decisions. GOV.UK. 2022; [cited 2022 Nov 21]. Available from: https://www.gov.uk/government/publications/mhra-guidance-on-the-use-of-real-world-data-in-clinical-studies-to-support-regulatory-decisions/mhra-guideline-on-randomised-controlled-trials-using-real-world-data-to-support-regulatory-decisions.

 Google Scholar

Therapeutic Goods Administration. Real world evidence and patient reported outcomes in the regulatory context. Published online 2021; [cited 2022 Nov 21]. Available from: https://www.tga.gov.au/sites/default/files/real-world-evidence-and-patient-reported-outcomes-in-the-regulatory-context.pdf.

 Google Scholar

Jandhyala R. The multiple stakeholder approach to real-world evidence (RWE) generation: observing multidisciplinary expert consensus on quality indicators of rare disease patient registries (RDRs). Curr Med Res Opin. 2021;37(7):1249–1257.

 PubMed Web of Science ®Google Scholar

Armstrong RA, Mouton R. Definitions of anaesthetic technique and the implications for clinical research. Anaesthesia. 2018;73(8):935–940.

 PubMed Web of Science ®Google Scholar

Data availability statement

The authors confirm that the data supporting the findings of this study are available within the article.