The multiple stakeholder approach to real-world evidence (RWE) generation: observing multidisciplinary expert consensus on quality indicators of rare disease patient registries (RDRs)

Abstract

Introduction

Those invested in the outcome of specific decisions may seek to inform the decision-maker by generating relevant evidence. With multiple decision-makers, the evidence generator benefits from understanding whether their needs differ. This was explored using relevant stakeholders’ preferences on quality indicators (QIs) of rare disease patient registries (RDRs), a common vehicle for generating evidence, to support the adoption of new medicines.

Methods

The Jandhyala method was used to compare the observed consensus on RDR quality indicators from each stakeholder group (SHG) and between the combined pharmaceutical (pharma) and non-pharmaceutical (non-pharma) SHG population.

Results

All non-pharmaceutical stakeholder groups were equally unique – suggesting QIs peculiar to themselves – Patients (13/102; 12.75%); Payors (5/102; 4.90%); Prescribers (7/102; 6.86%) to the same degree. Non-pharmaceutical stakeholders generated 31/102 (30.39%) unique QIs. Pharma stakeholder generated 25/102 (24.51%). This totals 56/102 (54.90%) QIs without a match. Two QIs were unanimously suggested and agreed upon by all SHGs (7/7; 100%) – “Engages with patients and gains their buy-in” and “Includes a core data set as part of outcomes”. There was homogenous consensus on common QIs between: Commercial – Market Access (13/52; 25%; p = .005) SHGs; Commercial – Medical Affairs (9/47; 19.15%; p = .001) SHGs; Commercial – Payor (11/41; 21.15%; p = .012) SHGs, and Market Access – Regulatory Affairs (10/41; 19.61%; p = .008) SHGs.

Conclusions

A multiple stakeholder approach for generating real-world evidence can be justified. The potential mismatch of 54.9% between pharma and non-pharma stakeholders indicates a critical misalignment between generator and consumer of its RWE. Pharmaceutical SHG’s were highlighted for greater alignment with Patients and Payor (non-pharma) groups.

Keywords:

• Real-world evidence
• rare disease patient registries
• pharmaceutical stakeholders
• non-pharmaceutical stakeholders
• Jandhyala method
• multiple stakeholder approach

Introduction

Evidence is the product of research conducted to answer a particular research question. Such evidence is of the greatest value to its “consumers.” These are defined by the relevance of the research question to them. This includes the value that the answers will have in supporting their decision making¹.

Some may assume that evidence-based decision-making is a risk-mitigating approach². Consumers of evidence may fall into groups based on two factors: (1) Their common unanswered research or policy questions; and (2) The effect that the evidence is expected to have on an environment.

Consumers may generate relevant evidence themselves or have it generated for them by another interested party. This party may be driven by the shared benefits, to themselves and to the consumer, of answering a specific set of consumer-relevant questions.

The pharmaceutical industry is one such interested party. It manufactures medicines for the benefit of patients, in return for revenue generated by its adoption³. However, before these benefits can be broadly realized, key hurdles of regulatory approval, reimbursement and prescribing practices need to be addressed. This needs to be done with evidence tailored to their gatekeeper consumers’ needs.

Some argue that these gatekeepers, more constructively termed “stakeholders,” require a dedicated strategy to meet their specific evidence needs. Consequently, a multiple stakeholder approach to planning and delivering evidence is required. This needs to maximize the return on investment, in terms of time and resources. The way to do this is by ensuring that as many stakeholders as possible can benefit from the evidence generated⁴.

Patient registries, or observational cohort clinical studies of prospective and/or retrospective type, have the potential to generate answers to a large number of research questions in real-world patients⁵. This is especially so if these questions are appropriately understood and incorporated in the planning phase^6,7.

Their value can be amplified in rare diseases. This occurs when, in order to increase the quality of the evidence generated, small, discrete patient populations cared for by different centres can be combined and studied as a larger, single population. Orphan medicines are also among the more costly drugs and are consequently under greater scrutiny⁸.

Therefore, well-targeted evidence which encourages orphan medicine adoption by supporting the approval, reimbursement, and prescribing, can maximize the chances for rare populations to receive this life-changing treatment.

The pharmaceutical industry has evolved by recognizing the need to engage with key non-pharma (PH−) stakeholders, such as the Payor (Payo), Patient (Pati) and Prescriber (Pres). It has achieved this by aligning its internal functions to mirror the needs of these stakeholders. This has ensured that their perspective remains in focus for the company^9,10. This has been attempted by defining functions such as Regulatory Affairs (RAff), Market Access (MAcc), Commercial (Comm) and Medical Affairs (MAff). To engage specifically with patient organizations, several companies have also implemented patient advocacy functions.

Study objectives and research questions

The value and versatility of patient registries in rare diseases for generating real-world evidence (RWE) is clear¹¹. Furthermore, Berger, Sox¹² also underscores the importance of the decisions made by stakeholders involved in making new medicines available. Thus, three research questions have been developed as the subject of this work.

Research questions

• Research Question 1: by confirming the differences in the stakeholder needs between them, is a multiple stakeholder approach required?

• Research Question 2: what is the exact nature of the differences and of the level of agreement on any common items across the groups?

• Research Question 3a: is pharma aligned with PH − stakeholders or not?

• Research Question 3b: is there an alignment between the individual stakeholder groups (SHGs), especially between those Pharma (PH+) SHGs intended to mirror specific PH − SHGs?

Study aim and objectives

The overarching aim of this project is to answer the research questions above. The alignment of answers to the questions above will be measured by comparing the detailed lists of quality indicators (QIs) of rare disease patient registries (RDRs), generated using the Jandhyala method from each stakeholder group. This will enable consensus to be observed among the stakeholder groups on the quality indicators that should be included. Hence, the study has two main two objectives, to facilitate achieving the study aim.

• Objective 1: to identify the quality indicators for a successful RDR from the perspective of multiple relevant stakeholders.

• Objective 2: to assess the heterogeneity among stakeholders.

The outputs from this work are expected to both inform the planning of RWE generating strategies, and determine whether a multiple stakeholder approach is justified.

Methods

Jandhyala method

This original research was conducted using the Jandhyala method¹³. This is a novel way of observing proportional group awareness and consensus of answers arising from list-generating questioning¹⁴. The Jandhyala method uses two anonymized (Awareness and Consensus) rounds of surveys without face-to-face interaction between rounds. It is differentiated from other consensus-generating methodologies due to two key factors. Firstly, in its passive approach in observing and not forcing consensus, through neither providing feedback between rounds nor including the option of adding or dropping items after the initial stage. This enables the original set of answers to be appreciated and not undergo modification through several rounds before a set threshold of agreement is reached. The second factor is calculating the awareness index, enabling each participant’s suggestions to be considered part of the whole list, thereby inferring their proportional awareness of the subject matter. Hence, being limited to only two survey rounds without a face-to-face meeting, the Jandhyala method lends itself to the rapid completion of data collection and the minimization of study participant attrition.

Using the Jandhyala method in this study, the expert consensus was achieved by observing levels of awareness and consensus relating to QIs. These are solicited via two anonymized online surveys and the calculation of an awareness index (AI) and consensus index (CI) for each QI, respectively. During the Awareness Round (1) survey, participants were asked to respond to the list-generating question: “What is important to planning for, or judging the success of, a Rare Disease Registry?” They were asked to provide a minimum of three, and a maximum of ten, free-text answers. The participants’ responses from this Awareness Round (1) were analyzed per group. They were then refined into a mutually exclusive list of quality indicators by three researchers using a process of content analysis and open coding. The codes were then attributed to the relevant participants’ answers by one researcher. They were then confirmed by a second, with disagreements being resolved by the author.

The participants who completed the Awareness Round (1) survey were asked to participate in the Consensus Round (2) survey. They were asked to rate their level of agreement with the inclusion of the quality indicators arising from the Awareness Round (1) survey. They were to use a five-point Likert scale (strongly agree; agree; neither agree nor disagree; disagree; or strongly disagree). Quality indicators reaching a consensus level of >50% (CI > 0.5) were retained in the final list. There were no formal endpoints in this study, as the outcomes were consensus on QIs that indicated successful RDR quality.

Study participants

Study participants were invited from the author’s professional network and via advertisements placed on professional social media networking sites. They were chosen based on experience in RWE in rare diseases from the following industry and non-industry stakeholder groups: Medical Affairs experts, Market Access experts, Commercial experts, Regulatory Affairs experts, Patients, Prescribers and Payors.

Study variables

In line with the published method and its reporting convention, the awareness index, consensus index, and an index score indicating prompting, were calculated for each generated QI within their stakeholder group.

The AI, measured in the Awareness Round (1), was calculated using the frequency of a QI concerning the most frequently occurring QI. The minimal awareness threshold was defined as an AI > 50%. The CI was calculated as the percentage of participants who agreed or strongly agreed with a QI in the Consensus Round (2).

The Index score measured whether prompting occurred during the Consensus Round (2). Prompting occurred if the absolute difference between the AI and the CI was >0.05 (or 5%). Prompting was considered negligible if the absolute difference was <0.05 (or 5%). Unprompted consensus occurred when most participants suggested a QI in the Awareness Round (1), and the majority agreed, or strongly agreed, that the QI was important in the Consensus Round (2). Any QI that was not suggested by the participants during the Awareness Round (1), but which was agreed to be important in the Consensus Round (2), was prompted completely. Quality indicators achieving a CI >0.50 were retained.

The different SHGs were recruited at different time points. To send the Consensus Round (2) survey out in a timely fashion, it was necessary to code the Awareness Round (1) data for each group individually. Consequently, the generated QIs could not be harmonized across all groups until the end of the data collection period, and an additional round of analysis had been carried out to standardize the generated QIs and to avoid repetition.

The list of QIs for a successful registry from the perspective of multiple stakeholders was duly generated. QIs could be considered “unique” in three different respects, each aligned to a comparison outlined in the statistical analysis section below. These were “unique” to a single SHG only, to a pair of SHGs only and to a population, PH + or PH−, of stakeholder groups only.

Statistical analysis

Absolute heterogeneity in stakeholder responses

At least one unique QI observed is a threshold for absolute heterogeneity of a stakeholder group. The number of unique QIs generated by each SHG, the total number of QIs generated by each SHG, and the proportion of unique QIs are presented.

Two proportions tests were performed to see whether the proportion of unique QIs generated by each SHG was significantly different than those of others (combined). A p-value <.05 indicated a significant difference.

Relative heterogeneity

The heterogeneity among stakeholders for each QI was examined using individual- or expert-level consensus rating. For this analysis, the consensus ratings of the individual stakeholders were further classified as “agree” (expert-level consensus ratings of 4 or 5 in the Jandhyala method) and “disagree” (expert-level consensus rating of 1–3 in the Jandhyala method). For each QI, these classified ratings were tabulated among the SHGs that provided consensus for the QI. Chi-square tests were performed on the tabulated data, and a p-value <.05 indicated heterogeneity among stakeholders for consensus rating on the QI.

The SHGs were combined into PH+; Commercial, Market Access, Medical Affairs, and Regulatory Affairs; and PH−; Patients, Prescribers, and Payors. Chi-square testing was also performed on the combined tabulated data. A p-value <.05 indicated heterogeneity in consensus on each QI between the PH + and PH − stakeholders for the QI.

Heterogeneity in common pairwise stakeholder QIs

The heterogeneity between stakeholder pairs for the list of QIs generated and consented upon by each SHG within the pairs were examined. This analysis used QI-level consensus score (CS) from the Jandhyala method, ranging from 1 to 4: Complete consensus (CI = 1.00) – CS 1; Consensus + (0.50 < CI) – CS 2; Consensus – (0 < CI ≤ 0.50) – CS 3; and No consensus (CI = 0) – CS 4. The category numbers for each generated list of QIs were tabulated between each stakeholder pair. Chi-square tests were performed on the tabulated data, and a p-value <.05 indicated homogeneity between the stakeholder pair in their consensus on the QI.

All statistical analyses were performed using R 3.6.2.

Results

Study participants

Recruitment took place in May 2019 and January 2020. A total of 55 participants were recruited. PH − stakeholder participants were made up of 9 Prescribers, 7 Payors and 9 Patients. PH + stakeholder participants comprised 13 Medical Affairs experts, 5 Market Access experts, 7 Commercial experts, and 5 Regulatory Affairs experts. Non-industry regulatory functions were uniformly unable to participate in the study on the grounds of conflict of interest. Due to recruitment difficulties, the Regulatory Affairs experts were not always specialized in rare diseases, and one Payor had to be recruited from outside the European Union (EU).

Jandhyala method

A definitive list of QIs for each stakeholder group examined was achieved through the execution of the Jandhyala method.

Statistical analysis

Heterogeneity in SHG QIs

Absolute heterogeneity

Responses to awareness round (1) survey

From the Awareness Round (1) survey, a total of 102 QIs were generated and received an AI and a CI (Supplements 1 and 2). Of these, the following numbers of QIs were unique to either an SHG or a population of SHGs: PH + stakeholders generated 25/102 (24.51%): Commercial experts (4/102; 3.92%), Market Access experts (6/102; 5.88%), Medical Affairs experts (4/102; 3.92%), and Regulatory Affairs experts (4/102; 3.92%) and 7/102 (6.86%) suggested by >1 PH + SHG; while PH − stakeholders generated 31/102 (30.39%): Patients (13/102; 12.75%), Payors (5/102; 4.90%), and Prescribers (7/102; 6.86%) and 6/102 (5.88%) suggested by >1 PH − SHG (Table 1; Figure 1). All stakeholder groups met the threshold for absolute heterogeneity within the overall SHG result.

Figure 1. Pairwise degree of consensus and heterogeneity of quality indicators constituting a rare disease patient registry. Abbreviations. Comm, Commercial; MAcc, Market Access; MAff, Medical Affairs; RAff, Regulatory Affairs; Pati, Patients; Payo, Payors; Pres, Prescribers.

Table 1. The number of unique QIs suggested by SHGs as a proportion of the total number of QIs suggested.

SHG	SHG size	Unique QIs (a)	Non-unique QIs (b)	Total QIs (c)	a/c (%)	p-Value	a/Total overall unique QIs^†† (%)	a/Total overall QIs* (%)
PH−
Pati	9	13	32	45	28.89	.123	30.23	12.75
Payo	7	5	27	32	15.63	.968	11.63	4.9
Pres	9	7	26	33	21.21	.825	16.28	6.86
Total	25	25	52*	77*	32.47	NA	58.14	24.51
Mean	8.33	8.33	36.67	28.33	22.72	NA	19.37	8.17
Total	25	31^†	46^†	77*	40.26	NA	NA	30.39
PH+
Comm	7	4	33	37	10.81	.427	9.3	3.92
MAcc	5	6	29	35	17.14	.999	13.95	5.88
MAff	13	4	21	25	16	.999	9.3	3.92
RAff	5	4	31	35	11.43	.505	9.3	3.92
Total	30	18	53*	71*	25.35	NA	41.86	17.65
Mean	7.50	4.5	28.5	33	13.64	NA	10.47	4.41
Total	30	25^†	46^†	71*	35.21	NA	NA	24.51
Overall (PH − and PH+)
Total	55	43^††	59*	102*	42.16	NA	100	42.16
Mean per SHG	7.86	6.14	34.57	28.43	21.6	NA	14.28	6.02
Total ^†		56	46	102	54.9	NA	NA	54.9

Abbreviations. SHG, Stakeholder Group; SH, Stakeholder; QI, Quality Indicator; Comm, Commercial; MAcc, Market Access; MAff, Medical Affairs; RAff, Regulatory Affairs; Pati, Patients; Payo, Payors; Pres, Prescribers.

*Totals are adjusted for the overlapping number of quality indicators in combined populations of stakeholder groups.

^†Numbers presented for combined populations of non-parma and pharma stakeholder groups.

^††Total overall quality indicators unique to any one stakeholder group only.

Out of the total number of unique QIs suggested (n = 56), the PH + group generated 25 unique QIs: Commercial experts (4/43; 9.3%), Market Access experts (6/43; 13.95%), Medical Affairs experts (4/43; 9.3%) and RAff (4/43; 9.3%); while the PH − group generated 31 unique QIs: Patients (13/43; 30.23%), Payors (5/43; 11.63%), and Prescribers (7/43; 16.28%) (Table 1; Figure 1).

On further analysis, the numbers of unique QIs to total QIs within each SHGs were dissimilar. This led to differences in proportions of unique QIs among SHGs. Stakeholders in the PH + group generated the following unique and total QIs: Comm (4/37; 10.81%), Market Access experts (6/35; 17.14%), Medical Affairs experts (4/25; 16.00%), and Regulatory Affairs experts (4/35; 11.43%); while the PH − group generated: Patients (13/45; 28.89%), Payors (5/32; 15.63%), and Prescribers (7/33; 21.21) (Table 1; Figure 1).

When compared to the total number of QIs, the total number of unique QIs per SHG generated across both PH + and PH − stakeholder groups was 43/102 (42.16%) (Table 1; Figure 1). The potential mismatch of unique QIs between the stakeholder groups of PH + and PH − was 56/102 (54.9%).

Relative heterogeneity

The overall mean of unique QIs (PH+; PH−) suggested per SHG was 6.14 (4.5; 8.33), whereas the overall mean of non-unique QIs was 28.43. (28.5; 28.33). The mean SHG size was 7.86 participants per SHG (7.50; 8.33). For the PH − stakeholder groups: The Patients SHG suggested both the largest number of unique and total QIs (13/45; 28.89%) followed by Prescribers (7/33; 21.21%) and Payors (5/32; 15.63%), respectively.

For the PH + stakeholder groups: The Medical Affairs SHG had the greatest number of participants (13/55) but suggested the smallest number of unique and total QIs (4/25; 16.00%). The proportion of unique QIs to total QIs for the other PH + stakeholders were: Commercial experts (4/37; 10.81%), Market Access experts (6/35; 17.14%), and Regulatory Affairs experts (4/35; 11.43%) (Table 1; Figure 1). No statistically significant difference was observed in the proportion of unique QIs across all SHGs, whether PH + or PH−. This means that they were all equally unique.

Heterogeneity in QI consensus

Responses to consensus round (2) survey

A consensus score of ≤2 was reached in 23 out of 102 QIs in four or more SHGs (Table 2). A degree of commonality and heterogeneity exists between the consensus levels of these common QIs.

Table 2. Analysis of QIs above consensus threshold for inclusion (CI > 0.50) in ≥4 SHGs on heterogeneity of consensus scores.

QI No.	Quality indicator (QI)	No. SHGs Aware of QI	No. SHGs with CS = 1 or 2 on QI	Proportion of QIs Reaching CI > 0.50	Overall heterogeneity (p-value)	PH − vs. PH + heterogeneity (p-value)
9	Generates a large number of publications	4	4	100	.061	.097
13	Achieves a representative demographic of the population of interest	4	4	100	.083	.23
21	Sponsor funds provides appropriate level of funding	4	3	75	.149	.305
25	Offers site specific support	4	4	100	.218	.198
32	Utilizes online electronic data capture tool	4	4	100	.092	.232
47	Allows patients to access their own data	4	4	100	.364	.172
50	Reports on data quality (completeness)	4	4	100	.124	.289
54	Incorporates clear and transparent governance processes	4	3	75	.019*	.283
4	Collects data on quality of life	5	4	80	.015*	.5
7	Collects information on all available treatments	5	4	80	.022*	.03*
11	Allows sites to access their own data	5	5	100	.381	.446
12	Maximizes recruitment of eligible patient population of interest	5	5	100	.205	.839
19	Implements data quality procedures (completeness)	5	5	100	.059	.786
33	Engages with regulators and gains their buy-in	5	5	100	.267	.377
34	Engages with payors and gains their buy-in	5	5	100	.267	.377
46	Declares sponsorship in a clear and transparent manner	5	3	60	.006*	.014*
2	Collects patient relevant [reported] outcomes [symptoms]	6	5	83.33	.022*	.152
18	Implements data management procedures (edit checks for accuracy)	6	6	100	.118	.044*
24	Reflects clear rationale and objectives in its design	6	6	100	.029*	.602
29	Engages with prescribers and gains their buy-in	6	6	100	.031*	.496
30	Utilizes data entry forms that at easy to use	6	6	100	.401	.949
27	Engages with patients [organizations] and gains their buy-in	7	7	100	.347	.999
39	Includes a core data set as part of outcomes	7	7	100	.475	.502

Abbreviations. SHG, Stakeholder Group; QI, Quality Indicator; No., Number; Comm, Commercial; MAcc, Market Access; MAff, Medical Affairs; RAff, Regulatory Affairs; Pati, Patients; Payo, Payors; Pres, Prescribers; CS, Consensus Score; CI, Consensus Index.

*Quality indicators meeting statistically significant heterogeneity (p < .05) for consensus scores across contributing stakeholder groups.

Two QIs reached consensus in all seven SHGs for inclusion in the list QIs of RDRs: “Includes a core dataset as part of the outcomes” and “Engages with patients (organisations) and gains their buy-in” (Table 2; Figure 2).

Figure 2. Distribution of suggested QIs listed in QI numbering ascending order. (a) Stakeholder: uniqueness, consensus index; (b) heterogeneity of quality indicators by the level of stakeholder consensus; (c) heterogeneity by pharma or non-pharma stakeholder group consensus threshold for inclusion (CI > 0.05) within their discipline listed in quality indicator numbering ascending order. Abbreviations. QI, Quality Indicator; Comm, Commercial; MAcc, Market Access; MAff, Medical Affairs; RAff, Regulatory Affairs; Pati, Patients; Payo, Payors; Pres, Prescribers.

Heterogeneity in pairwise stakeholder consensus

When conducting pairwise analysis, it was found that the Commercial – Regulatory Affairs pair had the lowest number of unique QIs (n = 8), while the Patients – Regulatory Affairs pair had the highest (n = 22). Aligned PH − and PH + pairwise comparisons were: Market Access – Payors (12/54; 22.22%), Commercial – Prescribers (11/54; 20.37%), and Medical Affairs – Patients (18/60; 30%).

All pairs of SHGs were examined for homogeneity in their consensus score for each common QI. The seven SHGs yielded 21 comparisons, with four of these comparisons being statistically significant and therefore, homogeneous: Commercial – Market Access (p = .005); Commercial – Payors (p = .012); Commercial – Medical Affairs (p = .001); and Market Access – Regulatory Affairs (p = .008) (Table 3 and Figure 2).

Table 3. Pairwise analysis of SHGs on the heterogeneity of consensus scores for QIs that are common.

	Pairwise	Unique items (a)	Non-unique items (b)	p-Value	(a/a + b) %
1	Comm-MAcc	13	39	.005*	25
2	Comm-MAff	9	38	.001*	19.15
3	Comm-Pati	19	47	.569	28.79
4	Comm-Payo	11	41	.012*	21.15
5	Comm-Pres	11	43	.193	20.37
6	Comm-RAff	8	48	.249	14.29
7	MAcc-MAff	11	37	.157	22.92
8	MAcc-Pati	19	49	.104	27.94
9	MAcc-Payo	12	42	.495	22.22
10	MAcc-Pres	13	39	.063	25
11	MAcc-RAff	10	41	.008*	19.61
12	MAff-Pati	18	42	.779	30
13	MAff-Payo	9	40	.999	18.37
14	MAff-Pres	12	35	.235	25.53
15	MAff-RAff	9	41	.214	18
16	Pati-Payo	21	42	.794	33.33
17	Pati-Pres	21	42	.348	33.33
18	Pati-RAff	22	39	.155	36.07
19	Payo-Pres	14	39	.601	26.42
20	Payo-RAff	10	41	.074	19.61
21	Pres-RAff	11	41	.052	21.15

Abbreviations. SHG, Stakeholder Group; QI, Quality Indicator; No., Number; Comm, Commercial; MAcc, Market Access; MAff, Medical Affairs; RAff, Regulatory Affairs; Pati, Patients; Payo, Payors; Pres, Prescribers.

*Stakeholder group pairs meeting statistically significant homogeneity (p < .05) on consensus scores for quality indicators in common.

Discussion

This research sought to confirm the presence of discrete PH − stakeholder groups with equally unique needs, to validate the need for a multiple stakeholder approach in evidence generation. However, this was confirmed in both PH + and PH − stakeholders. Despite the unique QIs mapped out per SHG, several common QIs existed. An understanding of both common and unique QIs enables evidence generation to be planned to satisfy the needs of more than one PH − stakeholder. This improves the evidence generator’s return-on-investment in both time and resource and the availability of medicines from which a group of patients will benefit. Two QIs of rare disease registries met with unanimous agreement across all stakeholder groups and should be considered “core” to the success of this type of research: (1) the need to have a core dataset and (2) the need to both engage with patients and obtain their buy-in.

The core dataset has clear significance in the context of the “Achilles heel” of observational clinical studies – that of missing data from the data requested. The issue here is the central requirement that an observational study must not mandate data collection through unnecessary interventions on patients in the real world. Instead, it must simply observe their standard diagnostic and monitoring practice. There is a common theme running through these stakeholders’ appreciation for a key QI of a potential powerhouse of evidence generation. The theme is the inevitability of missing data and the limitation this places on the quality of the evidence generated. The development of core datasets in rare diseases is currently an area of interest for the author. Work is ongoing regarding how these can be delivered with multiple stakeholders in mind.

Regarding the work presented here, it should be recognized that PH + stakeholders have evolved, in alignment with, and primarily as a result of, the recognition of the various gatekeeper PH − SHGs above.

The results of the dedicated pairwise analysis of common QIs between all possible pairs of stakeholders were surprising. The largest number of common QIs between a PH+/ PH − pair was Patients – Regulatory Affairs (22/61; 36.07%). Expected pairs failed to demonstrate homogeneity in consensus on common QIs. This was despite the low background level of commonality, most notably Payors – Market Access, where other pairs yielded unexpected and, arguably, less useful alignment, such as Market Access – Commercial.

The above findings were in the context of a broader and more worrying lack of alignment in QIs between the PH + and PH − stakeholders; 56/102 (54.90%) QIs were without a corresponding partner on either side of this divide. This suggests a critical incongruence in QIs between evidence generator and consumer. Finally, despite the recognition that patients were unanimously agreed to be an SHG that needed to be engaged with, no PH + stakeholder group pairwise test yielded homogeneity in common QIs with the Patients stakeholder group.

The ultimate beneficiary of optimally generated evidence, in this setting, is the patient, through being treated with medicines from which they would receive a benefit. Failure to provide the necessary evidence to satisfy the gatekeepers, that is, the regulator, the payor and the prescriber, of this benefit may result in the medicine being withheld from these patients. Therefore, the importance of this research highlights not only the critical mismatch between the perceptions of the evidence generator and the external stakeholders they are trying to influence but the exact composition of the mismatch. This detailed knowledge is expected to inform decision-making and reduce or eradicate this mismatch, resulting in more significant patient benefit.

Limitations

The study recruited 55 participants from the EU into seven of the eight intended stakeholder groups. This highlighted the potential limitations of the study regarding the generalisability of the findings. The small size of the examined population must be recognized at both the stakeholder level and the geographical level and should be taken into account when interpreting the results. Furthermore, the inability to recruit regulators to participate in the study has left the pairwise comparisons incomplete. Consequently, a larger number of QIs need to be solicited from the PH + stakeholders.

Conclusion

This original research supports the existence of discrete SHGs involved in the adoption of new medicine. They fall into two clear functional categories, that is, being internal or external to the pharmaceutical company, which depends on whether they generate or consume evidence. Each SHG has unique needs from patient registries, and the RWE generated from them needs to be observed. Though a degree of commonality has been identified, no two SHGs can be considered the same. Therefore, it is recommended that pharmaceutical companies adopt a sensitive multiple stakeholder approach to ensure evidence utility to the intended consumer. In this regard, a critical mismatch has been observed between PH + and PH − stakeholders, which is reflected by the redundancies of the QIs on both sides of this generator/consumer divide. This calls into question the ability of the former to entirely meet the evidence needs of the latter and whether much of their current effort is not, in fact, wasted by not meeting consumer needs.

Further work is encouraged in this area, first to confirm these findings, and then to understand how they can be implemented to bring the generator and the consumer of RWE into closer alignment, to realize the benefits of RWE more efficiently. The unanimous agreement on the need to engage with patients, and their high number of unique QIs, should prioritize this group for specific attention.

Transparency

Declaration of funding

No external funding sources supported this work.

Declaration of financial/other relationships

The author is the Founder and CEO of Medialis Ltd., a medical affairs consultancy and contract research organization involved in the design and delivery of real-world evidence. All members of the Medialis Multidisciplinary RWE Advisory Panel were offered remuneration for their time.

Author contributions

RJ conceived the research, designed the study, and is the sole author of this manuscript.

Ethics approval and consent to participate

All participants provided written informed consent to participate in this research study, and it was determined using the NHS Research Ethics Committee (REC) review tool that the study did not require ethical approval to be conducted (certification available upon request).

Acknowledgements

RJ would like to thank the biostatisticians and medical writers at Medialis Ltd for their statistical and editorial support and the members of the RWE Multidisciplinary Advisory Panel for participating in this research.

Data availability statement

All data generated or analyzed during this study are included in this published article and its supplementary information files.