Real world evidence for public health decision-making on vaccination policies: perspectives from an expert roundtable

ABSTRACT

Introduction

Influenza causes significant morbidity and mortality, but influenza vaccine uptake remains below most countries’ targets. Vaccine policy recommendations vary, as do procedures for reviewing and appraising the evidence.

Areas Covered

During a series of roundtable discussions, we reviewed procedures and methodologies used by health ministries in four European countries to inform vaccine recommendations. We review the type of evidence currently recommended by each health ministry and the range of approaches toward considering randomized controlled trials (RCTs) and real-world evidence (RWE) studies when setting influenza vaccine recommendations.

Expert Opinion

Influenza vaccine recommendations should be based on data from both RCTs and RWE studies of efficacy, effectiveness, and safety. Such data should be considered alongside health-economic, cost-effectiveness, and budgetary factors. Although RCT data are more robust and less prone to bias, well-designed RWE studies permit timely evaluation of vaccine benefits, effectiveness comparisons over multiple seasons in large populations, and detection of rare adverse events, under real-world conditions. Given the variability of vaccine effectiveness due to influenza virus mutations and increasing diversification of influenza vaccines, we argue that consideration of both RWE and RCT evidence is the best approach to more nuanced and timely updates of influenza vaccine recommendations.

KEYWORDS:

• Influenza
• influenza vaccination
• vaccine effectiveness
• vaccination recommendations
• evidence base
• evaluation methodology
• public health

1. Introduction

Annually, seasonal influenza causes approximately 1 billion infections worldwide, 3–5 million severe cases and up to 646,000 influenza-associated deaths [1,2]. There is a widespread public perception that influenza is a routine illness of mild severity [3]. However, the mortality risk associated with influenza approaches that of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) (which has declined since the height of the pandemic) [4–6]. Cases requiring hospitalization occur most often in older adults aged ≥65 years and children younger than 2 years, the latter further implicated in onward transmission to older adults [7,8]. Adults younger than 65 years are also susceptible to influenza infections, and since they are responsible for the majority of care-giving for sick household members, they carry the highest rate of influenza-related absenteeism among all age groups [7–9]. Individuals of any age with certain long-term health conditions, including cardiovascular disease, metabolic disorders such as diabetes and obesity, chronic lung disease, immunocompromised status, and pregnancy, are at high risk of influenza [9,10].

Influenza viruses exhibit high mutability, leading to annual surges in influenza infections during the winter months in temperate areas of the Northern and Southern Hemispheres and throughout the year in equatorial regions, predominantly during wet seasons [11,12]. The annual resurgence of influenza results from antigenic drift, a rapid and continuous evolutionary process in which selection pressure from the human immune system promotes mutations in hemagglutinin and neuraminidase, permitting new strains to arise that can evade immune responses. In addition, reassortant zoonotic influenza A viruses occasionally infect humans, in rare instances generating influenza pandemics [11–13]. Viruses descended from novel influenza A strains that caused pandemics in 1968 (A(H3N2)) and 2009 (A(H1N1)pdm09) remain in circulation today alongside influenza B strains [14–17].

Vaccination reduces the risk of influenza infection and severe disease and complications, including cardiovascular events [18–20]. Persons at high risk of influenza complications receive priority status for influenza vaccination in many countries [10,20–24]. The recommended antigenic composition of influenza vaccines is reviewed annually (once per hemisphere) based on recommendations from the World Health Organization (WHO), which evaluates influenza surveillance and vaccine effectiveness (VE) data to predict which circulating influenza strains are most likely to predominate in the following season. In any given season, however, ongoing antigenic drift and egg adaptation (i.e. antigenic changes in vaccine viruses grown in chicken eggs) may lead to a mismatch between circulating strains and those contained within seasonal influenza vaccines [21,25–28]. This is especially true of influenza seasons dominated by A(H3N2) [29–31]. Viral mismatch negatively affects VE and may contribute to the occurrence of high severity influenza seasons, with elevated rates of influenza-related hospitalizations and deaths [8,32–44].

The belief that influenza is not a serious disease and that vaccines are not effective are two major reasons vaccination rates are often far below targets set by the WHO [3,45]. To address these gaps, the WHO and other public health authorities maintain worldwide influenza surveillance networks to track influenza viruses and also employ vaccination campaigns to encourage influenza vaccine uptake. The WHO Global Influenza Surveillance and Response System (GISRS) coordinates influenza-related activities among WHO Global Influenza Program member states’ Collaborating Centers for Reference and Research on Influenza. These actions include seasonal and pandemic influenza surveillance data reporting and laboratory analysis; monitoring of zoonotic influenza viruses and pandemic risk assessment; selection of vaccine virus strains and development of candidate vaccine viruses, which are disseminated to manufacturers; and pandemic preparedness programs. Similar WHO activities exist for SARS-CoV-2 and respiratory syncytial virus (RSV) but are substantially less well-developed [46,47].

Public health authorities at the international and national level are entrusted to formulate and release influenza vaccination recommendations. To develop these recommendations, these agencies have at their disposal real-world evidence (RWE) of VE obtained from influenza surveillance and healthcare databases (real-world data [RWD]) that are used in observational studies. Results based on RWD might be influenced by a wide variety of factors, both vaccine- and nonvaccine-related (Figure 1) [48–50]. In addition, public health authorities examine influenza vaccine efficacy data obtained from randomized controlled trials (RCTs). Both observational studies and RCTs are used by health authorities to inform influenza vaccine recommendations, although far fewer contemporaneous RCTs exist.

Figure 1. Factors influencing vaccine effectiveness [48–50].

In March 2023, in a series of virtual roundtable discussions, the authors of the present article met to share insights into how vaccination policies and recommendations are formulated in their home countries (Germany, Italy, Spain, and the United Kingdom); to review how clinical evidence, including RWE, is generated and used to guide public health decision-making on vaccination policies; to discuss the methodologies and frameworks used by European countries and national immunization technical advisory groups (NITAGs) for assessing clinical evidence and developing vaccine recommendations; and to develop expert consensus on how to enable an appropriate, effective, and efficient use of RWE for vaccination policies. This review represents a summary of those discussions, wherein we describe the basis for a set of recommendations for the use of different types of evidence in the development of vaccine recommendations.

2. Public health decision making for influenza immunization

2.1. Development of immunization policies and vaccine recommendations in Europe

Despite coordination and encouragement from GISRS, influenza vaccine recommendations are inconsistent from country to country, even in Europe. In most developed nations, a NITAG develops vaccine recommendations [51]. In Germany, for example, the Standing Committee on Vaccination (STIKO) defines vaccination recommendations using strict, evidence-based criteria that rate RCTs as high-quality evidence, and RWE studies as low-quality; the latter may be upgraded to moderate quality if the RWE study is considered to be extremely well-designed [52,53]. Influenza vaccination is not generally recommended in Germany but rather only for persons older than 60 years, chronically ill persons, and persons with frequent contact with other people, such as health care workers and teachers, among others. For older adults (≥60 years), the STIKO recommends the high-dose quadrivalent influenza vaccine (QIV-HD), while no specific vaccine is preferred for younger persons. For children between 2 and 17 years of age, live attenuated influenza vaccine (LAIV) may also be used. All other vaccines such as the cell culture-based vaccines, adjuvanted vaccines, or recombinant vaccines can be used as well depending on the approval of the preparation but are not preferred over the egg-based standard vaccine [54]. Most German states generally follow the STIKO recommendations, although some may adopt the full set for the general population whereas others focus only on at-risk groups. However, Saxony has its own State Committee on Vaccination (SIKO), which makes recommendations based on expert consensus and is more willing to consider RWE alongside RCT evidence.

In Spain, the Committee for Immunization Program and Registry issues vaccine recommendations after briefings by ad hoc Working Groups. All Working Groups follow a protocol developed by the Spanish Health Technology Assessment Network, which includes evaluation of disease burden, VE and vaccine safety, the impact of any proposed modifications to vaccination policy, ethical aspects, RWE, and economic impact [55,56]. In 2023, for example, influenza vaccination was added to the pediatric vaccination schedule through this process [57]. Final recommendations are issued and endorsed by the Inter-territorial Health Council, a policy body that includes Spain’s Minister of Health and the top health political authorities of each of the 17 autonomous regions and two autonomous cities in Spain [58]. According to the agreement of the Interterritorial Council of the National Health System, influenza vaccination is a priority measure to prevent complications associated with influenza and reduce the health and social impact of the COVID-19 pandemic [59]. The 2021–2022 influenza vaccination campaign reached 69.5% coverage of people aged ≥65 years, 76.0% of those ≥75 years, 60.0% of healthcare personnel, and 55.3% of pregnant women [60]. The Presentation of Program and Registration of Vaccinations recently extended influenza vaccination to children between 6 and 59 months, achieving coverage of over 40% in the autonomous regions of Spain [61,62].

In the UK, vaccination recommendations are issued in England by the Joint Committee on Vaccination and Immunization (JCVI), translated into policy by the Department of Health and Social Care (DHSC), and operationalized by the National Health Service (NHS) England; the same recommendations are usually adopted in-full in Wales, Scotland, and Northern Ireland, sometimes with minor operational differences. There is a statutory obligation on the Secretary of State for Health and Social Care to implement JCVI recommendations in England if they are affordable and practical. The JCVI evaluation is always based on clinical data (with routine reliance on RWE) and cost-effectiveness, with an annual review of VE data, followed by updated recommendations. There is clear recognition by JCVI that influenza vaccine is no longer a simple commodity where all products are roughly equal; instead the rapid diversification of formulations and platforms is well-recognized (e.g. egg vs. cell based or recombinant, trivalent vs. quadrivalent, standard vs. high-dose). JCVI now expresses specific vaccine choices for older people, including adjuvanted quadrivalent influenza vaccine (aQIV), QIV-HD, and recombinant quadrivalent influenza vaccine (QIVr). Currently in the 2023–2024 winter season, only aQIV and QIVr are available for NHS reimbursement. JCVI further recommends cell-based quadrivalent influenza vaccine (QIVc) or QIVr for adults at-risk and predominantly LAIV for children [63,64]. In England, general practitioners procure vaccines directly from suppliers (often discounted) and obtain NHS reimbursement (at full price) plus fixed administration fees, provided national guidelines are followed. Inactivated egg-grown vaccines are increasingly seen by JCVI as undesirable because of egg-adaptation unless there are shortages of preferred alternatives [63]. In general, JCVI considers and accepts data from well-performed, independently conducted observational studies, the commonest of which are the routine test negative case–control studies performed by the UK Health Security Agency, and specifically evaluating VE in patient subgroups and, where possible, by vaccine type [65].

The Italian Ministry of Health occasionally asks the Italian NITAG to provide advice on influenza vaccination recommendations, but the NITAG’s role is not statutory, and the process of developing recommendations is not guideline-driven or indeed standardized. There is also no official position on consideration of RCTs and RWE as the basis for vaccine policy. Consequently, recommendations in Italy are not regularly updated based on new information. The new National Immunization Plan [66] foresees that vaccine recommendations may be updated on a yearly basis. Concerns over budget implications also hamper the addition of new population groups to the recommendations. As far as influenza vaccine recommendations are concerned, every year the Ministry of Health releases a relevant document including the target population and the vaccines authorized for use in specific groups. The most recent recommendations have suggested, for the first time in Italy, the preferential use of enhanced influenza vaccines in older adults (QIV-HD in those aged ≥60 or aQIV in those aged ≥65 years), without further specification of product choice or hierarchy [67]. In Italy, the choice of vaccine product is left to each region, which is independently responsible for vaccine procurement.

2.2. Generation of VE data to support vaccine recommendations in Europe

As with vaccine recommendations, European nations take different approaches to the ongoing assessment of VE in different populations and determination of whether vaccine recommendations should be changed.

In Germany, the Robert Koch Institute (RKI) monitors public health on behalf of the government and has issued annual influenza surveillance reports through the 2018–2019 season but has not done so since [68]. Currently, the RKI participates in STIKO as an external body but has no other clearly defined role in vaccine evaluation. The Paul Ehrlich Institute Federal Institute of Vaccines and Biomedicines is responsible for approval of vaccines and monitoring of adverse events, but it does not review effectiveness data [69]. Germany participates in the European influenza vaccine evaluation project Influenza – Monitoring Vaccine Effectiveness in Europe (I-MOVE), which publishes preliminary and end-of-season effectiveness data using a test-negative design based on the data obtained by networks of sentinel physicians in primary care [70–73]. However, although the STIKO takes VE into account in its recommendations, it does not currently review influenza disease or VE every year. Thus, new vaccines are generally recommended later in Germany than in other countries, and on average, more than 6 years may pass before an approved vaccine is recommended.

In Spain, the Ministry of Health evaluates influenza vaccination programs through the Presentation of Program and Registration of Vaccinations, which proposes recommendations based on scientific evidence and the epidemiology of vaccine-preventable diseases. Both RCT and nonrandomized studies of interventions (NRSI) – primarily test-negative design studies – are taken into account [55,56]. In addition, the Carlos III Health Institute has participated since its inception in the I-MOVE system [70–73]. The Carlos III Health Institute produces infographics that summarize the annual impact of influenza and influenza vaccines [74,75]. Several research groups in Spain based in Navarra, Catalonia, and the Valencian Community also regularly publish results on the effectiveness of the influenza vaccine in preventing various outcomes [76–80].

In the UK, the UK Health Security Agency evaluates VE on an ongoing basis using RWE [81], based on test-negative design studies. The vaccine landscape is itself largely divided into three population segments: LAIV for schoolchildren, cell and recombinant vaccines for at-risk adults <65 years of age, and adjuvanted vaccine for adults ≥65 years (although cell and recombinant vaccines are also allowed and used in smaller quantities). Because these products are used in separate patient groups, assessing VE by product is possible. A JCVI influenza subcommittee considers all available VE data and makes recommendations to the main JCVI committee, which typically accepts them and reports the recommendations to DHSC. The DHSC then works with NHSE to produce an annual ‘flu letter’ for general practitioners and pharmacists, indicating the vaccines to be procured in each patient category and which vaccines will and will not be reimbursed [82]. Following this advice, healthcare providers order vaccines from manufacturers.

Like the other countries mentioned, Italy participates in I-MOVE [70–73]. No Italian governmental agency continuously assesses VE or evaluates the outcomes of current vaccine recommendations, which at the national level have changed very slowly in the past. Some health authorities rely on data analyzed by individual research institutes using Health Technology Assessment (HTA) or similar methodologies. Recent reports include evaluations of LAIV, QIVc, aQIV, and HD-QIV [83–86]. Since implementation of vaccination campaign and procurement activities are fully delegated to regional health authorities, use of evidence varies widely in Italy. By law, procurement agencies must take into consideration quality along with price in the choice of vaccine products, but there is no official guideline on how quality should be assessed. As a result, vaccine price and budget availability are the main drivers for the decision.

2.3. Harmonization of influenza vaccine recommendations between public health authorities at the national and international level

Influenza vaccine recommendations from different authorities are usually consistent within nations but often not between nations. In Europe, the European Medicines Agency (EMA) serves as a regulator for the European Union (EU), and I-MOVE evaluates VE across 15 European Union/European Economic Area Member States [70–73], which together provide a consistent set of factors on which to base vaccine recommendations. In addition, the Development of Robust and Innovative Vaccine Effectiveness (DRIVE) project is a public–private partnership that works to develop networks for large-scale VE studies in Europe [87]. Recently, DRIVE published a formal evaluation of 42 nations’ structural capabilities in the delivery of high-quality vaccination services on the population and individual level [88]. The European Center for Disease Prevention and Control (ECDC) is working to establish the Vaccine Effectiveness, Burden and Impact Studies (VEBIS) infrastructure, which will be used to monitor COVID-19 and influenza VE and perform disease burden and other studies in different settings, countries, and regions [89]. However, no platforms are available from the ECDC or European Commission to bring different NITAGs or other national agencies together to reach agreement or harmonization on vaccine policies. Because vaccine recommendations have budgetary implications, European authorities may be reluctant to intervene in these decisions. Each country has its own unique healthcare system and its own way of financing vaccination programs. Recommendations are tied to reimbursement in some countries but not in others.

In addition, differences in decision-making processes between and within countries are linked to the role of each stakeholder in different countries (e.g. whether NITAG recommendations are binding or not), to the actors involved in the decision, and to its implementation within countries. Where regional organizations are in place (such as in Italy), decisions taken at the regional level may follow different processes, and thus make it possible that not all vaccines are equally available across regions. For example, in Spain, there is a high level of agreement and harmonization among the central and regional public health authorities, which are the payers and policymakers; however, due to budgetary constraints or health priorities at the regional level, dissimilarities can occur in the willingness to pay and make available some new vaccines. A clear example is the differential availability of the recombinant herpes zoster vaccine or the staggered inclusion of influenza in the childhood vaccination schedule. In the UK, the Medicines and Healthcare Products Regulatory Agency licenses vaccines, which is a fully separate process from the JCVI recommendations. In Germany, the recommendations are made by the STIKO (or in Saxony by the SIKO). The federal states usually incorporate the recommendations into the vaccination guidelines, and the health insurers usually reimburse the vaccines in the vaccination guidelines. But exceptions occur where recommended vaccines are not included in the vaccination guidelines or are not reimbursed, or alternatively, vaccination guidelines include vaccines or health insurance companies reimburse vaccines that were not recommended by the STIKO.

2.4. Vaccine recommendation development in the US

In the United States, the Advisory Committee on Immunization Practices (ACIP) follows a strict data evaluation method based on Grading of Recommendations Assessment, Development, and Evaluation (GRADE) and considers both RCTs and RWE but also relies on open and transparent interaction and exchange among public health stakeholders. The ACIP recently performed an evidence-based evaluation of enhanced influenza vaccines for older adults, which resulted in updated vaccine recommendations [90–93]. In their evaluation, the ACIP considered the body of evidence available for each vaccine assessed, and any relevant clinical evidence regarding a certain vaccine was included and subjected to a quality assessment. RWE and RCTs were included, as well as both laboratory-confirmed and clinically diagnosed influenza endpoints (e.g. International Statistical Classification of Diseases [ICD] codes). This scientifically sound and pragmatic approach enabled inclusion of clinical evidence that may have provided critical insights into the efficacy/effectiveness and safety of a vaccine, but which would not be evaluated if a certain type of evidence was excluded a priori (e.g. RWE or non–laboratory-confirmed endpoints). Also included was the complete body of evidence available regarding the evaluation of vaccine performance on critically important clinical endpoints (e.g. influenza-related hospitalizations) that are typically evaluated within RWE studies and that in clinical practice are not evaluated with the same stringent criteria applied in an RCT (e.g. laboratory-confirmation of influenza).

3. Appraisal of evidence for decision-making in influenza vaccination

As described in Section 2, European public health agencies rely on both RCTs and RWE studies for vaccine policy decisions, but with differing levels of emphasis. STIKO in Germany emphasizes RCTs over RWE data, whereas the UK JCVI relies more on RWE studies [52,53,63–65]. The Spanish and Italian Ministries of Health consider a broad set of data. In Spain, RWE VE data are utilized when RCTs are unavailable, but when available, RCTs are highly influential in the decision-making process [55,56,66,67]. In Italy, in the absence of agreed-upon protocols for evidence appraisal, both RCTs and RWE are considered for decision-making. When available, filtered information-like systematic reviews or HTA documents are highly influential.

3.1. Strengths and limitations of different types of vaccine studies

3.1.1. RCTs

The highly controlled conditions of RCT designs provide good control of bias and confounding, leading to accurate estimates of efficacy and safety based on well-defined outcomes in the specific study population (Table 1). RCTs are well-suited for measuring effects between two therapies (i.e. absolute vaccine efficacy, or whether vaccine A is better than vaccine B or better than no vaccine/placebo) within the specific study population as well as for capturing acute adverse events, particularly in the first 6 weeks post-vaccination [94,98].

Table 1. Strengths and limitations of RCTs and RWE.

	RCTs	RWE
Strengths	Unbiased if excellent performance and reporting Good control of confounding Excellent for estimations of efficacy and safety in study population based on prespecified outcomes Performance under highly controlled conditions^a	Real-world conditions Permits analysis of groups not represented or under-represented in RCTs Permits post-marketing evaluation of effectiveness and safety
Limitations	Difficult to include large numbers of patients from at-risk subgroups Due to ethical considerations, conducting placebo-controlled trials for influenza vaccines, which are recommended for specific high-risk groups, is not appropriate Limited follow-up Limited population exposure due to stringent criteria for inclusion (older age, pregnancy, immunodeficiencies, underlying conditions often excluded) Performance under highly controlled conditions^a Performance under not usual health care provision conditions: more intensive and continuous contact during the study period	Prone to information bias Prone to selection bias Prone to confounding
Barriers to data generation	Recruitment, low influenza attack rates, other aspects of good clinical practice (GCP) [94–96]	Potential lack or incompleteness of necessary data Risk of bias Access to reliable and representative data/all aspects of GCP (i.e. ROBINS-I) [97]
Monitoring of vaccine efficacy/effectiveness and safety over multiple, consecutive influenza seasons and in broad cohorts/populations	Difficult and expensive, even economically unviable [94]	Feasible and relatively cheap [94]
Assessment of incidence of rare safety outcomes	Potentially not possible even in large RCTs (N > 10,000)	Large study populations (N > 100,000) enable assessment of the incidence of rare events
Endpoints relevant to public health decision making	Efficacy in preventing laboratory-confirmed infection, transmission, mild and severe illness (hospital admission, ICU admission, death, length of stay, antibiotic use, other relevant outcomes)	Effectiveness in preventing medical encounters at different points of care (outpatient, inpatient, ICU) or impact on several infection complications (e.g. pneumonia, cardiovascular outcomes, worsening frailty, death) Incidence of rare safety outcomes

Abbreviations: GCP, good clinical practice; ICU, intensive care unit.

^aPotentially a strength and a weakness.

RCTs are typically considered the gold standard for studies of pharmaceutical products, but influenza vaccine RCTs have several unique limitations (Table 1). Seasonal variation in the match or mismatch between vaccine and circulating influenza strains, and other factors beyond manufacturers’ control, may limit reproducibility of results. The same vaccine might show a profoundly protective effect in a well-matched season and little or no benefit if the next season is highly mismatched. Consequently, regulators do not require that influenza vaccine RCTs demonstrate efficacy but rather only immunogenicity and safety. Efficacy RCTs are also costly and may be challenging to run, so vaccine manufacturers have limited willingness to perform them, especially since they are not required for annual re-approval. A limitation common to all RCTs includes relatively small study populations that often exclude at-risk groups – for influenza vaccines, these would include older adults, pregnant persons, and those with immunodeficiency or comorbidities putting them at high risk of influenza complications. Eventually, RCTs may not be large enough to detect relevant serious adverse effects. In addition, the highly controlled conditions of RCTs do not reflect the real-world day-to-day conditions of health care [94,98].

3.1.2. RWE studies

By definition, RWE studies evaluate vaccines as they are used in the real world (Table 1). These observational studies typically analyze data from registries or healthcare databases and thus are more feasible, practical, and substantially less costly than RCTs. They also usually involve much larger study populations (hundreds of thousands or millions vs. thousands in an RCT), including subgroups and cohorts usually excluded from RCTs, such as at-risk populations, and they can more easily cover multiple-influenza seasons than RCTs. These larger, more inclusive study populations confer increased statistical power and the ability to detect rare adverse events or evaluate prevention of influenza-related outcomes such as medical visits or hospitalizations. RWEs are well-suited for monitoring the effectiveness and safety in the post-marketing phase of a vaccine as well as for measuring relative VE between different vaccines, including head-to-head comparisons of vaccines used in the same population during the same season [94,98]. Nonetheless, RWEs (like all observational studies) are prone to bias and confounding. Accuracy of the findings may be reduced depending on the study design and the data sources and analysis [95,97]. In addition, there are mathematical methods to reduce if not eliminate bias in studies. The study by Wang et al. compared 32 RCTs with RWE generated by the authors. Using methods to reduce bias, they found a high correlation between both study types [99].

3.2. Clinical endpoints used to assess vaccine performance

RCTs typically report the incidence of influenza in terms of laboratory-confirmed cases, whereas RWEs may report laboratory-confirmed influenza or cases of influenza-like illness (Table 1). In addition, both studies may evaluate prevention of related endpoints such as influenza-related outpatient visits, hospitalizations, ICU admissions, or death, or secondary complications such as pneumonia and cardiovascular events [100]. Nevertheless, the evaluation of these rarer ‘public health’ endpoints is harder using RCTs due to generally much smaller study sizes.

3.3. Barriers to data generation

The primary barriers to data generation in RCTs (Table 1) include the impracticability of recruiting sufficient numbers of patients to consistently evaluate the efficacy and safety in different influenza seasons [94–96]. For example, the 2005–2006 influenza season was very mild, with an attack rate that was far below the 4% rate that was assumed in the design of a placebo-controlled RCT conducted in the Czech Republic. Consequently, the study population was too small to include enough influenza cases to show any differences between the influenza vaccine and placebo [96].

Barriers to data generation in RWE studies include lack of high-quality RWD, a lack of guidelines on the proper generation and use of RWE, and the potential for bias and confounding inherent in observational studies (Table 1) [94,101,102]. The risk of bias can be appropriately detected by using tools such as the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) tool [97]. Another hurdle is that RCTs are clearly defined, whereas ‘RWE study’ is not a well-defined term, and the category is often considered to include more or less everything that is not an RCT. NRSI represents another category of quantitative studies that estimate the effectiveness of an intervention (either harm or benefit) but do not use randomization to allocate units (individuals or clusters of individuals) to intervention groups. Such studies often occur in the course of usual treatment decisions or according to people’s choices and are sometimes referred to as observational studies. NRSI are used to evaluate a wide range of interventions, from drugs and hospital procedures to diverse community health interventions and health systems at a national level. NRSI are often included in systematic reviews when RCTs are not available or do not fully address the research question. They are particularly useful for evaluating long-term or rare outcomes, different populations, or settings. NRSI can take various forms, including cohort studies, case–control studies, controlled before-and-after studies, and interrupted-time-series studies [103]. The terms RWE and NRSI are often used interchangeably. However, by definition, NRSI includes ‘non-randomization and ‘intervention,’ whereas RWE is associated with RWD, which in turn refers to large electronic data sets containing information on patients and healthcare encounters. NRSI encompasses any study generating RWE of health interventions.

3.4. Value of RWE in the context of influenza vaccination policy

RWE and RCTs are complementary; many research questions that cannot be addressed by RCTs can be answered with RWE (Table 1). Comparing data over many consecutive influenza seasons is impractical with RCTs, whereas RWE studies permit multi-season evaluations with relative ease, because the data already exist and need only to be analyzed. Whether retrospective or prospective, RWE studies can be designed to examine endpoints not covered by RCTs, including the impact of vaccination on rarer influenza-related outcomes, the relative effectiveness and safety of vaccines used under real-world conditions and in specific subgroups typically excluded from RCTs, and the incidence of rare adverse events, with frequencies too low to be detected in RCTs [101,102]. In addition, efforts such as the ECDC VEBIS studies can provide evidence derived from large populations in a shorter timeframe than RCTs, facilitating timely public health decision-making [89].

The COVID-19 pandemic demonstrated the value of RWE for public health decision-making. For much of the pandemic, RWE studies provided the majority of evidence on VE [104]. Public health agencies operated under emergency conditions, and initially available RCTs were very short-term in limited (albeit relatively large) study populations. Eventually, RCTs demonstrated the efficacy of the vaccines and lower rates of hospitalizations after vaccination, but research questions on outcomes such as mortality, pregnancy outcomes, booster vaccination schemes, and effectiveness against virus variants were evaluated using RWE studies, and relevant vaccine recommendations relied entirely on this type of data [105,106]. In the UK, test-negative case–control studies were the mainstay and provided sufficient evidence to judge whether the vaccines were working, when VE began to wane, and which populations should receive a booster [107–109]. Observational data also helped identify rare side effects (e.g. thromboembolism, myocarditis) with COVID-19 vaccines that could not be detected in RCTs [105,110].

4. Tools for evidence assessment

Useful tools for evaluating the quality of vaccine studies include the Revised Tool for Assessing Risk of Bias (RoB 2), ROBINS-I, and Good Research for Comparative Effectiveness (GRACE) [97,111,112]. The GRADE system, Evidence to Decision (EtD) or Evidence to Recommendations (EtR) frameworks, and HTA may be used to develop recommendations [103,113–117].

Across Europe, various NITAGs consider recommendations made by the WHO, the Strategic Advisory Group of Experts on Immunization (SAGE), the ACIP and other groups in developing their own recommendations. Spanish and Italian authorities rely on HTA evaluations and cost-effectiveness analyses along with systematic reviews, meta-analyses, and network meta-analyses that incorporate both RCTs and RWE as the standard for decision-making. The German STIKO relies on the GRADE methodology for its recommendations, and similarly the ECDC provide a technical review of vaccine evidence using GRADE.

5. Conclusion

Despite being a major cause of morbidity and mortality throughout the world, influenza is often dismissed as ‘just the flu’ and influenza vaccination rates remain well below the targets set by the WHO and other health authorities. This calls the attention to the need to strengthen vaccination policies and appropriately consider the available evidence to develop recommendations, which should rely on the most up-to-date evidence for VE and adverse events. RCTs remain vital for vaccine licensure, however, given the variable nature of influenza viruses and seasons, consideration of RCT and RWE findings together is the best approach to producing timely updates of influenza vaccine recommendations. In fact, RWE studies permit the relatively fast evaluation of vaccine benefits and safety, particularly for detection of rare adverse events. Some notes of caution are warranted. There should be a thorough and comprehensive process to identify data sources that are fit for the intended purpose prior to analysis, and this process should adhere to specific standards that guarantee the suitability and high quality of the data for addressing the research question or purpose. This step does not contradict the core principle of utilizing RWE for its opportunity and timeliness but ensures that the data meet necessary fit-for-purpose criteria, which include relevance, accuracy, completeness, timeliness, representativeness, consistency, accessibility, and ethical compliance. Meanwhile, RCT data remain vital for vaccine licensure, but given the variable nature of influenza viruses and VE, consideration of RCT and RWE findings together is the best approach to producing timely updates of influenza vaccine recommendations.

6. Expert opinion

A more balanced and coordinated approach to influenza policy making, which considers evidence from observational RWE studies alongside RCTs, would enable countries and regions to promptly update influenza vaccine recommendations, which in turn would strengthen public health agencies’ efforts to increase vaccination coverage rates and protect more people from influenza infections and serious complications.

The main barrier to wider consideration of RWE in public health policy making is the belief that these studies produce low-quality evidence. In fact, when the two approaches have been used to study the same problem, the results are hardly ever substantially different [99,118]. From a methodological perspective, RCTs produce more robust evidence, and RCT results should be preferred if they address the appropriate target, are policy-relevant, and are timely. However, these three conditions rarely apply at the same time, and relying on RCTs alone – and therefore waiting for the perfect answer – may delay decisions and prevent timely policymaking. Instead, public health stakeholders should acknowledge that both RCTs and RWE studies have strengths and weaknesses and thus they should be considered as complementary in evaluations of the efficacy/effectiveness and safety of influenza vaccines. For example, efforts to control the COVID-19 pandemic relied on both RWE and RCT data, which enabled timely decision-making that preserved lives and lessened the impact on health services. Moreover, extensive use of RWE, especially before long-term follow-up data from RCTs were available, demonstrated that RWE studies can provide sufficient evidence if the data are carefully and thoughtfully generated with good methodologies and used with full cognizance of potential biases and weaknesses [117].

The need for coordination and harmonization between NITAGs and other decision-making bodies is important. Development of guidelines that define high-quality RWE study designs at the international level that are endorsed by national authorities may help harmonize recommendations across regions.

A standardized method to incorporate expert opinion into the evidence review would also help with the timeliness and harmonization of vaccine recommendations, especially in years when new evidence on vaccine products is scarce or contradictory. The ACIP approach described in Section 2.4 may serve as a model for Europe and other regions of the world.

Future efforts are needed to define and evaluate clinical endpoints relevant to influenza, including those directly related to influenza (e.g. infection, illness, hospitalization, ICU admission, death), secondary complications (e.g. myocardial infarction and stroke), and other related endpoints such as frailty and quality of life. This information should be widely available in healthcare system databases. RWE would be further strengthened by improved general data protection regulations and other privacy regulations, which could facilitate the use of registries in RWE analyses.

In 5–10 years, wide acceptance of RWE may foster development of more appropriate and better targeted influenza interventions. By this time, it is reasonable to anticipate that vaccines for RSV will have been deployed in the same patient populations, that further COVID-19 vaccine boosters might be required, and that new influenza vaccine modalities may be available. Inclusion of RWE in public health decision-making should parallel implementation of a systematic, standardized, transparent, and internationally harmonized effort to update vaccine recommendations. To achieve this goal, stakeholders (i.e. NITAGS, national health authorities, HTAs, private sector institutions, etc.) should develop consensus and agreement on all steps and requirements for RWE similar to Good Clinical Practice principles and proceedings for RCTs.

Article highlights

• European health ministries develop national influenza vaccine recommendations based on evidence from randomized controlled trials (RCTs), real-world evidence (RWE) studies, and analyses of disease burden and economic impact

• The degree to which any given type of evidence is used varies by country, with some countries placing greater emphasis on RCTs while others rely more on RWE studies

• Although RCT data are more robust and less prone to bias, well-designed RWE studies permit timely evaluation of vaccine benefits, effectiveness comparisons over multiple seasons in large populations, and detection of rare adverse events, under real-world conditions

• Consideration of both RWE and RCTs evidence would permit health ministries to develop more nuanced and timely updates of influenza vaccine recommendations

Abbreviations

ACIP, US Advisory Committee on Immunization Practices; aQIV, adjuvanted quadrivalent influenza vaccine; DHSC, Department of Health and Social Care; DRIVE, Development of Robust and Innovative Vaccine Effectiveness; ECDC, European Center for Disease Prevention and Control; EMA, European Medicines Agency; EtD, Evidence to Decision; EtR, Evidence to Recommendations; EU, European Union; GISRS, Global Influenza Surveillance and Response System; GRACE, Good Research for Comparative Effectiveness; GRADE, Grading of Recommendations Assessment, Development, and Evaluation; HTA, Health Technology Assessment; I-MOVE, Influenza – Monitoring Vaccine Effectiveness in Europe; JCVI, Joint Committee on Vaccination and Immunization; LAIV, live attenuated influenza vaccine; NHS, National Health Service; NITAG, national immunization technical advisory group; NRSI, nonrandomized studies of interventions; QIV-HD, high-dose quadrivalent influenza vaccine; QIVc, cell-based quadrivalent influenza vaccine; QIVr, recombinant quadrivalent influenza vaccine; RCT, randomized controlled trial; RKI, Robert Koch Institute; RoB 2, Revised Tool for Assessing Risk of Bias; ROBINS-I, Risk of Bias in Non-randomized Studies of Interventions; RSV, respiratory syncytial virus; RWE, real-world evidence; SAGE, Strategic Advisory Group of Experts on Immunization; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; SIKO, State Committee on Vaccination; STIKO, Standing Committee on Vaccination; VE, vaccine effectiveness VEBIS, Vaccine Effectiveness, Burden and Impact Studies; WHO, World Health Organization.

Declaration of interests

C de Waure reports receiving honoraria from CSL Seqirus and MSD for research and advisory activities. BC Gärtner reports receiving honoraria for lectures and advisory boards from CSL Seqirus, Sanofi, BionTech, Moderna, and Viatris. PLL reports honoraria from CLS Seqirus, GSK, Moderna, MSD, Novavax, Pfizer, and Sanofi as advisor or speaker in conference symposia and educational activities. J Puig-Barberà reports honoraria from CSL Seqirus, Novavax, Sanofi and HIPRA as advisor and speaker in educational activities. JS Nguyen-Van-Tam was seconded to the Department of Health and Social Care, England (DHSC) from October 2017 – March 2022. After ending this role, he reports consulting fees from CSL Seqirus Ltd, and lecture fees from AstraZeneca and Sanofi Pasteur, all of whom manufacture influenza vaccines. From May 2023 onwards he reports consulting fees from Moderna Therapeutics Inc., which is developing future influenza vaccines. The views expressed in this article are not necessarily those of DHSC or its agencies, nor of the companies mentioned above. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or material discussed in the manuscript apart from those disclosed.

Reviewer disclosures

A peer reviewer on this manuscript has disclosed that they receive grant funding from Sanofi. Peer reviewers on this manuscript have no other relevant financial or other relationships to disclose.

Author contributions

All authors participated in the discussion and the development of this manuscript and reviewed and approved the final manuscript.

Acknowledgments

The authors would like to thank medical consultant C. Gordon Beck for medical writing assistance in the preparation of this paper.

Data availability statement

Data are derived from public domain resources and are referenced in the manuscript.

Additional information

Funding

CSL Seqirus funded all costs associated with the development and publishing of the present manuscript.