Assessment of the comprehensiveness of paediatric national immunisation programmes in Europe: expert validation and future perspectives

ABSTRACT

Background

The breadth of protection of National Immunisation Programmes (NIPs) across Europe varies, however, this has not been assessed within published literature. Therefore, a framework was developed to assess the comprehensiveness of pediatric NIPs in Europe. This study aimed to validate and further develop criteria used to cluster countries into three tiers.

Research design and methods

Independent Europe-based experts (n = 23) in the field of pediatric vaccination were invited to participate in a double-blinded modified Delphi panel, with two online survey rounds and a virtual consensus meeting. Consensus was defined as ≥ 80% of experts rating their agreement/disagreement on a 9-point Likert scale.

Results

The number of preventable diseases covered by an NIP, simplification of the vaccination calendar, strengthened protection by increasing serotype, degree of funding and epidemiological factors were considered key concepts for consideration of the comprehensiveness of pediatric NIPs in Europe. Experts highlighted that the framework should be extended to include adolescent vaccines and populations up to 18 years of age. Consensus regarding further amendments to the framework was also reached.

Conclusions

This Delphi panel validated a framework to assess the comprehensiveness of European NIPs. The framework can be used to facilitate discussions to help countries improve and expand the breadth of protection provided by their NIP.

KEYWORDS:

• Vaccination
• Europe
• immunization
• programmes
• assessment
• comprehensiveness

1. Introduction

1.1. Background and rationale

Globally, it is estimated that 700,000 children under five years of age die annually from vaccine preventable diseases (VPD) [1]. Approximately 99% of children that die from VPD are from low- and middle-income countries [1,2]. Infectious diseases confer a substantial burden among children as they have a higher susceptibility for developing more severe symptoms due to an immature immune system and their frequent exposure to varying environments [3].

In Europe, a country’s National Immunisation Programme (NIP) reflects the vaccines included and available within their respective national schedule. Whilst NIPs are widely developed and implemented at the European level, decisions related to the inclusion of vaccines in vaccination schedules, program implementation and year-on-year budgets are frequently made nationally [4]. Consequently, a lack of harmonization exists between countries in terms of the type of vaccines that are offered, the types of healthcare professionals authorized to administer the vaccine, the timing of vaccination and the setting in which it is administered. In addition, the COVID-19 pandemic has resulted in unprecedented disruptions to routine immunization programs worldwide and it is anticipated that this will impact future efforts to progress and expand NIPs.

A recent study on the status of human papillomavirus (HPV) vaccination recommendation, funding, and coverage in European countries covered by the World Health Organization (WHO) Regional Office for Europe (2018–2019, 53 countries) was one of the first successful attempts to systematically map and describe HPV vaccination programs across Europe as well as factors related to vaccination funding and implementation [5]. However, this study focused solely on the HPV vaccination and did not extend to other pediatric vaccines [5]. The comprehensiveness of NIPs has not been defined and/or measured in published literature. Highlighting differences in immunization programs and demonstrating where vaccination calendars can be expanded to offer more protection against infectious diseases can support scientific discussions and policy engagement for inclusion of the appropriate vaccines in clinical guidelines and/or NIPs to address disease burden in the country/region. Therefore, a framework to categorize the comprehensiveness of pediatric NIPs in Europe-based on clearly defined and scientifically robust criteria was developed.

1.2. Introduction to the PEDVAC assessment

1.2.1. Framework development

The assessment framework was initially developed to map and assess the comprehensiveness of routine pediatric NIPs in Europe for which the first dose of the schedule is administered before 24 months. The countries included in the assessment framework were aligned with the EU geographic/political responsibility, which included the UK (as at the time of project initiation, the UK was still a member of the EU) and other non-EU countries such as Albania, Norway, Switzerland, and Serbia. The assessment from this point will be referred to as the pediatric vaccination (PEDVAC) assessment. The data on immunization calendars for this assessment was collected via a web-based targeted literature search, for available information on NIPs and calendars in targeted European countries. The latest NIPs were identified in the public domain and further validated by local experts, to ensure the most relevant, up to date data were presented (Supplementary material, Table S6)

The methodology for the search was the same used by Bonanni et al. (2020), which captured national recommendations for HPV vaccination [5]. However, the PEDVAC assessment is not limited to HPV, including a full overview of NIPs, and is continuously updated to reflect the most recent NIPs across Europe.

Following completion of the targeted literature review for the assessment of NIPs, a score was developed and applied to each country. The score for each country comprised three component parts:

• BASELINE – corresponding to the most common pediatric infectious disease (PID) programmes

• EXTENDED – corresponding to less common PID programmes

• ENHANCEMENT – corresponding to specific measures undertaken by a country to upgrade their NIP (extending target population, extended protection through higher number of antigens/serotypes, simplification of vaccine schedules by using combination vaccines).

Within the assessment, vaccines, referred to henceforth as antigens, are allocated into either a ‘baseline’ or ‘extended’ bucket. Antigens flagged as priorities in the WHO/European Union (EU) European Vaccine Action Plan were assumed as the benchmark for a comprehensive pediatric immunization program and were included in the baseline bucket.

1.2.2. Scoring and tiering

Each antigen is associated with a certain number of points. For instance, a country which recommends and funds 10 out of 11 vaccines in a baseline ‘bucket’ and 1 out of 4 vaccines in an extended ‘bucket’ would be assigned the following points:

• Total baseline score: 1.5 points x 10 antigens = 15 points

• Total extended score: 3 points x 1 antigen = 3 points.

Enhancement scores then add some percentage of increase on the antigen’s score when the conditions are respected. The enhancement score criteria are outlined below:

• Extension of an NIP to universal population (50%)

• Adding antigens together in combination vaccines (10%)

• Strengthening protection by increasing valencies (25%)

Total scores are then used to tier and map countries. The original ranges for each tier are outlined below:

• TIER 3: countries for which their total score is less than the maximum baseline score (default value range: less than 16.5)

• TIER 2: countries for which their total score is greater than maximum baseline score but less than maximum baseline score + 50% of maximum extended and enhancement scores (default value range: 16.5 to less than 25)

• TIER 1: countries for which their total score is greater than maximum baseline score + 50% of maximum extended and enhancement scores (default value range: 25 or more)

1.3. Study aim

Following completion of the literature review and development of the PEDVAC assessment and scoring framework, validation by experts was required. This study aimed to validate and further develop the criteria which are currently used to cluster countries into three tiers, utilizing the assistance of a number of individual experts in the field of immunization. Additionally, this study aimed to score and map the European NIPs, to outline what ‘gold standard’ practice looks like and to help support further evolution of European NIPs.

2. Methodology

2.1. Study population

Recruitment was performed by a third-party agency, to ensure both the study sponsor and study investigators remained blinded from the identities of recruited experts, to avoid compromising the study integrity, bias, and conflict of interest.

This study recruited Europe-based experts in the field of vaccination, with 10–30 years of experience, to participate in a modified Delphi panel conducted between July 2022 and February 2023. Recruited experts were independent experts that were not affiliated with the study sponsor. Experts were recruited from geographic clusters based on vaccine market access pathway characteristics outlined in Laigle et al. (2021) to ensure the study sample was representative [6]. The clusters were as follows:

• Cluster 1: Countries with national and regional decision-making and mandatory funding (Belgium, Italy, Spain, and Sweden)

• Cluster 2: Countries with national decision-making and national tendering (Cyprus, Malta, and the UK)

• Cluster 3: Countries with individual reimbursement (Czech Republic, France, Germany, Greece, and Slovakia)

• Cluster 4: Countries with national decision-making and mandatory funding (GDP higher; Austria, Denmark, Estonia, Finland, Ireland, Luxembourg, The Netherlands, and Slovenia)

• Cluster 5: Countries with national decision-making and mandatory funding (GDP lower; Bulgaria, Croatia, Hungary, Latvia, Lithuania, Poland, Portugal, and Romania)

In Laigle et al. (2021), ‘mandatory’ was defined as legally binding. Further details relating to the cluster characteristics can be also found in Laigle et al. (2021) [6].

Study screeners were used to assess experts’ eligibility for inclusion. Key criteria for participation were as follows: 10–30 years of experience working in NIPs and a strong understanding of the preventable infectious diseases covered by the NIP and the recommendations for funding in their respective countries.

The study follows the principles of the declaration of Helsinki. Informed consent was obtained from all experts, who were then provided with a study brief designed to provide background information on the study, the study rationale, an overview of the Delphi panel process, and their role and responsibilities throughout the Delphi panel rounds.

Prior to the commencement of the Delphi panel, participating experts were allocated a unique identification number by the third-party agency, which was used for all rounds of the Delphi panel. This was deemed necessary to avoid compromising the study integrity, bias, and conflict of interest. The study sponsor and investigators were therefore unaware of the identity of those participating.

2.2. Steering committee involvement

The Delphi study enlisted the assistance of three key opinion leaders (KOLs) who formed the steering committee and are listed as authors. The KOLs have extensive experience in prevention of infectious diseases, vaccination strategy and public health policymaking at an international level. The KOLs were recruited to ensure that the questions were relevant and correctly represented the most up-to-date NIPs across Europe. They were not included in the Delphi panel survey rounds and were not present during the consensus meeting.

2.3. Delphi panel

A traditional Delphi panel involves multiple rounds (at least two) of anonymized questionnaires, with controlled feedback at each stage to encourage a reduction in the variation of responses, moving toward consensus. In healthcare research, the established approach to provide robust actionable results within a prescribed timeframe is to employ a modified Delphi technique.

In this study, two rounds of survey were conducted to allow for the generation and consolidation of expert opinions, followed by a consensus meeting with the purpose to ensure that, wherever possible, statements were discussed until consensus was reached (or there was agreement within the meeting that consensus would not be reached) on each topic. Consensus was determined using a Likert scale and defined as: ≥80% rating their ‘disagreement’ as 1–3 or ‘agreement’ as 7–9 (on a 9-point scale) – this was referred to as ‘consensus agreement.’

This study sought to explore five key concepts for validation by recruited experts, across both the first- and second-round surveys:

• Key concepts for consideration of a comprehensiveness score for vaccination NIPs in Europe

• Overview and recap of the PEDVAC assessment

• Exploration of the PEDVAC scoring system for assessing the comprehensiveness of a NIP

• Enhancement score assignment

• Dividing NIPs into tiers of comprehensiveness.

It should be noted that factors related to implementation such as vaccine coverage rates were beyond the scope of the PEDVAC assessment. Vaccine requirements (mandatory or voluntary) as a concept was originally included in the PEDVAC assessment to reflect the engagement of authorities and/or policy makers in NIPs.

A detailed overview of the methodology used for the Delphi panel rounds can be found in the Supplementary Material.

3. Results

3.1. Delphi panel results

3.1.1. Expert characteristics

In the first-round, 23 experts agreed to participate and were included. Following this, 22 participated in the second-round, due to the attrition of one Austrian expert in cluster 4 leaving the process due to a lack of responsiveness (Table 1). Finally, 11 experts were able to participate in the consensus meeting. Nearly all (n = 22, 95.7%) experts had experience in diagnosis and/or treatment of PID, and just under half had experience in the decision-making process for funding and recommendations for vaccines (n = 10, 43.5%) and/or developing guidelines for pediatric infectious disease (n = 11, 47.8%).

Table 1. Geographical distribution of recruited experts, by clusters outlined in Laigle et al., (2021) [6].

	Cluster 1	Cluster 2	Cluster 3	Cluster 4*	Cluster 5
Cluster	National and regional decision-making and mandatory funding	National decision-making and national tendering	Individual reimbursement	National decision-making and mandatory funding (GDP higher)	National decision-making and mandatory funding (GDP lower)
Experts	N=3 from Spain N=2 from Italy	N=5 from the UK	N=2 from France N=2 from Greece	N=3 from the Netherlands N=1 from Austria	N=4 from Portugal N=1 from Poland
Total (N=23)	5	5	4	4	5

3.1.2. Key concepts for consideration of a comprehensiveness score for vaccination NIPs in Europe

In the Delphi panel survey rounds, experts reached agreement that key concepts include the number of preventable diseases covered by an NIP; the simplification of the calendar; strengthening protection using the vaccine with incremental benefit (i.e. increased serotype protection); the degree of funding (fully funded/partially funded/out of pocket) and funding (recommended vs recommended and funded, share of funding).

During the consensus meeting, experts’ opinions regarding the comprehensiveness of a pediatric NIP when targeting risk-groups vs universal populations varied. This topic was debated extensively; however, consensus was not reached for this concept, with experts highlighting that the choice of target population is a multi-faceted and complex challenge and may not be applicable to all countries NIPs. Experts raised that across different disease areas, determining the appropriateness of targeting either population would be dependent on the VPD’s activity and epidemiological factors, such as disease prevalence.

Additionally, experts were unable to agree upon the importance of vaccination requirements (mandatory vs voluntary vaccination) as a key concept for consideration when assessing the comprehensiveness of NIPs. Some experts felt there was little evidence that imposing mandatory vaccination would influence vaccine uptake, whereas other experts felt that, for some cultures, mandatory vaccination would be more appropriate.

The importance of epidemiological factors, when considering the comprehensiveness of an NIP, was also explored at the consensus meeting. The responses received varied, with some experts emphasizing the need for the harmonization of NIP schedules across Europe using the roll out of COVID-19 vaccines as an example. Experts highlighted that during the pandemic, complications arose as a consequence of differing vaccination policies, such as difficulties when traveling and restricted access to countries. They noted that to achieve harmonization of schedules, Europe should ideally be considered as a singular entity, meaning epidemiological factors would be redundant. However, other experts raised that variations in the epidemiology of VPD and high-risk patient groups in different European countries meant that some diseases were a greater challenge for particular countries. For example, the experts noted that incidence of celiac disease was greater in the UK than most of Europe, meaning there is a greater need to vaccinate against pneumococcal infection as individuals with celiac disease are at a higher risk of pneumococcal infection.

3.1.3. Assignment of antigens

When asked to consider the antigens included within the baseline and extended buckets, experts agreed during the Delphi panel process that adolescent (up to 18 years of age) vaccines (HPV, tetanus, diphtheria, and pertussis or meningococcal vaccines) should be included in the baseline bucket, moving the focus from pediatric NIPs alone to those administered within adolescent years.

Due to the potential morbidity/mortality associated with severe disease, nearly all experts (n = 10) felt the inclusion of meningococcal subtypes as antigens within the baseline bucket was appropriate. Further, no experts objected to rotavirus being included in the baseline bucket, due to the high prevalence of the disease in Europe, and no experts objected to influenza and varicella remaining in the extended bucket.

3.1.4. Exploration of the PEDVAC scoring system for assessing the comprehensiveness of an NIP

When questioned about the scoring values assigned to the baseline and extended buckets, experts agreed that the values assigned for absence, recommendation only and recommendations and funding were appropriate for the baseline bucket. At the consensus meeting, experts agreed the weighting of the extended bucket should not be equal to the baseline bucket. Whilst exact values were not discussed, the attainment of consensus on this approach to scoring provides justification for the use of the scores originally proposed.

3.1.5. Enhancement score assignment

In the first- and second-round survey, experts revised scores for the assigned enhancement scores, reaching agreement for the following scores:

• 20% for adding antigens together in combination vaccines,

• 25% for strengthening protection by increasing valencies and,

• 50% for extension of an NIP to universal populations.

In the consensus meeting, the moderator noted that, following the completion of the Delphi panel, the assigned enhancement scores may be altered as elements which experts agreed were important/not important are added/removed from the PEDVAC assessment. When experts were asked if, in this scenario, the enhancement scores would need to be re-validated, all agreed that re-validation was not needed.

During the consensus meeting, a concept raised by experts in the first-round, namely an ‘ideal vaccination calendar,’ was explored. Experts discussed the main barriers to developing such a calendar and the cooperation needed across Europe needed to achieve this. One barrier identified was achieving agreement between countries upon which vaccines should be included within the baseline bucket. Further, agreement amongst countries upon the timing of each vaccine would present challenges, as the age at which vaccines are currently administered differs. Finally, experts highlighted an ideal calendar that, perhaps, mandates vaccination of certain diseases across Europe, may not be suitable for countries where there is a growing attitude of vaccine hesitancy. To avoid this, the panel noted that European countries would need to work together to reduce the spread of misinformation regarding vaccination.

3.1.6. Dividing NIPs into tiers

Across the first- and second-round, the majority of experts agreed that the values assigned for the three tiers outlined in section 1.2.1 were appropriate and there would be no need to reduce or increase the number of tiers. However, it was raised by one expert that assigning countries into three tiers was a reductive exercise and unable to demonstrate the nuance of disease variation and economic status of European countries.

3.1.7. Overview of consensus statements

A full overview of consensus statements achieved across the two Delphi survey rounds and consensus meeting are presented in Table 2.

Table 2. Overview of all statements which reached consensus across the two Delphi surveys and consensus meeting.

Area of survey	Consensus statement
Key concepts for consideration of the comprehensiveness of pediatric NIPs in Europe	The number of preventable diseases covered by an NIP is an important element when considering the comprehensiveness of a pediatric NIP.*
	Simplification of the calendar (combination vaccines vs. single shots) is an important element when considering the comprehensiveness of a pediatric NIP.*
	Strengthened protection by using the vaccine with incremental benefit (vaccines with broadest serotype coverage) is an important element when considering the comprehensiveness of a pediatric NIP.
	The degree of funding (fully funded/partially funded/out of pocket) is an important element when considering the comprehensiveness of a pediatric NIP.
	Funding (recommended not funded vs. recommended and funded vaccines) is an important element when considering the comprehensiveness of a pediatric NIP.
	It is crucial that epidemiological factors should be considered when assessing the comprehensiveness of a pediatric vaccination calendar.
Overview and recap on PEDVAC assessment	Human papillomavirus (HPV) should be included within the baseline antigen bucket.
	Adolescent vaccines (for example for HPV, tetanus, diphtheria, and pertussis or meningococcal vaccines) should be included in the baseline antigen bucket.
	The PEDVAC assessment should be extended to consider populations up to 18 years, to provide a more reliable overview of the comprehensiveness of a NIP.
	Meningococcal disease should be included in the baseline bucket.
	Rotavirus should be included within the baseline bucket.
	Varicella should remain in the extended bucket.
	Influenza should remain in the extended bucket.
Exploration of the PEDVAC scoring system for assessing the comprehensiveness of a NIP	The values assigned for absence, recommendation and recommendations and funding were appropriate for the baseline bucket.*
	The weighting of the extended bucket should not be equal to the baseline bucket.^†
Enhancement score assignment	The suggested enhancement score of 20% for adding antigens together in combination vaccines, is appropriate.
	The suggested enhancement score of 25% for strengthening protection by increasing valencies, is appropriate.
	The suggested enhancement score of 50% for extension of an NIP to universal populations, is appropriate.
	A model “ideal vaccination calendar” should be developed for the EU. Countries with NIPs that follow the calendar, or adhere to it closely, would have a greater score.
	The assigned enhancement scores would not need to be re-validated if enhancement scores were altered because of elements which experts agreed were important/not important were added/removed from the PEDVAC assessment.
Dividing NIPs	The ranges presented in the tiers were appropriate/did not require changes.*
	The values assigned for tier 1 (25 to 33.525) are appropriate.*
	The values assigned for tier 2 (16.5 to <25) are appropriate.
	The values assigned for tier 3 (<16.5) are appropriate.
	The number of tiers presented are appropriate.

*Consensus reached in the first-round.

^†While there was no consensus that the values assigned for the extended bucket were “appropriate,” the panel did reach consensus that the extended scores should not be equal to the baseline bucket.

HPV: Human papillomavirus; NIP: National Immunization Program.

3.2. Implications for the PEDVAC assessment

Following the Delphi panel, amendments were made to the PEDVAC assessment to reflect the consensus achieved throughout the survey rounds.

3.2.1. Assignment of antigens

Table 3 highlights which antigens would be included in the baseline and extended buckets, using consensus achieved in this Delphi study. As discussed in section 3.1.3 consensus was reached by experts to move rotavirus, meningococcal disease, and HPV to the baseline bucket.

Table 3. Updated assignment of antigens in baseline and extended buckets.

Bucket	Disease	Score per individual antigen (recommended in NIP)	Score per individual antigen (recommended and funded in the NIP)
Baseline	Poliomyelitis, Tetanus, Diphtheria, Measles, Pertussis, Rubella, Hepatitis B, Mumps, Haemophilus Influenzae Type B, Pneumococcal disease, Tuberculosis, Rotavirus infection*, Meningococcal disease (subtypes C/B/ACWY)*, Human Papilloma Virus*	0.5	1.5
Extended	Varicella, Influenza	1.5	3

*Antigens which moved from the extended bucket to the baseline bucket following attainment of consensus through early survey rounds and discussions within the consensus meeting.

NIP: National Immunization Program.

Consequently, the minimum and maximum scores were altered. The range for the baseline bucket would be 12 points as the minimum and 21 points as the maximum. The range for the extended bucket would be 0 points as the minimum and 6 points as the maximum.

3.2.2. Acknowledging epidemiological factors

In the second-round survey, the panel consented that epidemiological factors should be considered when assessing the comprehensiveness of a vaccination calendar. The assignment of antigens has been unified across Europe to reflect the emphasis that the panel had placed on harmonization across vaccinations programs within Europe.

3.2.3. Enhancement score system

Table 4 presents the updated enhancement score scoring system, incorporating expert consensus from this Delphi study.

Table 4. Updated assignment of enhancement scores.

Criteria	%	Scope	Bucket and scoring impact
Extension of NIP to universal population	+50%	Varicella (universal vs at-risk)	+50% of extended score = 1.5pts
		Rotavirus (universal vs at-risk)	+50% of baseline score = 0.75pts
		HPV (gender-neutral vs female only)	+50% of baseline score = 0.75pts
Schedule simplification by combination usage	+20%	Hexa vs Penta	+20% of HepB baseline score = 0.30pts
Strengthening protection by increasing serotypes/serogroups	+25%	MenACWY = MenB vs MenC + MenB	+25% of baseline score = 0.375pts
		QIV vs TIV	+25% of baseline score = 0.375pts
		PCV15 vs PCV10 9vHPV vs 2vHPV	+25% of baseline score = 0.375pts

HepB: Hepatitis B; HPV: Human Papillomavirus; MenACWY: Meningococcal Conjugate (serotypes A, C, W and Y); MenB: Meningococcal Serotypes B; MenC: Meningococcal Serotypes C; PCV: Pneumococcal Conjugate Vaccine; QIV: Quadrivalent Influenza Vaccine; TIV: Trivalent Influenza Vaccine.

As discussed in section, experts reached consensus that the following enhancement scores were appropriate to be used in the PEDVAC assessment:

• 20% for adding antigens together in combination vaccines

• 25% for strengthening protection by increasing valencies

• 50% for extension of an NIP to universal populations.

3.2.4. Assigned tiering system

As a consequence of the changes detailed in sections 3.2.1 and 3.2.3, the ranges of the tiers would alter to the following:

• Tier 3: score of < 21, countries reaching a score below the maximum baseline score

• Tier 2: score of 21–26.6, countries reaching above maximum baseline score but below this same baseline score + 50% of (extended + enhancement) score

• Tier 1: score of > 26.6, countries reaching above upper limit (maximum baseline + 50% of extended and enhancement).

3.2.5. Updated PEDVAC scores

The data used to inform the scoring system and populate the scoring system is outlined in the appendix. Additionally, the total baseline, extended and enhancement scores for each country, used to inform the tiering within the PEDVAC assessment is outlined in Table 5.

Table 5. The total baseline, extended and enhancement scores for each country, used to inform the tiering within the PEDVAC framework of the assessment of the comprehensiveness of NIPs.

Country	Total baseline score	Total extended score	Total enhancement score	Total score	Tier
Albania	19.50	0.00	1.43	20.93	3.00
Austria	19.50	3.00	3.30	25.80	2.00
Belgium	19.50	0.00	2.55	22.05	2.00
Bulgaria	19.50	1.00	1.43	21.93	2.00
Croatia	18.00	0.00	1.43	19.43	3.00
Cyprus	18.00	3.00	3.30	24.30	2.00
Czech Republic	18.50	0.00	1.68	20.18	3.00
Denmark	15.00	3.00	2.25	20.25	3.00
Estonia	18.00	0.00	1.43	19.43	3.00
Finland	16.50	6.00	3.75	26.25	2.00
France	21.00	3.00	3.68	27.68	1.00
Germany	19.50	3.00	3.68	26.18	2.00
Greece	19.50	3.00	4.05	26.55	2.00
Hungary	19.50	4.00	3.00	26.50	2.00
Ireland	19.50	3.00	3.68	26.18	2.00
Italy	19.50	3.00	4.43	26.93	1.00
Latvia	19.50	6.00	4.05	29.55	1.00
Lithuania	21.00	0.00	2.55	23.55	2.00
Malta	18.00	0.00	1.80	19.80	3.00
Netherlands	19.50	0.00	1.80	21.30	2.00
Norway	18.00	0.00	2.18	20.18	3.00
Poland	20.00	2.00	1.63	23.63	2.00
Portugal	18.00	0.00	1.50	19.50	3.00
Romania	18.00	0.00	1.05	19.05	3.00
Serbia	18.00	0.00	1.50	19.50	3.00
Slovakia	16.50	0.00	1.05	17.55	3.00
Slovenia	18.00	3.00	1.80	22.80	2.00
Spain	18.00	6.00	4.43	28.43	1.00
Sweden	18.00	0.00	2.55	20.55	3.00
Switzerland	18.00	3.00	3.30	24.30	2.00
United Kingdom	19.50	3.00	3.68	26.18	2.00

3.2.6. Updated PEDVAC country map

Following the insights obtained from the Delphi panel, the country map has also been updated (Figure 1) to reflect the new antigen scores and subsequent tiering changes, outlined in sections 3.2.1, 3.2.3 and 3.2.4.

Figure 1. Updated PEDVAC map.

The PEDVAC assessment found that the countries with the most comprehensive NIPs were France, Italy, Latvia, and Spain. All tier 1 countries recommended and funded at least 12 of the 14 antigens included in the baseline bucket and Latvia and Spain recommended and funded both antigens in the extended bucket. For Italy and France, whilst they only recommended and funded one extended antigen, they recommended 13 and 14 antigens from the baseline bucket, respectively. Furthermore, all countries achieved high enhancement scores, with Spain, Italy and Latvia achieving the three largest enhancement scores across assessed European countries (4.43, 4.43 and 4.05, respectively). Italy and Latvia achieved the maximum enhancement scores for all criteria, except for strengthening protection by increasing serotypes/serogroups. They received the maximum enhancement score assignment based on their gender-neutral vaccination of HPV, universal vaccination of rotavirus and the use of hexavalent vaccination.

Tier 2 contained the most countries of any tier (n = 15; Austria, Belgium, Bulgaria, Cyprus, Finland, Germany, Greece, Hungary, Ireland, Lithuania, Netherlands, Poland, Slovenia, Switzerland, and the United Kingdom). Within tier 2, the majority of countries recommended and funded 12/13 baseline antigens with the exception of Lithuania, which recommended all 14 baseline antigens, and Finland which recommended and funded 11. Of the tier 2 countries, 12 recommended at least one extended antigen, with the exception of Belgium, Lithuania and the Netherlands which did not recommend or fund any extended antigens. In contrast, Finland was the only tier 2 country which recommended both extended antigens.

The least comprehensive, tier 3 countries were Albania, Croatia, Czech Republic, Denmark, Estonia, Malta, Norway, Portugal, Romania, Serbia, Slovakia and Sweden. Countries within tier 3 often did not fulfill many enhancement score criteria or recommend and fund many antigens from the baseline bucket, therefore scoring lower than countries in tiers 1 and 2. For example, within this tier, the range in enhancement scores was 1.05–2.55 and the majority of countries recommended and funded fewer than 13 antigens in the baseline bucket. Many countries require only small changes to their NIPs to move from tier 3 into tier 2, with Albania, Czech Republic, Norway and Sweden less than a point away from tier 2. For example, Albania has one of the lowest enhancement scores of 1.43 (assigned for extension of rotavirus to a universal population, schedule simplification by combination usage and by strengthening protection by increasing serotypes/serogroups. Therefore, Albania may target improving enhancement scores by extending recommended and funded vaccines to universal populations and adding vaccines from the extended bucket to their NIP.

3.2.7. Removal of HPV antigen

Following the introduction of HPV to the baseline antigen bucket, the population covered in the PEDVAC assessment framework would extend beyond the initial pediatric population (<24 months) to adolescents up to the age of 18. To explore how the results of the PEDVAC assessment framework would change if the population remained only pediatric, the HPV antigen was removed from the baseline bucket. To account for the loss of an antigen from the baseline bucket, the tier limits were adjusted (Tier 2: 19.50, Tier 1: 24.55).

Following the application of these changes, the impact on the overall tiering of countries was limited (Figure 2). The removal of the HPV antigen changed tiering in Italy, from tier 1 to tier 2, and the Netherlands, and Belgium, which both moved from tier 2 to tier 3. The change was primarily driven by reduction in enhancement score associated with HPV such as strengthening protection by increasing serotypes/serogroups (2 valent vaccine vs 9 valent vaccine) and extension of NIP to universal population criteria (gender neutral vaccination vs girls only).

Figure 2. PEDVAC map after the removal of HPV antigen.

4. Discussion

This modified Delphi panel obtained expert consensus to validate a framework that aims to assess the comprehensiveness of a pediatric European country’s vaccination program. To our knowledge, there is no validated assessment of comprehensiveness that visualizes all European NIPs in the public domain and is readily available to assess comprehensiveness. The validated assessment outlines numerous unique features, including archetype analysis which characterizes countries based on distinct market access pathways across Europe. Visualization and assessment of NIPs will prompt discussion on why differences between NIP scores exist and promote the dissemination of countries justifications for their decisions within public literature. Further, the assessment of comprehensiveness serves as a resource to highlight key areas that European decision makers may wish to target to improve their current NIPs, with the aim of improving overall public health.

4.1. Summary of key findings

The first area explored with experts sought to identify the key concepts for consideration regarding the comprehensiveness of pediatric NIPs in Europe. Consensus was reached on six statements, which highlighted that the key concepts were: the number of preventable diseases covered by an NIP; the simplification of the calendar; strengthening protection using the vaccine with incremental benefit; the degree of funding (fully funded/partially funded/out of pocket) and funding (recommended vs recommended and funded, share of funding). Target population and vaccine requirements were highlighted as important, however, considerations varied, and experts were unable to reach consensus. Experts emphasized that determining the appropriate target population and vaccine requirement is situational, with several country specific differences in the concepts considered important for developing a vaccine calendar. This has been previously reflected in an archetype analysis of 34 countries in adult immunization decision-making by Privor-Dumm et al (2020), which highlighted that the UK and Netherlands placed a greater importance on economics when decision-making than other countries, whereas for others, such as Germany, disease burden was the primary driver [7]. This highlighted the multi-faceted complexity of assessing the comprehensiveness of an NIP.

When considering the antigens included in the baseline and extended buckets, consensus was reached for three statements. The panel agreed that the PEDVAC assessment should be extended to consider adolescent vaccines and populations up to 18 years, to provide a more reliable overview of the comprehensiveness of a NIP. Additionally, through the double-blinded Delphi panel process, experts decided independently that HPV and meningococcus should be included in the baseline bucket. When considering the integration of HPV into the baseline bucket, it is important to consider the insights from the consensus meeting that explicitly specified the need for differential considerations across disease areas when considering a targeted population vs universal. A key implication of this finding in the context of HPV would be the considerations warranted for a gender-neutral vaccination strategy or a female-only vaccination strategy, with additional points assigned to a gender-neutral vaccination strategy. For example, in Spain, the HPV vaccine was recently extended to all adolescents, including boys over 12, meaning that additional points would be assigned given that both genders are included within the NIP [8]. In regard to meningococcus, the panel emphasized that it should be included within the baseline bucket due to the severity of the disease. Whilst meningococcal vaccination was traditionally targeted to vaccinate populations at sub-regional levels where the disease was endemic, the severity of the disease has seen many European countries include it within their NIPs, even if the vaccines are not cost-effective [9].

At the consensus meeting, there was further discussion regarding the composition of the baseline and extended buckets, with no experts objecting to rotavirus being included in the baseline bucket. Despite rotavirus and varicella both being ubiquitous viruses, no experts objected to varicella remaining in the extended bucket. A similar rationale was given by the panel for keeping influenza vaccines in the extended bucket, with one expert raising that ‘the benefit demonstrated’ by varicella and influenza vaccines ‘is less,’ with no further experts objecting. Further, experts achieved consensus in the second-round survey that the PEDVAC assessment should be extended to consider populations up to 5 years of age. However, at the consensus meeting, no expert objected to the extension of the PEDVAC assessment further to 18 years, aligning with the statement to include adolescent vaccines.

When questioned about the values assigned to the baseline and extended bucket, experts agreed that the values assigned for absence, recommendation only and recommendations and funding were appropriate for the baseline bucket. At the consensus meeting, experts agreed the weighting of the extended bucket should not be equal to the baseline bucket. Whilst no exact scores were agreed upon, the attainment of consensus on this approach to scoring provides justification for the use of the scores originally proposed.

When exploring the assignment of enhancement scores, experts agreed upon a revised score of 20% for adding antigens together in combination vaccines, 25% for strengthening protection by increasing valencies and the proposed 50% for extension of an NIP to universal populations. In the first-round survey, the concept of an ‘ideal vaccination calendar’ was also highlighted, with country’s adhering closely to it receiving an additional enhancement score. The consensus meeting sought to clarify what would be the barrier to developing such a calendar, with countries agreement upon the vaccines included in the baseline bucket, the timing of vaccine administration and compliance highlighted. One expert noted that the PEDVAC assessment may be of use to help align countries and overcome such barriers by outlining best practice. Additionally, no expert at the consensus meeting felt that the framework would need to be re-validated if assigned enhancement scores were altered as elements which experts agreed were important/not important were added or removed.

Finally, experts all agreed that the values and ranges presented for all three tiers were appropriate and felt the three tiers were enough.

4.2. Updated map and tiering of countries

Following completion of the Delphi panel, amendments to the assessment framework altered the tiering of European countries included. The countries reported as the most comprehensive were France, Italy, Latvia, and Spain, with the majority of countries falling into the second tier. The poorer scores within these latter countries were a consequence of minimal enhancement scores, (e.g. as a result of targeting ‘at-risk’ populations), and low coverage of baseline and extended vaccines.

The PEDVAC assessment provides a structured framework for countries to improve the comprehensiveness of their NIPs and highlights the ‘gold standard’ or ‘best practice’ that a country aspires to adhere to or achieve to improve the comprehensiveness of their NIP, and subsequent protection. Tier 1 countries can be considered as the ‘gold standard’ or those adhering to ‘best practice.’ These countries not only recommend and fund all baseline antigens (in accordance with the WHO EU Vaccine Action Plan), but are highly comprehensive, achieving considerable enhancement scores by vaccinating a universal population or using vaccine combos, for example.

Further, the PEDVAC tool provides a visual overview of the comprehensiveness of NIPs across the European region, highlighting inequalities that may exist or potential areas for improvements for countries within lower tiers. Reducing the inequalities that currently exist in regard to the comprehensiveness of a countries NIP has been set as an important objective of WHO’s Immunisation Agenda 2030 (IA2030). The IA2030 is a global strategy set by WHO to improve universal health care coverage by strengthening immunization systems, which builds on their previous immunization strategy (Expanded Program on Immunization [EPI]). When developing IA2030, WHO acknowledged that since the establishment of the EPI, immunization coverage has increased and the breadth of protection provided by vaccination has expanded, with the addition of several new vaccines that have contributed to providing protection. As the baseline and extended buckets of the PEDVAC assessment aligns with the breadth of protection assessed in the IA2030, decision makers may use the PEDVAC assessment to map inequalities at a regional level and use the numerical values provided to understand what changes are required to amend them [10].

The numeric value provided in the PEDVAC assessment indicates the breadth of protection that a country NIP provides, indicates how far away a country is from progression from the next tier of comprehensiveness, and details if a country could add another baseline or extended antigen to provide higher protection. Additionally, the numeric value allows for easy comparison between countries within a given geographic region and can be updated to monitor how countries progress through the tiers. The PEDVAC assessment may therefore be used by countries to improve the comprehensiveness of their NIP and subsequently strengthen NIPs to align with IA2030 targets, with the aim of reducing mortality and morbidity caused by diseases preventable through vaccination and increasing equitable access to new and existing vaccines. For the worst performing countries, immediate improvement can be attained by recommending and funding any antigen in the baseline bucket. Further, improving enhancement scores can be achieved by strengthening protection, for example, by extending the targeted population of the 9-valent HPV, rotavirus and varicella vaccine from ‘high risk’ to ‘universal.’

One of the lower scoring countries in the PEDVAC assessment criteria was Denmark, which was recently selected by the European Agency for Health and Digitalisation (HaDEA) for having one of the five best initiatives in the project ‘Overcoming barriers to vaccination’ [11]. Denmark was selected due to the high vaccination coverage, easy access and effective use of data to monitor its NIP, and subsequent success in preventing the spread of infectious diseases such as measles, mumps, rubella and polio, achieved by building trust via a comprehensive information campaign [11]. Vaccination coverage rates are indicators of the success of NIP implementation. In the PEDVAC framework, efforts pertaining to the implementation of NIP to enhance vaccination coverage was not assessed; rather the PEDVAC framework sought to assess, map and tier the comprehensiveness of European NIPs (referring to the broadness of antigen covered, the sophistication and modernism of the NIPs) [11]. Whilst Denmark was acknowledged for the high coverage of its NIP, it was still amongst the lowest scoring countries in the PEDVAC assessment criteria, based on the comprehensiveness of its NIP. This highlights the important distinction that should be considered when assessing the comprehensiveness of an NIP versus indicators of successful implementation [11].

The scenario, whereby the HPV antigen was removed from the baseline bucket, highlights the stability of the PEDVAC assessment framework when subjected to changes in scoring. In the scenario, in which HPV was excluded and scoring was adjusted to account for its removal, only Italy moved between tiers, from tier 1 to tier 2, and the Netherlands, and Belgium, which both moved from tier 2 to tier 3. Additionally, excluding one antigen highlights how changes in recommending and funding to just one antigen can improve countries’ scores and move them up through the tiers outlined in the PEDVAC assessment framework. Therefore, this scenario demonstrates that the PEDVAC framework is a valid and reliable assessment to explore several scenarios, to assess possible improvement in countries NIP comprehensiveness.

The comprehensiveness of NIPs is constantly evolving. For example, Ireland recently published a health technology assessment (HTA) on varicella, in which they recommend implementation of a universal varicella immunization program. If this recommendation was to be implemented, it would receive an additional three points in the PEDVAC assessment framework and move to tier 1 with a total score of 29.18. Further, the inclusion of varicella would make Ireland the second highest scoring country in Europe, behind Latvia. As the countries’ NIPs are constantly evolving, there is a need for the PEDVAC assessment framework to be kept continuously up to date. Equally, the PEDVAC assessment framework could be shared publicly, and used under the responsibility of a ‘public’ owner, scientific organization, or governmental authority.

Finally, the PEDVAC assessment is a qualitative and simplified approach to assessing the comprehensiveness of pediatric NIPs across Europe. However, NIP decision-making should further consider factors not included within the framework, including disease epidemiology, burden, vaccine efficacy and vaccine cost-effectiveness.

4.2.1. Strengths and limitations

The Delphi panel is a structured communication technique that enabled in-depth conversation in a controlled environment regarding assessing the comprehensiveness of NIPs. This modified Delphi process was conducted in a robust and reliable manner, with 23 experts completing the first-round of the Delphi survey and the attrition of one expert in the second-round. Whilst recruitment was based upon previous market access archetypes noted within the literature [6], the recruitment across clusters was unevenly distributed in some instances. For example, five experts were recruited from the UK within cluster 2; this cluster also included Cyprus and Malta. However, the archetype analysis characterized countries based on distinct market access pathways across Europe. Basing the recruitment of the Delphi panel sample on this archetyping analysis therefore ensured that the sample was representative of all market access vaccine archetypes across Europe.

For the consensus meeting, a smaller number of experts were available for inclusion. The reduction in sample size was due to limited availability of experts around the date the consensus meeting was held. However, the reduction in sample size aligns with Delphi methodology to ensure that discussion between experts is manageable and that all experts can contribute, to collect greater insight on discussion topics. The number of experts included in this Delphi consensus panel, however, is aligned with previously published studies and exceeds the minimum requirements in published literature regarding the attainment of consensus within this type of study [12–14].

In terms of attrition, one expert left the Delphi panel process between the first- and second-round, due to a lack of response, which limited cluster four to including only three experts in the second-round, however, based on the Delphi methodology, all opinions expressed, and voices heard are allocated the same weighting, therefore the impact of attrition should not impact the generalizability of findings. Further, by using the screening criteria, all archetypes outlined by Laigle et al. (2021) were represented and maintained throughout all Delphi rounds, reducing the impact of attrition on the generalizability of findings [6].

An assessment metric, used to evaluate the maturity of national immunization programs across the life course across all 194 WHO member states, has been previously piloted by Goldin et al. (2023) [15]. The assessment metric was developed through expert consultations, with five vaccines selected to represent delivery across the life course [15]. Like the PEDVAC assessment framework, this assessment metric was intended to monitor the trajectory of NIP comprehensiveness across the life course in the years ahead. Our efforts with the PEDVAC assessment further this aim, by specifically focusing on countries within the EU28, expanding the number of criteria used to assess maturity, increasing the number of vaccines considered in the assessment and using a more rigorous methodology to assess comprehensive in greater detail.

A steering committee was used to support the Delphi panel; however, the steering committee had no role in influencing the participants or achieving any desired outcomes. The members of the steering committee were recruited to ensure that the questions were relevant and represented the most up to date NIPs across Europe, therefore strengthening the rigor and validity of the study, by ensuring that the questions that were asked were relevant and would provide the most valuable insights.

In the current version of the PEDVAC assessment, vaccine coverage rates were considered out of scope. Whilst vaccine coverage rates may be considered when assessing NIP comprehensiveness, heterogeneity currently exists in the extent and proficiency of monitoring vaccine coverage in Europe. Inclusion of vaccine coverage rates may therefore have affected the reliability of the scoring provided by the PEDVAC assessment framework, and therefore was not included. Further, it was agreed that inclusion of implementation factors would expand the remit focus of the tool. Future research should therefore focus on the incorporation of these components to highlight how efforts in improving both the comprehensiveness of NIPs, alongside efforts to improve factors relating to implementation may help strengthen the breadth of protection against such infectious diseases across the European region.

NIPs and attitudes to vaccines are rapidly evolving, with some countries amending their policies at the end of 2022, as such this Delphi represents the view of the panel about NIPs at the time of study. Consequently, emerging vaccines such as long-acting monoclonal antibodies and RSV vaccines have not being included. However, the consented logic obtained in this Delphi panel can be applied to emerging vaccines when updating the PEDVAC assessment.

The growing interest in assessing the comprehensiveness of NIPs, and being able to compare across countries, is documented through the development of the PEDVAC assessment framework and the work conducted by Goldin et al. (2023), with the importance of this concept detailed in the WHOs IA2030 plan.

5. Conclusion

This Delphi panel has refined and validated a framework to assess the comprehensiveness a European countries vaccination program, to visualize and assess NIPs. The comprehensiveness assessment is intended to prompt discussion on why differences between NIP scores exist and promote increased dissemination of countries justifications for their decisions within public literature. Further, the PEDVAC assessment can be used as resource to highlight key areas that European countries may wish to target to improve their current program with the aim of improving overall public health.

Declaration of interest

U Sabale is an employee of MSD Lithuania and owns stock in Merck & Co., Inc., Rahway, NJ, U.S.A., V Laigle is an employee of MSD France and owns stock in Merck & Co., Inc., Rahway, NJ, U.S.A., J Murtagh is an employee of MSD Ireland and owns stock in Merck & Co., Inc., Rahway, NJ, U.S.A.. Research execution was delivered by J Cochrane, D Riley, R Perry and L Heron who were employed by AV PROVE at the time this research was commissioned by MSD. P Bonanni, J Antonio Navarro Alonso and J Eskola were reimbursed for serving as steering committee members. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or material discussed in the manuscript apart from those disclosed.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

Study design and conceptualization: Ugne Sabale, Valerie Laigle, Janice Murtagh, Paolo Bonanni, Jose Antonio Navarro Alonso, Juhani Eskola, Louise Heron, Richard Perry, Danielle Riley, James Cochrane; b. Acquisition of data: Ugne Sabale, Valerie Laigle, Janice Murtagh, Louise Heron, Richard Perry, Danielle Riley, James Cochrane; c. Analysis and interpretation of data: Ugne Sabale, Valerie Laigle, Janice Murtagh, Louise Heron, Richard Perry, Danielle Riley, James Cochrane. Drafting the manuscript: Ugne Sabale, Valerie Laigle, Janice Murtagh, Louise Heron, Richard Perry, Danielle Riley, James Cochrane; b. Revising for intellectual content: Ugne Sabale, Valerie Laigle, Janice Murtagh, Paolo Bonanni, Jose Antonio Navarro Alonso, Juhani Eskola, Louise Heron, Richard Perry, Danielle Riley, James Cochrane. Final approval of the completed manuscript: Ugne Sabale, Valerie Laigle, Janice Murtagh, Paolo Bonanni, Jose Antonio Navarro Alonso, Juhani Eskola, Louise Heron, Richard Perry, Danielle Riley, James Cochrane.

Acknowledgments

The authors would like to acknowledge the researchers at Adept Field Solutions, with special thanks to Jessica McGuinness as the project manager, who facilitated the fieldwork for the study.

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14760584.2024.2324939

Additional information

Funding

This work was funded by Merck Sharp & Dohme LLC, a subsidiary of Merck & Co, Inc., Rahway, NJ, USA. The independent (non-sponsor affiliated) experts were reimbursed for their time based on fair market value.