Construction of an indicator framework for vaccine inclusion in public health programs: A Delphi-entropy method study

ABSTRACT

Governments must decide which vaccine priority to include in their public health programs. Using the modified Delphi and entropy method, we developed an indicator framework for vaccine inclusion at the national, provincial, municipal, and district/county levels, each containing three dimensions. In total, 4 primary indicators, 17 secondary indicators, and 45 tertiary indicators were selected, covering vaccine-preventable diseases, candidate vaccines, and social drivers of the supply and demand sides. From a subjective perspective, there was no significant weighting difference in the primary and secondary indicators at all administrative levels. “Vaccine-preventable diseases” as a primary indicator had the greatest weight in the peer group, of which “Health burden” had the highest weight among the secondary indicators. From the objective perspective, the social drivers on the supply and demand sides of the primary indicators accounted for 65% and higher. Among the secondary indicators, “the characteristics of the candidate vaccine” and “vaccine-related policies on the supply side” held 8% of weights or more at both national and provincial levels. “Demographic characteristics” held the highest weight at the municipal (13.50) and district/county (15.45) level. This study indicates that China needs different considerations when using WHO-recommended vaccines at the national, provincial, municipal, and district/county levels. In addition, this study highlights that behavioral and social drivers are important indicators that need to be considered for decision-making. This framework provides a tool for policymakers to determine the inclusion priority of candidate vaccines.

KEYWORDS:

• Expanded program on immunization
• public health program
• routine immunization program
• vaccines
• behavioral and social drivers

Introduction

Immunization is one of the most cost-effective methods for improving health, achieving sustainable development,¹ and preventing 3.5 to 5 million deaths annually because of infectious diseases such as diphtheria, tetanus, pertussis, influenza, and measles.² The National Immunization Program (NIP) is a public health intervention program maintaining the well-being of human beings, increasing vaccination coverage, maximizing vaccination efficacy, and ensuring public health and support coordinated economic and social development.³ Despite the significant achievements of China’s NIP, challenges remain in incorporating new vaccines and improving implementation efficiency. Particularly, issues related to disease burden, financing, supply, and informatization need to be addressed. As the protective effects of new vaccines increase, the eligible population for vaccination and the demand in high-burden populations grow. It is necessary to overcome the challenges and incorporate more World Health Organization (WHO) -recommended vaccines into routine plans to achieve the ambitious goals of Immunization Agenda 2030⁴ and Healthy China 2030. Thus, it is essential to establish a scientific and feasible evaluation method, such as an evaluation indicator system, to assist public health decision-making and determine which vaccines are eligible or urgently needed for prioritization in public health programs to gain the utmost benefit.^5,6

Many factors must be taken into account for vaccine inclusion in public health programs. There is no uniform and well-defined tool or methodology to evaluate the indicators and priority of vaccine inclusion. Based on the WHO Principles and Considerations for Adding Vaccines to National Immunization Programs, three key factors must be considered. The first factor is the vaccine-preventable disease (VPDs), and the high disease burden and the low effectiveness of other public health and social interventions aimed at preventing or controlling VPDs should be considered. In China, some high-burden VPDs, such as influenza, which leads to an average of 88,000 (95% CI: 84000–92,000) excess respiratory disease-related deaths annually⁷, and pneumococcal infection, which is responsible for 2.55% of community-acquired pneumonia⁸ and 8,000 (5,500–8,900) deaths⁹, are not covered in the routine immunization program. The second factor is the vaccine supply capacity, safety, efficacy, and economic attributes, such as cost and cost-effectiveness. In 2018, several NIP vaccines, including Bacille Calmette-Guérin (BCG), inactivated poliovirus vaccine (IPV), and measles-mumps-rubella (MMR) vaccine, had supply shortages for 5–10 months. Moreover, Chinese district/county-level administrative authorities bear a large proportion of the national immunization program costs. The financial capacity of counties in the Eastern and Western regions varies greatly, making it difficult to achieve balance and sustainability. The third factor is the ability to provide new vaccines in the primary immunization service system in the long term. The levels of socioeconomic development and considerations differ from regions, as do the vaccines included in their public health programs. The Joint Committee on Vaccination and Immunization (JCVI) in the United Kingdom primarily emphasizes the health economics of vaccines, reminding that the recommended vaccines must be cost-effective.^10,11 In Australia, the federal, state, and grassroots governments focus on vaccine safety, quality, and efficacy.¹² The Advisory Committee on Immunization Practices (ACIP) in the United States,^13,14 the Ständigen Impfkommission in Germany,¹⁵ and the Dutch government³ referred not only to WHO recommendations but also to population values and preferences and economic analysis.¹⁴

In the past four decades, China’s immunization program has made great achievements, evidenced by a significant decrease in the incidence of 11 VPDs.^16,17 However, the inclusion of new vaccines in public health programs is limited by the lack of evidence, innovation in decision-making, and pricing in some regions. Since 1978, China adopted the EPI, which has included 16 vaccines so far and mainly covers children younger than 6 years.¹⁸ Fewer vaccines have been suggested for adolescents and adults in NIP than in developed countries,¹⁸ and most of them are monovalent.¹⁹ Many vaccines recommended to be included in public health programs for free vaccination by WHO, such as pneumococcal conjugate vaccine, rotavirus vaccine, Hib vaccine, and influenza vaccine, are not yet included in China NIP, and mostly require complete self-payment. This increases the financial burden on individuals and decreases the vaccination rate. Non-NIP vaccine coverage and use varies between regions.²⁰ Data from provincial, municipal, and district/county pilot studies are used to introduce vaccines. Therefore, before vaccine inclusion in the NIP, provinces, cities, and counties can introduce vaccines to local public health programs based on the local situation, operating freedom, convenience, and accessibility. National-level decision-makers can comprehensively evaluate the progress and effectiveness of vaccination at different administrative levels and finally add vaccines to national public health programs. Previous studies on vaccine inclusion focused on epidemiological effect analysis and economic evaluation,^21–23 neglecting public compliance and evaluation indicators based on a multi-regional perspective.

An indicator framework for vaccine inclusion in national or regional public health programs can facilitate vaccine decision-making. The framework should identify the indicators to be evaluated and then assign weights to each indicator. While economic indicators, such as cost-effectiveness, generally receive the utmost attention by countries, they should not be the only or foremost criteria for prioritizing inclusion in different regions.²⁴ The involvement of a broad range of stakeholders is considered to ensure transparency and accountability in decision-making and support an equitable priority-setting process. Qualitative evaluation, quantitative evaluation, and their combination can be applied to assign indicator weights.²⁵ Qualitative results rely on the evaluator’s subjectivity, and inferential analysis reflects the footprint of knowledge and practical experience, which lacks an objective and reasonable scientific basis.²⁶ Quantitative results are objective analyses based on sample data.²⁷ Their combination standardizes the results and makes them numerical, combining the advantages of subjectivity and objectivity.²⁸ Subjective assignments bias the weights of subsystems and indicators, and there is not a complete systematic evaluation of VPDS burden in China.

Hence, a fully quantitative or qualitative approach would not be completely suitable, and a combination of both approaches may provide a stronger support and structure. Our study aimed to combine the subjective Delphi and objective entropy weighting methods to determine different levels of evaluation indicators and weights and build an evaluation indicator framework suitable for vaccine inclusion in public health programs. Our results can provide a decision-making tool to be widely used for vaccination.

Methods

Summary of study design

The modified Delphi-entropy weight method was applied to construct the indicator frameworks for vaccine inclusion. After the literature review and group discussions, questionnaires containing drafted background information on initial indicators for vaccine inclusion in public health programs were sent to experts before the formal professional consultation instead of collecting information on indicators from the first round of expert consultation in traditional Delphi. Based on expert opinions in the comprehensive evaluation of the consultation feedback on the indicators, the weights of indicators were determined (Figure 1).

Figure 1. Flowchart of constructing an indicator framework. The dotted box in the top half marks the process of selection and determining evaluation indicators for vaccine inclusion in public health programs. The dashed box in the bottom half marks the process of assigning weights to all metrics, subjectively relying on the expert score product and objectively based on the entropy weight.

Selection of indicators

In order to compile a comprehensive set of initial indicators for expert review, the indicator pool was constructed through multiple channels, primarily derived from literature reviews and group discussions, with some inputs from previous vaccine development reports and social surveys. Chinese and English literature databases, including PubMed, Web of Science, Scopus, Google Scholar, China Biomedical Literature Database (http://www.sinomed.ac.cn/), China Knowledge Network (CNKI, http://www.sinomed.ac.cn/), and Wanfang (https://www.wanfangdata.com.cn/), were searched for published articles from 2013 to 2023. Searches were conducted using the following keywords and terms alone or in combination with “OR” and/or “AND” operators: “vaccine” “national immunized program” “expanded program on immunization” “vaccines inclusion in public health program” “indicator system, indicator framework,” “influenza vaccine, pneumococcal vaccine, HPV vaccine, vaccine-preventable diseases, vaccine hesitation.” We selected research articles and extracted the factors associated with the inclusion of vaccines in NIP. If the study area involved different administrative levels, the indicator was assigned to the indicator pool of the corresponding administrative unit. Together with the aspects considered in other countries, we included a complete set of factors that could potentially influence vaccine inclusion, both for the VPDs and vaccine application. Members of the discussion group screened the available indicators according to the systematicity, typicality, comparability, operability, and quantifiability of the indicator framework construction, eliminated indicators with similar meanings, and provided a preliminary indicator pool for expert consultation. The indicator pool contained four levels: nation, province, city, and district/county, with indicators in three dimensions for each level.

Selection of experts

Based on previous studies and suggestions of several experts invited by the Delphi method,^29,30 we invited experts from medical institutions across China from September to November 2022 using a convenience sampling method. The experts were from center for disease control, universities, and associations engaged in vaccine policy and management, or from enterprises engaged in vaccine production and sales, ensuring the representativeness and authority. The inclusion criteria were as follows: 1) ≥5 years of experience in the field of vaccines; 2) familiarity with VPDs and vaccine applications; 3) holding a master’s degree or higher; and 4) holding a senior title.

Experts’ consultation

There were two rounds of expert consultation. In the first round, the indicator that needed to be retained or adjusted in level or dimension in the preliminary pool were selected. The research group collected feedback from all experts for group discussion and modification and revised the indicators they suggested to be added or deleted. When opposing views existed regarding the retention or removal of a specific indicator, other experts were consulted, and the indicator was retained in the pool only if more than 90% of the experts agreed. All retained and removed indicators were then presented to experts, and we proceeded to the next step in the absence of objections. In case of objections, the aforementioned steps were repeated until a consensus was reached. Thus, the indicators for each dimension and level were screened based on the expert consultation and recommendations.

The second round of consultation was divided into two sections. In the first section, experts were asked to assess the importance of each indicator using a 5-point Likert scale.⁵ The score of “very important” was 5, “important” was 4, “generally important” was 3, “not very important” was 2, and “not important at all” was 1. A higher score meant the experts considered the indicator more important for the inclusion of vaccines in public health programs. The second section was experts’ scoring of indicator weights. The total weight score of the primary indicators was 100. Similarly, the total weight score of the secondary indicators for each primary indicator and that of the tertiary indicators for each secondary indicator were 100. If the secondary indicator contained only one tertiary indicator, the default weight score for that indicator was 100. The framework was presented as a table and sent to the experts by e-mail.

The positive coefficient was measured by the rate of expert feedback, namely the number of experts who gave feedback divided by the number of questionnaires sent out. The authority coefficient was expert’s self-evaluation, depending on how well they judged and were familiar with the research problem.^25,29 We classified the degree of familiarity with the research content into five levels: highly familiar, moderately familiar, somewhat familiar, slightly familiar, and unfamiliar, assigning corresponding values of 1.0, 0.8, 0.6, 0.4, and 0.2, respectively. The experts’ judgment basis was divided into five categories: theoretical analysis, work experience, reference literature, peer understanding, and intuitive feeling, with corresponding values of 1.0, 0.8, 0.6, 0.4, and 0.2, respectively. The sum of the judgment basis and familiarity scores, divided by 2, served as the authority coefficient, ranging from 0 to 1. A higher value indicated a higher degree of expert authority, more valuable opinions, and increased reliability of conclusions.

The Kendall’s coordination coefficient (W) represented the consistency of opinions from different experts, determining whether there was a large disagreement among experts in rating the same indicator. Generally, W accepted a value between 0 and 1, and a larger value showed higher consistency and data credibility. W < 0.2 indicated a poor level of consistency; 0.2–0.4 indicated a fair level of consistency; 0.4–0.6 indicated a moderate level of consistency; and 0.6–0.8 indicated a strong level of consistency. Considering that experts did not provide identical evaluations for all indicators, the nonparametric consistency test of K relevant samples was conducted, with P < .05 indicating that the rating levels were consistent and the results can be considered credible. The variation coefficient represented the degree of concentration of experts’ opinions. If the variation coefficient of an indicator was less than 0.2, the degree of acceptability was the highest.

The weight of indicators

The weight coefficient is a quantitative value that weighs the relative importance of each indicator. We evaluated the weight coefficients from objective and subjective perspectives. In subjective terms, the product method based on the scores obtained from the second round of expert consultation was used. The weight coefficients of primary indicators referred to the percentage of the scores. The combined weight coefficients of secondary indicators referred to the weight coefficient of primary indicators multiplied by the percentage of secondary indicator scores. The combined weighting coefficients of tertiary indicators were the combined weighting coefficient of its secondary indicators multiplied by the percentage of its tertiary indicator scores. In objective terms, the dispersion of each indicator score was applied to calculate information entropy and weights. Entropy is an objective measure of uncertainty. More evaluated information leads to smaller uncertainty, smaller corresponding entropy, and larger indicator weight coefficients. Compared to the traditional Delphi, this combined method not only saves time for expert consultation and accelerates the research process,³¹ but also partly prevents the subjective bias of assigning indicator weights. The entropy weighting method had the following main steps:

In this study, there was a matrix for each indicator at the four levels (national, provincial, municipal, and county/district level). For example, if m experts scored n indicators of the primary dimension for the priority of vaccine inclusion at the national level, matrix R_mn was formed as Equation (1)(1) $\mathit{R}=\left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1\mathit{n}}\\ r_{21}& r_{22}& \cdots & r_{2\mathit{n}}\\ ⋮& ⋮& \cdots & ⋮\\ r_{\mathit{m}1}& r_{\mathit{m}2}& \cdots & r_{\mathit{mn}}\end{array}\right]$(1) . The evaluation matrix was standardized. Indicators positively contributing to vaccine inclusion were treated according to Equation (2)(2) $r_{\mathit{ij}}=\frac{{\mathit{r}}_{\mathit{ij}}^{\prime }-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)}{\mathit{Max}\left(r_{\mathit{ij}}^{\prime }\right)-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)},\left(\mathit{i}=1,2,\dots ,\mathit{m}.\mathit{j}=1,2,\dots ,\mathit{n}\right)$(2) , and conversely according to Equation (3)(3) $r_{\mathit{ij}}=\frac{\mathit{Max}\left({\mathit{r}}_{\mathit{ij}}^{\prime }\right)-r_{\mathit{ij}}^{\prime }}{\mathit{Max}\left(r_{\mathit{ij}}^{\prime }\right)-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)}$(3) . Entropy measures the randomness and uncertainty of information, while utility values gauge the impact and value of information on decision-making, thereby enhancing the quality and effectiveness of decisions. Both of the s values are involved in the calculation of objective weights. Equations (4)(4) $H_j=-\mathit{k}{\sum }_{\mathit{i}=1}^m{f}_{\mathit{ij}}\cdot \ln f_{\mathit{ij}}$(4) and (5(5) $f_{\mathit{ij}}=\mathit{\hspace{0.17em}}\frac{{\mathit{r}}_{\mathit{ij}}}{{\sum }_{\mathit{i}=1}^m=r_{\mathit{ij}}}\mathit{\hspace{0.17em}},\mathit{k}=\frac{1}{ln\mathit{m}}$(5) ) were used to calculate the entropy value of each indicator, where $f_{\mathit{ij}}$ was the evaluated scores of expert i on the indicator j. Equation (6)(6) ${\mathrm{d}}_{\mathrm{j}}=1-{\mathrm{H}}_j$(6) was the calculation of the utility values and the Equation (7)(7) $w_j=\frac{1-H_j}{{\sum }_{\mathit{j}=1}^n\left(1-H_j\right)}=\frac{1-H_j}{\mathit{n}-{\sum }_{\mathit{j}=1}^n{H}_j}$(7) estimated the entropy weight of indicator j.

(1) $\mathit{R}=\left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1\mathit{n}}\\ r_{21}& r_{22}& \cdots & r_{2\mathit{n}}\\ ⋮& ⋮& \cdots & ⋮\\ r_{\mathit{m}1}& r_{\mathit{m}2}& \cdots & r_{\mathit{mn}}\end{array}\right]$(1)

(2) $r_{\mathit{ij}}=\frac{{\mathit{r}}_{\mathit{ij}}^{\prime }-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)}{\mathit{Max}\left(r_{\mathit{ij}}^{\prime }\right)-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)},\left(\mathit{i}=1,2,\dots ,\mathit{m}.\mathit{j}=1,2,\dots ,\mathit{n}\right)$(2)

(3) $r_{\mathit{ij}}=\frac{\mathit{Max}\left({\mathit{r}}_{\mathit{ij}}^{\prime }\right)-r_{\mathit{ij}}^{\prime }}{\mathit{Max}\left(r_{\mathit{ij}}^{\prime }\right)-\mathit{Min}\left(r_{\mathit{ij}}^{\prime }\right)}$(3)

(4) $H_j=-\mathit{k}{\sum }_{\mathit{i}=1}^m{f}_{\mathit{ij}}\cdot \ln f_{\mathit{ij}}$(4)

(5) $f_{\mathit{ij}}=\mathit{\hspace{0.17em}}\frac{{\mathit{r}}_{\mathit{ij}}}{{\sum }_{\mathit{i}=1}^m=r_{\mathit{ij}}}\mathit{\hspace{0.17em}},\mathit{k}=\frac{1}{ln\mathit{m}}$(5)

(6) ${\mathrm{d}}_{\mathrm{j}}=1-{\mathrm{H}}_j$(6)

(7) $w_j=\frac{1-H_j}{{\sum }_{\mathit{j}=1}^n\left(1-H_j\right)}=\frac{1-H_j}{\mathit{n}-{\sum }_{\mathit{j}=1}^n{H}_j}$(7)

Statistics

Excel 2020 software (Microsoft Corporation, Redmond, USA) was used to collect the expert consultation results and calculate the evaluation coefficients. R studio version 4.1.2 (Rstudio Team, PBC, Boston, MA, USA) was used to perform Kendall’s W test on weighted importance score at all levels of indicators and calculate the entropy value. The test level was α = 0.05.

Results

Basic expert information

We invited 22 experts from 17 medical institutions in 15 regions to participate in this study. The composition and expertise of the participants in this study adhere to the Delphi method requirements, encompassing a diverse range of medical institutions at national, provincial, city, and district/county levels. The participants’ expertise spans various fields, including epidemiology and health statistics, public health or immunization planning, health economics, clinical medicine, and vaccine management or production. The study comprises 12 male and 10 female participants, with 90.91% (20 individuals) holding a master’s degree or higher. Additionally, 86.36% (19 individuals) possess over a decade of work experience in their respective positions (Supplementary Table S1).

The rationality of experts’ consultation

The quality of experts’ consultation showed moderate or higher levels of response, authority, coordination, and acceptance. The positive coefficients for the first and second consultation rounds were 86.36% (19/22) and 85.71 (6/7), respectively, and the authority coefficients for both rounds of expert consultation exceeded 0.79. No statistically significant differences were observed in the coordination coefficients of the primary indicators at the city and district/county levels. However, at other levels and dimensions, Kendall’s coefficient ranged from 0.40 to 0.95 (P < .05), suggesting that experts’ scores were consistent and the results were credible (Supplementary Table S2). The variation coefficients for the importance scores of secondary and tertiary indicators were predominantly below 0.2, with extremely high acceptance of the indicators by experts, but that for primary indicators were all less than 0.4, with moderate acceptance (Supplementary Tables S3–S6).

Framework for vaccine inclusion in public health program

In total, 4 primary indicators, 17 secondary indicators, and 45 tertiary indicators were included in the framework (Tables 1 and 2).

Table 1. Subjective weighting of vaccines in public health programs at each administrative level.

Primary indicators	Weight				Secondary indicators	Weight				Tertiary indicators	Weight
	National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level
1. Vaccine–preventable diseases	32.50	31.67	29.17	29.17	1.1 Health burden	11.71	11.42	10.46	–	1.1.1 High–risk group	1.57	1.53	1.80	–
										1.1.2 Incidence areas	1.20	1.17	1.60	–
										1.1.3 Incidence rate	1.93	1.89	2.17	–
										1.1.4 Mortality rate	1.78	1.74	1.92	–
										1.1.5 Case fatality rate	1.35	1.31	1.79	–
										1.1.6 Disability rate	1.22	1.20	0.35	–
										1.1.7 Hospitalization rate	1.33	1.30	0.42	–
										1.1.8 Severe disease rate	1.33	1.30	0.42	–
					1.2 Economic burden	9.21	8.92	8.50	–	1.2.1 Direct economic burden	5.35	5.18	5.05	–
										1.2.2 Indirect economic burden	3.86	3.74	3.45	–
					1.3 Impact level	7.38	7.21	6.18	–	1.3.1 Public health priorities	5.05	4.83	3.86	–
										1.3.2 Public health emergencies	2.33	2.38	2.33	–
					1.4 Non–vaccine preventive and control measures	4.21	4.13	4.03	29.17	1.4.1 Cost	2.04	2.00	2.01	14.58
										1.4.2 Effectiveness	2.17	2.13	2.01	14.58
2. Candidate vaccines	29.17	28.33	25.00	25.00	2.1 Characteristics	12.33	12.04	12.63	11.13	2.1.1 Immunogenicity	1.56	1.50	1.50	1.10
										2.1.2 Protective efficacy	1.70	1.66	1.76	1.36
										2.1.3 Protective effect	3.06	3.02	3.24	3.04
										2.1.4 Durability of protection	1.75	1.70	1.73	1.71
										2.1.5 Safety	3.06	3.02	3.24	3.04
										2.1.6 Type of vaccine	1.20	1.14	1.15	0.88
					2.2 Application	8.04	7.75	7.25	7.25	2.2.1 Target group of vaccination	3.47	3.33	3.29	3.29
										2.2.2 Cost–effectiveness/effectiveness ratio	4.57	4.43	3.96	3.96
					2.3 Ethics	2.85	3.10	0.25	–	2.3.1 Ethical compliance	2.85	3.10	–	–
					2.4 National/regional experience	5.94	5.44	4.88	6.63	2.4.1 WHO/national/provincial/municipal/district/county recommendation for inclusion	3.63	3.21	2.69	3.74
										2.4.2 Number of countries/provinces/municipalities/counties that have been included in public health programs	2.32	2.23	2.19	2.89
3. BeSD from the supply side	23.33	23.33	26.67	26.67	3.1 Industrial supply	6.67	6.67	5.67	5.67	3.1.1 Maximum supply (capacity)	2.06	–	–	–
										3.1.2 Production efficiency	1.78	–	–	–
										3.1.3 Transportation conditions	1.34	2.73	2.33	2.33
										3.1.4 Storage conditions	1.49	3.18	2.75	2.75
					3.2 Finance	6.17	6.17	7.25	7.25	3.2.1 Financial support and input	3.43	3.43	4.14	4.14
										3.2.2 Financing sustainability	2.73	2.73	3.11	3.11
					3.3 Advocacy policies	3.67	3.67	4.25	4.25	3.3.1 Efforts of policy advocacy and promotion	1.92	2.05	3.21	4.25
										3.3.2 Domestic/import procurement ratio	1.75	1.62	1.04	–
					3.4 Service capacity	4.17	4.17	6.25	6.25	3.4.1 Allocation of professional staff in CDC	0.98	1.05	1.78	1.78
										3.4.2 Allocation of professional staff in vaccination units	1.12	1.12	1.85	1.85
										3.4.3 Surveillance system	1.12	1.05	1.36	1.36
										3.4.4 Information management system	0.95	0.95	1.26	1.26
					3.5 Vaccine prices	3.00	3.00	3.92	3.92	3.5.1 Vaccine price	3.00	3.00	3.92	3.92
4. BeSD from the demand side	15.00	16.67	19.17	19.17	4.1 Demographic characteristics*	–	2.83	3.75	3.75	4.1.1 Gender/age structure	–	2.83	3.75	3.75
					4.2 Social factors	4.17	3.92	4.50	4.50	4.2.1 Community/unit current vaccination rate	3.83	3.58	3.13	3.13
										4.2.2 Religion	–	–	1.37	1.37
					4.3 Motivation/willingness	5.50	5.00	5.50	5.50	4.3.1 Willingness to vaccinate	5.50	2.42	2.62	5.50
										4.3.2 Recommendations of doctors in vaccination–related departments	–	1.36	1.54	–
										4.3.3 Recommendation of doctors in non–vaccination–related departments	–	1.23	1.34	–
					4.4 Behavior	5.33	4.92	5.42	5.42	4.4.1 Consistency between willingness and behavior of vaccination	5.33	4.92	5.42	–

The symbol “*” indicates the indicator is for adult vaccines only. The symbol “- “indicates the indicator is excluded from consideration within the administrative region and is not assigned a weight.

Table 2. Objective weights for vaccine inclusion in public health programs at each administrative level.

Primary indicators	Weight				Secondary indicators	Weight				Tertiary indicators	Weight
	National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level
1. Vaccine–preventable diseases	11.66	8.73	13.57	13.57	1.1 Health burden	3.78	3.45	8.29	–	1.1.1 High–risk group	2.57	2.50	1.68	–
										1.1.2 Incidence areas	1.07	1.04	1.65	–
										1.1.3 Incidence rate	2.45	2.38	1.75	–
										1.1.4 Mortality rate	3.53	3.43	1.68	–
										1.1.5 Case fatality rate	1.08	1.05	1.63	–
										1.1.6 Disability rate	8.47	8.24	–	–
										1.1.7 Hospitalization rate	5.65	5.50	–	–
										1.1.8 Severe disease rate	5.65	5.50	–	–
					1.2 Economic burden	7.02	6.41	4.15	–	1.2.1 Direct economic burden	0.86	0.84	1.21	–
										1.2.2 Indirect economic burden	8.48	8.25	1.93	–
					1.3 Impact level	4.17	3.81	4.18	–	1.3.1 Public health priorities	1.32	1.50	1.37	–
										1.3.2 Public health emergencies	1.27	1.27	2.24	–
					1.4 Non–vaccine preventive and control measures	3.29	3.00	3.85	8.35	1.4.1 Cost	0.86	0.84	2.31	3.75
										1.4.2 Effectiveness	0.86	0.84	2.31	3.75
2. Candidate vaccines	10.63	17.07	21.65	21.65	2.1 Characteristics	9.74	8.88	4.29	4.85	2.1.1 Immunogenicity	3.84	3.74	7.05	2.25
										2.1.2 Protective efficacy	2.25	2.19	2.80	2.04
										2.1.3 Protective effect	1.16	1.13	1.50	3.11
										2.1.4 Durability of protection	2.18	2.12	2.80	4.59
										2.1.5 Safety	2.44	2.37	3.14	2.26
										2.1.6 Type of vaccine	2.45	2.39	3.14	2.73
					2.2 Application	6.61	6.03	4.47	5.04	2.2.1 Target group of vaccination	2.23	2.17	2.88	4.69
										2.2.2 Cost–effectiveness/effectiveness ratio	1.36	1.32	1.73	2.81
					2.3 Ethics	2.85	8.04		–	2.3.1 Ethical compliance	1.79	1.74	–	–
					2.4 National/regional experience	6.20	9.45	3.34	10.85	2.4.1 WHO/national/provincial/municipal/district/county recommendation for inclusion	1.09	2.12	1.64	2.67
										2.4.2 Number of countries/provinces/municipalities/counties that have been included in public health programs	1.09	1.12	1.64	2.67
3. BeSD from the supply side	47.65	44.05	37.89	37.89	3.1 Industrial supply	6.61	6.03	5.49	–	3.1.1 Maximum supply (capacity)	4.28	–	–	–
										3.1.2 Production efficiency	1.16	–	–	–
										3.1.3 Transportation conditions	2.18	5.66	7.50	12.21
										3.1.4 Storage conditions	1.27	1.46	1.93	3.15
					3.2 Finance	6.33	5.78	8.25	9.44	3.2.1 Financial support and input	1.52	1.48	1.96	3.19
										3.2.2 Financing sustainability	1.03	1.00	1.33	2.16
					3.3 Advocacy policies	11.13	10.16	7.28	8.33	3.3.1 Efforts of policy advocacy and promotion	2.23	1.49	1.93	3.75
										3.3.2 Domestic/import procurement ratio	1.36	1.23	4.46	–
					3.4 Service capacity	8.10	7.39	5.87	6.72	3.4.1 Allocation of professional staff in CDC	3.56	2.13	3.53	5.75
										3.4.2 Allocation of professional staff in vaccination units	1.10	1.07	1.96	3.19
										3.4.3 Surveillance system	3.28	1.12	1.17	1.91
										3.4.4 Information management system	0.97	0.94	2.53	4.11
					3.5 Vaccine prices	3.29	3.00	8.29	9.49	3.5.1 Vaccine price	4.69	4.56	6.04	9.83
4. BeSD from the demand side	30.06	30.16	26.89	26.89	4.1 Demographic characteristics*		4.24	13.50	15.45	4.1.1 Gender/age structure	–	1.74	2.31	3.75
					4.2 Social factors	5.70	4.18	5.26	6.02	4.2.1 Community/unit current vaccination rate	1.79	1.74	1.82	2.96
										4.2.2 Religion	–	–	1.82	2.96
					4.3 Motivation/willingness	3.67	3.45	4.15	4.75	4.3.1 Willingness to vaccinate	1.79	3.47	4.59	3.75
										4.3.2 Recommendations of doctors in vaccination–related departments	–	1.48	1.96	–
										4.3.3 Recommendation of doctors in non–vaccination–related departments	–	2.13	2.82	–
					4.4 Behavior	11.51	6.73	4.15	10.72	4.4.1 Consistency between willingness and behavior of vaccination	1.79	1.74	2.31	–

The symbol “*” indicates the indicator is for adult vaccines only. The symbol “- “indicates the indicator is excluded from consideration within the administrative region and is not assigned a weight.

Subjectively, there was no difference in the weight ranking of primary and secondary indicators at the national, provincial, municipal, and district/county levels. Among primary indicators, VPDs were the priority for inclusion. Among secondary indicators, health burden, economic burden, candidate vaccine characteristics, and applications had the greatest weight. Third-dimension indicators showed significant differences in weighting at the administrative levels. At the national level, public willingness to vaccinate (5.50), direct economic burden (5.35), and consistency between willingness and behavior of vaccination (5.33) were prioritized. At the provincial level, direct economic burden (5.18), willingness to vaccinate & voluntary vaccination concordance (social drivers) (4.92), and whether it is a public health priority (4.83) were prioritized. At the municipal level, willingness to vaccinate & voluntary vaccination consistency (5.42), direct economic burden (5.05), and financial support and investment (4.14) were prioritized. At the district/county level, cost (14.58) and effectiveness of non-vaccination prevention and control measures (14.58) were prioritized (Table 1).

Objectively, the weights of social drivers on the supply side and demand side were high for the primary indicator at the national, provincial, municipal, and district/county levels (Table 2). The priority factors for the secondary indicator varied at different administrative levels. The top three weightings at the national level were the behavior of the vaccine demand side (11.51), policies related to the vaccine supply side (11.13), and candidate vaccine characteristics (9.74). At the provincial level, candidate vaccine supply side-related policies (10.16), other regional experiences (9.45), and vaccine characteristics (8.88) were prioritized, respectively. At the municipal level, the top three prioritized indicators were regional demographic characteristics (13.50), health burden of VPDs (8.29), and vaccine prices (8.29). At the district/county level, the top three prioritized indicators were regional demographic characteristics (15.45), experience in other regions (10.85), and public vaccination behavior (10.72). The entropy and utility values that determine the objective weights of all dimension indicators were showed in the supplementary Tables S7–S10.

Discussion

Prevention, control, and eradication of VPDs is a top priority for WHO and its member states. Given the population density and high mobility of people in China, it is needed to prioritize the inclusion of new vaccines in public health programs. Governments and local public health teams face increasingly complex and challenging decisions on which services to invest in. The prioritization framework assists local authorities in the systematic design of public health programs and reduces task affordability and complexity. This study summarized the considerations that need to be taken into account for vaccine inclusion and constructed an evaluation framework for vaccine inclusion in national and regional public health programs, providing a hierarchical and multidimensional assessment tool to follow up the prioritization of non-NIP vaccine use.

Factors associated with VPDS, the vaccine candidate itself, and different aspects of vaccine delivery make vaccine inclusion complex. The results of this study showed that indicators affecting VPDS and vaccine characteristics, such as health burden, economic burden, vaccine efficacy, safety, and cost-effectiveness, hold the greatest weight in vaccine inclusion at all levels and dimensions. Consistent with most national considerations, a systematic review of articles published in the last decade found that the most common indicators included in national frameworks were disease burden, vaccine efficacy/effectiveness, vaccine safety, economic assessment, vaccine impact on health and non-health outcomes, and cost-effectiveness of alternative measures.³² These findings are consistent with the key considerations in the WHO guideline as well, which was designed based on the experience of many countries introducing new vaccines. The data of disease burden can directly reflect the impact of VPDs on public health and is the most important factor for indicating the need for vaccination. Vaccine safety and efficacy are important considerations as well. The risks will outweigh the benefits if a vaccine is not safe enough and has too many side effects. Only after a vaccine has been scientifically validated to be safe and effective could it be recommended for widespread use. Cost-effectiveness assessments can determine if the vaccine is cost-effective and feasible relative to other public health interventions in a resource-limited setting. If costs are too high and benefits are limited, the new vaccine may not be the optimal strategy. With limited resources, public health programs need to select vaccines to maximize the control of serious diseases. Therefore, commonly considered indicators can reflect the public health value of a vaccine.

The emphasis on social drivers was reflected in the top list of subjective weighting of experts’ scores from the national to the municipal level. This weighting was more prominent in the objective evaluation results, especially in the third dimension at the city level. Social drivers are the practices, perceptions, and experiences specific to vaccination and can be altered to improve vaccination rates.³³ However, political priorities, vaccine delivery issues, vaccine supply, funding sources, and the processes and stakeholders involved in the behavioral and social drivers (BeSDs) are not often considered in national frameworks and are more likely to function in vaccine inclusion.³² For example, COVID-19 vaccination is the largest and fastest campaign in history, but many at-risk populations remain unprotected. Herein, only 28% of older people and 37% of healthcare workers in low-income countries have received primary vaccination, and most of them have not yet received a booster dose.³⁴ Despite China’s well-supplied and active incentives, COVID-19 vaccine booster rates for people over 80 years of age is 40–50%,³⁵ and a similar status is observed for influenza vaccination in Beijing, a mega first-tier city.³⁶ By comprehensively considering BeSDs, immunization programs can better address potential barriers and strategically tailor their interventions to maximize vaccine acceptance and program success. Therefore, it is imperative to integrate these drivers into the decision-making process to optimize the inclusion of new vaccines in public programs.

The results of the objective evaluation indicated that vaccination policy advocacy and the proportion of domestic and imported vaccines were heavily weighted at the national and provincial levels. Vaccine price and transport capacity were heavily weighted at the municipal and district/county level. Some international practice in relation to the above factors can benefit policymakers at different levels of administration. In the Country-led Assessment for Prioritization on Immunization, 12 countries across Africa, Asia, and America collaboratively developed a Decision-Support Tool for National Immunization Programs by emphasizing stakeholder engagement and regional contextualization.³⁷ On vaccine policy support, the Global Alliance for Vaccines and Immunisation worked with governments to strengthen policy-making and took a leading role in vaccine introduction, particularly in many low- and middle-income countries. Key features of vaccine adoption policy-making include strong policy dialogue among national stakeholders and institutions, amiable partnerships between the public and private sectors, and addressing reimbursement mechanisms for problematic vaccines related to procurement and production. In terms of vaccine-related funding, developing countries are increasingly interested in weighing the potential health impacts due to the introduction of new vaccines and the required investments, among which the need and demand for technical support and capacity building around economic assessment is high.³⁸ Despite the recognized benefits and cost-effectiveness of vaccination programs, it remains vulnerable to budget cuts.³⁹ Sustainable financing is one of the many components that contribute to the success of vaccination programs, which requires consistent resources and long-term commitment.¹¹

This study provided a framework for assessing the priority of new vaccines at different administrative levels in China. However, some limitations remain: The experts we invited were all from China, missing immunization experts from other countries to provide international insights⁴⁰ and help us avoid risk factors that were overlooked. Experience and research have shown that vaccination strategies work well when culturally suitable approaches are recruited.⁴¹ Therefore, the Chinese framework may not apply to other regions, but the factors considered in this framework can be referred to in other frameworks. Application of the framework will be limited by the lack of local evidence, and data from other regions can be used as a substitution. In addition, the Delphi method relies on expert intuition, which inevitably introduces a subjective bias that may affect the objectivity of the results. Although we combined Delphi with the entropy method to avoid subjective weight scores and group decision-making limitations, they could not be completely avoided because the indicators were selected through experts’ opinions.

Conclusions

This study constructed a systematic framework for the inclusion of new vaccines in public health programs in China, comprising a national-provincial-city-district/county hierarchical system. Either objectively or subjectively, social drivers of both supply and demand sides of the inclusion chain are gradually receiving more attention and weight in decision-making, with increasing attention to vaccine policy advocacy, financial investment, vaccine prices, public willingness to vaccinate, and behavioral consistency among the drivers. When implementing this framework for public health programs, cultural factors, resource availability, delivery infrastructure, and combination of subjective and objective weighting frameworks should be considered. Continuous monitoring and evaluation are needed to ensure that the framework improves priority management and decision-making for immunization programs.

Authors’ contributions

QW: Conceptualization; Methodology; Writing – Review & Editing. PD: Formal analysis; Data Curation. MJ (Mengmeng Jia) & MJ (Mingyue Jiang): Investigation & Data Curation. JL: Software, Resources. WY & LF: Supervision; Writing – Review & Editing; Project administration.

Ethics approval and consent to participate

All data analyzed in this study were collected from experts’ consultation and no individual information was included. Thus, this study was exempted from ethics approval.

Abbreviations

NIP	=	National Immunization Program
EPI	=	Expanded Program on Immunization
WHO	=	World Health Organization
VPDs	=	Vaccine-preventable diseases
BeSDs	=	Behavioral and social drivers

Acknowledgments

We thank all the experts who provided their valuable suggestions for the framework design.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

The datasets used and/or analyzed for this study are available from the corresponding author upon reasonable request.

Supplementary data

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/21645515.2023.2272539.

Additional information

Funding

This study was supported by the the Non-profit Central Research Institute Fund of the Chinese Academy of Medical Sciences [2021-RC330-002.], the Bill & Melinda Gates Foundation [INV-024911] and Beijing Natural Science Foundation-Haidian Original Innovation Joint Fund [L222028]. The funding sources had no role in data collection or the interpretation or decisions relating to data publication.