Construction of an indicator framework for vaccine inclusion in public health programs: A Delphi-entropy method study

Figures & data

Figure 1. Flowchart of constructing an indicator framework. The dotted box in the top half marks the process of selection and determining evaluation indicators for vaccine inclusion in public health programs. The dashed box in the bottom half marks the process of assigning weights to all metrics, subjectively relying on the expert score product and objectively based on the entropy weight.

Table 1. Subjective weighting of vaccines in public health programs at each administrative level.

Primary indicators	Weight				Secondary indicators	Weight				Tertiary indicators	Weight
	National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level
1. Vaccine–preventable diseases	32.50	31.67	29.17	29.17	1.1 Health burden	11.71	11.42	10.46	–	1.1.1 High–risk group	1.57	1.53	1.80	–
										1.1.2 Incidence areas	1.20	1.17	1.60	–
										1.1.3 Incidence rate	1.93	1.89	2.17	–
										1.1.4 Mortality rate	1.78	1.74	1.92	–
										1.1.5 Case fatality rate	1.35	1.31	1.79	–
										1.1.6 Disability rate	1.22	1.20	0.35	–
										1.1.7 Hospitalization rate	1.33	1.30	0.42	–
										1.1.8 Severe disease rate	1.33	1.30	0.42	–
					1.2 Economic burden	9.21	8.92	8.50	–	1.2.1 Direct economic burden	5.35	5.18	5.05	–
										1.2.2 Indirect economic burden	3.86	3.74	3.45	–
					1.3 Impact level	7.38	7.21	6.18	–	1.3.1 Public health priorities	5.05	4.83	3.86	–
										1.3.2 Public health emergencies	2.33	2.38	2.33	–
					1.4 Non–vaccine preventive and control measures	4.21	4.13	4.03	29.17	1.4.1 Cost	2.04	2.00	2.01	14.58
										1.4.2 Effectiveness	2.17	2.13	2.01	14.58
2. Candidate vaccines	29.17	28.33	25.00	25.00	2.1 Characteristics	12.33	12.04	12.63	11.13	2.1.1 Immunogenicity	1.56	1.50	1.50	1.10
										2.1.2 Protective efficacy	1.70	1.66	1.76	1.36
										2.1.3 Protective effect	3.06	3.02	3.24	3.04
										2.1.4 Durability of protection	1.75	1.70	1.73	1.71
										2.1.5 Safety	3.06	3.02	3.24	3.04
										2.1.6 Type of vaccine	1.20	1.14	1.15	0.88
					2.2 Application	8.04	7.75	7.25	7.25	2.2.1 Target group of vaccination	3.47	3.33	3.29	3.29
										2.2.2 Cost–effectiveness/effectiveness ratio	4.57	4.43	3.96	3.96
					2.3 Ethics	2.85	3.10	0.25	–	2.3.1 Ethical compliance	2.85	3.10	–	–
					2.4 National/regional experience	5.94	5.44	4.88	6.63	2.4.1 WHO/national/provincial/municipal/district/county recommendation for inclusion	3.63	3.21	2.69	3.74
										2.4.2 Number of countries/provinces/municipalities/counties that have been included in public health programs	2.32	2.23	2.19	2.89
3. BeSD from the supply side	23.33	23.33	26.67	26.67	3.1 Industrial supply	6.67	6.67	5.67	5.67	3.1.1 Maximum supply (capacity)	2.06	–	–	–
										3.1.2 Production efficiency	1.78	–	–	–
										3.1.3 Transportation conditions	1.34	2.73	2.33	2.33
										3.1.4 Storage conditions	1.49	3.18	2.75	2.75
					3.2 Finance	6.17	6.17	7.25	7.25	3.2.1 Financial support and input	3.43	3.43	4.14	4.14
										3.2.2 Financing sustainability	2.73	2.73	3.11	3.11
					3.3 Advocacy policies	3.67	3.67	4.25	4.25	3.3.1 Efforts of policy advocacy and promotion	1.92	2.05	3.21	4.25
										3.3.2 Domestic/import procurement ratio	1.75	1.62	1.04	–
					3.4 Service capacity	4.17	4.17	6.25	6.25	3.4.1 Allocation of professional staff in CDC	0.98	1.05	1.78	1.78
										3.4.2 Allocation of professional staff in vaccination units	1.12	1.12	1.85	1.85
										3.4.3 Surveillance system	1.12	1.05	1.36	1.36
										3.4.4 Information management system	0.95	0.95	1.26	1.26
					3.5 Vaccine prices	3.00	3.00	3.92	3.92	3.5.1 Vaccine price	3.00	3.00	3.92	3.92
4. BeSD from the demand side	15.00	16.67	19.17	19.17	4.1 Demographic characteristics*	–	2.83	3.75	3.75	4.1.1 Gender/age structure	–	2.83	3.75	3.75
					4.2 Social factors	4.17	3.92	4.50	4.50	4.2.1 Community/unit current vaccination rate	3.83	3.58	3.13	3.13
										4.2.2 Religion	–	–	1.37	1.37
					4.3 Motivation/willingness	5.50	5.00	5.50	5.50	4.3.1 Willingness to vaccinate	5.50	2.42	2.62	5.50
										4.3.2 Recommendations of doctors in vaccination–related departments	–	1.36	1.54	–
										4.3.3 Recommendation of doctors in non–vaccination–related departments	–	1.23	1.34	–
					4.4 Behavior	5.33	4.92	5.42	5.42	4.4.1 Consistency between willingness and behavior of vaccination	5.33	4.92	5.42	–

The symbol “*” indicates the indicator is for adult vaccines only. The symbol “- “indicates the indicator is excluded from consideration within the administrative region and is not assigned a weight.

Table 2. Objective weights for vaccine inclusion in public health programs at each administrative level.

Primary indicators	Weight				Secondary indicators	Weight				Tertiary indicators	Weight
	National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level		National	Provincial	Municipal	District/county-level
1. Vaccine–preventable diseases	11.66	8.73	13.57	13.57	1.1 Health burden	3.78	3.45	8.29	–	1.1.1 High–risk group	2.57	2.50	1.68	–
										1.1.2 Incidence areas	1.07	1.04	1.65	–
										1.1.3 Incidence rate	2.45	2.38	1.75	–
										1.1.4 Mortality rate	3.53	3.43	1.68	–
										1.1.5 Case fatality rate	1.08	1.05	1.63	–
										1.1.6 Disability rate	8.47	8.24	–	–
										1.1.7 Hospitalization rate	5.65	5.50	–	–
										1.1.8 Severe disease rate	5.65	5.50	–	–
					1.2 Economic burden	7.02	6.41	4.15	–	1.2.1 Direct economic burden	0.86	0.84	1.21	–
										1.2.2 Indirect economic burden	8.48	8.25	1.93	–
					1.3 Impact level	4.17	3.81	4.18	–	1.3.1 Public health priorities	1.32	1.50	1.37	–
										1.3.2 Public health emergencies	1.27	1.27	2.24	–
					1.4 Non–vaccine preventive and control measures	3.29	3.00	3.85	8.35	1.4.1 Cost	0.86	0.84	2.31	3.75
										1.4.2 Effectiveness	0.86	0.84	2.31	3.75
2. Candidate vaccines	10.63	17.07	21.65	21.65	2.1 Characteristics	9.74	8.88	4.29	4.85	2.1.1 Immunogenicity	3.84	3.74	7.05	2.25
										2.1.2 Protective efficacy	2.25	2.19	2.80	2.04
										2.1.3 Protective effect	1.16	1.13	1.50	3.11
										2.1.4 Durability of protection	2.18	2.12	2.80	4.59
										2.1.5 Safety	2.44	2.37	3.14	2.26
										2.1.6 Type of vaccine	2.45	2.39	3.14	2.73
					2.2 Application	6.61	6.03	4.47	5.04	2.2.1 Target group of vaccination	2.23	2.17	2.88	4.69
										2.2.2 Cost–effectiveness/effectiveness ratio	1.36	1.32	1.73	2.81
					2.3 Ethics	2.85	8.04		–	2.3.1 Ethical compliance	1.79	1.74	–	–
					2.4 National/regional experience	6.20	9.45	3.34	10.85	2.4.1 WHO/national/provincial/municipal/district/county recommendation for inclusion	1.09	2.12	1.64	2.67
										2.4.2 Number of countries/provinces/municipalities/counties that have been included in public health programs	1.09	1.12	1.64	2.67
3. BeSD from the supply side	47.65	44.05	37.89	37.89	3.1 Industrial supply	6.61	6.03	5.49	–	3.1.1 Maximum supply (capacity)	4.28	–	–	–
										3.1.2 Production efficiency	1.16	–	–	–
										3.1.3 Transportation conditions	2.18	5.66	7.50	12.21
										3.1.4 Storage conditions	1.27	1.46	1.93	3.15
					3.2 Finance	6.33	5.78	8.25	9.44	3.2.1 Financial support and input	1.52	1.48	1.96	3.19
										3.2.2 Financing sustainability	1.03	1.00	1.33	2.16
					3.3 Advocacy policies	11.13	10.16	7.28	8.33	3.3.1 Efforts of policy advocacy and promotion	2.23	1.49	1.93	3.75
										3.3.2 Domestic/import procurement ratio	1.36	1.23	4.46	–
					3.4 Service capacity	8.10	7.39	5.87	6.72	3.4.1 Allocation of professional staff in CDC	3.56	2.13	3.53	5.75
										3.4.2 Allocation of professional staff in vaccination units	1.10	1.07	1.96	3.19
										3.4.3 Surveillance system	3.28	1.12	1.17	1.91
										3.4.4 Information management system	0.97	0.94	2.53	4.11
					3.5 Vaccine prices	3.29	3.00	8.29	9.49	3.5.1 Vaccine price	4.69	4.56	6.04	9.83
4. BeSD from the demand side	30.06	30.16	26.89	26.89	4.1 Demographic characteristics*		4.24	13.50	15.45	4.1.1 Gender/age structure	–	1.74	2.31	3.75
					4.2 Social factors	5.70	4.18	5.26	6.02	4.2.1 Community/unit current vaccination rate	1.79	1.74	1.82	2.96
										4.2.2 Religion	–	–	1.82	2.96
					4.3 Motivation/willingness	3.67	3.45	4.15	4.75	4.3.1 Willingness to vaccinate	1.79	3.47	4.59	3.75
										4.3.2 Recommendations of doctors in vaccination–related departments	–	1.48	1.96	–
										4.3.3 Recommendation of doctors in non–vaccination–related departments	–	2.13	2.82	–
					4.4 Behavior	11.51	6.73	4.15	10.72	4.4.1 Consistency between willingness and behavior of vaccination	1.79	1.74	2.31	–

The symbol “*” indicates the indicator is for adult vaccines only. The symbol “- “indicates the indicator is excluded from consideration within the administrative region and is not assigned a weight.

Data availability statement

The datasets used and/or analyzed for this study are available from the corresponding author upon reasonable request.