Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists

Figures & data

Table 1. PubMed, and Cochrane Library search strategies

Search	Quarry
	PubMed
#1	Search immunization schedule[mh]
#2	Search (immunisation coverage[tiab] OR immunisation rate[tiab] OR immunisation uptake [tiab] OR immunization coverage[tiab OR immunization rate[tiab] OR immunization uptake [tiab] OR vaccination coverage[tiab] OR vaccination rate[tiab] OR vaccination uptake [tiab] OR vaccine coverage [tiab] OR vaccine uptake [tiab])
#3	Search (#1 OR #2)
#4	Search (meta-analysis[mh] OR meta analysis[pt] OR meta-review[ti] OR meta-analysis[tiab] OR meta-analysis [ti] OR metaanalysis [ti] OR meta-analyses [tiab])
#5	Search (review[tiab] OR systematic review[tiab]OR review[pt] OR systematic reviews[pt] OR literature review[tiab])
#6	Search (#4 OR #5)
#7	Search (#3 AND #6)
	Cochrane Library
#1	MeSH descriptor: [Immunization] explode all trees
#2	MeSH descriptor: [Immunization Schedule] explode all trees
#3	MeSH descriptor: [Immunization, Secondary] explode all trees
#4	MeSH descriptor: [Immunization Programs] explode all trees
#5	MeSH descriptor: [Vaccination] explode all trees
#6	immunisation coverage OR immunisation rate OR immunisation uptake OR immunization coverage OR immunization rate OR immunization uptake OR vaccination coverage OR vaccination rate OR vaccination uptake OR vaccine coverage OR vaccine uptake:ti,ab,kw
#7	#1 or #2 or #3 or #4 or #5 or #6

Table 2. ROBIS assessment of systematic reviews of interventions to improve immunization coverage

	Phase 2				Phase 3
Review	1: Study Eligibility Criteria	2: Identification and selection of studies	3: Data collection and study appraisal	4: Synthesis and findings	Judging risk of Bias
Adams 2015
Aigbogun 2015
Arditi
Batt 2004
Bassani 2013
Bordley 2000
Briss 2000
Chamla 2015
Corace 2016
Elkan 2000
Fu 2014
Gera 2016
Giuffrida 1999
Glenton 2011
Groom 2015
Gyorkos 1994
Harvey 2015
Hollmeyer 2012
Jacob 2016
Jacobson Van 2005
Johri 2015
Kauffman 2013
Kendrick 2000
Lagarde 2007
Lagarde 2009
Lassi 2015
Lau 2012
Lee 2016
Levin 2010
Lystra 2016
Maglione 2002
Mureed 2015
Ndiaye 2005
Niccolai 2015
Odone 2015
Oyo-ita 2016
Peacock 2013
Pegurri 2005
Poorman 2015
Rashid 2016
Rayman 2008
Saeterdal 2014
Schmidt 2013
Shea 1996
Shea 2009
Smulian 2016
Stone 2002
Szilagyi 2000
Thomas 2010
Thomas 2014
Voss 2016
Walling 2016
Wang 2016
Whittaker 2002
Wigham 2014
Williams 2011
Wong 2016

Low risk, High risk, Unclear risk

Figure 1. PRISMA search flow diagram

Figure 2. AMSTAR-based PCA showing the relationship between methodological quality and risk of bias of reviews

Figure 3. ROBIS-based PCA showing the relationship between methodological quality and risk of bias of reviews

Figure 4. Scree plot of AMSTAR-based PCA showing total variance in the data as explained by each principal component

Figure 5. Scree plot of ROBIS-based PCA showing total variance in the data as explained by each principal component

Table A1. AMSTAR checklist

No	Review question	Y	N	Unc	NI	Page No.
	Y = Yes; N = No; Unc = Unclear; NI = No information NB: add comment below row when response is unclear
1	Was an ‘a priori’ design for eligible studies provided? The research question and inclusion criteria should be established before the conduct of the review.
2	Was there duplicate study selection and data extraction? There should be at least two independent data extractors and a consensus procedure for disagreements should be in place.
3	Was a comprehensive literature search performed? At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.
4	Was the status of publication (i.e. grey literature) used as an inclusion criterion? The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language etc.
5	Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided.
6	Were the characteristics of the included studies provided? In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported.
7	Was the scientific quality of the included studies assessed and documented? ‘A priori’ methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.
8	Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations.
9	Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test for homogeneity, I-squared statistic). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e. is it sensible to combine?).
10	Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test).
11	Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.
Total “Yes” score

Table A2. Quality assessment using ROBIS tool R

Phase 2: Identifying concerns with the review process
Domain 1: Study Eligibility Criteria Describe the study eligibility criteria, any restrictions on eligibility and whether there was evidence that objectives and eligibility criteria were pre-specified. Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
1.1 Did the review adhere to pre-defined objectives and eligibility criteria?
1.2 Were the eligibility criteria appropriate for the review question?
1.3 Were eligibility criteria unambiguous?
1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate (e.g. date, sample size, study quality, outcomes measured)?
1.5 Were any restrictions in eligibility criteria based on sources of information appropriate (e.g. publication status or format, language, availability of data)?
Concerns regarding specification of study eligibility criteria		LOW
		HIGH
		UNCLEAR
Rationale for concern: study eligibility appropriate
Domain 2: Identification and selection of studies Describe methods of study identification and selection (e.g. number of reviewers involved). Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
2.5 Were efforts made to minimize error in selection of studies?
Concerns regarding methods used to identify and/or select studies		LOW
		HIGH
		UNCLEAR
Rationale for concern: Identification and selection of studies appropriate
Domain 3: Data collection and study appraisal Describe methods of data collection, what data were extracted from studies or collected through other means, how risk of bias was assessed (e.g. number of reviewers involved) and the tool used to assess risk of bias. Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
3.1 Were efforts made to minimize error in data collection?
3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?
3.3 Were all relevant study results collected for use in the synthesis?
3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria?
3.5 Were efforts made to minimize error in risk of bias assessment?
Concerns regarding methods used to collect data and appraise studies		LOW
		HIGH
		UNCLEAR
Rationale for concern: Data collection and study appraisal good
Domain 4: Synthesis and findings. Describe synthesis methods Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
4.1 Did the synthesis include all studies that it should?
4.2 Were all pre-defined analyses reported or departures explained?
4.3 Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies?
4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?
4.5 Were the findings robust, e.g. as demonstrated through funnel plot or sensitivity analyses?
4.6 Were biases in primary studies minimal or addressed in the synthesis?
Concerns regarding the synthesis and findings		LOW
		HIGH
		UNCLEAR
Rationale for concern: Synthesis and findings good
Phase 3: Judging Risk of Bias
Summarize the concerns identified during the Phase 2 assessment:
Domain	Concern	Rationale for concern
1. Concerns regarding specification of study eligibility criteria
2. Concerns regarding methods used to identify and/or select studies
3. Concerns regarding used to collect data and appraise studies
4. Concerns regarding the synthesis and findings
RISK OF BIAS IN THE REVIEW
Describe whether conclusions were supported by the evidence Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
A. Did the interpretation of findings address all of the concerns identified in Domains 1 to 4?
B. Was the relevance of identified studies to the review’s research question appropriately considered?
C. Did the reviewers avoid emphasizing results on the basis of their statistical significance?
Risk of bias in the review		LOW
		HIGH
		UNCLEAR

Table

Component	Eigenvalue	Difference	Proportion	Cumulative
Comp1	4.06112	2.60331	0.3124	0.3124
Comp2	1.45782	.149729	0.1121	0.4245
Comp3	1.30809	.373597	0.1006	0.5252
Comp4	1.27073	.311338	0.0977	0.6229
Comp5	.959391	.0672598	0.0738	0.6967
Comp6	.892131	.133032	0.0686	0.7653
Comp7	.759099	.116533	0.05884	0.8237
Comp8	.642566	.18544	0.0494	0.8731
Comp9	.457126	.0569896	0.0352	0.9083
Comp10	.400137	.0218823	0.0308	0.9391
Comp11	.378254	.0746761	0.0291	0.9682
Comp12	.303578	.193621	0.0234	0.9915
Comp13	.109958	.	0.0085	1.0000

Table

Variable	Comp1	Comp2	Comp3	Comp4	Comp5	Comp6	Comp7	Comp8	Comp9	Comp10	Comp11
item1	0.1759	0.2721	0.1748	0.6267	0.1467	−0.0338	0.6104	−0.1295	−0.0985	0.14751	−0.1678
item2	0.3640	−0.2305	0.0955	−0.0933	0.0700	−0.5084	0.2085	0.5846	−0.1659	0.0218	0.3462
item3	0.2952	0.1327	−0.5285	0.0818	0.4330	0.0163	−0.0915	−0.3536	−0.0439	−0.1131	0.5222
item4	0.4008	−0.4187	−0.2304	0.0076	−0.0078	0.7178	0.2717	0.2693	−0.2289	−0.1446	−0.0657
item5	0.2820.	0.0451	0.2902	0.1593	−0.0811	0.0736	−0.5374	−0.1097	−0.6267	0.1083	−0.1023
Item6	0.2821	−0.4916	0.2660	−0.2089	−0.0844	0.0350	0.1604	−0.4984	0.2348	0.4538	0.1439
Item7	0.2729	0.3805	0.4697	−0.2653	−0.2317	0.0237	0.0473	0.1715	0.0199	0.5882	−0.2396
Item8	0.3327	0.1750	−0.0338	−0.0310	−0.7212	−00526	0.1649	−0.1951	0.0743	−0.4951	0.1334
Item9	0.3643	0.1005	0.1271	−0.4724	0.4295	−0.0857	0.1197	−0.1184	0.0534	−0.3597	−0.5152
Item10	0.1765	0.4423	0.4654	−0.1949	0.1173	0.4522	−0.0789	0.2581	0.3435	0.0528	0.3657
Item11	0.3575	−0.2347	−0.1355	0.4661	0.0126	−0.0555	−0.3704	0.1858	0.5778	−0.0562	−0.2637

Table

Component	Eigenvalue	Difference	Proportion	Cumulative
Comp1	4.7725	2.13981	0.2273	0.2273
Comp2	2.63269	.306424	0.1254	0.3526
Comp3	2.32627	.647554	0.1108	0.4634
Comp4	1.67871	.0656661	0.0799	0.5433
Comp5	1.61305	.399984	0.0768	0.6202
Comp6	1.21306	.0793917	0.0578	0.6779
Comp7	1.13367	.270353	0.0540	0.7319
Comp8	.863317	.0641371	0.0411	0.7730
Comp9	.79918	.143115	0.0312	0.8111
Comp10	.656064	.016294	0.0312	0.8423
Comp11	.63977	.115759	0.0305	0.8728
Comp12	.524011	.0710757	0.0250	0.8977
Comp13	.452936	.0629754	0.0216	0.9193
Comp14	.38996	.0610955	0.0186	0.9379
Comp15	.328865	.0636681	0.0157	0.9535
Comp16	−265,197	0.0479669	0.0126	0.9662
Comp17	.21723	.0401903	0.0103	0.9765
Comp18	.17704	.0328356	0.0084	0.9849
Comp19	.144204	.0545696	0.0069	0.9918
Comp20	.0896344	.00699254	0.0043	0.9961
Comp21	0.0826419	.	0.0039	1.0000

Table

Variable	Comp1	Comp2	Comp3	Comp4	Comp5	Comp6	Comp7	Comp8	Comp9	Comp10	Comp11	Comp 12
Item1	0.3134	−0.2456	0.1615	0.1713	−0.2439	0.0485	0.0435	−0.0842	−0.0616	0.3157	0.0339
Item2	0.2607	−0.1960	−0.3837	−0.0382	−0.0371	0.0031	−0.087	−0.2552	−0.0108	0.2403	0.0355	0.2045
Item3	0.2777	−0.1566	−0.1519	0.1390	−0.0464	−0.1550	−0.4408	0.1197	−0.1121	−0.2963	0.1531	0.0045
Item4	0.2072	−0.2522	0.3909	−0.0453	0.1297	0.1480	0.1752	0.0803	0.1273	−0.1620	0.1454	−0.3470
Item5	0.0725	0.1960	−0.1914	0.4294	0.3695	−0.0261	0.2030	−0.1107	0.1662	−0.0626	0.0019	0.0227
Item6	0.1717	0.2657	0.2169	−0.0248	0.2592	−0.3665	−0.3040	0.0594	0.1030	0.2131	0.1329	−0.2445
Item7	0.2161	−0.0939	0.2986	−0.0867	0.2957	−0.0988	−0.2681	−0.1389	0.1896	0.3569	−0.1159	0.3656
Item8	0.2145	−0.2248	0.1490	−0.2998	0.1580	0.0862	0.0688	0.5448	−0.2158	0.0657	0.0415	0.1252
Item9	0.0284	−0.0271	0.0404	0.5547	0.3098	0.1406	0.2236	0.2337	−0.0257	0.1859	0.3204	0.1111
Item10	0.2193	0.1741	−0.2663	−0.1830	0.1075	−0.3695	0.0917	0.3168	0.1616	−0.1187	−0.0694	0.2344
Item11	0.1507	0.1434	−0.3582	−0.1316	0.3022	−0.1621	0.2816	−0.0388	−0.1241	−0.0377	−0.2227	−0.3196
Item12	0.2260	−0.1882	−0.1083	−0.3820	0.1703	0.0011	0.2662	−0.2691	−0.0813	−0.1158	0.3931	−0.1317
Item13	0.2751	0.0291	−0.0010	0.1504	−0.4395	−0.0624	0.2752	0.2592	−0.0158	0.3339	−0.0329	−0.1938
Item14	0.2368	0.2627	0.3467	0.1192	−0.0303	0.0032	0.0544	−0.1087	−0.1944	−0.1771	−0.1317	−0.0380
Item15	0.2749	0.1794	0.1107	0.0376	0.0824	0.2223	0.0921	−0.1196	−0.5408	−0.1820	−0.2543	0.3696
Item16	0.1476	0.1156	0.2264	0.0184	−0.3665	−0.4254	0.3149	−0.0315	0.2219	−0.1759	−0.0056	0.2338
Item17	0.2427	0.0493	−0.2105	0.1241	−0.0715	0.3029	−0.2995	0.3305	0.2298	−0.3210	−0.1484	−0.1012
Item18	0.2113	−0.3457	0.0922	0.0909	0.0549	0.1080	0.1172	−0.2335	0.4714	−0.2703	−0.2206	0.0793
Item19	0.1350	0.3270	−0.0211	−0.2555	0.0236	0.4466	0.0643	0.0314	0.3031	0.2991	−0.2797	−0.0912
Item20	0.0573	0.4057	−0.0520	−0.1657	−0.1196	0.2804	0.0232	−0.0155	0.1974	−0.1030	0.5976	0.2956
Item21	0.3311	0.2209	0.0588	0.0778	−0.1342	0.0328	−0.2287	−0.2948	0.1101	0.0309	0.1360	−0.3126