https://orcid.org/0000-0002-1273-4779
ABSTRACT
Introduction: Systematic reviews (SRs) are the backbone of evidence-based health care, but no gold standard exists to assess their methodological quality. Although the AMSTAR tool is accepted for analyzing the quality of SRs, the ROBIS instrument was recently developed. This study compared the capacity of both instruments to capture the quality of SRs of interventions for improving vaccination coverage.
Methods: We conducted a comprehensive literature search in the Cochrane Library and PubMed. Two reviewers independently screened the search output, assessed study eligibility, and extracted data from eligible SRs; resolving differences through consensus. We conducted Principal Component Analysis (PCA) in Stata 14 to determine similarities and differences between AMSTAR and ROBIS.
Results: A total of 2322 records were identified through the search and 75 full-text publications were assessed for eligibility, of which 57 met inclusion criteria. Using AMSTAR, we found 32%, 60% and 9% of SRs to have high, moderate and low quality, respectively. With ROBIS, we judged 74%, 14% and 12% of SRs to have low, unclear and high risk of bias. PCA showed that SRs with low risk of bias in ROBIS clustered together with SRs having high-quality in AMSTAR, and SRs with high risk of bias in ROBIS clustered with low-quality SRs in AMSTAR.
Conclusions: Our findings suggest that there is an association between methodological quality and risk of bias in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.
KEYWORDS:
Background
Vaccination is a very powerful public health tool for improving human survival from infectious diseasesCitation1-Citation3 The use of vaccination as a public health strategy began when the World Health Organization (WHO) launched the Expanded Programme on Immunization (EPI) in 1974.Citation4 When the EPI was launched, the WHO recommended a standard vaccination schedule covering six basic vaccines i.e., tuberculosis (Bacille Calmette-Guerin (BCG)), polio, diphtheria, tetanus, pertussis, and measles.Citation5 Vaccination has the potential to increase uptake and coverage of newly available vaccines in EPI programs of low and middle-income countries.Citation1,Citation6-Citation8 However, the achievement so far is described as “fragile” as judged by outbreaks of some of these diseases in low, middle and high-income countries.Citation7 The outbreaks reflect the existence of communities with partially vaccinated or unvaccinated children.Citation7
A systematic review (SR), which is a type of study designed to synthesize available evidence can be used to answer questions about the effects of interventions used to improve the coverage rates of different vaccines.Citation9 Therefore, the quality of SRs has to be assessed before they can be used to make any clinical recommendations. Various assessment tools have been developed to assess the methodological quality of SRs, with the most commonly used ones being the “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) and “Risk Of Bias In Systematic reviews” (ROBIS) checklists.Citation10,Citation11 AMSTAR consists of 11 items () used as a checklist added up into an overall score while ROBIS which is used to evaluate risk of bias in SRs is based on three phases.Citation10,Citation11 Phase one is optional, phase two covers four domains (i.e., study eligibility criteria, identification and selection of studies, data collection and study appraisal and synthesis and findings) through which bias may be introduced in an SR and phase three, which has four domains (which summarizes the concerns identified during the phase two assessment), assesses the overall risk of bias (). AMSTAR reports the level of methodological quality as low, moderate and high while ROBIS reports SRs as having low or high risk of bias.Citation10 Therefore, the aim of this study was to compare the methodological quality of systematic reviews of interventions for improving vaccination coverage, using AMSTAR and ROBIS tools.
Table 1. PubMed, and Cochrane Library search strategies
Table 2. ROBIS assessment of systematic reviews of interventions to improve immunization coverage
Results
Search results
The literature search yielded a total number of 2322 of records of which 75 full-text publications were retrieved and assessed for eligibility. After full-text assessment, 57 were considered eligible for inclusion. The search and selection of reviews are shown in .
Figure 1. PRISMA search flow diagram
General characteristics
Among the 57 included reviews, 11(19.3%) were Cochrane reviews. The first authors of the SRs were from the following countries: UK (10 reviews),Citation12-Citation21 Canada (9 reviews),Citation6,Citation22-Citation29 Norway (3 reviews),Citation30-Citation32 Greece (1 review),Citation33 Australia (3 reviews),Citation34-Citation36 Switzerland (2 reviews),Citation37,Citation38 USA (9 reviews),Citation38-Citation46 Thailand (1 review),Citation47 Hong Kong (1 review),Citation48 China (1 review)Citation49 and Nigeria (1 review).Citation50
Results using AMSTAR instrument
Using the AMSTAR tool, we judged 18(32%) reviews as being of high methodological quality, 34(60%) as moderate quality, and 5(9%) as low quality. Cochrane reviews were all high quality (total scores ranged from 9 to 11) while the quality of non-Cochrane reviews ranged from low to high quality (scores from 3 to 10).
Results using ROBIS instrument
Using the ROBIS tool, we classified 42(74%) reviews to have low risk of bias, 8(14%) as unclear risk and 7(12%) as having high risk of bias. None of the Cochrane reviews had a high risk of bias.
Principal component analysis
We used principal component analysis (PCA) to convert vectors of the 11 AMSTAR item scores and 21 ROBIS signaling questions to two principal components (PCs), respectively. Subsequently, we used PCA to determine the relationship between AMSTAR and ROBIS. PC1 and PC2 projections where SRs of similar methodological quality and risk of bias clustered together are shown in and . There was an overlap of SRs with low risk of bias in ROBIS and high quality in AMSTAR (robis = 1/amstar = 3), high risk of bias and low quality (robis = 3/amstar = 1), and unclear risk of bias and moderate quality (robis = 2/amstar = 2) ( and ).
Figure 2. AMSTAR-based PCA showing the relationship between methodological quality and risk of bias of reviews
Figure 3. ROBIS-based PCA showing the relationship between methodological quality and risk of bias of reviews
The AMSTAR-based PCA results showed that in the first component, the 5th AMSTAR item “Was a list of studies (included and excluded) provided?” explained most of the variability (40%) while the 8th AMSTAR item “Was the scientific quality of the included studies used appropriately in formulating conclusions?” explained most variability (17%) in the second component (, Appendix II). Concerning the ROBIS-based PCA, data showed that in the first component, question 21 “Were biases in primary studies minimal or addressed in the synthesis?” contributed more than 48% variability while the second component showed that item 20 “Were the findings robust, e.g., as demonstrated through funnel plot or sensitivity analyses?” contributed 26% of variability in the data (, ).
Figure 4. Scree plot of AMSTAR-based PCA showing total variance in the data as explained by each principal component
Figure 5. Scree plot of ROBIS-based PCA showing total variance in the data as explained by each principal component
Discussion
The two validated tools were used for the assessment of methodological quality and risk of bias of SRs in this study. AMSTAR is a tool that consists of 11-items interpreted individually or as the sum of reported items, i.e., an overall score which assesses methodological quality while ROBIS is a domain-based instrument consisting of 21 signaling questions designed to evaluate the risk of bias. There are two primary differences between AMSTAR and ROBIS. AMSTAR does not evaluate how authors collected relevant data for the review and ROBIS does not assess anything about authors’ conflict of interests and funding sources.
Our results show that most systematic reviews that assess the effects of interventions aimed at improving vaccination coverage have been conducted in the United States of America. To our knowledge, this is the first study to compare the results of the AMSTAR and ROBIS tools for assessing risk of bias and methodological quality in SRs concerning interventions to improve vaccination coverage. Our findings report that AMSTAR judged most of the SRs as having high and moderate quality while it considered only 9% of the reviews as low methodological quality. In addition, we have observed that all Cochrane reviews were of high methodological quality while the non-Cochrane reviews were of low, moderate and high quality. The current results seem to be consistent with other studies whose objectives were also to assess methodological quality of SRs using the two tools, although focusing on different fields.Citation51,Citation52
These previous studies also used the AMSTAR tool to evaluate the methodological quality of SRs of health-care interventions. Furthermore, they showed that AMSTAR rated Cochrane reviews as having high methodological quality compared to non-Cochrane reviews.Citation51,Citation52 Taken together, these data suggest that AMSTAR is a useful tool for assessing the quality of Cochrane and non-Cochrane SRs.
Concerning bias, most of the reviews, including all Cochrane reviews, were judged to have low risk of bias (74%) while the rest had unclear or high risk of bias (). These results suggest that Cochrane reviews of interventions are of higher quality compared to non-Cochrane reviews. This study suggests that ROBIS has fair consistency and good legitimacy to appraise risk of bias in systematic reviews.
Regarding principal component analysis (PCA), the AMSTAR-based PCA data showed that SRs classified as having high methodological quality by AMSTAR clustered together with those considered to have low risk of bias by ROBIS. Furthermore, SRs judged as having low methodological quality clustered together with those having high risk of bias (). The ROBIS-based PCA also showed similar findings (). These results suggest that there is an association between SRs with high methodological quality and those with low risk of bias. Although these tools are designed to assess two different concepts, our data imply that low risk of bias translates to high methodological quality of SRs. Our findings are different from a study conducted by,Citation53 whose PCA data showed that reviews on psoriasis interventions, classified as having high and moderate quality were considered as having high risk of bias. This previous study thus suggested that it is possible for SRs with high methodological quality to have high risk of bias.Citation53
With regard to variability of data, the fifth item of AMSTAR explained the highest variability (40%) of data which implies that this item varies significantly between systematic reviews when it comes to methodological quality. Therefore, this also suggests that the fifth item is the best to distinguish between SRs of low, moderate and high quality. Similarly, contribution of 48% variability by signaling question 21 in ROBIS infers that this item varies significantly between SRs and is also the best to differentiate between SRs of low versus high risk of bias. Our results are different from those obtained by,Citation53 who reported that two of ROBIS signaling questions, i.e., the 14th and 15th items contributed most variability in their data. The findings by this previous study suggested that both these items were the best to differentiate between SRs of high and low risk of bias.Citation53
Other qualitative studies, e.g., survey methodology studies have used PCA to determine the different factors that contribute to variations in the survey population structures or to changes in public health or changes in behavior. These included research that set out to identify the contribution of different categories of mothers’ characteristics (e.g., birth order, mother’s education and household wealth index) to an increase in proportion of births attended by skilled personnel;Citation54 contribution of maternal variables (e.g., age, education level, Medicaid insurance status, infant birth order), the social environment and physical environment to non-Hispanic black and white individuals in preterm birth;Citation55 variations in the prevalence of missed opportunities for vaccination (MOVs) among children of poor and non-poor mothers in sub-Saharan African countriesCitation56 and contribution of dietary patterns to the risk of developing colorectal cancer.Citation53
Similar to the present investigation, the above-mentioned studies successfully identified the factors that mainly contributed to changes in the number of births attended by skilled health-care workers, preterm births in non-Hispanic black and white individuals, prevalence of MOVs and risk of developing colorectal cancer, using PCA. Applying PCA, the current study has added to the body of knowledge regarding identifying the AMSTAR and ROBIS items that mostly contribute to variation in the quality of systematic reviews aimed at increasing vaccination coverage.
Practical implications of the study
ROBIS is a newly developed and approved instrument for appraising risk of bias in SRs and hence has been reported to have overcome limitations of prior instruments. Although many studies have assessed the methodological quality of SRs using AMSTAR in a variety of research fields, this study suggests that SRs use both AMSTAR and ROBIS to evaluate methodological quality of reviews. In addition, this paper recommends the use of PCA to identify the AMSTAR and ROBIS items that contribute to variance in the quality of SRs. The use of PCA also identifies the AMSTAR items and ROBIS signaling questions that contribute the most to the final judgment of methodological quality and risk of bias.
Limitations of the study
This study conducted a comprehensive search to procure a sample of published systematic reviews on interventions to increase vaccination coverage. Therefore, one of the limitations of this investigation includes the fact that we did not conduct a search in gray literature databases. The latter implies that we could not assess the methodological quality in the unpublished research. Additionally, the search was conducted in June 2016, suggesting that we may have missed recently published data on systematic reviews of interventions aimed at increasing vaccination coverage. With regards to the methodological quality assessment, one significant limitation is related to the indeterminate conversion of the AMSTAR total scores into categorical rankings, e.g., low, middle and high methodological quality. This would potentially make it difficult to measure the differences in methodological quality across systematic reviews.
Methods
This manuscript reports a pre-specified sub-study of an overview of systematic reviews of interventions aimed at improving vaccination coverage, whose protocol is registered in the International Prospective Register of Systematic Reviews (PROSPERO), with registration number CRD42018090342.Citation57 We included peer-reviewed systematic reviews which assessed the effects of interventions to improve vaccination coverage; irrespective of the type of study designs, participants, interventions, or vaccines concerned.
A comprehensive literature search was conducted in June 2016 in the Cochrane Library and PubMed using a combination of keywords, including vaccination, immunization, vaccine, uptake, and coverage (). Reference lists of relevant publications were also screened for potentially eligible reviews. Two authors, Anelisa Jaca (AJ) and Valantine Ngum Ndze (VNN), independently screened all search records for potentially eligible reviews. Full texts of articles judged to be potentially eligible by either author were retrieved and independently assessed by the two authors for inclusion. The two authors (AJ and VNN) independently extracted data on the citation details and AMSTAR and ROBIS quality items from eligible systematic reviews. In each case, the two authors resolved discrepancies by discussion and consensus.
AMSTAR is an 11-item validated tool used to assess the methodological quality of systematic reviews (). Each item judged as “yes” was awarded 1 point and a total score was recorded. A review with a total score of 0 to 4 was indicated as low quality, 5 to 8 as moderate quality, and 9 to 11 as high quality.
The ROBIS tool consists of three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help evaluate concerns about potential biases in the review. The ratings from these signaling questions help assessors to judge overall risk of bias. The signaling questions are answered as “Yes”, “Probably Yes”, “Probably No”, “No” and “No Information”, with “Yes” indicating low concerns of risk of bias. The level of concern about bias associated with each domain is then judged as “low,” “high,” or “unclear”Citation58 ().
Principal Component Analysis (PCA) is a statistical method which has been used by different studies to reduce a large set of variables to a small set of variables called principal components. Furthermore, PCA is used to identity differences and similarities in a dataset and identify the components of the dataset that contribute to variation.Citation59 PCA was performed in Stata 14 to determine similarities and differences between AMSTAR and ROBIS. The 11 AMSTAR item scores and 21 ROBIS signaling questions were, respectively, converted into two sets of uncorrelated values called principal components per review and each review was tagged based on ROBIS and AMSTAR final classification. For the purpose of PCA, SRs with low, moderate and high methodological quality were, respectively, coded as 1, 2 and 3. In addition, SRs with low, unclear and high risk of bias were coded as 1, 2 and 3, respectively. Subsequently, biplots were constructed to determine the relationship between methodological quality and risk of bias as per the final classification scores of the two tools (ROBIS and AMSTAR).
Conclusions
Our results indicate that there is an association between methodological quality (measured using AMSTAR) and risk of bias (measured using ROBIS) in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.
Disclosure of potential conflicts of interest
No potential conflicts of interest were disclosed.
Declarations
Ethics approval and consent to participate
This study was based on an analysis of existing survey data with all identifier information removed; therefore, no ethical clearance was required.
Consent for publication
Not applicable.
Acknowledgments
The authors would like to acknowledge Yusentha Balakrishna, the Biostatistician who assisted with conducting the Principle Component Analysis.
Additional information
Funding
Related Research Data
References
- Andre FE, Booy R, Bock HL, Clemens J, Datta SK, John TJ. Vaccination greatly reduces disease, disability, death and inequity worldwide. Vol. 86. Belgium: Bulletin of the World Health Organization; 2008. p. 140–46.
- Clements CJ, Nshimirimanda D, Gasasira A. Using immunization delivery strategies to accelerate progress in Africa towards achieving the millennium development goals. Vaccine. 1926–33;26:2008.
- Okwo-Bele JM. Integrating immunization with other health interventions for greater impact: the right strategic choice. J. Infect. Dis. 2012;205:4–5. doi:10.1093/infdis/jis213.
- Wiysonge CS, Uthman OA, Ndumbe PM, Hussey GD. A bibliometric analysis of childhood immunization research productivity in Africa since the onset of the expanded program on immunization in 1974. bmc med. 2013;11:66. doi:10.1186/1741-7015-11-66.
- Maharani A, Kuroda Y. Determinants of immunization status among 12- to 23-month-old children in Indonesia (2008 – 2013): a multilevel analysis. BioMed Central; 2018. p. 1–11.
- Lau D, Hu J, Majumdar S. Interventions to improve influenza and pneumococcal vaccination rates among community-dwelling adults: a systematic review and meta-analysis. Ann Fam Med. 2012;10(6):538–47. doi:10.1370/afm.1405.
- Philippe D, Jean-Marie O-B, Marta G-D, Thomas C. Global immunization: status, progress, challenges and future. BMC Int Health Hum Rights. 2009;9(Suppl 1):S2. doi:10.1186/1472-698X-9-S1-S2.
- Machingaidze S, Wiysonge CS, Hussey GD. Strengthening the expanded programme on immunization in Africa: looking beyond 2015. PLOS Medicine; 2015;10(3):1–5.
- Ryś P, Władysiuk M, Skrzekowska-Baran I, Małecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Pol. Arch. Med. WEWNETRZNEJ. 2009;119:148–56.
- Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J. Clin. Epidemiol. 2016;69:225–34. doi:10.1016/j.jclinepi.2015.06.005.
- Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Syst. Rev. 2016;5:1. doi:10.1186/s13643-016-0289-2.
- Aigbogun NW, Hawker JI, Stewart A. Interventions to increase influenza vaccination rates in children with high-risk conditions – a systematic review. Vaccine. 2015;33(6):759–70. doi:10.1016/j.vaccine.2014.12.013.
- Adams J, Bateman B, Becker F, Cresswell T, Flynn D, McNaughton R. Effectiveness and acceptability of parental financial incentives and quasi-mandatory schemes for increasing uptake of vaccinations in preschool children: systematic review, qualitative study and discrete choice experiment. Health Technol Assess (Rockv). 2015;19:94.
- Elkan R, Kendrick D, Hewitt M, Robinson JJA, Tolley K, Blair M. The effectiveness of domiciliary health visiting: a systematic review of international studies and a selective review of the British literature. Health Technol Assess (Rockv). 2000;4(13):i-v+1–330.
- Harvey H, Reissland N, Mason J. Parental reminder, recall and educational interventions to improve early childhood immunisation uptake: a systematic review and meta-analysis. Vaccine. 2015;33(25):2862–80. doi:10.1016/j.vaccine.2015.04.085.
- Kendrick D, Hewitt M, Dewey M, Elkan R, Blair M, Robinson J. The effect of home visiting programmes on uptake of childhood immunization: a systematic review and meta-analysis. J Public Health Med. 2000;22(1):90–98.
- Pegurri E, Fox-Rushby JA, Damian W. The effects and costs of expanding the coverage of immunisation services in developing countries: a systematic literature review. Vaccine. 2005;23(13):1624–35. doi:10.1016/j.vaccine.2004.02.029.
- Williams N, Woodward H, Majeed A, Saxena S. Primary care strategies to improve childhood immunisation uptake in developed countries: systematic review. JRSM Short Rep. 2011;2:1–21.
- Whittaker K. Lay workers for improving the uptake of childhood immunization. Br J Community Nurs. 2002;7(9):474–79. doi:10.12968/bjcn.2002.7.9.10659.
- Wigham S, Ternent L, Bryant A, Robalino S, Sniehotta FF, Adams J. Parental financial incentives for increasing preschool vaccination uptake: systematic review. Pediatrics. 2014;134(4):e1117–28. doi:10.1542/peds.2013-3604.
- Batt K, Fox-Rushby JA, Castillo-Riquelme M. The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: systematic review of the grey literature. Bulletin of the World Health Organization. 2004;011437(04).
- Bassani DG, Arora P, Wazny K, Gaffey MF, Lenters L, Bhutta ZA. Financial incentives and coverage of child health interventions: a systematic review and meta-analysis. BMC Public Health. 2013;13(SUPPL.3):S30. doi:10.1186/1471-2458-13-S3-S30.
- Corace KM, Srigley JA, Hargadon DP, Yu D, MacDonald TK, Fabrigar LR. Using behavior change frameworks to improve healthcare worker influenza vaccination rates: a systematic review. Vaccine. 2016;34(28):3235–42. doi:10.1016/j.vaccine.2016.10.045.
- NP JMPMACSJP. Strategies to increase the demand for childhood vaccination in low-and middle-income countries: a systematic review and meta-analysis. TT -. Bull World Health Organ. 2015;93(5):339–46. doi:10.2471/BLT.14.146951.
- Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Carsley J, Franco ED, Delage G. Evaluation of the Effectiveness of Immunization Delivery Methods. Canadian Journal of Public Health / Revue Canadienne de Santé Publique. 2016;85:S14–S30.
- Lee C, Robinson JL. Systematic review of the effect of immunization mandates on uptake of routine childhood immunizations. J. Infect. 2016;72(6):659–66. doi:10.1016/j.jinf.2015.11.003.
- Peacock S, Konrad S, Watson E, Nickel D, Muhajarine N. Effectiveness of home visiting programs on child outcomes: a systematic review. BMC Public Health. 2011;13:1.
- Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. Emerg. Infect. Dis. 1996;3:399–409.
- R.E. T, M. R, D. L. Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database Syst. Rev. 2010;9(7):CD005188.
- Glenton C, Scheel IB, Lewin S, Swingler GH. Can lay health workers increase the uptake of childhood immunisation? Systematic review and typology. Trop. Med. Int. Heal. 2011;16(9):1044–53. doi:10.1111/j.1365-3156.2011.02813.x.
- Lewin S, Munabi-Babigumira S, Glenton C, Daniels K, Bosch-Capblanch X, Van Wyk B. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst. Rev. 2010;3(3):CD004015.
- Saeterdal I, Lewin S, Austvoll-Dahlgren A, Glenton C, Munabi-Babigumira S. Interventions aimed at communities to inform and/or educate about early childhood vaccination (Review). Cochrane Database Syst. Rev. 2014;6(11):1–95.
- Lytras T, Kopsachilis F, Mouratidou E, Papamichail D, Bonovas S. Interventions to increase seasonal influenza vaccine coverage in healthcare workers: a systematic review and meta-regression analysis. Hum Vaccin Immunother. 2016;12(3):671–81. doi:10.1080/21645515.2016.1208328.
- Z.S. L, B.A. H, Z.A. B. Community-based intervention packages for reducing maternal and neonatal morbidity and mortality and improving neonatal outcomes. Cochrane Database Syst. Rev. 2010;11(11):CD007754.
- Kaufman J, Synnot A, Ryan R, Hill S, Horey D, Willis N. Face to face interventions for informing or educating parents about early childhood vaccination. Cochrane Database Syst. Rev. 2013;7(5):1–89.
- Rashid H, Yin JK, Ward K, King C, Seale H, Booy R. Assessing interventions to improve influenza vaccine uptake among health care workers. Health Aff. 2016;35(2):284–92. doi:10.1377/hlthaff.2015.1087.
- Hollmeyer H, Hayden F, Mounts A, Buchholz U. Review: interventions to increase influenza vaccination among healthcare workers in hospitals. Influenza Other Respi Viruses. 2013;7(4):604–21. doi:10.1111/irv.12002.
- Arditi C, Rège-Walther M, Wyatt J, Durieux P, Burnand B. Computer-generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes (review). Cochrane Database Syst. Rev. 2012;8(12):1–99.
- Smulian EA, Mitchell KR, Stokley S. Interventions to increase HPV vaccination coverage: a systematic review. Hum Vaccin Immunother. 2016;12(6):1566–88. doi:10.1080/21645515.2016.1208328.
- Walling EB, Benzoni N, Dornfeld J, Bhandari R, BA S, Garbutt J. Interventions to improve hpv vaccine uptake: a systematic review. Pediatrics. 2016;138(1):e20153863–e20153863. doi:10.1542/peds.2015-3863.
- Fu LY, Bonhomme LA, Cooper SC, Joseph JG, Zimet GD. Educational interventions to increase HPV vaccination acceptance: a systematic review. Vaccine. 2014;32(17):1901–20. doi:10.1016/j.vaccine.2014.01.091.
- Giuffrida A, Gosden T, Forland F, Kristiansen I, Sergison M, Leese B. Target payments in primary care: effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. 2009;20(4):1–12.
- Jacob N, Coetzee D. Missed opportunities for immunisation in health facilities in Cape Town, South Africa. S. Afr. Med. J. 2015 Oct 10;105(11):917. doi:10.7196/SAMJ.2015.v105i11.10194.
- Poorman E, Gazmararian J, Parker RM, Yang B, Elon L. Use of text messaging for maternal and infant health: a systematic review of the literature. Matern Child Health J. 2015;19(5):969–89. doi:10.1007/s10995-014-1595-8.
- Ryman TK, Dietz V, Cairns KL. Too little but not too late: results of a literature review to improve routine immunization programs in developing countries. BMC Health Serv Res. 2008;8:1–11. doi:10.1186/1472-6963-8-134.
- Targonski PV, Poland GA. Review: patient reminder or recall systems improve immunization rates. ACP J. Club. 2003;139:18.
- Mureed S, Somronghtong R, Kumar R, Ghaffar A, Chapman RS. Enhanced immunization coverage through interventions for childhood cluster diseases in developing countries. J Ayub Med Coll Abbottabad. 2015;27(1):223–27.
- Wong VWY, Lok KYW, Tarrant M. Interventions to increase the uptake of seasonal influenza vaccination among pregnant women: a systematic review. Vaccine. 2016;34(1):20–32. doi:10.1016/j.vaccine.2015.11.020.
- Wang S, Smith H, Peng Z, Xu B, Wang W. Increasing coverage of hepatitis B vaccination in China: a systematic review of interventions and implementation experiences. Med. (United States). 2016;95:1–15.
- Oyo-Ita A, Wiysonge C, Oringanje C, Nwachukwu CEC, Oduwole O, Meremikwu MMM. Interventions for improving coverage of childhood immunisation in low- and middle-income countries (Review). Cochrane Database Syst. Rev. 2016;(7):CD008145.
- Bühn S, Mathes T, Prengel P, Wegewitz U, Ostermann T, Robens S, Pieper D. The risk of bias in systematic reviews tool showed fair reliability and good construct validity. J. Clin. Epidemiol. 2017;91:121–28. doi:10.1016/j.jclinepi.2017.06.019.
- Pollock M, Fernandes RM, Hartling L. Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions. BMC Med. Res. Methodol. 2017;17(1):48. doi:10.1186/s12874-017-0325-5.
- Gómez-García F, Ruano J, Gay-Mimbrera J, Aguilar-Luque M, Sanz-Cabanillas JL, Alcalde-Mellado P, Maestre-López B, Carmona-Fernández PJ, González-Padilla M, García-Nieto AV, et al. Most systematic reviews of high methodological quality on psoriasis interventions are classified as high risk of bias using ROBIS tool. J. Clin. Epidemiol. 2017;92:79–88. doi:10.1016/j.jclinepi.2017.08.015.
- Bosomprah S, Aryeetey GC, Nonvignon J, Adanu RM. A decomposition analysis of change in skilled birth attendants, 2003 to 2008, Ghana demographic and health surveys. BMC Pregnancy and Childbirth. 2014;14:415.
- Benmarhnia T, Huang J, Basu R, Wu J, Bruckner TA. Decomposition analysis of Black–White disparities in birth outcomes : the relative contribution of air pollution and social factors in California. Environmental Health Perspectives. 2010;1–7.
- Ndwandwe D, Uthman OA. Decomposing the gap in missed opportunities for vaccination between poor and non-poor in sub-Saharan Africa: a Multicountry Analyses. Hum Vaccin Immunother. 2018;1–7.
- Jaca A, Ndze VN, Wiysonge C, Jaca A, Ndze VN, Wiysonge C. International prospective register of systematic reviews: an overview of systematic reviews of interventions aimed at improving vaccination coverage Citation Review question Participants/population Intervention (s), exposure (s) Comparator (s)/con. 2017;1–4.
- Whiting P, Davies P, Savović J, Caldwell D, Churchill R. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews. Journal of Clinical Epidemiology. 2016;4(69):225–234.
- Lever J, Krzywinski M, Altman N. Principal component analysis. Nat. Publ. Gr. 2017;14:641–42.
Appendix I.
Table A1. AMSTAR checklist
Table A2. Quality assessment using ROBIS tool R
Appendix II.
The AMSTAR-based PCA data
Principal components/correlation Number of obs = 57.
Number of comp = 13.
Trace = 13.
Rotation: (unrotated = principal). Rho = 1.0000.
Principal components (eigenvectors).
Appendix III.
The ROBIS-based PCA data
Principal components/correlation Number of obs = 57.
Number of comp = 21.
Trace = 21.
Rotation: (unrotated = principal) Rho = 1.0000.
Principal components (eigenvectors).