Mixing and matching: using qualitative methods to improve quantitative impact evaluations (IEs) and systematic reviews (SRs) of development outcomes

References

• 3ie. n.d.a. Checklist for Making Judgements about How Much Confidence to Place in a Systematic Review of Effect (Adapted Version of SURE Checklist). London: International Initiative for Impact Evaluation (3ie).
   Google Scholar
• 3ie. n.d.b. Principles of Impact Evaluation. New Delhi: International Initiative for Impact Evaluation (3ie).
   Google Scholar
• American Institutes for Research. 2017. Humanitarian Cash Transfers in the Democratic Republic of the Congo: Evidence from UNICEF’s ARCC II Programme. Washington, DC: Author.
   Google Scholar
• Anderson, L., M. Petticrew, E. Rehfuess, R., Armstrong, E., Ueffing, P., Baker, D., Francis, and P. Tugwell. 2011. “Using Logic Models to Capture Complexity in Systematic Reviews.” Research Synthesis Methods 2 (1): 33-42. https://doi.org/10.1002/jrsm.32 .
   PubMed Web of Science ®Google Scholar
• Bamberger, M. 2015. “Innovations in the Use of Mixed Methods in Real-World Evaluation.” Journal of Development Effectiveness 7 (3): 327–335. doi:10.1080/19439342.2015.1068832.
   Web of Science ®Google Scholar
• Bamberger, M., J. Rugh, and L. Mabry. 2012. Realworld Evaluation: Working under Budget, Time, Data, and Political Constraints. 2nd ed. California: SAGE Publications.
   Google Scholar
• Berg, R. C., and E. Denison. 2012. “Interventions To Reduce The Prevalence Of Female Genital Mutilation/Cutting In African Countries.” Campbell Systematic Reviews 2012: 9. doi:10.4073/Csr.2012.9.
   Google Scholar
• Bonilla, J., R. C. Zarzur, S. Handa, C. Nowlin, A. Peterman, H. Ring, and D. Seidenfeld. 2017. “Cash for Women’s Empowerment? A Mixed-Methods Evaluation of the Government of Zambia’s Child Grant Program.” World Development 95: 55–72. doi:10.1016/j.worlddev.2017.02.017.
   PubMed Web of Science ®Google Scholar
• Brody, C., T. De Hoop, M. Vojtkova, R. Warnock, M. Dunbar, P. Murthy, and S. Dworkin. 2017. “Can Economic Self-Help Group Programs Improve Women’s Empowerment? A Systematic Review.” Journal Of Development Effectiveness 9 (1): 15–40. doi:10.1080/19439342.2016.1206607.
   Web of Science ®Google Scholar
• Cameron, D., A. Mishra, and A. Brown. 2016. “The Growth of Impact Evaluation for International Development: How Much Have We Learned?” Journal of Development Effectiveness 8 (1): 1–21. doi:10.1080/19439342.2015.1034156.
   Web of Science ®Google Scholar
• Carr-Hill, R., C. Rolleston, R. Schendel, and H. Waddington. 2018. “The Effectiveness of School-Based Decision Making in Improving Educational Outcomes: A Systematic Review.” Journal of Development Effectiveness 10 (1): 61–94. doi:10.1080/19439342.2018.1440250.
   Web of Science ®Google Scholar
• Chinen, M., and M. Elmeski. 2016. Evaluation of The Transformative Potential Of Positive Gender Socialization In Education For Peacebuilding. Washington, D.C.: Air.
   Google Scholar
• Creswell, J. W., A. C. Klassen, V. L. Plano Clark, and K. C. Smith. 2011. Best Practices For Mixed Methods Research In The Health Sciences, 2094–2103. Bethesda (Maryland): National Institutes Of Health.
   Google Scholar
• Creswell, J. W. 2014. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. London: Sage Publications.
   Google Scholar
• Davies, D., and J. Dodd. 2002. “Qualitative Research and the Question of Rigor.” Qualitative health research 12 (2): 279–289. doi:10.1177/104973230201200211.
   PubMed Web of Science ®Google Scholar
• De Buck, E., H. Van Remoortel, K. Hannes, T. Govender, S. Naidoo, B. Avau, A. Vande Veegaete, et al. 2017. “Approaches to Promote Handwashing and Sanitation Behaviour Change in Low- and Middle-Income Countries: A Mixed Method Systematic Review.” Campbell Systematic Reviews 2017: 7. doi:10.4073/Csr.2017.7.
   Google Scholar
• Denzin, N. K., and Y. S. Lincoln. 2018. The Sage Handbook of Qualitative Research. 5th ed. Los Angeles: Sage.
   Google Scholar
• Evans, D., S. Hausladen, K. Kosec, and N. Reese. 2014. Community-Based Conditional Cash Transfers in Tanzania: Results from A Randomized Trial. Washington, DC: World Bank Publications.
   Google Scholar
• Gertler, P. J., S. Martinez, P. Premand, L. B. Rawlings, and C. M. J. Vermeersch. 2016. Impact Evaluation in Practice. Washington, DC: World Bank Group.
   Google Scholar
• Glynn, A. N., and N. Ichino. 2014. “Using Qualitative Information to Improve Causal Inference.” American Journal of Political Science 59 (4): 1055–1071. doi:10.1111/ajps.12154.
   Web of Science ®Google Scholar
• Greene, J. C., V. J. Caracelli, and W. F. Graham. 1989. “Toward A Conceptual Framework for Mixed-Method Evaluation Designs.” Educational Evaluation And Policy Analysis 11: 255–274. doi:10.3102/01623737011003255.
   Google Scholar
• Haynes, A., and F. Merttens. 2017. Evaluation of The Uganda Social Assistance Grants for Empowerment (SAGE) Programme (No.354). International Policy Centre for Inclusive Growth. Brazil: United Nations Development Programme(UNDP).
   Google Scholar
• Hombrados, J. G., and H. Waddington. 2012. Tool to Assess Risk of Bias and Internal Validity of Social Experiments and Quasi-Experiments. Mimeo. London: International Initiative For Impact Evaluation (3ie).
   Google Scholar
• Hulland, K., N. Martin, R. Dreibelbis, J. Debruicker Valliant, and P. Winch. 2015. “What Factors Affect Sustained Adoption of Safe Water, Hygiene and Sanitation Technologies?” In A Systematic Review of Literature. London: Eppi-centre, Social Science Research Unit, Ucl Institute Of Education, University College London
   Google Scholar
• Humphreys, M., and A. M. Jacobs. 2015. “Mixing Methods: A Bayesian Approach.” American Political Science Review 109: 653–673. doi:10.1017/S0003055415000453.
   Web of Science ®Google Scholar
• King, E., C. Samii, and B. Snilstveit. 2010. Interventions to Promote Social Cohesion in Sub-Saharan Africa. Synthetic Review 002. New Delhi: International Initiative For Impact Evaluation.
   Google Scholar
• Kneale, D., D. Gough, M. Bangpan, H. Waddington, and J. Thomas. 2018. Causal Chain Analysis in Systematic Reviews of International Development Interventions. Unpublished Manuscript. London: Centre of Excellence for Development Impact and Learning (Cedil).
   Google Scholar
• Langer, L. 2017. “Mixed-Methods Critical Appraisal Tool.” PhD diss.,University of Johannesburg. 1–19. Johannesburg: Africa Centre For Evidence.
   Google Scholar
• Langford, R., and C. Panter-Brick. 2013. “A Health Equity Critique of Social Marketing: Where Interventions Have Impact but Insufficient Reach.” Social Science & Medicine 83: 133–141. doi:10.1016/j.socscimed.2013.01.036.
   PubMed Web of Science ®Google Scholar
• Lawry, S., C. Samii, R. Hall, A. Leopold, D. Hornby, and F. Mtero. 2017. “The Impact of Land Property Rights Interventions on Investment and Agricultural Productivity in Developing Countries: A Systematic Review.” Journal of Development Effectiveness 9 (1): 61–81. doi:10.1080/19439342.2016.1160947.
   Web of Science ®Google Scholar
• Lewin, S., A. D. Oxman, J. N. Lavis, and A. Fretheim. 2009. “Support Tools for Evidence-Informed Health Policymaking (Stp) 8: Deciding How Much Confidence to Place in A Systematic Review.” Health Research Policy And Systems 7 (Suppl 1): S8. doi:10.1186/1478-4505-7-S1-S4.
   PubMedGoogle Scholar
• Masset, E., F. Rathinam, M. Nath., and B. Wood. 2018. Successful Impact Evaluations: Lessons from DFID and 3ie. CEDIL Inception Paper. London: Centre of Excellence for Development Impact and Learning (CEDIL).
   Google Scholar
• Maxwell, J., M. Chmiel, and S. Rogers. 2015. “Designing Integration in Multimethod and Mixed Methods Research.” In The Oxford Handbook of Multimethod and Mixed Methods Research Enquiry, edited by S. Hesse-Biber and R. Johnson. 1st ed. 1–70. New York: Oxford University Press.
   Google Scholar
• Miles, M. B., and A. M. Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. California: Sage Publications.
   Google Scholar
• Mmari, K., and T. J. H. University. 2006. “Using Qualitative Methods for Monitoring and Evaluation.”Power point presentation presented as a part of the course, Fundamentals of Program Evaluation at Johns Hopkins University. Accessed 24th October 2018. http://ocw.jhsph.edu/courses/FundamentalsProgramEvaluation/PDFs/Lecture11.pdf
   Google Scholar
• Munro, S. A., S. A. Lewin, H. J. Smith, M. E. Engel, A. Fretheim, and J. Volmink. 2007. “Patient Adherence To Tuberculosis Treatment: A Systematic Review Of Qualitative Research.” Plos Med 4 (7): E238. doi:10.1371/journal.pmed.0040238.
   PubMed Web of Science ®Google Scholar
• Nielsen, N., K. Godden, P. Leguene, D. Ruegenberg, and J. Rüdiger. 2010. Wfp Cambodia School Feeding 2000–2010: A Mixed Method Impact Evaluation. Rome: World Food Programme Office Of Evaluation.
   Google Scholar
• Nisbett, N., R. Longhurst, I. Barnett, F. Feruglio, J. Gordon, J. Hoddinott, F. Jahan, N. Karachiwalla, S. Roy, and V. Shah. 2016. Impact Evaluation of The Dfid Programme to Accelerate Improved Nutrition for The Extreme Poor in Bangladesh. Brighton: Maximising the Quality of Scaling Up Nutrition Programme (MQSUN)
   Google Scholar
• Noblit, G. W., and R. D. Hare. 1988. Meta-Ethnography: Synthesizing Qualitative Studies. Qualitative Research Methods. Vol. 11. Newbury Park, California: Sage Publications.
   Google Scholar
• Noyes, J., J. Popay, A. Pearson, K. Hannes, and A. Booth, On Behalf of the Cochrane Qualitative Research Methods Group. 2011. “Qualitative Research and Cochrane Reviews.” Chapter 20 in Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0, edited by J. P. T. Higgins and S. Green. London: The Cochrane Collaboration.
   Google Scholar
• Pawson, R., T. Greenhalgh, G. Harvey, and K. Walshe. 2005. “Realist Review – a New Method of Systematic Review Designed for Complex Policy Interventions.” Journal of Health Services Research & Policy 10: 21–34. doi:10.1258/1355819054308530.
   PubMedGoogle Scholar
• Phillips, D., H. Waddington, and H. White. 2015. “Better Targeting of Farmers as a Channel for Poverty Reduction: A Systematic Review of Farmer Field Schools Targeting.” Development Studies Research 1 (1): 113–136. doi:10.1080/21665095.2014.924841.
   Google Scholar
• Piza, C., T. Cravo, L. Taylor, L. Gonzalez, I. Musse, I. Furtado, A. C. Sierra, and S. Abdelnour. 2016. Business Support for Small and Medium Enterprises in Low- and Middle-Income Countries: A Systematic Review, 3ie Systematic Review. Vol. 25. London: International Initiative For Impact Evaluation.
   Google Scholar
• Pluye, Pierre, E. Robert, M. Cargo, Gillian Bartlett, A. O’cathain, Frances Griffiths, Felicity Boardman, Marie-Pierre Gagnon, and M. C. Rousseau. (2011). “Proposal: A mixed methods appraisal tool for systematic mixed studies reviews.” 1–8. Montréal: McGill University.
   Google Scholar
• Pullin, A. S., M. Bangpan, S. Dalrymple, K. Dickson, N. R. Haddaway, J. R. Healey, H. Hauari, et al. 2013. “Human Well-being Impacts of Terrestrial Protected Areas.” Environmental Evidence 2 (19): 1–41. doi:10.1186/2047-2382-2-19.
   Google Scholar
• Rosenfield, P. L. 1992. “The Potential of Transdisciplinary Research for Sustaining and Extending Linkages between the Health and Social Sciences.” Social Science & Medicine 35: 1343–1357. doi:10.1016/0277-9536(92)90038-R.
   PubMed Web of Science ®Google Scholar
• Schulte-Mecklenbeck, M., A. Kühberger, and J. G. Johnson, eds. 2011. A Handbook of Process Tracing Methods for Decision Research: A Critical Review and User’s Guide. Psychology Press.
   Google Scholar
• Shea, B., B. C. Reeves, G. Wells, M. Thuku, C. Hamel, J. Moran, D. Moher, et al. 2017. “Amstar 2: A Critical Appraisal Tool for Systematic Reviews that Include Randomised or Non-Randomised Studies of Healthcare Interventions, or Both.” British Medical Journal 358: J4008. doi:10.1136/bmj.j4008.
   PubMed Web of Science ®Google Scholar
• Skalidou, D., and C. Oya. 2018. “The Challenges of Screening and Synthesising Qualitative Research in a Mixed-Methods Systematic Review. The Case of the Impact of Agricultural Certification Schemes.” Journal of Development Effectiveness 10 (1): 39–60. doi:10.1080/19439342.2018.1438495.
   Web of Science ®Google Scholar
• Snilstveit, B. 2012. “Systematic Reviews: From ‘Bare Bones’ Reviews to Policy Relevance.” Journal of Development Effectiveness 4 (3): 388–408. doi:10.1080/19439342.2012.709875.
   Web of Science ®Google Scholar
• Sutcliffe, K., J. Thomas, G. Stokes, K. Hinds, and M. Bangpan. 2015. “Intervention Component Analysis (ICA): A Pragmatic Approach for Identifying the Critical Features of Complex Interventions.” Systematic Reviews 4: 140. doi:10.1186/s13643-015-0126-z.
   PubMedGoogle Scholar
• Thomas, J., A. Harden, A. Oakley, S. Oliver, K. Sutcliffe, R. Rees, G. Brunton, and J. Kavanagh. 2004. “Integrating Qualitative Research with Trials in Systematic Reviews.” British Medical Journal 28: 1010. doi:10.1136/Bmj.328.7446.1010.
   Google Scholar
• Thorne, S., J. Stephens, and T. Truant. 2016. “Building Qualitative Study Design Using Nursing’s Disciplinary Epistemology.” Journal of advanced nursing 72 (2): 451–460. doi:10.1111/jan.2016.72.issue-2.
   PubMed Web of Science ®Google Scholar
• Waddington, H., B. Snilstveit, J. G. Hombrados, M. Vojtkova, D. Phillips, P. Davies, and H. White. 2014. “Farmer Field Schools for Improving Farming Practices and Farmer Outcomes in Low-And Middle-Income Countries: A Systematic Review.” Campbell Systematic Reviews, 1–336.
   Google Scholar
• Waddington, H., E. Masset, and E. Jimenez. 2018. “What Have We Learned From 10 Years of Systematic Reviews in International Development?” Journal Of Development Effectiveness 10 (1): 1–16. doi:10.1080/19439342.2018.1441166.
   Web of Science ®Google Scholar
• Watson, J., R. Byrne, J. Opazo, F. Tsang, M. Morgan-Jones, and S. Diepeveen. 2011. “What Are the Major Barriers to Increased Use of Modern Energy Services Among the World’s Poorest People and are Interventions to Overcome These Effective?” Cee Protocol 11-004. Collaboration For Environmental Evidence. www.Environmentalevidence.Org/Sr11004.Html
   Google Scholar
• Welch, V. A., E. Ghogomu, A. Hossain, S. Awasthi, Z. A. Bhutta, C. Cumberbatch, R. Fletcher, et al. 2016. “Deworming and Adjuvant Interventions for Improving the Developmental Health and Well-Being of Children in Low- and Middle-Income Countries: A Systematic Review and Network Meta-Analysis.” Campbell Systematic Reviews 2016: 7. doi:10.4073/csr.2016.7.
   Google Scholar
• White, H. 2008. “Of Probits and Participation: The Use of Mixed Methods in Quantitative Impact Evaluation.” IDS Bulletin 39: 98–109. doi:10.1111/j.1759-5436.2008.tb00436.x.
   Web of Science ®Google Scholar
• White, H. 2009. “Theory-Based Impact Evaluation: Principles and Practice.” Journal of Development Effectiveness 1 (3): 271–284. doi:10.1080/19439340903114628.
   Web of Science ®Google Scholar
• White, H. 2018. “Theory Based Systematic Reviews.” Journal Development Effectiveness 10: 1. doi:10.1080/19439342.2018.1439078.
   Web of Science ®Google Scholar
• White, H., and D. A. Raitzer. 2017. Impact Evaluation of Development Interventions: A Practical Guide. Manila: Asian Development Bank. doi:10.22617/Tcs179188-2.
   Google Scholar
• White, H., and H. Waddington. 2012. “Why Do We Care about Evidence Synthesis? An Introduction to The Special Issue on Systematic Reviews.” Journal of Development Effectiveness 4 (3): 351-358. doi: 10.1080/19439342.2012.711343.
   Web of Science ®Google Scholar