Theory-based systematic reviews

ABSTRACT

It has been common to say that studies should address not just the question of what works, but also how, where, for whom and at what cost? A unifying framework for such an approach is the theory of change. This paper lays out an approach for using such a theory-based approach to systematic reviews, discussing issues which arise in mixed-methods causal chain analysis. I illustrate the funnel of attrition which is a heuristic device to understand why effect sizes are lower at the higher reaches of the causal chain, including why participation is less than usually expected. Examples are given from the international development sector.

KEYWORDS:

• Systematic reviews
• impact evaluation
• theory-based approach
• causal chain analysis

Acknowledgements

Thanks to Hugh Waddington for encouraging me to write the paper and supplying examples, and to two anonymous reviewers for comments. The usual disclaimer applies.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. See, for example, http://www.tools4dev.org/resources/theory-of-change-vs-logical-framework-whats-the-difference-in-practice/.

2. This seems to be a common feature of Cochrane logic models. Indeed, the CIDG note states that a logic model ‘shows ‘how the different outcomes included in the review relate to each other’ (CIDG, n.d.: 3).

3. For more discussion of criticisms of being linear see White (2010).

4. This problem is identified by Dreze in his discussion of India’s midday meal programme (Drèze 2017).

5. Later social funds made payments via local authorities, but such designs emerged only in the late 90s.

6. See Goldacre (2012) for the same argument in the medical context.

7. Rumours that researchers collecting blood are vampires continue in many countries; see blog by Thomas de Hoop and Radhika Menon ‘Of rumours related to blood, poison and researchers’ http://blogs.3ieimpact.org/of-rumours-related-to-blood-poison-and-researchers/.

8. Protests against projects and project sabotage are under-documented and researched. See discussion in White (1999).

9. The confidence intervals are calculated using standard meta-analysis approaches.

10. Campbell standards used to recommend this focus, but now encourage collection indicators across the causal chain.

11. I am in agreement with Harden (2010) that it often makes less sense to talk about exclusively quantitative or qualitative studies as many studies contain both types of data and analysis.

12. http://www.dailymail.co.uk/news/article-1269463/Afghanistan-PowerPoint-slide-Generals-left-baffled-PowerPoint-slide.html. I am grateful to Rick Davies for bringing this example to my attention.

13. One way in which I have run this exercise is to say, suppose it is three years from now and the evaluation finds the programme to have had no impact, what might have gone wrong to explain this?.

14. The International Initiative for Impact Evaluation (3ie) requires researchers to submit annual reports during the course of their studies, which typically span 3–4 years. During my time at 3ie I would regularly read reports of problems in implementing both the intervention and the evaluation which were not mentioned at all in the final study. In one specific case I requested a researcher to include implementation evidence to help explain an anomalous finding, but he refused on the grounds that ‘no journal will require or even want that information’.

Additional information

Notes on contributors

Howard White

Howard White is CEO of the Campbell Collaboration. He has published widely on the effectiveness of aid and anti-poverty programmes, and evaluation and systematic review methods. He is former editor of the Journal of Development Effectiveness and Journal of Development Studies.