Case selection for robust generalisation: lessons from QuIP impact evaluation studies

ABSTRACT

What wider lessons can be drawn from a single impact evaluation study? This article examines how case study and source selection contribute to useful generalisation. Practical suggestions for making these decisions are drawn from a set of qualitative impact studies. Generalising about impact is a deliberative process of building, testing and refining useful theories about how change happens. To serve this goal, purposive selection can support more credible generalisation than random selection by systematically and transparently drawing upon prior knowledge of variation in actions, contexts, and outcomes to test theory against diverse, deviant and anomalous cases.

KEYWORDS:

• Aid – Monitoring and evaluation, Aid effectiveness, Accountability
• Methods

Acknowledgements

The paper was made possible by all those who contributed to QuIP studies, as acknowledged in Copestake, Morsink, and Remnant (2019). Gary Goertz, Steve Powell and Fiona Remnant also commented on an earlier draft. The author is a Director and co-founder of Bath Social and Development Research, a non-profit company set up, under licence to the University of Bath, to promote better evaluation through adaptation and use of the QuIP.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 For fuller reviews of methodological options, see Bond (2015) and Stern et al. (2012). Copestake, Morsink, Remnant (2019, Chapter 2) also compares the QuIP with 32 other approaches.

2 This “CMO” terminology does not map perfectly onto the “ZXY” shorthand used here, because project actions (X) are defined by realist evaluators as part of the context (C), while the term “mechanism” refers to the mostly unobservable cognitive causal links through which context (combining X and Z) generate outcomes Y (Blamey and Mackenzie 2007).

3 Interviews elicit narrative stories about drivers of change from individuals, while focus groups are organised by age and gender to elicit explanations for change experienced more widely within the location. Individual interviews are usually conducted first mainly to reduce contamination of narrative statements across sources.

4 This lies somewhere between the mechanical thematic coding of self-evidently real “diamonds in the sand” and the more creative “organic” coding of meanings (Braun and Clarke 2016).

5 As the dataset becomes larger it becomes harder for the analyst to select codes inductively in a way that is informed by immersion in all the data (Copestake, Davies, and Remnant 2019). For this reason, larger projects are better evaluated by more than one QuIP study, each coded independently, followed by meta-analysis across them.

6 Thematic saturation can in principle be measured. For example, Hagaman and Wutich (2016) found that “ … 16 or fewer interviews were enough to identify common themes from sites with relatively homogeneous groups.” However, such findings are likely to be highly context-specific (Braun and Clarke 2016). Smaller samples may be sufficient for narrowly focused confirmatory studies, as revealed in the earlier discussion of process tracing.

7 A large literature explores selection of cases within a 2 × 2 matrix comprising X = 1 or X = 0, and dY = 1 or dY = 0, and where X = 0 signifies non-participation, and dY = 1 signifies a positive outcome (Goertz 2017). QuIP mostly selects sources where X = 1, but selecting cases, where X = 0 but dY = 1 (equifinality) may also be useful.

Additional information

Funding

The original design and development of the QuIP was funded by research grant ES/J018090/1 jointly from the UK Department for International Development (DFID) and the Economic and Social Research Council (ESRC).

Notes on contributors

James Copestake

James Copestake is Professor of International Development at the University of Bath, UK, and has a particular interest in development finance modalities and their evaluation. He is also a trustee of INTRAC.