Representing theories of change: technical challenges with evaluation consequences

ABSTRACT

This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation. The paper then outlines a range of different ways of addressing these problems that could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.

KEYWORDS:

• Evaluation
• theory of change
• evaluability
• representation

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. https://en.wikipedia.org/wiki/Adaptive_radiation .

2. http://www.kstoolkit.org/Theory+of+Change+%26+Theory+of+Action .

3. The one exception to the need for a Theory of Change might be Goal-Free Evaluation (Scriven 1991).

4. As can be seen in a Google Image search, Theories of Change are sometimes represented in more metaphoric forms, using landscapes, houses, trees etc. They suffer from essentially the same problems as seen with more diagrammatic representations discussed here.

5. For my criticism of the text content of ‘boxes’, especially in Logical Framework models, see http://mande.co.uk/category/lists/the-logical-framework/#editor .

6. Along with dates and who participated in the revisions. This will create a trail of evidence on how change was perceived and managed over the course of a given intervention (Shaw 2018).

7. Insufficient but Necessary part of a combination that is Unnecessary but Sufficient.

8. For example, by well exceeding a targeted outcome value.

9. For a review of 30 packages available in 2015, https://www.kdnuggets.com/2015/06/top-30-social-network-analysis-visualization-tools.html .

10. https://en.wikipedia.org/wiki/Centrality .

11. Betweenness Centrality is the number of the shortest paths between other nodes in a network that pass through a given node. See https://en.wikipedia.org/wiki/Betweenness_centrality .

12. https://en.wikipedia.org/wiki/Confusion_matrix .

13. Selective as informed by a clear case selection strategy e.g. https://evalc3.net/how-it-works/within-case-analysis/ .

14. https://en.wikipedia.org/wiki/Fuzzy_cognitive_map .

15. The exceptions being varieties of goal free evaluation (Scriven 1991).

Additional information

Funding

This paper was contracted as an Inception Paper for the Centre of Excellence for Development Impact and Learning (CEDIL), UK.

Notes on contributors

Rick Davies

Dr Rick Davies is an independent Evaluation Consultant, based in Cambridge, United Kingdom. His work focuses on international development aid programmes funded or implemented by bilateral and multilateral aid agencies, and NGOs.