Effective evaluation of projects on violence against women and girls

ABSTRACT

What makes evaluations useful for interventions on violence against women and girls (VAWG)? This article reports on a qualitative comparative analysis of 39 evaluations, and shows that it takes a combination of elements to produce good evaluation effects. It identifies eight configurations of conditions which have generated effective evaluations. Key elements were the evaluation context, the evaluators’ sensitivity to gender and to the participants’ rights and security, and consultation with persons considered to be the ultimate “beneficiaries” of the intervention. Both qualitative and quantitative approaches could lead to effective evaluation. The article concludes with recommendations for evaluation users.

KEYWORDS:

• Civil society – Participation
• Aid – Monitoring and Evaluation
• Gender and diversity
• Methods

Acknowledgements

We are grateful for the interest and support of our DFID counterparts and the External Reference Group that has accompanied our work (see note 7). We are also indebted to Julian Brückner (Humboldt University) who provided technical advice on QCA at different stages of the review, and to the highly qualified research assistants Miruna Bucurescu, Scout Burghardt, Astrid Matten, Sanja Kruse and Paula Pustulka, who coded thousands of pages of evaluation reports. The review would not have been possible without the time and effort generously contributed by more than 200 development and evaluation professionals, who responded to our survey, provided evaluation reports, gave interviews, pre-tested tools, facilitated contacts, and offered thoughtful comments on our blog (www.evawreview.de).

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Michaela Raab and Wolfgang Stuppert are evaluation specialists based in Germany. Michaela Raab has 30 years of international development and evaluation experience, particularly in the fields of social development and human rights. Wolfgang Stuppert is a social scientist and activist with more than 10 years of experience in quantitative and qualitative research projects. He is currently affiliated with Camino gGmbH, a social research firm in Berlin.

Notes

1 For the sake of easier readability, we use the terms “project” and “intervention” interchangeably. We are aware that some evaluations might focus on a specific intervention within a larger project, or that an evaluation might examine several projects that come together under a large-scale intervention.

2 “Sufficient” meaning that we had answers from those implementing the evaluated intervention, as well as from at least one other stakeholder – for instance, a donor representative or an evaluator.

3 We use the term “beneficiary” in quotation marks, as it frames the persons it designates as passive recipients rather than active stakeholders in a project. We have decided to use this term here to avoid confusion. In our research, “stakeholders” refers chiefly to those involved in commissioning, designing, and implementing evaluations (who, in most cases, are not the intended “beneficiaries” of the evaluated project).

4 All evaluations are listed in our review report, available at: http://r4d.dfid.gov.uk/pdf/outputs/misc_gov/61259-Raab_Stuppert_Report_VAWG_Evaluations_Review_DFID_20140626.pdf. Evaluation reports we found via a web search and those we have been authorised to share can be downloaded at: http://vawreview.blogspot.de/2014/01/evaluations-identified-for-first-coding.html.

5 The external reference group consisted of evaluation experts linked to organisations with a long track record in VAWG programming: Joëlle Barbot (CIDA), Krishna Belbase (UNICEF), Valeria Carou-Jones (UNFPA), Katie Chapman (DFID), Jennifer Leith (DFID), Helen Lindley (Womankind), Judith McFarlane, Jodi Nelson (Gates Foundation), Fiona Power (DFID), Amanda Sim (International Rescue Committee), Inga Sniukaite (UN Women), Joe Stephenson (DFID), and Jeanne Ward.

6 The percentages add up to more than 100% because some cases are covered by several paths, i.e. several cases are present in several paths.

7 We conducted a series of in-depth interviews with diverse stakeholders in five evaluations representing the most frequent paths, to understand in more detail how the conditions have interacted to generate effects.

Additional information

Funding

This work was supported by the UK Department for International Development under Grant 4007 4464: “Review of Evaluation Approaches and Methods for Violence Against Women and Girls Interventions” and under grant number PO 4007 7375.