Evaluation in Conflict Zones: Methodological and Ethical Challenges

Abstract

This article explores the methodological and ethical challenges particular to the conduct and use of evaluations in conflict zones. It does this through examining the synergistic interaction of conflict dynamics and the four domains of evaluation — ethics, methodology, logistics, and politics. Drawing on evaluation theory and practice, as well as field experience, the article seeks to contribute to the building of a more methodologically self-conscious sub-field of evaluation in conflict zones — with implications not only for the field of evaluation, but also for researchers and practitioners in the fields of development, humanitarianism, peacebuilding, and private sector investment.

Keywords:

• evaluation
• methodology
• ethics
• politics
• conflict zones
• fieldwork
• conflict theory
• peacebuilding
• development
• complexity

Notes

¹ Noteworthy recent initiatives include the establishment of the Learning Portal for Design, Monitoring and Evaluation for Peacebuilding (http://dmeforpeace.org/) and the US Institute for Peace funded Peacebuilding Evaluation Project (http://www.allianceforpeacebuilding.org/?page = workpep). A particularly important contribution to this field of work is the OECD's (2012) Evaluating Peacebuilding Activities in Settings of Conflict and Fragility.

² The term and idea of PCIA is introduced in Bush (1998). Elaboration is provided in Bush (2003; 2005). For a series of critical debates on the theory and practice of PCIA, see the two sets of commissioned essays in the Berghof Centre Handbook for Constructive Conflict Transformation: http://www.berghof-handbook.net/dialogue-series/

³ For example, the research undertaken, or supported, by International Alert and IDRC — see International Alert et al. (2004). This PCIA research has also been incorporated into programmes such as the PEACE III Programme in Northern Ireland. See: http://www.seupb.eu/Libraries/PEACE_III_Practical_Project_Guidlines/PIII_paper_practical_project_guidelines_090519__Aid_for_Peace_Approach.sflb.ashx

^*We would like to thank the following people for their contributions to refining our thinking about the issues addressed in this paper: Colin Knox; Rick Davies; the students of the evaluation modules we taught at the INCORE Summer School (2010–2012); the participants in workshops on ethics in conflict zones in the Community of Evaluators Meeting (Kathmandu, February 2013) and the African Evaluation Association Meeting (Accra, January 2012); and two anonymous peer reviewers.

⁴ Our use of the term ‘impact’ corresponds with the commonly accepted definition: ‘Positive and negative, primary and secondary long-term effects produced by [an] … intervention, directly or indirectly, intended or unintended’ (OECD 2002).

⁵ In development studies, see Scheyvens and Storey (2003) and Devereux and Hoddinott (1993). In the anthropology of violence, see Kovats-Bernat (2002) and Belousov et al. (2007). More practically, see UN (1998) and Cutts and Dingle (1995).

⁶ This is one of many stories emerging from our ongoing project on the ‘ethical tipping points of evaluators in conflict zones’. Evaluators spoke on the condition of anonymity. Skype Interview Date, 14 April 2013.

⁷ This passage is culled from the unpublished research prospectus prepared by Janaka Jayawickrama for a project on the evaluation of research in conflict zones (see Jayawickrama & Strecker, 2013).

⁸ This passage is culled from the unpublished research prospectus prepared by Janaka Jayawickrama for a project on the evaluation of research in conflict zones (see Bush & Duggan forthcoming).

⁹ Kenneth Bush, field notes, 2002.

¹⁰ The heart-breaking footnote to this case is that the parents of the village subsequently burned the school to the ground. As far as I know, the large NGO responsible for the project is still receiving funding from bilateral agencies for projects using the same flawed blueprint.

¹¹ Kenneth Bush interview with Father Paul Satkunayagam, Director and Co-founder of the Butterfly Peace Garden, February 2002.

¹² For excellent overviews, discussion, and tools of the panoply of approaches see Rick Davies, Monitoring and Evaluation News (http://mande.co.uk/) and The Learning Portal for Design, Monitoring, and Peacebuilding (http://dmeforpeace.org/).

¹³ Ibid.

¹⁴ The IDRC's outcome mapping is one example of a methodology that uses a systems approach to untangle the problems of evaluating research. Applying complexity theory has been more challenging mainly in terms of translating theory into a useable framework for practitioners. See Ramalingam et al. (2008) and Verkoren (2008).

¹⁵ This is not arguing (or supporting the argument) that militarised conflict zones are anarchic or lacking in social, political, or economic structures. Rather, we are arguing that such structures may be subordinated to, or transformed by, protracted dirty war.

¹⁶ D-2: ‘Abide by current professional ethics, standards, and regulations regarding confidentiality, informed consent, and potential risks or harms to participants’.

¹⁷ For a copy of the standards see: http://www.evaluationcanada.ca/site.cgi?en:6:10

¹⁸ http://www.eval.org/publications/guidingprinciples.asp

¹⁹ Also noteworthy in this context is Chapter 11, on ethics, in Church and Rogers (2006).

²⁰ For details, see: http://www.eval.org/publications/AJEcontribcats.asp