The challenges of screening and synthesising qualitative research in a mixed-methods systematic review. The case of the impact of agricultural certification schemes

ABSTRACT

The number of mixed-methods systematic reviews in international development is growing in recent years. By recognising the value of qualitative research in providing valuable evidence on causal mechanisms, barriers, facilitators and the importance of context, mixed-methods systematic reviews go beyond the ‘what works’ question. However, appropriate methods to screen and synthesise qualitative evidence in these reviews are still in a development phase, and the methodological literature dealing with reviewing qualitative evidence in the field of development studies is scarce and under-developed. This paper aims to contribute to this gap by discussing the methodological and practical challenges of including qualitative evidence in a mixed-methods systematic review in international development. In particular, this article makes a contribution in terms of offering reviewers and users of systematic reviews a full account of the process of screening and synthesising a very large volume of heterogeneous qualitative studies. Using as an example a review on the effects of certification schemes for agricultural production, we report on each reviewing step, describing the problems encountered and solutions found. The paper proposes ways of extracting a large volume of data and integrating the qualitative synthesis with the evidence from the related quantitative effectiveness review.

KEYWORDS:

• Mixed-methods systematic review
• methodology
• agricultural certification
• qualitative synthesis

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. These types of questions are also called ‘process’ questions as they explore the ‘conditions of programme implementation and the mechanisms that mediate between processes and outcomes as a means to understand when and how programmes work' (Weiss 1997:41). See also Moore et al. (2014) on process evaluations in the context of health.

2. See Geertz (1973).

3. See Thomas and Harden (2008) on why it makes sense to search both quantitative and qualitative studies together before initiating separated screening.

4. However, there is an increasing tendency in the methodology literature to agree on basic principles of quality that can correspond to the standard validity criteria used for quantitative research. Hannes (2011) mentions the criteria of credibility (qualitative equivalent for internal validity); transferability (external validity); dependability (reliability); and confirmability (objectivity). Reflexivity and plausibility may also be considered in addition to the core criteria above, especially in cases where the epistemological orientation of the researcher prevents from conforming to the principle of confirmability (Bryman 2012).

5. Node is the term that Nvivo uses for ‘a collection of references about a specific theme, place, person or other area of interest’ (http://help-nv10.qsrinternational.com/desktop/concepts/about_nodes.htm).

Additional information

Funding

This work was supported by the International Initiative for Impact Evaluation [SR6.1158].

Notes on contributors

Dafni Skalidou

Dafni Skalidou is a PhD candidate at the School of International Development, University of East Anglia (UEA). She holds a MSc in Impact Evaluation from the UEA, a MA in International Development for the Polytechnic University of Catalonia and a BA in Banking and Financial Management from the University of Piraeus. She has worked with producers’ and consumers’ organisations in Ecuador, Bolivia, Mexico and Spain. In terms of research, Dafni has coordinated and conducted studies in cocoa and banana value chains in Cameroon, Ghana and Ecuador, and was involved in two Campbell Collaboration Systematic Reviews, as part of the 3ie-Systematic Reviews team and as a freelance consultant. She is currently finishing her doctoral research on placement and selection processes of cocoa certification programmes in the Ghanaian context.

Carlos Oya

Carlos Oya is Reader in the Political Economy of Development at SOAS, University of London. He has also lectured at the University Eduardo Mondlane in Mozambique, Universidad Complutense (Madrid), and at the Graduate Institute of International and Development Studies in Geneva. His main research interests are: employment, poverty, agrarian political economy, political economy of development, development policy, and research methodology. His current research interests include the assessment of the effectiveness of agricultural certification schemes in developing countries, and the employment effects of Chinese investments and firms in sub-Saharan Africa. Carlos has published widely in journals such as Journal of Development Studies, Journal of Agrarian Change, Journal of Peasant Studies, Feminist Economics, Journal of Development Effectiveness, Review of African Political Economy, Journal of Modern African Studies, and Third World Quarterly, and co-edited the 2015 Routledge volume Rural Wage Employment in Developing Countries. He is also co-editor of the Journal of Agrarian Change, a leading journal in agrarian political economy.