Policy Evaluation in Polarized Polities: The Case of Randomized Controlled Trials

Abstract

This paper provides a political-economic analysis of policy evaluation. We focus on Randomized Controlled Trials (RCTs) as a subset of policy evaluations and argue that they are used instrumentally by decision-makers in order to improve perceptions of reforms and help secure policy legacy. We theorize that this ’credibility premium’ is more valuable for incumbents in politically polarized societies, which we empirically examine using two methods. First, we provide a series of vignettes of prominent randomized evaluations embedded by governments in policy roll-outs and a detailed case study of the Liberian government’s decision to commission a third-party RCT evaluation of a proposed primary school privatization reform. Second, we have compiled a unique cross-country panel data set on RCTs in development policy since 1996, with which we demonstrate that RCTs are more likely to occur in polarized societies, and that the effect is amplified by the degree of political competition.

Keywords:

• Randomized controlled trials
• RCTs
• political polarization
• political competition

Acknowledgements

The authors are gratefully acknowledge the helpful comments from seminar participants at the the Université Paris - Dauphine (DIAL), Université Libre de Bruxelles, the Université Paris I Panthéon Sorbonne, the Università di Roma La Sapienza, Higher School of Economics (St. Petersburg), and the Central European University (Budapest). In particular, the authors thank Mariyana Angelova, Ágnes Batory, Caitlin Brown, Gabriel Cepaluni, Quentin David, Amanda Driscoll, Andreas Madestam, Mikael Melki, Pierre-Guillaume Méon, Edward Miguel, Arieda Muco, and Anand Murugesan for comments and conversations that have improved this paper. Part of this research was carried out while Dorsch was visiting the Université Paris-Panthéon Assas and he thanks them for their hospitality and financial support. The authors thank the 3ie for providing us with the meta data from their online repository and for responding to our queries. Supplementary Materials can be found on the Corresponding Author’s personal website: https://sites.google.com/view/dorsch/research?authuser=0

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, MTD, upon reasonable request.

Notes

1 See Corduneanu-Huci et al. (2021) for an overview of how the supply side of RCTs may be affected by the political environment in the hosting country. They also show that a large share of RCTs are executed with governmental partners. See also the meta-study of Corduneanu-Huci, Dorsch, and Maarek (2022).

2 Technically, the measure is one minus the Herfindahl index of ethnic concentration. For country j, $\mathit{\text{fract}}{.}_j=1-{\sum }_{i=1}^N{s}_{\mathit{\text{ij}}}^2,$ where s_ij is the share of ethnic group i in the population of country j.

3 See Bhavnani and Miodownik (2009); Chandra (2005), and Mozaffar, Scarritt, and Galaich (2003) for examples of papers that use ethnic fractionalization as a proxy for political polarization, especially in developing economies.

4 The standard deviation of the vote share variable is 27.01, multiplied by the coefficient -0.002 in the high polarization sub-sample gives -0.054. The unconditional mean of the dependent variable is 0.188.