Replication Studies of Development Impact Evaluations

Abstract

Six years ago, the International Initiative for Impact Evaluation (3ie) launched a programme to promote and fund replication studies of impact evaluations in international development. We designed the programme with the objective of improving the quality of evidence for development policy-making, using replication research to both validate the results of published impact evaluations and build the incentives for more transparent and high quality research going forward. The programme’s focus is internal replication, which uses the original data from a study to address the same question as that study. This Journal of Development Studies special issue compiles the majority of completed 3ie-funded replication studies initiated in the first years of the programme. In all cases the pure replication components of these studies are generally able to reproduce the results published in the original article. Most of the measurement and estimation analyses confirm the robustness of the original articles or call into question just a subset of the original findings. These replication studies mostly focus on providing additional information about the impacts of the interventions – especially additional information that can be important for interpreting the articles for the purpose of policy-making.

1. Introduction

Six years ago, the International Initiative for Impact Evaluation (3ie) launched a programme to promote and fund replication studies of impact evaluations in international development. We designed the programme with the objective of improving the quality of evidence for development policy-making, using replication research to both validate the results of published impact evaluations and build the incentives for more transparent and high quality research going forward.¹ The programme’s focus is internal replication, which is research that uses the original data from a study to address the same question as that study. We argued the need for internal replication research in international development and proposed three categories of replication exercises in Brown, Cameron, and Wood (2014). A major element of the programme to date has been the funding of replication research through competitive grants windows. This Journal of Development Studies special issue compiles the majority of completed 3ie-funded replication studies initiated in the first years of the programme.²

In Brown et al. (2014), we argue there are four challenges in social science that replication helps to solve. The first challenge is simply that to err is human. Every researcher makes mistakes. Internal replication studies can help to uncover and correct mistakes, and even just the increased probability of replication should improve the incentives of original authors (and their editors) to check for, and be up front about, mistakes in their work. Several of the replication studies in this issue indeed found mistakes, such as coding errors, as part of their pure replication exercises. However, these mistakes were generally not critical.

The second challenge is that no science is a perfect science. This challenge is particularly true when statistics and econometrics are used, as these methods require assumptions and they also evolve over time. Replication studies that include measurement and estimation analysis can check the robustness of analytical choices made in the original study. Methodological considerations around measurement are just as important to examine as those around estimation, especially in cases where qualitative concepts are represented with quantitative measures. All of the replication studies in this issue include measurement and estimation analysis, and while many of the checks support the robustness of the original results, there are some cases where the analyses attenuate the published results in important ways.

The third challenge is the challenge faced by researchers to publish or perish. Unfortunately, the desire among journals to publish articles that report both significant and statistically significant results can distort the incentives of researchers and editors, causing reporting bias for the former and publication bias for the latter. Pre-registration can help with both, but only if the authors elect to pre-register in advance of their analysis. Once an article is published, replication research can help by exploring the selectivity and robustness of those results reported by the researchers and accepted by the editors. We have seen cases among the 3ie-funded studies where, for example, original authors explained to replication researchers that the reason certain findings in the replication study were not included in the original article is because the journal editor asked for them to be cut.

The final challenge we call ‘policy recommendations please’. This challenge, which is the request or requirement by some research funders and journals for authors to draw policy conclusions from their results, may be particularly acute in international development where the stakeholders are eager to see concrete returns on research investments, and these conclusions can influence programme and research agendas. Our concern is that these requirements often push authors to conclude that their results support a particular theory of change (which then supports a particular policy or programme) without carefully considering other possible theories of change. We define theory of change analysis as an approach to replication research in order to distinguish it from the more neutral diagnostic approach (for more discussion of these two approaches, see Brown & Wood, 2018). Among the replication studies in this issue are several in which the replication researchers motivate their empirical analysis by carefully considering the purported theory of change and alternate theories of change for the programme or policy evaluated. Some of the results of these theory of change analyses do call into question popular policy recommendations.

Most agree that these challenges to social science research exist, and, following the increased attention around replication in the last decade, many also now agree that replication research can help to address them. Yet, the publication of replication studies in journals is still quite rare. Part of the problem is that most journal editors do not want to publish them. Duvendack, Palmer-Jones, and Reed (2015) reviewed statements on journal websites and emailed journal editors and found that only 10 of 333 economics journals explicitly publish replication studies. Even worse, journal editors are biased to only publishing replication studies that challenge or overturn the original results. Gertler, Galiani, & Romero surveyed the editors of the top 11 economics journals, in which only 11 internal replication studies have been published since 2011, and found that 35 of the 88 editors said they would publish a replication study that overturned the results of the original, while only nine said they would publish a confirmatory replication study. We know that for some of the studies in this issue, the researchers approached the journal in which the original article was published to inquire about submitting a comment based on the replication study, and they were turned down.

We are pleased to have the following eight replication studies accepted by Journal of Development Studies, and we hope that having these studies, which include some that are primarily confirmatory, published as a set will help the development research community see the bigger picture of what can be learned from replication research. In this introduction, we describe the features of the 3ie programme that funded these studies, summarise each of the studies included in this issue, and briefly discuss both what is learned from the studies and what we have learned from the experience of running the replication programme.

2. 3ie replication programme

The grants windows through which the majority of these studies were funded were simply named replication window 1 (RW1) and replication window 2 (RW2).³ To apply for these grants, replication researchers needed to select an original study from a list provided by 3ie. For RW1 and RW2, 3ie staff created the list with inputs from crowd sourcing through the 3ie website. We aimed to have a list that was representative across various sectors of development and included published articles that have been influential. By providing a candidate studies list instead of having applicants select studies entirely on their own, we hoped to reduce possible adverse incentives to produce a replication study that contradicts the results of the original article.

At the beginning of the grant period, each grantee was required to submit a replication plan, akin to a pre-analysis plan, which was posted on the 3ie website. Grantees produced these plans by revising the research plan portion of their grant applications according to comments from the application reviewers, the external project advisor, and 3ie staff. We then requested that grantees clearly explain in their final reports if, where, and why their analysis deviated from their plan.

As mentioned, another feature of the programme was that each grantee was assigned an external project advisor (a recognised researcher from academe or a research institution) who, along with 3ie staff, provided reviews at multiple stages of the replication research, including for the replication plan, the pure replication, and the draft final replication study. We included comments from an additional, anonymous, external reviewer at the draft final replication study stage. The agreement document for these grants included a requirement to adhere to the programme’s notification and communication policy, which set out standard procedures for communication between the replication researchers and the original authors.⁴ These procedures, which aimed to reduce frictions within the replication process, include sharing the pure replication results with the original authors upon completion of that first stage of the replication research.

Before publishing the final reports in our paper series, we worked with the replication researchers on their word choice and tone in an effort to make the replication studies non-combative (Brown & Wood, 2014). Also, as outlined in the notification and communication policy, we gave the original authors the opportunity to write a reply to the replication study when the final report was complete, and we posted the original author reply concurrently with each replication study. The original author replies to the final reports from which these eight articles are drawn may be found on the 3ie website.

The replication studies published here went through the same peer-review process as all articles submitted to the Journal of Development Studies with additional comments provided by the guest editors. The anonymous referees were not notified that the submissions were for a special issue on replication, and it is notable that three of the articles are published in note form here because the referees felt there was little merit in publishing replications that largely confirm the original studies.

3. Study summaries

Basurto et al.’s replication study ‘Walking on Solid Ground. A replication study of Cattaneo et al.’ re-examines Cattaneo, Galiani, Gertler, Martinez, and Titiunik (2009) research on housing improvements. The original article has been influential in Mexico where the intervention took place and is used by the NGO Un Techo para me Pais to promote their mission across Latin America. The original authors evaluate the effectiveness of a government intervention to replace dirt with cement flooring in Mexican households. They report statistically significant health improvements for families who receive cement floors. The original authors also find that in comparison to the control households, the households who received cement floors reported, on average, fewer signs of depression.

Basurto et al.’s pure replication generally reproduces the original results. The replication researchers conduct a number of measurement and estimation analyses. One of their checks tests the robustness of the original results to alternative imputation strategies for missing data, since the imputation strategies used in the original article have been shown to introduce bias. Basurto et al. find the published findings robust to their imputation reanalysis. Another of the replication researchers’ reanalyses explores possible heterogeneous impacts of the intervention based on the initial physical conditions of the households. They interact the treatment variable with households who had more than the median share of cement flooring at the time of baseline data collection. The results provide suggestive evidence that households with higher levels of cement floor coverage at baseline benefited less from this intervention than those with an initially lower level of cement flooring.

In ‘Heterogeneous effects of urban land titling: a replication of “property rights for the poor”’, Cameron, Whitney, and Winters’ replication study reexamines Galiani and Schargrodsky’s (2010) research on property rights. Galiani and Schardrodsky’s article is influential partly because it was among the first to question the theory that land rights benefit the poor through increased access to credit. The original authors examine a high profile government intervention that gave land rights to squatters in Argentina. They find that titling substantially increased housing investment. Furthermore, they determine that land titling instigated a number of social impacts through a slow channel of increased physical and human capital investment rather than expanded land collateralisation, as commonly assumed. Cameron, Whitney, and Winters reproduce the original results in their pure replication. In their measurement and estimation analysis, the replication researchers use an alternative propensity score estimation. They also include gender and education variables in their propensity scores and find very similar results to the original publication after conducting these robustness tests.

Cameron, Whitney, and Winters also explore gender and education variables in reanalysis focusing on the theory of change. They find that household improvements were more likely when the squatter was male. They also find that household improvements were more likely when the squatter had at least a primary level of education. Cameron, Whitney, and Winters caution that their theory of change results are speculative given a limited number of observations in the data, but nevertheless provide suggestive extensions to and qualifications of the original findings.

Carvalho and Rokicki’s replication study ‘The impact of India’s Janani Suraksha Yojana conditional cash transfer programme: A replication study’ reexamines Lim et al.’s (2010) research on India’s Janani Suraksha Yojana (JSY) intervention. JSY is a large-scale conditional cash transfer programme designed to incentivise pregnant women to use formal birthing facilities. Lim et al. use a coarsened exact matching approach to create the counterfactual comparison group in their evaluation. Their original findings identify large JSY programme effects on a number of pregnancy related outcomes including number of antenatal care visits, institutional deliveries, and skilled birth attendance. Carvalho and Rokicki’s pure replication closely reproduces the original results. They conduct a number of measurement and estimation analyses. In one analysis, they vary the estimation strategy by using propensity score matching instead of coarsened exact matching. The replication researchers find the original results robust to changes in matching estimation strategies.

In another analysis, Carvalho and Rokicki test the generalisability of the original findings by examining the results by Indian states. They use a state-level multi-level estimation strategy to review differences in JSY’s influence on health outcomes. Their multi-level estimator identifies state-level differences in health coverage and mortality outcomes. This finding suggests uneven implementation of the JSY programme across states.

Donato and Garcia Mosqueira’s replication study ‘Information improves provider behaviour: a replication study of a community-based monitoring programme in Uganda’ re-examines Björkman and Svensson’s (2009) work on local governance. In the original article, Björkman and Svensson evaluate whether a public information campaign that encouraged communities to monitor health care providers influenced provider service quality, community utilisation of the facilities, and subsequent health outcomes. Björkman and Svensson find improvements in service quality, facility utilisation, and some health outcomes. The original article has been highly influential, particularly among other researchers who have designed similar interventions based on Björkman and Svensson (2009). Raffler, Posner, and Parkerson (2018) test a dramatically scaled up version of the intervention, also implemented in Uganda.

Donato and Garcia Mosqueira reproduce the original results in their pure replication. In one of their measurement and estimation analyses, the replication researchers run a risk of bias check by examining the pre-intervention balance of the experiment. They test the pretreatment balance between the treatment and control populations with variables measuring participant wealth and vaccination rates. Donato and Garcia Mosqueira find balance on the wealth variables between treatment and control households. However, they also discover some imbalance between the immunisation rates of children in treatment and control households. In another one of their measurement and estimation analyses, the replication researchers account for the presence of community-based organisations (CBOs) in some of the treatment and control communities before the start of the experiment. After they control for presence of CBOs, Donato and Garcia Mosqueira find a decrease in some of the health outcomes results.

Iverson and Palmer-Jones’ replication study ‘All you need is cable TV?’ reassesses Jensen and Oster’s (2009) work on how access to cable TV influences the status of women in rural India. The original paper finds that the introduction of cable TV corresponded to decreases in the acceptability of domestic violence, fertility, and son preference and increases in women’s autonomy. During their pure replication process, Iversen and Palmer-Jones document some coding errors and at some points question the original authors’ variable construction. Ultimately, their pure replication reproduces some but not all of the published results. Some of their robustness analyses focus on the question of measurement. Iversen and Palmer-Jones deconstruct the aggregated indicators used in the original analysis and they conduct tests for gender and education heterogeneous effects. They conclude that cable television access has no influence on uneducated women. They also use additional data to re-construct some of the variables. Based on their analysis using the reconstructed variables, they question the link between cable TV and school enrolment. Another of their analyses explores possible theories of change for the intervention. Their review of the media effects literature questions the original recommendation of broadening access to cable TV across India. Instead Iversen and Palmer-Jones suggest further research be conducted to determine the causal mechanisms behind the observed associations between cable TV and changes in social behaviours and attitudes.

In ‘Evidence of behavioural compensation in internal replication of male circumcision trial to reduce HIV acquisition in Kisumu, Kenya,’ Korte, Djimeu, and Calvo conduct a replication study that reevaluates Bailey et al.’s (2007) work on HIV prevention. The original researchers examine the link between voluntary medical male circumcision (VMMC) and HIV infections. They determine that VMMC decreases the likelihood of HIV acquisition. The replication researchers’ pure replication finds very similar results to the original paper. In their measurement and estimation analyses, Korte et al. conduct a series of robustness checks. For one, they break the sample into two groups based on age and test for heterogeneous treatment effects. Their age reanalysis produces very similar results as the published findings. Korte et al. also use instrumental variables and other econometric based methods to assess the possibility of risk compensation among treatment individuals. The replication researchers detect the presence of risk compensation within the treatment group, a finding not discovered by the original authors. This finding suggests VMMC interventions may actually lead to more risky behaviour.

In ‘Information reduces corruption and improves enrolment (but not schooling): A replication study of a newspaper campaign in Uganda’, Kuecken and Valfort conduct a replication study of Reinikka and Svensson, (2005) research on social services governance. The original authors evaluate if a Ugandan anti-corruption newspaper campaign increased funding to schools. They find that the campaign improved school enrolments and significantly reduced leakages of funds. In their pure replication, Kuecken and Valfort reproduce the published results. In one of their measurement and estimation analyses, the replication researchers use additional data to test the parallel trends assumption of the original authors’ difference-in-difference estimation procedure. They determine that the lack of correlation between key variables before the intervention supports the validity of the parallel trends assumption. One of the hypotheses Kuecken and Valfort test in their theory of change analysis focuses on the number of teachers in different schools by geographic location. Examining the two separate periods of data analysed in the original evaluation, they find suggestive evidence that the newspaper campaign generated a fairer distribution of teachers across rural and urban schools.

Wood and Dong’s replication study ‘Recalling extra data: A replication study of Finding Missing Markets’ reexamines Ashraf, Giné, and Karlan’s (2009) work on agricultural commercialisation. The original authors analyse the influence of specific crop incentive programmes in Kenya on household income. They find that new cash crop adopters experienced statistically significant increases in income levels. Wood and Dong’s pure replication identifies some typological errors in the published results but generally reproduces the findings. The original authors increased their sample size by expanding their endline sample and asking those additional households to recall their household responses from the time of the baseline. One of the focuses of Wood and Dong’s measurement and estimation analysis is the use of these recall data. The replication researchers separate the baseline data into the original baseline observations and the baseline observations created through recall surveys. They identify a number of statistically significant differences between these two samples. Wood and Dong also test an alternative theory of change. The original authors argue that the incentive programmes lead farmers to grow specific cash crops, and these crops lead to the observed increases in income. Wood and Dong explore whether selling any crops at markets rather than, or in addition to, adopting new crops might explain improved outcomes. They find that market participation does also increase household incomes.

4. What did we learn from the studies?

In spite of the sometimes angry responses from the original authors, the replication studies here are often supportive of the original articles. In all cases the pure replications are generally able to reproduce the results published in the original article, and some of the exceptions are coding or typographical errors. Even most of the measurement and estimation analyses confirm the robustness of the original articles or call into question just a subset of the original findings. Rather than seeking to ‘overturn’ the original findings, these replication studies mostly focus on providing additional information about the impacts of the interventions – especially additional information that can be important for interpreting the articles for the purpose of policy-making.

For example, by decomposing indexes and looking at heterogeneous outcomes, Iverson and Palmer-Jones suggest that the theory of change for how cable TV affects women in India is more complex, and possibly different, than we may have thought. This is important to understand if we want to consider cable TV as a mechanism to be used for change. Wood and Dong’s exploration of the recall data in Ashraf et al. (2009) does raise concerns about the risk of bias in the original article, but their results also suggest, like Iverson and Palmer-Jones’, that the theories at play are more complicated than we thought.

These replication studies also highlight the usefulness of reanalysing data with alternative estimation strategies. Korte, Djimeu, and Calvo use an instrumental variables approach instead of the traditional health estimation methods and, contrary to the original article, find evidence suggestive of risk compensation, which has been an important policy concern. Their finding is a clear example of the potential for cross-disciplinary methodological approaches to replication research. Other researchers, like Cameron, Whitney, and Winters, test the robustness of published findings to different specifications of the original estimation strategies. Their conclusion that the original results are robust to an alternative propensity score matching specification should provide the development community with additional confidence in the published findings.

Some of the replication researchers focus on pre-intervention household and community characteristics as a way to provide added policy-maker relevance to their research. Kuecken and Valfort report suggestive evidence that the anti-corruption campaign resulted in a fairer distribution of teachers to rural areas. Donato and Garcia Mosqueira conclude that pre-intervention wealth did not influence intervention outcomes but that accounting for baseline immunisation rates decreases the strength of the published health findings. Basurto et al.’s suggestive evidence of differential effects of the intervention based on existing household conditions might support a more targeted approach to similar future interventions. These replication studies provide policy-makers with more nuanced guidance about what interventions work, where, and for whom.

An issue of great importance to many policy-makers is generalisability. Researchers often focus their studies on one geographic level. Carvalho and Rokicki’s state-level analysis of a national intervention demonstrates the importance of accounting for heterogeneity in programme implementation. Their findings suggested the averaging of the health outcome results might be understating the impact. Carvalho and Rokicki’s replication study may provide additional confidence to policy-makers considering this type of intervention.

The studies presented here also provide some great examples of replication exercises that replication researchers can conduct. In Brown and Wood (2018) we present a diagnostic approach to replication research that includes four categories: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. We use examples from the papers published here and others to illustrate the approaches that replication researchers might consider when developing their replication plans.

5. What did we learn from the programme?

3ie established the replication programme, and we designed the policies and practices of the programme, with the belief that a third party operating between original authors and replication researchers could help to change the contentious culture of internal replication research. To be frank, sometimes it still seems that it is impossible to change that culture in practice, even though many researchers now claim to be proponents of replication. In Brown et al. (2014) we briefly introduce the various incentive issues for original authors, replication researchers, and journal editors that hinder replication research. Our experiences over the last six years have thrown those issues into stark relief.

In Brown and Wood (2016) we lay out in more detail the major sources of tension in replication research, including conceptual differences, conflicting incentives, and claims about ethics, and we examine whether a third party can resolve or at least minimise these sources of tension.⁵ Journals often have not helped. At one extreme, they let original authors post corrections on personal websites without acknowledging the replication researchers who identified the mistakes instead of the journal publishing a comment (or full article) authored by the replication researchers. At the other extreme, as found by Gertler, Galiani, and Romero (2018), they favour publishing replication research that overturns the results of the original, although it is rarely the same journal that publishes the replication study. We saw this example when the Davey, Aiken, Hayes, and Hargreaves (2015) and Aiken, Davey, Hargreaves, and Hayes (2015) replication studies challenging some of the results of the highly influential Miguel and Kremer (2004) evaluation of mass deworming in Kenya were published, while less contradictory 3ie-funded replication studies were rejected for publication by journals. Even for this supplement, we were forced to publish some of the studies as notes because referees felt the findings were too confirmatory. These extremes further place original authors and replication researchers at odds with each other, and exacerbate the incentives and reactions of both sides.

We also found that the ‘policy recommendations please’ challenge for original authors seems to exacerbate their resistance to replication research. Understandably, the cost of acknowledging concerns raised in replication studies can be particularly high to authors who widely promote the policy recommendations of their studies, or for others who widely promote these recommendations. The solution needs to include both less reliance on single studies and more replication research before any single study becomes highly influential.

Our back and forth with original authors and replication researchers did lead to much good. We developed the notification and communication policy in response to, and greatly informed by, a conflict that arose early on in the programme when a replication study presented at a seminar hosted by 3ie was confused for a study funded by 3ie. When the first grant-funded replication studies were complete, we hosted a consultation event bringing together proponents of replication with some of the critics of our programme. That event motivated later innovations to the programme, including the development of a push button replication (PBR) protocol and adding PBR requirements to all 3ie funded replication studies. PBR is a check on whether the original authors’ programme code can be run on the original data to reproduce the published results (see Wood, Müller, and Brown, 2018 for more details).

Criticism of the programme by one group of researchers led them to conduct a very revealing survey of journal editors, which provides evidence that journal editors are more likely to publish replication studies that overturn the original articles than that confirm them (Gertler et al., 2018). This evidence does not solve the problem, but helps shine a light on it. The public debates also had positive spillovers. The worm wars debate ignited by the Davey et al. (2015) and Aiken et al. (2015) replication studies motivated useful comparisons of how different fields conduct empirical analysis as well as thoughtful critiques of how researchers report their results and address corrections (see, for example, Powell-Jackson et al., 2018; Humphreys, 2015).

There is growing support among scientists and social scientists for research transparency, including replication. Perhaps more pertinent, there is growing demand from policy-makers and research funders for research transparency. If scientists want their results to be credible, they need to accept that independent verification, through replication, is a part of the scientific process. If replication is to become the norm, however, researchers need incentives to conduct replication research. A big incentive is publication. This special issue of the Journal of Development Studies is an important step, but there is a long way to go.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Brown directed the 3ie Replication Programme from its beginning until 2016 when she left 3ie. Wood managed the 3ie Replication Programme starting in 2012, and served as the director from Brown’s departure until July 2018.

2. One study has already been published as two articles (Aiken et al., 2015; Davey et al., 2015).

3. One of the studies (Iverson and Palmer-Jones) was funded as a pilot and followed different procedures. Two of the studies (Korte, Djimeu, & Calvo and Wood & Dong) were conducted by 3ie staff but did follow the same procedures other than the initial grant application.

4. The current version may be found here: http://www.3ieimpact.org/media/filer_public/2016/06/28/rw-fsp-contracts-notification-communication-policy.pdf.

5. Brown (2016) elaborates on the idea of replication ethics.