Thou shalt be given… but how? A replication study and extended cost-effectiveness analysis of a randomized experiment on food assistance in Northern Ecuador

ABSTRACT

The original cluster-randomised controlled trial took place in two provinces of Northern Ecuador. The trial compared cash, food or voucher assistance and found them to be effective. The results of the original paper are completely replicable and the findings of the original analysis are robust; however, we additionally find that treatments are more effective in the poorer province. We then extend the cost-effectiveness analysis by taking into account the parameter uncertainty in a probabilistic sensitivity analysis (PSA). Moreover, we provide information about the usefulness of additional research to reduce uncertainty concerning the cost-effectiveness analysis by calculating the expected value of perfect information (EVPI). Our extended cost-effectiveness analysis reveals that for some outcomes decision makers with a relatively small willingness-to-pay maybe a risk to choose a less cost-effective intervention. Furthermore, our findings indicate that further research into the comparative cost-effectiveness of these interventions could be worthwhile. This is in particular true for low-resource settings where decision makers face stricter budget constraints to finance interventions. In particular, policy makers with a small budget have a high risk of choosing a comparably less cost-effective treatment. Hence, further research to reduce uncertainty of effect estimates could aid decision-making.

KEYWORDS:

• Replication
• food assistance
• cost-effectiveness
• uncertainty

Acknowledgments

We would like to thank the participants of the 3ie workshop in Washington (D.C.) and the external project adviser. We also would like to thank the original authors, and in particular Melissa Hidrobo, for providing us with their data, code, and general support to carry out the replication analysis.

Disclosure statement

No potential conflict of interest was reported by the authors.

Replication code

The code used for the replication is available under the following URL https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/0W3VCA

Supplementary material

Supplemental data for this article can be accessed here

Notes

1. Barrios are existing administrative units within the urban centres with oversight over social services and other administrative functions.

2. This was confirmed by the authors in a personal communication.

3. Using 30% of the parameter values as a conservative estimate of the standard error is a rule-of-thumb approach that ensures that a parameter is still considered statistically significant, i.e. the confidence interval based on this assumed standard error does not overlap with zero. A slightly larger assumption about the standard error, e.g. 40% of the parameter in question, would translate into considering the parameter statistically not significant at the 1-% level anymore.

4. The net monetary benefit (NMB) is defined as NMB = WTP * Effect – Cost. That is, we calculate the value of the achieved effect for a decision maker’s willingness-to-pay and subtract the cost. If the WTP times the expected effect is larger than the costs, an intervention is cost effective for this level of WTP.

5. This was confirmed by the authors in a personal communication.

Additional information

Funding

This work was supported by the International Initiative for Impact Evaluation (3ie) under Replication Window 4.

Notes on contributors

Stefan K. Lhachimi

Stefan K. Lhachimi is head of the Research Group for Evidence Based Public Health. He obtained a Master in Public Policy from Duke University and a PhD in public health from Erasmus University Rotterdam. He has active experience in reviewing and drafting health economic evaluations for NICE (UK) and IQWiG (Germany) and he published widely on low- and middle-income countries.

Till Seuring

Till Seuring is a post-doctoral fellow at the Research Group for Evidence-Based Public Health.