Multiple Testing and the Distributional Effects of Accountability Incentives in Education

Abstract

This article proposes bootstrap-based multiple testing procedures for quantile treatment effect (QTE) heterogeneity under the assumption of selection on observables, and shows its asymptotic validity. Our procedure can be used to detect the quantiles and subgroups exhibiting treatment effect heterogeneity. We apply the multiple testing procedures to data from a large-scale Pakistani school report card experiment, and uncover evidence of policy-relevant heterogeneous effects from information provision on child test scores. Furthermore, our analysis reinforces the importance of preventing the inflation of false positive conclusions because 63% of statistically significant QTEs become insignificant once corrections for multiple testing are applied.

Keywords:

• Accountability
• Bootstrap tests
• Information
• Multiple testing
• Quantile treatment effects
• Student performance

Acknowledgments

We thank Jishnu Das, Jeff Smith, and seminar and conference participants at the University of Georgia, Hunter College, the University of North Carolina Greensboro, RWI, Sciences Po Paris, Tilburg University, AEA, CLSRN, ESAM, ESNASM, and SOLE/EALE for helpful comments and suggestions. Jacob Schwartz and Thor Watson provided excellent research assistance. Computer code used to generate the results in this article are available in either Stata or MATLAB on request. We thank Co-Editor and two anonymous referees for valuable comments. All remaining errors are our own. Lehrer and Song, respectively, thank SSHRC for research support. An online appendix to this article providing further details regarding (1) additional motivation for the testing procedure, (2) the asymptotic validity of the multiple testing procedure, and (3) robustness checks for the main result is available at https://rvpohl.github.io/files/LehrerPohlSong_Multiple_App.pdf.

Additional information

Funding

Lehrer and Song, respectively, thank SSHRC for research support