Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions That Need to Be Powered to Detect Small Impacts

References

• Agodini, R., & Harris, B. (2010). An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness, 3(3), 199–253. https://doi.org/10.1080/19345741003770693
   Google Scholar
• Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567(7748), 305–307. https://doi.org/10.1038/d41586-019-00857-9
   PubMed Web of Science ®Google Scholar
• Armstrong, T. B., & Kolesár, M. (2018). Optimal inference in a class of regression models. Econometrica, 86(2), 655–683. https://doi.org/10.3982/ECTA14434
   Web of Science ®Google Scholar
• Armstrong, T. B., & Kolesár, M. (2020). Simple and honest confidence intervals in nonparametric regression. Quantitative Economics, 11(1), 1–39. https://doi.org/10.3982/QE1199
   Web of Science ®Google Scholar
• Balu, R., Zhu, P., Doolittle, F., Schiller, E., Jenkins, J., & Gersten, R. (2015). Evaluation of response to intervention practices for elementary school reading. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Barreca, A. I., Lindo, J. M., & Waddell, G. R. (2016). Heaping-induced bias in regression discontinuity designs. Economic Inquiry, 54(1), 268–293. https://doi.org/10.1111/ecin.12225
   Web of Science ®Google Scholar
• Bartalotti, O. (2019). Regression discontinuity and heteroskedasticity robust standard errors: Evidence from a fixed-bandwidth approximation. Journal of Econometric Methods, 8(1), 1–26. https://doi.org/10.1515/jem-2016-0007
   Google Scholar
• Bartalotti, O., Calhoun, G., & He, Y. (2017). Bootstrap confidence intervals for sharp regression discontinuity designs. In M. D. Cattaneo & J. C. Escanciano (Eds.), Advances in econometrics: Regression discontinuity designs theory and applications (Vol. 38, pp. 421–453). Emerald Publishing Limited.
   Google Scholar
• Bloom, H. S. (1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556. https://doi.org/10.1177/0193841X9501900504
   Web of Science ®Google Scholar
• Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). Russell Sage Foundation.
   Google Scholar
• Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59. https://doi.org/10.3102/0162373707299550
   Web of Science ®Google Scholar
• Branson, Z., Rischard, M., Bornn, L., & Miratrix, L. W. (2019). A nonparametric Bayesian methodology for regression discontinuity designs. Journal of Statistical Planning and Inference, 202, 14–30. https://doi.org/10.1016/j.jspi.2019.01.003
   Web of Science ®Google Scholar
• Calonico, S., Cattaneo, M. D., & Farrell, M. H. (2020). Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs. The Econometrics Journal, 23(2), 192–210. https://doi.org/10.1093/ectj/utz022
   Web of Science ®Google Scholar
• Calonico, S., Cattaneo, M., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326. https://doi.org/10.3982/ECTA11757
   Web of Science ®Google Scholar
• Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading and mathematics software products: Findings from two student cohorts. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Cattaneo, M. D., Frandsen, B. R., & Titiunik, R. (2015). Randomization inference in the regression discontinuity design: An application to party advantages in the U.S. Senate. Journal of Causal Inference, 3(1), 1–24. https://doi.org/10.1515/jci-2013-0010
   Web of Science ®Google Scholar
• Cattaneo, M. D., Titiunik, R., & Vasquez-Bare, G. (2017). Comparing inference approaches for RD designs: A reexamination of the effect of Head Start on child mortality. Journal of Policy Analysis and Management, 36(3), 643–681. https://doi.org/10.1002/pam.21985
   PubMed Web of Science ®Google Scholar
• Chen, Y., Feng, S., Heckman, J. J., & Kautz, T. (2020). Sensitivity of self-reported noncognitive skills to survey administration conditions. Proceedings of the National Academy of Sciences, 117(2), 931–935. https://doi.org/10.1073/pnas.1910731117
   PubMed Web of Science ®Google Scholar
• Chiang, H., Wellington, A., Hallgren, K., Speroni, C., Herrmann, M., Glazerman, S., & Constantine, J. (2015). Evaluation of the Teacher Incentive Fund: Implementation and impacts of pay-for-performance after two years. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
   Google Scholar
• Constantine, J., Player, D., Silva, T., Hallgren, K., Grider, M., & Deke, J. (2009). An evaluation of teachers trained through different routes to certification, final report. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Deke, J., & Dragoset, L. (2012). Statistical power for regression discontinuity designs in education: Empirical estimates of design effects relative to randomized controlled trials. Mathematica Policy Research.
   Google Scholar
• Gelman, A., & Imbens, G. (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447–456.
   Web of Science ®Google Scholar
• Greenberg, D., & Barnow, B. S. (2014). Flaws in evaluations of social programs: Illustrations from randomized controlled trials. Evaluation Review, 38(5), 359–387. https://doi.org/10.1177/0193841X14545782
   PubMed Web of Science ®Google Scholar
• He, Y., & Bartalotti, O. (2020). Wild bootstrap for fuzzy regression discontinuity designs: Obtaining robust bias-corrected confidence intervals. The Econometrics Journal, 23(2), 211–231. https://doi.org/10.1093/ectj/utaa002
   Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706
   Web of Science ®Google Scholar
• Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x
   Web of Science ®Google Scholar
• Imbens, G. W., & Kalyanaraman, K. (2012). Optimal bandwidth choice for the regression discontinuity estimator. The Review of Economic Studies, 79(3), 933–959. https://doi.org/10.1093/restud/rdr043
   Web of Science ®Google Scholar
• Imbens, G., & Wager, S. (2019). Optimized regression discontinuity designs. The Review of Economics and Statistics, 101(2), 264–278. https://doi.org/10.1162/rest_a_00793
   Web of Science ®Google Scholar
• James-Burdumy, S., Deke, J., Gersten, R., Lugo-Gil, J., Newman-Gonchar, R., Dimino, J., Haymond, K., & Liu, A. Y.-H. (2012). Effectiveness of four supplemental reading comprehension interventions. Journal of Research on Educational Effectiveness, 5(4), 345–383. https://doi.org/10.1080/19345747.2012.698374
   Web of Science ®Google Scholar
• James-Burdumy, S., Deke, J., Lugo-Gil, J., Carey, N., Hershey, A., Gersten, R., & Faddis, B. (2010). Effectiveness of selected supplemental reading comprehension interventions: Findings from two student cohorts. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• James-Burdumy, S., Dynarski, M., & Deke, J. (2008). After-school program effects on behavior: Results from the 21st Century Community Learning Centers program national evaluation. Economic Inquiry, 46(1), 13–18. https://doi.org/10.1111/j.1465-7295.2007.00074.x
   Web of Science ®Google Scholar
• Kane, T. J. (2015). Frustrated with the pace of progress in education? Invest in better evidence. The Brookings Institution.
   Google Scholar
• Kolesár, M., & Rothe, C. (2018). Inference in regression discontinuity designs with a discrete running variable. American Economic Review, 108(8), 2277–2304. https://doi.org/10.1257/aer.20160945
   Web of Science ®Google Scholar
• Leamer, E. (2010). Tantalus on the road to asymptopia. Journal of Economic Perspectives, 24(2), 31–46. https://doi.org/10.1257/jep.24.2.31
   Web of Science ®Google Scholar
• Lee, D. S., & Card, D. (2008). Regression discontinuity inference with specification error. Journal of Econometrics, 142(2), 655–674. https://doi.org/10.1016/j.jeconom.2007.05.003
   Web of Science ®Google Scholar
• Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms. National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Murray, D. M. (1998). Design and analysis of group-randomized trials. Oxford University Press.
   Google Scholar
• Noack, C., & Rothe, C. (2020). Bias-aware inference in fuzzy regression discontinuity designs. Working paper.
   Google Scholar
• Pei, Z., Lee, D. S., Card, D., & Weber, A. (2020). Local polynomial order in regression discontinuity designs. NBER working paper 27424.
   Google Scholar
• Puma, M. J., Olsen, R. B., Bell, S. H., & Price, C. (2009). What to do when data are missing in group randomized controlled trials. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Sales, A. C., & Hansen, B. B. (2019). Limitless regression discontinuity. Journal of Educational and Behavioral Statistics, 20(10), 1–32.
   Google Scholar
• Schochet, P. Z. (2008a). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87. https://doi.org/10.3102/1076998607302714
   Web of Science ®Google Scholar
• Schochet, P. Z. (2008b). Technical methods report: Statistical power for regression discontinuity designs in education evaluations. National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
   Web of Science ®Google Scholar
• What Works Clearinghouse. (2008). WWC procedures and standards handbook (version 2.0). Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• What Works Clearinghouse. (2013). Assessing attrition bias (version 2.1). Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• What Works Clearinghouse. (2014). Assessing attrition bias—addendum (version 3.0). Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• What Works Clearinghouse. (2020). WWC standards handbook (version 4.1). Institute of Education Sciences, U.S. Department of Education.
   Google Scholar