An Applied Researcher’s Guide to Estimating Effects from Multisite Individually Randomized Trials: Estimands, Estimators, and Estimates

References

• Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. (2017). When should you adjust standard errors for clustering? [Technical report]. National Bureau of Economic Research.
   Google Scholar
• Aronow, P. M., Green, D. P., & Lee, D. K. (2014). Sharp bounds on the variance in randomized experiments. The Annals of Statistics, 42(3), 850–871. https://doi.org/10.1214/13-AOS1200
   Web of Science ®Google Scholar
• Bloom, H. S., Raudenbush, S. W., Weiss, M. J., & Porter, K. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10(4), 817–842. https://doi.org/10.1080/19345747.2016.1264518
   Web of Science ®Google Scholar
• Bloom, H. S., & Spybrook, J. (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 10(4), 877–902. https://doi.org/10.1080/19345747.2016.1271069
   Web of Science ®Google Scholar
• Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3), 414–427. https://doi.org/10.1162/rest.90.3.414
   Web of Science ®Google Scholar
• Clark, M. A., Gleason, P., Tuttle, C. C., & Silverberg, M. K. (2011). Do charter schools improve student achievement? Evidence from a national randomized study [Working paper]. Mathematica Policy Research, Inc.
   Google Scholar
• Cochran, W. G. (1977). Sampling techniques (3rd ed.). John Wiley and Sons.
   Google Scholar
• Edmunds, J. A., Unlu, F., Glennie, E., Bernstein, L., Fesler, L., Furey, J., & Arshavsky, N. (2017). Smoothing the transition to postsecondary education: The impact of the early college model. Journal of Research on Educational Effectiveness, 10(2), 297–325. https://doi.org/10.1080/19345747.2016.1191574
   Web of Science ®Google Scholar
• Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
   Google Scholar
• Hodges, J. S., & Clayton, M. K. (2011). Random effects old and new. http://www.biostat.umn.edu/∼hodges/PubH8492/Hodges-ClaytonREONsubToStatSci.pdf
   Google Scholar
• Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.
   Google Scholar
• Kern, H. L., Stuart, E. A., Hill, J., & Green, D. P. (2016). Assessing methods for generalizing experimental impact estimates to target populations. Journal of Research on Educational Effectiveness, 9(1), 103–127. https://doi.org/10.1080/19345747.2015.1060282
   PubMed Web of Science ®Google Scholar
• Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics, 7(1), 295–318. https://doi.org/10.1214/12-AOAS583
   Web of Science ®Google Scholar
• Lohr, S. L. (2019). Sampling: Design and analysis. CRC Press.
   Google Scholar
• Mayer, A., Patel, R., Rudd, T., & Ratledge, A. (2015). Designing scholarships to improve college success: Final report on the performance-based scholarship demonstration. MDRC.
   Google Scholar
• Miratrix, L., Sekhon, J. S., Theodoridis, A., & Campos, L. (2018). Worth weighting? How to think about and use sample weights in survey experiments. Political Analysis, 26(3), 275–291. https://doi.org/10.1017/pan.2018.1
   Web of Science ®Google Scholar
• Miratrix, L. W., Sekhon, J. S., & Yu, B. (2013). Adjusting treatment effect estimates by post-stratification in randomized experiments. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 75(2), 369–396. https://doi.org/10.1111/j.1467-9868.2012.01048.x
   Web of Science ®Google Scholar
• Olsen, R. B., Orr, L. L., Bell, S. H., & Stuart, E. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107–121. https://doi.org/10.1002/pam.21660
   PubMed Web of Science ®Google Scholar
• Orr, L. L., Olsen, R. B., Bell, S. H., Schmid, I., Shivji, A., & Stuart, E. A. (2019). Using the results from rigorous multisite evaluations to inform local policy decisions. Journal of Policy Analysis and Management, 38(4), 978–1003. https://doi.org/10.1002/pam.22154
   Web of Science ®Google Scholar
• Parsons, J., Weiss, C., & Wei, Q. (2016). Impact evaluation of the adolescent behavioral learning experience (ABLE) program. Vera Institute of Justice. https://www.vera.org/publications/rikersadolescent-behavioral-learning-experience-evaluation.
   Google Scholar
• Pashley, N. E., & Miratrix, L. W. (2020). Block what you can, except when you shouldn't. arXiv preprint arXiv:2010.14078.
   Google Scholar
• Pashley, N. E., & Miratrix, L. W. (In Press). Insights on variance estimation for blocked and matched pairs designs. The Journal of Educational and Behavioral Statistics.
   Google Scholar
• Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36(4), 672–683. https://doi.org/10.1080/07350015.2016.1247004
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bloom, H. S. (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475–499. https://doi.org/10.1177/1098214015600515
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Schwartz, D. (2020). Randomized experiments in education, with implications for multilevel causal inference. Annual Review of Statistics and Its Application, 7(1), 177–208. https://doi.org/10.1146/annurev-statistics-031219-041205
   Google Scholar
• Richburg-Hayes, L., Visher, M., & Bloom, D. (2008). Do learning communities effect academic outcomes? Evidence from an experiment in a community college. Journal of Research on Educational Effectiveness, 1(1), 33–65. https://doi.org/10.1080/19345740701692472
   Google Scholar
• Rosenbaum, P. R. (2010). Design of observational studies. Springer.
   Google Scholar
• Sarndal, C.-E., Swensson, B., & Wretman, J. (2003). Model assisted survey sampling. Springer.
   Google Scholar
• Schochet, P. Z. (2016). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs [2nd ed. (NCEE 2015–4011)]. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. http://ies.ed.gov/ncee/edlabs.
   Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Mifflin and Company.
   Google Scholar
• Tipton, E., & Olsen, R. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47(8), 516–524. https://doi.org/10.3102/0013189X18781522
   Web of Science ®Google Scholar
• Weiss, M. J., Bloom, H. S., & Brock, T. (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808. https://doi.org/10.1002/pam.21760
   Web of Science ®Google Scholar
• Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719
   Web of Science ®Google Scholar
• Weiss, M. J., Ratledge, A., Sommo, C., & Gupta, H. (2019). Supporting community college students from start to degree completion: Long-term evidence from a randomized trial of CUNY’s ASAP. American Economic Journal: Applied Economics, 11(3), 253–297. https://doi.org/10.1257/app.20170430
   Web of Science ®Google Scholar
• Yeager, D. S., Hanselman, P., Paunesku, D., Hulleman, C., Dweck, C., Muller, C., Crosnoe, R., Walton, G. M., Tipton, E., & Duckworth, A. (2017). National study of learning mindsets – One year impact analysis. https://osf.io/tn6g4/.
   Google Scholar