Using Data from Randomized Trials to Assess the Likely Generalizability of Educational Treatment-Effect Estimates from Regression Discontinuity Designs

References

• Angrist, J. D., & Pischke, J. S. (2015). Mastering metrics: The path from cause to effect. Princeton, NJ: Princeton University Press.
   Google Scholar
• Angrist, J. D., & Rokkanen, M. (2015). Wanna get away? Regression discontinuity estimation of exam school effects away from the cutoff. Journal of the American Statistical Association, 110(512), 1331–1344.
   Web of Science ®Google Scholar
• Black, A. R., Somers, M. A., Doolittle, F., Unterman, R., & Grossman, J. B. (2009). The evaluation of enhanced academic instruction in after-school programs: Final report (NCEE 2009–4077). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Bloom, H. S. (2012). Modern regression discontinuity analysis. Journal of Research on Educational Effectiveness, 5(1), 43–82. doi:10.1080/19345747.2011.578707
   Web of Science ®Google Scholar
• Bloom, H. S., & Michalopoulos, C. (2011). When is the story in the subgroups? Strategies for interpreting and reporting intervention effects on subgroups. Prevention Science, 14(2), 179–188.
   Google Scholar
• Bloom, H. S., Raudenbush, S. W., Weiss, M. J., & Porter, K. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10(4), 817–842. doi:10.1080/19345747.2016.1264518
   Web of Science ®Google Scholar
• Bloom, H. S., & Spybrook, J. (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 10(4), 877. doi:10.1080/19345747/2016.1271069
   Web of Science ®Google Scholar
• Bloom, H. S., & Unterman, R. (2014). Can small high schools of choice improve educational prospects for disadvantaged students? Journal of Policy Analysis and Management, 33(2), 290–319. doi:10.1002/pam.21748
   Web of Science ®Google Scholar
• Bloom, H. S., & Weiland, C. (2015). Quantifying variation in Head Start effects on young children’s cognitive and socio-emotional skills using data from the National Head Start Impact Study (MDRC Working Papers on Research Methodology). Retrieved from https://www.mdrc.org/sites/default/files/quantifying_variation_in_head_start.pdf
   Google Scholar
• Booher-Jennings, J. (2005). Below the bubble: Educational triage and the Texas Accountability System. American Educational Research Journal, 42(2), 231–268. doi:10.3102/00028312042002231
   Web of Science ®Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Brown, R. S., & Dale, S. B. (2007). The research design and methodological issues for the cash and counselling evaluation. Health Services Research, 42(1Pt2), 414–445.
   PubMedGoogle Scholar
• Chaplin, D. D., Cook, T. D., Zurovac, J., Coopersmith, J. S., Finucane, M. M., Vollmer, L. N., & Morris, R. E. (2018). The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study comparisons. Journal of Policy Analysis and Management, 37(2), 403–429.
   Web of Science ®Google Scholar
• Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally.
   Google Scholar
• Cooper, H. M., & Hedges, L. V. (1994). The handbook of research synthesis. New York, NY: Russell Sage Foundation.
   Google Scholar
• Cunha, F., & Heckman, J. J. (2007). The technology of skill formation. American Economic Review, 97(2), 31–47. doi:10.1257/aer.97.2.31
   Web of Science ®Google Scholar
• Dong, Y., & Lewbel, A. (2015). Identifying the effect of changing the policy threshold in regression discontinuity models. The Review of Economics and Statistics, 97(5), 1081–1092.
   Web of Science ®Google Scholar
• Edmunds, J., Unlu, F., Glennie, E., Bernstein, L., Fesler, L., Furey, J., & Arshavsky, N. (2017). Smoothing the transition to postsecondary education: The impact of the early college model. Journal of Research on Educational Effectiveness, 10(2), 297–325. doi:10.1080/19345747.2016.1191574
   Web of Science ®Google Scholar
• Goodman, J. (2008). Who merits financial aid? Massachusetts' Adams Scholarship. Journal of Public Economics, 92(10–11), 2121–2131. doi:10.1016/j.jpubeco.2008.03.009
   Web of Science ®Google Scholar
• Hahn, J., Todd, P., & Van Der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression discontinuity design. Econometrica, 69(1), 201–209. doi:10.1111/1468-0262.00183
   Web of Science ®Google Scholar
• Hill, J. (2011). Bayesian non-parametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1), 217. doi:10.1198/jcgs.2010.08162
   Web of Science ®Google Scholar
• Hotelling, H. (1951). A generalized T test and measure of multivariate dispersion. Chapel Hill, NC: University of North Carolina.
   Google Scholar
• Imbens, G., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615–635. doi:10.1016/j.jeconom.2007.05.001
   Web of Science ®Google Scholar
• Lee, D. S. (2008). Randomized experiments from non-random selection in U.S. Senate House elections. Journal of Econometrics, 142(2), 807–828. doi:10.1016/j.jeconom.2007.05.004
   Web of Science ®Google Scholar
• Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. doi:10.1257/jel.48.2.281
   Web of Science ®Google Scholar
• Nomi, T., & Raudenbush, S. W. (2016). Making a success of “Algebra for All”: The impact of extended instructional time and classroom peer skill in Chicago. Educational Evaluation and Policy Analysis, 38(2), 431–451. doi:10.3102/0162373716643756
   Web of Science ®Google Scholar
• Nunnally, J. C. (1967). Psychometric theory. New York, NY: McGraw-Hill Book Company.
   Google Scholar
• Puma, M., Bell, S., Cook, R., Heid, C., Shapiro, G., Broene, P., & Friedman, J. (2010). Head Start Impact Study final report. Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/hs_impact_study_final.pdf
   Google Scholar
• Ramey, C. T., & Ramey, S. L. (1998). Early intervention and early experience. American Psychologist, 53(2), 109–120. doi:10.1037/0003-066X.53.2.109
   PubMed Web of Science ®Google Scholar
• Raudenbush, S. W., & Bloom, H. S. (2015). Learning about and from cross-site distributions of program impacts using multi-site trials. American Journal of Evaluation, 36(4), 475. doi:10.1177/1098214015600515
   Web of Science ®Google Scholar
• Rothwell, P. M. (2005). Subgroup analysis in randomised controlled trials: Importance, indications, and interpretation. The Lancet, 365(9454), 176–186. doi:10.1016/S0140-6736(05)17709-5
   PubMed Web of Science ®Google Scholar
• Somers, M.-A., Corrin, W., Sepanik, S., Salinger, T., Levin, J., & Zmach, C. (2010). The Enhanced Reading Opportunities Study Final Report: The impact of supplemental literacy courses for struggling ninth-grade readers (NCEE 2010–4021). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex-post facto experiment. Journal of Educational Psychology, 51(6), 309–317. doi:10.1037/h0044319
   Web of Science ®Google Scholar
• Tipton, E., Yeager, D. S., Iachan, R., & Schneider, B. (2019). Designing probability samples to study treatment effect heterogeneity. In P. J. Lavrakas (Ed.), Experimental methods in survey research: Techniques that combine random sampling with random assignment. New York, NY: Wiley.
   Google Scholar
• U.S. Department of Education. (2017). Institute of Education Sciences, What Works Clearinghouse. Standards Handbook, version 4.0. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf.
   Google Scholar
• Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. doi:10.1080/19345747.2017.13000719
   Web of Science ®Google Scholar
• Wing, C., & Cook, T. D. (2013). Strengthening the regression discontinuity design using additional design elements: A within-study comparision. Journal of Policy Analysis and Management, 32(4), 853–877. doi:10.1002/pam.21721
   Web of Science ®Google Scholar
• Wooldridge, J. F. (2009). Introductory econometrics: A modern approach (4th ed.). Mason, OH: South-Western CENGAGE Learning.
   Google Scholar