Assessing the Precision of Multisite Trials for Estimating the Parameters of a Cross-Site Population Distribution of Program Effects

References

• Agodini, R., Harris, B., Thomas, M., Murphy, R., Gallagher, L., Pendleton, A. (2010). Achievement effects of four early elementary school math curricula (NCEE 2011-4001). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Angrist, J. D., Pathak, P. A., & Walters, C. R. (2013). Explaining charter school effectiveness. American Economic Journal: Applied Economics, 5(4), 1–27.
   PubMed Web of Science ®Google Scholar
• Black, A. R., Somers, M. A., Doolittle, F., Unterman, R., & Grossman, J. B. (2009). The evaluation of enhanced academic instruction in after-school programs final report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Bloom, H. S. (1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556.
   Web of Science ®Google Scholar
• Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In H. S. Bloom (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115–172). New York, NY: Russell Sage Foundation.
   Google Scholar
• Bloom, H. S., Bos, J., & Lee, S. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23(4), 445–469.
   PubMed Web of Science ®Google Scholar
• Bloom, H., Hill, C., Black, A. R., & Lipsey, M. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328.
   Google Scholar
• Bloom, H. S., Hill, C. J., & Riccio, J. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare‐to‐work experiments. Journal of Policy Analysis and Management, 22(4), 551–575.
   Web of Science ®Google Scholar
• Bloom, H. S., Raudenbush, S. W., Weiss, M. J., & Porter, K. (2017). Using multi-site experiments to study cross-site variation in effects of program assignment. Journal of Research on Educational Effectiveness. Advance online publication. doi:10.1080/19345747.2016.1264518
   Google Scholar
• Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59.
   Web of Science ®Google Scholar
• Bloom, H. S., & Weiland, C. (2015). Quantifying variation in Head Start effects on young children's cognitive and socio-emotional skills using data from the National Head Start Impact Study. New York, NY: MDRC.
   Google Scholar
• Crits-Christoph, P., Tu, X., & Gallop, R. (2003). Therapists as fixed versus random effects—Some statistical and conceptual issues: A comment on Siemer and Joormann (2003). Psychological Methods, 8(4), 518–523.
   PubMed Web of Science ®Google Scholar
• Dong, N., & Maynard, R. A. (2013). PowerUp!: A tool for calculating minimum detectable effect size differences and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24–67. doi:10.1080/19345747.2012.673143.
   Web of Science ®Google Scholar
• Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final report (NCEE 2010-4029). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Hamilton, G. (2002). Moving people from welfare to work: Lessons from the National Evaluation of Welfare-to-Work Strategies. Washington, DC: U.S. Department of Health and Human Services.
   Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.
   Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489.
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Pigott, T. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203–217.
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426–445.
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Rhoads, C. (2010). Statistical power analysis in education research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177.
   Web of Science ®Google Scholar
• Jacob, R., Zhu, P., & Bloom, H. (2010). New empirical evidence for the design of group-randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198.
   Google Scholar
• James-Burdumy, S., & National Center for Education Evaluation and Regional Assistance. (2009). Effectiveness of selected supplemental reading comprehension interventions: Impacts on a first cohort of fifth-grade students. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Institute of Education Sciences and National Science Foundation. (2013). Common guidelines for education research and development. Retrieved from http://ies.ed.gov/pdf/CommonGuidelines.pdf
   Google Scholar
• Kemple, J. J., & Haimson, J. (1994). Florida's Project Independence: Program implementation, participation patterns, and first-year impacts, New York, NY: MDRC.
   Google Scholar
• Konstantopoulos, S. (2008). Computing power of tests of the variance of treatment effects in designs with two levels of nesting. Multivariate Behavioral Research, 43(2), 327–352.
   PubMed Web of Science ®Google Scholar
• Morebeek, M., & Teerenstra, S. (2016). Power analysis of trials with multilevel data. Boca Raton, FL: CRC Press.
   Google Scholar
• Olsen, R., Bein, E., & Judkins, D. (2015, November). Sample size requirements for education multi-site RCTs that select sites randomly. Paper presented at 37th Annual Conference of the Association for Public Policy Analysis and Management, Miami, FL.
   Google Scholar
• O'Muircheartaigh, C., & Hedges, L. V. (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63(2), 195–210.
   Web of Science ®Google Scholar
• Peck, L. R., & Bell, S. H. (2014). The role of program quality in determining Head Start's impact on child development (OPRE Report #2014-10). Washington DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
   Google Scholar
• Puma, M., Bell, S., Cook, R., Heid, C., Shapiro, G., Broene, P., … Spier, E. (2010). Head start impact study: Final report. Washington, DC: U.S. Department of Health and Human Services, Administration for Children & Families.
   Google Scholar
• Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173.
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bloom, H. S. (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475–499.
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213.
   PubMed Web of Science ®Google Scholar
• Riccio, J. A., & Friedlander, D. (1992). GAIN: Program strategies, participation patterns, and first-year impacts in six counties: Executive summary. New York, NY: MDRC.
   Google Scholar
• Schochet, P. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87.
   Web of Science ®Google Scholar
• Schochet, P. Z. (2013). Estimators for clustered education RCTs using the Neyman Model for causal inference. Journal of Educational and Behavioral Statistics, 38(3), 219–238.
   Web of Science ®Google Scholar
• Schochet, P. Z. (2015). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs. Retrieved from http://files.eric.ed.gov/fulltext/ED556496.pdf
   Google Scholar
• Senn, S. (2007). Statistical Issues in Drug Development. Chichester, West Sussex, England: Wiley.
   Google Scholar
• Serlin, R. C., Wampold, B. E., & Levin, J. R. (2003). Should providers of treatment be regarded as a random factor? If it ain't broke, don't “fix” it: A comment on Siemer and Joormann (2003). Psychological Methods, 8(4), 524–534.
   PubMed Web of Science ®Google Scholar
• Siemer, M., & Joorman, J. (2003). Assumptions and consequences of treating providers in therapy studies as fixed versus random effects: Reply to Crits-Christoph, Tu, and Gallop (2003) and Serlin, Wampold, and Levin (2003). Psychological Methods, 8(4), 535–544.
   PubMed Web of Science ®Google Scholar
• Spybrook, J. (2014). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82(3), 334–357.
   Web of Science ®Google Scholar
• Spybrook, J., & Raudenbush, S. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318.
   Web of Science ®Google Scholar
• Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 1–13.
   Web of Science ®Google Scholar
• Spybrook, J., Westine, C. D., & Taylor, J. A. (2016). Design parameters for impact research in science education: A multi-state analysis. AERA Open, 2(1), doi:10.1177/2332858415625975.
   PubMedGoogle Scholar
• Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386.
   Web of Science ®Google Scholar
• Tipton, E. (2013). Improving generalizations from experiments using propensity score subclassification assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.
   Web of Science ®Google Scholar
• Tipton, E., Fellers, L., Caverly, S., Vaden-Kiernan, M., Borman, G., Sullivan, K., & Ruiz De Castilla, V. (2016). Site selection in experiments: An assessment of site recruitment and generalizability in two scale-up studies. Journal of Research on Educational Effectiveness, 9(S1), 209–228.
   Web of Science ®Google Scholar
• Weiss, M. J., Bloom, H. S., Verbitsky Savitz, N., Gupta, H., Vigil, A., & Cullinan, D. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Advance online publication. doi:10.1080/19345747.2017.1300719
   Google Scholar
• Westine, C., Spybrook, J., & Taylor, J. (2013). An empirical investigation of design parameters for planning cluster randomized trials of science achievement. Evaluation Review, 37(6), 490–519.
   PubMed Web of Science ®Google Scholar
• Word, E., Johnston, J., Bain, H., Fulton, D. B., Boyd-Zaharias, J., Lintz, M. N., … Breda, C. (1990). Student/teacher achievement ratio (STAR): Tennessee's K–3 class-size study. Nashville, TN: Tennessee State Department of Education.
   Google Scholar
• Zhu, P., Jacob, R., Bloom, H., & Xu, Z. (2012). Designing and analyzing studies that randomize schools to estimate intervention effects on student academic outcomes without classroom-level information. Educational Evaluation and Policy Analysis, 34(1), 45–68.
   Web of Science ®Google Scholar