Optimal Sample Allocation in Group-Randomized Mediation Studies with a Group-Level Mediator

References

• Allison, P. (1995). Exact variance of indirect effects in recursive linear models. Sociological Methodology, 25, 253–266. doi: 10.2307/271069
   Web of Science ®Google Scholar
• Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. doi: 10.1037/0022-3514.51.6.1173.
   PubMed Web of Science ®Google Scholar
• Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29, 30–59. doi: 10.3102/0162373707299550.
   Web of Science ®Google Scholar
• Bollen, K. A., & Stine, R. (1990). Direct and indirect effects: Classical and bootstrap estimates of variability. Sociological Methodology, 20, 115–140. doi: 10.2307/271084.
   Google Scholar
• Desimone, L. M., & Hill, K. L. (2017). Inside the black box: Examining mediators and moderators of a middle school science intervention. Educational Evaluation and Policy Analysis, 39, 511–536. doi: 10.3102/0162373717697842.
   Web of Science ®Google Scholar
• Eden, M. (2017). School discipline reform and disorder: Evidence from New York City Public Schools. The Education Digest, 83, 22–28.
   Google Scholar
• Enders, C., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138. doi: 10.1037/1082-989X.12.2.121.
   PubMed Web of Science ®Google Scholar
• Gagne, P., & Hancock, G. R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41, 65–83.
   PubMed Web of Science ®Google Scholar
• Hayes, A. F., & Scharkow, M. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science, 24, 1918–1927. doi: 10.1207/s15327906mbr4101_5.
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Borenstein, M. (2014). Conditional optimal design in three- and four-level experiments. Journal of Educational and Behavioral Statistics, 39, 257–281. doi: 10.3102/1076998614534897.
   Web of Science ®Google Scholar
• Hedges, L., & Hedberg, E. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi: 10.3102/0162373707299706.
   Web of Science ®Google Scholar
• Hoffman, L. L., Hutchinson, C. J., & Reiss, E. (2009). On improving school climate: Reducing reliance on rewards and punishment. International Journal of Whole Schooling, 5, 13.
   Google Scholar
• Hox, J. J., Moerbeek, M., Kluytmans, A., & van de Schoot, R. (2014). Analyzing indirect effects in cluster randomized trials: The effect of estimation method, number of groups and group sizes on accuracy and power. Frontiers in Psychology, 5(78), 1–7.
   PubMedGoogle Scholar
• Imai, K., Keele, L., & Tingley, D. (2010). A general approach to causal mediation analysis. Psychological Methods, 15, 309–334. doi: 10.1037/a0020761.
   PubMed Web of Science ®Google Scholar
• Institute of Education Sciences, U.S. Department of Education & National Science Foundation (2013). Common Guidelines for Education Research and Development (NSF 13-126).
   Google Scholar
• Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally-defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42, 499–530. doi: 10.3102/1076998617695506.
   Web of Science ®Google Scholar
• Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017). Statistical power for causally-defined mediation in group-randomized studies. Multivariate Behavioral Research, Advance online publication.
   PubMed Web of Science ®Google Scholar
• Kelcey, B., & Phelps, G. (2014). Strategies for improving power in school randomized studies of professional development. Evaluation Review, 37, 520–554. doi: 10.1177/0193841X14528906.
   Web of Science ®Google Scholar
• Kelcey, B., & Phelps, G. (2013). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35, 370–390. doi: 10.3102/0162373713482766.
   Web of Science ®Google Scholar
• Kelcey, B., Phelps, G., Spybrook, J., Jones, N., & Zhang, J. (2017). Designing large-scale multisite and cluster-randomized studies of professional development. Journal of Experimental Education, 85, 389–410. doi: 10.1080/00220973.2016.1220911.
   Web of Science ®Google Scholar
• Kelcey, B., & Shen, Z. (2016). Multilevel design of school effectiveness studies in sub-Saharan Africa. School Effectiveness and School Improvement, 27, 492–510. doi: 10.1080/09243453.2016.1168855.
   Web of Science ®Google Scholar
• Konstantopoulos, S. (2009). Incorporating cost in power analysis for three-level cluster-randomized designs. Evaluation Review, 33, 335–357. doi: 10.1177/0193841X09337991.
   PubMed Web of Science ®Google Scholar
• Korendijk, E. J. H., Moerbeek, M., and Maas, C. J. M. (2010). The robustness of designs for trials with nested data against incorrect initial intracluster correlation coefficient estimates. Journal of Educational and Behavioral Statistics, 35, 566–585. doi: 10.3102/1076998609360774
   Web of Science ®Google Scholar
• Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21. doi: 10.1207/s15327906mbr3001_1.
   PubMed Web of Science ®Google Scholar
• Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36, 249–277. doi: 10.1207/S15327906MBR3602_06.
   PubMed Web of Science ®Google Scholar
• Li, X., & Beretvas, S. N. (2013). Sample size limits for estimating upper level mediation models using multilevel SEM. Structural Equation Modeling: A Multidisciplinary Journal, 20, 241–264. doi: 10.1080/10705511.2013.769391.
   Web of Science ®Google Scholar
• Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. doi: 10.1037/a0012869.
   PubMed Web of Science ®Google Scholar
• MacKinnon, D. P. (2008). Introduction to statistical mediation analysis. Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104. doi: 10.1037/1082-989X.7.1.83.
   PubMed Web of Science ®Google Scholar
• MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 37–67. doi: 10.1207/s15327906mbr3901_4.
   PubMed Web of Science ®Google Scholar
• Oehlert, G. W. (1992). A note on the delta method. The American Statistician, 46, 27–29.
   Web of Science ®Google Scholar
• Osher, D., Bear, G. G., Sprague, J. R., & Doyle, W. (2010). How can we improve school discipline? Educational Researcher, 39, 48. doi: 10.3102/0013189X09357618
   Web of Science ®Google Scholar
• Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670. doi: 10.1177/0049124112460380.
   Web of Science ®Google Scholar
• Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6, 77–98. doi: 10.1080/19312458.2012.679848.
   Google Scholar
• Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185. doi: 10.1037/1082-989X.2.2.173.
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Sadoff, S. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154. doi: 10.1080/19345740801982104.
   Google Scholar
• Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., & Shavelson, R. J. (2007). Estimating causal effects using experimental and observational designs (report from the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association.
   Google Scholar
• Schochet, P. Z. (2011). Do typical RCTs of education interventions have sufficient statistical power for linking impacts on teacher practice and student achievement outcomes?. Journal of Educational and Behavioral Statistics, 36, 441–471. doi: 10.3102/1076998610375840.
   Web of Science ®Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference (2nd ed.). Boston: Houghton Mifflin.
   Google Scholar
• Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312. doi: 10.2307/270723.
   Google Scholar
• Spybrook, J., Bloom, H., Congdon, R., Hill, C., Martinez, A., Raudenbush, S. W. (2011). Optimal design plus empirical evidence: Documentation for the “Optimal Design” software version 3.0. Available from http://www.wtgrantfoundation.org/resources/optimal-design.
   Google Scholar
• Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences. International Journal of Research and Method in Education, 39, 255–267. doi: 10.1080/1743727X.2016.1150454.
   Web of Science ®Google Scholar
• Spybrook, J., & Raudenbush, S. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the institute of education sciences. Educational Evaluation and Policy Analysis, 31, 298–318. doi: 10.3102/0162373709339524.
   Web of Science ®Google Scholar
• VanderWeele, T. J. (2015). Explanation in causal inference: Methods for mediation and interaction. New York, NY: Oxford University Press.
   Google Scholar
• Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and Psychological Measurement, 73, 913–934.
   Web of Science ®Google Scholar
• Yoshikawa, H., Leyva, D., Snow, C. E., Treviño, E., Barata, M. C., Weiland, C., … Arbour, M. C. (2015). Experimental impacts of a teacher professional development program in Chile on preschool classroom quality and child outcomes. Developmental Psychology, 51, 309–322. doi: 10.1177/0013164413495237.
   PubMed Web of Science ®Google Scholar
• Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719. doi: 10.1177/1094428108327450.
   Web of Science ®Google Scholar