Power and Sample Size Determination for Multilevel Mediation in Three-Level Cluster-Randomized Trials

References

• Aarons, G. A., Glisson, C., Green, P. D., Hoagwood, K., Kelleher, K. J., & Landsverk, J. A. (2012). The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implementation Science, 7(1), 56. doi:10.1186/1748-5908-7-56
   PubMed Web of Science ®Google Scholar
• Aguinis, H., Edwards, J. R., & Bradley, K. J. (2017). Improving our understanding of moderation and mediation in strategic management research. Organizational Research Methods, 20(4), 665–685. doi:10.1177/1094428115627498
   Web of Science ®Google Scholar
• Baldwin, S. A., Murray, D. M., Shadish, W. R., Pals, S. L., Holland, J. M., Abramowitz, J. S., Andersson, G., Atkins, D. C., Carlbring, P., Carroll, K. M., Christensen, A., Eddington, K. M., Ehlers, A., Feaster, D. J., Keijsers, G. P. J., Koch, E., Kuyken, W., Lange, A., Lincoln, T., … Watson, J. (2011). Intraclass correlation associated with therapists: estimates and applications in planning psychotherapy research. Cognitive Behaviour Therapy, 40(1), 15–33. doi:10.1080/16506073.2010.520731
   PubMedGoogle Scholar
• Beasley, T. (2014). Test of mediation: paradoxical decline in statistical power as a function of mediator collinearity. The Journal of Experimental Education, 82(3), 283–306. doi:10.1080/00220973.2013.813360
   PubMed Web of Science ®Google Scholar
• Brincks, A., Enders, C., Llabre, M., Bulotsky-Shearer, R., Prado, G., & Feaster, D. (2017). Centering predictor variables in three-level contextual models. Multivariate Behavioral Research, 52(2), 149–163. doi:10.1080/00273171.2016.1256753
   PubMed Web of Science ®Google Scholar
• Brincks, A. M., Feaster, D. J., & Mitrani, V. B. (2010). A multilevel mediation model of stress and coping for women with HIV and their families. Family Process, 49(4), 517–529. doi:10.1111/j.1545-5300.2010.01337.x
   PubMed Web of Science ®Google Scholar
• Brown, E. C., Hawkins, J. D., Rhew, I. C., Shapiro, V. B., Abbott, R. D., Oesterle, S., Arthur, M. W., Briney, J. S., & Catalano, R. F. (2014). Prevention system mediation of communities that care effects on youth outcomes. Prevention Science, 15(5), 623–632. doi:10.1007/s11121-013-0413-7
   PubMed Web of Science ®Google Scholar
• Bulus, M., Dong, N., Kelcey, B., Spybrook, J. (2019). PowerUpR Shiny App for Experimental and Quasi-Experimental Study Design (Version 1.0.4) [Software]. https://powerupr.shinyapps.io/index/
   Google Scholar
• Cegala, D. J., & Post, D. M. (2009). The impact of patients’ participation on physicians’ patient-centered communication. Patient Education and Counseling, 77(2), 202–208. doi:10.1016/j.pec.2009.03.025
   PubMed Web of Science ®Google Scholar
• Černe, M., Jaklič, M., & Škerlavaj, M. (2013). Authentic leadership, creativity, and innovation: A multilevel perspective. Leadership, 9(1), 63–85. doi:10.1177/1742715012455130
   Web of Science ®Google Scholar
• Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Erlbaum.
   Google Scholar
• Cosby, R., Howard, M., Kaczorowski, J., Willan, A., & Sellors, J. (2003). Randomizing patients by family practice: sample size estimation, intracluster correlation and data analysis. Family Practice, 20(1), 77–82. doi:10.1093/fampra/20.1.77
   PubMed Web of Science ®Google Scholar
• Curenton, S. M., Dong, N., & Shen, X. (2015). Does aggregate school-wide achievement mediate fifth grade outcomes for former early childhood education participants? Developmental Psychology, 51(7), 921–934. doi:10.1037/a0039295
   PubMed Web of Science ®Google Scholar
• Dong, N., Reinke, W. M., Herman, K. C., Bradshaw, C. P., & Murray, D. W. (2016). Meaningful effect sizes, intraclass correlations, and proportions of variance explained by covariates for planning two-and three-level cluster randomized trials of social and behavioral outcomes. Evaluation Review, 40(4), 334–377. doi:10.1177/0193841X16671283
   PubMed Web of Science ®Google Scholar
• Eden, D. (2017). Field experiments in organizations. Annual Review of Organizational Psychology and Organizational Behavior, 4(1), 91–122. doi:10.1146/annurev-orgpsych-041015-062400
   Google Scholar
• Eden, D., Stone-Romero, E. F., & Rothstein, H. R. (2015). Synthesizing results of multiple randomized experiments to establish causality in mediation testing. Human Resource Management Review, 25(4), 342–351. doi:10.1016/j.hrmr.2015.02.001
   Web of Science ®Google Scholar
• Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138. doi:10.1037/1082-989X.12.2.121
   PubMed Web of Science ®Google Scholar
• Evans, D. (2003). Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing, 12(1), 77–84. doi:10.1046/j.1365-2702.2003.00662.x
   PubMed Web of Science ®Google Scholar
• Gottfredson, D. C., Cook, T. D., Gardner, F. E., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16(7), 893–926. doi:10.1007/s11121-015-0555-x
   PubMed Web of Science ®Google Scholar
• Harvey, A. G., & Gumport, N. B. (2015). Evidence-based psychological treatments for mental disorders: Modifiable barriers to access and possible solutions. Behaviour Research and Therapy, 68, 1–12. doi:10.1016/j.brat.2015.02.004
   PubMed Web of Science ®Google Scholar
• Hayes, A. F., & Scharkow, M. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science, 24(10), 1918–1927. doi:10.1177/0956797613480187
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2013). Intraclass correlations and covariate outcome correlations for planning two-and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. doi:10.1177/0193841X14529126
   PubMed Web of Science ®Google Scholar
• Hedges, L., & Hedberg, E. (2007). Intraclass correlation values for planning school-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi:10.3102/0162373707299706
   Web of Science ®Google Scholar
• Hedges, L., Hedberg, E., & Kuyper, A. (2012). The variance of intraclass correlations in three- and four-level models. Educational and Psychological Measurement, 72(6), 893–909. doi:10.1177/0013164412445193
   Web of Science ®Google Scholar
• Imai, K., Tingley, D., & Yamamoto, T. (2013). Experimental designs for identifying causal mechanisms. Journal of the Royal Statistical Society: Series A (Statistics in Society)), 176(1), 5–51. doi:10.1111/j.1467-985X.2012.01032.x
   Web of Science ®Google Scholar
• Institute of Education Sciences (IES). (2019). Institute of Education Sciences Request for Applications: Education Research Grants CFDA Number: 84.305A. https://ies.ed.gov/funding/pdf/2019_84305A.pdf
   Google Scholar
• Jacob, R., Zhu, P., & Bloom, H. (2010). New empirical evidence for the design of group randomized trials in education. Journal of Research on Educational Effectiveness, 3(2), 157–198. doi:10.1080/19345741003592428
   Google Scholar
• Kazdin, A. E. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology, 3(1), 1–27. doi:10.1146/annurev.clinpsy.3.022806.091432
   PubMed Web of Science ®Google Scholar
• Keeley, R. D., Brody, D. S., Engel, M., Burke, B. L., Nordstrom, K., Moralez, E., Dickinson, L. M., & Emsermann, C. (2016). Motivational interviewing improves depression outcome in primary care: A cluster randomized trial. Journal of Consulting and Clinical Psychology, 84(11), 993–1007. https://psycnet.apa.org/doi/10.1037/ccp0000124 doi:10.1037/ccp0000124
   PubMed Web of Science ®Google Scholar
• Kelcey, B., & Carlisle, J. F. (2013). Learning about teachers’ literacy instruction from classroom observations. Reading Research Quarterly, 48(3), 301–317. doi:10.1002/rrq.51
   Web of Science ®Google Scholar
• Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42(5), 499–530. doi:10.3102/1076998617695506
   Web of Science ®Google Scholar
• Kelcey, B., Dong, N., Spybrook, J., & Shen, Z. (2017). Experimental power for indirect effects in group-randomized studies with group-level mediators. Multivariate Behavioral Research, 52(6), 699–719.
   Google Scholar
• Kelcey, B., Hill, H. C., & Chin, M. J. (2019). Teacher mathematical knowledge, instructional quality, and student outcomes: a multilevel quantile mediation analysis. School Effectiveness and School Improvement, 30(4), 398–431. doi:10.1080/09243453.2019.1570944
   Web of Science ®Google Scholar
• Kelcey, B., & Phelps, G. (2013a). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35(3), 370–390. doi:10.3102/0162373713482766
   Web of Science ®Google Scholar
• Kelcey, B., & Phelps, G. (2013b). Strategies for improving power in school-randomized studies of professional development. Evaluation Review, 37(6), 520–554. doi:10.1177/0193841X14528906
   PubMed Web of Science ®Google Scholar
• Kelcey, B., & Shen, Z. (2016). Multilevel design of school effectiveness studies in sub-Saharan Africa. School Effectiveness and School Improvement, 27(4), 492–510. doi:10.1080/09243453.2016.1168855
   Web of Science ®Google Scholar
• Kenny, D. A., & Judd, C. M. (2014). Power anomalies in testing mediation. Psychological Science, 25(2), 334–339. doi:10.1177/0956797613502676
   PubMed Web of Science ®Google Scholar
• Kisbu-Sakarya, Y., MacKinnon, D. P., & Miočević, M. (2014). The distribution of the product explains normal theory mediation confidence interval estimation. Multivariate Behavioral Research, 49(3), 261–268. doi:10.1080/00273171.2014.903162
   PubMed Web of Science ®Google Scholar
• Kozina, A. (2018a). School-based prevention of anxiety using the “MyFRIENDS” emotional resilience program: Six-month follow-up. International Journal of Psychology, 55(1), 1–8. doi:10.1002/ijop.12553
   Web of Science ®Google Scholar
• Kozina, A. (2018b). Can the “My FRIENDS” anxiety prevention programme also be used to prevent aggression? A six-month follow-up in a school. School Mental Health, 10(4), 500–509. doi:10.1007/s12310-018-9272-5
   Web of Science ®Google Scholar
• Kozlowski, S. W., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes.
   Google Scholar
• Kreft, I. G., De Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30(1), 1–21. doi:10.1207/s15327906mbr3001_1
   PubMed Web of Science ®Google Scholar
• Krull, J. L., & MacKinnon, D. P. (1999). Multilevel mediation modeling in group-based intervention studies. Evaluation Review, 23(4), 418–444. doi:10.1177/0193841X9902300404
   PubMed Web of Science ®Google Scholar
• Layard, R., & Clark, D. M. (2014). Thrive: The power of evidence-based psychological therapies. Penguin.
   Google Scholar
• Livert, D., Rindskopf, D., Saxe, L., & Stirratt, M. (2001). Using multilevel modeling in the evaluation of community-based treatment programs. Multivariate Behavioral Research, 36(2), 155–184. doi:10.1207/S15327906MBR3602_02
   PubMed Web of Science ®Google Scholar
• MacKinnon, D. P. (2008). Multivariate applications series. Introduction to statistical mediation analysis. Taylor & Francis Group/Lawrence Erlbaum Associates.
   Google Scholar
• MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7(1), 83–104. doi:10.1037/1082-989X.7.1.83
   PubMed Web of Science ®Google Scholar
• Mierlo, H., Rutte, C. G., Vermunt, J. K., Kompier, M. A. J., & Doorewaard, J. A. C. M. (2007). A multi-level mediation model of the relationships between team autonomy, individual task design and psychological well-being. Journal of Occupational and Organizational Psychology, 80(4), 647–664. doi:10.1348/096317907X196886
   Web of Science ®Google Scholar
• Mowbray, C. T., Woodward, A. T., Holter, M. C., MacFarlane, P., & Bybee, D. (2009). Characteristics of users of consumer-run drop in centers versus clubhouses. The Journal of Behavioral Health Services & Research, 36(3), 361–371. doi:10.1007/s11414-008-9112-8
   PubMed Web of Science ®Google Scholar
• National Longitudinal Survey of Children and Youth, Statistics Canada. (2019). Government of Canada, Statistics. National Longitudinal Survey of Children and Youth (NLSCY). http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=4450
   Google Scholar
• Phelps, G., Kelcey, B., Liu, S., & Jones, N. (2016). Informing estimates of program effects for studies of mathematics professional development using teacher content knowledge outcomes. Evaluation Review, 40, 383–409.
   Google Scholar
• Pituch, K. A., Murphy, D. L., & Tate, R. L. (2009). Three-level models for indirect effects in school- and class-randomized experiments in education. The Journal of Experimental Education, 78(1), 60–95. doi:10.1080/00220970903224685
   Web of Science ®Google Scholar
• Pituch, K. A., & Stapleton, L. M. (2008). The performance of methods to test upper-level mediation in the presence of nonnormal data. Multivariate Behavioral Research, 43(2), 237–267. doi:10.1080/00273170802034844
   PubMed Web of Science ®Google Scholar
• Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670. doi:10.1177/0049124112460380
   Web of Science ®Google Scholar
• Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41(3), 367–400. doi:10.1207/s15327906mbr4103_5
   PubMed Web of Science ®Google Scholar
• Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98. doi:10.1080/19312458.2012.679848
   Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
   Google Scholar
• Roche, A. M., & Freeman, T. (2004). Brief interventions: good in theory but weak in practice. Drug and Alcohol Review, 23(1), 11–18. doi:10.1080/09595230410001645510
   PubMed Web of Science ®Google Scholar
• Roth, K. J., Wilson, C. D., Taylor, J. A., Stuhlsatz, M. A., & Hvidsten, C. (2019). Comparing the Effects of Analysis-of-Practice and Content-Based Professional Development on Teacher and Student Outcomes in Science. American Educational Research Journal, 56(4), 1217–1253. doi:10.3102/0002831218814759
   Web of Science ®Google Scholar
• SAMSHA. (2007). National Survey on Drug Use and Health. http://www.oas.samhsa.gov/NSDUHlatest.htm
   Google Scholar
• Schwartz, R. (2010). Motivational interviewing (patient-centered counseling) to address childhood obesity. Pediatric Annals, 39(3), 154–158. doi:10.3928/00904481-20100223-06
   PubMed Web of Science ®Google Scholar
• Sikorski, C., Luppa, M., König, H. H., van den Bussche, H., & Riedel-Heller, S. G. (2012). Does GP training in depression care affect patient outcome?-A systematic review and meta-analysis. BMC Health Services Research, 12(1), 10. doi:10.1186/1472-6963-12-10
   PubMed Web of Science ®Google Scholar
• Slade, M., Bird, V., Clarke, E., Le Boutillier, C., McCrone, P., Macpherson, R., Pesola, F., Wallace, G., Williams, J., & Leamy, M. (2015). Supporting recovery in patients with psychosis through care by community-based adult mental health teams (REFOCUS): a multisite, cluster, randomised, controlled trial. The Lancet Psychiatry, 2(6), 503–514. doi:10.1016/S2215-0366(15)00086-3
   PubMed Web of Science ®Google Scholar
• Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312. doi:10.2307/270723
   Google Scholar
• Spybrook, J., Kelcey, B., & Dong, N. (2016). Power analyses for detecting treatment by moderator effects in cluster randomized trials. Journal of Educational and Behavioral Statistics, 6, 605–627.
   Google Scholar
• Spybrook, J., Shi, R., & Kelcey, B. (2016). Progress in the past decade: an examination of the precision of cluster randomized trials funded by the US Institute of Education Sciences. International Journal of Research & Method in Education, 39(3), 255–267. doi:10.1080/1743727X.2016.1150454
   Web of Science ®Google Scholar
• Sterba, S. K. (2017). Partially nested designs in psychotherapy trials: A review of modeling developments. Psychotherapy Research, 27(4), 425–436. doi:10.1080/10503307.2015.1114688
   PubMed Web of Science ®Google Scholar
• Van Voorhees, B. W., Fogel, J., Houston, T. K., Cooper, L. A., Wang, N.-Y., & Ford, D. E. (2005). Beliefs and attitudes associated with the intention to not accept the diagnosis of depression among young adults. Ann Fam Med., 3(1), 38–46. doi:10.1370/afm.273
   Google Scholar
• VanderWeele, T. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544. doi:10.1177/0049124110366236
   PubMed Web of Science ®Google Scholar
• VanderWeele, T. J., Hong, G., Jones, S. M., & Brown, J. L. (2013). Mediation and spillover effects in group-randomized trials: A case study of the 4Rs educational intervention. Journal of the American Statistical Association, 108(502), 469–482. doi:10.1080/01621459.2013.779832
   PubMed Web of Science ®Google Scholar
• VanderWeele, T. J., & Vansteelandt, S. (2009). Conceptual issues concerning mediation, interventions and composition. Statistics and Its Interface, 2(4), 457–468. doi:10.4310/SII.2009.v2.n4.a7
   Web of Science ®Google Scholar
• Westine, C. D. (2016). Finding efficiency in the design of large multisite evaluations: Estimating variances for science achievement studies. American Journal of Evaluation, 37(3), 311–325. doi:10.1177/1098214015624014
   Web of Science ®Google Scholar
• Westine, C., Spybrook, J., & Taylor, J. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37(6), 490–519. doi:10.1177/0193841X14531584
   PubMed Web of Science ®Google Scholar
• Willenbring, M. L., Kivlahan, D., Kenny, M., Grillo, M., Hagedorn, H., & Postier, A. (2004). Beliefs about evidence-based practices in addiction treatment: A survey of Veterans Administration program leaders. Journal of Substance Abuse Treatment, 26(2), 79–85.(03)00161-2 doi:10.1016/S0740-5472(03)00161-2
   PubMed Web of Science ®Google Scholar
• Wilt, J. (2012). Mediators and mechanisms of psychotherapy: Evaluating criteria for causality. Graduate Student Journal of Psychology, 14, 53–60. https://www.researchgate.net/profile/Joshua_Wilt/publication/263041559_Mediators_and_Mechanisms_of_Psychotherapy_Evaluating_Criteria_for_Causality/links/00b495399d3778fc88000000.pdf
   Google Scholar
• Windgassen, S., Goldsmith, K., Moss-Morris, R., & Chalder, T. (2016). Establishing how psychological therapies work: the importance of mediation analysis. Journal of Mental Health, 25(2), 93–99. doi:10.3109/09638237.2015.1124400
   PubMed Web of Science ®Google Scholar
• Xu, B. D., Zhao, S. K., Li, C. R., & Lin, C. J. (2017). Authentic leadership and employee creativity: testing the multilevel mediation model. Leadership & Organization Development Journal, 38(3), 482–498. doi:10.1108/LODJ-09-2015-0194
   Web of Science ®Google Scholar
• Zhang, Z., Zyphur, M., & Preacher, K. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12(4), 695–719. doi:10.1177/1094428108327450
   Web of Science ®Google Scholar