Bootstrap Confidence Intervals for Multilevel Standardized Effect Size

References

• American Educational Research Association (AERA). (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35, 33–40. doi:10.3102/0013189X035006033
   Google Scholar
• American Psychological Association (APA). (2010). Publication manual of the American Psychological Association (6th ed.).
   Google Scholar
• Ames, A. J. (2013). Accuracy and precision of an effect size and its variance from a multilevel model for cluster randomized trials: A simulation study. Multivariate Behavioral Research, 48(4), 592–618. doi:10.1080/00273171.2013.802978
   PubMed Web of Science ®Google Scholar
• Anderson, S. F., & Maxwell, S. E. (2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305–324. doi:10.1080/00273171.2017.1289361
   PubMed Web of Science ®Google Scholar
• Angst, F., Aeschlimann, A., & Angst, J. (2017). The minimal clinically important difference raised the significance of outcome effects above the statistical level, with methodological implications for future studies. Journal of Clinical Epidemiology, 82, 128–136. doi:10.1016/j.jclinepi.2016.11.016
   PubMed Web of Science ®Google Scholar
• Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology. American Psychologist, 73(1), 3–25. doi:10.1037/amp0000191
   PubMed Web of Science ®Google Scholar
• Azzalini, A. (2013). The skew-normal and related families (with the collaboration of Antonella Capitanio). Cambridge University Press. 10.1017/CBO9781139248891
   Google Scholar
• Azzalini, A. (2019). The R package sn: The skew-normal and related distributions such as the skew-t (version 1.5-4) [Computer software manual]. http://azzalini.stat.unipd.it/SN
   Google Scholar
• Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Beretvas, S. N. (2011). Cross-classified and multiple membership models. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). Routledge.
   Google Scholar
• Biesanz, J. C., Falk, C. F., & Savalei, V. (2010). Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research, 45(4), 661–701. doi:10.1080/00273171.2010.498292
   PubMed Web of Science ®Google Scholar
• Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61(1), 27–41. doi:10.1037/0003-066X.61.1.27
   PubMed Web of Science ®Google Scholar
• Bolier, L., Ketelaar, S. M., Nieuwenhuijsen, K., Smeets, O., Gärtner, F. R., & Sluiter, J. K. (2014). Workplace mental health promotion online to enhance well-being of nurses and allied health professionals: A cluster-randomized controlled trial. Internet Interventions, 1(4), 196–204. doi:10.1016/j.invent.2014.10.002
   Google Scholar
• Bull, S. S., Levine, D. K., Black, S. R., Schmiege, S. J., & Santelli, J. (2012). Social media-delivered sexual health intervention: A cluster randomized controlled trial. American Journal of Preventive Medicine, 43(5), 467–474. doi:10.1016/j.amepre.2012.07.022
   PubMed Web of Science ®Google Scholar
• Byrd, J. K. (2007). A call for statistical reform in EAQ. Educational Administration Quarterly, 43(3), 381–391. doi:10.1177/0013161X06297137
   Web of Science ®Google Scholar
• Candel, M. J. J. M., & Breukelen, G. J. P. V. (2009). Varying cluster sizes in trials with clusters in one treatment arm: Sample size adjustments when testing treatment effects with linear mixed models. Statistics in Medicine, 28(18), 2307–2324. doi:10.1002/sim.3620
   PubMed Web of Science ®Google Scholar
• Canty, A., & Ripley, B. D. (2017). boot: Bootstrap R (S-Plus) functions (R package version 1.3-20) [Computer software]. https://cran.r-project.org/web/packages/boot
   Google Scholar
• Carpenter, J. R., Goldstein, H., & Rasbash, J. (2003). A novel bootstrap procedure for assessing the relationship between class size and achievement. Journal of the Royal Statistical Society: Series C (Applied Statistics)), 52(4), 431–443. doi:10.1111/1467-9876.00415
   Web of Science ®Google Scholar
• Chen, L.-T., & Peng, C.-Y J. (2015). The sensitivity of three methods to nonnormality and unequal variances in interval estimation of effect sizes. Behavior Research Methods, 47(1), 107–126. doi:10.3758/s13428-014-0461-3
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
   Google Scholar
• Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312. doi:10.1037/0003-066X.45.12.1304
   Web of Science ®Google Scholar
• Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. doi:10.1177/0956797613504966
   PubMed Web of Science ®Google Scholar
• Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University.
   Google Scholar
• Deen, M., de Rooij, M. (2018). Clusterbootstrap: Analyze clustered data with generalized linear models using the cluster bootstrap (R package version 1.0.0) [Computer software manual]. https://CRAN.R-project.org/package=ClusterBootstrap
   Google Scholar
• Deen, M., & de Rooij, M. (2019). ClusterBootstrap: An R package for the analysis of hierarchical data using generalized linear models with the cluster bootstrap. Behavior Research Methods. doi:10.3758/s13428-019-01252-y
   PubMed Web of Science ®Google Scholar
• Efron, B. (1982). The jackknife, the bootstrap, and other resampling plans. Society for industrial and applied mathematics.
   Google Scholar
• Efron, B. (1987). Better Bootstrap Confidence Intervals. Journal of the American Statistical Association, 82(397), 171–185. doi:10.1080/01621459.1987.10478410
   Web of Science ®Google Scholar
• Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman and Hall. 10.1111/1467-9639.00050
   Google Scholar
• Feingold, A. (2015). Confidence interval estimation for standardized effect sizes in multilevel and latent growth modeling. Journal of Consulting and Clinical Psychology, 83(1), 157–168. doi:10.1037/a0037721
   PubMed Web of Science ®Google Scholar
• Ferguson, C. J. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532–538. doi:10.1037/a0015808
   Web of Science ®Google Scholar
• Field, C. A., Pang, Z., & Welsh, A. H. (2010). Bootstrapping robust estimates for clustered data. Journal of the American Statistical Association, 105(492), 1606–1616. doi:10.1198/jasa.2010.tm09541
   Web of Science ®Google Scholar
• Flynn, T. N., & Peters, T. J. (2004). Use of the bootstrap in analysing cost data from cluster randomised trials: Some simulation results. BMC Health Services Research, 4(1), 33. doi:10.1186/1472-6963-4-33
   PubMed Web of Science ®Google Scholar
• Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2–18. doi:10.1037/a0024338
   PubMed Web of Science ®Google Scholar
• Goldstein, H. (2011). Bootstrapping in multilevel models. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 163–171). Routledge.
   Google Scholar
• Grissom, R. J., & Kim, J. J. (2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). Routledge.
   Google Scholar
• Harrell, F. E. Jr. (2019). rms: Regression modeling strategies (R package version 5.1-3.1) [Computer software manual]. https://CRAN.R-project.org/package=rms
   Google Scholar
• Haug, S., Paz Castro, R., Kowatsch, T., Filler, A., Dey, M., & Schaub, M. P. (2017). Efficacy of a web- and text messaging-based intervention to reduce problem drinking in adolescents: Results of a cluster-randomized controlled trial. Journal of Consulting and Clinical Psychology, 85(2), 147–159. doi:10.1037/ccp0000138
   PubMed Web of Science ®Google Scholar
• Hayes, A. F., & Scharkow, M. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis. Psychological Science, 24(10), 1918–1927. doi:10.1177/0956797613480187
   PubMed Web of Science ®Google Scholar
• Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 107–128. doi:10.3102/10769986006002107
   Google Scholar
• Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. doi:10.3102/1076998606298043
   Web of Science ®Google Scholar
• Hedges, L. V. (2008). What are effect sizes and why do we need them? Child Development Perspectives, 2(3), 167–171. doi:10.1111/j.1750-8606.2008.00060.x
   Web of Science ®Google Scholar
• Hedges, L. V. (2009). Effect sizes in nested designs. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 337–355). Russell Sage Foundation.
   Google Scholar
• Hedges, L. V. (2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36(3), 346–380. doi:10.3102/1076998610376617
   Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi:10.3102/0162373707299706
   Web of Science ®Google Scholar
• Hinkley, D. V., & Shi, S. (1989). Importance sampling and the nested bootstrap. Biometrika, 76(3), 435–446. doi:10.1093/biomet/76.3.435
   Web of Science ®Google Scholar
• Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). Routledge.
   Google Scholar
• Huang, F. L. (2018). Using cluster bootstrapping to analyze nested data with a few clusters. Educational and Psychological Measurement. 78(2), 297–318. doi:10.1177/0013164416678980
   Google Scholar
• Jones, S. M., Brown, J. L., Hoglund, W. L. G., & Aber, J. L. (2010). A school-randomized clinical trial of an integrated social–emotional learning and literacy intervention: Impacts after 1 school year. Journal of Consulting and Clinical Psychology, 78(6), 829–842. doi:10.1037/a0021383
   PubMed Web of Science ®Google Scholar
• Kelley, K. (2005). The effects of nonnormal distributions on confidence intervals around the standardized mean difference: Bootstrap and parametric confidence intervals. Educational and Psychological Measurement, 65(1), 51–69. doi:10.1177/0013164404264850
   Web of Science ®Google Scholar
• Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17(2), 137–152. doi:10.1037/a0028086
   PubMed Web of Science ®Google Scholar
• Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). American Psychological Association.
   Google Scholar
• Lai, M. H. C. (2019). bootmlm: Bootstrap resampling for multilevel models (R Package Version 0.0.1) [Computer software]. doi:10.5281/zenodo.1879127.
   Google Scholar
• Lai, M. H. C., & Kwok, O.-M. (2014). Standardized mean differences in two-level cross-classified random effects models. Journal of Educational and Behavioral Statistics, 39(4), 282–302. doi:10.3102/1076998614532950
   Web of Science ®Google Scholar
• Lai, M. H. C., & Kwok, O.-M. (2016). Estimating standardized effect sizes for two-and three-level partially nested data. Multivariate Behavioral Research, 51(6), 1–756. doi:10.1080/00273171.2016.1231606
   PubMed Web of Science ®Google Scholar
• Littell, R. C. (2006). SAS for mixed models. SAS Institute.
   Google Scholar
• Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. doi:10.1027/1614-2241.1.3.86
   Google Scholar
• MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39(1), 99–128. doi:10.1207/s15327906mbr3901_4
   PubMed Web of Science ®Google Scholar
• Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9(2), 147–163. doi:10.1037/1082-989X.9.2.147
   PubMed Web of Science ®Google Scholar
• National Center for Educational Statistics (NCES). (2012). NCES statistical standards (rev. ed.). U. S. Department of Education.
   Google Scholar
• Norusis, M. J. M. J. (2012). IBM SPSS statistics 19 advanced statistical procedures companion. Prentice Hall.
   Google Scholar
• Owen, A. B., & Eckles, D. (2012). Bootstrapping data arrays of arbitrary order. The Annals of Applied Statistics, 6(3), 895–927. doi:10.1214/12-AOAS547
   Web of Science ®Google Scholar
• Peng, C.-Y J., Chen, L.-T., Chiang, H.-M., & Chiang, Y.-C. (2013). The impact of APA and AERA guidelines on effect size reporting. Educational Psychology Review, 25(2), 157–209. doi:10.1007/s10648-013-9218-2
   Web of Science ®Google Scholar
• Perugini, M., Gallucci, M., & Costantini, G. (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9(3), 319–332. doi:10.1177/1745691614528519
   Web of Science ®Google Scholar
• Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36(4), 717–731. doi:10.3758/BF03206553
   PubMedGoogle Scholar
• R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.1) [Computer software]. R Foundation for Statistical Computing.
   Google Scholar
• Rasbash, J., Steele, F., Browne, W. J., Goldstein, H. (2019). A user’s guide to MLwiN, v3.03. http://www.bristol.ac.uk/cmm/media/software/mlwin/downloads/manuals/3-03/manual-web.pdf
   Google Scholar
• Roberts, J. K., & Fan, X. (2004). Bootstrapping within the multilevel/hierarchical linear modeling framework: A primer for use with SAS and SPLUS. Multiple Linear Regression Viewpoints, 30, 23–34.
   Google Scholar
• Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231–244). Russel Sage Foundation.
   Google Scholar
• Schulz, K. F., Altman, D. G., Moher, D., & the CONSORT Group (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. PLoS Medicine, 7(3), e1000251. doi:10.1371/journal.pmed.1000251
   PubMed Web of Science ®Google Scholar
• Searle, S. R., Casella, G., & McCulloch, C. E. (2006). Variance components (2nd ed.). Wiley.
   Google Scholar
• Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61(4), 605–632. doi:10.1177/00131640121971392
   Web of Science ®Google Scholar
• Snijders, T. A. B., & Bosker, R. J. (1994). Modeled variance in two-level models. Sociological Methods & Research, 22(3), 342–363. doi:10.1177/0049124194022003004
   Web of Science ®Google Scholar
• Thai, H.-T., Mentré, F., Holford, N. H. G., Veyrat-Follet, C., & Comets, E. (2013). A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed-effects models. Pharmaceutical Statistics, 12(3), 129–140. doi:10.1002/pst.1561
   PubMed Web of Science ®Google Scholar
• Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 25–32. doi:10.3102/0013189X031003025
   Google Scholar
• Thompson, B. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44(5), 423–432. doi:10.1002/pits.20234
   Web of Science ®Google Scholar
• Ukoumunne, O. C., Davison, A. C., Gulliford, M. C., & Chinn, S. (2003). Non-parametric bootstrap confidence intervals for the intraclass correlation coefficient. Statistics in Medicine, 22(24), 3805–3821. doi:10.1002/sim.1643
   PubMed Web of Science ®Google Scholar
• Vallejo Seco, G., Ato García, M., Fernández García, M. P., & Livacic Rojas, P. E. (2013). Multilevel bootstrap analysis with assumptions violated. Psicothema, 25, 520–528. 10.7334/psicothema2013.58
   PubMed Web of Science ®Google Scholar
• Van der Leeden, R., Meijer, E., & Busing, F. M. T. A. (2008). Resampling multilevel models. In J. de Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 401–433). Springer.
   Google Scholar
• van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20(3), 293–309. doi:10.1037/met0000025
   PubMed Web of Science ®Google Scholar
• Wang, J., Carpenter, J. R., & Kepler, M. A. (2006). Using SAS to conduct nonparametric residual bootstrap multilevel modeling with a small number of groups. Computer Methods and Programs in Biomedicine, 82(2), 130–143. doi:10.1016/j.cmpb.2006.02.006
   PubMed Web of Science ®Google Scholar
• White, H. R., Mun, E. Y., & Morgan, T. J. (2008). Do brief personalized feedback interventions work for mandated students or is it just getting caught that works? Psychology of Addictive Behaviors, 22(1), 107–116. doi:10.1037/0893-164X.22.1.107
   PubMed Web of Science ®Google Scholar
• Wu, C. F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. The Annals of Statistics, 14(4), 1261–1295. doi:10.1214/aos/1176350142
   Web of Science ®Google Scholar