Multilevel Mediation With Small Samples: A Cautionary Note on the Multilevel Structural Equation Modeling Framework

REFERENCES

• Aroian, L. A. (1947). The probability function of the product of two normally distributed variables. The Annals of Mathematical Statistics, 18, 265–271. doi:10.1214/aoms/1177730442
   Google Scholar
• Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1982. doi:10.1037/0022-3514.51.6.1173
   PubMed Web of Science ®Google Scholar
• Bauer, D. J. (2003). Estimating multilevel linear models as structural equation models. Journal of Educational and Behavioral Statistics, 28, 135–167. doi:10.3102/10769986028002135
   Web of Science ®Google Scholar
• Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11, 142–163. doi:10.1037/1082-989X.11.2.142
   PubMed Web of Science ®Google Scholar
• Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology, 10, 1–11. doi:10.1027/1614-2241/a000062
   Web of Science ®Google Scholar
• Biesanz, J. C., Falk, C. F., & Savalei, V. (2010). Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research, 45, 661–701. doi:10.1080/00273171.2010.498292
   PubMed Web of Science ®Google Scholar
• Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144–152. doi:10.1111/bmsp.1978.31.issue-2
   Web of Science ®Google Scholar
• Browne, W. J., & Draper, D. (2006). A comparison of Bayesian and likelihood-based methods for fitting multilevel models. Bayesian Analysis, 1, 473–514. doi:10.1214/06-BA117
   Web of Science ®Google Scholar
• Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90, 414–427. doi:10.1162/rest.90.3.414
   Web of Science ®Google Scholar
• Carr, E., & Chung, H. (2014). Employment insecurity and life satisfaction: The moderating influence of labour market policies across Europe. Journal of European Social Policy, 24, 383–399. doi:10.1177/0958928714538219
   Web of Science ®Google Scholar
• Cheung, M. W. L. (2013). Implementing restricted maximum likelihood estimation in structural equation models. Structural Equation Modeling, 20, 157–167. doi:10.1080/10705511.2013.742404
   Web of Science ®Google Scholar
• Clarke, P. (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology & Community Health, 62, 752–758. doi:10.1136/jech.2007.060798
   PubMed Web of Science ®Google Scholar
• Craig, C. C. (1936). On the frequency function of xy. The Annals of Mathematical Statistics, 7, 1–15. doi:10.1214/aoms/1177732541
   Google Scholar
• Curran, P. J. (2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38, 529–569. doi:10.1207/s15327906mbr3804_5
   PubMed Web of Science ®Google Scholar
• Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., … Lee, R. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79, 69–102. doi:10.3102/0034654308325581
   Web of Science ®Google Scholar
• Depaoli, S. (2014). The impact of inaccurate “informative” priors for growth parameters in Bayesian growth mixture modeling. Structural Equation Modeling, 21, 239–252. doi:10.1080/10705511.2014.882686
   Web of Science ®Google Scholar
• Depaoli, S., & Clifton, J. P. (2015). A Bayesian approach to multilevel structural equation modeling with continuous and dichotomous outcomes. Structural Equation Modeling, 22, 327–351. doi:10.1080/10705511.2014.937849
   Web of Science ®Google Scholar
• Depaoli, S., & van de Schoot, R. (2015). Improving transparency and replication in Bayesian statistics: The WAMBS-checklist. Psychological Methods. Advance online publication. doi:10.1037/met0000065
   PubMed Web of Science ®Google Scholar
• Enders, C. K., Fairchild, A. J., & MacKinnon, D. P. (2013). A Bayesian approach for estimating mediation effects with missing data. Multivariate Behavioral Research, 48, 340–369. doi:10.1080/00273171.2013.784862
   PubMed Web of Science ®Google Scholar
• Ferron, J. M., Bell, B. A., Hess, M. R., Rendina-Gobioff, G., & Hibbard, S. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41, 372–384. doi:10.3758/BRM.41.2.372
   PubMed Web of Science ®Google Scholar
• Fritz, M. S., & MacKinnon, D. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18, 233–239. doi:10.1111/j.1467-9280.2007.01882.x
   PubMed Web of Science ®Google Scholar
• Fritz, M. S., Taylor, A. B., & MacKinnon, D. P. (2012). Explanation of two anomalous results in statistical mediation analysis. Multivariate Behavioral Research, 47, 61–87. doi:10.1080/00273171.2012.640596
   PubMed Web of Science ®Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi:10.3102/0162373707299706
   Web of Science ®Google Scholar
• Hox, J. J., & Maas, C. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8, 157–174. doi:10.1207/S15328007SEM0802_1
   Web of Science ®Google Scholar
• Hox, J. J., Maas, C. J., & Brinkhuis, M. J. (2010). The effect of estimation method and sample size in multilevel structural equation modeling. Statistica Neerlandica, 64, 157–170. doi:10.1111/j.1467-9574.2009.00445.x
   Web of Science ®Google Scholar
• Hox, J. J., van de Schoot, R., & Matthijsse, S. (2012). How few countries will do? Comparative survey analysis from a Bayesian perspective. Survey Research Methods, 6, 87–93.
   Web of Science ®Google Scholar
• Kish, L. (1965). Survey sampling. New York, NY: Wiley.
   Google Scholar
• Kreft, I. G. (1996). Are multilevel techniques necessary? An overview, including simulation studies (Working paper). Los Angeles, CA: California State University.
   Google Scholar
• Krull, J. L., & MacKinnon, D. P. (1999). Multilevel mediation modeling in group-based intervention studies. Evaluation Review, 23, 418–444. doi:10.1177/0193841X9902300404
   PubMed Web of Science ®Google Scholar
• Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36, 249–277. doi:10.1207/S15327906MBR3602_06
   PubMed Web of Science ®Google Scholar
• Kruschke, J. K., Aguinis, H., & Joo, H. (2012). The time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15, 722–752. doi:10.1177/1094428112457829
   Web of Science ®Google Scholar
• Li, P., & Redden, D. T. (2015). Small sample performance of bias‐corrected sandwich estimators for cluster‐randomized trials with binary outcomes. Statistics in Medicine, 34, 281–296. doi:10.1002/sim.v34.2
   PubMed Web of Science ®Google Scholar
• Maas, C. J., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-2241.1.3.86
   Google Scholar
• MacKinnon, D. P., Fritz, M. S., Williams, J., & Lockwood, C. M. (2007). Distribution of the product confidence limits for the indirect effect: Program PRODCLIN. Behavior Research Methods, 39, 384–389. doi:10.3758/BF03193007
   PubMed Web of Science ®Google Scholar
• MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104. doi:10.1037/1082-989X.7.1.83
   PubMed Web of Science ®Google Scholar
• McNeish, D. (2014). Modeling sparsely clustered data: Design-based, model-based, and single-level methods. Psychological Methods, 19, 552–563. doi:10.1037/met0000024
   PubMed Web of Science ®Google Scholar
• McNeish, D. (2016a). Using data-dependent priors to mitigate small sample bias in latent growth models. Journal of Educational and Behavioral Statistics, 41, 27–56. doi:10.3102/1076998615621299
   Web of Science ®Google Scholar
• McNeish, D. (2016b). On using Bayesian methods to address small sample problems. Structural Equation Modeling, 23, 750–773. doi:10.1080/10705511.2016.1186549
   Web of Science ®Google Scholar
• McNeish, D., & Stapleton, L. M. (2016). The effect of small sample size on two level model estimates: A review and illustration. Educational Psychology Review, 28, 295–314. doi:10.1007/s10648-014-9287-x
   Web of Science ®Google Scholar
• Mehta, P. D., & Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10, 259–284. doi:10.1037/1082-989X.10.3.259
   PubMed Web of Science ®Google Scholar
• Miranda, R., MacKillop, J., Treloar, H., Blanchard, A., Tidey, J. W., Swift, R. M., … Monti, P. M. (2016). Biobehavioral mechanisms of topiramate’s effects on alcohol use: An investigation pairing laboratory and ecological momentary assessments. Addiction Biology, 21, 171–182. doi:10.1111/adb.12192
   PubMed Web of Science ®Google Scholar
• Morel, J. G., Bokossa, M. C., & Neerchal, N. K. (2003). Small sample correction for the variance of GEE estimators. Biometrical Journal, 45, 395–409. doi:10.1002/bimj.200390021
   Web of Science ®Google Scholar
• Muthén, B. O. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557–585. doi:10.1007/BF02296397
   Web of Science ®Google Scholar
• Muthén, B. O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28, 338–354. doi:10.1111/j.1745-3984.1991.tb00363.x
   Web of Science ®Google Scholar
• Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376–398. doi:10.1177/0049124194022003006
   Web of Science ®Google Scholar
• Muthén, B. O., & Asparouhov, T. (2008). Growth mixture modeling: Analysis with non-Gaussian random effects. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 143–165). Boca Raton, FL: Chapman & Hall/CRC.
   Google Scholar
• Muthen, L.K., & Muthen, B,O. (2010) Growth modeling with latent variables using Mplus: Introductory and intermediate growth models. Retrieved from http://www.statmodel.com/download/Topic3-v.pdf.
   Google Scholar
• Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. doi:10.2307/271070
   Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide: Statistical analysis with latent variables (7th ed.). Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Pituch, K. A., & Stapleton, L. M. (2011). Hierarchical linear and structural equation modeling approaches to mediation analysis in randomized field experiments. In M. Williams & P. Vogt (Eds.), The Sage handbook of innovation in social research methods (pp. 590–619). Thousand Oaks, CA: Sage.
   Google Scholar
• Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41, 367–400. doi:10.1207/s15327906mbr4103_5
   PubMed Web of Science ®Google Scholar
• Pituch, K. A., Whittaker, T. A., & Stapleton, L. M. (2005). A comparison of methods to test for mediation in multisite experiments. Multivariate Behavioral Research, 40, 1–23. doi:10.1207/s15327906mbr4001_1
   PubMed Web of Science ®Google Scholar
• Polansky, A. M. (1999). Upper bounds on the true coverage probability of bootstrap percentile type confidence intervals. The American Statistician, 53, 362–369.
   Web of Science ®Google Scholar
• Preacher, K. J. (2011). Multilevel SEM strategies for evaluating mediation in three-level data. Multivariate Behavioral Research, 46, 691–731. doi:10.1080/00273171.2011.589280
   PubMed Web of Science ®Google Scholar
• Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6, 77–98. doi:10.1080/19312458.2012.679848
   Google Scholar
• Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2011). Alternative methods for assessing mediation in multilevel data: The advantages of multilevel SEM. Structural Equation Modeling, 18, 161–182. doi:10.1080/10705511.2011.557329
   Web of Science ®Google Scholar
• Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209–233. doi:10.1037/a0020141
   PubMed Web of Science ®Google Scholar
• Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modeling. Psychometrika, 69, 167–190. doi:10.1007/BF02295939
   Web of Science ®Google Scholar
• Snijders, T. A., & Bosker, R. J. (1993). Standard errors and sample sizes for two-level research. Journal of Educational and Behavioral Statistics, 18, 237–259. doi:10.3102/10769986018003237
   Web of Science ®Google Scholar
• Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312. doi:10.2307/270723
   Google Scholar
• Springer, M. D., & Thompson, W. E. (1966). The distribution of products of independent random variables. SIAM Journal on Applied Mathematics, 14, 511–526. doi:10.1137/0114046
   Web of Science ®Google Scholar
• Tofighi, D., & MacKinnon, D. P. (2011). RMediation: An R package for mediation analysis confidence intervals. Behavior Research Methods, 43, 692–700. doi:10.3758/s13428-011-0076-x
   PubMed Web of Science ®Google Scholar
• van de Schoot, R., Broere, J. J., Perryck, K. H., Zondervan-Zwijnenburg, M., & van Loey, N. E. (2015). Analyzing small data sets using Bayesian estimation: The case of posttraumatic stress symptoms following mechanical ventilation in burn survivors. European Journal of Psychotraumatology, 6.
   Web of Science ®Google Scholar
• van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & Aken, M. A. (2014). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85, 842–860. doi:10.1111/cdev.12169
   PubMed Web of Science ®Google Scholar
• van de Schoot, R., Winter, S., Ryan, O., Zondervan-Zwijnenburg, M., & Depaoli, S. (2017). A systematic review of Bayesian papers in psychology: The last 25 years. Psychological Methods. Advance online publication. doi:10.1037/met0000100
   PubMedGoogle Scholar
• van der Leeden, R., Meijer, E., & Busing, F. M. T. A. (2008). Resampling multilevel models. In J. De Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 401–433). New York, NY: Springer.
   Google Scholar
• van Noorden, R., Maher, B., & Nuzzo, R. (2014). The top 100 papers. Nature, 514, 550–553. doi:10.1038/514550a
   PubMed Web of Science ®Google Scholar
• Wang, J., Carpenter, J. R., & Kepler, M. A. (2006). Using SAS to conduct nonparametric residual bootstrap multilevel modeling with a small number of groups. Computer Methods and Programs in Biomedicine, 82, 130–143. doi:10.1016/j.cmpb.2006.02.006
   PubMed Web of Science ®Google Scholar
• Yuan, Y., & MacKinnon, D. P. (2009). Bayesian mediation analysis. Psychological Methods, 14, 301–322. doi:10.1037/a0016972
   PubMed Web of Science ®Google Scholar
• Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719. doi:10.1177/1094428108327450
   Web of Science ®Google Scholar
• Zyphur, M. J., & Oswald, F. L. (2015). Bayesian estimation and inference: A user’s guide. Journal of Management, 41, 390–420. doi:10.1177/0149206313501200
   Web of Science ®Google Scholar