Multilevel Mediation With Small Samples: A Cautionary Note on the Multilevel Structural Equation Modeling Framework

Abstract

Multilevel structural equation modeling (ML-SEM) for multilevel mediation is noted for its flexibility over a system of multilevel models (MLMs). Sample size requirements are an overlooked limitation of ML-SEM (100 clusters is recommended). We find that 89% of ML-SEM studies have fewer than 100 clusters and the median number is 44. Furthermore, 75% of ML-SEM studies implement 2–1–1 or 1–1–1 models, which can be equivalently fit with MLMs. MLMs theoretically have lower sample size requirements, although studies have yet to assess small sample performance for multilevel mediation. We conduct a simulation to address this pervasive problem. We find that MLMs have more desirable small sample performance and can be trustworthy with 10 clusters. Importantly, many studies lack the sample size and model complexity to necessitate ML-SEM. Although ML-SEM is undeniably more flexible and uniquely positioned for difficult problems, small samples often can be more effectively and simply addressed with MLMs.

Keywords:

• multilevel mediation
• multilevel SEM
• small sample

Notes

¹ This equivalence is frequently cited in the context of growth models where there is the most overlap between the modeling frameworks. However, the equivalence can be more tenuous for other types of models (Bauer, 2003). Even within the context of growth models, there are a handful of models that can only be specified in one framework but not the other. For instance, the latent basis model, which estimates loadings from the slope factor to the observed variables to empirically model nonlinear growth, is easily fit as a SEM but cannot be fit as an MLM (Curran, 2003). Thus, even though there can be a large degree of overlap between the frameworks, there is not always a strict one-to-one mapping. A presentation by Muthén and Muthén (2010) further differentiates the MLM and SEM frameworks (Slides 29–38, https://www.statmodel.com/download/Topic3-v.pdf).

² Additionally, there are no attempts to extend the Kenward–Roger correction to SEM, presumably because a primary component of the correction is the degrees of freedom for inferential tests. SEM software tends to use asymptotic Z tests for inference instead of finite sample test statistics that require degrees of freedom.

³ Although such manipulations are justifiable, the oft-cited paper by Curran (2003) questions the utility of such a transformation. As Curran (2003) noted, “Of most importance, I believe that if no other elements of the SEM are incorporated in a MLM then the SEM approach has nothing unique to offer over the standard MLM. … My recommendation is that if a particular research hypothesis can be fully evaluated using a standard MLM, by all means use the MLM approach” (pp. 564–565). Essentially, if SEMs necessarily convert models to MLMs to implement REML, researchers might as well cut out the intermediate SEM step and resort to MLMs from the beginning.

⁴ Our focus here is multilevel mediation broadly, so we do not provide extensive background detail on MCMC estimation and we assume that readers have a general understanding of Bayesian methods. For readers unfamiliar with Bayesian methods, readable introductions can be found in Kruschke, Aguinis, and Joo (2012), van de Schoot et al. (2014), and Zyphur and Oswald (2015).

⁵ Although these power values might seem high for data with few clusters, keep in mind that the path from M to Y occurs at Level 1. In the smallest sample size condition in the simulation design, the overall Level 1 sample size is about 100 and the 25-cluster, small cluster size condition contains about 250 individuals. The values obtained in this study are approximately equal to values obtained by MacKinnon et al. (2002) if the effective sample sizes in this study are compared to their single-level sample sizes. The effective sample size approximates the amount of information present (in terms of sample size) if there were no clustering where where m is the average cluster size, N is the total sample size, N_eff is the effective sample size, and ρ is the intraclass correlation (Kish, 1965).