Investigating Sources of Heterogeneity with Three-Step Multilevel Factor Mixture Modeling: Beyond Testing Measurement Invariance in Cross-National Studies

Abstract

We propose the three-step multilevel factor mixture modeling (ML FMM) to test measurement invariance (MI) across many groups and furthermore to model predictors of latent class membership that possibly induce measurement noninvariance. This Monte Carlo simulation study found that information criteria such as Bayesian Information Criterion tended to select a more complex model when sample size was very large. Thus, the adequacy of three-step ML FMM regarding the correct MI detection was demonstrated with an empirically derived information criterion for large data. However, the number of latent classes was overestimated when intraclass correlation was large. For the test of covariate effects, Type I error was well controlled and power was generally adequate when a correct model was identified at Step 1. Using background variables selected from Trends in International Mathematics and Science Study 2011, the application of three-step ML FMM to a cross-national MI test is demonstrated.

Keywords:

• cross-cultural comparative studies
• measurement invariance
• multilevel factor mixture modeling
• three-step approach

Notes

¹ In terms of measurement invariance level, noninvariance size, and covariate effect, the selected conditions include all conditions (2 × 2) under large covariate effect; large scalar noninvariance condition under the small covariate effect; and large metric noninvariance condition under no covariate effect. For each of these selected conditions, ICC (3) × NC (3) × CS (2) are fully crossed for the two latent class conditions, but only NC = 60 was included, that is, ICC (3) × NC (1) × CS (2) for the three latent class conditions. Total 144 conditions were included.

² We treated the variables with four response categories as continuous. When we conducted the proposed ML FMM with selected replications using the continuous variables (before creating four response categories) and the variables with four response categories, the differences in parameter estimates between two types of data were negligible.

³ www.usf.edu/education/faculty/faculty-profiles/eun-sook-kim.aspx

⁴ The need of new IC emerged ad hoc based on the simulation results. Because the proposal of the new IC is not part of the purpose of this study, the details of new IC are reported in Appendix B.

⁵ The traditional ICs such as AIC and BIC also supported the 2-class configural invariance model.

Additional information

Funding

This research was supported by a grant from the American Educational Research Association which receives funds for its “AERA Grants Program” from the National Science Foundation under NSF: [Grant Number DRL-0941014]. Opinions reflect those of the author(s) and do not necessarily reflect those of the granting agencies.