Testing Factorial Invariance in Multilevel Data: A Monte Carlo Study

Abstract

Testing factorial invariance has recently gained more attention in different social science disciplines. Nevertheless, when examining factorial invariance, it is generally assumed that the observations are independent of each other, which might not be always true. In this study, we examined the impact of testing factorial invariance in multilevel data, especially when the dependency issue is not taken into account. We considered a set of design factors, including number of clusters, cluster size, and intraclass correlation (ICC) at different levels. The simulation results showed that the test of factorial invariance became more liberal (or had inflated Type I error rate) in terms of rejecting the null hypothesis of invariance held between groups when the dependency was not considered in the analysis. Additionally, the magnitude of the inflation in the Type I error rate was a function of both ICC and cluster size. Implications of the findings and limitations are discussed.

Keywords:

• measurement invariance
• factorial invariance
• multilevel models
• multiple group analysis
• intra-class correlation

Notes

^aAnalysis type = TWOLEVEL.

^bAnalysis type = COMPLEX.

^cFor the conditions in which admissible solution rates were below .70, the number of replications increased to achieve a reasonable ASR. Because the power and Type I error did not show significant difference after increasing ASR, the original results are reported.