A Comprehensive Comparison of Model Selection Methods for Testing Factorial Invariance

Abstract

This study compared the performance of (1) model selection methods based on a point estimate (MSM-P), (2) partially Bayesian methods, and (3) fully Bayesian methods, for testing factorial invariance. The MSM-P considered included the likelihood ratio test (LRT) and a number of information criteria (IC). The Bayesian methods tested included the deviance information criterion (DIC), widely available information criterion (WAIC), and leave-one-out cross-validation (LOO). We investigated true positive and false positive rates of each model selection method under conditions varying in sample size, proportion of non-invariant items, and pattern, location and magnitude of non-invariance. The impact of various prior choices on Bayesian model selection results was also examined. Results showed that DIC, WAIC, and LOO performed comparably well, had a good balance between true and false positive rates, were flexible given the prior choices, and may complement MSM-P in factorial invariance testing. Implications and recommendations based on the findings were discussed.

Keywords:

• Factorial invariance
• Bayesian model selection
• widely available information criterion
• leave-one-out cross-validation

Acknowledgement

We thank the editor and two anonymous reviewers whose insightful comments helped improve and clarify this manuscript.

Authors’ Note

The second author is currently employed at the Educational Testing Service, Princeton, New Jersey.

Notes

¹ Note that “BIC (or BIC’) provides a simple and accurate approximation to Bayes factors” (Raftery, 1995, p. 155). For this reason, BIC is considered a Bayesian information criteria in some previous studies (e.g., Lu et al., 2017). In our study, we consider BIC as a MSM-P due to that BIC is computed as a function of only a point estimate, instead of using pointwise information from the entire posterior distribution.

² When the number of non-invariant items was 1, only the decreasing pattern can be evaluated and thus 16 conditions were examined: 2 locations of non-invariance × 2 magnitudes of non-invariance × 4 sample sizes. With 2, 4, 6, or 8 non-invariant items, 32 (16 × 2 locations of non-invariance) conditions for each were examined. In total, 144 (16 + 32 × 4) conditions were created for non-invariant data generation.