A Comprehensive Comparison of Model Selection Methods for Testing Factorial Invariance

References

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. doi:10.1109/TAC.1974.1100705
   Web of Science ®Google Scholar
• Barendse, M., Albers, C., Oort, F., & Timmerman, M. (2014). Measurement bias detection through Bayesian factor analysis. Frontiers in Psychology, 5, 1–9. doi:10.3389/fpsyg.2014.00001
   PubMed Web of Science ®Google Scholar
• Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
   PubMed Web of Science ®Google Scholar
• Bollen, K. A., Harden, J. J., Ray, S., & Zavisca, J. (2014). BIC and alternative Bayesian information criteria in the selection of structural equation models. Structural Equation Modeling, 21, 1–19. doi:10.1080/10705511.2014.856691
   PubMed Web of Science ®Google Scholar
• Brannick, M. T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16, 201–213. doi:10.1002/(ISSN)1099-1379
   Web of Science ®Google Scholar
• Brooks, S., Smith, J., Vehtari, A., Plummer, M., Stone, M., Robert, C. P., … Dawid, A. (2002). Discussion on the paper by Spiegelhalter, Best, Carlin and van der Linde. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 64, 616–639.
   Web of Science ®Google Scholar
• Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York, NY: Springer-Verlag.
   Google Scholar
• Carlin, B. P., & Louis, T. A. (2001). Bayes and empirical Bayes methods for data analysis (2nd ed.). Boca Raton, FL: Chapman and Hall/CRC.
   Google Scholar
• Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464–504. doi:10.1080/10705510701301834
   Web of Science ®Google Scholar
• Chen, Q., Luo, W., Palardy, G. J., Glaman, R., & McEnturff, A. (2017). The efficacy of common fit indices for enumerating classes in growth mixture models when nested data structure is ignored: A Monte Carlo study. SAGE Open, 7, 1–19. doi:10.1177/2158244017700459
   Web of Science ®Google Scholar
• Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255. doi:10.1207/S15328007SEM0902_5
   Web of Science ®Google Scholar
• Choi, I.-H., Paek, I., & Cho, S.-J. (2017). The impact of various class-distinction features on model selection in the mixture Rasch model. The Journal of Experimental Education, 85, 411–424. doi:10.1080/00220973.2016.1250208
   Web of Science ®Google Scholar
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. doi:10.1007/BF02310555
   Web of Science ®Google Scholar
• Da Silva, M. A., Bazán, J. L., & Huggins-Manley, A. C. (2018). Sensitivity analysis and choosing between alternative polytomous IRT models using Bayesian model comparison criteria. Communications in Statistics-Simulation and Computation, 48, 601–620. doi:10.1080/03610918.2017.1390126
   Web of Science ®Google Scholar
• De Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2013). Assessing parameter invariance in the BLIM: Bipartition models. Psychometrika, 78, 710–724. doi:10.1007/s11336-013-9325-5
   PubMed Web of Science ®Google Scholar
• Denwood, M. J. (2016). runjags: An R package providing interface utilities, model templates, parallel computing methods and additional distributions for MCMC models in JAGS. Journal of Statistical Software, 71, 1–25. doi:10.18637/jss.v071.i09
   Web of Science ®Google Scholar
• Depaoli, S. (2014). The impact of inaccurate “informative” priors for growth parameters in Bayesian growth mixture modeling. Structural Equation Modeling, 21, 239–252. doi:10.1080/10705511.2014.882686
   Web of Science ®Google Scholar
• Finch, W. H., & Miller, J. (2019). The use of incorrect informative priors in the estimation of MIMIC model parameters with small sample sizes. Structural Equation Modeling, 26, 497–508. doi:10.1080/10705511.2018.1553111
   Web of Science ®Google Scholar
• French, B. F., & Finch, W. H. (2008). Multigroup confirmatory factor analysis: Locating the invariant referent sets. Structural Equation Modeling, 15, 96–113. doi:10.1080/10705510701758349
   Web of Science ®Google Scholar
• Geisser, S., & Eddy, W. F. (1979). A predictive approach to model selection. Journal of the American Statistical Association, 74, 153–160. doi:10.1080/01621459.1979.10481632
   Web of Science ®Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis (3rd Ed. ed.). Boca Raton, FL: CRC press.
   Google Scholar
• Gelman, A., Hwang, J., & Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24, 997–1016. doi:10.1007/s11222-013-9416-2
   Web of Science ®Google Scholar
• Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. doi:10.1214/ss/1177011136
   Google Scholar
• Ghosh, J., & Dunson, D. B. (2009). Default prior distributions and efficient posterior computation in Bayesian factor analysis. Journal of Computational and Graphical Statistics, 18, 306–320. doi:10.1198/jcgs.2009.07145
   PubMed Web of Science ®Google Scholar
• Hoijtink, H., & Chow, S.-M. (2017). Bayesian hypothesis testing: Editorial to the Special Issue on Bayesian data analysis. Psychological Methods, 22, 211. doi:10.1037/met0000143
   PubMed Web of Science ®Google Scholar
• Huang, P.-H. (2017). Asymptotics of AIC, BIC, and RMSEA for model selection in structural equation modeling. Psychometrika, 82, 407–426. doi:10.1007/s11336-017-9572-y
   PubMed Web of Science ®Google Scholar
• Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76, 297–307. doi:10.1093/biomet/76.2.297
   Web of Science ®Google Scholar
• Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural Equation Modeling, 23(4), 555–566. doi:10.1080/10705511.2016.1154793
   Google Scholar
• Jöreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409–426. doi:10.1007/BF02291366
   Web of Science ®Google Scholar
• Jorgensen, T. D. (2017). Applying permutation tests and multivariate modification indices to configurally invariant models that need respecification. Frontiers in Psychology, 8, 1–9. doi:10.3389/fpsyg.2017.00001
   PubMedGoogle Scholar
• Jorgensen, T. D., Kite, B. A., Chen, P.-Y., & Short, S. D. (2018). Permutation randomization methods for testing measurement equivalence and detecting differential item functioning in multiple-group confirmatory factor analysis. Psychological Methods, 23, 708–728. doi:10.1037/met0000152
   PubMed Web of Science ®Google Scholar
• Kim, E. S., Cao, C., Wang, Y., & Nguyen, D. T. (2017). Measurement invariance testing with many groups: A comparison of five approaches. Structural Equation Modeling, 24, 524–544. doi:10.1080/10705511.2017.1304822
   Web of Science ®Google Scholar
• Kim, E. S., & Yoon, M. (2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling, 18, 212–228. doi:10.1080/10705511.2011.557337
   Web of Science ®Google Scholar
• Kim, E. S., Yoon, M., & Lee, T. (2012). Testing measurement invariance using MIMIC: Likelihood ratio test with a critical value adjustment. Educational and Psychological Measurement, 72, 469–492. doi:10.1177/0013164411427395
   Web of Science ®Google Scholar
• Kim, E. S., Yoon, M., Wen, Y., Luo, W., & Kwok, O.-M. (2015). Within-level group factorial invariance with multilevel data: Multilevel factor mixture and multilevel MIMIC models. Structural Equation Modeling, 22, 603–616. doi:10.1080/10705511.2014.938217
   Web of Science ®Google Scholar
• Kim, S.-H., Cohen, A. S., Cho, S.-J., & Eom, H. J. (2019). Use of information criteria in the study of group differences in trace lines. Applied Psychological Measurement, 43, 95–112. doi:10.1177/0146621618772292
   PubMed Web of Science ®Google Scholar
• Kite, B. A., Jorgensen, T. D., & Chen, P.-Y. (2018). Random permutation testing applied to measurement invariance testing with ordered-categorical indicators. Structural Equation Modeling, 25, 573–587. doi:10.1080/10705511.2017.1421467
   Web of Science ®Google Scholar
• Lee, S. Y. (2007). Bayesian estimation of structural equation models. In S. Y. Lee (Ed.), Structural equation modeling: A Bayesian approach (pp. 67–109). Chichester, England: John Wiley & Sons Inc.
   Google Scholar
• Lee, T., Cai, L., & Kuhfeld, M. (2016). A poor person’s posterior predictive checking of structural equation models. Structural Equation Modeling, 23, 206–220. doi:10.1080/10705511.2015.1014041
   Web of Science ®Google Scholar
• Li, T., Xie, C., & Jiao, H. (2017). Assessing fit of alternative unidimensional polytomous IRT models using posterior predictive model checking. Psychological Methods, 22, 397–408. doi:10.1037/met0000082
   PubMed Web of Science ®Google Scholar
• Liang, X., Yang, Y., & Huang, J. (2018). Evaluation of structural relationships in autoregressive cross-lagged models under longitudinal approximate invariance: A Bayesian analysis. Structural Equation Modeling, 25, 558–572. doi:10.1080/10705511.2017.1410706
   Web of Science ®Google Scholar
• Lin, L.-C., Huang, P.-H., & Weng, L.-J. (2017). Selecting path models in SEM: A comparison of model selection criteria. Structural Equation Modeling, 24, 855–869. doi:10.1080/10705511.2017.1363652
   Web of Science ®Google Scholar
• Liu, Y., & West, S. G. (2018). Longitudinal measurement non-Invariance with ordered-categorical indicators: How are the parameters in second-order latent linear growth models affected? Structural Equation Modeling, 25, 762–777. doi:10.1080/10705511.2017.1419353
   Web of Science ®Google Scholar
• Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Lu, Z. H., Chow, S. M., & Loken, E. (2016). Bayesian factor analysis as a variable-selection problem: Alternative priors and consequences. Multivariate Behavioral Research, 51, 519–539. doi:10.1080/00273171.2016.1168279
   PubMed Web of Science ®Google Scholar
• Lu, Z.-H., Chow, S.-M., & Loken, E. (2017). A comparison of Bayesian and frequentist model selection methods for factor analysis models. Psychological Methods, 22, 361–381. doi:10.1037/met0000145
   PubMed Web of Science ®Google Scholar
• Lubke, G., & Neale, M. (2008). Distinguishing between latent classes and continuous factors with categorical outcomes: Class invariance of parameters of factor mixture models. Multivariate Behavioral Research, 43, 592–620. doi:10.1080/00273170802490673
   PubMed Web of Science ®Google Scholar
• Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10, 21–39. doi:10.1037/1082-989X.10.1.21
   PubMed Web of Science ®Google Scholar
• Luo, Y. (2019). LOO and WAIC as model selection methods for polytomous items. Psychological Test and Assessment Modeling, 61, 161–185.
   Google Scholar
• Luo, Y., & Al-Harbi, K. (2017). Performances of LOO and WAIC as IRT model selection methods. Psychological Test and Assessment Modeling, 59, 183–205.
   Google Scholar
• McDonald, R. P. (1989). An index of goodness-of-fit based on noncentrality. Journal of Classification, 6, 97–103. doi:10.1007/BF01908590
   Web of Science ®Google Scholar
• Meade, A. W., Johnson, E. C., & Braddy, P. W. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568–592. doi:10.1037/0021-9010.93.3.568
   PubMed Web of Science ®Google Scholar
• Meade, A. W., & Lautenschlager, G. J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11, 60–72. doi:10.1207/S15328007SEM1101_5
   Web of Science ®Google Scholar
• Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. doi:10.1007/BF02294825
   Web of Science ®Google Scholar
• Meredith, W., & Teresi, J. A. (2006). An essay on measurement and factorial invariance. Medical Care, 44, 69–77. doi:10.1097/01.mlr.0000245438.73837.89
   PubMed Web of Science ®Google Scholar
• Merkle, E., Furr, D., & Rabe-Hesketh, S. (2019). Bayesian comparison of latent variable models: Conditional versus marginal likelihoods. Psychometrika, 84, 802–829. doi:10.1007/s11336-019-09679-0
   Google Scholar
• Merkle, E. C., & Rosseel, Y. (2018). blavaan: Bayesian structural equation models via parameter expansion. Journal of Statistical Software, 85, 1–30. doi:10.18637/jss.v085.i04
   PubMed Web of Science ®Google Scholar
• Merkle, E. C., Rosseel, Y., Garnier-Villarreal, M., Jorgensen, T. D., Hoofs, H., & van de Schoot, R. (2018). Bayesian latent variable analysis. Retrieved from https://cran.r-project.org/web/packages/blavaan/blavaan.pdf
   Google Scholar
• Millsap, R. E. (2011). Statistical approaches to measurement invariance. New York, NY: Routledge.
   Google Scholar
• Millsap, R. E., & Cham, H. (2012). Investigating factorial invariance in longitudinal data. In B. Laursen, T. D. Little, & N. A. Card (Eds.), Handbook of developmental research methods (pp. 109–126). New York, NY: Guilford.
   Google Scholar
• Millsap, R. E., & Kwok, O.-M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93–115. doi:10.1037/1082-989X.9.1.93
   PubMed Web of Science ®Google Scholar
• Muthén, B. O., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi:10.1037/a0026802
   PubMed Web of Science ®Google Scholar
• Muthén, B. O., & Asparouhov, T. (2013). BSEM measurement invariance analysis. Mplus Web Notes 17. Retrieved from https://www.statmodel.com/examples/webnotes/webnote17.pdf
   Google Scholar
• Muthén, L. K., & Muthén, B. O. (1998-2010). Mplus user’s guide (Version 6th). Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London A, 231, 289–337. doi:10.1098/rsta.1933.0009
   Google Scholar
• Nye, C. D., Bradburn, J., Olenick, J., Bialko, C., & Drasgow, F. (2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22, 678–709. doi:10.1177/1094428118761122
   Web of Science ®Google Scholar
• Nye, C. D., & Drasgow, F. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96, 966–980. doi:10.1037/a0022955
   PubMed Web of Science ®Google Scholar
• Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569. doi:10.1080/10705510701575396
   Web of Science ®Google Scholar
• Olivera-Aguilar, M. (2013). Impact of violations of longitudinal measurement invariance in latent growth models and autoregressive quasi-simplex models. Arizona State University. Retrieved from https://repository.asu.edu/items/18699
   Google Scholar
• Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Proceedings of the 3rd International Workshop On Distributed Statistical Computing. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.13.3406
   Google Scholar
• Pornprasertmanit, S. (2018). A note on effect size for measurement invariance. Retrieved from https://cran.r-project.org/web/packages/semTools/vignettes/partialInvariance.pdf
   Google Scholar
• Preinerstorfer, D., & Formann, A. K. (2012). Parameter recovery and model selection in mixed Rasch models. British Journal of Mathematical and Statistical Psychology, 65, 251–262. doi:10.1111/j.2044-8317.2011.02020.x
   PubMed Web of Science ®Google Scholar
• Raftery, A. E. (1995). Bayesian model selection in social research. Sociological Methodology, 25, 111–164. doi:10.2307/271063
   Web of Science ®Google Scholar
• Raykov, T., Marcoulides, G. A., & Millsap, R. E. (2013). Factorial invariance in multiple populations: A multiple testing procedure. Educational and Psychological Measurement, 73, 713–727. doi:10.1177/0013164412451978
   Web of Science ®Google Scholar
• Revuelta, J., & Ximenez, C. (2017). Bayesian dimensionality assessment for the multidimensional nominal response model. Frontiers in Psychology, 8, 1–16. doi:10.3389/fpsyg.2017.00001
   PubMed Web of Science ®Google Scholar
• Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Journal of Statistical Software, 48, 1–36. doi:10.18637/jss.v048.i02
   Web of Science ®Google Scholar
• Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applies statistician. The Annals of Statistics, 12, 1151–1172. doi:10.1214/aos/1176346785
   Web of Science ®Google Scholar
• Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21, 167–180. doi:10.1080/10705511.2014.882658
   Web of Science ®Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464. doi:10.1214/aos/1176344136
   Web of Science ®Google Scholar
• Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333–343. doi:10.1007/BF02294360
   Web of Science ®Google Scholar
• Shi, D., Song, H., Liao, X., Terry, R., & Snyder, L. A. (2017). Bayesian SEM for specification search problems in testing factorial invariance. Multivariate Behavioral Research, 52, 430–444. doi:10.1080/00273171.2017.1306432
   PubMed Web of Science ®Google Scholar
• Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society. Series B (statistical Methodology), 64, 583–639. doi:10.1111/1467-9868.00353
   Web of Science ®Google Scholar
• Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292. doi:10.1037/0021-9010.91.6.1292
   PubMed Web of Science ®Google Scholar
• Tanabe, A. S. (2011). Kakusan4 and Aminosan: Two programs for comparing nonpartitioned, proportional and separate models for combined molecular phylogenetic analyses of multilocus sequence data. Molecular Ecology Resources, 11, 914–921. doi:10.1111/j.1755-0998.2011.03021.x
   PubMed Web of Science ®Google Scholar
• Teo, T. (2015). Comparing pre-service and in-service teachers’ acceptance of technology: Assessment of measurement invariance and latent mean differences. Computers & Education, 83, 22–31. doi:10.1016/j.compedu.2014.11.015
   Web of Science ®Google Scholar
• Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432. doi:10.1007/s11222-016-9696-4
   Web of Science ®Google Scholar
• Vrieze, S. I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17, 228–243. doi:10.1037/a0027127
   PubMed Web of Science ®Google Scholar
• Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594.
   Web of Science ®Google Scholar
• Yoon, M., & Millsap, R. E. (2007). Detecting violations of factorial invariance using data-based specification searches: A Monte Carlo study. Structural Equation Modeling, 14, 435–463. doi:10.1080/10705510701301677
   Web of Science ®Google Scholar