Measurement Invariance Testing with Many Groups: A Comparison of Five Approaches

REFERENCES

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. doi:10.1109/TAC.1974.1100705
   Web of Science ®Google Scholar
• Asparouhov, T., & Muthén, B. O. (2010). Bayesian analysis of latent variable models using Mplus. Retrieved from http://statmodel.com/download/BayesAdvantages18.pdf
   Google Scholar
• Asparouhov, T., & Muthén, B. O. (2012). Computing the strictly positive Satorra–Bentler chi-square test in Mplus (Mplus Web Notes No. 12). Retrieved from http://statmodel.com/examples/webnotes/SB5.pdf
   Google Scholar
• Asparouhov, T., & Muthén, B. O. (2014). Multiple-group factor analysis alignment. Structural Equation Modeling, 21, 495–508. doi:10.1080/10705511.2014.919210
   Web of Science ®Google Scholar
• Beierlein, C., Davidov, E., Schmidt, P., Schwartz, S. H., & Rammstedt, B. (2012). Testing the discriminant validity of Schwartz’ Portrait Value Questionnaire items: A replication and extension of Knoppen and Saris (2009). Survey Research Methods, 6, 25–36. doi:10.18148/srm/2012.v6i1.5092#sthash.P8YCGK8T.dpuf
   Web of Science ®Google Scholar
• Bentler, P. M., & Bonnet, D. G. (1980). Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–600. doi:10.1037/0033-2909.88.3.588
   Web of Science ®Google Scholar
• Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 34, 155–175. doi:10.1080/10705510701301834
   Google Scholar
• Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255. doi:10.1207/S15328007SEM0902_5
   Web of Science ®Google Scholar
• Christensen, R., Johnson, W. O., Branscum, A. J., & Hanson, T. E. (2010). Bayesian ideas and data analysis: An introduction for scientists and statisticians. Boca Raton, FL: CRC.
   Google Scholar
• Davidov, E., Dülmer, H., Schlüter, E., Schmidt, P., & Meuleman, B. (2012). Using a multilevel structural equation modeling approach to explain cross-cultural measurement noninvariance. Journal of Cross-Cultural Psychology, 43, 558–575. doi:10.1177/0022022112438397
   Web of Science ®Google Scholar
• Davidov, E., Meuleman, B., Cieciuch, J., Schmidt, P., & Billiet, J. (2014). Measurement equivalence in cross-national research. Annual Review of Sociology, 40, 55–75. doi:10.1146/annurev-soc-071913-043137
   Web of Science ®Google Scholar
• Desa, D. (2014). Evaluating measurement invariance of TALIS 2013 complex scales: A comparison between continuous and categorical multiple-group confirmatory factor analyses (EDU/WKP[2014]2). Paris, France: OECD Publishing.
   Google Scholar
• Dunson, D. B., Palomo, J., & Bollen, K. A. (2005). Bayesian structural equation modeling (Tech. Report No. 2005-5). Research Triangle Park, NC: Statistical and Applied Mathematical Sciences Institute.
   Google Scholar
• Elsworth, G. R., Beauchamp, A., & Osborne, R. H. (2016). Measuring health literacy in community agencies: A Bayesian study of the factor structure and measurement invariance of the Health Literacy Questionnaire (HLQ). BMC Health Services Research, 16, 508. doi:10.1186/s12913-016-1754-2
   PubMed Web of Science ®Google Scholar
• Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparing with Mantel–Haenszel, SIBTEST, and IRT likelihood ratio. Applied Psychological Measurement, 29, 278–295. doi:10.1177/0146621605275728
   Web of Science ®Google Scholar
• Gelman, A., & Pardoe, I. (2006). Bayesian measures of explained variance and pooling in multilevel (hierarchical) models. Technometrics, 48, 241–251. doi:10.1198/004017005000000517
   Web of Science ®Google Scholar
• Hancock, G. R., Lawrence, F. R., & Nevitt, J. (2000). Type I error and power of latent mean methods and MANOVA in factorially invariant and noninvariant latent variable systems. Structural Equation Modeling, 7, 534–556. doi:10.1207/S15328007SEM0704_2
   Web of Science ®Google Scholar
• He, J., & Kubacka, K. (2015). Data comparability in the Teaching and Learning International Survey (TALIS) 2008 and 2013 (OECD Education Working Papers, No. 124). Paris, France: OECD. doi:10.1787/5jrp6fwtmhf2-en
   Google Scholar
• Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118
   Web of Science ®Google Scholar
• Jak, S., Oort, F. J., & Dolan, C. V. (2013). A test for cluster bias: Detecting violations of measurement invariance across clusters in multilevel data. Structural Equation Modeling, 20, 265–282. doi:10.1080/10705511.2013.769392
   Web of Science ®Google Scholar
• Jak, S., Oort, F. J., & Dolan, C. V. (2014). Measurement bias in multilevel data. Structural Equation Modeling, 21, 31–39. doi:10.1080/10705511.2014.856694
   Web of Science ®Google Scholar
• Jennrich, R. (2006). Rotation to simple loadings using component loss functions: The oblique case. Psychometrika, 71, 173–191. doi:10.1007/s11336-003-1136-B
   Web of Science ®Google Scholar
• Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70, 631–639. doi:10.2307/2285946
   Web of Science ®Google Scholar
• Kaplan, D. (2014). Bayesian statistics for the social sciences. New York, NY: Guilford.
   Google Scholar
• Kaplan, D., & Depaoli, S. (2012). Bayesian structural equation modeling. In R. Hoyle (Ed.), Handbook of structural equation modeling (pp. 650–673). New York, NY: Guilford.
   Google Scholar
• Kim, E. S., Joo, S.-H., Lee, P., Wang, Y., & Stark, S. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling, 23, 870–887. doi:10.1080/10705511.2016.1196108
   Google Scholar
• Kim, E. S., & Willson, V. L. (2014). Measurement invariance across groups in latent growth modeling. Structural Equation Modeling, 21, 408–424. doi:10.1080/10705511.2014.915374
   Web of Science ®Google Scholar
• Kim, E. S., Yoon, M., Wen, Y., Luo, W., & Kwok, O. (2015). Within-level group factorial invariance with multilevel data: Multilevel factor mixture and multilevel MIMIC models. Structural Equation Modeling, 22, 603–616. doi:10.1080/10705511.2014.938217
   Web of Science ®Google Scholar
• Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.
   Google Scholar
• Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767–778. doi:10.1093/biomet/88.3.767
   Web of Science ®Google Scholar
• Lubke, G. H., & Muthén, B. O. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10, 21–39. doi:10.1037/1082-989X.10.1.21
   PubMed Web of Science ®Google Scholar
• Lukočienė, O., Varriale, R., & Vermunt, J. K. (2010). The simultaneous decision(s) about the number of lower- and higher-level classes in multilevel latent class analysis. Sociological Methodology, 40, 247–283. doi:10.1111/j.1467-9531.2010.01231.x
   Web of Science ®Google Scholar
• Lukočienė, O., & Vermunt, J. K. (2010). Determining the number of components in mixture models for hierarchical data. In A. Fink, L. Berthold, W. Seidel, & A. Ultsch (Ed.), Advances in data analysis, data handling and business intelligence, studies in classification, data analysis, and knowledge organizations (pp. 241–249). Berlin, Germany: Spirnger. doi:10.1007/978-3-642-01044-622
   Google Scholar
• Maas, C. M., & Hox, J. J. (2005). Sufficient sample size for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 86–92. doi:10.1027/1614-2241.1.3.86
   Google Scholar
• McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York, NY: Wiley.
   Google Scholar
• Meade, A., Johnson, E., & Braddy, P. (2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568–592. doi:10.1037/0021-9010.93.3.568
   PubMed Web of Science ®Google Scholar
• Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543. doi:10.1007/BF02294825
   Web of Science ®Google Scholar
• Muthén, B. O., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313–335. doi:10.1037/a0026802
   PubMed Web of Science ®Google Scholar
• Muthén, B. O., & Asparouhov, T. (2013a). BSEM measurement invariance analysis. Retrieved from https://www.statmodel.com/examples/webnotes/webnote17.pdf
   Google Scholar
• Muthén, B. O., & Asparouhov, T. (2013b). New methods for the study of measurement invariance with many groups. Retrieved from http://ww.statmodel2.com/download/PolAn.pdf
   Google Scholar
• Muthén, B. O., & Asparouhov, T. (2014). IRT studies of many groups: The alignment method. Frontiers in Psychology, 5, Article 978. doi: 10.3389/fpsyg.2014.00978
   Google Scholar
• Muthén, B. O., & Muthén, L. K. (2012). Mplus 7.1 [Computer software]. Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Nagengast, B., & Marsh, H. W. (2013). Motivation and engagement in science around the globe: Testing measurement invariance with multigroup SEMs across 57 countries using PISA 2006. In L. Rutkowski, M. Von Davier, & D. Rutkowski (Eds.), A handbook of international large-scale assessment (pp. 317–344). Boca Raton, FL: CRC.
   Google Scholar
• Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569. doi:10.1080/10705510701575396
   Web of Science ®Google Scholar
• Redner, R. A., & Walker, H. F. (1984). Mixture densities, maximum likelihood, and the EM algorithm. SIAM Review, 26, 195–239.
   Web of Science ®Google Scholar
• Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74, 31–57. doi:10.1177/0013164413498257
   Web of Science ®Google Scholar
• Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. Von Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for development research (pp. 399–419). Newbury Park, CA: Sage.
   Google Scholar
• Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled chi-square test statistic. Psychometrika, 75, 243–248. doi:10.1007/s11336-009-9135-y
   PubMed Web of Science ®Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. doi:10.2307/2958889
   Web of Science ®Google Scholar
• Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333–343. doi:10.1007/BF02294360
   Web of Science ®Google Scholar
• Sinharay, S., Johnson, M. S., & Stern, H. S. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30, 298–321. doi:10.1177/0146621605285517
   Web of Science ®Google Scholar
• Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London, UK: Sage.
   Google Scholar
• Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society, 64, 583–639. doi:10.1111/1467-9868.00353
   Google Scholar
• Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91, 1292–1306. doi:10.1037/0021-9010.91.6.1292
   PubMed Web of Science ®Google Scholar
• Stephens, M. (2000). Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62, 795–809. doi:10.1111/1467-9868.00265
   Web of Science ®Google Scholar
• Tay, L., Dienner, E., Drasgow, F., & Vermunt, J. K. (2011). Multilevel mixed-measurement IRT analysis: An explication and application to self-reported emotions across the world. Organizational Research Methods, 14, 177–207. doi:10.1177/1094428110372674
   Web of Science ®Google Scholar
• Tein, J. Y., Coxe, X., & Cham, H. (2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling, 20, 640–657. doi:10.1080/10705511.2013.824781
   PubMed Web of Science ®Google Scholar
• van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthén, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4, 1–15. doi:10.3389/fpsyg.2013.00770
   PubMed Web of Science ®Google Scholar
• Woods, C. M. (2009). Evaluation of MIMIC model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44, 1–27. doi:10.1080/00273170802620121
   PubMed Web of Science ®Google Scholar
• Zercher, F., Schmidt, P., Cieciuch, J., & Davidov, E. (2015). The comparability of the universalism value over time and across countries in the European Social Survey: Exact vs. approximate measurement invariance. Frontiers in Psychology, 6, 733. doi:10.1037/t08228-000
   Web of Science ®Google Scholar