Evaluating the Performance of the K-fold Cross-Validation Approach for Model Selection in Growth Mixture Modeling

References

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723.
   Web of Science ®Google Scholar
• Anguita, D., Ghelardoni, L., Ghio, A., Oneto, L., & Ridella, S. (2012). The ‘K’in K-fold cross validation. Paper presented at the proceedings, European symposium on artificial neural networks, computational intelligence and machine learning, Bruges, Belgium.
   Google Scholar
• Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.
   Google Scholar
• Asparouhov, T., & Muthén, B. (2014). Auxiliary variables in mixture modeling: Three-step approaches using M plus. Structural Equation Modeling, 21(3), 329–341.
   Web of Science ®Google Scholar
• Bauer, D. J. (2007). Observations on the use of growth mixture models in psychological research. Multivariate Behavioral Research, 42(4), 757–786.
   Web of Science ®Google Scholar
• Bengio, Y., & Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5,1089–1105.
   Web of Science ®Google Scholar
• Braga-Neto, U. M., & Dougherty, E. R. (2004). Is cross-validation valid for small-sample microarray classification? Bioinformatics, 20(3), 374–380.
   PubMed Web of Science ®Google Scholar
• Chen, Q., Kwok, O.-M., Luo, W., & Willson, V. L. (2010). The impact of ignoring a level of nesting structure in multilevel growth mixture models: A Monte Carlo study. Structural Equation Modeling, 17(4), 570–589.
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Colder, C. R., Mehta, P., Balanda, K., Campbell, R. T., Mayhew, K., Stanton, W. R., … Flay, B. R. (2001). Identifying trajectories of adolescent smoking: An application of latent growth mixture modeling. Health Psychology, 20(2), 127–135.
   PubMed Web of Science ®Google Scholar
• Depaoli, S. (2013). Mixture class recovery in GMM under varying degrees of class separation: Frequentist versus Bayesian estimation. Psychological Methods, 18(2), 186–219.
   PubMed Web of Science ®Google Scholar
• Diallo, T. M., Morin, A. J., & Lu, H. (2016). Performance of growth mixture models in the presence of time-varying covariates. Behavior Research Methods, 49(5), 1951–1962.
   Web of Science ®Google Scholar
• Elliott, D. S., Huizinga, D., & Menard, S. (1989). Multiple problem youth: Delinquency, substance use, and mental health problems. New York, NY: Springer Science & Business Media.
   Google Scholar
• Fushiki, T. (2011). Estimation of prediction error by using K-fold cross-validation. Statistics and Computing, 21(2), 137–146.
   Web of Science ®Google Scholar
• Grimm, K. J., Mazza, G. L., & Davoudzadeh, P. (2017). Model selection in finite mixture models: A k-fold cross-validation approach. Structural Equation Modeling, 24(2), 246–256.
   Web of Science ®Google Scholar
• Grimm, K. J., Ram, N., & Estabrook, R. (2016). Growth modeling: Structural equation and multilevel modeling approaches. New York, NY: Guilford Publications.
   Google Scholar
• Guerra-Peña, K., & Steinley, D. (2016). Extracting spurious latent classes in growth mixture modeling with nonnormal errors. Educational and Psychological Measurement, 76(6), 933–953.
   PubMed Web of Science ®Google Scholar
• Hallquist, M., & Wiley, J. (2016). MplusAutomation: Automating Mplus model estimation and interpretation (version 0.6-4) [Computer software]. Retrieved from https://www.R-project.org/
   Google Scholar
• Hamilton, J. (2009). An investigation of growth mixture models when data are collected with unequal selection probabilities: A Monte Carlo study (Unpublished doctoral dissertation). College park, MD: University of Maryland.
   Google Scholar
• Henson, J. M., Reise, S. P., & Kim, K. H. (2007). Detecting mixtures from structural model differences using latent variable mixture modeling: A comparison of relative model fit statistics. Structural Equation Modeling, 14(2), 202–226.
   Web of Science ®Google Scholar
• Hix-Small, H., Duncan, T. E., Duncan, S. C., & Okut, H. (2004). A multivariate associative finite growth mixture modeling approach examining adolescent alcohol and marijuana use. Journal of Psychopathology and Behavioral Assessment, 26(4), 255–270.
   Web of Science ®Google Scholar
• Hu, J., Leite, W. L., & Gao, M. (2017). An evaluation of the use of covariates to assist in class enumeration in linear growth mixture modeling. Behavior Research Methods, 49(3), 1179–1190.
   PubMed Web of Science ®Google Scholar
• Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76(2), 297–307.
   Web of Science ®Google Scholar
• Jung, T., & Wickrama, K. (2008). An introduction to latent class growth analysis and growth mixture modeling. Social and Personality Psychology Compass, 2(1), 302–317.
   Google Scholar
• Jung, Y., & Hu, J. (2015). AK-fold averaging cross-validation procedure. Journal of Nonparametric Statistics, 27(2), 167–179.
   PubMed Web of Science ®Google Scholar
• Kim, S.-Y. (2014). Determining the number of latent classes in single-and multiphase growth mixture models. Structural Equation Modeling, 21(2), 263–279.
   PubMed Web of Science ®Google Scholar
• Li, L., & Hser, Y.-I. (2011). On inclusion of covariates for class enumeration of growth mixture models. Multivariate Behavioral Research, 46(2), 266–302.
   PubMed Web of Science ®Google Scholar
• Li, M. (2015). Investigating methods of incorporating covariates in growth mixing modeling: A simulation study (Unpublished doctoral dissertation). College Park, MD: University of Maryland.
   Google Scholar
• Li, M., & Harring, J. R. (2017). Investigating approaches to estimating covariate effects in growth mixture modeling a simulation study. Educational and Psychological Measurement, 77(5), 766–791.
   PubMed Web of Science ®Google Scholar
• Li, M., Harring, J. R., & Macready, G. B. (2014). Investigating the feasibility of using Mplus in the estimation of growth mixture models. Journal of Modern Applied Statistical Methods, 13(1), 484–513.
   Google Scholar
• Liu, J. (2012). A systematic investigation of within-subject and between-subject covariance structures in growth mixture models (Unpublished doctoral dissertation). College Park, MD: University of Maryland.
   Google Scholar
• Liu, M. (2011). Using latent profile models and unstructured growth mixture models to assess the number of latent classes in growth mixture modeling (Unpublished doctoral dissertation). College Park, MD: University of Maryland.
   Google Scholar
• Liu, M., & Hancock, G. R. (2014). Unrestricted mixture models for class identification in growth mixture modeling. Educational and Psychological Measurement, 74(4), 557–584.
   Web of Science ®Google Scholar
• Liu, Y., Luo, F., & Liu, H. (2014). Factors of piecewise growth mixture model: distance and pattern. Acta Psychologica Sinica, 46(9), 1400–1412.
   Google Scholar
• Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88(3), 767–778.
   Web of Science ®Google Scholar
• Lubke, G., & Grimm, K. J. (2017). Introduction to novel approaches in mixture modeling. Structural Equation Modeling, 24(2), 157–158.
   Web of Science ®Google Scholar
• Lubke, G., & Luningham, J. (2017). Fitting latent variable mixture models. Behaviour Research and Therapy, 98, 91–102.
   Google Scholar
• Lubke, G., & Muthén, B. (2007). Performance of factor mixture models as a function of model size, covariate effects, and class-specific parameters. Structural Equation Modeling, 14(1), 26–47.
   Web of Science ®Google Scholar
• Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person-and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling, 16(2), 191–225.
   Web of Science ®Google Scholar
• Martin, D. P., & Von Oertzen, T. (2015). Growth mixture models outperform simpler clustering algorithms when detecting longitudinal heterogeneity, even with small sample sizes. Structural Equation Modeling, 22(2), 264–275.
   Web of Science ®Google Scholar
• McLachlan, G. (1999). Mahalanobis distance. Resonance, 4(6), 20–26.
   Google Scholar
• McLachlan, G., & Peel, D. (2000). Finite mixture models. New York, NY: John Wiley & Sons.
   Google Scholar
• Meijer, R. J., & Goeman, J. J. (2013). Efficient approximate k‐fold and leave‐one‐out cross‐validation for ridge regression. Biometrical Journal, 55(2), 141–155.
   PubMed Web of Science ®Google Scholar
• Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Newbury Park, CA: Sage Publications.
   Google Scholar
• Muthén, B., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55(2), 463–469.
   PubMed Web of Science ®Google Scholar
• Muthén, L., & Muthén, B. (1988-2015). Mplus Version 7.4 [statistical software]. Los Angeles, CA: Muthén & Muthén.
   Google Scholar
• Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14(4), 535–569.
   Web of Science ®Google Scholar
• Pastor, D., & Gagné, P. (2013). Mean and covariance structure mixture models. In G. R. Hancock & R. O. Mueller (Eds.), Quantitative methods in education and the behavioral sciences: Issues, research, and teaching. Structural equation modeling: A second course (pp. 343–393). Charlotte, NC: IAP Information Age Publishing.
   Google Scholar
• Peugh, J., & Fan, X. (2012). How well does growth mixture modeling identify heterogeneous growth trajectories? A simulation study examining GMM’s performance characteristics. Structural Equation Modeling, 19(2), 204–226.
   Web of Science ®Google Scholar
• Peugh, J., & Fan, X. (2015). Enumeration index performance in generalized growth mixture models: A Monte Carlo test of Muthén’s (2003) hypothesis. Structural Equation Modeling, 22(1), 115–131.
   Web of Science ®Google Scholar
• Preacher, K. J., & Merkle, E. C. (2012). The problem of model selection uncertainty in structural equation modeling. Psychological Methods, 17(1), 1–14.
   PubMed Web of Science ®Google Scholar
• R Core Team. (2017). R: A language and environment for statistical computing (Version 3.3.3). Vienna, Austria: R Foundation for Statistical Computing.
   Google Scholar
• Ram, N., & Grimm, K. J. (2009). Methods and measures: Growth mixture modeling: A method for identifying differences in longitudinal change among unobserved groups. International Journal of Behavioral Development, 33(6), 565–576.
   PubMed Web of Science ®Google Scholar
• Refaeilzadeh, P., Tang, L., & Liu, H. (2009). Cross-validation In Liu L., Özsu M.T. (Eds.), Encyclopedia of Database Systems. Boston, MA: Springer.
   Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.
   Web of Science ®Google Scholar
• Tofighi, D., & Enders, C. K. (2008). Identifying the correct number of classes in growth mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 317–341). Charlotte, NC: Information Age Publishing.
   Google Scholar
• Tolvanen, A. (2007). Latent growth mixture modeling: A simulation study (Unpublished doctoral dissertation). Finland: University of Jyväskylä.
   Google Scholar
• Usami, S. (2014). Performance of information criteria for model selection in a latent growth curve mixture model. Journal of the Japanese Society of Computational Statistics, 27(1), 17–48.
   Google Scholar
• Vermunt, J. K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469.
   Web of Science ®Google Scholar
• Wang, M., & Bodner, T. E. (2007). Growth mixture modeling: Identifying and predicting unobserved subpopulations with longitudinal data. Organizational Research Methods, 10(4), 635–656.
   Web of Science ®Google Scholar
• Wang, R. (2011). Performance of model selection statistics in growth mixture modeling of homogeneous data. Stony Brook, NY: The Graduate School, Stony Brook University.
   Google Scholar
• Wickrama, K. K., Lee, T. K., O’Neal, C. W., & Lorenz, F. O. (2016). Higher-order growth curves and mixture modeling with Mplus: A practical guide. New York, NY: Routledge.
   Google Scholar