Dynamic Fit Index Cutoffs for Hierarchical and Second-Order Factor Models

References

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, and NCME). (2014). Standards for educational and psychological testing. American Psychological Association.
   Google Scholar
• Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. The American Psychologist, 73, 3–25. https://doi.org/10.1037/amp0000191
   PubMed Web of Science ®Google Scholar
• Arend, M. G., & Schäfer, T. (2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24, 1–19. https://doi.org/10.1037/met0000195
   PubMed Web of Science ®Google Scholar
• Barrett, P. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42, 815–824. https://doi.org/10.1016/j.paid.2006.09.018
   Web of Science ®Google Scholar
• Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. https://doi.org/10.1037/0033-2909.88.3.588
   Web of Science ®Google Scholar
• Bollen, K. A. (1989). Structural equations with latent variables. Wiley.
   Google Scholar
• Bollen, K. A., & Stine, R. A. (1992). Bootstrapping goodness-of-fit measures in structural equation models. Sociological Methods & Research, 21, 205–229. https://doi.org/10.1177/0049124192021002004
   Web of Science ®Google Scholar
• Brown, T. A. (2015). Confirmatory factor analysis for applied research. Guilford.
   Google Scholar
• Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230–258. https://doi.org/10.1177/0049124192021002005
   Web of Science ®Google Scholar
• Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). SAGE.
   Google Scholar
• Browne, M. W., MacCallum, R. C., Kim, C. T., Andersen, B. L., & Glaser, R. (2002). When fit indices and residuals are incompatible. Psychological Methods, 7, 403–421. https://doi.org/10.1037//1082-989x.7.4.403
   PubMed Web of Science ®Google Scholar
• Brunner, M., Nagy, G., & Wilhelm, O. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796–846. https://doi.org/10.1111/j.1467-6494.2011.00749.x
   PubMed Web of Science ®Google Scholar
• Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytical studies. Cambridge University Press.
   Google Scholar
• Caspi, A., Houts, R. M., Belsky, D. W., Goldman-Mellor, S. J., Harrington, H., Israel, S., Meier, M. H., Ramrakha, S., Shalev, I., Poulton, R., & Moffitt, T. E. (2014). The p factor: One general psychopathology factor in the structure of psychiatric disorders? Clinical Psychological, 2, 119–137. https://doi.org/10.1177/2167702613497473
   Google Scholar
• Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36, 462–494. https://doi.org/10.1177/0049124108314720
   PubMed Web of Science ®Google Scholar
• D’Urso, D. E., Maassen, E., van Assen, M. A. L. M., Nuijten, M. B., De Roover, K., Wicherts, J. M. (2022). The dire disregard of measurement invariance testing in psychological science. PsyArXiv. Retrieved July 29, 2022, from https://psyar xiv. com/n3f5u/.
   Google Scholar
• DeYoung, C. G. (2006). Higher-order factors of the Big Five in a multi-informant sample. Journal of Personality and Social Psychology, 91, 1138–1151. https://doi.org/10.1037/0022-3514.91.6.1138
   PubMed Web of Science ®Google Scholar
• Digman, J. M. (1997). Higher-order factors of the Big Five. Journal of Personality and Social Psychology, 73, 1246–1256. https://doi.org/10.1037//0022-3514.73.6.1246
   PubMed Web of Science ®Google Scholar
• Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indexes to misspecified structural or measurement model components: Rationale of two-index strategy revisited. Structural Equation Modeling: A Multidisciplinary Journal, 12, 343–367. https://doi.org/10.1207/s15328007sem1203_1
   Web of Science ®Google Scholar
• Fan, X., & Sivo, S. A. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42, 509–529. https://doi.org/10.1080/00273170701382864
   Web of Science ®Google Scholar
• Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8, 370–378. https://doi.org/10.1177/1948550617693063
   Web of Science ®Google Scholar
• Green, P., & MacLeod, C. J. (2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7, 493–498. https://doi.org/10.1111/2041-210X.12504
   Web of Science ®Google Scholar
• Greiff, S., & Heene, M. (2017). Why psychological assessment needs to start worrying about model fit. European Journal of Psychological Assessment, 33, 313–317. https://doi.org/10.1027/1015-5759/a000450
   Web of Science ®Google Scholar
• Groskurth, K., Bhaktha, N., Lechner, C. (2022). Making model judgments ROC(K) solid: Tailored cutoffs for fit indices through simulation and ROC analysis in structural equation modeling. PsyArXiv. Retrieved May 16, 2023, from https://psyarxiv.com/62j89/.
   Google Scholar
• Hancock, G. R., & Mueller, R. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educational and Psychological Measurement, 71, 306–324. https://doi.org/10.1177/0013164410384856
   Web of Science ®Google Scholar
• Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Buhner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319–336. https://doi.org/10.1037/a0024917
   PubMed Web of Science ®Google Scholar
• Hu, L. T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453. https://doi.org/10.1037/1082-989X.3.4.424
   Web of Science ®Google Scholar
• Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. https://doi.org/10.1080/10705519909540118
   Web of Science ®Google Scholar
• Hu, L. T., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112, 351–362. https://doi.org/10.1037/0033-2909.112.2.351
   PubMed Web of Science ®Google Scholar
• Jackson, D. L., Gillaspy, J. A., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14, 6–23. https://doi.org/10.1037/a0014694
   PubMed Web of Science ®Google Scholar
• Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., van Hell, J., Kennedy, E., Sullivan, G. F., Taylor, C. L., Ward, T., & Beaty, R. E. (2022). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior Research Methods, advance online publication. https://doi.org/10.3758/s13428-022-01986-2
   Web of Science ®Google Scholar
• Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42, 448–457. https://doi.org/10.1080/02796015.2013.12087465
   Web of Science ®Google Scholar
• Kim, H., & Millsap, R. (2014). Using the Bollen-Stine bootstrapping method for evaluating approximate fit indices. Multivariate Behavioral Research, 49, 581–596. https://doi.org/10.1080/00273171.2014.947352
   PubMed Web of Science ®Google Scholar
• Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). Guilford.
   Google Scholar
• Krueger, R. F., Markon, K. E., Patrick, C. J., Benning, S. D., & Kramer, M. D. (2007). Linking antisocial behavior, substance use, and personality: An integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116, 645–666. https://doi.org/10.1037/0021-843X.116.4.645
   PubMed Web of Science ®Google Scholar
• Markon, K. E., Krueger, R. F., & Watson, D. (2005). Delineating the structure of normal and abnormal personality: An integrative hierarchical approach. Journal of Personality and Social Psychology, 88, 139–157. https://doi.org/10.1037/0022-3514.88.1.139
   PubMed Web of Science ®Google Scholar
• Marsh, H. W., & Craven, R. G. (2006). Reciprocal effects of self-concept and performance from a multidimensional perspective: Beyond seductive pleasure and unidimensional perspectives. Perspectives on Psychological Science, 1, 133–163. https://doi.org/10.1111/j.1745-6916.2006.00010.x
   Web of Science ®Google Scholar
• Marsh, H. W., Hau, K. T., & Grayson, D. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares & J. McArdle (Eds.), Psychometrics (pp. 275–340). Erlbaum Associates.
   Google Scholar
• Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s(1999) Findings. Structural Equation Modeling, 11, 320–341. https://doi.org/10.1207/s15328007sem1103_2
   Web of Science ®Google Scholar
• Marsh, H. W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept: First-and higher order factor models and their invariance across groups. Psychological Bulletin, 97, 562–582. https://doi.org/10.1037/0033-2909.97.3.562
   Web of Science ®Google Scholar
• Martin, A. J. (2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. The British Journal of Educational Psychology, 77, 413–440. https://doi.org/10.1348/000709906X118036
   PubMed Web of Science ®Google Scholar
• Maydeu-Olivares, A., Shi, D., & Rosseel, Y. (2018). Assessing fit in structural equation models: A Monte-Carlo evaluation of RMSEA versus SRMR confidence intervals and tests of close fit. Structural Equation Modeling, 25, 389–402. https://doi.org/10.1080/10705511.2017.1389611
   Web of Science ®Google Scholar
• McDonald, R. P., & Ho, M. H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7, 64–82. https://doi.org/10.1037/1082-989x.7.1.64
   PubMed Web of Science ®Google Scholar
• McGrew, K. S. (2009). CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1–10. https://doi.org/10.1016/j.intell.2008.08.004
   Web of Science ®Google Scholar
• McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100, 43–52. https://doi.org/10.1080/00223891.2017.1281286
   PubMed Web of Science ®Google Scholar
• McNeish, D. (2023). Generalizability of dynamic fit index, equivalence testing, and Hu & Bentler cutoffs for evaluating fit in factor analysis. Multivariate Behavioral Research, 58, 195–219. advance online publication. https://doi.org/10.1080/00273171.2022.2163477
   PubMed Web of Science ®Google Scholar
• McNeish, D., & Wolf, M. G. (2021). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, advance online publication.
   Web of Science ®Google Scholar
• McNeish, D., & Wolf, M. G. (2022). Dynamic fit cutoffs for one-factor models. Behavior Research Methods, advance online publication.
   Web of Science ®Google Scholar
• McNeish, D. (in press). Dynamic fit index cutoffs for factor analysis with Likert, ordinal, or binary responses. https://psyarxiv.com/tp35s/
   Google Scholar
• Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. https://doi.org/10.1037/0022-006X.46.4.806
   Web of Science ®Google Scholar
• Miles, J., & Shevlin, M. (2007). A time and a place for incremental fit indices. Personality and Individual Differences, 42, 869–874. https://doi.org/10.1016/j.paid.2006.09.022
   Web of Science ®Google Scholar
• Millsap, R. E. (2007). Structural equation modeling made difficult. Personality and Individual Differences, 42, 875–881. https://doi.org/10.1016/j.paid.2006.09.021
   Web of Science ®Google Scholar
• Millsap, R. E. (2013). A simulation paradigm for evaluating model fit. In M. Edwards & R. MacCallum (Eds.), Current issues in the theory and application of latent variable models (pp. 165–182). Routledge.
   Google Scholar
• Mulaik, S. A. (2009). Foundations of factor analysis. McGraw-Hill.
   Google Scholar
• Murray, A. L., & Johnson, W. (2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41, 407–422. https://doi.org/10.1016/j.intell.2013.06.004
   Web of Science ®Google Scholar
• Nieman, T., Mai, R. (2023). FCO: Flexible cutoffs for model fit evaluation in covariance-based structural models. [Software]. Retrieved from CRAN. https://cran.r-project.org/web/packages/FCO/index.html
   Google Scholar
• Niemand, T., & Mai, R. (2018). Flexible cutoff values for fit indices in the evaluation of structural equation models. Journal of the Academy of Marketing Science, 46, 1148–1172. https://doi.org/10.1007/s11747-018-0602-9
   Web of Science ®Google Scholar
• Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548–570. https://doi.org/10.1177/1094428110368562
   Web of Science ®Google Scholar
• Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Design and implementation. Structural Equation Modeling, 8, 287–312. https://doi.org/10.1207/S15328007SEM0802_7
   Web of Science ®Google Scholar
• Pornprasertmanit, S., Wu, W., & Little, T. D. (2013). Using a Monte Carlo approach for nested model comparisons in structural equation modeling. In R. E. Millsap, L. A. van der Ark, D. M. Bolt, & C. M. Woods (Eds.), New developments in quantitative psychology (pp. 187–197). Springer.
   Google Scholar
• Pornprasertmanit, S., Miller, P., Schoemann, A., Jorgensen, T., Quick, C. (2022). SIMulated structural equation modeling (R package version 0.5–16). http://CRAN.R-project.org/package=simsem
   Google Scholar
• Reynolds, M. R., & Keith, T. Z. (2017). Multi-group and hierarchical confirmatory factor analysis of the Wechsler Intelligence Scale for Children—Fifth Edition: What does it measure? Intelligence, 62, 31–47. https://doi.org/10.1016/j.intell.2017.02.005
   Web of Science ®Google Scholar
• Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373. https://doi.org/10.1037/a0029315
   PubMed Web of Science ®Google Scholar
• Rijmen, F. (2010). Formal relations and an empirical comparison among the bi‐factor, the testlet, and a second‐order multidimensional IRT model. Journal of Educational Measurement, 47, 361–372. https://doi.org/10.1111/j.1745-3984.2010.00118.x
   Web of Science ®Google Scholar
• Robitzsch, A. (2020, October) Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5, 589965. https://doi.org/10.3389/feduc.2020.589965
   Google Scholar
• Ropovik, I. (2015). A cautionary note on testing latent variable models. Frontiers in Psychology, 6, 1715. https://doi.org/10.3389/fpsyg.2015.01715
   PubMed Web of Science ®Google Scholar
• Saris, W. E., Satorra, A., & Sörbom, D. (1987). The detection and correction of specification errors in structural equation models. Sociological Methodology, 17, 105–129. https://doi.org/10.2307/271030
   Google Scholar
• Saris, W. E., Satorra, A., & Van der Veld, W. M. (2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling, 16, 561–582. https://doi.org/10.1080/10705510903203433
   Web of Science ®Google Scholar
• Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72, 910–932. https://doi.org/10.1177/0013164412452564
   Web of Science ®Google Scholar
• Schönemann, P. H. (1981). Power as a function of communality in factor analysis. Bulletin of the Psychonomic Society, 17, 57–60. https://doi.org/10.3758/BF03333667
   Web of Science ®Google Scholar
• Schreiber, J. B. (2008). Core reporting practices in structural equation modeling. Research in Social & Administrative Pharmacy, 4, 83–97. https://doi.org/10.1016/j.sapharm.2007.04.003
   PubMed Web of Science ®Google Scholar
• Shi, D., Lee, T., & Maydeu-Olivares, A. (2019). Understanding the model size effect on SEM fit indices. Educational and Psychological Measurement, 79, 310–334. https://doi.org/10.1177/0013164418783530
   PubMed Web of Science ®Google Scholar
• Shi, D., Lee, T., & Terry, R. A. (2018). Revisiting the model size effect in structural equation modeling. Structural Equation Modeling, 25, 21–40. https://doi.org/10.1080/10705511.2017.1369088
   Web of Science ®Google Scholar
• Silvia, P. J. (2008). Another look at creativity and intelligence: Exploring higher-order models and probable confounds. Personality and Individual Differences, 44, 1012–1021. https://doi.org/10.1016/j.paid.2007.10.027
   Web of Science ®Google Scholar
• Sivo, S. A., Fan, X., Witta, E. L., & Willse, J. T. (2006). The search for “optimal” cutoff properties: Fit index criteria in structural equation modeling. The Journal of Experimental Education, 74, 267–288. https://doi.org/10.3200/JEXE.74.3.267-288
   Web of Science ®Google Scholar
• Tanaka, J. S. (1987). “How big is big enough?”: Sample size and goodness of fit in structural equation models with latent variables. Child Development, 58, 134–146. https://doi.org/10.2307/1130296
   Web of Science ®Google Scholar
• Thoemmes, F., Rosseel, Y., & Textor, J. (2018). Local fit evaluation of structural equation models using graphical criteria. Psychological Methods, 23, 27–41. https://doi.org/10.1037/met0000147
   PubMed Web of Science ®Google Scholar
• Tomarken, A. J., & Waller, N. G. (2003). Potential problems with “well fitting” models. Journal of Abnormal Psychology, 112, 578–598. https://doi.org/10.1037/0021-843X.112.4.578
   PubMed Web of Science ®Google Scholar
• West, S. G., Wu, W., McNeish, D., & Savord, A. (2023). Model fit in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (2nd ed., pp. 184–205) Guilford Press.
   Google Scholar
• Wolf, M. G., & McNeish, D. (2023). dynamic: An R package for deriving dynamic fit index cutoffs for factor analysis. Multivariate Behavioral Research, 58, 189–194. https://doi.org/10.1080/00273171.2022.2163476
   PubMed Web of Science ®Google Scholar
• Wolf, M. G., McNeish, D. (2021). Dynamic Model Fit (version 1.1.0). [Software]. Available from www.dynamicfit.app.
   Google Scholar
• Wolf, M. G., McNeish, D. (2022). dynamic: DFI cutoffs for latent variables models (version 1.1.0). [Software]. Retrieved from CRAN. https://cran.r-project.org/web/packages/dynamic.
   Google Scholar
• Wright, A. G., Lukowitsky, M. R., Pincus, A. L., & Conroy, D. E. (2010). The higher order factor structure and gender invariance of the Pathological Narcissism Inventory. Assessment, 17, 467–483. https://doi.org/10.1177/1073191110373227
   PubMed Web of Science ®Google Scholar
• Yung, Y. F., Thissen, D., & McLeod, L. D. (1999). On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika, 64, 113–128. https://doi.org/10.1007/BF02294531
   Web of Science ®Google Scholar
• Zhang, M. F., Dawson, J. F., & Kline, R. B. (2021). Evaluating the use of covariance‐based structural equation modelling with reflective measurement in organizational and management research: A review and recommendations for best practice. British Journal of Management, 32, 257–272. https://doi.org/10.1111/1467-8551.12415
   Web of Science ®Google Scholar
• Zinbarg, R. E., Barlow, D. H., & Brown, T. A. (1997). Hierarchical structure and general factor saturation of the Anxiety Sensitivity Index: Evidence and implications. Psychological Assessment, 9, 277–284. https://doi.org/10.1037/1040-3590.9.3.277
   Web of Science ®Google Scholar
• Zyphur, M. J., Bonner, C. V., & Tay, L. (2023). Structural equation modeling in organizational research: The state of our science and some proposals for its future. Annual Review of Organizational Psychology and Organizational Behavior, 10, 495–517. https://doi.org/10.1146/annurev-orgpsych-041621-031401
   Google Scholar