Dynamic Fit Index Cutoffs for Hierarchical and Second-Order Factor Models

Abstract

A recent review found that 11% of published factor models are hierarchical models with second-order factors. However, dedicated recommendations for evaluating hierarchical model fit have yet to emerge. Traditional benchmarks like RMSEA <0.06 or CFI >0.95 are often consulted, but they were never intended to generalize to hierarchical models. Through simulation, we show that traditional benchmarks perform poorly at identifying misspecification in hierarchical models. This corroborates previous studies showing that traditional benchmarks do not maintain optimal sensitivity to misspecification as model characteristics deviate from those used to derive the benchmarks. Instead, we propose a hierarchical extension to the dynamic fit index (DFI) framework, which automates custom simulations to derive cutoffs with optimal sensitivity for specific model characteristics. In simulations to evaluate performance, results showed that the hierarchical DFI extension routinely exceeded 95% classification accuracy and 90% sensitivity to misspecification whereas traditional benchmarks applied to hierarchical models rarely exceeded 50% classification accuracy and 20% sensitivity.

Keywords:

• CFI
• hierarchical factor model
• model fit
• RMSEA
• scale validation
• second-order model

Notes

1 For didactic resources on software implementation of the DFI method, also see the DFI frequently asked questions document (https://psyarxiv.com/2zk7g/) or the vignettes located at https://github.com/melissagwolf/dynamic.

2 Beta versions of functions described here can be downloaded from the dynamic GitHub package, https://github.com/melissagwolf/dynamic prior to being pushed to the CRAN version of the dynamic, which will occur after this article is published.

3 Data files with simulation results for each condition are available from the first authors Open Science Framework page for this project, https://osf.io/m4c6z.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through R305D220003 to Arizona State University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.