Caught off Base: A Note on the Interpretation of Incremental Fit Indices

Figures & data

Table 1. Study design: Two estimated models cross-fitted across three data-generating models.

		Data characteristics							Estimated model
Data-generating model	df	n	I	B	Ib	rb	$\left|\mathbf{R}\right|$	$\overline{r}$	Correctly specified	Misspecified
Scenario 1
M1: one-factor	54	200	12	1	12	.40	.02	.40	One-factor	Multi-factor
M2: multi-factor equal $\left|\mathbf{R}\right|$	54	200	12	3	4	.53	.02	.14	Multi-factor	One-factor
M3: multi-factor equal rb	54	200	12	3	4	.40	.11	.11	Multi-factor	One-factor
Scenario 2
M1: one-factor	54	200	12	1	12	.20	.27	.20	One-factor	Multi-factor
M2: multi-factor equal $\left|\mathbf{R}\right|$	54	200	12	3	4	.30	.27	.08	Multi-factor	One-factor
M3: multi-factor equal rb	54	200	12	3	4	.20	.55	.05	Multi-factor	One-factor

Note. df degrees of freedom of the data-generating model [i.e., $I(I+1)/2$ sufficient statistics − 24 estimated parameters]; sample size n; I number of manifest indicator variables; B number of independent cluster blocks; I_b number of indicators per block; r_b within-block correlation; $\left|\mathbf{R}\right|$ determinant of the model-implied population correlation matrix as an expression of the degree of multivariate dependence; $\overline{r}$ average model-implied correlation. Non-zero factor loadings in data-generating models are constrained to $\sqrt{r_b};$ estimated models have no such equality constraints. Multi-factor models are orthogonal with an independent cluster structure (cf. blocks).

Figure 1. ${\chi }^2$ distribution under correctly- and misspecified models. The dotted line corresponds to the 5% critical value ${\chi }_{df=54}^2=72.15$ With ${\chi }_m^2$ chisquare of the model of interest; $\left|\mathbf{R}\right|$ determinant of the model-implied population correlation matrix as an expression of the degree of multivariate dependence; r_b within-block correlation. In both scenarios, sample size n = 200. The misspecified model is a multi-factor model for the one-factor model, and vice versa (see also Table 1).

Figure 2. CFI distribution under correctly- and misspecified models. The dotted line corresponds to the commonly adopted .95 CFI rule of thumb. With $\left|\mathbf{R}\right|$ determinant of the model-implied population correlation matrix as an expression of the degree of multivariate dependence; r_b within-block correlation. In both scenarios, sample size n = 200. The misspecified model is a multi-factor model for the one-factor model, and vice versa (see also Table 1).

Table B.1. Model comparison results for the set of competing models.

	${\mathrm{M}}_0$	${\mathrm{M}}_1$	${\mathrm{M}}_{2a}$	${\mathrm{M}}_{2b}$	${\mathrm{M}}_3$	${\mathrm{M}}_S$
${\chi }^2$	919	312	154	85	35	0
df	36	27	27	24	18	0
p	<.001	<.001	<.001	<.001	0.010	1.000
λ	883	285	127	61	17	0
${\text{CFI}}_{(m,0)}$	0.00	0.68	0.86	0.93	0.98	1.00
$\left|\mathbf{R}\right|=0.047$, n = 301, I = 9.

Note. λ = non-centrality for the estimated model (i.e., $\lambda ={\chi }^2-\text{df}$); CFI = CFI value for estimated model (i.e., ${\text{CFI}}_{(m,0)}=1-\frac{{\lambda }_m}{{\lambda }_0}$); $\left|\mathbf{R}\right|$ = determinant of the observed correlation matrix (i.e., the degree of multivariate dependence); n = sample size; I = number of items. Here one would typically further clarify the model specifications and highlight the differences among the models. Yet, to keep the appendix compact see the text above.