Fit Difference Between Nonnested Models Given Categorical Data: Measures and Estimation

ABSTRACT

In structural equation modeling (SEM) studies with categorical data, researchers often use the root mean square error of approximation (RMSEA), comparative fit index (CFI), or standardized root mean squared residual (SRMR) to compare rival models. Model selection based on $\mathrm{\Delta }\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\mathrm{A}$, $\mathrm{\Delta }\mathrm{C}\mathrm{F}\mathrm{I}$, or $\mathrm{\Delta }\mathrm{S}\mathrm{R}\mathrm{M}\mathrm{R}$ is meaningful because (a) finding a better model is more scientific and easier than establishing a “good” model, (b) it avoids the problems with cutoffs for fit indices, (c) one is less likely to overlook other equally substantively plausible models, and (d) information criteria (e.g., AIC, BIC) are not applicable to categorical data SEM. In this paper, we propose point estimators and confidence intervals (CIs) for $\mathrm{\Delta }\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\mathrm{A}$, $\mathrm{\Delta }\mathrm{C}\mathrm{F}\mathrm{I}$, and $\mathrm{\Delta }\mathrm{S}\mathrm{R}\mathrm{M}\mathrm{R}$ under categorical data. Our methods are applicable to nonnested models and do not need a true model. Simulation results show our point estimators and CIs are all trustworthy, whereas the bias is large when estimating $\mathrm{\Delta }\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\mathrm{A}$ ($\mathrm{\Delta }\mathrm{C}\mathrm{F}\mathrm{I}$, $\mathrm{\Delta }\mathrm{S}\mathrm{R}\mathrm{M}\mathrm{R}$) based on the common estimators in the current literature for RMSEA (CFI, SRMR).

KEYWORDS:

• Fit index
• confidence interval
• categorical data
• nonnested models

Notes

¹ A saturated structure for thresholds simply means the number of model parameters for the thresholds is the same as the number of thresholds in data, namely, a case where ${\theta }_1,{\theta }_2,\cdots ,{\theta }_K$ are used to explain ${\tau }_1,{\tau }_2,\cdots ,{\tau }_K$. However, this does not dictate ${\theta }_1={\tau }_1$, ${\theta }_2={\tau }_2$, …, ${\theta }_K={\tau }_K$, because the model parameters are still free to take on different values in model fitting.

² All the code and files for our simulation studies are accessible at https://bit.ly/38sIWKz.

³ Models D, E, and G have the same degrees of freedom (see Table 7), and therefore it is impossible for them to be nested models. Moreover, because they do not always have the same fit (see Table 7), they are not equivalent models either. Regarding the model pair FD, $d_D>d_F$ means it is only possible for Model D to be nested within Model F, but the fact that $F_D^{\ast }<F_F^{\ast }$ can happen rules out the possibility of nesting. The same argument applies to model pair FG. The last pair is FE. Because d_E > d_F, if we can find a correlation matrix to which Model E fits better than does Model F, then we can rule out that Model E is nested within Model F. To find such an example, consider the ${\mathbf{P}}^{\mathbf{\ast }}$ we created in Misfit Situation 1 (see the online supplement in Footnote 2 for the values of this ${\mathbf{P}}^{\mathbf{\ast }}$). Fitting Model E to ${\mathbf{P}}^{\mathbf{\ast }}$ yields ${\theta }_E^{\ast }$ and the population model-implied matrix ${\mathbf{P}}_E({\theta }_E^{\ast })$. Then, Model E fits perfectly to ${\mathbf{P}}_E({\theta }_E^{\ast })$ but Model F cannot. Because $d_E>d_F$ and Model E can fit better than Model F sometimes, Model E is not nested within Model F.