Examining Chi-Square Test Statistics Under Conditions of Large Model Size and Ordinal Data

Abstract

This study examined the effect of model size on the chi-square test statistics obtained from ordinal factor analysis models. The performance of six robust chi-square test statistics were compared across various conditions, including number of observed variables (p), number of factors, sample size, model (mis)specification, number of categories, and threshold distribution. Results showed that the unweighted least squares (ULS) robust chi-square statistics generally outperform the diagonally weighted least squares (DWLS) robust chi-square statistics. The ULSM estimator performed the best overall. However, when fitting ordinal factor analysis models with a large number of observed variables and small sample size, the ULSM-based chi-square tests may yield empirical variances that are noticeably larger than the theoretical values and inflated Type I error rates. On the other hand, when the number of observed variables is very large, the mean- and variance-corrected chi-square test statistics (e.g., based on ULSMV and WLSMV) could produce empirical variances conspicuously smaller than the theoretical values and Type I error rates lower than the nominal level, and demonstrate lower power rates to reject misspecified models. Recommendations for applied researchers and future empirical studies involving large models are provided.

Keywords:

• chi-square test statistics
• large models
• ordinal factor analysis

ACKNOWLEDGMENTS

We acknowledge the Research Computing Center at the University of South Carolina, the University of Alabama Research Computing Portal and the High-Performance Computing Center at Weifang University for providing the computing resources that contributed to the results of this paper.

Notes

¹ Sample size was set to N = 200 observations for all misspecified models to ensure that the power had not reached an asymptote so that differences in power could be observed (Forero et al., 2009).

² Relative bias (RB) was computed as: , where represents the average sample estimates across all replications, and indicates the theoretical values.

³ When fitting multiple factor models (e.g., f ≥5), almost all replications were nonconvergent if the factor loadings were low (i.e., λ = 0.40). Therefore, as investigating the effects of f, we only summarized and reported the results for the conditions where λ = 0.80.