New Effect Size Measures for Structural Equation Modeling

Abstract

Effect size is crucial for quantifying differences and a key concept behind Type I errors and power, but measures of effect size are seldom studied in structural equation modeling (SEM). While fit indices such as the root mean square error of approximation may address the severity of model misspecification, they are not a direct generalization of commonly used effect size measures such as Cohen’s d. Moreover, with violations of normality and when a test statistic does not follow a noncentral chi-square distribution, measures of misfit that are defined through the assumed distribution of the test statistic are no longer valid.

In this study, two new classes of effect size measures for SEM are developed by generalizing Cohen’s d. The first class consists of definitions that are theoretically equivalent to $F_{ML}({\mathrm{\Sigma }}_0,\mathrm{\Sigma }(\theta ))$, the population counterpart of the normal-distribution-based discrepancy function. The second class of effect size measures bears a stricter resemblance to Cohen’s d in its original form. Several versions of these generalizations are investigated to identify the one that is least affected by sample size and population distribution but most sensitive to model misspecification. Their performances under violated distributional assumptions, severity of model misspecification, and various sample sizes are examined using both normal maximum likelihood estimation and robust M-estimation. Monte Carlo results indicate that one measure in the first class of effect size measures is little affected by sample size and distribution while preserving sensitivity to model misspecification and thus is recommended for researchers to report in publications.

Keywords:

• Cohen’s d
• effect size
• model fit
• model misspecification

Notes

¹ Consistency at large holds when the effect size measure and its estimator approach the same value, although both their limit values might be infinity.

² Cases were considered as nonconverged when the maximum iteration limit of 1000 was exceeded in the Fisher scoring algorithm to estimate the model parameters. This occurred when the difference between the model parameter estimates in the previous step and the current step exceeded $1\times {10}^{-10}$ for all 1000 iterations.

Additional information

Funding

This work was supported by the National Science Foundation: [Grant Number SES-1461355].