The Performance of Maximum Likelihood and Weighted Least Square Mean and Variance Adjusted Estimators in Testing Differential Item Functioning With Nonnormal Trait Distributions

Abstract

The relative performance of the maximum likelihood (ML) and weighted least square mean and variance adjusted (WLSMV) estimators was investigated by studying differential item functioning (DIF) with ordinal data when the latent variable () was not normally distributed. As the ML estimator, ML with robust standard errors (labeled MLR in Mplus) was chosen and implemented with 2 link functions (logit vs. probit). The Type I error and power of χ² tests were evaluated under various simulation conditions including the shape of the distributions for the reference and focal groups. Type I error was better controlled with MLR estimators than WLSMV. The error from WLSMV was inflated when there was a large difference in the shape of the distribution between the 2 groups. In general, the power remained quite stable across different distribution conditions regardless of the estimators. WLSMV and MLR-probit showed comparable power, whereas MLR-logit performed the worst.

Keywords:

• differential item functioning
• limited information
• nonnormality
• ordinal response

Notes

¹ MLR provides ML estimates with standard errors and a chi-square test statistic (when applicable) that are robust to nonnormality and nonindependence of observations, whereas ML provides ML estimates with conventional standard errors and chi-square test statistic. The MLR standard errors are computed using a sandwich estimator (Muthén & Muthén, 2010).

² The true value for is –0.5. Therefore, a negative value of bias in Table 2 indicates that the estimate (e.g., –0.6) is smaller than –0.5, but the absolute value of the estimate is greater than 0.5. In this regard, the author used the term overestimated.

³ As shown in Table 3, Type I errors were overestimated in some conditions and underestimated in others. As power is affected by the Type I error rate, power is also over- or underestimated. Because the power should be obtained conditional on controlling the Type I error rate, the term power in this article should be interpreted as the rejection rates (when DIF was present), rather than the intrinsic meaning of power.

⁴ The Bayes estimator is not feasible for the current application in Mplus.