Evaluation of Six Effect Size Measures of Measurement Non-Invariance for Continuous Outcomes

Abstract

Measurement invariance is assessed in the factor analytic framework by testing differences in model fit of a sequential series of models; however, the statistical significance of these differences is influenced by many factors, including sample size. Effect sizes are independent of sample size and can be used to determine the magnitude and practical importance of an effect. We developed four new effect size measures of measurement non-invariance for continuous outcomes. To test the properties of these effect sizes and of two existing effect sizes of non-invariance, we conducted a simulation study. We varied group sample sizes, location of the latent distributions, magnitude of non-invariance and type of non-invariance (e.g., metric invariance). Three of the effect sizes were unbiased in all conditions and all six were consistent. Recommendations for their use and future directions are discussed.

Keywords:

• Measurement invariance
• effect size
• DIF
• factor analysis

Acknowledgments

The authors would like to thank Jenn-Yun Tein and Samantha Anderson for their input on this project.

Notes

¹ The different formulations of the pooled standard deviation lead to different effect size values; however, the general conclusions (e.g., consistency) of our simulation study did not change.

² We ran all analyses using a pooled standard deviation (corrected in the cases of d_MACS and d_{MACS_Signed}) in the denominator for all six studied effect sizes and the conclusions did not change. Additionally, the pooled and not pooled versions of the effect sizes were correlated .99 for all effect sizes. Thus, we chose the denominator that was more interpretable.

³ The Monte Carlo standard error (a measure of between-simulation variability) was calculated for each effect size within each design cell. The largest value was 0.004. Thus, the number of replications was deemed acceptable for making conclusions based on the second decimal place.

⁴ Using different identification constraints (e.g., standardizing the factor for one group or using a reference variable) does not affect the value of the effect size.