A Monte Carlo Confidence Interval Method for Testing Measurement Invariance

Abstract

We propose a new method, the Monte Carlo confidence interval (MCCI) method, for studying measurement invariance. This method allows researchers to examine the invariance of all items simultaneously by comparing the observed between-group differences in parameters with those obtained under the null hypothesis of invariance. We compare the performance of our method to two other methods: the multigroup confirmatory factor analysis (MG-CFA) and the forward method using confidence intervals (forward CI). The results show that our method performed better than the MG-CFA method and comparably with the forward CI method. The code to implement our method and an empirical example are provided in the Supplementary materials.

Keywords:

• Forward confidence interval method
• measurement invariance
• Monte Carlo confidence interval method
• Monte Carlo test
• multigroup confirmatory factor analysis
• parametric bootstrap

Disclosure statement

We have no conflicts of interests to disclose. The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

1 When this nested model approach is used to detect noninvariant items, variation exists in the sequences and types of constrained models examined. Alternative strategies can be found in Stark et al. (2006).

2 In our pilot analyses, we tried the settings of K = 1000, K = 2000, and K= 5000 and obtained essentially the same results. In addition, many previous studies using the MC test also set the number of simulated samples to 1000 (e.g., Jalal & Bentler, 2018; Yuan et al., 2021). Hence, 1000 simulated samples were used in this study.

3 In our simulations, the factor means and variances between the two groups were specified to be equal. As suggested by a reviewer, we also tried several conditions in which the factor means and variances are unequal (Group 1: the factor means and variances are set to 0 and 1, respectively; and Group 2: the factor means and variances are set to 0.2 and 1.3, respectively). The results are displayed in Part A of the online Supplementary materials. We found that assuming different factor means and variances did not change the results.

4 The rate was calculated by subtracting the power value of the forward CI method from that of the MCCI method and then dividing it by the power of the forward CI method.

Additional information

Funding

The present research was financially supported by the National Natural Science Foundation of China (grant number: 32071091).