Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons

Abstract

It is frequently stated in the literature that measurement invariance is a prerequisite for the comparison of group means or standard deviations of the latent variable in factor models. This article argues that measurement invariance is not necessary for meaningful and valid comparisons across groups. There is unavoidable ambiguity in how researchers can define comparisons if measurement invariance is violated. Moreover, there is no support for preferring the partial invariance approach over competing approaches, such as invariance alignment, robust linking, or Bayesian approximate invariance. Furthermore, we also argue why an intentionally misspecified multiple-group factor model with invariant item parameters can be justified if measurement invariance is violated.

Keywords:

• Factor analysis
• group comparisons
• measurement invariance
• partial invariance
• validity

1. Introduction

In the social and behavioral sciences, concepts are frequently measured utilizing questionnaires. Different indicators (i.e., questions or items) are used to infer values of a latent factor variable from a set of item responses. The unidimensional factor model is typically used for relating items to the latent variable of interest (Mulaik, 2009). In many applications, it is of interest to compare the distribution of the factor variables across different groups of subjects. Frequently used grouping variables are demographic variables, such as gender, social status, migration background, or different cultures or countries. In the psychometric literature, it is emphasized that the concept of measurement invariance (Flake et al., 2017; Meredith, 1993; Millsap, 2011; Molenaar & Borsboom, 2013; Somaraju et al., 2022; Vandenberg & Lance, 2000; Wicherts, 2016) plays a vital role in justifying meaningful and valid comparisons of means and standard deviations of the latent factor across groups (e.g., Boer et al., 2018; D’Urso et al., 2022; Kankaraš & Moors, 2014). Different levels of measurement invariance are discussed in the literature, depending on the restrictions that are imposed on the parameters of the factor model across groups. In the context of group mean comparisons, the focus is typically on scalar (also labeled strong or strict) measurement invariance. Scalar invariance assumes the existence of equal (i.e., invariant) item intercepts and item loadings in the factor model across groups.

The current social science consensus is that establishing measurement invariance is a necessary prerequisite for group comparisons because it ensures that items have the same meaning across groups (Davidov et al., 2014; Lacko et al., 2022; Meredith & Teresi, 2006; Meuleman et al., 2022; Putnick & Bornstein, 2016; but see for opposing views Funder, 2020; Protzko, 2022; Welzel & Inglehart, 2016). In his popular textbook on structural equation modeling, Kline (2016, p. 398) argues that “strong invariance is the minimal level required for meaningful interpretation of group mean contrasts.” Boer et al. (2018) explicitly write that “only scalar invariance allows validly comparing scale mean scores across cultures.” Furthermore, they also note that “comparing scale mean scores across cultures (using t-tests, multivariate ANOVAs, SEM with mean structures, or multilevel analyses) is appropriate only if scalar invariance is established.” Unequivocally, Davidov and Meuleman (2019) also state that “scalar invariance is a necessary condition to make valid comparisons of latent [group] means.” In line with these statements, Putnick and Bornstein (2016) suggest that “…measurement invariance is fast becoming de rigueur in psychological and developmental research.” Given these quotes from the literature, it seems legitimate to say that there exist strong claims that measurement invariance would be a prerequisite for meaningful and valid comparisons across groups.

In this article, we question the relevance of measurement invariance testing for meaningful and valid group comparisons. Our argument proceeds in two steps. In the first step, we clarify that the assumption of scalar measurement invariance across groups imposes the absence of item-by-group interactions in item parameters. However, we challenge the importance of establishing scalar measurement invariance and provide reasons why, in our view, the presence of item-by-group interactions does not preclude group comparisons. In addition, empirical research has shown that “scalar invariance, however, has been found to rarely fit the data well, especially in the case of many groups (Muthén & Asparouhov, 2018, p. 640). In response to this, different approaches for dealing with violations of measurement invariance (e.g., partial invariance, invariance alignment) have been proposed. These approaches allow for group comparisons under the assumption of partial (Byrne et al., 1989) or approximate (Lek et al., 2019; Muthén & Asparouhov, 2018; Seddig & Leitgöb, 2018) measurement invariance. However, in the second step, we clarify that group mean comparisons under partial or approximate invariance rely on strong—and often untestable—assumptions about the structure of the item-by-group interactions. Thus, there is ambiguity in defining group comparisons when the assumption of full measurement invariance is not met (i.e., measurement non-invariance) because none of the competing estimation approaches can be generally preferred. Consequently, we consider statistical techniques to investigate measurement invariance only as tools for exploring data that might help understand questionnaire data. However, we are convinced that the violation of measurement invariance typically does not threaten validity.

The article is structured as follows. In Section 2, we review the unidimensional factor analysis model for multiple groups. Section 3 discusses the assumptions of measurement invariance, and it is argued in Section 4 why we think that measurement invariance has only minor relevance for ensuring meaningful and valid group comparisons. Furthermore, we discuss different estimation strategies for dealing with measurement non-invariance in Section 5. We argue that there is ambiguity in defining group comparisons because a researcher must individually make untestable model assumptions. The paper closes with a discussion in Section 6.

2. Unidimensional Factor Analysis

Assume that there are I items $X_1,\dots ,X_I$ that are measured in a finite number of G groups. The unidimensional factor model for the I items in group g is given as (see, e.g., Meredith, 1993) (1) $$X_i={\nu }_{ig}+{\lambda }_{ig}F+e_i\mathrm{ },\text{ E}\left(F\right)={\alpha }_g\mathrm{ },\mathrm{ }\mathrm{SD}\left(F\right)={\psi }_g,$$(1)

where F is the latent factor, υ_ig and λ_ig are item intercepts and item loadings for item i in group g, respectively. The item residual variables e_i ($i=1,\dots ,I$) are assumed to have zero means and are uncorrelated with each other and uncorrelated with F. Note that the group-specific mean α_g and standard deviation ψ_g cannot be identified unless additional constraints on item intercepts or item loadings are imposed. Under a multivariate normal distribution assumption, the group-specific mean vector μ_g and the covariance matrix Σ_g are sufficient statistics of the factor model in group g. In the rest of the article, we deal with population-level data and do not distinguish sample statistics from population statistics because we focus on identification conditions and asymptotic bias discussions for population-level data.

The typical step in the assessment of measurement invariance starts with fitting the unidimensional factor model (1) within each group g (see, e.g., Schroeders & Gnambs, 2020; van de Schoot et al., 2012). One sets α_g = 0 and ψ_g = 1 as identification constraints. The group-specific item means µ_i,g = E(X_i | G = g) provide saturated estimates of identified item intercepts ${\nu }_{i,g}^{*}$ (i.e., µ_i,g = ${\nu }_{i,g}^{*}$). The covariances σ_ij,g = Cov(X_i, X_j | G = g) for pairs of items i and j in group g follow the restricted model (2) $${\sigma }_{ij,g}={\lambda }_{i,g}^{*}{\lambda }_{j,g}^{*},$$(2)

if the unidimensional factor model holds. In Equation (2)(2) $${\sigma }_{ij,g}={\lambda }_{i,g}^{*}{\lambda }_{j,g}^{*},$$(2) , ${\lambda }_{i,g}^{*}$ are the identified group-specific item loadings in each of the groups. The item loadings can be identified from observed covariances (Steyer, 1989) (3) $${\lambda }_{i,g}^{*}=\sqrt{\frac{{\sigma }_{ij,g}{\sigma }_{ik,g}}{{\sigma }_{jk,g}}}$$(3) for two items j and k that are distinct from i, assuming that all covariances are positive.¹ Overall, these properties make clear that the identified group-specific intercepts ${\nu }_{i,g}^{*}$ and loadings ${\lambda }_{i,g}^{*}$ in the unidimensional factor model are only functions of observed sufficient statistics µ_i,g and σ_ij,g.

For ordinal items, factor analysis can be based on item thresholds and polychoric correlations if the probit link function and the graded response model are used (Muthén, 1984). In the case of ordinal items, violations of measurement invariance are also labeled as differential item functioning (DIF; Holland & Wainer, 1993). In the following, we focus on the case that items are treated as continuous variables. Nevertheless, the main arguments in this paper also apply to ordinal items.

3. Absence of Item-By-Group Interactions Under Full Invariance

If the unidimensional factor model is fitted separately for each group, group-specific differences in factor mean α_g and standard deviations ψ_g (g = 1,…,G) are represented in identified item intercepts ${\nu }_{i,g}^{*}$ and loadings ${\lambda }_{i,g}^{*}.$ Hence, some identification constraints must be imposed to enable the computation (i.e., identification) of α_g and ψ_g. One popular identification constraint is assuming full invariance of item parameters across groups; that is, the existence of common item intercepts and loadings across groups. In the literature, two types of measurement invariance are distinguished (Millsap, 2011): Measurement invariance regarding item loadings (metric invariance or weak invariance) and measurement invariance regarding item intercepts and loadings (scalar invariance or strong invariance). Both invariance types will also be referred to as full invariance if the conditions exactly hold.

Metric invariance assumes equal loadings across groups, which enables the identification of group-specific standard deviations ψ_g of the factor variable. The assumption of the existence of common item loadings λ_i,0 results in (4) $${\lambda }_{i,g}^{*}={\lambda }_{i,0}^{}{\psi }_g^{}.$$(4)

For model identification, we set the standard deviation ψ₁ of the first group equal to 1. With positive item loadings,² we can use the notation $l_{i,g}^{*}=\log {\lambda }_{i,g}^{*},$ l_i,0 = log λ_i,0, and p_g = log ψ_g. We get from Equation (4)(4) $${\lambda }_{i,g}^{*}={\lambda }_{i,0}^{}{\psi }_g^{}.$$(4) (5) $$l_{i,g}^{*}=l_{i,0}^{}+p_g^{}.$$(5)

It can be seen that (5) is an additive model for two-way data $l_{i,g}^{*}$ (i = 1,…,I; g = 1,…,G), which is equivalent to presupposing the absence of interactions in an analysis of variance (ANOVA) model for logarithmized item loadings $l_{i,g}^{*}$ (see Robitzsch & Lüdtke, 2020; van der Linden, 1994). Note that condition (4) holds if (6) $$\frac{{\lambda }_{i,g}^{*}}{{\lambda }_{i,h}^{*}}=\frac{{\lambda }_{j,g}^{*}}{{\lambda }_{j,h}^{*}}$$(6) for all items i and j and pairs of groups g and h. Thus, metric invariance across groups is established if the ratio of item loadings is constant across items for all pairs of groups. The literature often argues that metric invariance is required for a meaningful and valid comparison³ of group standard deviations.

Scalar invariance presupposes the existence of common item intercepts ν_i,0 in addition to common item loadings λ_i,0 across groups. This property allows the identification of group means α_g of the factor variable. Assuming that metric invariance holds (i.e., the existence of common item loadings λ_i,0), the assumption of common item intercepts results in (7) $${\nu }_{i,g}^{\mathrm{*}}={\nu }_{i,0}^{}+{\lambda }_{i,0}^{}{\alpha }_g^{}$$(7)

By dividing the two sides of Equation (7)(7) $${\nu }_{i,g}^{\mathrm{*}}={\nu }_{i,0}^{}+{\lambda }_{i,0}^{}{\alpha }_g^{}$$(7) by λ_i,0 and using the notation for transformed item intercepts $u_{i,g}^{*}={\nu }_{i,g}^{*}/{\lambda }_{i,0}^{},$ $u_{i,0}^{}={\nu }_{i,0}^{}/{\lambda }_{i,0}^{}$ and $a_g^{}={\alpha }_g^{}/{\lambda }_{i,0}^{},$ we obtain the additive model (8) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}$$(8)

For model identification, the mean α₁ of the first group is set to 0. Again, Equation (8)(8) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}$$(8) corresponds to an ANOVA model for transformed item intercepts $u_{i,g}^{*}$ with only main effects. Hence, scalar invariance excludes the possibility of item-by-group interactions in transformed item intercepts $u_{i,g}^{*}.$ In the literature, it is often argued that scalar invariance is required for a meaningful and valid comparison of group means.

To sum up, we have shown that the assumptions of metric and scalar invariance correspond to ANOVA models without interaction effects of transformed group-specific item loadings and intercepts. We critically reflect on the relevance of this property in the following Section 4.

4. Why Full Invariance Has Limited Relevance

In this section, we present three arguments why the assumption that metric and scalar measurement invariance exactly holds (i.e., full invariance) has only limited relevance for ensuring a meaningful and valid comparison of group means and standard deviations.

First, we question the usefulness of the implied assumption of the absence of interaction effects in ANOVA models for transformed item loadings and item intercepts (see Equations 5(5) $$l_{i,g}^{*}=l_{i,0}^{}+p_g^{}.$$(5) and 8(8) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}$$(8) ). Of course, one can compute average effects in ANOVA models if interaction effects occur. However, additional constraints (i.e., weighting schemes) must be imposed for interaction effects (Maxwell et al., 2017). In the causal effects literature, it is pointed out that average treatment effects are well-defined if there are heterogeneous treatment effects (i.e., the treatment effect is moderated by covariates; see, e.g. Morgan & Winship, 2015). It is sometimes argued that one compares apples and oranges if measurement invariance is violated (see, e.g., Chen, 2008, or Greiff & Scherer, 2018, for a discussion). If items function differentially across groups (i.e., there are item-by-group interactions in item parameters), items do not measure “the same concept” (i.e., factor; see also Little, 2013). This would threaten the validity and preclude researchers from making meaningful group comparisons. We think such statements can only be regarded as metaphorical claims without statistical or empirical meaning (see Robitzsch & Lüdtke, 2022, for a discussion). For example, Meuleman et al. (2022) argue that group comparisons “should reflect true differences rather than measurement differences.” It remains unclear what is meant by “true differences” because these have to be identified from “measurement differences” (i.e., observed data) in any case. Such statements indicate that those researchers simply believe there should be no differential functioning of the items because this would compromise meaningful and valid group comparisons. In the same way, Meredith and Teresi (2006) argue that “the failure of invariance to hold is […] evidence that the manifest variables [i.e., observed variables] fail to measure the same latent attributes [i.e., latent variables] in the same way in different situations.” As it is also evident in this quote, the definition of latent variables becomes circular because they do not even exist in the statistical model and must always be inferred from manifest variables.⁴ Instead, latent variables only emerge (i.e., they are represented by model parameters in the multiple-group factor analysis) from restrictions in the mean vector and covariance matrix of observed variables. The typical reasoning seems to imply that “true differences” are attributed to a causal interpretation of the latent factor variable (Borsboom, 2008). Notably, the substitution of assessing the adequacy of the measurement process with demonstrating acceptable model fit of psychometric (factor) models can be questioned (Uher, 2021; see also Borgstede & Eggert, 2023, or Edelsbrunner, 2022).

Second, it might be argued that full invariance is desirable because group-specific factor means and standard deviations were invariant if only a subset of items would be used in an analysis. Hence, empirical results of group comparisons do not depend on the particular choice of items in a questionnaire. We think the choice of items will impact the concept to be measured in any case (VanderWeele, 2022). However, in many applications (e.g., comparing self-concept scales between gender groups or countries), the scale and, hence, the concrete choice of items is fixed. With a fixed scale, the property of invariance concerning subsets of items is simply irrelevant. It seems plausible that group comparisons could change if a self-concept scale is measured with four items selected from a five-item questionnaire. However, we would argue that the measured concept will change with a subscale of items, and it is unclear why the concept of interest should be restricted to fewer items.

Third, we question whether the call for the absence of item-by-group interactions in the ANOVA model for transformed item loadings and intercepts even poses a threat to validity if the differential functioning of items is construct-relevant (Camilli, 1993; Shealy & Stout, 1993).⁵ By excluding items with item-by-group interactions, validity would decrease because the purposefully constructed measurement instrument would be changed in statistical analysis (De Los Reyes et al., 2022; El Masri & Andrich, 2020; Robitzsch & Lüdtke, 2022). We further elaborate on this issue when discussing the role of partial invariance.

In empirical research, full invariance seems to be as rare as unicorns. Metric invariance seems much more frequently fulfilled than scalar invariance (Rutkowski & Svetina, 2014). The following Section 5 discusses different approaches for dealing with violations of measurement invariance. It is argued that there is unavoidable ambiguity in identifying group means and standard deviations if the assumption of full invariance is not met. We assert that it is also unnecessary for meaningful group comparisons to demonstrate that measurement invariance is only violated to some tolerable extent.

5. Ambiguity in Choosing Identification Constraints in Violations of Full Invariance

In practice, full invariance will almost always be violated to some extent (e.g., Ellis, 1993; Leitgöb et al., 2023). As argued in Section 3, the extent of non-invariance is characterized by interaction effects of the ANOVA models defined in Equations (5)(5) $$l_{i,g}^{*}=l_{i,0}^{}+p_g^{}.$$(5) and (8)(8) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}$$(8) . The interactions can be regarded as DIF effects in item loadings (i.e., ${\tilde{l}}_{i,g}^{}$) and item intercepts (i.e., ${\tilde{u}}_{i,g}^{}$) (9) $$l_{i,g}^{\mathrm{*}}=l_{i,0}^{}+p_g^{}+{\tilde{l}}_{i,g}^{}$$(9) (10) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}+{\tilde{u}}_{i,g}^{}$$(10)

For transformed item loadings $l_{i,g}^{*}$ and item intercepts $u_{i,g}^{*}$ in Equations (9)(9) $$l_{i,g}^{\mathrm{*}}=l_{i,0}^{}+p_g^{}+{\tilde{l}}_{i,g}^{}$$(9) and (10)(10) $$u_{i,g}^{\mathrm{*}}=u_{i,0}^{}+a_g^{}+{\tilde{u}}_{i,g}^{}$$(10) , there are I × G input parameters, but there are I + G main effects (i.e., p_g and a_g) and I × G interaction effects (i.e., ${\tilde{l}}_{i,g}^{}$ and ${\tilde{u}}_{i,g}^{}$) in model equations. Hence, the number of the estimated parameters would exceed the number of input parameters. This implies that identification constraints must be imposed in the violation of measurement invariance because there are more unknown parameters than input parameters. The statistical approaches for handling non-invariance are distinguished in different aspects. First, we can distinguish approaches that specify a multiple-group factor model in one step from approaches that first estimate group-wise factor models and link the results onto a common metric in a subsequent linking approach (Kolen & Brennan, 2014).⁶ It has been demonstrated that two-step methods can be alternatively estimated as one-step methods with appropriate constraints of interaction effects in (9) and (10) (see Little et al., 2006 or von Davier & von Davier, 2007). Second, approaches can be distinguished whether or not they explicitly model DIF effects. As we will argue below, this distinction is practically irrelevant because it is rather the choice of the estimation function that matters than the mere fact of including DIF effects in the factor model (see Robitzsch, 2022). Third, estimation approaches can be distinguished whether their resulting group comparisons are robust regarding the presence of a few interaction effects in transformed item loadings or item intercepts (9) and (10). This distinction is the most important because it classifies the approaches into robust and nonrobust linking of groups (Robitzsch, 2021). Importantly, the former allows some items or their item parameters to be excluded from comparisons, while the latter does not exclude or downweigh items. We now classify competing approaches to handling non-invariance regarding this aspect.

5.1. Partial Invariance Approach and Robust Linking

Among the robust linking approaches, the partial invariance approach is likely the most frequently employed solution to measurement non-invariance (Byrne et al., 1989; Steenkamp & Baumgartner, 1998). In this approach, it is assumed that only a few interaction effects in (9) and (10) differ from zero, while most interaction effects are equal to zero. Modification indices can be used to detect which of the parameters should be estimated as different from zero. The multiple-group factor model is reestimated in a second analysis step by allowing some group-specific item parameters (Byrne et al., 1989). It is often argued that partial invariance ensures meaningful and valid group comparisons if full invariance is violated. From the perspective of ANOVA estimation with interaction effects, it is interesting that partial invariance corresponds to an estimation problem with a loss function that minimizes the number of interaction effects (i.e., using an L₀ loss function; see Davies, 2014; Robitzsch, 2020a; Robitzsch & Lüdtke, 2020). We have pointed out elsewhere that the partial invariance approach comes with the disadvantage that comparing pairs of groups could effectively depend on different sets of items because items whose group-specific parameters were freed do not contribute to the linking process (Robitzsch & Lüdtke, 2022). For example, in a cross-cultural study, the comparison of Poland and Germany could depend on a different set of items than the comparison of Poland and Austria.⁷ We consider this property as a threat to validity. Thus, we would argue that there is a danger of comparing apples and oranges when using the partial invariance approach in applications with more than two groups (Robitzsch & Lüdtke, 2022). In contrast, the set of items is fixed for all group comparisons if a misspecified multiple-group factor model with invariant item parameters is fitted to all groups.

5.2. Regularization Techniques

As an alternative to the two-step method of partial invariance, the regularization technique with a lasso-type penalty function (see, e.g., Finch, 2022, for an overview) can be utilized for fitting the multiple-group factor model (Bauer et al., 2020; Geminiani et al., 2021; Huang, 2018; Liang & Jacobucci, 2020; Schauberger & Mair, 2020). Like in partial invariance, it is also assumed that DIF effects are sparsely distributed; that is, the majority of group-specific item parameters must be equal to zero to enable an unbiased estimation of group means and standard deviations. One can expect that partial invariance and regularization will provide very similar results in applications (Robitzsch, 2022).

5.3. Invariance Alignment

Minor violations of measurement invariance are also referred to as approximate invariance in the literature (Arts et al., 2021; Lomazzi, 2021; Luong & Flake, 2022; Meitinger et al., 2020; Muthén & Asparouhov, 2018; van de Schoot et al., 2013). This label involves approaches that acknowledge deviations from full invariance in multiple-group factor analysis with DIF effects. Invariance alignment is a two-step estimation procedure in which the group-specific identified item parameters are linked in a second step such that the majority of DIF effects are small while there are only a few large DIF effects (Asparouhov & Muthén, 2014; Muthén & Asparouhov, 2014). In this regard, invariance alignment is ideally suited to situations close to partial invariance. The alignment method may be considered to automate the partial invariance approach (Leitgöb et al., 2023). In fact, invariance alignment falls into the class of robust linking techniques popular in item response theory (Robitzsch, 2020a). Alternative techniques, such as robust Haberman linking, might be preferred over invariance alignment for statistical reasons (Robitzsch, 2020a). The vital aspect of robust linking techniques, such as invariance alignment is that some large DIF effects are treated as outliers and should be eliminated or down-weighted in the estimation of group means and standard deviations (Magis & De Boeck, 2011; Robitzsch, 2022; Robitzsch & Lüdtke, 2020; Wang et al., 2022). The similarity of lasso-type regularization and robust linking has also been pointed out by Chen et al. (2021).⁸ Because invariance alignment may be viewed as an alternative implementation of the partial invariance approach, our critique of comparing apples with oranges in partial invariance also applies to the alignment, robust linking, and regularization approaches.

5.4. Bayesian Approximate Invariance with Normal Prior Distributions

As an alternative, Bayesian approximate invariance (Muthén & Asparouhov, 2018) has been proposed that specifies normal prior distributions with known variances in differences between group-specific item parameters (van de Schoot et al., 2013). It has been shown that normal prior distributions of parameter differences correspond to normal prior distributions of DIF effects (Battauz, 2020; Robitzsch, 2022). Alternatively, the variance of the prior distribution can also be estimated, referred to as a multiple-group factor analysis with random effects (De Boeck, 2008; De Jong et al., 2007; Fox & Verhagen, 2010). Importantly, using normal prior distributions presupposes that DIF effects are normally distributed. In particular, it is assumed that they follow a symmetric distribution. This assumption is in stark contrast to the assumption of partial invariance in which the majority of DIF effects is zero (or very small). With only a few items for a factor (say, smaller than 20), it is very unlikely that statistical modeling will help find the correct distributional assumption of DIF effects. Hence, it is up to the researcher whether he or she believes in a particular strategy for modeling measurement non-invariance (Robitzsch, 2022). It should be emphasized that robust linking and, hence, invariance alignment treat DIF effects as outliers and essentially remove items from some group comparisons. This assumption is not imposed in Bayesian approximate invariance. All items enter the determination of group means and group standard deviations but are somehow weighted according to the size of DIF effects.

5.5. Intentionally Misspecified Models

Another option would be to intentionally not model DIF effects (Robitzsch, 2022). That is, invariant item parameters are assumed in a statistical model, and there is no attempt to introduce additional item parameters for DIF effects. The estimation function of the multiple-group factor model weighs sampling error and model error. M-estimation theory provides the statistical framework for obtaining valid statistical inference for misspecified models (Berk et al., 2014; Boos & Stefanski, 2013; Huber, 1964; White, 1982). It is well-known that maximum likelihood estimation will be the most efficient estimation method in the case of correctly specified models (i.e., under full invariance). However, under violations of invariance, alternative estimation methods, such as diagonally weighted least squares or unweighted least squares⁹ can be used (Robitzsch, 2022). Using unweighted least squares, model errors referring to DIF effects would be equally weighted in the computation of group means. This might be seen as a desirable property. If deviations in DIF effects should be down-weighted or essentially eliminated from computing group means and standard deviations, robust estimation functions, such as least absolute deviation estimation can be utilized (Robitzsch, 2022; Siemsen & Bollen, 2007).

5.6. Summary

Overall, we would like to argue that there is unavoidable ambiguity in handling situations of measurement non-invariance. We find that recommendations that particular approaches enable (approximately) unbiased estimation of group means are ill-advised because individual researchers define how to handle DIF effects by choosing an estimation function (Robitzsch, 2022). We do not think there are statistical arguments for preferring a partial invariance or an invariance alignment approach over estimating the multiple-group factor analysis model assuming invariant item parameters and utilizing unweighted least squares estimation. Such recommendations depend on how the truth is defined. It is unlikely that there will be any consistent frontrunner among competitive approaches for dealing with measurement non-invariance (see Muthén & Asparouhov, 2018; Pokropek et al., 2019, 2020). Notably, the choice of estimation methods under measurement non-invariance relies on assumptions about the structure of DIF effects. There can be a few large but outlying DIF effects, or DIF effects can be rather symmetrically distributed. The former case corresponds to a partial invariance approach, while the latter case refers to Bayesian approximate invariance with normal prior distributions. With only a few items, it is unlikely to determine the true data-generating structure. In contrast, an estimation method explicitly defines the researcher’s belief about the structure of DIF effects and is “strategically chosen” (Asparouhov & Muthén, 2023). Asparouhov and Muthén (2023) argue that alignment-based methods will typically provide less biased group comparisons compared to Bayesian approaches with normal prior distributions on DIF effects. We find such statements as recommendations for practitioners highly questionable because they come with an impetus of general validity that could never be seriously justified.

6. Discussion

In this article, we have argued that the relevance of measurement invariance for ensuring meaningful and valid group comparisons is likely to be overstated (Welzel et al., 2021). It was pointed out why full invariance should not be considered a prerequisite for meaningful and valid group comparisons. Nevertheless, some kind of measurement equivalence of items (such as translational equivalence) must be ensured for validity reasons (van de Vijver, 2018). However, measurement invariance testing should not replace the difficult non-statistical task of demonstrating measurement equivalence (but see Fischer et al., 2022). Several researchers erroneously equate the concepts of measurement invariance and measurement equivalence (Boer et al., 2018; Geiser, 2020). A large part of the literature argues that measurement invariance is necessary but not sufficient for ensuring meaningful and valid group comparisons (Fischer et al., 2022, Meitinger, 2017; van de Vijver, 2018). This article argues against this perspective: measurement invariance is not a necessary condition (see our arguments in Section 4).

We want to highlight that measurement non-invariance can only detect item-specific functioning across groups. Bias due to systematic effects, such as response styles cannot typically be detected with measurement invariance assessment. Nevertheless, assessing measurement non-invariance can prove a viable screening tool for detecting potential issues in scale indicators. However, in our view, items should not be automatically removed from group comparisons or should receive group-specific item parameters in the partial invariance approach if measurement non-invariance would not be determined as construct-irrelevant. Researchers should provide substantive (and non-statistical) evidence of why they believe items should be removed from the comparisons. There should not be mechanistic rules for group comparisons solely determined through statistical criteria. In this sense, we think applying psychometric models should always be carried out from the perspective of (external) validity (Kane, 2013). We feel it is inappropriate to accuse other researchers of conducting non-rigorous research that does not test for measurement invariance (e.g., Boer et al., 2018; D’Urso et al., 2022). Importantly, we do not want to claim that a justification for selecting particular items for measuring a construct is not required. However, we firmly believe that measurement invariance assessment is not necessarily a prerequisite for establishing sufficient measurement quality of the operationalization of a construct.

We also argued that there is unavoidable ambiguity in defining identification constraints if the assumption of full invariance is violated (i.e., measurement non-invariance). There are competing plausible methods for handling non-invariance,¹⁰ and it is usually up to the individual researcher to select one among the methods (Robitzsch, 2022; Zitzmann & Loreth, 2021). In particular, we are not convinced by preferring partial invariance over alternative approaches because this results in comparing apples with oranges for more than two groups. In addition, applying the partial invariance approach implies that some noninvariant items are excluded from group comparisons, which can cause a threat to validity. Moreover, it is unlikely that sparsely distributed DIF effects frequently occur in practice, although this setting is most frequently employed in simulation studies (e.g., Asparouhov & Muthén, 2014; Steinmetz, 2013). In our experience with cognitive data and with many test items, it is much more likely that DIF effects are symmetrically distributed and closely follow a normal distribution.

Furthermore, we question whether a statistical test of full invariance or model fit effect sizes has relevance in applied research. In contrast, the specified multiple-group factor analysis with invariant item parameters is intentionally misspecified, and the assumption of measurement invariance is only made for identification reasons. We have argued that unweighted least squares estimation might be preferred over maximum likelihood estimation in the multiple-group factor model because the former equally weighs model errors (i.e., DIF effects). Hence, we also think that the emphasis on approximate measurement approaches in violations of full invariance is misguided (e.g., Leitgöb et al., 2023) because misspecified multiple-group factor analysis with assumed (but empirically violated) invariance is equally defensible. Notably, the assumed pattern of DIF effects (i.e., sparsely or symmetrically densely distributed) is crucial for the choice of analysis.

6.1. Why Longitudinal and Multilevel Measurement Invariance Are Also Not Required

We think the concept of measurement invariance is also not necessary for ensuring meaningful and valid comparisons of similar types of analysis in factor models. Establishing measurement invariance regarding time points in longitudinal data (Geiser, 2020; Little, 2013; Wicherts et al., 2004; Winter & Depaoli, 2020) or the combination of groups and time points in longitudinal data (Avvisati, 2020; Koc & Pokropek, 2022; Lee & von Davier, 2020; Seddig et al., 2020) does not seem a prerequisite to us for making comparisons across different time points by applying our arguments in the case of groups to time points. Moreover, we also do not believe that measurement invariance across levels in multilevel levels (Jak, 2019; Jak & Jorgensen, 2017; Ryu, 2014; Ryu & Mehta, 2017) is required, nor that the absence of cluster bias (Jak et al., 2013) seems to be a prerequisite for analyzing multilevel data in multilevel factor models.

6.2. Latent Class and Mixture Models for Handling Measurement Noninvariance

Alternative approaches stick to the case of considering measurement invariance for a larger number of groups (say, more than ten). Latent class or mixture models can be applied that define clusters (i.e., latent classes) of invariant groups (De Roover, 2021; De Roover et al., 2022; Eid, 2019; see also Zieger et al., 2019). Hence, these approaches suggest that comparisons would only be meaningful for subsets of groups that share the same item parameters in the measurement model. Hence, one would typically conclude in applications that comparisons across all groups would not be meaningful and the original research question cannot be answered. We argued in this article why this reasoning is unjustified.

6.3. Why Simulation Studies Have Limited Relevance for Recommendations for Practitioners

A lot of simulation research demonstrates how group comparisons are biased in particular violations of measurement invariance (e.g., Steinmetz, 2013). We think that simulation studies have limited relevance in generating recommendations for practitioners in applied research in measurement non-invariance. Simulation studies always define the truth by assuming specific data-generating parameters. The essential point is that the simulators already know the structure of DIF effects in the measurement situation. However, as we argued in the previous section, applied researchers only have empirical data. They cannot typically test competing assumptions on the structure of measurement non-invariance because there is always ambiguity on how to believe about the structure of DIF effects. Applied researchers can only argue why they prefer particular treatments of measurement non-invariance by choosing a particular estimation method. Notably, there are no definite criteria for legitimate statements that using the partial or invariance alignment approach will generally provide less biased group comparisons in empirical research. Simulation studies can help methodological and applied researchers understand different estimation methods in different data-generating models. Nevertheless, the choice of a statistical method in a concrete empirical application cannot be defended based on results from simulation studies.

In fact, most simulation studies utilized the partial invariance model as the data-generating model. It comes as no surprise as a finding of such a simulation study that partial invariance or invariance alignment approach were appropriate estimation methods in this case. However, with a different data-generating model, a quite different conclusion would be drawn.

6.4. Measurement Invariance with Multiple Covariates

The original definition of measurement invariance dates back to Mellenbergh (1989) and Meredith (1993). Measurement invariance for a set of item responses X is fulfilled if the conditional distribution of item responses given the factor F does not depend on any observed or unobserved covariates summarized in a vector V. More formally, it is required for measurement invariance that P(X | F,V = P(X | F) holds for all possible covariates V. In principle, V could define any subpopulation of persons from the original population. This property led Meredith (1993) to define measurement invariance under the concept of selection invariance. Selection invariance implies that a measurement model holds (after utilizing appropriate identification constraints) in any subpopulation of the original population. In particular, this property also rules out the possibility of a latent grouping variable like in factor mixture models (Cohen & Bolt, 2005; Cole et al., 2019; Lubke & Muthén, 2007). Suppose one would believe in the requirement of the more strict measurement invariance definition of Mellenbergh and Meredith. In that case, a multitude of statistical models must be fitted to rule out any kind of non-invariance. In principle, V could be a high-dimensional vector of discrete and continuous variables and statistical techniques based on machine learning or regularization (Bauer et al., 2020; Brandmaier & Jacobucci, 2023; Jacobucci et al., 2019; Tutz & Schauberger, 2015) might help to detect non-invariance. No doubt, there are feasible and valid statistical approaches that can handle a high-dimensional covariate vector V for investigating DIF. However, it remains questionable what the purpose of such statistical enterprises is. Why should one believe that a data-driven modified model provides more meaningful results than applying inference based on a priori clearly defined single-factor model?

6.5. Assessing Measurement Invariance for Ordinal Items

In typical measurement invariance analyses, Likert scale items with four-point, five-point, or continuous bounded scales are used. We have argued elsewhere that it is defensible using the factor model in a continuous treatment of the items by analyzing Pearson correlations (or covariances of the raw items) instead of an ordinal treatment based on polychoric correlations (Robitzsch, 2020b; see also Kampen & Swyngedouw, 2000). To us, the choice of the metric of the latent variable (i.e., treating items as continuous or ordinal) must be based on substantive and not statistical reasons, and it is not legitimate to say that one of the two methods will typically lead to biased results because such reasoning depends on how researchers define the truth. Consequently, researchers should not generally prefer an ordinal treatment of items in the assessment of measurement invariance (see Svetina et al., 2020, for a tutorial). Moreover, we believe that it is inappropriate to ask whether violations of measurement invariance are the consequence of using a continuous instead of an ordinal treatment (see, e.g., Chen et al., 2020; Meuleman et al., 2022; Welzel et al., 2021, 2022, for such discussions). In our view, the choice of the metric of the latent variable is independent of assessing measurement invariance. Our conclusion that measurement invariance is not necessary for a meaningful and valid comparison across groups also applies to ordinal treatments of measurement invariance based on polychoric correlations or using item response models estimated with maximum likelihood. Of course, the reasoning would also apply to factor models that utilize bounded distributions for items (Noel & Dauvier, 2007; Revuelta et al., 2022).

6.6. Is Scalar Invariance Sufficient for Comparing Regression Coefficients Across Groups?

Moreover, we would like to point out that establishing full invariance in a unidimensional factor model does not guarantee the invariance of relationships of a factor variable with covariates across groups. In such a case, invariance analysis must be performed for items by testing for potential interaction effects of groups and the covariate of interest. Unfortunately, incorrect statements that metric invariance in a multiple-group model ensures the comparability of covariances of the factor variable and covariates across groups can be frequently found in the literature (e.g., Davidov & Meuleman, 2019; He et al., 2017; Leitgöb et al., 2023). For example, Leitgöb et al. (2023) write: “If metric invariance is given, one may compare unstandardized associations (i.e., covariances or unstandardized regression coefficients between the latent variables of interest for which metric invariance holds). Considering our previous example, one may compare the covariance or the unstandardized regression coefficient between the variable ‘age’ and the latent variable ‘attitudes toward immigration.’” Moreover, He et al. (2017) note that “with metric invariance of the scales established, correlational analysis among metric invariant constructs and achievement can be considered valid.” These statements imply that it would only be necessary to demonstrate metric invariance for all involved latent variables separately to ensure invariance of unstandardized associations regarding selecting subsets of items. However, this is incorrect because one must show that in the mentioned example in Leitgöb et al. (2023) that in a joint model involving “age” and the indicators of “attitudes toward immigration,” residual correlations between age and the indicators must be equal to zero. This assumption remains untested if metric invariance is only independently shown for all latent variables. This frequently found inconsistency in the literature is particularly annoying because “Methodologists […] insist with increasing vigor that detecting ‘non-invariance’ in […] is an infallible sign of […] in-equivalences in how respondents understand the items” (Welzel et al., 2021). Hence, the usual steps in measurement invariance literature are anything but rigorous. However, we argued that we generally question the methodological concept of measurement invariance.

6.7. Quantifying Measurement Non-invariance as an Additional Source of Uncertainty

Finally, we believe it might be advantageous to quantify the extent of non-invariance as an additional source of uncertainty in estimated parameters, such as group means and standard deviations. Such an approach has been discussed as linking errors for applications in educational assessment based on item response models (Monseur & Berezner, 2007; Robitzsch & Lüdtke, 2019).¹¹ As an alternative approach in the multiple-group factor model, the model error can be parametrized as an additional source of uncertainty in the estimation function (Wu, 2010; Wu & Browne, 2015). This approach results in an uncertainty quantification of estimated parameters that comprises sampling and model errors. This approach should be more widely implemented in empirical research involving (multiple-group) confirmatory factor analysis and structural equation modeling.

Notes

1 In typical applications, this restriction is not a serious limitation because item loadings can all be made positive by multiplying item scores by −1. An example might be the presence of positively and negatively worded statements. The item scores of all negatively worded items can be multiplied by −1.

2 Ibid.

3 In the following, we use the terms “valid” and “meaningful” interchangeably.

4 Formally, one can define true scores as intraindividual distributions of manifest items in stochastic measurement theory (SMT; Eid, 2000; Steyer, 1989; Steyer et al., 2015). Factor models are defined by imposing some restrictions on the true score distributions in SMT. However, SMT does not resolve identification issues in factor models in general and measurement non-invariance in particular. SMT just includes an additional and redundant layer for defining true scores (and latent variables) as a notational exercise, but arrives at the same identification conditions compared without these definitional exercises. Hence, ambiguity in defining group comparisons in measurement non-invariance situations cannot be resolved by utilizing SMT.

5 An anonymous reviewer argued that assessing and eliminating DIF items would be particularly relevant in high-stakes assessment tests. We tend to disagree with such a statement. As described by Camilli (2006), test fairness should be clearly distinguished from DIF (i.e., measurement non-invariance). Test fairness is related to unsystematic (i.e., item-specific) and systematic disadvantages of particular groups of test takers. It is evident that DIF assessment only focuses on item-specific DIF and does not test for systematic group disadvantages. If DIF exists, high-stakes testing is a good showcase that removing construct-relevant DIF items from the test is questionable. According to Camilli (2006) and our conviction, only construct-irrelevant DIF items should be removed from the test because only those items are related to test unfairness.

6 Note that two-step linking approaches are much more popular in item response modeling than in applications of structural equation modeling.

7 Zieger et al. (2019) argue that in cross-cultural studies researchers should identify groups of countries with invariant item parameters in order to allow comparisons among them. However, in the violation of invariance, not all countries should be compared simultaneously.

8 Indeed, the recently proposed penalized structural equation modeling (Asparouhov & Muthén, 2023) can be viewed as a regularized estimation approach.

9 Unweighted least squares estimation may be justified for standardized variables or if all items were measured using the same Likert scale.

10 See, e.g., Pokropek and Pokropek (2022), for more sophisticated approaches based on deep learning techniques.

11 In fact, linking errors can be interpreted as summary statistics of the expected parameter change (Oberski, 2014) in group means and standard deviations.