Evaluating Factorial Invariance: An Interval Estimation Approach Using Bayesian Structural Equation Modeling

Abstract

In this study, we introduce an interval estimation approach based on Bayesian structural equation modeling to evaluate factorial invariance. For each tested parameter, the size of noninvariance with an uncertainty interval (i.e. highest density interval [HDI]) is assessed via Bayesian parameter estimation. By comparing the most credible values (i.e. 95% HDI) with a region of practical equivalence (ROPE), the Bayesian approach allows researchers to (1) support the null hypothesis of practical invariance, and (2) examine the practical importance of the noninvariant parameter. Compared to the traditional likelihood ratio test, simulation results suggested that the proposed Bayesian approach could offer additional insight into evaluating factorial invariance, thus, leading to more informative conclusions. We provide an empirical example to demonstrate the procedures necessary to implement the proposed method in applied research. The importance of and influences on the choice of an appropriate ROPE are discussed.

Keywords:

• Bayesian SEM
• parameter estimation
• factorial invariance
• highest density interval
• region of practical equivalence

Article information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was partly supported by Grant SES-1659936 from the National Science Foundation.

Role of the funders/sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The authors would like to thank Sarah Depaoli and the anonymous reviewers for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions or the National Science Foundation is not intended and should not be inferred.

Notes

1 Different procedures have been developed for conducting LRT. In this study, we focused on the procedure in which researchers first select at least one item as a reference indicator (RI), and fit the baseline model with all other parameters freely estimated. Then, factorial invariance tests are conducted by fitting a series of models by imposing increasingly restrictive equality constraints (i.e. the free baseline approach). Alternatively, one can begin such tests by fitting a model with all of the parameters constrained to be equal, and then progressively relaxing certain equality constraints (i.e. the constrained baseline approach). Further information on the constrained baseline approach can be found in Stark, Chernyshenko, and Drasgow (2006) and Kim and Yoon (2011). In addition, noninvariance can also be detected by applying the iterative procedures (Cheung & Rensvold, 1998), in which each single item serves, in turn, as an RI (see also Cheung & Lau, 2012).

2 That is, the difficulty parameter (b) decreased by .1 in Group 2.

3 Using Google scholar, through 04/09/2018, the number of citations of Cheung & Rensvold (2002), Chen (2007), and Meade et al. (2008) are 6,595, 2,125, and 584, respectively.

4 Cheung & Rensvold (2002) recommended that |Δ CFI| ≥ .01 implied noninvariance; Meade et al. (2008) suggested that |Δ CFI| ≥ .002 implied noninvariance. Chen (2007) proposed cutoffs based on sample size; that is, the cutoffs for noninvariance were |Δ CFI| ≥ .005 for N ≤ 300, and |Δ CFI| ≥ .01 for N > 300.

5 While these cutoffs provide guidelines for interpretation, it is noted, however, that that the conventional cutoffs for evaluating RMSEA are believed to be overly stringent for the purpose of assessing invariance. As such, Yuan and Chan (2016) recommended use of the adjusted cutoff values presented above for assessing factorial invariance. For computation details, see Yuan and Chan (2016).

6 For example, the same RMSEA value (say 0.05) may hold a different meaning in terms of the model misspecification when models differ in terms of the magnitude of factor loadings and model size (Chen, Curran, Bollen, Kirby, & Paxton, 2008; Maydeu-Olivares, Shi, & Rosseel, 2018; Savalei, 2012; Shi, Lee, & Maydeu-Olivares, 2018).

7 When fitting a CFA model, the metric of the latent variables must be set to identify the model. In testing for factorial invariance, a common method for identification is to use (at least) one item as a reference indicator (RI; Cheung & Rensvold, 1999; Johnson, Meade, & DuVernet, 2009; Steiger, 2002). Specifically, an arbitrary group is selected as the reference group and its factor variance to set to one (for models with a mean structure, the factor mean of the reference group should also be fixed to zero). In addition, the factor loadings (as well as the intercepts for models with mean structures) of the RI(s) are constrained to be equal across all groups. In so doing, there is only one set of estimated coefficients that optimally reproduces the data. In other words, a multiple-group model is identified. Meanwhile, since other parameters are estimated in reference to the standardized factor in the reference group and selected RI(s), the scale of the multiple-group model is set so that the corresponding parameters are comparable across groups. (Cheung & Rensvold, 1999; Johnson et al., 2009; Meade & Wright, 2012). Research has shown when an inappropriate item is chosen to be a RI, severe Type I or Type II errors are expected in testing factorial invariance; that is, truly invariant items could be detected erroneously as noninvariant items and vice versa (Johnson et al., 2009; Yoon & Millsap, 2007). Selection of a RI determines whether the true status of invariance could be detected using the multiple-group CFA method. Methodologists have proposed a number of methods which allow researchers to select the proper RI (see Meade & Wright, 2012; Rivas, Stark, & Chernshenko, 2009; Shi, 2016; Shi, Song, Liao, Terry & Snyder, 2017; Woods, 2009). It is noted that other approaches were proposed, which allow researchers to test invariance without using any specific item as RI (e.g. Raykov, Marcoulides, & Millsap, 2013). For the proposed BSEM approach, we assume researchers could identify the multiple group model by selecting the proper RI.

8 We applied the cutoff suggested by Cheung & Rensvold (2002) as this is the most widely cited criterion.

9 The proposed BSEM method could be extended to test factorial invariance when the number of groups is larger than two by using the Bayesian hierarchical prior (see Verhagen & Fox, 2013). Future studies are expected to explore this extension.

10 We recognize that since the number response categories was small (i.e. less than five), to better account the ordinal nature of the data; ordinal factor analysis models (or polytomous IRT models) should be used (DiStefano & Morgan, 2014; Rhemtulla, Brosseau-Liard & Savalei, 2012). For demonstration purposes, we treated the outcome variables as continuous.