The SEM Reliability Paradox in a Bayesian Framework

Abstract

Within the frequentist structural equation modeling (SEM) framework, adjudicating model quality through measures of fit has been an active area of methodological research. Complicating this conversation is research revealing that a higher quality measurement portion of a SEM can result in poorer estimates of overall model fit than lower quality measurement models, given the same structural misspecifications. Through population analysis and Monte Carlo simulation, we extend the earlier research to recently developed Bayesian SEM measures of fit to evaluate whether these indices are susceptible to the same reliability paradox, in the context of using both uninformative and informative priors. Our results show that the reliability paradox occurs for RMSEA, and to some extent, gamma-hat and PPP (measures of absolute fit); but not CFI or TLI (measures of relative fit), across Bayesian (MCMC) and frequentist (maximum likelihood) SEM frameworks alike. Taken together, these findings indicate that the behavior of these newly adapted Bayesian fit indices map closely to their frequentist analogs. Implications for their utility in identifying incorrectly specified models are discussed.

Keywords:

• Bayesian
• fit indices
• goodness-of-fit
• latent variable models
• model fit
• structural equation modeling

Acknowledgements

We would like to acknowledge Research Computing at The University of Virginia for providing computational resources and technical support that have contributed to the results reported within this publication. URL: https://rc.virginia.edu. We are also grateful to the editor and anonymous reviewers for their detailed and constructive feedback.

Notes

1 Although Hoofs et al. (2018) first introduced a Bayesian form of RMSEA, the RMSEA proposed by Garnier-Villarreal and Jorgensen (2020) as shown in (5) minimizes the influence of sample size on the index.

2 Other R blavaan goodness-of-fit results are provided in the supplemental materials, which showed the same results pattern as those from Mplus.

3 Mplus invokes an automatic convergence criterion based on the potential scale reduction (PSR; Brooks & Gelman, 1998) that is monitored at every 100th iteration (Asparouhov & Muthen, 2010) such that all model parameters must reach PSR values of < 1.1 for iterations to stop. We kept this setting in the current study.

4 We added code in blavaan’s model fitting process such that the burn-in was iteratively increased by 1,000 samples until all model parameters reached PSRs ≤ 1.05, similar to Garnier-Villarreal & Jorgensen 2020).

5 The default non-informative prior for loadings in R blavaan (Merkle et al., 2021) are based on Stan defaults, unless otherwise specified, and are N(0,10)[sd]. Mplus’ default non-informative priors for loadings are wider: N(0, 10¹⁰).

6 In all our Bayesian analyses, since factors were specified as standardized, their covariances were treated as correlations with priors that were non-informative. B(1,1) is the recommended non-informative prior for correlations in blavaan, which can be translated to U(-1,1) and is similar to Mplus’ default factor correlation matrix prior of IW(0,-p-1) (Asparouhov & Muthén, 2021). We note that we could not use the IW as a prior for our simulations because IS1 and IS2 analyses place constraints on at least one of the correlations in the psi matrix.

7 In all our Bayesian analyses, we used a non-informative prior of IG(3,1) for indicator residual variances to closely approximate blavaan’s default non-informative residual variance prior G(1,.5). The IG and G distributions are positively skewed and cannot be negative. The IG prior we used in Mplus specifies a distribution with a mode of 0.25, mean of 0.50, and standard deviation of 0.50. The G prior in blavaan specifies a distribution with a mode of 0, a mean of 0.50, and a standard deviation of 0.50.

8 The default non-informative prior for loadings in R blavaan (Merkle et al., 2021) are based on Stan defaults, unless otherwise specified, and are N(0,10)[sd]. Mplus’ default non-informative priors for loadings are similar but wider: N(0, 10¹⁰).