Detecting Model Misspecification in Bayesian Piecewise Growth Models

Abstract

Bayesian estimation has become increasingly more popular with piecewise growth models because it can aid in accurately modeling nonlinear change over time. Recently, new Bayesian approximate fit indices (BRMSEA, BCFI, and BTLI) have been introduced as tools for detecting model (mis)fit. We compare these indices to the posterior predictive p-value (PPP), and also examine the Bayesian information criterion (BIC) and the deviance information criterion (DIC), to identify optimal methods for detecting model misspecification in piecewise growth models. Findings indicated that the Bayesian approximate fit indices are not as reliable as the PPP for detecting misspecification. However, these indices appear to be viable model selection tools rather than measures of fit. We conclude with recommendations regarding when researchers should be using each of the indices in practice.

Keywords:

• Bayesian fit indices
• model fit
• model misspecification
• model selection
• piecewise growth models

Notes

1 For the interested reader, there are additional methods for specifying prior distributions on the parameters included in this investigation. For example, separation strategy priors can be used for the latent factor covariance matrix (Depaoli, 2021; Liu et al., 2016), and there are a variety of prior settings that can be used for variance parameters (Gelman, 2006). To keep the simulation conditions manageable, and focus on the priors most commonly modified in applied research, we opted to only examine different settings for the latent growth factor means.

2 It is important to note that sometimes the DIC is considered to only be partially Bayesian because it does not use the entire posterior. Rather, the DIC uses the mean of the simulated values from $D(\mathit{\theta }).$ As a result, many other indices have been developed within the Bayesian estimation framework. However, we restrict our investigation to these indices because they are currently the most widely used. For more information on the shortcomings of the DIC, see Spiegelhalter et al. (2014). In addition, the DIC is computed differently in popular Bayesian software packages (Merkle et al., 2019). For example, Mplus and the R blavaan package compute the marginal DIC, in which the likelihood component is integrated over the latent variables. Other software packages, such as BUGS and JAGS, use the conditional DIC, in which the likelihood is conditional on the latent variables. The magnitude of the two types of DICs varies across models and may not favor the same one. In the current study, we focus on the marginal DIC, as it has been implemented in Mplus and has been recommended in hierarchical Bayesian models for its ability to evaluate a model’s generalizability beyond the observed individuals (Merkle et al., 2019).

3 Although we implemented a 90% CI here, it is important to note that other interval widths could have also been implemented. It is possible that substantive conclusions would differ with the implementation of a different interval width (e.g., more inconclusive decisions may be made if wider widths are used). However, we selected 90% to be consistent with the defaults in Mplus. We felt that this would be the most informative setting because it maintains consistency with how the methods will likely be applied in the literature.

4 For full results, please see the OSF page for this project: https://osf.io/myrds/

5 For the interested reader, we have added an additional plot in the online supplementary material. This plot showcases how the PPP and the DIC align with respect to model evaluation. Indeed, the more misspecified models (corresponding to lower PPP values) were selected less frequently by the DIC as compared to the true model (which corresponded to a higher PPP overall).