Detecting Prior-Data Disagreement in Bayesian Structural Equation Modeling

Abstract

The choice of prior specification plays a vital role in any Bayesian analysis. Prior-data disagreement occurs when the researcher’s prior knowledge is not in agreement with the evidence provided by the data. We examined the ability of the Data Agreement Criterion (DAC) and Bayes Factor (BF) to detect prior-data disagreement in SEM through a simulation study. The design included four sample size levels and 49 prior specifications. Findings suggested that prior-data disagreement still affects posterior estimates when samples are relatively large. Further, comparing multiple prior specifications sheds more light on the presence of prior-data disagreement than assessing a single prior specification. The DAC was easy to implement but cannot assess interactions between priors within one specification. The BF takes these interactions into account, resulting in a global assessment of prior-data disagreement. However, the BF became challenging to compute with larger sample sizes. Implications for applied researchers are discussed.

Keywords:

• Bayesian estimation
• prior-data disagreement
• prior distributions
• structural equation models

Disclosure statement

We have no known conflict of interest to disclose.

Notes

1 It should be noted that prior-data disagreement would not pose a problem to a true subjective Bayesian, who would simply update their prior belief. However, prior-data disagreement can cause computational issues for the pragmatic evidence-based subjective Bayesian, which is why it is the focus of the current investigation.

2 Other distance measures can also be used, however the KL divergence appears to perform best (Lek & van de Schoot, 2019).

3 As the marginal likelihood is likely support a diffuse prior over any informative prior, it is likely that several prior specifications need to be compared to the diffuse benchmark prior to gain insight into the relative ranking of the researcher-specified priors.

4 This prior was selected after a pilot study demonstrated that this prior had minimal impact on the posterior distribution but did not result in convergence issues. Other priors investigated were: N(0, 10), N(0, 100), and U(−100, 100).

5 These values represent data dependent priors (DDP; McNeish, 2016), which we selected by first generating 100 samples of n = 50 from the population model. Next, we used these 100 samples to estimate the population model with maximum likelihood estimation (MLE). we averaged the standard error estimates of the intercept mean and slope mean across the 100 samples. we used those values to specify the standard deviation hyperparameters of the moderately informative priors. we used the smallest sample size conditions of the main simulation design to find the standard error estimate that represented the highest level of uncertainty. The same values were used across all sample size levels in the main simulation. In applied research, DDPs are somewhat controversial, as the researcher technically double-dips by using their data to specify the priors that are subsequently used to analyze their data. However, they can aide in model estimation under certain circumstances (e.g., McNeish, 2016).