The Impact of Moderate Priors For Bayesian Estimation and Testing of Item Factor Analysis Models When Maximum Likelihood is Unsuitable

Abstract

In psychological research, available data are often insufficient to estimate item factor analysis (IFA) models using traditional estimation methods, such as maximum likelihood (ML) or limited information estimators. Bayesian estimation with common-sense, moderately informative priors can greatly improve efficiency of parameter estimates and stabilize estimation. There are a variety of methods available to evaluate model fit in a Bayesian framework; however, past work investigating Bayesian model fit assessment for IFA models has assumed flat priors, which have no advantage over ML in limited data settings. In this paper, we evaluated the impact of moderately informative priors on ability to detect model misfit for several candidate indices: posterior predictive checks based on the observed score distribution, leave-one-out cross-validation, and widely available information criterion (WAIC). We found that although Bayesian estimation with moderately informative priors is an excellent aid for estimating challenging IFA models, methods for testing model fit in these circumstances are inadequate.

Keywords:

• Bayesian estimation
• Item factor analysis
• small sample
• model fit

Notes

¹ This is because limited information estimation is based on polychoric correlation coefficients, which are highly sensitive to low frequencies in item-by-item contingency tables (Olsson, 1979; Savalei, 2011).

² However, the choice of (1) results in a model that is only locally identified because the signs for the factor loadings could be either positive or negative (Bollen & Bauldry, 2010; Loken, 2005).

³ Though more computationally demanding, ML estimation is superior to limited information approaches when frequencies are sparse because limited information estimators are sensitive to low frequencies in bivariate contingency tables, whereas ML estimation is sensitive to univariate (item-level) sparse frequencies (Wirth & Edwards, 2007).

⁴ This prior for factor loadings requires specifying the sign of the relationship between the latent factor and each indicator, this restriction can often reasonably be made a priori and has computational advantages, but may be relaxed (Bainter, 2017; Fox & Glas, 2003 also constrain loadings to positive values).

⁵ “Moderately informative” is meant to distinguish from so-called “informative” and “uninformative” priors, which refer to peaked and diffuse priors, respectively. The informative versus uninformative labeling is itself problematic, as a flat prior may be counterintuitively influential in some cases and the level of information in a particular prior varies on a case by case basis. For comparison, the default “non-informative” prior in Mplus for item loadings is N(0,∞), but this prior is not recommended for parameters on the logit scale, especially when sample size is not large (Asparouhov & Muthén, 2010). In contrast, an “informative” prior may be centered on an expected value with variance weighted according to the strength of the prior belief or evidence (e.g. Gelman & Hill, 2007, pp. 392–393).

⁶ We checked that 200 replications were sufficient to yield stable results by plotting the running median across converged replications in each condition.

⁷ This restriction is computationally advantageous to avoid sign indeterminacy, but may be relaxed if not reasonable substantively (see Bainter, 2017).

⁸ We encountered a non-zero number of divergent transitions after warm-up for a number of replications, depending on condition (some not reported here). Divergent transitions may be an indication that results are untrustworthy and warns of potential pathologies (Monnahan, Thorson, & Branch, 2017). Decreasing the step size may eliminate divergent transitions, but also increases computational time. We reran a subset of conditions with decreased step size, which almost completely eliminated divergent transitions. We then compared replications that had one or more divergent transitions before the adjustment and zero divergent transitions after the adjustment, and we found no difference. Therefore, we retained our original results with the default step size for this simulation, but for a single replication (and when the population-generating model is not available for comparison), decreasing the step size can help eliminate divergent transitions in these conditions.

⁹ Limited information fit tests such as M² are better, though they require the same sample sizes needed for stable model convergence, and our models did not converge using FlexMIRT software in order to obtain these fit indices.

¹⁰ To further investigate overall parameter estimate efficiency, we also examine root mean square error and the median absolute deviation about the median. These results are consistent with the pattern demonstrated in Figure 5 and with previous literature (Bainter, 2017), so they are omitted here for brevity, but they are available for interested readers in supplemental materials, tables S1 and S2.

¹¹ In the Stan output, this posterior standard deviation is labeled “sd.” The Monte Carlo standard error is also printed and labeled “se_mean”; this value quantifies accuracy in the calculation of the posterior and is not related to uncertainty of the parameters.