Advantages of Spike and Slab Priors for Detecting Differential Item Functioning Relative to Other Bayesian Regularizing Priors and Frequentist Lasso

ABSTRACT

An important step in scale development and assessment is to evaluate differential item functioning (DIF) across segments of the population. Recent approaches use lasso regularization to simultaneously detect DIF in all items and avoid incorrect anchor item assumptions that incur inflated error rates for classical DIF evaluation methods. Although promising, lasso methods cause underestimated standard errors and incorrect p-values. An alternative is Bayesian regularization that provides empirical standard errors. However, we point out that using empirical criteria such as credible intervals for selecting DIF parameters has limited validity. We argue that using a spike-and-slab prior with an inclusion probability criterion provides more theoretically coherent DIF selection and inference over Bayesian regularizing priors with empirical selection rules or frequentist lasso. We demonstrate this by simulation studies with Multi-group Item Response Theory and Moderated Nonlinear Factor Analysis models. Practical utility of the spike-and-slab prior selection criterion is discussed.

KEYWORDS:

• Differential item functioning
• measurement invariance
• Bayesian regularization
• integrative data analysis

Acknowledgments

We thank David Thissen, Patrick Curran, and two anonymous reviewers for helpful suggestions on this work. Correspondence concerning this article should be addressed to Siyuan Marco Chen, Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, 235 E. Cameron Avenue, Chapel Hill, NC 27599-3270. E-mail: [email protected]

Supplemental data

Supplemental data for this article can be accessed on the publisher’s website

Notes

¹ The baseline latent factor SD is the square root of the baseline variance and $\beta$ in the SD formulation here are 1/2 the $\beta$ in the variance formulation.

² To express a traditional two-group 2-parameter logistic item response theory (2PL IRT) model as a special case of MNLFA, $g(\cdot )$ will be a logit link function, $q=1$, and $x_i$ is a scalar that only takes the value 0 or 1 for two group membership.

³ The above SSP parameter selection and estimation procedures have been referred to in a broader context as stochastic search variable selection (e.g., Bainter et al., 2020) and Bayesian model averaging (Raftery et al., 1997).

⁴ The alternative, to omit zero DIF estimates from accuracy calculations, would leave too few DIF estimates available across replications, which would cause instability in calculating the empirical standard deviation and SE accuracy (often giving accuracy values above 3). Additionally, in frequentist lasso conditions, only estimates considered non-zero at least 5 times across all replications are included in the accuracy calculation to avoid having minuscule empirical parameter SD and outlier SE accuracy incidents.

⁵ The low convergence rates in Bayesian lasso and adaptive lasso may have contributed to downward bias (better accuracy) on their SE accuracy and upward bias on their coverage results, because the replications with less regular data (e.g., correlated DIF covariate effects, weaker effect sizes) are likely to cause divergence in the simulation without more detailed model tuning. These replications may have been more demanding to estimate and, if not left out of the results, added to SE inaccuracies or coverage error.

⁶ We thank an anonymous reviewer for this suggestion.