Assessing Model Selection Uncertainty Using a Bootstrap Approach: An Update

Abstract

Model comparisons in the behavioral sciences often aim at selecting the model that best describes the structure in the population. Model selection is usually based on fit indexes such as Akaike’s information criterion (AIC) or Bayesian information criterion (BIC), and inference is done based on the selected best-fitting model. This practice does not account for the possibility that due to sampling variability, a different model might be selected as the preferred model in a new sample from the same population. A previous study illustrated a bootstrap approach to gauge this model selection uncertainty using 2 empirical examples. This study consists of a series of simulations to assess the utility of the proposed bootstrap approach in multigroup and mixture model comparisons. These simulations show that bootstrap selection rates can provide additional information over and above simply relying on the size of AIC and BIC differences in a given sample.

Keywords:

• AIC
• BIC
• bootstrap model selection rate
• model comparison

FUNDING

This work was supported by a grant from the National Institute on Drug Abuse (NIDA), DA-018673. The content is solely the responsibility of the authors and does not represent the official views of NIDA or the National Institutes of Health.

Notes

¹ Note that the discrepancy due to approximation is not necessarily decreasing with model complexity if models are not nested. For instance, a misspecified four-factor latent path model with 72 parameters had a much larger chi-square test of model fit.

Additional information

Funding

This work was supported by a grant from the National Institute on Drug Abuse (NIDA), DA-018673. The content is solely the responsibility of the authors and does not represent the official views of NIDA or the National Institutes of Health.