Inference Based on the Best-Fitting Model Can Contribute to the Replication Crisis: Assessing Model Selection Uncertainty Using a Bootstrap Approach

Abstract

Inferences and conclusions drawn from model fitting analyses are commonly based on a single “best fitting” model. If model selection and inference are carried out using the same data model selection, uncertainty is ignored. We illustrate the Type I error inflation that can result from using the same data for model selection and inference, and we then propose a simple bootstrap-based approach to quantify model selection uncertainty in terms of model selection rates. A selection rate can be interpreted as an estimate of the replication probability of a fitted model. The benefits of bootstrapping model selection uncertainty are demonstrated in growth mixture analyses of data from the National Longitudinal Study of Youth, and a 2-group measurement invariance analysis of the Holzinger–Swineford data.

Keywords:

• bootstrap procedure
• model selection uncertainty
• replication crisis

FUNDING

Gitta H. Lubke was supported by DA-018673 (National Institute on Drug Abuse) and FP7-602768 (European Union). The computational work was partially done on clusters acquired through BCS-1229450 (National Science Foundation).

Additional information

Funding

Gitta H. Lubke was supported by DA-018673 (National Institute on Drug Abuse) and FP7-602768 (European Union). The computational work was partially done on clusters acquired through BCS-1229450 (National Science Foundation).