A Criterion for Optimal Predictive Model Selection

Abstract

AIC is known in the literature as the best for estimation of the regression function, BIC is the best for identification—i.e., when there is no model uncertainty, BIC consistently identifies the true model —but both of them do not necessarily have the optimal out-of-sample predictive ability. In the presence of model uncertainty, Bayesian model averaging (BMA) is the gold standard for making out-of-sample predictions and inferences, but it does not select a model because it averages over a given collection of models. Therefore, BMA does not have interpretability—a property that is often desirable for a statistical procedure. We propose a procedure for model selection that trades off between the prediction accuracy and the interpretability. The procedure seeks a model—thus, it has the interpretability—that has a predictive distribution closest to that of BMA, thus the selected model has better predictive performance than any other model. The suggested procedure can be easily and efficiently implemented by using a Markov Chain Monte Carlo algorithm. We present a number of examples in a linear regression framework for both real data and simulated data. It is shown in these examples that our procedure selects models with optimal predictive performance. In the extreme case where there is no model uncertainty, our procedure selects the same model as BIC does.

Keywords:

• Bayesian model averaging
• Markov chain Monte Carlo
• Model selection
• Optimal predictive model

Mathematics Subject Classification:

• Primary 62J05
• Secondary 62F15

Acknowledgment

The author would like to thank an anonymous referee for his/her careful reading and helpful comments. The author is very grateful to Marcus Hutter for sharing his ideas and discussion.

Notes

MUI = .71 shows that there is high model uncertainty.

MUI = .61 shows that there is moderate model uncertainty.

MUI = .56 shows that there is moderate model uncertainty.

MUI = .11: there is no model uncertainty.

MUI = .68: there is model uncertainty.