An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

Abstract

In this work, we propose a novel approximated collapsed variational Bayes approach to model selection in linear regression. The approximated collapsed variational Bayes algorithm offers improvements over mean field variational Bayes by marginalizing over a subset of parameters and using mean field variational Bayes over the remaining parameters in an analogous fashion to collapsed Gibbs sampling. We have shown that the proposed algorithm, under typical regularity assumptions, (a) includes variables in the true underlying model at an exponential rate in the sample size, or (b) excludes the variables at least at the first order rate in the sample size if the variables are not in the true model. Simulation studies show that the performance of the proposed method is close to that of a particular Markov chain Monte Carlo sampler and a path search based variational Bayes algorithm, but requires an order of magnitude less time. The proposed method is also highly competitive with penalized methods, expectation propagation, stepwise AIC/BIC, BMS, and EMVS under various settings. Supplementary materials for the article are available online.

Keywords:

• Collapsed Gibbs sampling
• Consistency
• Markov chain Monte Carlo

Supplementary Materials

R Codes: The supplemental files for this article include R code and all datasets which can be used to replicate the simulation study included in the article. Please read file README contained in the zip file for more details. (ACVBcode.zip, zip archive)

Appendix: The supplemental files include the Appendix which gives the derivation of (2.4) and the proof of Theorem 1. (ACVBappendix.pdf, pdf file)

Acknowledgments

We thank the reviewers and associate editor for improving the manuscript.

Additional information

Funding

The following sources of funding are gratefully acknowledged: Australian Research Council Discovery Project grant (DP210100521) to JO. Finally, we thank José Miguel Hernández-Lobato for sharing his implementation of EP for linear regression models with spike and slab priors.