An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

References

• Akaike, H. (1973), “Information Theory and an Extension of the Maximum Likelihood Principle,” in Second International Symposium on Information Theory (Tsahkadsor, 1971), pp. 267–281. Akadémiai Kiadó.
   Google Scholar
• Bishop, C. M. (2006), Pattern Recognition and Machine Learning, New York: Springer.
   Google Scholar
• Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003), “Latent Dirichlet Allocation,” Journal of Machine Learning Research, 3, 993–1022.
   Web of Science ®Google Scholar
• Bottolo, L., and Richardson, S. (2010), “Evolutionary Stochastic Search for Bayesian Model Exploration,” Bayesian Analysis, 5, 583–618.
   Web of Science ®Google Scholar
• Carbonetto, P., and Stephens, M. (2011), “Scalable Variational Inference for Bayesian Variable Selection in Regression, and its Accuracy in Genetic Association Studies,” Bayesian Analysis, 6, 1–42.
   PubMedGoogle Scholar
• Carbonetto, P., and Stephens, M. (2012), “Scalable Variational Inference for Bayesian Variable Selection in Regression, and its Accuracy in Genetic Association Studies,” Bayesian Analysis, 7, 73–108.
   Web of Science ®Google Scholar
• Claeskens, G., and Hjort, N. L. (2008), “Model Selection and Model Averaging, Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.
   Google Scholar
• Dresdner, G., Shekhar, S., Pedregosa, F., Locatello, F., and Rätsch, G. (2021), “Boosting Variational Inference with Locally Adaptive Step-Sizes,” Proceedings of the 30th International Joint Conference on Artificial Intelligence, 2337–2343.
   Google Scholar
• Fan, J., and Li, R. (2001), “Variable Selection via on Concave Penalized Likelihood and its Oracle Properties,” Journal of the American Statistical Association, 96, 1348–1360.
   Web of Science ®Google Scholar
• Fan, J., and Lv, J. (2008), “Sure Independence Screening for Ultrahigh Dimensional Feature Space,” Journal of The Royal Statistical Society, Series B, 70, 849–911.
   PubMed Web of Science ®Google Scholar
• Fan, J., and Lv, J. (2010), “A Selective Overview of Variable Selection in High Dimensional Feature Space,” Statistica Sinica, 20, 101–148.
   PubMed Web of Science ®Google Scholar
• Fan, J., and Peng, H. (2004), “Nonconcave Penalized Likelihood with a Diverging Number of Parameters,” The Annals of Statistics, 32, 928–961.
   Web of Science ®Google Scholar
• Feldkircher, M., and Zeugner, S. (2009), “Benchmark Priors Revisited: On Adaptive Shrinkage and the Supermodel Effect in Bayesian Model Averaging,” IMF Working Papers, International Monetary Fund, 9.
   Google Scholar
• Griffiths, T. L., and Steyvers, M. (2004), “Finding Scientific Topics,” Proceedings of the National Academy of Sciences of the United States of America, 101, 5228–5235.
   Google Scholar
• Guo, F., Wang, X., Fan, K., Broderick, T., and Dunson, D. B. (2016), “Boosting Variational Inference,” CoRR, abs/1611.05559.
   Google Scholar
• Han, S., Liao, X., Dunson, D., and Carin, L. (2016), “Variational Gaussian Copula Inference,” in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Volume 51 of Proceedings of Machine Learning Research, pp. 829–838.
   Google Scholar
• Hans, C., Dobra, A., and West, M. (2007), “Shotgun Stochastic Search for “large p” Regression,” Journal of the American Statistical Association, 102, 507–516.
   Web of Science ®Google Scholar
• Hastie, T., and Efron, B. (2013), “lars 1.2. Least Angle Regression, Lasso and Forward Stagewise Regression,” R package. http://cran.r-project.org.
   Google Scholar
• Hastie, T., Tibshirani, R., and Tibshirani, R. J. (2017), “Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso,” arXiv preprint arXiv:1707.08692.
   Google Scholar
• Hernández-Lobato, J. M., Hernández-Lobato, D., and Suárez, A. (2015), “Expectation Propagation in Linear Regression Models with Spike-and-Slab Priors,” Machine Learning, 99, 437–487.
   Web of Science ®Google Scholar
• Huang, X., Wang, J., and Liang, F. (2016), “A Variational Algorithm for Bayesian Variable Selection,” arxiv.
   Google Scholar
• Kingma, D. P., and Welling, M. (2014), “Auto-Encoding Variational Bayes,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, eds. Y. Bengio and Y. LeCun, Conference Track Proceedings.
   Google Scholar
• Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., and Blei, D. M. (2017), “Automatic Differentiation Variational Inference,” Journal of Machine Learning Research, 18, 430–474.
   Web of Science ®Google Scholar
• Li, F., and Zhang, N. R. (2010), “Bayesian Variable Selection in Structured High-Dimensional Covariate Spaces with Applications in Genomics,” Journal of the American Statistical Association, 105, 1202–1214.
   Web of Science ®Google Scholar
• Locatello, F., Khanna, R., Ghosh, J., and Ratsch, G. (2018), “Boosting Variational Inference: An Optimization Perspective,” in Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, Volume 84 of Proceedings of Machine Learning Research, pp. 464–472.
   Google Scholar
• Mallows, C. L. (1973), “Some Comments on Cp,” Technometrics, 15, 661–675.
   Web of Science ®Google Scholar
• Miller, A. C., Foti, N. J., and Adams, R. P. (2017), “Variational Boosting: Iteratively Refining Posterior Approximations,” in Proceedings of the 34th International Conference on Machine Learning, Volume 70 of Proceedings of Machine Learning Research, pp. 2420–2429.
   Google Scholar
• Minka, T. P. (2001), “Expectation Propagation for Approximate Bayesian Inference,” in Proceedings of the International Conference on Uncertainty in Artificial Intelligence (Vol. 17), pp. 362–369.
   Google Scholar
• Minka, T., and Lafferty, J. (2002), “Expectation-Propagation for the Generative Aspect Model,” in Proceedings of the International Conference on Uncertainty in Artificial Intelligence (Vol. 18), pp. 352–359.
   Google Scholar
• Müller, S., and Welsh, A. H. (2010), “On Model Selection Curves,” International Statistical Review, 78, 240–256.
   Web of Science ®Google Scholar
• Murphy, K. P. (2012), Machine Learning: A Probabilistic Perspective, London: The MIT Press.
   Google Scholar
• Ngo, G. H., Eickhoff, S. B., Fox, P. T., and Yeo, B. T. T. (2016), “Collapsed Variational Bayesian Inference of the Author-Topic Model: Application to Large-Scale Coordinate-based Meta-Analysis,” International Workshop on Pattern Recognition in Neuroimaging, pp. 1–4.
   Google Scholar
• Ormerod, J. T., and Wand, M. P. (2010), “Explaining Variational Approximations,” The American Statistician, 64, 140–153.
   Web of Science ®Google Scholar
• Ormerod, J. T., You, C., and Müller, S. (2017), “A Variational Bayes Approach to Variable Selection,” Electronic Journal of Statistics, 11, 3549–3594.
   Web of Science ®Google Scholar
• Piironen, J., and Vehtari, A. (2017), “Comparison of Bayesian Predictive Methods for Model Selection,” Statistics and Computing, 27, 711–735.
   Web of Science ®Google Scholar
• Ranganath, R., Gerrish, S., and Blei, D. (2014), “Black Box Variational Inference,” in Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Volume 33 of Proceedings of Machine Learning Research, pp. 814–822.
   Google Scholar
• Ranganath, R., Tran, D., and Blei, D. (2016), “Hierarchical Variational Models,” in Proceedings of The 33rd International Conference on Machine Learning, Volume 48 of Proceedings of Machine Learning Research, pp. 324–333.
   Google Scholar
• Rezende, D., and Mohamed, S. (2015), “Variational Inference with Normalizing Flows,” in Proceedings of the 32nd International Conference on Machine Learning, Volume 37 of Proceedings of Machine Learning Research, pp. 1530–1538.
   Google Scholar
• Ročková, V., and George, E. I. (2014), “EMVS: The EM Approach to Bayesian Variable Selection,” Journal of the American Statistical Association, 109, 828–846.
   Web of Science ®Google Scholar
• Schwarz, G. (1978), “Estimating the Dimension of a Model,” The Annals of Statistics, 6, 461–464.
   Web of Science ®Google Scholar
• Smith, M. S., Loaiza-Maya, R., and Nott, D. J. (2020), “High-Dimensional Copula Variational Approximation through Transformation,” Journal of Computational and Graphical Statistics, 29, 729–743.
   Web of Science ®Google Scholar
• Stingo, F. C., and Vannucci, M. (2011), “Variable Selection for Discriminant Analysis with Markov Random Field Priors for the Analysis of Microarray Data,” Bioinformatics, 27, 495–501. DOI: 10.1093/bioinformatics/btq690.
   PubMed Web of Science ®Google Scholar
• Sung, J., Ghahramani, Z., and Bang, S. (2008), “Latent-Space Variational Bayes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 2236–2242. DOI: 10.1109/TPAMI.2008.157.
   PubMed Web of Science ®Google Scholar
• Tan, L. (2021), “Use of Model Reparametrization to Improve Variational Bayes,” Journal of the Royal Statistical Society, Series B, 83, 30–57.
   Google Scholar
• Teh, Y. W., Newman, D., and Welling, M. (2006), “A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation,” in Neural Information Processing Systems, pp. 1353–1360.
   Google Scholar
• Tibshirani, R. (1996), “Regression Shrinkage and Selection via the Lasso,” Journal of the Royal Statistical Society, Series B, 58, 267–288.
   Google Scholar
• Titsias, M. K., and Lázaro-Gredilla, M. (2014), “Doubly Stochastic Variational Bayes for Non-conjugate Inference,” in Proceedings of the 31st International Conference on International Conference on Machine Learning – Volume 32, ICML’14, pp. II–1971–II–1980.
   Google Scholar
• Wand, M. P., Ormerod, J. T., Padoan, S. A., and Frühwirth, R. (2011), “Mean Field Variational Bayes for Elaborate Distributions,” Bayesian Analysis, 6, 847–900.
   Web of Science ®Google Scholar
• Wang, H. (2009), “Forward Regression for Ultra-High Dimensional Variable Screening,” Journal of the American Statistical Association, 104, 1512–1524.
   Web of Science ®Google Scholar
• Wang, P., and Blunsom, P. (2013a), “Collapsed Variational Bayesian Inference for Hidden Markov Models,” International Conference on Artificial Intelligence and Statistics, 599–607.
   Google Scholar
• Wang, P., and Blunsom, P. (2013b),“Collapsed Variational Bayesian Inference for PCFGs,” in Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 173–182.
   Google Scholar
• Wang, Y., and Blei, D. M. (2019), “Frequentist Consistency of Variational Bayes,” Journal of the American Statistical Association, 114, 1147–1161.
   Web of Science ®Google Scholar
• Wang, Z., Ma, S., Zappitelli, M., Parikh, C., Wang, C.-Y., and Devarajan, P. (2016), “Penalized Count Data Regression with Application to Hospital Stay after Pediatric Cardiac Surgery,” Statistical Methods in Medical Research, 25, 2685–2703. DOI: 10.1177/0962280214530608.
   PubMed Web of Science ®Google Scholar
• Xu, S. (2007), “An Empirical Bayes Method for Estimating Epistatic Effects of Quantitative Trait Loci,” Biometrics, 63, 513–521. DOI: 10.1111/j.1541-0420.2006.00711.x.
   PubMed Web of Science ®Google Scholar
• Yin, M., and Zhou, M. (2018), “Semi-Implicit Variational Inference,” in Proceedings of the 35th International Conference on Machine Learning, Volume 80 of Proceedings of Machine Learning Research, pp. 5660–5669.
   Google Scholar
• Yu, W., Azizi, L., and Ormerod, J. T. (2020), “Variational Nonparametric Discriminant Analysis,” Computational Statistics & Data Analysis, 142, 106817.
   Web of Science ®Google Scholar
• Yu, W., Ormerod, J. T., and Stewart, M. (2020), “Variational DSiscriminant Analysis with Variable Selection,” Statistics and Computing, 30, 933–951.
   Web of Science ®Google Scholar
• Zeugner, S., and Feldkircher, M. (2015), “Bayesian Model Averaging Employing Fixed and Flexible Priors: The BMS Package for R,” Journal of Statistical Software, 68, 1–37.
   Web of Science ®Google Scholar