Simultaneous Variable and Covariance Selection With the Multivariate Spike-and-Slab LASSO

Abstract

We propose a Bayesian procedure for simultaneous variable and covariance selection using continuous spike-and-slab priors in multivariate linear regression models where q possibly correlated responses are regressed onto p predictors. Rather than relying on a stochastic search through the high-dimensional model space, we develop an ECM algorithm similar to the EMVS procedure of Ročková and George targeting modal estimates of the matrix of regression coefficients and residual precision matrix. Varying the scale of the continuous spike densities facilitates dynamic posterior exploration and allows us to filter out negligible regression coefficients and partial covariances gradually. Our method is seen to substantially outperform regularization competitors on simulated data. We demonstrate our method with a re-examination of data from a recent observational study of the effect of playing high school football on several later-life cognition, psychological, and socio-economic outcomes. An R package, scripts for replicating examples in this article, and results from further simulation studies are provided in the supplementary materials available online.

Keywords:

• Bayesian shrinkage
• EM algorithm
• Gaussian graphical modeling
• Multivariate regression
• Nonconvex optimization

Acknowledgments

We would also like to thank the editor, associate editor, and referees for their very helpful suggestions.

Funding

Additional information

Funding

The authors gratefully acknowledge NSF grant DMS-1406563, a Simons Fellowship from the Isaac Newton Institute at the University of Cambridge, and the James S. Kemper Foundation Faculty Research Fund at the University of Chicago Booth School of Business.