Uncertainty Quantification for Modern High-Dimensional Regression via Scalable Bayesian Methods

ABSTRACT

Tremendous progress has been made in the last two decades in the area of high-dimensional regression, especially in the “large p, small n” setting. Such sample starved settings inevitably lead to models which are potentially very unstable and hence quite unreliable. To this end, Bayesian shrinkage methods have generated a lot of recent interest in the modern high-dimensional regression and model selection context. Such methods span the wide spectrum of modern regression approaches and include among others, spike-and-slab priors, the Bayesian lasso, ridge regression, and global-local shrinkage priors such as the Horseshoe prior and the Dirichlet–Laplace prior. These methods naturally facilitate tractable uncertainty quantification and have thus been used extensively across diverse applications. A common unifying feature of these models is that the corresponding priors on the regression coefficients can be expressed as a scale mixture of normals. This property has been leveraged extensively to develop various three-step Gibbs samplers to explore the corresponding intractable posteriors. The convergence of such samplers however is very slow in high dimensions settings, making them disconnected to the very setting that they are intended to work in. To address this challenge, we propose a comprehensive and unifying framework to draw from the same family of posteriors via a class of tractable and scalable two-step blocked Gibbs samplers. We demonstrate that our proposed class of two-step blocked samplers exhibits vastly superior convergence behavior compared to the original three-step sampler in high-dimensional regimes on simulated data as well as data from a variety of applications including gene expression data, infrared spectroscopy data, and socio-economic/law enforcement data. We also provide a detailed theoretical underpinning to the new method by deriving explicit upper bounds for the (geometric) rate of convergence, and by proving that the proposed two-step sampler has superior spectral properties. Supplementary material for this article is available online.

Keywords:

• Bayesian shrinkage
• Geometric ergodicity
• High-dimensional regression
• Hilbert–Schmidt operator
• Scale mixture of normals
• Uncertainty quantification

Additional information

Funding

Bala Rajaratnam and Doug Sparks are supported in part by the U.S. National Science Foundation under grants DMS-CMG-1025465, AGS-1003823, DMS-1106642, and DMS-CAREER-1352656, and by the US Air Force Office of Scientific Research grant award FA9550-13-1-0043. Kshitij Khare is supported in part by NSF grant DMS-15-11945.