Bayesian Bootstrap Spike-and-Slab LASSO

Abstract

The impracticality of posterior sampling has prevented the widespread adoption of spike-and-slab priors in high-dimensional applications. To alleviate the computational burden, optimization strategies have been proposed that quickly find local posterior modes. Trading off uncertainty quantification for computational speed, these strategies have enabled spike-and-slab deployments at scales that would be previously unfeasible. We build on one recent development in this strand of work: the Spike-and-Slab LASSO procedure. Instead of optimization, however, we explore multiple avenues for posterior sampling, some traditional and some new. Intrigued by the speed of Spike-and-Slab LASSO mode detection, we explore the possibility of sampling from an approximate posterior by performing MAP optimization on many independently perturbed datasets. To this end, we explore Bayesian bootstrap ideas and introduce a new class of jittered Spike-and-Slab LASSO priors with random shrinkage targets. These priors are a key constituent of the Bayesian Bootstrap Spike-and-Slab LASSO (BB-SSL) method proposed here. BB-SSL turns fast optimization into approximate posterior sampling. Beyond its scalability, we show that BB-SSL has a strong theoretical support. Indeed, we find that the induced pseudo-posteriors contract around the truth at a near-optimal rate in sparse normal-means and in high-dimensional regression. We compare our algorithm to the traditional Stochastic Search Variable Selection (under Laplace priors) as well as many state-of-the-art methods for shrinkage priors. We show, both in simulations and on real data, that our method fares very well in these comparisons, often providing substantial computational gains. Supplementary materials for this article are available online.

Keywords:

• Bayesian bootstrap
• Posterior contraction
• Spike-and-Slab LASSO
• Weighted likelihood bootstrap

Supplementary Materials

Appendix: File “appendix” containing proofs, discussion of connections to NPL, details of computational complexity analysis and additional experimental results mentioned in the article. (pdf file)

Code: File “code” containing R scripts to perform the simulations and experiments described in the article. (zipped file)

Notes

1 Here we are not necessarily assuming that $\left|\right|{\mathit{X}}_j|{|}_2^2=n$ and the above formula is hence, slightly different from Theorem 3.1 of Ročková and George (2018).

2 Note that if $w_1,w_2,\dots ,w_n\stackrel{\text{iid}}{\sim }\hspace{0.17em}\text{exp}\hspace{0.17em}(1)$, then $\frac{w_1}{\sum w_i},\frac{w_2}{\sum w_i},\dots ,\frac{w_n}{\sum w_i}\sim \text{Dir}(1,1,\dots ,1)$ which brings us back to the uniform Dirichlet distribution.

3 In fact, under the uniform Dirichlet distribution, the marginal distribution becomes $w_i\sim n\times \text{Beta}(1,n-1)$. Since $n\times \text{Beta}(1,n-1)\stackrel{\mathrm{d}}{\to }\text{Gamma}(1,1)$, the distribution of $-\frac{1}{w_i}$ converges to Inverse-Gamma(1,1) which exhibits a skewed shape, which is in sharp contrast to the symmetric Gaussian distribution of the true posterior.

4 In order for ${\eta }^{*}$ to be a bounded real number smaller than 1, we would need ${\lambda }_1/\left|\right|\mathit{X}\left|\right|\to \infty$. For example when $p\asymp n$, for random matrix where each element is generated independently by Gaussian distribution, we have $\left|\right|\mathit{X}\left|\right|=O_p(\sqrt{n}+\sqrt{p})$ (Vivo, Majumdar, and Bohigas 2007). So in order for such a sequence C_n (s.t. $C_n\to \infty$) to exist, we need ${\lambda }_1/(\sqrt{n}+\sqrt{p})\to \infty$. We can choose ${\lambda }_1=(\sqrt{n}+\sqrt{p})\sqrt{\hspace{0.17em}\text{log}\hspace{0.17em}p}$ and $C_1=\hspace{0.17em}\text{log}\hspace{0.17em}\frac{{\lambda }_1}{\left|\right|\mathit{X}\left|\right|}$ under such settings.

Additional information

Funding

The authors gratefully acknowledge the support from the James S. Kemper Faculty Fund at the Booth School of Business and the National Science Foundation (grant no. NSF DMS-1944740).