Fully Bayesian logistic regression with hyper-LASSO priors for high-dimensional feature selection

ABSTRACT

Feature selection arises in many areas of modern science. For example, in genomic research, we want to find the genes that can be used to separate tissues of different classes (e.g. cancer and normal). One approach is to fit regression/classification models with certain penalization. In the past decade, hyper-LASSO penalization (priors) have received increasing attention in the literature. However, fully Bayesian methods that use Markov chain Monte Carlo (MCMC) for regression/classification with hyper-LASSO priors are still in lack of development. In this paper, we introduce an MCMC method for learning multinomial logistic regression with hyper-LASSO priors. Our MCMC algorithm uses Hamiltonian Monte Carlo in a restricted Gibbs sampling framework. We have used simulation studies and real data to demonstrate the superior performance of hyper-LASSO priors compared to LASSO, and to investigate the issues of choosing heaviness and scale of hyper-LASSO priors.

KEYWORDS:

• High-dimensional
• feature selection
• non-convex penalties
• horseshoe
• heavy-tailed prior
• hyper-LASSO priors
• MCMC
• Hamiltonian Monte Carlo
• Gibbs sampling
• fully Bayesian

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Longhai Li   http://orcid.org/0000-0002-3074-8584

Additional information

Funding

The research of Longhai Li was supported by the fundings from Natural Sciences and Engineering Research Council of Canada (NSERC), and Canada Foundation for Innovations(CFI). The research of Weixin Yao was supported by NSF grant DMS-1461677.