Versatile Descent Algorithms for Group Regularization and Variable Selection in Generalized Linear Models

Abstract

This article proposes an adaptively bounded gradient descent (ABGD) algorithm for group elastic net penalized regression. Unlike previously proposed algorithms, the proposed algorithm adaptively bounds the Fisher information matrix, which results in a flexible and stable computational framework. In particular, the proposed algorithm (i) does not require orthogonalization of the predictors, and (ii) can be easily applied to any combination of exponential family response distribution and link function. The proposed algorithm is implemented in the grpnet R package (available from CRAN), which implements the approach for common response distributions (Gaussian, binomial, and Poisson), as well as several response distributions not previously considered in the group penalization literature (i.e., multinomial, negative binomial, gamma, and inverse Gaussian). Simulated and real data examples demonstrate that the proposed algorithm is as or more efficient than existing methods for Gaussian, binomial, and Poisson distributions. Furthermore, using two genomic examples, I demonstrate how the proposed algorithm can be applied to high-dimensional multinomial regression problems with grouped predictors. R code to reproduce the results is included as supplementary materials.

KEYWORDS:

• Elastic net
• Lasso
• MCP
• Multinomial regression
• SCAD

Disclosure Statement

No potential conflict of interest was reported by the author.

Notes

1 Non-unity weights can be easily incorporated by replacing ${\mathbf{X}}_k$ with ${\mathbf{W}}^{1/2}{\mathbf{X}}_k$ in these formulas.

2 These authors refer to the orthogonalized solution as the “standardized” solution in their paper.

3 This is sometimes referred to as the “symmetric” parameterization (Friedman, Hastie, and Tibshirani 2010), which differs from the typical multi-logit transformation that uses one response category as a reference.

Additional information

Funding

This work was partially supported by a sabbatical from the University of Minnesota and the following NIH grants: R01EY030890, R01MH115046, and U01DA046413.