Automatic Feature Selection via Weighted Kernels and Regularization

Abstract

Selecting important features in nonlinear kernel spaces is a difficult challenge in both classification and regression problems. This article proposes to achieve feature selection by optimizing a simple criterion: a feature-regularized loss function. Features within the kernel are weighted, and a lasso penalty is placed on these weights to encourage sparsity. This feature-regularized loss function is minimized by estimating the weights in conjunction with the coefficients of the original classification or regression problem, thereby automatically procuring a subset of important features. The algorithm, KerNel Iterative Feature Extraction (KNIFE), is applicable to a wide variety of kernels and high-dimensional kernel problems. In addition, a modification of KNIFE gives a computationally attractive method for graphically depicting nonlinear relationships between features by estimating their feature weights over a range of regularization parameters. The utility of KNIFE in selecting features through simulations and examples for both kernel regression and support vector machines is demonstrated. Feature path realizations also give graphical representations of important features and the nonlinear relationships among variables. Supplementary materials with computer code and an appendix on convergence analysis are available online.

Key Words:

• Reproducing kernel Hilbert space
• Kernel ridge regression
• Lasso
• Nonlinear regression
• Regularization paths
• Support vector machine

ACKNOWLEDGMENTS

The author is grateful to Robert Tibshirani for the helpful suggestions and advice in developing and testing this method. The author thanks Stephen Boyd for suggesting kernel linearization, Rahul Mazumder and Holger Hoefling for discussions on algorithm convergence, and Trevor Hastie for the helpful suggestions. The author also thanks three anonymous reviewers, the editor, and associate editor for suggestions that led to several improvements in this article.