Nonlinear Variable Selection via Deep Neural Networks

Abstract

This article presents a general framework for high-dimensional nonlinear variable selection using deep neural networks under the framework of supervised learning. The network architecture includes both a selection layer and approximation layers. The problem can be cast as a sparsity-constrained optimization with a sparse parameter in the selection layer and other parameters in the approximation layers. This problem is challenging due to the sparse constraint and the nonconvex optimization. We propose a novel algorithm, called deep feature selection, to estimate both the sparse parameter and the other parameters. Theoretically, we establish the algorithm convergence and the selection consistency when the objective function has a generalized stable restricted Hessian. This result provides theoretical justifications of our method and generalizes known results for high-dimensional linear variable selection. Simulations and real data analysis are conducted to demonstrate the superior performance of our method. Supplementary materials for this article are available online.

Keywords:

• Deep neural network
• ${\mathcal{l}}_0$
• Nonconvex
• Nonlinear
• Variable selection

Supplementary Materials

Proofs of Theorem 2.1 and Section 4.

Source code: Python and R codes used for experiments in the article.

Notes

¹ Our code is available at https://github.com/cyustcer/Deep-Feature-Selection