Improved Estimation of High-dimensional Additive Models Using Subspace Learning

Abstract

Additive models have been widely used as a flexible nonparametric regression method that can overcome the curse of dimensionality. Using sparsity-inducing penalty for variable selection, several methods are developed for fitting additive models when the number of predictors is very large, sometimes even larger than the sample size. However, despite good asymptotic properties, the finite sample performance of these methods may deteriorate considerably when the number of relevant predictors becomes moderately large. This article proposes a new method that reduces the number of unknown functions to be nonparametrically estimated through learning a predictive subspace representation shared by the additive component functions. The subspace learning is integrated with sparsity-inducing penalization in a penalized least squares formulation and an efficient algorithm is developed for computation involving Stiefel matrix manifold optimization and proximal thresholding operators on matrices. Theoretical convergence properties of the algorithm are studied. The proposed method is shown to be competitive with existing methods in simulation studies and a real data example. Supplementary materials for this article are available online.

Keywords:

• Adaptive group Lasso
• Dimensionality reduction
• Low rank approximation
• Polynomial splines
• Sparsity
• Variable selection

Supplementary Materials

A supplemental pdf file:

Details of construction the hyper basis functions, results of three additional simulation setups, and the technical proof of Theorem 1 with some auxiliary lemmas.

Code:

The R code used for experiments in the article.

Acknowledgments

The authors thank the editor, the associate editor, and the anonymous reviewers for their comments that helped significantly improve this work.

Additional information

Funding

This research was supported by Public Computing Cloud, Renmin University of China. The research of Shiyuan He was partially supported by National Natural Science Foundation of China (No.11801561). The research of Kejun He was partially supported by National Natural Science Foundation of China (No.11801560).