RaSE: A Variable Screening Framework via Random Subspace Ensembles

Abstract

Variable screening methods have been shown to be effective in dimension reduction under the ultra-high dimensional setting. Most existing screening methods are designed to rank the predictors according to their individual contributions to the response. As a result, variables that are marginally independent but jointly dependent with the response could be missed. In this work, we propose a new framework for variable screening, random subspace ensemble (RaSE), which works by evaluating the quality of random subspaces that may cover multiple predictors. This new screening framework can be naturally combined with any subspace evaluation criterion, which leads to an array of screening methods. The framework is capable to identify signals with no marginal effect or with high-order interaction effects. It is shown to enjoy the sure screening property and rank consistency. We also develop an iterative version of RaSE screening with theoretical support. Extensive simulation studies and real-data analysis show the effectiveness of the new screening framework.

Keywords:

• Ensemble learning
• High-dimensional data
• Random subspace method
• Rank consistency
• Sure screening property
• Variable screening
• Variable selection

Supplementary Materials

All the proofs and additional details of the paper are available in the online supplementary materials.

Notes

Acknowledgments

We are grateful to the editor, the AE, and anonymous reviewers for their insightful comments which have greatly improved the scope and quality of the article.

Notes

1 For more details, please refer to the toy example in Appendix A.3.1.

2 For SIS, ISIS, SIRS, DC-SIS, and HOLP, since the packages implementing them do not provide the option to use multi-cores, we ran them with a single core only.

3 For RaSE methods, sometimes there might be less than $[n/\hspace{0.17em}\text{log}\hspace{0.17em}n]$ variables which have positive selected proportion. In this case, we randomly choose from the remaining variables with 0 selected proportions to have the desired number of selected variables.

Additional information

Funding

Feng’s research is partially supported by NSF CAREER Grant DMS-2013789. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.