A novel bagging approach for variable ranking and selection via a mixed importance measure

ABSTRACT

At present, ensemble learning has exhibited its great power in stabilizing and enhancing the performance of some traditional variable selection methods such as lasso and genetic algorithm. In this paper, a novel bagging ensemble method called BSSW is developed to implement variable ranking and selection in linear regression models. Its main idea is to execute stepwise search algorithm on multiple bootstrap samples. In each trial, a mixed importance measure is assigned to each variable according to the order that it is selected into final model as well as the improvement of model fitting resulted from its inclusion. Based on the importance measure averaged across some bootstrapping trials, all candidate variables are ranked and then decided to be important or not. To extend the scope of application, BSSW is extended to the situation of generalized linear models. Experiments carried out with some simulated and real data indicate that BSSW achieves better performance in most studied cases when compared with several other existing methods.

KEYWORDS:

• Variable selection
• variable ranking
• bagging
• stepwise search algorithm
• ensemble learning
• selection accuracy
• variable selection ensemble

AMS SUBJECT CLASSIFICATIONS:

• 62J05
• 68T05
• 62F07

Acknowledgments

The authors are very grateful to the referees and the editor for their valuable comments which greatly helped to improve the paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research was supported by the National Natural Science Foundation of China [No. 11671317, 61572393, 11501049], the Program for International Science and Technology Cooperation Projects of China [No. 2015DFA81780], Shaanxi Science and Technology Co-ordination Innovation Project [No. 2014SZS20-K04], and the Open Research Fund Program of Shaanxi Key Laboratory of Non-Traditional Machining.