Tuning Variable Selection Procedures by Adding Noise

Abstract

Many variable selection methods for linear regression depend critically on tuning parameters that control the performance of the method, for example, “entry” and “stay” significance levels in forward and backward selection. However, most methods do not adapt the tuning parameters to particular datasets. We propose a general strategy for adapting variable selection tuning parameters that effectively estimates the tuning parameters so that the selection method avoids overfitting and underfitting. The strategy is based on the principle that overfitting and underfitting can be directly observed in estimates of the error variance after adding controlled amounts of additional independent noise to the response variable, then running a variable selection method. It is related to the simulation technique SIMEX found in the measurement error literature. We focus on forward selection because of its simplicity and ability to handle large numbers of explanatory variables. Monte Carlo studies show that the new method compares favorably with established methods.

KEY WORDS :

• Akaike information criterion
• Bayes information criterion
• Forward selection
• Mallows C_p
• Model selection
• Regression
• SIMEX