Can’t Ridge Regression Perform Variable Selection?

References

• Anzanello, M. J., and Fogliatto, F. S. (2014), “A Review of Recent Variable Selection Methods in Industrial and Chemometrics Applications,” European Journal of Industrial Engineering, 8, 619–645. DOI: 10.1504/EJIE.2014.065731.
   Web of Science ®Google Scholar
• Barcella, W., De Iorio, M., and Baio, G. (2017), “A Comparative Review of Variable Selection Techniques for Covariate Dependent Dirichlet Process Mixture Models,” Canadian Journal of Statistics, 45, 254–273. DOI: 10.1002/cjs.11323.
   Web of Science ®Google Scholar
• Chen, J., and Chen, Z. (2008), “Extended Bayesian Information Criteria for Model Selection With Large Model Spaces,” Biometrika, 95, 759–771. DOI: 10.1093/biomet/asn034.
   Web of Science ®Google Scholar
• Desboulets, L. D. D. (2018), “A Review on Variable Selection in Regression Analysis,” Econometrics, 6, 1–27. DOI: 10.3390/econometrics6040045.
   Web of Science ®Google Scholar
• Fan, J., and Li, R. (2001), “Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties,” Journal of the American Statistical Association, 96, 1348–1360. DOI: 10.1198/016214501753382273.
   Web of Science ®Google Scholar
• Fan, J., and Lv, J. (2010), “A Selective Overview of Variable Selection in High Dimensional Feature Space,” Statistica Sinica, 20, 101–148.
   PubMed Web of Science ®Google Scholar
• Frommlet, F., and Nuel, G. (2016), “An Adaptive Ridge Procedure for L0 Regularization,” PLOS ONE, 11, e0148620. DOI: 10.1371/journal.pone.0148620.
   PubMed Web of Science ®Google Scholar
• Hamidieh, K. (2018), “A Data-Driven Statistical Model for Predicting the Critical Temperature of a Superconductor,” Computational Materials Science, 154, 346–354. DOI: 10.1016/j.commatsci.2018.07.052.
   Web of Science ®Google Scholar
• Hastie, T. (2020), “Ridge Regularizaton: An Essential Concept in Data Science,” Technometrics (this issue), DOI: 10.1080/00401706.2020.1791959.
   Web of Science ®Google Scholar
• Heinze, G., Wallisch, C., and Dunkler, D. (2018), “Variable Selection—A Review and Recommendations for the Practicing Statistician,” Biometrical Journal, 60, 431–449. DOI: 10.1002/bimj.201700067.
   PubMed Web of Science ®Google Scholar
• Hoerl, A. E., and Kennard, R. W. (1970), “Ridge Regression: Biased Estimation for Nonorthogonal Problems,” Technometrics, 12, 55–67. DOI: 10.1080/00401706.1970.10488634.
   Web of Science ®Google Scholar
• Huang, J., Jiao, Y., Liu, Y., and Lu, X. (2018), “A Constructive Approach to L0 Penalized Regression,” Journal of Machine Learning Research, 19, 1–37.
   Web of Science ®Google Scholar
• Joseph, V. R., and Delaney, J. D. (2008), “Analysis of Optimization Experiments,” Journal of Quality Technology, 40, 282–298. DOI: 10.1080/00224065.2008.11917734.
   Web of Science ®Google Scholar
• Kirpich, A., Ainsworth, E. A., Wedow, J. M., Newman, J. R. B., Michailidis, G., and McIntyre, L. M. (2018), “Variable Selection in Omics Data: A Practical Evaluation of Small Sample Sizes,” PLOS ONE, 13, e0197910. DOI: 10.1371/journal.pone.0197910.
   PubMed Web of Science ®Google Scholar
• R Core Team (2018), R: A Language and Environment for Statistical Computing, Vienna, Austria: R Foundation for Statistical Computing.
   Google Scholar
• Shao, J., and Deng, X. (2012), “Estimation in High-Dimensional Linear Models With Deterministic Design Matrices,” The Annals of Statistics, 40, 812–831. DOI: 10.1214/12-AOS982.
   Web of Science ®Google Scholar
• Stefanski, L. A., Wu, Y., and White, K. R. (2014), “Variable Selection in Nonparametric Classification via Measurement Error Model Selection Likelihoods,” Journal of the American Statistical Association, 109, 574–589. DOI: 10.1080/01621459.2013.858630.
   PubMed Web of Science ®Google Scholar
• Talbot, D. (2019), “A Descriptive Review of Variable Selection Methods in Four Epidemiologic Journals: There Is Still Room for Improvement,” European Journal of Epidemiology, 34, 725–730. DOI: 10.1007/s10654-019-00529-y.
   PubMed Web of Science ®Google Scholar
• Tibshirani, R. (1996), “Regression Shrinkage and Selection via the Lasso,” Journal of the Royal Statistical Society, Series B, 58, 267–288. DOI: 10.1111/j.2517-6161.1996.tb02080.x.
   Google Scholar
• Wang, H., Li, R., and Tsai, C.-L. (2007), “Tuning Parameter Selectors for the Smoothly Clipped Absolute Deviation Method,” Biometrika, 94, 553–568. DOI: 10.1093/biomet/asm053.
   PubMed Web of Science ®Google Scholar
• Wipf, D. P., and Nagarajan, S. S. (2008), “A New View of Automatic Relevance Determination,” in Advances in Neural Information Processing Systems (Vol. 20), eds. J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, pp. 1625–1632, Curran Associates, Inc.
   Google Scholar
• Ye, J. (1998), “On Measuring and Correcting the Effects of Data Mining and Model Selection,” Journal of the American Statistical Association, 93, 120–131. DOI: 10.1080/01621459.1998.10474094.
   Web of Science ®Google Scholar
• Yuan, M., and Lin, Y. (2007), “On the Non-Negative Garrotte Estimator,” Journal of the Royal Statistical Society, Series B, 69, 143–161. DOI: 10.1111/j.1467-9868.2007.00581.x.
   Google Scholar
• Zhang, C.-H. (2010), “Nearly Unbiased Variable Selection Under Minimax Concave Penalty,” The Annals of Statistics, 38, 894–942. DOI: 10.1214/09-AOS729.
   Web of Science ®Google Scholar
• Zhang, H. H., and Lu, W. (2007), “Adaptive Lasso for Cox’s Proportional Hazards Model,” Biometrika, 94, 691–703. DOI: 10.1093/biomet/asm037.
   Web of Science ®Google Scholar
• Zou, H. (2006), “The Adaptive Lasso and Its Oracle Properties,” Journal of the American Statistical Association, 101, 1418–1429. DOI: 10.1198/016214506000000735.
   Web of Science ®Google Scholar