Cross-Validation: What Does It Estimate and How Well Does It Do It?

References

• Akaike, H. (1974), “A New Look at the Statistical Model Identification,” IEEE Transactions on Automatic Control, 19, 716–723. DOI: 10.1109/TAC.1974.1100705.
   Web of Science ®Google Scholar
• Allen, D. (1974), “The Relationship between Variable Selection and Data Augmentation and a Method of Prediction,” Technometrics, 16, 125–127. DOI: 10.1080/00401706.1974.10489157.
   Web of Science ®Google Scholar
• Austern, M., and Zhou, W. (2020), “Asymptotics of Cross-Validation,” arXiv preprint. arXiv:2001.11111.
   Google Scholar
• Bayle, P., Bayle, A., Mackey, L., and Janson, L. (2020), “Cross-Validation Confidence Intervals for Test Error,” in Conference on Neural Information Processing Systems.
   Google Scholar
• Bengio, Y., and Grandvalet, Y. (2004), “No Unbiased Estimator of the Variance of k-fold Cross-Validation,” Journal of Machine Learning Research, 5, 1089–1105.
   Web of Science ®Google Scholar
• Benkeser, D., Petersen, M., and van der Laan, M. J. (2020), “Improved Small-Sample Estimation of Nonlinear Cross-Validated Prediction Metrics,” Journal of the American Statistical Association, 115, 1917–1932. DOI: 10.1080/01621459.2019.1668794.
   PubMed Web of Science ®Google Scholar
• Blum, A., Kalai, A. T., and Langford, J. (1999), “Beating the Hold-Out: Bounds for k-fold and Progressive Cross-Validation,” in Proceedings of the Twelfth Annual Conference on Learning Theory.
   Google Scholar
• Celisse, A., and Guedj, B. (2016), “Stability Revisited: New Generalisation Bounds for the Leave-One-Out,” arXiv preprint. arXiv:1608.06412.
   Google Scholar
• Dietterich, T. G. (1998), “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms,” Neural Computation, 10, 1895–1923. DOI: 10.1162/089976698300017197.
   PubMed Web of Science ®Google Scholar
• Dua, D., and Graff, C. (2017), “UCI Machine Learning Repository.”
   Google Scholar
• Dudoit, S., and van der Laan, M. J. (2005), “Asymptotics of Cross-Validated Risk Estimation in Estimator Selection and Performance Assessment,” Statistical Methodology, 2, 131–154. DOI: 10.1016/j.stamet.2005.02.003.
   Google Scholar
• Efron, B. (1983), “Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation,” Journal of the American Statistical Association, 78, 316–331. DOI: 10.1080/01621459.1983.10477973.
   Web of Science ®Google Scholar
• Efron, B. (1986), “How Biased is the Apparent Error Rate of a Prediction Rule?” Journal of the American Statistical Association, 81, 461–470.
   Web of Science ®Google Scholar
• Efron, B. (2004), “The Estimation of Prediction Error,” Journal of the American Statistical Association, 99, 619–632.
   Web of Science ®Google Scholar
• Efron, B., and Tibshirani, R. (1997), “Improvements on Cross-Validation: The.632+ Bootstrap Method,” Journal of the American Statistical Association, 92, 548–560. DOI: 10.2307/2965703.
   Web of Science ®Google Scholar
• Efron, B. (1993), An Introduction to the Bootstrap, Boca Raton, FL: Chapman & Hall/CRC.
   Google Scholar
• Geisser, S. (1975), “The Predictive Sample Reuse Method with Applications,” Journal of the American Statistical Association, 70, 320–328. DOI: 10.1080/01621459.1975.10479865.
   Web of Science ®Google Scholar
• Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2019), “Surprises in High-Dimensional Ridgeless Least Squares Interpolation,” arXiv preprint. arXiv:1903.08560.
   Google Scholar
• Hastie, T., Tibshirani, R., and Friedman, J. (2009), The Elements of Statistical Learning (2nd ed.), New York: Springer.
   Google Scholar
• Kale, S., Kumar, R., and Vassilvitskii, S. (2011), “Cross-Validation and Mean-Square Stability,” in Proceedings of Innovations in Computer Science.
   Google Scholar
• Kumar, R., Lokshtanov, D., Vassilvitskii, S., and Vattani, A. (2013), “Near-Optimal Bounds for Cross-Validation via Loss Stability,” in Proceedings of the 30th International Conference on Machine Learning.
   Google Scholar
• LeDell, E., Petersen, M., and van der Laan, M. (2015), “Computationally Efficient Confidence Intervals for Cross-Validated Area under the ROC Curve Estimates,” Electronic Journal of Statistics, 9, 1583–1607. DOI: 10.1214/15-EJS1035.
   PubMed Web of Science ®Google Scholar
• Liu, S., and Dobriban, E. (2020), “Ridge Regression: Structure, Cross-Validation, and Sketching,” in International Conference on Learning Representations.
   Google Scholar
• Mallows, C. L. (1973), “Some Comments on Cp,” Technometrics, 15,661–675. DOI: 10.2307/1267380.
   Web of Science ®Google Scholar
• Markatou, M., Tian, H., Biswas, S., and Hripcsak, G. (2005), “Analysis of Variance of Cross-Validation Estimators of the Generalization Error,” Journal of Machine Learning Research, 6, 1127–1168.
   Web of Science ®Google Scholar
• Nadeau, C., and Bengio, Y. (2003), “Inference for the Generalization Error,” Machine Learning, 52, 239–281. DOI: 10.1023/A:1024068626366.
   Web of Science ®Google Scholar
• Reeves, G., Xu, J., and Zadik, I. (2019), “The All-or-Nothing Phenomenon in Sparse Linear Regression,” in Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, eds. A. Beygelzimer and D. Hsu, pp. 2652–2663.
   Google Scholar
• Rosset, S., and Tibshirani, R. J. (2020), “From Fixed-x to Random-x Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation,” Journal of the American Statistical Association, 115, 138–151. DOI: 10.1080/01621459.2018.1424632.
   Web of Science ®Google Scholar
• Schwarz, G. (1978), “Estimating the Dimension of a Model,” The Annals of Statistics, 6, 461–464. DOI: 10.1214/aos/1176344136.
   Web of Science ®Google Scholar
• Shao, J. (1993), “Linear Model Selection by Cross-Validation,” Journal of the American Statistical Association, 88, 486–494. DOI: 10.1080/01621459.1993.10476299.
   Web of Science ®Google Scholar
• Stein, C. M. (1981), “Estimation of the Mean of a Multivariate Normal Distribution,” The Annals of Statistics, 9, 1135–1151. DOI: 10.1214/aos/1176345632.
   Web of Science ®Google Scholar
• Stoica, P., Eykhoff, P., Janssen, P., and Soderstrom, T. (1986), “Model-Structure Selection by Cross-Validation,” International Journal of Control, 43, 1841–1878. DOI: 10.1080/00207178608933575.
   Web of Science ®Google Scholar
• Stone, M. (1977), “Cross-Validatory Choice and Assessment of Statistical Predictions,” Journal of the Royal Statistical Society, Series B, 36, 111–147. DOI: 10.1111/j.2517-6161.1974.tb00994.x.
   Google Scholar
• Wager, S. (2020), “Cross-Validation, Risk Estimation, and Model Selection: Comment on a Paper by Rosset and Tibshirani,” Journal of the American Statistical Association, 115, 157–160. DOI: 10.1080/01621459.2020.1727235.
   Web of Science ®Google Scholar
• Xu, Q.-S., and Liang, Y.-Z. (2001), “Monte Carlo Cross Validation,” Chemometrics and Intelligent Laboratory Systems, 56, 1–11. DOI: 10.1016/S0169-7439(00)00122-2.
   Web of Science ®Google Scholar
• Yang, Y. (2007), “Consistency of Cross Validation for Comparing Regression Procedures,” The Annals of Statistics, 35, 2450–2473. DOI: 10.1214/009053607000000514.
   Web of Science ®Google Scholar
• Yousef, W. A. (2020), “A Leisurely Look at Versions and Variants of the Cross Validation Estimator,” arXiv preprint. arXiv:1907.13413.
   Google Scholar
• Zhang, P. (1993), “Model Selection Via Multifold Cross Validation,” The Annals of Statistics, 21, 299–313. DOI: 10.1214/aos/1176349027.
   Web of Science ®Google Scholar
• Zhang, P. (1995), “Assessing Prediction Error in Non-parametric Regression,” Scandinavian Journal of Statistics, 22, 83–94.
   Web of Science ®Google Scholar