Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

References

• Akaike, H. (1973), “Information Theory and an Extension of the Maximum Likelihood Principle,” Second International Symposium on Information Theory, pp. 267–281.
   Google Scholar
• Ball, K. (1993), “The Reverse Isoperimetric Problem for Gaussian Measure,” Discrete & Computational Geometry, 10, 411–420.
   Web of Science ®Google Scholar
• Baranchik, A. (1964), “Multiple Regression and Estimation of the Mean of a Multivariate Normal Distribution,” Technical Report, Stanford University.
   Google Scholar
• Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013), “Valid Post-Selection Inference,” Annals of Statistics, 41, 802–837.
   Web of Science ®Google Scholar
• Bernau, C., Augustin, T., and Boulesteix, A.-L. (2013), “Correcting the Optimal Resampling-Based Error Rate by Estimating the Error Rate of Wrapper Algorithms,” Biometrics, 69, 693–702.
   PubMed Web of Science ®Google Scholar
• Breiman, L. (1992), “The Little Bootstrap and Other Methods for Dimensionality Selection in Regression: X-Fixed Prediction Error,” Journal of the American Statistical Society, 87, 738–754.
   Web of Science ®Google Scholar
• Candes, E. J., Sing-Long, C. M., and Trzasko, J. D. (2013), “Unbiased Risk Estimates for Singular Value Thresholding and Spectral Estimators,” IEEE Transactions on Signal Processing, 61, 4643–4657.
   Web of Science ®Google Scholar
• Cavalier, L., Golubev, Y., Picard, D., and Tsybakov, A. (2002), “Oracle Inequalities for Inverse Problems,” Annals of Statistics, 30, 843–874.
   Web of Science ®Google Scholar
• Chen, X., Lin, Q., and Sen, B. (2015), “On Degrees of Freedom of Projection Estimators with Applications to Multivariate Shape Restricted Regression,” arXiv: 1509.01877.
   Google Scholar
• Donoho, D. L., and Johnstone, I. M. (1994), “Ideal Spatial Adaptation by Wavelet Shrinkage,” Biometrika, 81, 425–455.
   Web of Science ®Google Scholar
• ——— (1995), “Adapting to Unknown Smoothness via Wavelet Shrinkage,” Journal of the American Statistical Association, 90, 1200–1224.
   Web of Science ®Google Scholar
• ——— (1998), “Minimax Estimation via Wavelet Shrinkage,” Annals of Statistics, 26, 879–921.
   Web of Science ®Google Scholar
• Efron, B. (1986), “How Biased is the Apparent Error Rate of a Prediction Rule?” Journal of the American Statistical Association, 81, 461–470.
   Web of Science ®Google Scholar
• ——— (2004), “The Estimation of Prediction Error: Covariance Penalties and Cross-Validation,” Journal of the American Statistical Association, 99, 619–632.
   Web of Science ®Google Scholar
• ——— (2010), Large-scale Simultaneous Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction, New York: Cambridge University Press.
   Google Scholar
• ——— (2014), “Estimation and Accuracy after Model Selection,” Journal of the American Statistical Association, 109, 991–1007.
   PubMed Web of Science ®Google Scholar
• Efron, B., and Hastie, T. (2016), Computer Age Statistical Inference: Algorithms, Inference, and Data Science, New York: Cambridge University Press.
   Google Scholar
• Fithian, W., Sun, D., and Taylor, J. (2014), “Optimal Inference after Model Selection,” arXiv: 1410.2597.
   Google Scholar
• Hoerl, A., and Kennard, R. (1970), “Ridge Regression: Biased Estimation for Nonorthogonal Problems,” Technometrics, 12, 55–67.
   Web of Science ®Google Scholar
• James, W., and Stein, C. (1961), “Estimation with Quadratic Loss,” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1, 361–379.
   Google Scholar
• Janson, L., Fithian, W., and Hastie, T. (2015), “Effective Degrees of Freedom: A Flawed Metaphor,” Biometrika, 102, 479–485.
   PubMed Web of Science ®Google Scholar
• Johnstone, I. M. (1999), “Wavelet Shrinkage for Correlated Data and Inverse Problems: Adaptivity Results,” Statistica Sinica, 9, 51–83.
   Web of Science ®Google Scholar
• ——— (2015), Gaussian Estimation: Sequence and Wavelet Models, New York: Cambridge University Press, draft version.
   Google Scholar
• Klivans, A., O’Donnell, R., and Servedio, R. (2008), “Learning Geometric Concepts via Gaussian Surface Area,” Foundations of Computer Science, 49, 541–550.
   Google Scholar
• Kneip, A. (1994), “Ordered Linear Smoothers,” Annals of Statistics, 22, 835–866.
   Web of Science ®Google Scholar
• Krstajic, D., Buturovic, L., Leahy, D., and Thomas, S. (2014), “Cross-Validation pitfalls when Selecting and Assessing Regression and Classification Models,” Journal of Cheminformatics, 6.
   PubMed Web of Science ®Google Scholar
• Lee, J., Sun, D., Sun, Y., and Taylor, J. (2016), “Exact Post-Selection Inference, with Application to the Lasso,” Annals of Statistics, 44, 907–927.
   Web of Science ®Google Scholar
• Li, K.-C. (1985), “From Stein’s Unbiased Risk Estimates to the Method of Generalized Cross-Validation,” Annals of Statistics, 14, 1352–1377.
   Google Scholar
• ——— (1986), “Asymptotic Optimality of CL and Generalized Cross-Validation in Ridge Regression with Application to Spline Smoothing,” Annals of Statistics, 14, 1101–1112.
   Web of Science ®Google Scholar
• ——— (1987), “Asymptotic Optimality for Cp, CL, Cross-Validation and Generalized Cross-Validation: Discrete Index Set,” Annals of Statistics, 15, 958–975.
   Web of Science ®Google Scholar
• Lockhart, R., Taylor, J., Tibshirani, R. J., and Tibshirani, R. (2014), “A Significance Test for the Lasso,” Annals of Statistics, 42, 413–468.
   PubMed Web of Science ®Google Scholar
• Mallows, C. (1973), “Some Comments on Cp,” Technometrics, 15, 661–675.
   Web of Science ®Google Scholar
• Mikkelsen, F. R., and Hansen, N. R. (2016), “Degrees of Freedom for Piecewise Lipschitz Estimators,” arXiv: 1601.03524.
   Google Scholar
• Nazarov, F. (2003), “On the Maximal Perimeter of a Convex set in Rn with Respect to Gaussian Measure,” Geometric Aspects of Functional Analysis, 1806, 169–187.
   Google Scholar
• Stein, C. (1981), “Estimation of the Mean of a Multivariate Normal Distribution,” Annals of Statistics, 9, 1135–1151.
   Web of Science ®Google Scholar
• Tian Harris, X. (2016), “Prediction Error after Model Selection,” arXiv: 1610.06107.
   Google Scholar
• Tibshirani, R. J. (2015), “Degrees of Freedom and Model Search,” Statistica Sinica, 25, 1265–1296.
   Web of Science ®Google Scholar
• Tibshirani, R. J., and Taylor, J. (2011), “The Solution Path of the Generalized Lasso,” Annals of Statistics, 39, 1335–1371.
   Web of Science ®Google Scholar
• ——— (2012), “Degrees of Freedom in Lasso Problems,” Annals of Statistics, 40, 1198–1232.
   Web of Science ®Google Scholar
• Tibshirani, R. J., Taylor, J., Lockhart, R., and Tibshirani, R. (2016), “Exact Post-Selection Inference for Sequential Regression Procedures,” Journal of the American Statistical Association, 111, 600–620.
   Web of Science ®Google Scholar
• Tibshirani, R. J., and Tibshirani, R. (2009), “A Bias Correction for the Minimum Error Rate in Cross-Validation,” Annals of Applied Statistics, 3, 822–829.
   Web of Science ®Google Scholar
• Tsamardinos, I., Rakhshani, A., and Lagani, V. (2015), “Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization,” International Journal on Artificial Intelligence Tools, 24.
   Web of Science ®Google Scholar
• Ulfarsson, M. O., and Solo, V. (2013a), “Tuning Parameter Selection for Nonnegative Matrix Factorization,” IEEE International Conference on Acoustics, Speech and Signal Processing.
   Google Scholar
• ——— (2013b), “Tuning Parameter Selection for Underdetermined Reduced-Rank Regression,” IEEE Signal Processing Letters, 20, 881–884.
   Web of Science ®Google Scholar
• Varma, S., and Simon, R. (2006), “Bias in Error Estimation When Using Cross-Validation for Model Selection,” BMC Bioinformatics, 7.
   PubMed Web of Science ®Google Scholar
• Xie, X., Kou, S., and Brown, L. (2012), “SURE Estimates for a Heteroscedastic Hierarchical Model,” Journal of the American Statistical Association, 107, 1465–1479.
   PubMed Web of Science ®Google Scholar
• Ye, J. (1998), “On Measuring and Correcting the Effects of Data Mining and Model Selection,” Journal of the American Statistical Society, 93, 120–131.
   Web of Science ®Google Scholar
• Zou, H., Hastie, T., and Tibshirani, R. (2007), “On the ‘Degrees of Freedom’ of the Lasso,” Annals of Statistics, 35, 2173–2192.
   Web of Science ®Google Scholar
• Zou, H., and Yuan, M. (2008), “Regularized Simultaneous Model Selection in Multiple Quantiles Regression,” Computational Statistics and Data Analysis, 52, 5296–5304.
   Web of Science ®Google Scholar