From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Correction

This article refers to:

From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Rejoinder

Saharon Rosset^a and Ryan J. Tibshirani^b

^aDepartment of Statistics and Operations Research, Tel Aviv University, Tel Aviv, Israel; ^bDepartment of Statistics and Data Science, Carnegie Mellon University, Pittsburgh, PA

In our article (Rosset and Tibshirani 2020) Theorem 3 is incorrect as stated. Recalling the definition earlier in the article$$V^{+}={\mathbb{E}}_{X,x_0}\text{var}\left({\widehat{f}}_n(x_0)|X,x_0\right)-{\mathbb{E}}_X\text{var}\left({\widehat{f}}_n(x_1)|X\right),$$the original Theorem states:

Assume that $x\sim Q$ is generated as follows: we draw $z\in {\mathbb{R}}^p$, having iid components $z_i\sim F,i=1,\dots ,p$, where F is any distribution with zero mean and unit variance, and then set $x={\Sigma }^{1/2}z$, where $\Sigma \in {\mathbb{R}}^{p\times p}$ is positive definite and ${\Sigma }^{1/2}$ is its symmetric square root. Consider an asymptotic setup where $p/n\to \gamma \in (0,1)$ as $n\to \infty$. Then, for the OLS estimator,$$V^{+}\to \frac{{\sigma }^2{\gamma }^2}{1-\gamma }\hspace{1em}\text{as}\mathrm{}n\to \infty .$$

This theorem is incorrect as stated because V ⁺ is defined in terms of an expectation over X. However, under the assumptions of the theorem, the eigenvalues of the empirical covariance matrix $X^{T}X$ may not be bounded away from zero, and consequently the almost sure convergence of the eigenspectrum used in the proof does not guarantee convergence (or existence) of the expectation. One can generate counterexamples along the lines of the classic examples of almost sure convergence not implying convergence in expectation.

We note that under the same conditions (with the added requirement that the fourth moment of F is finite), Hastie et al. (2021) proved (in their proof of Proposition 2) that$${\mathbb{E}}_{x_0}\text{var}\left({\widehat{f}}_n(x_0)|X,x_0\right)-\text{var}\left({\widehat{f}}_n(x_1)|X\right)\stackrel{\mathrm{a}.\mathrm{s}.X}{\to }\frac{{\sigma }^2{\gamma }^2}{1-\gamma }\hspace{1em}\text{as}\mathrm{}n\to \infty .$$

In words, for almost any sequence of training covariate matrices X generated by this mechanism, the random variable whose expectation is V ⁺ converges to this same fixed limit.

Thus, we conclude that the failure of Theorem 3 as written in the original article is really somewhat of an “edge case”: Although it may not apply to our formal definition of V ⁺, which integrates over the distribution of X, the excess variance result does apply in asymptotic almost sure sense over X. In terms of practical use of the corrections proposed later in the article, we believe it still supports their wide applicability in estimating Random-X prediction error.

Acknowledgments

We thank an anonymous reviewer of a separate article for pointing out the difficulty with the original result.

ORCID

Saharon Rosset http://orcid.org/0000-0002-4458-9545