Out-of-Sample R²: Estimation and Inference

Abstract

Out-of-sample prediction is the acid test of predictive models, yet an independent test dataset is often not available for assessment of the prediction error. For this reason, out-of-sample performance is commonly estimated using data splitting algorithms such as cross-validation or the bootstrap. For quantitative outcomes, the ratio of variance explained to total variance can be summarized by the coefficient of determination or in-sample R², which is easy to interpret and to compare across different outcome variables. As opposed to in-sample R², out-of-sample R² has not been well defined and the variability on out-of-sample ${\widehat{R}}^2$ has been largely ignored. Usually only its point estimate is reported, hampering formal comparison of predictability of different outcome variables. Here we explicitly define out-of-sample R² as a comparison of two predictive models, provide an unbiased estimator and exploit recent theoretical advances on uncertainty of data splitting estimates to provide a standard error for ${\widehat{R}}^2$. The performance of the estimators for R² and its standard error are investigated in a simulation study. We demonstrate our new method by constructing confidence intervals and comparing models for prediction of quantitative Brassica napus and Zea mays phenotypes based on gene expression data. Our method is available in the R-package oosse.

KEYWORDS:

• Bootstrap
• Coefficient of determination
• Cross-validation
• Prediction
• Standard error

Supplementary Materials

R-package: The R-code for calculation of out-of-sample ${\widehat{R}}^2$ and its standard error is available in the R-package oosse from CRAN (https://cran.r-project.org/web/packages/oosse). (url)

R-code: R-code for running all simulations and analyses is available at https://github.com/maerelab/Rsquared. (url)

Supplementary material Exhaustive simulation results, proofs, and software versions. (pdf)

Additional information

Funding

The work of SH in the lab of SM was supported through a research collaboration with Inari Agriculture NV funded in part by Flanders Innovation & Entrepreneurship (VLAIO, grant HBC.2019.2814). W.W. received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme. The funders had no role in study design, data analysis, decision to publish, or preparation of the manuscript.