Cross-Validated Loss-based Covariance Matrix Estimator Selection in High Dimensions

Abstract

The covariance matrix plays a fundamental role in many modern exploratory and inferential statistical procedures, including dimensionality reduction, hypothesis testing, and regression. In low-dimensional regimes, where the number of observations far exceeds the number of variables, the optimality of the sample covariance matrix as an estimator of this parameter is well-established. High-dimensional regimes do not admit such a convenience. Thus, a variety of estimators have been derived to overcome the shortcomings of the canonical estimator in such settings. Yet, selecting an optimal estimator from among the plethora available remains an open challenge. Using the framework of cross-validated loss-based estimation, we develop the theoretical underpinnings of just such an estimator selection procedure. We propose a general class of loss functions for covariance matrix estimation and establish accompanying finite-sample risk bounds and conditions for the asymptotic optimality of the cross-validation selector. In simulation studies, we demonstrate the optimality of our proposed selector in moderate sample sizes and across diverse data-generating processes. The practical benefits of our procedure are highlighted in a dimensionality reduction application to single-cell transcriptome sequencing data. Supplementary materials for this article are available online.

Keywords:

• Covariance matrix estimation
• Cross-validation
• Dimension reduction
• High-dimensional statistics
• Loss-based estimation

Supplementary Materials

Online Supplement: All proofs and additional simulation results are included in the online supplement. A brief review of popular covariance matrix estimators is also included.

cvCovEst R package: The cross-validated covariance matrix estimator selection procedure is implemented in cvCovEst (Boileau et al. 2021), an open-source R package (R Core Team 2021). This package is available via the Comprehensive R Archive Network (CRAN) at https://CRAN.R-project.org/package=cvCovEst. Internally, the cvCovEst package relies upon the generalized CV framework of the origami R package (Coyle and Hejazi 2018).

Simulation and Analysis Code: Results in Sections 4 and 5 were produced using version 0.1.3 of the cvCovEst R package. The code and data used to produce these analyses are publicly available on GitHub at https://github.com/PhilBoileau/pub_cvCovEst.

Disclosure Statement

The authors report there are no competing interests to declare.

Funding

Note

¹Anecdotally, one cannot help but find themself reminded that “Everyone is sure of this [that errors are normally distributed] $\dots$ since the experimentalists believe that it is a mathematical theorem, and the mathematicians that it is an experimentally determined fact.” (Poincaré 1912, p. 171)

Acknowledgments

We thank Brian Collica and Jamarcus Liu for significant contributions to the development of the cvCovEst R package. We also thank the anonymous reviewers and associate editor for their helpful comments and suggestions.

Additional information

Funding

PB is funded by the Fonds de recherche du Québec – Nature et technologies, and the Natural Sciences and Engineering Research Council of Canada. NSH gratefully acknowledges the support of the National Science Foundation (award no. DMS 2102840).