On Optimal Correlation-Based Prediction

Abstract

This note examines, at the population-level, the approach of obtaining predictors $\tilde{h}(\mathbf{X})$ of a random variable Y, given the joint distribution of $(Y,\mathbf{X})$, by maximizing the mapping $h\mapsto \kappa (Y,h(\mathbf{X}))$ for a given correlation function $\kappa (·,·)$. Commencing with Pearson’s correlation function, the class of such predictors is uncountably infinite. The least-squares predictor $h^{*}$ is an element of this class obtained by equating the expectations of Y and $h(\mathbf{X})$ to be equal and the variances of $h(\mathbf{X})$ and $\mathbf{E}(Y|\mathbf{X})$ to be also equal. On the other hand, replacing the second condition by the equality of the variances of Y and $h(\mathbf{X})$, a natural requirement for some calibration problems, the unique predictor $h^{**}$ that is obtained has the maximum value of Lin’s (1989) concordance correlation coefficient (CCC) with Y among all predictors. Since the CCC measures the degree of agreement, the new predictor $h^{**}$ is called the maximal agreement predictor. These predictors are illustrated for three special distributions: the multivariate normal distribution; the exponential distribution, conditional on covariates; and the Dirichlet distribution. The exponential distribution is relevant in survival analysis or in reliability settings, while the Dirichlet distribution is relevant for compositional data.

Keywords:

• Calibration
• Coefficient of determination
• Concordance correlation coefficient
• Least-squares predictor
• Maximal agreement predictor
• Pearson’s correlation

This article refers to:

Comment on “On Optimal Correlation-Based Prediction,” by Bottai et al. (2022)

Acknowledgments

We thank Dr. Ronald Christensen, who served as the Editor during the review process of the article, the associate editor, and the referees for their very useful and insightful comments and criticisms which considerably helped in revising the article. T. Kim thanks Professor Alexander Goldenshluger for providing the post-doctoral research opportunity at the University of Haifa in Israel.

Funding

E. Peña is currently Program Director in the Directorate of Mathematical and Physical Sciences (MPS), Division of Mathematical Sciences (DMS) at the US National Science Foundation (NSF). As a consequence of this position, he receives support for research, which included work in this article, under NSF Grant 2049691 to the University of South Carolina.

Disclosure statement

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Additional information

Funding

E. Peña is currently Program Director in the Directorate of Mathematical and Physical Sciences (MPS), Division of Mathematical Sciences (DMS) at the US National Science Foundation (NSF). As a consequence of this position, he receives support for research, which included work in this article, under NSF Grant 2049691 to the University of South Carolina.