Generalizing R² for deming regressions

Abstract

The simple linear regression model and the associated goodness-of-fit measure, the coefficient of determination, R², are only appropriate when all measurement errors are associated with the measurement of the data in the dependent variable. When measurement errors are assumed in both variables, a Deming regression can be used; however, there is no associated R²-type measure for this specific type of regression. In this paper, we propose a measure, $R_{\mathrm{g}}^2$ which utilizes the minimum percentage improvement of the Deming regression over either the horizontal or the vertical line through the centroid of the data. We investigate some properties of this measure and its relation to R². We also consider other candidate methodologies for a generalized R² measure for a Deming regression model and investigate strengths and weaknesses of each as a way of beginning the conversation about which measure is the best and for which applications it is most suited.

Keywords:

• Goodness-of-fit
• linear model
• errors in variable
• least-squares
• method comparison study

Notes

2 In a similar manner, if there are errors in X and the values of Y are exact, and one wishes to predict X given a particular value of Y, horizontal regression can be used and the corresponding coefficient of determination measures the percent improvement in the sum of squared errors in using the regression line over the comparison line $X=\overline{x}.$ Also the quantities SST_x and SSE_x analogous to SST_y and SSE_y can be similarly defined.

3 By $\lambda =\infty$ we mean $\text{Var}({ϵ}_X)\ne 0\mathrm{ }\text{and}\mathrm{ }\text{Var}({ϵ}_Y)=0$