A Unified Approach for Assessing Agreement for Continuous and Categorical Data

Abstract

This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.

Key Words:

• Accuracy
• CCC
• CP
• ICC
• Inter-agreement
• Intra-agreement
• Kappa
• MSD
• Precision
• TDI
• Total-agreement

ACKNOWLEDGMENT

The research work for this article is supported by National Science Foundation (NSF) Grants DMS-0103727 and DMS-0603761, National Institutes of Health (NIH) Grant P50-AT00155 (jointly supported by National Center for Complementary and Alternative Medicine, the Office of Dietary Supplements, the Office of Research on Women's Health, and National Institute of General Medicine) and Astellas USA Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the NSF and the NIH.

Notes

CCC*: values are the same as the kappa, both in estimation and in inference.

CCC*: values are the same as the weighted kappa with mean squared weight, both in estimation and in inference.

: calculated by Barnhart's method.

*: for all CCC, precision and accuracy indices, the 97.5% lower limits are reported, for all TDI indices, the 95% upper limits are reported.

*: for all CCC, precision and accuracy indices, the 97.5% lower limits are reported.

*: is the inverse cumulative normal distribution ^**: is a central Chi-square distribution with one degree of freedom.