Measures of Agreement and Concordance With Clinical Research Applications

Abstract

This article reviews measures of interrater agreement, including the complementary roles of tests for interrater bias and estimates of kappa statistics and intraclass correlation coefficients (ICCs), following the developments outlined by Landis and Koch (1977a; 1977b; 1977c). Category-specific measures of reliability, together with pairwise measures of disagreement among categories, are extended to accommodate multistage research designs involving unbalanced data. The covariance structure of these category-specific agreement and pairwise disagreement coefficients is summarized for use in modeling and hypothesis testing. These agreement/disagreement measures of intraclass/interclass correlation are then estimated within specialized software and illustrated for several clinical research applications. Further consideration is also given to measures of agreement for continuous data, namely the concordance correlation coefficient (CCC) developed originally by Lin (1989). An extension to this CCC was published by King and Chinchilli (2001b), yielding a generalized concordance correlation coefficient which is appropriate for both continuous and categorical data. This coefficient is reviewed and its use illustrated with clinical research data. Additional extensions to this CCC methodology for longitudinal studies are also summarized.

Keywords: :

• Interrater bias
• Intraclass correlation
• Kappa statistics
• Pairwise disagreement
• Reliability