A Statistical Approach to Assessment of Agreement Involving Multiple Raters

Abstract

Study of agreement between two or more comparable sets of measurements taken on each member of a population is needed in many areas. There is an impressive literature on the topic covering both the qualitative and quantitative features of study variables. In order to ascertain the extent of agreement between the quantitative ‘target’ variable X and the quantitative ‘test’ variable Y, coverage probability (CP) was introduced by Lin et al. (2002). CP is defined as the probability of (X, Y) falling into a strip along the direction of X = Y of a judiciously specified width 2d but symmetric with respect to X and Y. For a given d, the higher CP value is, the better agreement it indicates. This article dwells on the concept of agreement involving three or more comparable quantitative measurements taken on each member of a population. We contemplate on a situation wherein we have available a reference gold standard against which several competitors are to be compared with respect to their performance as judged by the CP. We also address the problem of judging the performance of the competitors in the absence of any prescribed gold standard.

Keywords:

• Assay validation
• Bioavailability
• Coverage probability (CP)
• Deviation
• Gold standard
• Kappa
• Measurement
• Multivariate normal distribution
• Target

Mathematics Subject Classification:

• Primary 62J15
• Secondary 62H20, 62F30

Acknowledgments

This research was primarily sponsored by the National Science Foundation (NSF) Grant DMS-0103727, DMS-0904125 and the National Institute of Health (NIH) Grant P50-AT00155 (jointly supported by the National Center for Complementary and Alternative Medicine, the Office of Dietary Supplements, the Office of Research on Women's Health, and the National Institute of General Medicine). The contents are solely the responsibility of the authors and do not necessarily represent the official view of NSF or NIH. The authors acknowledge three anonymous referees that greatly helped improve the article. The authors also acknowledge Dr. Lawrence Lin and Prof. H. X. Barnhart for the permission of using their data in this article. Bikas K. Shiha was on leave from Indian Statistical Institute, Kolkata, while the research investigation for this article was being carried out at University of Illinois at Chicago, Chicago.