An Overview on Assessing Agreement with Continuous Measurements

Abstract

Reliable and accurate measurements serve as the basis for evaluation in many scientific disciplines. Issues related to reliable and accurate measurement have evolved over many decades, dating back to the nineteenth century and the pioneering work of Galton (1886), Pearson (1896 1899 1901), and Fisher (1925). Requiring a new measurement to be identical to the truth is often impractical, either because (1) we are willing to accept a measurement up to some tolerable (or acceptable) error, or (2) the truth is simply not available to us, either because it is not measurable or is only measurable with some degree of error. To deal with issues related to both (1) and (2), a number of concepts, methods, and theories have been developed in various disciplines. Some of these concepts have been used across disciplines, while others have been limited to a particular field but may have potential uses in other disciplines. In this paper, we elucidate and contrast fundamental concepts employed in different disciplines and unite these concepts into one common theme: assessing closeness (agreement) of observations. We focus on assessing agreement with continuous measurements and classify different statistical approaches as (1) descriptive tools; (2) unscaled summary indices based on absolute differences of measurements; and (3) scaled summary indices attaining values between –1 and 1 for various data structures, and for cases with and without a reference. We also identify gaps that require further research and discuss future directions in assessing agreement.

Keywords:

• Accuracy
• Agreement
• Coefficient of individual agreement
• Concordance correlation coefficient
• Coverage probability
• Generalizability
• Intraclass correlation coefficient
• Limits of agreement
• Method comparison
• Precision
• Reliability
• Repeatability
• Reproducibility
• Tolerance interval
• Total deviation index
• Validity

ACKNOWLEDGMENT

This research is supported by the National Institutes of Health Grant R01 MH70028 (for Huiman X. Barnhart and Michael Haber). We greatly appreciate Josep Carrasco and Pankaj Choudhary for their timely review of this paper and their constructive comments that lead to improvement of the content and completeness of the literatures in this paper.