A Comparison of Confidence Interval Methods for the Concordance Correlation Coefficient and Intraclass Correlation Coefficient with Small Number of Raters

Abstract

The intraclass correlation coefficient (ICC) with fixed raters or, equivalently, the concordance correlation coefficient (CCC) for continuous outcomes is a widely accepted aggregate index of agreement in settings with small number of raters. Quantifying the precision of the CCC by constructing its confidence interval (CI) is important in early drug development applications, in particular in qualification of biomarker platforms. In recent years, there have been several new methods proposed for construction of CIs for the CCC, but their comprehensive comparison has not been attempted. The methods consisted of the delta method and jackknifing with and without Fisher's Z-transformation, respectively, and Bayesian methods with vague priors. In this study, we carried out a simulation study, with data simulated from multivariate normal as well as heavier tailed distribution (t-distribution with 5 degrees of freedom), to compare the state-of-the-art methods for assigning CI to the CCC. When the data are normally distributed, the jackknifing with Fisher's Z-transformation (JZ) tended to provide superior coverage and the difference between it and the closest competitor, the Bayesian method with the Jeffreys prior was in general minimal. For the nonnormal data, the jackknife methods, especially the JZ method, provided the coverage probabilities closest to the nominal in contrast to the others which yielded overly liberal coverage. Approaches based upon the delta method and Bayesian method with conjugate prior generally provided slightly narrower intervals and larger lower bounds than others, though this was offset by their poor coverage. Finally, we illustrated the utility of the CIs for the CCC in an example of a wake after sleep onset (WASO) biomarker, which is frequently used in clinical sleep studies of drugs for treatment of insomnia.

Key Words:

• Bayesian method
• Concordance correlation coefficient
• Confidence interval
• Delta method
• Intraclass correlation coefficient with fixed raters
• Jackknife

ACKNOWLEDGMENTS

We are grateful to Dr. Joseph Heyse for bringing the jackknife method to our attention and for his guidance and encouragement. We also thank Dr. Andy Liaw and Dr. Richard Raubertas for helpful discussions. Alexandre Coimbra is currently at Genentech, Inc., San Francisco, CA.

Notes

Note. Methods compared are abbreviated as follows: C: Carrasco and Jover (2003), CZ: Carrasco and Jover (2003) with Z-transformation, J: jackknife, JZ: jackknife with Z-transformation, BJ: Bayes with Jeffreys prior (Berger and Sun, 2008), BC: Bayes with normal-inverse-Wishart conjugate prior (Gelman et al.,2003). The method with coverage closest to the nominal 95% is denoted in bold.

Note. The method with coverage closest to the nominal 95% is denoted in bold.

Note. The method with coverage closest to the nominal 95% is denoted in bold.

Note. The method with coverage closest to the nominal 95% is denoted in bold.