Estimators of kappa-exact small sample properties

Abstract

Many estimators of the measure of agreement between two raters have been proposed. We consider four estimators which have been shown to be interpretable as corrected for chance and as intraclass correlation coefficients, namely Scott's, Cohen's, Mak's and Maxwell and Pilliner's. The exact values of bias, mean squared error, skewness and kurtosis of these estimators and their jackknifed versions are calculated for sample sizes from 20 to 250 under Bloch and Kraemer's (1989) model. Although no estimator is uniformly optimal, jackknifed estimators are shown to be better, on average, than the usual estimators. These results extend and correct those of Block and Kraemer (1989) who used only Monte Carlo simulation. Two estimators of variance are also evaluated.

Keywords:

• Interrater agreement
• kappa statistic
• 2 × 2 tables
• jackknife

^*Current address: Glaxo Canada Inc. Mississauga, Ontario, Canada.

^*Current address: Glaxo Canada Inc. Mississauga, Ontario, Canada.

Notes

^*Current address: Glaxo Canada Inc. Mississauga, Ontario, Canada.