Conceptualizing Interrater Agreement as Testing the Existence of Extra Variation in the Multinomial Model

Abstract

The existence of extra variation has long been of interest to many researchers. In some studies, it may be a nuisance variable that needs to be accounted for. In other cases, it is the parameter of interest, such as agreement among raters in a study. In this article, I demonstrate how the goodness-of-fit (GOF) procedure and a test statistic based on a large-sample variance from Bartfay and Donner (2001a) can be applied to assess whether there is agreement, as an example of extra variation, between 2 raters. Using a Monte Carlo simulation study, I compare the properties of these 2 procedures to the likelihood ratio test. The results show that the GOF procedure possesses satisfactory Type I error rate and power. I illustrate these methods using a published data set on assessing the agreement between two reviewers on accepting manuscripts for publication.