Correcting Measurement Error in Content Analysis

ABSTRACT

Conducting and reporting reliability tests has become a standard practice in content analytical research. However, the consequences of measurement error in coding data are rarely discussed or taken into consideration in subsequent analyses. In this article, we demonstrate how misclassification in content analysis leads to biased estimates and introduce matrix back-calculation as a simple remedy. Using Monte Carlo simulation, we investigate how different ways of collecting information about the misclassification process influence the effectiveness of error correction under varying conditions. The results show that error correction with an adequate set-up can often substantially reduce bias. We conclude with an illustrative example, extensions of the procedure, and some recommendations.

Notes

¹ We will refrain from joining the philosophical debate whether the “true” value of a variable for a coding unit exists and, if so, can be determined without error. Similarly, we note that even the most accurate coding process does not necessarily guarantee validity (see Krippendorff, 2004a, Ch. 13, for a detailed discussion). Instead, we take a pragmatic stand on the issue. The true value is defined as the category that has to be coded following the researchers who have developed the measurement instrument.

² See the subsequent section on the maximum possible accuracy assumption for details.

³ We follow the notation of Kuha and Skinner (1997) throughout the article.

⁴ An anonymous reviewer was more pessimistic about the availability of preferred standards: “In practice, accuracy is a form of reliability that is rarely measurable because standards are hard to come by.” We agree that standards in the sense of a truth criterion are in fact rare. We argue, however, that a preferred standard in the sense of a correct application of the measurement instructions can, and should, be established. This is implicitly done during coder training, when the researchers explain the instrument to the coders. It is then only a small step to explicitly establish a preferred standard by classifying a set of test units.

⁵ The simulation was implemented in R (R Core Team, 2016), and full replication code is available at OSF: https://dx.doi.org/10.17605/OSF.IO/E7M9Z.

⁶ In the original publication, three dichotomous indicator variables were used to compute an average score for the strategy frame, and an average reliability score. However, for the purpose of this illustration, this mean index is just a rescaled dichotomous variable, which does not change the interpretation of proportions.