Events Data as Bismarck's Sausages? Intercoder Reliability, Coders' Selection, and Data Quality

Abstract

Precise measurement is difficult but essential in the generation of high-quality data, and it is therefore remarkable that often so little attention is paid to intercoder reliability. It is commonly recognized that poor validity leads to systematic errors and biased inference. In contrast, low reliability is generally assumed to be a lesser concern, leading only to random errors and inefficiency. We evaluate the intercoder reliability of our recently collected data on governance events in UN peacekeeping and show how poor coding and low intercoder reliability can produce systematic errors and even biased inference. We also show how intercoder reliability checks are useful to improve data quality. Continuous testing for intercoder reliability ex post enables researchers to create better data and ultimately improves the quality of their analyses.

Keywords:

• coders
• events data
• measurement
• peacekeeping
• reliability

Acknowledgments

A previous version of this paper was presented at the ISA annual conference in New York, March 2009 and at the Folke Bernadotte UN PKO working group in Columbia University, New York, October 30–31, 2009. We acknowledge financial support from the Folke Bernadotte Akademin and the ESF/ESRC (RES-062-23-0259). We thank the participants at the Folke Bernadotte UN PKO working group for their comments. We also thank Kristian Skrede Gleditsch, María Belén González, Birger Heldt, Spyros Kosmidis, Nikolay Marinov, Will H. Moore, Vera Troeger, and Steffen Weiss and three anonymous reviewers of International Interactions for their comments. Replication data are available on the International Interactions dataverse page at http://dvn.iq.harvard.edu/dvn/dv/internationalinteractions.

Notes

¹The findings of the survey are discussed in more detail in the concluding section. See also Table 7.

²In this article we pay special attention to events data; yet, the issue of intercoder reliability applies more broadly to research methods. Even qualitative data, like interviews, archival research or even participatory observation, require “coding,” that is, the selection and interpretation of information (Kalyvas 2006:393–422). Although often probably unfeasible for practical reasons, in principle researchers could independently code the “raw” information in order to gauge the reliability of qualitative research.

³Once measurement bias has been identified, it is straightforward to evaluate the impact of low validity on the statistical analysis and even to correct for any bias. However, it is far from straightforward to determine whether unreliable data have significant impact on the statistical analysis or how to use information on reliability to improve data quality. Baugh (2003) has developed a method that allows the adjustment of coefficients using the estimated data reliability.

⁴Conflict and cooperation do not refer to the general background of the governance event, but must be in direct response to an event. A deteriorating security situation is likely to influence governance, but conflict is only coded if the governance actors or public goods are explicitly targeted. The baseline categories are when no conflict (cooperation) is recorded.

⁵For more information on the other variables please refer to the codebook downloadable from http://privatewww.essex.ac.uk/~hdorus/.

⁶In the second stage, not all reports were double coded; Angola was double coded for the 93% of the reports, Burundi for 28%, Central African Republic 100%, and Democratic Republic of Congo 24%.

⁷As one of the reviewers pointed out, event selection may well be the major difference between high frequency data, such as TABARI, BCOW or the UN Peacekeeping events data, and country-year data, like the UCDP-PRIO data.

⁸ Hruschka et al. (2004) suggest fairly stringent cutoffs of Kappa ≥ 0.80 or 0.90, while Landis and Koch (1977:165) suggest a range of intercoder reliability based on Kappa, distinguishing between poor (< 0.0), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (0.81–1.00).

⁹The probit models of Tables 5 and 6 largely replicate these findings in Dorussen and Gizelis (2008), at least for the “good” coders. This is noteworthy since Dorussen and Gizelis (2008) rely exclusive on data collected in stage 2, use a more extensive model, disaggregate conflict/cooperation levels and estimate the effects on conflict and cooperation simultaneously.

¹⁰Though we provide only graphic tests for heteroskedasticity, we have also consistent findings using heteroskedasticity probit models (Alvarez and Brehm 1995) where we use the “good versus bad” variable to model the variance of the residuals. Indeed, we find that using “good” coders shrinks the residuals' variance.

¹¹We classify an article as quantitative if it includes at least one regression table. We excluded in our category “quantitative” articles that introduce new datasets, if they did not satisfy our criterion of at least one regression table. Moreover, we excluded formal theory articles that did not have any empirics.