Inter-Rater Agreement between Trachoma Graders: Comparison of Grades Given in Field Conditions versus Grades from Photographic Review

ABSTRACT

Purpose: Trachoma surveillance is most commonly performed by direct observation, usually by non-ophthalmologists using the World Health Organization (WHO) simplified grading system. However, conjunctival photographs may offer several benefits over direct clinical observation, including the potential for greater inter-rater agreement. This study assesses whether inter-rater agreement of trachoma grading differs when trained graders review conjunctival photographs compared to when they perform conjunctival examinations in the field.

Methods: Three trained trachoma graders each performed an independent examination of the everted right tarsal conjunctiva of 269 children aged 0–9 years, and then reviewed photographs of these same conjunctivae in a random order. For each eye, the grader documented the presence or absence of follicular trachoma (TF) and intense trachomatous inflammation (TI) according to the WHO simplified grading system.

Results: Inter-rater agreement for the grade of TF was significantly higher in the field (kappa coefficient, κ, 0.73, 95% confidence interval, CI 0.67–0.80) than by photographic review (κ = 0.55, 95% CI 0.49–0.63; difference in κ between field grading and photo grading 0.18, 95% CI 0.09–0.26). When field and photographic grades were each assessed as the consensus grade from the three graders, agreement between in-field and photographic graders was high for TF (κ = 0.75, 95% CI 0.68–0.84).

Conclusions: In an area with hyperendemic trachoma, inter-rater agreement was lower for photographic assessment of trachoma than for in-field assessment. However, the trachoma grade reached by a consensus of photographic graders agreed well with the grade given by a consensus of in-field graders.

• Diagnosis
• inter-rater agreement
• observer variation
• photography
• trachoma

ACKNOWLEDGMENTS

We thank Melkam Andualem for her expert trachoma grading. We also thank Donald Everett (National Eye Institute, Bethesda, MD, USA), who was the program officer for the underlying clinical trial; the data safety and monitoring committee including William Barlow (University of Washington, Seattle, WA, USA; Chair), Donald Everett (National Eye Institute, Bethesda, MD, USA), Larry Schwab (International Eye Foundation, Kensington, MD, USA), Arthur Reingold (University of California, Berkeley, CA, USA), Serge Resnikoff (Brien Holden Vision Institute, Sydney, Australia, and International Health and Development, Geneva, Switzerland), and Patricia Buffler (University of California, Berkeley); the Goncha Siso Enese Woreda Health Office, including Abreham Tadesse and Tsegaye Tsehay; the head of Amhara Regional State Health Bureau, Ayeligne Mulualem; and the Ethiopian Federal Ministry of Health.

DECLARATION OF INTEREST

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

This work was supported by the National Institutes of Health [NEI U10 EY016214 and NIH/NEI K23EY019071], the Bernard Osher Foundation, That Man May See, the Harper Inglis Trust, the Bodri Foundation, the South Asia Research Fund, Research to Prevent Blindness, and the International Trachoma Initiative. The trachoma control program in Amhara is supported by the Amhara Regional Health Bureau, the Lions-Carter Center SightFirst Initiative and many individual donors.