Inter-rater reliability of occupational exposure assessment in a case-control study of female breast cancer

Abstract

The objective of this paper was to estimate the inter-rater reliability of expert assessments of occupational exposures. An inter-rater reliability sub-study was conducted within a population-based case-control study of postmenopausal breast cancer. Detailed information on lifetime occupational histories was obtained from participants and two industrial hygienists assigned exposures to 185 jobs using a checklist of 293 agents. Experts rated exposure for each job–agent combination according to exposure status (unexposed/exposed), confidence that the exposure occurred (possible/probable/definite), intensity (low/medium/high), and frequency (% time per week). The statistical unit of observation was each job–agent assessment (185 jobs × 293 agents = 54,205 assessments per expert). Crude agreement, Gwet AC1/2 statistics, and Cohen’s Kappa were used to estimate inter-rater agreement for confidence and intensity; for frequency, the intra-class correlation coefficient (ICC) was used. The majority of job–agent combinations were evaluated by the two experts to be not exposed (crude agreement >98% of decisions). The degree of agreement between the experts for the confidence of exposure status was Gwet AC1/2 = 0.99 (95% CI: 0.99–0.99), and for intensity, a Gwet AC2 = 0.99 (95% CI: 0.99–0.99). For frequency, an ICC of 0.31 (95% CI: 0.26–0.35) was found. A sub-analysis restricted to job–agent combinations for which the two experts agreed on exposure status revealed a moderate agreement for confidence of exposure (Gwet AC2 = 0.66) and high agreement for intensity (Gwet AC2 = 0.96). For frequency, the ICC was 0.52 (95% CI: 0.47–0.57). A high level of inter-rater agreement was found for identifying exposures and for coding intensity, but agreement was lower for the coding of frequency of exposure.

Keywords:

• Epidemiology
• hygiene
• occupational health
• retrospective studies
• women

Acknowledgments

Exposure assessment methods were conducted by Robert Bourbonnais and Mounia Rhazi. Consultation and aid provided by Louise Nadon and Dora Rodriguez was instrumental in the success of this study. A large number of research assistants and interviewers ensured the careful collection of data. The authors would also like to express their gratitude for the collaboration of all Montreal area hospitals and all participants.

Competing interests

No potential conflict of interest was reported by the author(s).

Data availability

Due to the nature of this research, participants of this study did not agree for their data to be shared publicly, so supporting data is not available.

Additional information

Funding

Funding for the expert assessment was provided by the Institut de recherche Robert-Sauvé en santé et en sécurité du travail (grant No. 2017-0038). The original data collection was funded by the Quebec Breast Cancer Foundation. Vikki Ho holds a Sex and Gender Science Chair in Cancer Research from the Canadian Institutes for Health Research. She was supported by the Cancer Research Society, the Fonds de recherche du Québec - Santé and the Ministère de l’Économie, de la Science et de l'Innovation du Québec during the conduct of this work. Currently, Vikki Ho is the recipient of a Research Scholar J2 Award from the Fonds de recherche du Québec - Santé.