Inter-Rater Reliability of Assessed Prenatal Maternal Occupational Exposures to Solvents, Polycyclic Aromatic Hydrocarbons, and Heavy Metals

Abstract

Because direct measurements of past occupational exposures are rarely available in population-based case-control studies, exposure assessment of job histories by multiple expert raters is frequently used; however, the subjective nature of this method makes measuring reliability an important quality control step. We evaluated inter-rater reliability of 7729 retrospective jobs reported in the National Birth Defects Prevention Study. Jobs were classified as exposed, unexposed, or exposure unknown by two independent industrial hygienists; exposed jobs were further evaluated for intensity, frequency, and routes. Exposure prevalence ranged from 0.1–9.8%. Inter-rater reliability for exposure (yes/no), assessed by kappa coefficients, was fair to good for cadmium (κ = 0.46), chlorinated solvents (κ = 0.59), cobalt (κ = 0.54), glycol ethers (κ = 0.50), nickel compounds (κ = 0.65), oil mists (κ = 0.63), and Stoddard Solvent (κ = 0.55); PAHs (κ = 0.24) and elemental nickel (κ = 0.37) had poor agreement. After a consensus conference resolved disagreements, an additional 4962 jobs were evaluated. Inter-rater reliability improved or stayed the same for cadmium (κ = 0.51), chlorinated solvents (κ = 0.81), oil mists (κ = 0.63), PAHs (κ = 0.52), and Stoddard solvent (κ = 0.92) in the second job set. Inter-rater reliability varied by exposure agent and prevalence, demonstrating the importance of measuring reliability in studies using a multiple expert rater method of exposure assessment.

Key words:

• birth defects
• exposure assessment
• industrial hygienist

ACKNOWLEDGMENTS

The authors would like to recognize Marianne Yencken and James Catalano of the Battelle Centers for Public Health and Evaluation for their contributions to the ratings process and interpretation. We particularly thank the 10 NBDPS centers who collected the data. This work was supported by the Centers for Disease Control and Prevention (U50/CCU 713238; U01/DD000492); and the National Institute for Occupational Safety and Health (contract 200-2000-08018).

The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.

Notes

^A ACGIH threshold limit value. All values listed are 8-hr time-weighted averages.

^B Inhalable fraction.

^C NA, not applicable; quantitative exposure levels were not assigned to the intensity scores.

^D Includes EGME (ethylene-glycol-mono-methyl-ethers, TLV = 0.1 ppm) and EGEE (ethylene-glycol-mono-ethyl-ethers, TLV = 5 ppm) only.

^E TLV for soluble inorganic compounds is 0.1 μg/m³; TLV for insoluble inorganic compounds is 0.2 μg/m³.

^F No job was assessed as having this intensity.

^G There is no TLV for polycyclic aromatic hydrocarbons (PAHs) as a group; coal tar pitch volatiles (TLV = 0.2 μg/m³) were used as the marker for PAHs.

^A Results exclude 22–25 jobs per agent coded as unknown due to insufficient information to make an exposure determination.

^B Two of the three raters independently assessed exposure for each agent.

^C Observed agreement (p_o) is the proportion of jobs where the two raters agreed on the exposure status (exposed, not exposed).

^D Negative specific agreement (p_neg) is the ratio of the number of jobs rated as not exposed by both raters and the mean number of jobs rated as not exposed by each rater.

^E Positive specific agreement (p_pos) is the ratio of the number of jobs rated as exposed by both raters and the mean number of jobs rated as exposed by each rater.

^F Simple (unweighted) kappa coefficient (κ) is a measure of agreement correcting for chance.

^G Includes EGME (ethylene-glycol-mono-methyl-ethers, TLV = 0.1 ppm) and EGEE (ethylene-glycol-mono-ethyl-ethers, TLV = 5 ppm) only.

^A For direct and indirect intensity ratings (ordinal from 0 to 4), the table entry is the quadratic kappa statistic weighted by frequency (weighting discrepancies with the square of the difference between the paired assessments). The remaining kappa coefficients are simple (unweighted) coefficients. Four categories were considered for direct (not exposed, direct, indirect, both) and continuous (not exposed, continuous, intermittent, both) exposure ratings. Two categories were considered for dermal and ingestion ratings (not exposed/not likely and likely).

^B The intra-class correlation coefficient (ICC) was obtained via the MIXED procedure in SAS. The model considered job as a random effect, rater as a fixed effect, and specified an unstructured covariance matrix.

^C Weighted intensity = (direct intensity exposure score) × (fraction of time directly exposed) + (indirect intensity exposure score) × (fraction of time indirectly exposed), where intensity score was rated on an ordinal scale from 0 to 4.

^D Includes EGME (ethylene-glycol-mono-methyl-ethers, TLV = 0.1 ppm) and EGEE (ethylene-glycol-mono-ethyl-ethers, TLV = 5 ppm) only.

^E Kappa could not be computed because there was no variation in exposure (i.e., all jobs were rated as not exposed by ingestion).

^A Rater A, Rater C; NE, not exposed; E, exposed.

^B Perfect agreement. Kappa could not be computed because there was no variation in exposure (i.e., all jobs rated as not exposed).

^C Kappa could not be computed because there was no variation in exposure for Rater A (i.e., Rater A rated all jobs as not exposed).

^A E = exposed, NE = not exposed, D = discordant.

^B Only two raters completed the Step 2 ratings, while all three raters participated in the Step 3 consensus ratings.

^A Weighted exposure intensity score = (direct intensity exposure score) × (fraction of time directly exposed) + (indirect intensity exposure score) × (fraction of time indirectly exposed), where intensity score was rated on an integer scale from 0 to 4.

^B Number (percentage) of jobs rated by consensus as exposed.

^C Jobs were rated as direct, indirect, or both. Exposure was considered as direct here if it was rated as direct or both.

^D Jobs were rated as continuous, intermittent, or both. Exposure was considered as continuous here if it was rated as continuous or both.