An Examination of the Inter-Rater Reliability and Rater Accuracy of the Level of Service/Case Management Inventory

ABSTRACT

Correctional agencies use risk assessment instruments for a wide range of purposes, including to help classify, manage, and treat offenders. The literature on offender risk assessment largely focuses on assessing for predictive accuracy, and far less research examines reliability in scoring. This study adds to this gap in knowledge by assessing how reliably and accurately a group of trained raters score one particular risk assessment tool, the Level of Service/Case Management Inventory (LS/CMI). Findings reveal an adequate to strong level of inter-rater reliability across the domains of the LS/CMI. These results also suggest there is a wide range of rater accuracy across the items and domains of the LS/CMI. The policy and practical implications of these findings are discussed.

KEYWORDS:

• Risk assessment
• reliability
• community corrections
• probation

Acknowledgement

The authors wish to thank Kimberly Bernard for her editorial comments. The authors also wish to thank the two anonymous reviewers for their helpful and constructive comments.

Notes

1. For more on the discussion of the generations in risk assessments see Bonta and Andrews (2017, pp. 192–222).

2. Most of the other scholarship on reliability of the LS instruments are found in unpublished conference presentations, theses, and dissertations. These other works primarily involve examinations of alpha (α) levels, which can help inform how consistently the items within a domain are scored as a group, but not how reliably the assessment is scored by raters. For more information on this research see Andrews et al. (2010).

3. Some researchers suggest a different criterion for interpreting the ICC values. Baird et al. (2013), for example, use 0.0 to 0.2 = poor, 0.3 to 0.4 = fair, 0.5 to 0.6 = moderate, 0.7 to 0.8 = strong, and > 0.8 = almost perfect.