Assessing the confidence-accuracy relationship in risk assessments using the START: introducing calibration analysis

ABSTRACT

A risk assessor’s confidence has been shown to influence both the rater as well as those evaluating the risk assessment. It is important to consider the impact of confidence on accuracy in the risk assessment field given the significant implications of risk assessments for the assessed (e.g. available treatment options vs. restrictions of liberties) and the safety of the public. While prior research in the risk assessment field has used correlations to investigate the confidence-accuracy (C-A) relationship, a number of fields in psychology have introduced additional techniques, including calibration analysis, to understand this relationship. In this study, we examined the C-A relationship across various adverse clinical outcomes using the Short-Term Assessment of Risk and Treatability (START) and compared and contrasted the C-A relationship using conventional methods (i.e. correlations/ROC analysis) and calibration. Raters completed START assessments for a sample of 106 civil psychiatric inpatients. Overall, calibration provided greater detail into the C-A relationship compared to correlations/ROC analysis. Our results also suggested that the C-A relationship varied as a function of the outcome assessed (e.g. violence, substance abuse, unauthorized leave). These results provide insights into the C-A relationship in medico-legal settings and can inform best practices for risk assessment training and implementation.

KEYWORDS:

• Risk assessment
• confidence-accuracy relationship
• calibration
• patient safety
• START

Acknowledgments

This study was funded by a Canadian Institute of Health Research grant to TN. We are also grateful for the funding provided by the British Columbia Mental Health and Substance Use Services, the Michael Smith Foundation for Health Research, and the MIND Foundation. Dr. Nicholls would like to acknowledge the Canadian Institutes of Health Research for her Foundation Grant and express her gratitude for salary support from CIHR and the Michael Smith Foundation in the form of a New Investigator Award and a Career Scholar Award.

The authorship team would like to thank Dr. Luna for his generous and helpful responses to questions about his confidence paper (Luna, K., & Martín-Luengo, B. (2012). Confidence-accuracy calibration with general knowledge and eyewitness memory cued recall questions. Applied Cognitive Psychology, 26(2), 289–295. doi:10.1002/acp.1822) which helped us clarify our understanding of our results.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Sexual aggression levels 1 to 3 include sexually inappropriate behaviors (e.g. sexual comments) but do not meet the criteria for the HCR-20 definition of violent behaviours, and therefore were not included in the violence variable. In addition, START does not include a Specific Risk Estimate for sexually inappropriate behaviour.

Additional information

Funding

This work was supported by the Canadian Institutes of Health Research [210515]; Michael Smith Foundation for Health Research [CI-SCH-00924(06-1].