Prediction is Local: The Benefits of Risk Assessment Optimization

Abstract

In most states and jurisdictions, risk assessments are incorporated into justice system practice. Despite decades of use, the methods of tool development are rarely translated to the field. Many agencies implement ‘off-the-shelf’ versions, where a tool developed with a unique set of methods and subjects demonstrates prediction shrinkage when applied to a new jurisdiction. Using a large, 10-state sample of assessed youth (N=494,050), we isolate, test, and evaluate the relative impact of notable risk assessment variations, including: item selection, response weighting, outcome definition/duration, and jurisdiction. We further combined approaches to evaluate an ‘optimized’ development approach. Findings revealed substantial gains with each variation tested, where optimized models provided a full effect size predictive improvement. We discuss best practices for the future of risk assessment, noting the predictive accuracy lost when implementing tools off-the-shelf, and describe how optimization techniques substantially improve risk prediction, specifying a given tool to an agency’s needs.

Keywords:

• risk/needs assessment,
• optimization
• predictive validity
• multistate
• justice-involved youth

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Notably, Rice and Harris (2005) provide statistical crosswalks that we similarly tie to the noted effect size ranges.

2 It should be noted that while the current study does not test generational variations, ‘fourth generation’ is a well-known term (see Andrews et al., 2006) representing a tool that assesses risk, need, and responsivity and is often equated as a minimum standard for contemporary tools, noted here as a descriptive reference.

3 Additional descriptives, by site, can be provided upon request.

4 Readers should see Hamilton, Kigerl, et al. (2020) for a more detailed description of sites

5 Predictive metrics are computed for select hypothesis comparisons and may include Area Under the Curve (AUC) Accuracy (ACC), H measure, calibration, false positives, false negatives, true positives, true negatives, and a combined measure (SAR) that assesses the joint calibration, accuracy and model discrimination.

6 While there are many types of regression models available, the intent was not to examine a specific type but to remove prediction noise thought to be the result of bivariate selection.

7 While more development methods exist, we outline those of three common contemporary tools, notably the YLS/CMI (bivariate feature selection), OYAS (multivariate feature selection), and COMPAS Youth (multivariate feature selection and weighting).

8 Each score is averaged together; however, RMSE is reverse coded (i.e., 1 – RMSE) such that it became a maximization metric prior to averaging into the SAR score.

9 Items selected and model coefficients for the multivariate weighted model are provided in Appendix B.

10 Note that all models used a 12-month follow-up, operationalized in the methods section.

11 Items selected and model coefficients are provided in Appendix D.

12 Items selected and model coefficients are provided in Appendix D.

13 Items selected and model coefficients are provided in Appendix E.

14 Readers should note that two states did not have all eight outcomes available for testing; however, the best model was selected from those available.

15 Items selected and model coefficients are provided in Appendix F.

16 Note, due to space limitations, these results are not displayed but may be made available upon request.