The Use of Meta-Analysis to Compare and Select Offender Risk Instruments: A Commentary on Singh, Grann, and Fazel (2011)

ABSTRACT

Criminal justice researchers and agencies often look to empirical reviews and meta-analyses when evaluating offender risk instruments. However, variations in meta-analytical methods can influence the interpretation of statistical findings, resulting in potentially misleading conclusions. Through our re-examination of a meta-analysis by Singh, Grann, and Fazel (2011), we outline common methodological problems that may occur and demonstrate how alternative interpretations might be derived. We conclude by providing recommendations for conducting meta-analyses. Suggestions are made for researchers to consider alternative strategies when examining the validity of risk instruments, and for correctional agencies to consider, but go beyond, predictive validity when selecting offender risk instruments.

KEYWORDS:

• Risk assessment
• binary decision-making
• meta-analysis

Notes

¹ Although the PCL-R provides thresholds used to categorize individuals as “psychopathic” or “not psychopathic”, research shows that psychopathy is a non-binary construct, suggesting that binary cut scores should be avoided in research applications and reserved only for pragmatic decisions (e.g., Edens, Marcus, Lilienfeld, & Poythress, 2006). Consequently, Hare and numerous colleagues have included a third “moderate” psychopathy group in some predictive validity studies of the PCL-R (e.g., Hart, Kropp & Hare, 1988; Loucks & Zamble, 2000; Serin & Amos, 1995).

² It is noted that PPV is a dichotomized prediction statistic and therefore is subject to the previously noted criticisms. Since PPVs were reported by Singh et al., they are used in Figure 1 for illustrative purposes only (i.e., IQRs).

³ In the absence of necessary raw data, our method of re-analysis follows recommendations by Cumming and Finch (2005) using the 95% CI as an alternative to more rigorous significance testing (see also Schenker & Gentleman, 2001).