Using a Mobile App to Identify Base Rates and Monitor Bias in Forensic Evaluation

ABSTRACT

The potential for bias in forensic mental health evaluations can threaten the objectivity of psycholegal opinions. To date, individual evaluators have few mechanisms to identify bias in their own work. The present study examines the use of a recently developed iPhone App designed for this purpose. Evaluators from 18 states used this platform to enter information related to their forensic evaluations, allowing them to receive real-time data about the rates of their opinions across a variety of variables. This study discusses results from 831 evaluations entered into the app and outlines the benefits of using handheld technology in practice. Some variables appeared to influence forensic outcomes (such as the payment of a fee to an evaluator), while others did not (such as defendant ethnicity and race). Analyses also revealed a 49.6% base rate of incompetency to stand trial opinions across all competency evaluations entered into the app.

KEYWORDS:

• Bias
• competency to stand trial evaluations
• forensic evaluation
• technology
• competency

Disclosure statement

The authors are the owners of the mobile application discussed in this article, and they therefore have a financial interest in the application. We have an approved and effective mechanism to mitigate this conflict as approved by the publishers. The data utilized, analyzed, and presented in this article represents the full data set of every variable entered into the mobile application, with no data being excluded from any analysis unless otherwise disclosed.

Notes

¹ In accordance with Taylor & Francis policy and my ethical obligation as a researcher, I am reporting that I have a financial and/or business interest in a company that may be affected by the research reported in the enclosed paper. I have disclosed those interests fully to Taylor & Francis, and I have in place an approved plan for managing any potential conflicts arising from that involvement.

² At present, the app only provides the user with the base rates of their individual forensic opinions. Such a comparison would require users to compare their individualized data to the available literature.

³ Although [name withheld] provides definitions for how to code most of these variables, ultimately the variables are entered by the evaluators themselves and are therefore subject to their own judgments. Examples of outcomes that are “favorable to the defense attorney” might include opinions recommending substantial mitigation, insanity, or incompetency; however, due to evaluators entering information on their own, there is no way to make these determinations completely uniform.

⁴ [name withheld] does not yet collect information as to the sequencing of evaluations, so it is unknown how many CST evaluations were initial versus subsequent evaluations of the same defendant.

⁵ To align with developing standards regarding the definition of race, this study recognizes race as a political and social construct that often serves as a proxy for the impact of racist practices and structural inequality. It is not a biological variable and is examined in the current paper with this premise in mind. Pre-determined options for race, ethnicity, and gender in the app are limited to mutually exclusive, broadly defined categories that may not appropriately represent the multiple identities of some evaluators or examinees. This is especially true for ethnicity, which provides a dichotomous selection of either Hispanic or Latino/a versus not Hispanic or Latino/a. We specially recognize Idia B. Thurston for the language found in this acknowledgment (found in Buchanan et al., 2021, p. 1103).

⁶ A posthoc observed power analysis was conducted for the non-significant “fee charged” variable, and it revealed more than 90% power to reveal a small to medium effect size. It is possible that the true effect size is minimal, or that the interpretation of non-significance is a result of Type II error. However, given the sample size, and assuming even a small effect size, it is likely more probable that there is no true effect. Other non-significant findings specifically referenced in this study (race, gender, ethnicity, etc.) had larger sample sizes and our conclusions regarding power analyses are identical to the “fee charged” variable.

⁷ We decline to provide the state and the numbers of evaluators responsible for those 87 evaluations to preserve anonymity.

⁸ The overlap between defense attorneys and retaining attorneys is likely substantial, given that most mental health referrals are generated from defense counsel.