Three-Way ROCs for Forensic Decision Making

Abstract

Firearm examiners use a comparison microscope to judge whether bullets or cartridge cases were fired by the same gun. Examiners can reach one of three possible conclusions: Identification (a match), Elimination (not a match), or Inconclusive. Numerous error rate studies report that firearm examiners commit few errors when they conduct these examinations. However, the studies also report many inconclusive judgments (> 50%), and how to score these responses is controversial. There have recently been three Signal Detection Theory (SDT) primers in this domain. Unfortunately, these analyses rely on hypothetical data and fail to address the inconclusive response issue adequately. This article reports an SDT analysis using data from a large error rate study of practicing firearm examiners. First, we demonstrate the problem of relying on the traditional two-way SDT model, which either drops or combines inconclusive responses; in addition to lacking ecological validity, this approach leads to implausible results. Second, we introduce readers to the three-way SDT model. We demonstrate this approach in the forensic firearms domain. While the three-way approach is statistically complicated, it is well suited to evaluate performance for any forensic domain in which three possible decision categories exist.

KEYWORDS:

• Firearms
• Forensic science
• Inconclusive decisions
• Signal detection theory

Notes

1 An additional approach is to report 1- PPV (positive predictive value) (Hofmann, Carriquiry, and Vanderplas 2021). However, 1-PPV is dependent on the base rate, which is arbitrarily determined in research studies, unknowable in casework, relies on non-scientific determinations (Scurich and John 2012; Thompson et al. 2013), and confounds the performance of the examiner with the base rate (Albright 2022).