Three-Way ROCs for Forensic Decision Making

References

• AFTE. (2022), “AFTE Range of Conclusions,” available at https://afte.org/about-us/what-is-afte/afte-range-of-conclusions
   Google Scholar
• Albright, T. D. (2022), “How to Make Better Forensic Decisions,” Proceedings of the National Academy of Sciences, 119, e2206567119. DOI: 10.1073/pnas.2206567119.
   PubMed Web of Science ®Google Scholar
• Arkes, H. R., and Koehler, J. J. (2022a), “Inconclusives Are Not Errors: A Rejoinder to Dror,” Law, Probability and Risk, 21, 89–90. DOI: 10.1093/lpr/mgac009.
   Web of Science ®Google Scholar
• Arkes, H. R., and Koehler, J. J. (2022b), “Inconclusives and Error Rates in Forensic Science: A Signal Detection Theory Approach,” Law, Probability and Risk, 20, 153–168. DOI: 10.1093/lpr/mgac005.
   Web of Science ®Google Scholar
• Arkes, H. R., and Koehler, J. J. (2023), “Inconclusive Conclusions in Forensic Science: rejoinders to Scurich, Morrison, Sinha and Gutierrez,” Law, Probability and Risk, 21, 175–177. DOI: 10.1093/lpr/mgad002.
   Web of Science ®Google Scholar
• Arkes, H. R., and Mellers, B. A. (2002), “Do Juries Meet Our Expectations?,” Law and Human Behavior, 26, 625–639. DOI: 10.1023/a:1020929517312.
   PubMed Web of Science ®Google Scholar
• Baldwin, D. P., Bajic, S. J., Morris, M. D., and Zamzow, D. S. (2023), “A Study of Examiner Accuracy in Cartridge Case Comparisons Part 2: Examiner Use of the AFTE Range of Conclusions,” Forensic Science International, 349, 111739. DOI: 10.1016/j.forsciint.2023.111739.
   PubMed Web of Science ®Google Scholar
• Best, B. A., and Gardner, E. A. (2022), “An Assessment of the Foundational Validity of Firearms Identification Using Ten Consecutively Button-Rifled Barrels,” AFTE Journal, 54, 28–37.
   Google Scholar
• Biedermann, A., and Kotsoglou, K. N. (2021), “Forensic Science and the Principle of Excluded Middle: “Inconclusive” Decisions and the Structure of Error Rate Studies,” Forensic Science International. Synergy, 3, 100147. DOI: 10.1016/j.fsisyn.2021.100147.
   PubMedGoogle Scholar
• Bowers, A. J., and Zhou, X. (2019), “Receiver Operating Characteristic (ROC) Area under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes,” Journal of Education for Students Placed at Risk (JESPAR), 24, 20–46. DOI: 10.1080/10824669.2018.1523734.
   Web of Science ®Google Scholar
• Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993), 509 U.S. 579
   Google Scholar
• Dorfman, A. H., and Valliant, R. (2022a), “Inconclusives, Errors, and Error Rates in Forensic Firearms Analysis: Three Statistical Perspectives,” Forensic Science International. Synergy, 5, 100273. DOI: 10.1016/j.fsisyn.2022.100273.
   PubMedGoogle Scholar
• Dorfman, A. H., and Valliant, R. (2022b), “A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study,” Statistics and Public Policy, 9, 175–184. DOI: 10.1080/2330443X.2022.2120137.
   Web of Science ®Google Scholar
• Dreiseitl, S., Ohno-Machado, L., and Binder, M. (2000), “Comparing Three-Class Diagnostic Tests by Three-Way ROC Analysis,” Medical Decision Making: An International Journal of the Society for Medical Decision Making, 20, 323–331. DOI: 10.1177/0272989X0002000309.
   PubMed Web of Science ®Google Scholar
• Dror, I. E. (2022), “The Use and Abuse of the Elusive Construct of Inconclusive Decisions,” Law, Probability and Risk, 21, 85–87. DOI: 10.1093/lpr/mgac008.
   Web of Science ®Google Scholar
• Dror, I. E., and Scurich, N. (2020), (“Mis)Use of Scientific Measurement in Forensic Science,” Forensic Science International Synergy, 2, 333–338. DOI: 10.1016/j.fsisyn.2020.08.006.
   PubMedGoogle Scholar
• Durose, M. R., and Burch, A. M. (2016), “Publicly funded forensic crime laboratories: resources and Services. Bureau of Justice Statistics,” available at: https://bjs.ojp.gov/content/pub/pdf/pffclrs14.pdf.
   Google Scholar
• Edwards, D. C., and Metz, C. E. (2006), “Analysis of Proposed Three-Class Classification Decision Rules in Terms of the Ideal Observer Decision Rule,” Journal of Mathematical Psychology, 50, 478–487. DOI: 10.1016/j.jmp.2006.05.004.
   Web of Science ®Google Scholar
• Faigman, D. L., Scurich, N., and Albright, T. D. (2022), “The Field of Firearms Forensics is Flawed,” Scientific American.
   Google Scholar
• Garrett, B. L., Tucker, E., and Scurich, N. (in press), “Judging Firearms Evidence,” Southern California Law Review, 97.
   Google Scholar
• Glossary of the Association of Firearm and Tool Mark Examiners. (2013), 6th edition. Available at https://afte.org/uploads/documents/AFTE/textunderscoreGlossary/textunderscoreVersion/textunderscore6.091922/textunderscoreFINAL/textunderscoreCOPYRIGHT.pdf
   Google Scholar
• Green, D. M., and Swets, J. A. (1966), Signal Detection Theory and Psychophysics, New York: Wiley.
   Google Scholar
• Hanley, J. A., and McNeil, B. J. (1982), “The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve,” Radiology, 143, 29–36. DOI: 10.1148/radiology.143.1.7063747.
   PubMed Web of Science ®Google Scholar
• He, X., Song, X., and Frey, E. C. (2008), “Application of Three-Class ROC Analysis to Task-Based Image Quality Assessment of Simultaneous Dual-Isotope Myocardial Perfusion SPECT (MPS),” IEEE Transactions on Medical Imaging, 27, 1556–1567. DOI: 10.1109/TMI.2008.928921.
   PubMed Web of Science ®Google Scholar
• Hicklin, R. A., Eisenhart, L., Richetelli, N., Miller, M. D., Belcastro, P., Burkes, T. M., Parks, C. L., Smith, M. A., Buscaglia, J., Peters, E. M., Perlman, R. S., Abonamah, J. V., and Eckenrode, B. A. (2022), “Accuracy and Reliability of Forensic Handwriting Comparisons,” Proceedings of the National Academy of Sciences of the United States of America, 119, e2119944119. DOI: 10.1073/pnas.2119944119.
   PubMed Web of Science ®Google Scholar
• Hofmann, H., Carriquiry, A., and Vanderplas, S. (2020), “Treatment of Inconclusives in the AFTE Range of Conclusions,” Law, Probability and Risk, 19, 317–364. DOI: 10.1093/lpr/mgab002.
   Web of Science ®Google Scholar
• Kaye, D. H., Antill, G., Emmerich, E., Ishida, C., Lowe, M., and Perler, R. (2022), “Toolmark-Comparison testimony: A report to the Texas Forensic Science Commission,” available at: https://papers.ssrn.com/sol3/papers.cfm?abstract/_id=4108012
   Google Scholar
• Law, E. F., and Morris, K. B. (2021), “Evaluating Firearm Examiner Conclusion Variability Using Cartridge Case Reproductions,” Journal of Forensic Sciences, 66, 1704–1720. DOI: 10.1111/1556-4029.14758.
   PubMed Web of Science ®Google Scholar
• Monson, K. L., Smith, E. D., and Bajic, S. J. (2022), “Planning, Design and Logistics of a Decision Analysis Study: The FBI/Ames Study Involving Forensic Firearms Examiners,” Forensic Science International: Synergy, 4, 100221. DOI: 10.1016/j.fsisyn.2022.100221.
   PubMedGoogle Scholar
• Monson, K. L., Smith, E. D., and Peters, E. M. (2023), “Accuracy of Comparison Decisions by Forensic Firearms Examiners,” Journal of Forensic Sciences, 68, 86–100. DOI: 10.1111/1556-4029.15152.
   PubMed Web of Science ®Google Scholar
• Mossman, D. (1999), “Three-Way ROCs,” Medical Decision Making: An International Journal of the Society for Medical Decision Making, 19, 78–89. DOI: 10.1177/0272989X9901900110.
   PubMed Web of Science ®Google Scholar
• Neuman, M., Hundl, C., Grimaldi, A., Eudaley, D., Stein, D., and Stout, P. (2022), “Blind Testing in Firearms: Preliminary Results from a Blind Quality Control Program,” Journal of Forensic Sciences, 67, 964–974. DOI: 10.1111/1556-4029.15031.
   PubMed Web of Science ®Google Scholar
• Scurfield, B. K. (1996), “Multiple-Event Forced-Choice Tasks in the Theory of Signal Detectability,” Journal of Mathematical Psychology, 40, 253–269. DOI: 10.1006/jmps.1996.0024.
   PubMed Web of Science ®Google Scholar
• Scurfield, B. K. (1998), “Generalization of the Theory of Signal Detectability to n-Event m-Dimensional Forced-Choice Tasks,” Journal of Mathematical Psychology, 42, 5–31. DOI: 10.1006/jmps.1997.1183.
   PubMed Web of Science ®Google Scholar
• Scurich, N. (2022), “Inconclusives in Firearm Error Rate Studies Are Not “a Pass,” Law, Probability and Risk, 21, 123–127. DOI: 10.1093/lpr/mgac011.
   Web of Science ®Google Scholar
• Scurich, N., and Dror, I. E. (2020), “Continued Confusion about Inconclusives and Error Rates: Reply to Weller and Morris,” Forensic Science International. Synergy, 2, 703–704. DOI: 10.1016/j.fsisyn.2020.10.005.
   PubMedGoogle Scholar
• Scurich, N., Garrett, B. L., and Thompson, R. M. (2022), “Surveying Practicing Firearm Examiners,” Forensic Science International: Synergy, 4, 100228. DOI: 10.1016/j.fsisyn.2022.100228.
   PubMedGoogle Scholar
• Scurich, N., and John, R. S. (2012), “Prescriptive Approaches to Communicating the Risk of Violence in Actuarial Risk Assessment,” Psychology, Public Policy, and Law, 18, 50–78. DOI: 10.1037/a0024592.
   Web of Science ®Google Scholar
• Smith, A. M., and Neal, T. M. (2021), “The Distinction between Discriminability and Reliability in Forensic Science,” Science & Justice: Journal of the Forensic Science Society, 61, 319–331. DOI: 10.1016/j.scijus.2021.04.002.
   PubMed Web of Science ®Google Scholar
• Stanislaw, H., and Todorov, N. (1999), “Calculation of Signal Detection Theory Measures,” Behavior Research Methods, Instruments, & Computers: A Journal of the Psychonomic Society, Inc, 31, 137–149. DOI: 10.3758/bf03207704.
   PubMedGoogle Scholar
• Swets, J. A. (1988), “Measuring the Accuracy of Diagnostic Systems,” Science, 240, 1285–1293. DOI: 10.1126/science.3287615.
   PubMed Web of Science ®Google Scholar
• Swets, J. A., Dawes, R. M., and Monahan, J. (2000), “Psychological Science Can Improve Diagnostic Decisions,” Psychological Science in the Public Interest: a Journal of the American Psychological Society, 1, 1–26. DOI: 10.1111/1529-1006.001.
   PubMedGoogle Scholar
• Swets, J. A., and Pickett, R. M. (1982), Evaluation of Diagnostic Systems: Methods from Signal Detection Theory, New York: Academic Press.
   Google Scholar
• Thompson, W. C., Vuille, J., Biedermann, A., and Taroni, F. (2013), “The Role of Prior Probability in Forensic Assessments,” Frontiers in Genetics, 4, 220–223. DOI: 10.3389/fgene.2013.00220.
   PubMedGoogle Scholar
• Ulery, B. T., Hicklin, R. A., Buscaglia, J., and Roberts, M. A. (2011), “Accuracy and Reliability of Forensic Latent Fingerprint Decisions,” Proceedings of the National Academy of Sciences of the United States of America, 108, 7733–7738. DOI: 10.1073/pnas.1018707108.
   PubMed Web of Science ®Google Scholar
• Weller, T. J., and Morris, M. D. (2020), “Commentary on: I. Dror, N Scurich “(Mis) Use of Scientific Measurements in Forensic Science,” Forensic Science International: Synergy, 2, 701. DOI: 10.1016/j.fsisyn.2020.08.006.
   PubMedGoogle Scholar
• Wixted, J. T. (2020), “The Forgotten History of Signal Detection Theory,” Journal of Experimental Psychology. Learning, Memory, and Cognition, 46, 201–233. DOI: 10.1037/xlm0000732.
   PubMed Web of Science ®Google Scholar
• Wixted, J. T., and Mickes, L. (2012), “The Field of Eyewitness Memory Should Abandon Probative Value and Embrace Receiver Operating Characteristic Analysis,” Perspectives on Psychological Science: a Journal of the Association for Psychological Science, 7, 275–278. DOI: 10.1177/1745691612442906.
   PubMedGoogle Scholar
• Wixted, J. T., and Mickes, L. (2015), “ROC Analysis Measures Objective Discriminability for Any Eyewitness Identification Procedure,” Journal of Applied Research in Memory and Cognition, 4, 329–334. DOI: 10.1016/j.jarmac.2015.08.007.
   Web of Science ®Google Scholar