The total operating characteristic to measure diagnostic ability for multiple thresholds

Abstract

The relative operating characteristic (ROC) is a popular statistical method to measure the association between observed and diagnosed presence of a characteristic. The diagnosis of presence or absence depends on whether the value of an index variable is above a threshold. ROC considers multiple possible thresholds. Each threshold generates a two-by-two contingency table, which contains four central entries: hits, misses, false alarms, and correct rejections. ROC reveals for each threshold only two ratios, hits/(hits + misses) and false alarms/(false alarms + correct rejections). This article introduces the total operating characteristic (TOC), which shows the total information in the contingency table for each threshold. TOC maintains desirable properties of ROC, while TOC reveals strictly more information than ROC in a manner that makes TOC more useful than ROC. We illustrate the concepts with an application to land change science.

Keywords:

• area under the curve (AUC)
• diagnosis
• land use and land cover change (LUCC)
• relative operating characteristic (ROC)
• total operating characteristic (TOC)

Acknowledgments

The United States’ National Science Foundation (NSF) supported this research via award # OEC-1238212 entitled The PIE-LTER. Researchers at the Noel Kempff Mercado Natural History Museum developed the case study’s data from Landsat TM imagery. The data are available at www.clarku.edu/~rpontius and in the tutorial for the software Idrisi (www.clarklabs.org).