Comparing Performance of Multiclass Classification Systems with ROC Manifolds: When Volume and Correct Classification Fails

References

• Dreiseitl, S., Ohno-Machado, L., Binder, M. (2000). Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making 20:323–331.
   PubMed Web of Science ®Google Scholar
• Edwards, D.C., Metz, C.E., Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-clcass classification. IEEE Tranactions on Medical Imaging 23:891–895.
   PubMed Web of Science ®Google Scholar
• Edwards, D.C., Metz, C.E., Nishikawa, R.M. (2005). The hypervolume under the ROC hypersurface of “near-guessing” and “near-perfect” observers in N-class classification tasks. IEEE Transactions on Medical Imaging 24:293–299.
   PubMed Web of Science ®Google Scholar
• Ferri, C., Hernandez-Orallo, J., Salido, M.A. (2003). Volume Under the ROC Surface for Multi-class Problems. Proceedings of the 14th European Conference on Machine Learning (ECML '03), Cavtat-Dubrovnik, Croatia, September 22–26, LNAI 2837. Springer-Verlag, pp. 108–120.
   Google Scholar
• Hanley, J.A., McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36.
   PubMed Web of Science ®Google Scholar
• He, X., Frey, E.C. (2007). The meaning and use of the volume under a three-class ROC surface (VUS). IEEE Transactions on Medical Imaging 27:577–588.
   Web of Science ®Google Scholar
• He, X., Metz, C.E., Tsui, B.M. W., Links, J.M., Frey, E.C. (2006). Three-class ROC analysis a decision theoretic approach under the ideal observer framework. IEEE Transactions on Medical Imaging 25:571–581.
   PubMed Web of Science ®Google Scholar
• Heckerling, P.S. (2001). Parametric three-way receiver operating characteristic surface analysis using mathematica. Medical Decision Making 21:409–417.
   PubMed Web of Science ®Google Scholar
• Landgrebe, T., Duin, R. (2007). A simplified volume under the ROC hypersurface. SAIEE Africa Research Journal 98:94–100.
   Google Scholar
• Li, J., Fine, J.P. (2008). ROC analysis with multiple classes and multiple tests: Methodology and its application in microarray studies. Biostatistics 9:566–576.
   PubMed Web of Science ®Google Scholar
• Mas, V.R., Mas, L.A., Archer, K.J., Yanek, K., King, A.L., Gibney, E.M., Cotterell, A., Fisher, R.A., Posner, M., Maluf, D.G. (2007). Evaluation of gene panel mRNAs in urine samples of kidney transplant recipients as a non-invasive tool of graft function. Molecular Medicine 13:315–324.
   PubMed Web of Science ®Google Scholar
• Mas, V.R., Mueller, T.F., Archer, K.J., Maluf, D.G. (2011). Identifying biomarkers as diagnostic tools in kidney transplantation. Expert Reviews in Molecular Diagnostics 11:183–196.
   PubMed Web of Science ®Google Scholar
• McClish, D.K. (2007). ROC volumes—Should they be used? Biostatistics 49:665–666.
   Google Scholar
• Mossman, D. (1999). Three-way ROCs. Medical Decision Making 19:78–89.
   PubMed Web of Science ®Google Scholar
• Nakas, C.T., Alonzo, T.A. (2007). ROC graphs for assessing the ability of a diagnostic marker to detect three disease classes with an umbrella ordering. Biometrics 63:603–609.
   PubMed Web of Science ®Google Scholar
• Nakas, C.T., Alonzo, T.A., Yiannoutsos, C.T. (2010). Accuracy and cut-off point selection in three-class classification problems using a generalization of the Youden Index. Statistics in Medicine 29:2946–2955.
   PubMed Web of Science ®Google Scholar
• Nakas, C.T., Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine 23:3437–3449.
   PubMed Web of Science ®Google Scholar
• Perkins, N.J., Schisterman, E.F. (2006). The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology 163:670–675.
   PubMed Web of Science ®Google Scholar
• Sampat, M.P., Patel, A.C., Wang, Y., Gupta, S., Kan, C., Bovik, A.C., Markey, M.K. (2009). Indexes for three-class classification performance assessment—An empirical comparison. IEEE Transactions on Information Technology in Biomedicine. 13:300–312.
   PubMed Web of Science ®Google Scholar
• Schubert, C.M., Thorsen, S.N., Oxley, M.E. (2011). The ROC manifold for classification systems. Pattern Recognition 44:350–362.
   Web of Science ®Google Scholar
• Scurfiled, B.K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. Journal of Mathematical Psychology 40:253–269.
   PubMed Web of Science ®Google Scholar
• Skaltsa, K., Jover, L., Carrasco, J.L. (2010). Estimation of the diagnostic threshold accounting for decision costs and sampling uncertainty. Biometrical Journal 52:676–697.
   PubMed Web of Science ®Google Scholar
• Thorsen, S.N., Oxley, M.E. (2006). A description of competing fusion systems. International Journal of Information Fusion 7:347–360.
   Google Scholar
• Xiong, C., van Belle, G., Miller, J.P., Morris, J.C. (2006). Measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups. Statistics in Medicine 25:1251–1273.
   PubMed Web of Science ®Google Scholar
• Youden, W.J. (1950). Index for rating diagnostic tests. Cancer 3:32–35.
   PubMed Web of Science ®Google Scholar
• Yu, T. (2012). ROCS: Receiver operating characteristic surface for class-skewed high-throughput data. PLoS One 7(7):e40598.
   PubMed Web of Science ®Google Scholar