Receiver Operating Characteristic Plot and Area Under the Curve With Binary Classifiers: Pragmatic Analysis of Cognitive Screening Instruments

References

• Hanley JA , McNeilBJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36 (1982).
   PubMed Web of Science ®Google Scholar
• Hanley JA , McNeilBJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843 (1983).
   PubMed Web of Science ®Google Scholar
• Fawcett T . An introduction to ROC analysis. Pattern Recognit. Lett., 27(8), 861–874 (2006).
   Web of Science ®Google Scholar
• DeLong ER , DeLongDM, Clarke-PearsonDL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837–845 (1988).
   PubMed Web of Science ®Google Scholar
• Zweig MH , CampbellG. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem., 39(4), 561–577 (1993).
   PubMed Web of Science ®Google Scholar
• Zou KH , O’MalleyAJ, MauriL. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation, 115(5), 654–657 (2007).
   PubMed Web of Science ®Google Scholar
• Metz CE . Basic principles of ROC analysis. Semin. Nucl. Med., 8(4), 283–298 (1978).
   PubMed Web of Science ®Google Scholar
• Swets JA . Measuring the accuracy of diagnostic systems. Science, 240(4857), 1285–1293 (1988).
   PubMed Web of Science ®Google Scholar
• Jones CM , AthanasiouT. Summary receiver operating characteristic curve analysis techniques in the evaluation of diagnostic tests. Ann. Thorac. Surg., 79(1), 16–20 (2005).
   PubMed Web of Science ®Google Scholar
• Walter SD . Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Stat. Med., 21(9), 1237–1256 (2002).
   PubMed Web of Science ®Google Scholar
• Muschelli J . ROC and AUC with a binary predictor: a potentially misleading metric. arXivdoi:10.1007/s00357-019-09345-1 [ stat.CO] (2019).
   Google Scholar
• Glaveckaite S , ValevicieneN, PalionisDet al. Value of scar imaging and inotropic reserve combination for the prediction of segmental and global left ventricular functional recovery after revascularisation. J. Cardiovasc. Magn. Reson., 13(1), 35 (2011).
   PubMedGoogle Scholar
• Blumberg DM , DeMoraes CG, LiebmannJMet al. Technology and the glaucoma suspect. Invest. Ophthalmol. Vis. Sci., 57(9), OCT80–OCT85 (2016).
   PubMed Web of Science ®Google Scholar
• Budweg J , SprengerT, DeVere-Tyndall A, HagenkordA, StippichC, BergerCT. Factors associated with significant MRI findings in medical walk-in patients with acute headache. Swiss Med Wkly., 146, w14349 (2016).
   Web of Science ®Google Scholar
• Litvin TV , BresnickGH, CuadrosJA, SelvinS, KanaiK, OzawaGY. A revised approach for the detection of sight-threatening diabetic macular edema. JAMA Ophthalmol., 135(1), 62–68 (2017).
   PubMed Web of Science ®Google Scholar
• Mbizvo GK , BennettKH, SchnierC, SimpsonCR, DuncanSE, ChinRFM. The accuracy of using administrative healthcare data to identify epilepsy cases: a systematic review of validation studies. Epilepsia, 61(7), 1319–1335 (2020).
   PubMed Web of Science ®Google Scholar
• Hendry K , QuinnTJ, EvansJJ, StottDJ. Informant single screening questions for delirium and dementia in acute care – a cross-sectional test accuracy pilot study. BMC Geriatr., 15, 17 (2015).
   PubMed Web of Science ®Google Scholar
• Soysal P , UsarelC, IspirliG, IsikAT. Attended with and head-turning sign can be clinical markers of cognitive impairment in older adults. Int. Psychogeriatr., 29(11), 1763–1769 (2017).
   PubMed Web of Science ®Google Scholar
• Belmin J , Pariel-MadjlessiS, SurunPet al. The cognitive disorders examination (Codex) is a reliable 3-minute test for detection of dementia in the elderly (validation study in 323 subjects). Presse Med., 36(9 Pt 1), 1183–1190 (2007).
   PubMed Web of Science ®Google Scholar
• Bonello M , LarnerAJ. Applause sign: screening utility for dementia and cognitive impairment. Postgrad. Med., 128(2), 250–253 (2016).
   PubMed Web of Science ®Google Scholar
• Ghadiri-Sani M , LarnerAJ. Head turning sign. J. R. Coll. Physicians Edinb., 49(4), 323–326 (2019).
   PubMed Web of Science ®Google Scholar
• Ziso B , LarnerAJ. Codex (cognitive disorders examination) decision tree modified for the detection of dementia and MCI. Diagnostics (Basel), 9(2), E58 (2019).
   PubMedGoogle Scholar
• Larner AJ . The “attended alone” and “attended with” signs in the assessment of cognitive impairment: a revalidation. Postgrad. Med., 132(7), 595–600 (2020).
   PubMed Web of Science ®Google Scholar
• Hsieh S , McGroryS, LeslieFet al. The Mini-Addenbrooke’s Cognitive Examination: a new assessment tool for dementia. Dement. Geriatr. Cogn. Disord., 39(1–2), 1–11 (2015).
   PubMed Web of Science ®Google Scholar
• Larner AJ . MACE for diagnosis of dementia and MCI: examining cut-offs and predictive values. Diagnostics (Basel), 9(2), E51 (2019).
   PubMedGoogle Scholar
• Larner AJ . Screening for dementia: Q* index as a global measure of test accuracy revisited. medRxiv doi:10.1101/2020.04.01.20050567 (2020) ( Epub ahead of print).
   Google Scholar
• Noel-Storr AH , McCleeryJM, RichardEet al. Reporting standards for studies of diagnostic test accuracy in dementia: the STARDdem Initiative. Neurology, 83(4), 364–373 (2014).
   PubMed Web of Science ®Google Scholar
• Mallett S , HalliganS, ThompsonM, CollinsGS, AltmanDG. Interpreting diagnostic accuracy studies for patient care. BMJ, 344, e3999 (2012).
   Google Scholar
• Larner AJ . What is test accuracy? Comparing unitary accuracy metrics for cognitive screening instruments. Neurodegener. Dis. Manag., 9(5), 277–281 (2019).
   PubMed Web of Science ®Google Scholar
• Saito T , RehmsmeierM. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432 (2015).
   PubMed Web of Science ®Google Scholar
• Larner AJ . Defining “optimal” test cut-off using global test metrics: evidence from a cognitive screening instrument. Neurodegener. Dis. Manag., 10(4), 223–230 (2020).
   PubMed Web of Science ®Google Scholar
• Altman DG , RoystonP. The cost of dichotomising continuous variables. BMJ, 332(7549), 1080 (2006).
   PubMed Web of Science ®Google Scholar
• Pedregosa F , VaroquauxG, GramfortAet al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res., 12(85), 2825–2830 (2011).
   Google Scholar
• Du Z , HaoY. Package ‘reportROC’: an easy way to report ROC analysis (2017). https://mran.microsoft.com/snapshot/2017-10-04/web/packages/reportROC/reportROC.pdf
   Google Scholar
• Khan RA . ROCit: an R package for performance assessment of binary classifier with visualization (2020). https://cran.r-project.org/web/packages/ROCit/vignettes/my-vignette.html
   Google Scholar