Evaluating Binary Classifiers: Extending the Efficiency Index

Abstract

Aim: To further develop an ‘efficiency index’ (EI), the ratio of classifier accuracy to inaccuracy, to construct balanced EI (BEI) and unbiased EI (UEI) measures to evaluate a dementia screening test. EI formulations were compared with cognate formulations of the identification index (II). Materials & methods: A prospective pragmatic test accuracy study dataset examining Mini-Addenbrooke’s Cognitive Examination (MACE) was used. Results: EI, BEI and UEI varied with test cutoff. UEI was the most stringent measure, correcting for both disease prevalence and test threshold. Unlike II formulations, the boundary values of EI formulations (0,∞) ensure that negative values never occur. Conclusion: EI metrics may be useful for the evaluation of cognitive screening instruments and other diagnostic tests used for neurodegenerative disorders.

Plain language summary

The efficiency index (EI) is a recently introduced measure to evaluate tests used in medical diagnosis. It measures the likelihood of whether a test gives a correct (accurate) rather than incorrect (inaccurate) result. The result of the EI is easily interpreted: the higher the EI, the more likely the test gives the correct diagnosis. However, the measures upon which EI is based, accuracy and inaccuracy, can be influenced by factors such as the prevalence of disease in the population under investigation or the chosen test threshold. These biases can be to some extent factored out by using other measures, balanced accuracy and unbiased accuracy, which form the basis for extending EI to balanced (BEI) and unbiased (UEI) forms Using information from a study of a screening test for dementia, comparison of the various EI measures showed that BEI and UEI may be preferable to EI in the evaluation of diagnostic tests.

Keywords:

• binary classifier
• cognitive screening
• efficiency indexes
• identification indexes
• neurodegenerative disease

Financial & competing interests disclosure

The author has no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.