A statistical review: why average weighted accuracy, not accuracy or AUC?

Abstract

Sensitivity and specificity are key aspects in evaluating the performance of diagnostic tests. Accuracy and AUC are commonly used composite measures that incorporate sensitivity and specificity. Average Weighted Accuracy (AWA) is motivated by the need for a statistical measure that compares diagnostic tests from the medical costs and clinical impact point of view, while incorporating the relevant prevalence range of the disease as well as the relative importance of false-positive versus false-negative cases. We illustrate the testing procedures in four different scenarios: (i) one diagnostic test vs. the best random test, (ii) two diagnostic tests from two independent samples, (iii) two diagnostic tests from the same sample, and (iv) more than two diagnostic tests from different or the same samples. The impacts of sample size, prevalence, and relative importance on power and average medical costs/clinical loss are examined through simulation studies. Accuracy has the highest power while AWA provides a consistent criterion in selecting the optimal threshold and better diagnostic tests with direct clinical interpretations. The use of AWA is illustrated on a three-arm clinical trial evaluating three different assays in detecting Neisseria gonorrhoeae and Chlamydia trachomatis in the rectum and pharynx.

KEYWORDS:

• Diagnostic yield
• cost-utility
• diagnostic tests
• optimal threshold
• pragmatic assessment

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Institute of Allergy and Infectious Diseases, NIH [award number UM1AI104681]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (NIH).

Notes on contributors

Yunyun Jiang

Yunyun Jiang, Assistant Research Professor of Epidemiology and Biostatistics, Milken Institute School of Public Health at George Washington University. She is a biostatistician for the Antibacterial Resistance Leadership Group (ARLG) research network, a network that develops, designs and implements the transformational trials to change clinical practice and reduce the impact of antibacterial resistance and antimicrobial resistance. Her current research focuses on the development of novel statistical methodologies that will advance the ARLG mission, specifically innovative approaches to design, monitoring, and analyses of antibacterial studies (clinical trials, diagnostic studies, and master protocols). She received her Ph.D. in Biostatistics from Medical University of South Carolina, with her dissertation research focus on the design and implementation of Bayesian response adaptive randomization in phase III confirmatory clinical trials. Her research interests include clinical trial design, conduct and analysis.

Qing Pan

Qing Pan is tenured professor of Statistics at George Washington University. She received a B.S. in Biotechnology from Beijing University in 2000, an M.S. in Statistics from University of Georgia in 2003 and a Ph.D. in Biostatistics from University of Michigan in 2007. Dr. Pan’s research focuses on novel statistical and machine learning methods with applications in biostatistics and bioinformatics including survival analysis, electronic health records, network analysis and “omics” data. She was/is an important investigator in the Prospective Payment System for End Stage Renal Disease, Scientific Registry of Transplant Recipients, Diabetes Prevention Program, Antibacterial Resistance Leadership Group.

Ying Liu

Ying Liu obtained her Ph. D in applied statistics from UC Riverside. Her thesis topic is MCMC and Bayesian analyses. Then she worked with Dr. Scott Evans as a research associate at the center for Biostatistics in AIDS research at Harvard, during which she did research on sequentially multiple assignment randomized trials (SMART) and diagnostic studies. Currently she works for Biogen on Alzheimer's disease. Her research interests include propensity score methods, longitudinal analyses and Bayesian analyses.

Scott Evans

Scott Evans is a Professor of Epidemiology and Biostatistics and the Director of the George Washington University Biostatistics Center. Professor Evans interests include the design, monitoring, analyses, and reporting of and education in clinical trials and diagnostic studies. He is the author of more than 100 peer-reviewed publications and three textbooks on clinical trials including Fundamentals for New Clinical Trialists. He is the Director of the Statistical and Data Management Center (SDMC) for the Antibacterial Resistance Leadership Group (ARLG), a collaborative clinical research network that prioritizes, designs, and executes clinical research to reduce the public health threat of antibacterial resistance. Professor Evans is a member of the Board of Directors for the American Statistical Association (ASA) and the Society for Clinical Trials (SCT) and is a former member of the Board for the Mu Sigma Rho (the National Honorary Society for Statistics). He is a member of an FDA Advisory Committee, the Steering Committee of the Clinical Trials Transformation Initiative (CTTI), and serves as the Chair of the Trial of the Year Committee of the SCT. Professor Evans is the Editor-in-Chief of CHANCE and Statistical Communications in Infectious Diseases (SCID), and the Co-Editor of a Special Section of Clinical Infectious Diseases (CID) entitled Innovations in Design, Education, and Analysis (IDEA). Dr. Evans is a recipient of the Mosteller Statistician of the Year Award, the Robert Zackin Distinguished Collaborative Statistician Award, and is a Fellow of the American Statistical Association (ASA) and the Society for Clinical Trials (SCT).