Classification accuracy goals for diagnostic tests based on risk stratification

Abstract

In diagnostic test evaluation, performance goals are often set for classification accuracy measures such as specificity, sensitivity and diagnostic likelihood ratio. For tests that detect rare conditions, classification accuracy goals are attractive because they can be evaluated in case-control studies enriched for the condition. A neglected area of research is determining classification accuracy goals that confer clinical usefulness of a test. We determine classification accuracy goals based on desired risk stratification, i.e. the post-test risk of having the condition compared with the pre-test risk. We determine goals for rule-out tests, rule-in tests, and those that do both. Goals for negative and positive likelihood ratios (NLR, PLR) are emphasized because of their natural relationships with risk stratification via Bayes Theorem. Goals for specificity and sensitivity are implied by goals on NLR and PLR. Goals that confer superiority or non-inferiority of a test to a comparator are based on approximating risk differences and relative risks by functions of likelihood ratios. Inference is based on Wald confidence intervals for ratios of likelihood ratios. To illustrate, we consider hypothetical data on a fetal fibronectin assay for ruling out risk of pre-term birth and two human papillomavirus assays for detecting cervical cancer.

Trial registration

ClinicalTrials.gov identifier: NCT01931566.

KEYWORDS:

• Bayes theorem
• rare disease assumption
• likelihood ratio graph
• discrimination
• medical decision making

Acknowledgements

I would like to thank Drs. Frank Samuelson and Dandan Xu for reviewing an earlier version of this manuscript. Their thoughtful comments greatly improved the structure and content of the paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Gene Pennello

Gene Pennello, PhD is a research mathematical statistician for the Division of Imaging, Diagnostics, and Software Reliability at the Food and Drug Administration (FDA). He has extensive experience diagnostic device evaluation, including in vitro diagnostic tests, diagnostic imaging systems, and diagnostic software. He is a Fellow of the American Statistical Association (ASA), a past chair of the ASA Section on Medical Devices and Diagnostics, and a past President of the FDA Statistical Association. He earned his Ph.D. in Statistics from Oregon State University and was a postdoctoral training fellow at the National Cancer Institute, Division of Cancer Epidemiology and Genetics.