Number needed to test: quantifying risk stratification provided by diagnostic tests and risk predictions

Abstract

Risk stratification is the ability of a test or model to separate those at high vs. low risk of disease. There is no risk stratification metric that is in terms of the number of people requiring testing, which would help with considering the benefits, harms, and costs associated with the test and interventions. We introduce the expected number needed to test (NNtest) to identify one more disease case than by randomly selecting people for disease ascertainment. We show that NNtest measures risk stratification, allowing us to decompose NNtest into components that contrast the increase in risk upon testing positive (‘concern’) versus the decrease in risk upon testing negative (‘reassurance’). A graph of the reciprocals of concern vs. reassurance have linear contours of constant NNtest, visualizing the relative importance and tradeoff of each component to better understand the properties of risk thresholds with equal NNtest. We apply NNtest to the controversy over the risk threshold for who should get testing for BRCA1/2 mutations that cause high risks of breast and ovarian cancers. We show that risk thresholds between 0.78% and 5% optimize NNtest. At these thresholds, $NNtest\approx 120$ people will require risk-model evaluation to find one more mutation-carrier. However, these thresholds of equal NNtest provide very different concern and reassurance, with 0.78% providing much more reassurance (and thus much less concern) than 5%. Given that genetic testing costs are declining rapidly, the greater reassurance provided by the 0.78% threshold might be deemed more important than the greater concern provided by the 5% threshold.

Keywords:

• AUC
• diagnostic testing
• mean risk stratification
• number needed to treat
• risk prediction

Acknowledgments

This research was supported, in part, by the Intramural Research Program of the NIH/NCI. Dr. Saha-Chaudhuri and Mr. Dey are supported by FRQS salary award and NSERC discovery grant RGPIN-2017-06100. We are indebted to our late mentor and friend Sholom Wacholder for pointing out that that an expression we had derived is a harmonic mean. We thank Christine Fermo and Sue Pan for helping develop the Risk Stratification webtool http://analysistools.nci.nih.gov/biomarkerTools

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Canadian Network for Research and Innovation in Machining Technology, Natural Sciences and Engineering Research Council of Canada [ RGPIN-2017-06100] and Division of Cancer Epidemiology and Genetics, National Cancer Institute [none] and Fonds de Recherche du Québec-Société et Culture [none]. (http://analysistools.nci.nih.gov/biomarkerTools).

Notes on contributors

Hormuzd A. Katki

Hormuzd Katki is a Senior Investigator in the Division of Cancer Epidemiology and Genetics of the US National Cancer Institute.

Rajib Dey

Rajib Dey is a Doctoral student of Biostatistics at the Department of Epidemiology, Biostatistics, and Occupational Health at McGill University, Montreal, Canada.

Paramita Saha-Chaudhuri

Paramita Saha-Chaudhuri is an Associate Professor at the Department of Mathematics and Statistics, University of Vermont, Burlington, Vermont, USA.