A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators

Abstract

In this article, we analyze the three-way bootstrap estimate of the variance of the reader-averaged nonparametric area under the receiver operating characteristic (ROC) curve. The setting for this work is medical imaging, and the experimental design involves sampling from three distributions: a set of normal and diseased cases (patients), and a set of readers (doctors). The experiment we consider is fully crossed in that each reader reads each case. A reading generates a score that indicates the reader's level of suspicion that the patient is diseased. The distribution of scores for the normal patients is compared to the distribution of scores for the diseased patients via an ROC curve, and the area under the ROC curve (AUC) summarizes the reader's diagnostic ability to separate the normal patients from the diseased ones. We find that the bootstrap estimate of the variance of the reader-averaged AUC is biased, and we represent this bias in terms of moments of success outcomes. This representation helps unify and improve several current methods for multi-reader multi-case (MRMC) ROC analysis.

Keywords:

• Bias
• Multi-reader Multi-case (MRMC)
• Nonparametric AUC
• ROC analysis
• Three-way bootstrap
• Wilcoxon–Mann–Whitney statistic

Mathematics Subject Classification:

• Primary 62G05
• Secondary 62G09, 62C99, 91E45, 92C55, 62J10

Acknowledgments

The authors would like to thank Dr. Yulei Jiang for providing us with his high-quality dataset comparing radiologists with and without a computer aid to classify microcalcification clusters. We'd also like to thank the many colleagues at each institution that have contributed their time and expertise in discussions related to this work. This work is supported in part by grants # EB002106 and EB001694 to the University of Pittsburgh from the National Institute for Biomedical Imaging and Bioengineering, National Institutes of Health, Department of Health and Human Services.

Notes

Note: All estimates above are preceded by .