A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators

References

• Bandos , A. I. , Rockette , H. E. , Gur , D. ( 2007 ). Exact bootstrap variances of the area under the ROC curve . Commun. Statist. A Theor. 36 ( 13 ): 2443 – 2461 .
   Web of Science ®Google Scholar
• Barrett , H. H. , Kupinski , M. A. , Clarkson , E. ( 2005 ). Probabilistic foundations of the MRMC method . In: Eckstein , M. P. , Jiang , Y. , eds. Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment . Vol. 5749 . Proc. SPIE , pp. 21 – 31 .
   Google Scholar
• Beiden , S. V. , Wagner , R. F. , Campbell , G. ( 2000 ). Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis . Acad. Radiol. 7 ( 5 ): 341 – 349 .
   PubMed Web of Science ®Google Scholar
• Beiden , S. V. , Wagner , R. F. , Campbell , G. , Metz , C. E. , Jiang , Y. ( 2001 ). Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities . Acad. Radiol. 8 ( 7 ): 605 – 615 .
   PubMed Web of Science ®Google Scholar
• Clarkson , E. , Kupinski , M. A. , Barrett , H. H. ( 2006 ). A probabilistic model for the MRMC method. Part 1. Theoretical development . Acad. Radiol. 13 ( 11 ): 1410 – 1421 .
   PubMed Web of Science ®Google Scholar
• Davison , A. C. , Hinkley , D. V. (1997). Bootstrap Methods and Their Application . Cambridge : Cambridge University Press.
   Google Scholar
• Dorfman , D. D. , Alf , E. ( 1969 ). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data . J. Math. Psychol. 6 : 487 – 496 .
   Web of Science ®Google Scholar
• Dorfman , D. D. , Berbaum , K. S. , Metz , C. E. ( 1992 ). Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method . Invest. Radiol. 27 ( 9 ): 723 – 731 .
   PubMed Web of Science ®Google Scholar
• Dorfman , D. D. , Berbaum , K. S. , Lenth , R. V. ( 1995 ). Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis . Acad. Radiol. 2 ( 7 ): 626 – 633 .
   PubMed Web of Science ®Google Scholar
• Efron , B. ( 1982 ). The Jackknife, the Bootstrap and Other Resampling Plans . Philadelphia, PA : Society for Industrial and Applied Mathematics .
   Google Scholar
• Efron , B. , Tibshirani , R. J. ( 1993 ). An Introduction to the Bootstrap: Monographs on Statistics and Applied Probability . New York : Chapman & Hall .
   Google Scholar
• Gallas , B. D. ( 2006 ). One-shot estimate of MRMC variance: AUC . Acad. Radiol. 13 ( 3 ): 353 – 362 .
   PubMed Web of Science ®Google Scholar
• Gallas , B. D. , Brown , D. G. ( 2008 ). Reader studies for validation of CAD systems . Neur. Netw. 21 ( 2–3 ): 387 – 397 .
   PubMed Web of Science ®Google Scholar
• Gallas , B. D. , Pennello , G. A. , Myers , K. J. ( 2007 ). Multi-reader multi-case variance analysis for binary data . J. Opt. Soc. Amer. A 24 ( 12 ): B70 – B80 .
   Google Scholar
• Hillis , S. L. ( 2007 ). A comparison of denominator degrees of freedom methods for multiple observer ROC analysis . Statist. Med. 26 ( 3 ): 596 – 619 .
   PubMed Web of Science ®Google Scholar
• Hillis , S. L. , Obuchowski , N. A. , Schartz , K. M. , Berbaum , K. S. ( 2005 ). A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data . Statist. Med. 24 ( 10 ): 1579 – 1607 .
   PubMed Web of Science ®Google Scholar
• Jiang , Y. , Nishikawa , R. M. , Schmidt , R. A. , Metz , C. E. , Giger , M. L. , Doi , K. ( 1999 ). Improving breast cancer diagnosis with computer-aided diagnosis . Acad. Radiol. 6 ( 1 ): 22 – 33 .
   PubMed Web of Science ®Google Scholar
• Kupinski , M. A. , Clarkson , E. , Barrett , H. H. ( 2006 ). A probabilistic model for the MRMC method. Part 2. Validation and applications . Acad. Radiol. 13 ( 11 ): 1422 – 1430 .
   PubMed Web of Science ®Google Scholar
• Obuchowski , N. A. ( 1995a ). Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations . Acad. Radiol. 2 ( Suppl 1 ): S22 – S29 .
   PubMedGoogle Scholar
• Obuchowski , N. A. ( 1995b ). Multireader receiver operating characteristic studies: a comparison of study designs . Acad. Radiol. 2 ( 8 ): 709 – 716 .
   PubMed Web of Science ®Google Scholar
• Obuchowski , N. A. , Rockette , H. E. ( 1995 ). Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations . Commun. Statist. Simul. 24 ( 2 ): 285 – 308 .
   Web of Science ®Google Scholar
• Obuchowski , N. A. , Beiden , S. V. , Berbaum , K. S. , Hillis , S. L. , Ishwaran , H. , Song , H. H. , Wagner , R. F. ( 2004 ). Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods . Acad. Radiol. 11 ( 9 ): 980 – 995 .
   PubMed Web of Science ®Google Scholar
• Randles , R. H. , Wolfe , D. A. ( 1979 ). Introduction to the Theory of Nonparametric Statistics . New York : John Wiley and Sons .
   Google Scholar
• Roe , C. A. , Metz , C. E. ( 1997a ). Dorfman–Berbaum–Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation . Acad. Radiol. 4 : 298 – 303 .
   PubMed Web of Science ®Google Scholar
• Roe , C. A. , Metz , C. E. ( 1997b ). Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates . Acad. Radiol. 4 : 587 – 600 .
   PubMed Web of Science ®Google Scholar
• Song , X. , Zhou , X.-H. ( 2005 ). A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data . Biostatistics 6 ( 2 ): 303 – 312 .
   PubMed Web of Science ®Google Scholar
• Swets , J. A. , Pickett , R. M. ( 1982 ). Evaluation of Diagnostic Systems: Methods from Signal Detection Theory . New York : Academic Press .
   Google Scholar