Novel method of estimating AROC using an injury risk curve for biomechanical injury metric selection

ABSTRACT

Objective: Area under the receiver operating characteristic (AROC) is commonly used to evaluate an injury metric's ability to discriminate between injury and noninjury cases. However, AROC has limitations and may not handle censored data sets adequately. Survival methodology creates robust estimates of injury risk curves (IRCs) which accommodate censored data. We developed an observation-adjusted ROC (oaROC), an AROC-like statistic calculated from the IRC.

Methods: oaROC uses an observational distribution and an IRC to measure true positive rate (TPR) and false positive rate (FPR). The oaROC represents what the AROC would be with a large number of observations sampled from the IRC. We verified this using a limit test with simulated data sets at various sample sizes drawn from an assumed “true” IRC. For each sample size, 5,000 different data sets were created; a conventional AROC was calculated for each data set and compared with the single oaROC, which was calculated from the “true” IRC and not dependent on sample size.

Results: The oaROC, calculated from the simulated IRC, was 0.911. At a sample size of 20, the mean AROC was 0.930 (2.0% difference). At a sample size of 1,000, the mean AROC was 0.9114 (0.02% difference).

Conclusion: We verified that AROC approaches the oaROC with increasing sample sizes, and oaROC presents a measure of IRC discriminatory ability. Survival methodology can estimate IRCs using censored observations and the oaROC was designed with this in mind. The oaROC may be a useful measure of discrimination for data sets containing censored data. Further investigation is needed to evaluate oaROC calculated from estimated IRCs.

KEYWORDS:

• Injury risk
• AROC
• AUC
• IRC
• metric selection

Additional information

Funding

Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award UL1TR001420. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.