Exceedance Probability Score: A Novel Measure for Comparing Probabilistic Predictions

Abstract

Accurate prediction of exceedance probabilities is important in many applications. For example, in process planning and control, engineers should anticipate the risk that a product fails to meet its specification limits. Statistical comparison between candidate probability prediction methods is commonly performed using scoring rules, like the continuous ranked probability score (CRPS) and the logarithm score (LogS). In this work, a new scoring rule, the exceedance probability score, is proposed. The experiments in simulated and real industrial data show that the new scoring rule is useful in comparing and testing differences in the predictive accuracy of competitive probabilistic predictions in regression setting. The proposed scoring rule have some similarities with CRPS and LogS, but is more directly connected to the accuracy in the prediction of exceedance probabilities.

Keywords:

• 62J99
• 62H15
• 62P30

Keywords:

• CRPS
• Density forecast
• logarithm score
• Probability prediction
• Scoring rules

Acknowledgments

We are grateful to the Finnish Funding Agency for Technology and Innovation (TEKES), Ruukki Metals Oy, Valio Oy, Ovako Bar Oy, and Infotech Oulu for funding this research.