Evaluating Real-Time Probabilistic Forecasts With Application to National Basketball Association Outcome Prediction

Abstract

Motivated by the goal of evaluating real-time forecasts of home team win probabilities in the National Basketball Association, we develop new tools for measuring the quality of continuously updated probabilistic forecasts. This includes introducing calibration surface plots, and simple graphical summaries of them, to evaluate at a glance whether a given continuously updated probability forecasting method is well-calibrated, as well as developing statistical tests and graphical tools to evaluate the skill, or relative performance, of two competing continuously updated forecasting methods. These tools are demonstrated in an application to evaluate the continuously updated forecasts published by the United States-based multinational sports network ESPN on its principle webpage espn.com. This application lends statistical evidence that the forecasts published there are well-calibrated, and exhibit improved skill over several naïve models, but do not demonstrate significantly improved skill over simple logistic regression models based solely on a measurement of each teams’ relative strength, and the evolving score difference throughout the game.

Keywords:

• Brier score
• Calibration
• Functional data
• Probability forecasting
• Skill score

Supplementary Materials

The supplementary material document contains the proof of Theorem 1, rigorous definition of $p_i(t)$, a simulation study that examines the power of the proposed test, results for automatic bin selection, and a table that enumerates the discarded forecasts in Section 3.

Acknowledgments

We thank to two anonymous referees, an associate editor, and the editor Joshua M. Tebbs, for the helpful comments.

Additional information

Funding

This work has been partially supported by two Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants (GR, JAD), and the Ontario Graduate Scholarship provided by the Province of Ontario (YCK).