A Prediction Tournament Paradox

Abstract

In a prediction tournament, contestants “forecast” by asserting a numerical probability for each of (say) 100 future real-world events. The scoring system is designed so that (regardless of the unknown true probabilities) more accurate forecasters will likely score better. This is true for one-on-one comparisons between contestants. But consider a realistic-size tournament with many contestants, with a range of accuracies. It may seem self-evident that the winner will likely be one of the most accurate forecasters. But, in the setting where the range extends to very accurate forecasters, simulations show this is mathematically false, within a somewhat plausible model. Even outside that setting the winner is less likely than intuition suggests to be one of the handful of best forecasters. Though implicit in recent technical papers, this paradox has apparently not been explicitly pointed out before, though is easily explained. It perhaps has implications for the ongoing IARPA-sponsored research programs involving forecasting.

Keywords:

• Bradley–Terry model
• Brier score
• Competition
• Forecasting
• Probability assessment

Acknowledgments

I thank Seth Goldstein, Don Moore, Jens Witkowski, and two anonymous referees for valuable comments on an earlier draft.

Notes

¹In fact tournaments use Brier score, which is just $2\times$ the squared error, with modifications for multiple-choice questions.

²Asking whether “close to the true probabilities” is true in practice leads to basic issues in the philosophy of the meaning of true probabilities, not addressed here.

Additional information

Funding

The author gratefully acknowledges support by N.S.F. grant DMS-1504802.