Modeling Probability Forecasts via Information Diversity

ABSTRACT

Randomness in scientific estimation is generally assumed to arise from unmeasured or uncontrolled factors. However, when combining subjective probability estimates, heterogeneity stemming from people’s cognitive or information diversity is often more important than measurement noise. This article presents a novel framework that uses partially overlapping information sources. A specific model is proposed within that framework and applied to the task of aggregating the probabilities given by a group of forecasters who predict whether an event will occur or not. Our model describes the distribution of information across forecasters in terms of easily interpretable parameters and shows how the optimal amount of extremizing of the average probability forecast (shifting it closer to its nearest extreme) varies as a function of the forecasters’ information overlap. Our model thus gives a more principled understanding of the historically ad hoc practice of extremizing average forecasts. Supplementary material for this article is available online.

KEYWORDS:

• Expert belief
• Gaussian process
• Judgmental forecasting
• Model averaging
• Noise reduction

Supplementary Materials

Technical Details: The supplementary material includes (a) proofs for Propositions 3.1, 4.1, 4.2, and 5.1; (b) derivation of Equation (4(4) $$\begin{array}{c}\hfill \delta \in [0,1]\text{and}\lambda |\delta \in \left[max\left\{\frac{N-{\delta }^{-1}}{N-1},0\right\},1\right].\end{array}$$(4) ); and (c) instructions on how to estimate the model parameters under symmetric information. (.pdf file)

Acknowledgments

The authors would like to thank Edward George and Shane Jensen for helpful discussions.

Funding

This research was supported in part by NSF grant # DMS-1209117 and a research contract to the University of Pennsylvania and the University of California from the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior National Business Center contract number D11PC20061. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions expressed herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.