JUDGING JUDGMENT

Abstract

Philip E. Tetlock and I agree that forecasting tools are best evaluated in peer-reviewed settings and in comparison not only to expert judgments, but also to alternative modeling strategies. Applying his suggested standards of assessment, however, certain forecasting models not only outperform expert judgments, but also have gone head-to-head with alternative models and outperformed them. This track record demonstrates the capability to make significant, reliable predictions of difficult, complex events. The record has unfolded, contrary to Tetlock's contention, not only in government and business applications, but also in numerous peer-reviewed publications containing hundreds of real-time forecasts. Moreover, reliable prediction is achieved while avoiding significant false-positive or false-negative rates.

Notes

1. Consider, for instance, what a player looking four or five iterations beyond the idea of randomly distributed “votes,” with an average value of 50, would have submitted as the answer. Rather than doing the infinite regress—being, in Tetlock's terms, clever but not “too clever by half”—such a player might have deviated from the “too clever” answer of 0 and assumed that errors or sophisticated spreads would have produced a “winning” answer that was the average of only 4 or 5 regresses back from 50. If players, trying to get better odds of winning (that is, improving over 1/X), assumed others were doing the same and so randomized (playing mixed strategies) over the average of the first cut answer (2/3 of 50=33 1/3), and then each of the next 4 (2/3 of 33.333; 2/3*2/3*33.333; 2/3*2/3*2/3*33.333; and 2/3*2/3*2/3*2/3*33.333, then their answers would have been between 20.06 (4th regress) and 17.37 (5th regress), the average of which is 18.715; that is, just about the winning answer of 18. This example assumes a limit imposed on the potentially infinite regress leading to the Nash equilibrium value of 0, but it does illustrate how a clever but not “too clever by half” player could reasonably and strategically have ended up with 18 as the answer.

2. These questions were studied during specific time periods, of course, and made predictions for specific periods in the future. These details are left out here and in the Feder and the Ray and Russett studies. For the studies in peer-reviewed outlets the reader can check on these additional details.

3. The years of these predictions are classified. The CIA has declassified the fact that these questions were asked, but not when they were asked.

4. For example, Beck and Bueno de Mesquita 1985, 103–122,; Bueno de Mesquita 1990; Bueno de Mesquita and Iusi-Scarborough 1988; Bueno de Mesquita and Kim 1991; Bueno de Mesquita and Organski [1990] 1992; James 1998; Kugler 1987; Morrow, Bueno de Mesquita, and Wu 1993; Newman and Bridges 1994; Organski and Bueno de Mesquita 1993; Wu and Bueno de Mesquita 1994; Bueno de Mesquita and Stokman 1994; Kugler, Snider, and Longwell 1994.

5. The finished manuscript was submitted in the late summer of 2008 and the copyedited manuscript was ready for the printer around the beginning of summer, 2009.

6. There is some attrition in cases, though not very much, as several countries ceased to exist between 1980 and 2008, a question I did not address in these forecasts. I treated Russia as the successor state to the Soviet Union, and Germany as the successor state to East and West Germany (counting them as one country after unification).

7. The absolute weighted mean percentage error is calculated as ∣Predicted – Observed∣ with the weighted mean outcome computed as.

Additional information

Notes on contributors

Bruce Bueno de Mesquita

Bruce Bueno de Mesquita is the author, inter alia, of, The Predictioneer's Game (Random House, 2009)