Lessons from an escalation prediction competition

Figures & data

Table 1. Partitioning of data for estimating model weights, hyperparameter tuning, evaluation, and forecasting.

		Periodization
		Evaluation	True forecast
Training period	From:	τ = 121 (January 1990)	τ = 121 (January 1990)
	To:	τ = 408 (December 2013)	τ = 444 (December 2016)
Calibration period	From:	τ = 409 (January 2014)	τ = 445 (January 2017)
	To:	τ = 444 (December 2016)	τ = 488 (August 2020)
Testing/forecasting period	From:	τ = 445 (January 2017)	τ = 490 (October 2020)
	To:	τ = 480 (December 2019)	τ = 495 (March 2021)

The “evaluation” periodization is for testing models and ensembles the “true forecast” periodization for actual forecasting. The training set is used to train models and the calibration periods for hyperparameter tuning and estimating model weights. For true forecasting, we have fixed the calibration period to end at the last available month of UCDP-Candidate data release in time for the competition, August 2020. The third period, in turn, is for true forecasts or for out-of-sample evaluation of these, respectively. The true forecasting period starts in October 2020, whereas the testing period in the evaluation periodization commences immediately after the end of the calibration period. See Hegre et al. (2021) for more details on periodization.

Table 2. Evaluation scores for the benchmark and no-change models by step, cm.

Step	MSE benchmark	MSE no-change	TADDA benchmark	TADDA no-change
2	0.736	0.674	0.535	0.340
3	0.727	0.773	0.520	0.365
4	0.689	0.841	0.501	0.391
5	0.793	0.807	0.554	0.375
6	0.748	0.841	0.520	0.384
7	0.788	0.864	0.533	0.389

Table 3. Evaluation scores for the benchmark and no-change models by step, pgm.

Step	MSE benchmark	MSE no-change	TADDA benchmark	TADDA no-change
2	0.045	0.032	0.138	0.014
3	0.046	0.033	0.140	0.014
4	0.050	0.033	0.151	0.014
5	0.048	0.033	0.142	0.015
6	0.050	0.034	0.151	0.015
7	0.052	0.034	0.151	0.015

Figure 1. Maps of predicted change from September 2020 through March 2021, cm.

Figure 2. Maps of predicted change from September 2020 through March 2021, pgm.

Figure 3. Maps of observed change in fatalities from September 2020 to October 2020 (left), and March 2021 (right), cm (top), and pgm (bottom).

Figure 4. Predicted change from September 2020 through March 2021, for selected countries.

Figure 5. Predicted change from September 2020 through March 2021, for selected groups of grid cells (bottom).

Figure 6. Maps of predicted change from August 2020 through March 2021, weighted ensemble of the contributions, cm (left), and pgm (right).

Hegre, Håvard, Curtis Bell, Michael Colaresi, Mihai Catalin Croicu, Frederick Hoyles, Remco Jansen, Maxine Ria Leis, et al. 2021. “ViEWS: Revising and Evaluating the ViEWS Political Violence Early-Warning System.” Journal of Peace Research 58 (3): 599–611. doi:10.1177/0022343320962157.

 Web of Science ®Google Scholar