Combining p-values for Multivariate Predictive Ability Testing

Figures & data

Table 1 Test size.

		$\alpha =1\%$				$\alpha =5\%$				$\alpha =10\%$
	n:	10	50	100	200	10	50	100	200	10	50	100	200
Panel A: ${\sigma }_a=0$
$P_{r,n}$	CW	0.002	0.004	0.004	0.005	0.023	0.021	0.013	0.019	0.035	0.030	0.049	0.050
	DM	0.009	0.015	0.008	0.011	0.043	0.049	0.050	0.044	0.077	0.081	0.102	0.112
	GW	0.012	0.009	0.003	0.009	0.058	0.047	0.045	0.043	0.103	0.096	0.095	0.106
SF	CW	0.008	0.004	0.002	0.000	0.044	0.016	0.002	0.000	0.059	0.023	0.014	0.001
	DM	0.300	0.975	1.000	1.000	0.546	0.996	1.000	1.000	0.720	0.999	1.000	1.000
	GW	0.075	0.332	0.563	0.875	0.215	0.546	0.777	0.962	0.311	0.671	0.865	0.979
	MGW	0.116	0.747	0.991	1.000	0.357	0.912	0.998	1.000	0.551	0.972	1.000	1.000
Panel B: ${\sigma }_a=0.5$
$P_{r,n}$	CW	0.002	0.004	0.004	0.005	0.023	0.021	0.013	0.019	0.035	0.030	0.049	0.050
	DM	0.009	0.015	0.008	0.011	0.043	0.049	0.050	0.044	0.077	0.081	0.102	0.112
	GW	0.012	0.009	0.003	0.009	0.058	0.047	0.045	0.043	0.103	0.096	0.095	0.106
SF	CW	0.008	0.004	0.002	0.000	0.044	0.016	0.002	0.000	0.059	0.023	0.014	0.001
	DM	0.300	0.975	1.000	1.000	0.546	0.996	1.000	1.000	0.720	0.999	1.000	1.000
	GW	0.075	0.332	0.563	0.875	0.215	0.546	0.777	0.962	0.311	0.671	0.865	0.979
	MGW	0.116	0.747	0.991	1.000	0.357	0.912	0.998	1.000	0.551	0.972	1.000	1.000
Panel C: ${\sigma }_a=2$
$P_{r,n}$	CW	0.005	0.003	0.006	0.003	0.029	0.022	0.018	0.019	0.051	0.031	0.045	0.035
	DM	0.010	0.014	0.004	0.012	0.049	0.053	0.049	0.033	0.078	0.083	0.071	0.092
	GW	0.005	0.002	0.008	0.004	0.059	0.040	0.044	0.034	0.129	0.091	0.069	0.102
SF	CW	0.019	0.029	0.038	0.063	0.041	0.051	0.062	0.077	0.077	0.067	0.072	0.092
	DM	0.290	0.940	0.999	1.000	0.540	0.990	1.000	1.000	0.701	0.998	1.000	1.000
	GW	0.083	0.346	0.487	0.668	0.223	0.496	0.675	0.806	0.327	0.625	0.730	0.834
	MGW	0.190	0.891	0.998	1.000	0.474	0.973	1.000	1.000	0.636	0.994	1.000	1.000
Panel D: ${\sigma }_a=5$
$P_{r,n}$	CW	0.005	0.004	0.007	0.002	0.019	0.018	0.019	0.019	0.034	0.035	0.031	0.014
	DM	0.009	0.012	0.009	0.005	0.027	0.033	0.030	0.023	0.052	0.064	0.051	0.043
	GW	0.009	0.006	0.010	0.002	0.058	0.038	0.040	0.031	0.093	0.090	0.067	0.054
SF	CW	0.051	0.122	0.130	0.154	0.084	0.138	0.158	0.178	0.121	0.155	0.189	0.182
	DM	0.285	0.710	0.919	0.997	0.489	0.885	0.976	1.000	0.581	0.952	1.000	1.000
	GW	0.125	0.365	0.448	0.507	0.261	0.471	0.511	0.581	0.326	0.508	0.556	0.609
	MGW	0.665	0.992	1.000	1.000	0.844	1.000	1.000	1.000	0.925	1.000	1.000	1.000

NOTE: The table reports the size of intersection-union test ($P_{r,n}$) for different subtests (CW, DM, GW) and different degrees of dependence (σ_a). The Fisher statistic (S_F) is provided as a comparison. Results obtained through 5000 Monte Carlo iterations. CW stands for Clark and West (2007), DM for Diebold and Mariano (1995), GW for Giacomini and White (2006), and MGW for the multivariate GW test. To compute the latter, loss differentials were averaged in higher dimensions, as described in the supplementary material, to make the covariance matrix invertible.

Fig. 1 Power functions IU test low dimensions. The figure reports the conditional predictive ability test of Giacomini and White (2006), GW, the multivariate GW test, MGW, the Diebold and Mariano (1995) test, DM, the Mariano and Preve (2012) test, MP, and the Fisher (1934) test. The x-axis reports the absolute values of the boundaries imposed on the distribution of $\mathit{\Psi }$.

Fig. 2 Power functions IU test rejection accuracy. The figure reports the conditional predictive ability test of Giacomini and White (2006), GW, the multivariate GW test, MGW, the Diebold and Mariano (1995) test, DM, the Mariano and Preve (2012) test, MP, and the Fisher (1934) test. The x-axis reports the absolute number of the true null hypotheses.

Fig. 3 Power functions IU test high dimensions. The figure reports the conditional predictive ability test of Giacomini and White (2006), GW the Clark and West (2007) test, CW, and the Fisher (1934) test. The x-axis reports the mean of the first DGP.

Table 2 Rejections for each individual test.

	USD				EUR				GBP
	AR(2)	(%)	TVP	(%)	AR(2)	(%)	TVP	(%)	AR(2)	(%)	TVP	(%)
CW	1	2.6	16	41.0	0	0	5	21.7	1	4.5	6	27.3
DM	1	2.6	3	7.7	4	17.4	1	4.3	6	27.3	1	4.5
GW	2	5.1	4	10.3	0	0	1	4.3	4	18.2	0	0

NOTE: The table contains the total number of rejections for each test as well as the number of rejections in percent of the total number of forecasts. AR(2) refers to CC-AR(2) compared to CC-AR(1) forecasts, while TVP refers to TVP-AR(1) forecasts compared to CC-AR(1) forecasts.

Table 3 Test statistic for combined p-values.

Panel A: Individual currencies
	USD			EUR			GBP
	AR(2)	TVP	Comb.	AR(2)	TVP	Comb.	AR(2)	TVP	Comb.
CW	0.9	17.9*	8.9	0.1	12.2*	6.1	2.6	26.9**	13.4*
DM	0.9	2923.2***	1461.6***	4.3	1.3	2.2	6.9	1.5	3.4
GW	4.7	374.2***	187.1***	0.7	2.3	1.1	2.3	0.3	1.1
n	39	39	78	23	23	46	22	22	44
All	1.6	974.4***	487.2***	1.4	4.1	2.0	2.3	9.0	4.5
n	117	117	234	69	69	138	66	66	132
Panel B: Combined currencies
	USD-EUR			USD-GBP			EUR-GBP			All
	AR(2)	TVP	Comb.	AR(2)	TVP	Comb.	AR(2)	TVP	Comb.	AR(1)	TVP	Comb.
CW	1.0	11.4*	5.7	0.5	11.2*	5.6	1.3	13.1*	6.6	0.7	8.3	4.2
DM	2.5	1869.0***	934.5***	1.6	1838.8***	919.4***	3.4	0.8	1.7	1.8	1357.2***	678.6***
GW	3.0	239.3***	119.6***	2.9	235.4***	117.7***	1.1	1.2	0.6	2.2	173.7***	86.9**
n	62	62	124	61	61	122	45	45	90	84	84	168
All	1.0	612.9***	306.5***	1.0	623.0***	311.5***	1.1	4.4	2.2	0.7	452.4***	226.2***
n	186	186	372	183	183	366	135	135	270	252	252	504

NOTE: AR(2) refers to CC-AR(2) compared to CC-AR(1) forecasts, while TVP refers to TVP-AR(1) forecasts compared to CC-AR(1) forecasts. Panel A contains results from IU test for. GW stands for Giacomini-White test, CW for Clark-West test, and DM for Diebold-Mariano test. First two columns and first three rows contain test statistic of combined p-values from each tests for USD for each model. Third column contains test statistic for both models. Forth row contains test statistic for combined p-values from all tests for each model and for both models. Panel B contain test statistic for combined p-values from currency combinations for each test for each model and for both models as well as p-values from all tests combined for each model and for both models. Test rejects if statistic exceeds the critical values, obtained as $r/(\alpha (r-1))$: $\alpha =1\%$: 105.263, denoted by ${}^{***}$, $\alpha =5\%$: 21.053, denoted by ${}^{**},\alpha =10\%$: 10.526, denoted by ${}^{*}$.

Clark, T. E., and West, K. D. (2007), “Approximately Normal Tests for Equal Predictive Accuracy in Nested Models,” Journal of Econometrics, 138, 291–311. DOI: 10.1016/j.jeconom.2006.05.023.

 Web of Science ®Google Scholar

Diebold, F. X., and Mariano, R. S. (1995), Comparing Predictive Accuracy,” Journal of Business & Economic Statistics, 13, 253–265.

 Web of Science ®Google Scholar

Giacomini, R., and White, H. (2006), “Tests of Conditional Predictive Ability,” Econometrica, 74, 1545–1578. DOI: 10.1111/j.1468-0262.2006.00718.x.

 Web of Science ®Google Scholar

Mariano, R. S., and Preve, D. (2012), Statistical Tests for Multiple Forecast Comparison,” Journal of Econometrics, 169, 123–130. DOI: 10.1016/j.jeconom.2012.01.014.

 Web of Science ®Google Scholar

Fisher, R. A. (1934), Statistical Methods for Research Workers (Vol. 5), Edinburgh: Oliver and Boyd.

 Google Scholar