Testing the Out-of-Sample Forecasting Ability of a Financial Conditions Index for South Africa

Abstract

The importance of financial instability for the world economy has been severely demonstrated since the 2007–8 global financial crisis, highlighting the need for a better understanding of financial conditions. We consider a financial conditions index (FCI) for South Africa that is constructed from sixteen financial variables and test whether the FCI does better than its individual financial components in forecasting the key macroeconomic variables of output growth, inflation, and interest rates. Two sets of out-of-sample forecasts are obtained—one from a benchmark autoregressive (AR) model and one from a nested autoregressive distributed lag (ARDL) model that includes one financial variable at a time. This concept of forecast encompassing is used to examine the out-of-sample forecasting ability of these financial variables as well as of the FCI, while also controlling for data mining.

KEY WORDS:

• data mining
• financial conditions index
• forecast encompassing

Acknowledgments

The authors are grateful for comments by Christiane Baumeister and two anonymous referees.

Notes

1. Thompson et al. (2013) find that the estimated FCI has strong in-sample causality characteristics with respect to manufacturing output growth and the Treasury Bill yield.

2. Unit root test results are available from the authors upon request.

3. Standardizing the data enables analysis and comparison of the sizes of the effects of the FCIs.

4. In this instance, our FCI is the first extracted principal component, so .

5. Thompson et al. (2013) conduct this purging by using contemporaneous values of the macroeconomic variables. They test the use of past values in the purging process and find that there is no significant difference in the results. Due to the loss in sample size caused by lagging the macroeconomic variables, they rather use contemporaneous purging.

6. Thompson et al. (2013) also test the approaches of a simple weighted average and recursive PCA. These indexes however do not present adequate qualitative results and are therefore not explored further.

7. A Ludvigson and Ng (2009, 2010) assessment of the relevance of the individual components of the FCI over ten-year subsamples provides the impetus for testing the rolling-window approach. A host of alternative window sizes are tested in Thompson et al. (2013), but the 120-month rolling window presents the best results qualitatively and in terms of in-sample forecast tests.

8. Pesaran et al. (2006) list output growth, inflation, exchange rates, interest rates, and stock returns as typical series suffering from structural breaks—all of which we use in this research.

9. For a discussion and mapping of South African business cycle trends and the FCI, refer to Thompson et al. (2013).

10. This model can be used to conduct a test of the in-sample forecasting ability of by running a Wald test with H₀: . Rejection of the null hypothesis indicates that there is evidence of in-sample forecasting ability/Granger causality. See Thompson et al. (2013) for these in-sample results with respect to the FCI. Results pertaining to the individual financial series are available from the authors upon request.

11. Strictly speaking, Theil’s U uses a random-walk model as a benchmark. In our applications, we follow Rapach and Weber (2004) in using the AR model as benchmark, but we still refer to the ratio of the RMSFEs from the restricted and unrestricted models as Theil’s U.

12. The MSE-T and MSE-F statistics are assumed to be asymptotically normally distributed (West 1996). However, McCracken (2007) shows that they have a nonstandard asymptotic distribution at h = 1 when comparing nested models’ forecasts—as is the case in this application—and that the distribution is in fact a function of stochastic integrals of quadratics of Brownian motion for MSE-T and a function of stochastic integrals of Brownian motion for MSE-F. Clark and McCracken (2004) similarly show that the limiting distribution is also nonstandard for h > 1 when comparing nested models’ forecasts. Therefore, bootstrapped inference as proposed in Kilian (1999) is recommended.

13. Clark and McCracken (2001) show that for nested models and for h = 1, ENC-T has a nonstandard limiting distribution while ENC-NEW has a nonstandard asymptotic distribution. For h > 1 in nested models, Clark and McCracken (2004) show that ENC-T and ENC-NEW have nonstandard asymptotic distributions. Thus, bootstrapped inference is once again recommended.

14. The authors use extensive Monte Carlo simulations with nested models to ascertain these properties.

15. Hoover and Perez (2000) present the case that if data mining “must” be engaged in, then statistical inference should be adjusted so that critical values are made to be stricter.

16. In running the data-mining programs, we exclude the FCI as an explanatory variable since it contains the information of the sixteen financial variables.

17. Rapach and Weber (2004) claim that this is essentially similar to establishing the economic significance of a parameter estimate (versus the statistical significance).

18. Results are presented only for the forecast horizons deemed significant in Table 5.

19. In certain instances for x_t = M3 and x_t = SPR_MORT, is found to have inconsistent values (i.e., >1). We take the approach of Rapach and Weber (2004), who experienced similar issues in their research, and disregard these results.

20. Ex ante forecasting over this period is done by using the estimate of the model until January 2012 and forecasting without updating the estimates.

21. The RMSE statistic is calculated as the square root of the average of the squares of the errors. A similar forecasting exercise is conducted over the period of the financial crisis (December 2007–August 2009 are the official recession dates in South Africa), and the FCI again has the best forecasting performance (RMSE = 2.138; MAE = 1.638) compared to government bond volatility (RMSE = 2.652; MAE = 2.191) and the term spread (RMSE = 2.734; MAE = 2.148).