On the directional accuracy of inflation forecasts: evidence from South African survey data

ABSTRACT

We study the information content of South African inflation survey data by determining the directional accuracy of both short-term and long-term forecasts. We use relative operating characteristic (ROC) curves, which have been applied in a variety of fields including weather forecasting and radiology, to ascertain the directional accuracy of the forecasts. A ROC curve summarizes the directional accuracy of forecasts by comparing the rate of true signals (sensitivity) with the rate of false signals (one minus specifity). A ROC curve goes beyond market-timing tests widely studied in earlier research as this comparison is carried out for many alternative values of a decision criterion that discriminates between signals (of a rising inflation rate) and nonsignals (of an unchanged or a falling inflation rate). We find consistent evidence that forecasts contain information with respect to the subsequent direction of change of the inflation rate.

KEYWORDS:

• Inflation rate
• forecasting
• directional accuracy

JEL CLASSIFICATION:

• C53
• D82
• E37

Acknowledgments

We thank three anonymous reviewers for helpful comments. The usual disclaimer applies.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 Market-timing tests have been applied in a number of other contexts, including forecasting the real U.S. GNP ([55]; see also [58]) and inflation in South Korea ([2]). Lai [31] finds that exchange-rate forecasts have predictive value. Kolb and Stekler [28] present results on the directional accuracy of forecasts of U.S. interest rates, and find that forecasts are not superior relative to a naive random-walk model. Baghestani [6] finds that the directional accuracy of Blue Chip forecasts of the direction of change in the trade-weighted dollar exchange rate is low and not better than that of a random-walk model.

2 For an earlier working-paper version of this paper, see Pierdzioch et al. [45].

3 Sensitivity and specificity are linked to Peirce's [40] ‘science of the method’ and Youden's [61] index. For an introductory review, see Baker and Kramer [7]. See also Section 3 for an application to the survey data studied in this research.

4 It should be noted that, for a random-walk model, we have ${\pi }_{t,t+1}^e={\pi }_t$ for all t. Hence, for $c>0$, there are no signals, and for $c<0$, there are no nonsignals, implying that the combinations of sensitivity and specificity implied by a random-walk model can be represented by points on the bisecting line.

5 For an early review of the principles of ROC analysis, see Metz [37]. For introductory reviews, see also Zweig and Campbell [62], Greiner et al. [18], and Baker and Kramer [7]. For comprehensive analyses of ROC curves and a detailed exposition of several extensions, see Green and Swets [17] and Pepe [41].

6 Typically, forecasters submit their forecasts on different days. We do not have data on when the individual forecasts were returned. We do know, however, that all the forecasts are posted before the release of the actual historical data.

7 The CPI is a measure of the change in the prices of goods and services consumed by a representative household. One obtains the CPIX from the CPI by leaving aside interest-rate payments on mortgage bonds. See the internet page of Statistics South Africa (http://www.statssa.gov.za). See also Pierdzioch et al. [47]. We study both the CPIX and CPI indices because the SARB officially targeted CPIX from 2000 until 2008 and CPI thereafter. These two indices are slightly different proxies for underlying inflation, but a combination of the two can be described as the targeted inflation index over the period.

8 Bhundia and Ricci [12] analyze a range of political and economic factors that contributed to the crisis, including an acceleration in money growth and exchange rate overshooting, the South African Reserve Bank's net open forward book, the delay in the privatisation of Telkom (a wireline and wireless telecommunications provider), an announcement by the South African Reserve Bank that it would tighten exchange controls, and a slowdown of global economic activity.

9 Given the highly significant AUROC statistic it is not surprising that the classic market-timing test developed by Pesaran and Timmermann [42, PT] also yields highly significant results. The PT test assumes the value 46.2041*** for the CPI data and the value 28.3521*** for the CPIX data (full sample), where computations are based on a 3-by3 contingency table that accounts for the actual direction of change and forecasts of the direction of change assuming values <, =, and $>0$. In contrast to the AUROC analysis, computations for the PT test are based on a single decision criterion ($c=0$).

10 When we define the consensus forecast in terms of the cross-sectional median, the AUROC statistic assumes a value of 0.9479 (0.0181) for the CPI data, and a value of 0.8985 (0.0316) for the CPIX data.

11 Our data cover a limited period of time (2000/05–2014/06 for the CPI data, 2000/05–2008/12 for the CPIX data). Hence, it is hardly possible to carry out a full-fledged out-of-sample analysis of forecast accuracy for a benchmark model because, for such an analysis, we need a reasonably long sample period to train and estimate a benchmark model.

12 Results for an AR(12) model are similar (not reported).

13 The significance of the test is less strong for the CPIX than for the CPI data. It should also be mentioned that it is tempting to interpret the results for the AR model as evidence that forecasters, at least approximately, form their forecasts based on an autoregressive model of inflation dynamics. In our view, however, such an interpretation would stretch the evidence too far. In fact, evidence of behavioral biases in forecasts and cross-sectional heterogeneity of the shape of forecasters' loss functions reported in earlier research [46,47] suggests that, at the microeconomic level, the process of forecast formation may be quite complex.

14 When we use the original ROC curve, the optimal decision criterion for financial analysts is $c_Y=0.1$, as in the case of short-term forecasts.

15 Results of the PT market-timing test corroborate the results of the AUROC analysis. The point estimate of the PT test (based on a 2-by2 contingency table since there are no no-change forecasts) for financial analysts assumes the value 5.7257*** and is somewhat larger than the point estimates for the business sector (4.1872***) and trade unions (3.9832***). The test results are significant for all three groups of forecasters.

16 It should be noted that our empirical results are about being able to forecast the direction of change of inflation at certain forecast horizons, not about the causal effects of inflation expectations on actual inflation, et vice versa.

Additional information

Funding

Pierdzioch thanks to the German Science Foundation (Deutsche Forschungsgemeinschaft) for financial support (project ‘Macroeconomic Forecasting in Great Crises’) [no. 2677/4-2].