Value-at-risk forecasts under scrutiny—the German experience

Abstract

We present an analysis of the VaR forecasts and the P&L series of all 12 German banks that used internal models for regulatory purposes throughout the period from the beginning of 2001 to the end of 2004. One task of a supervisor is to estimate the ‘recalibration factor’, i.e. by how much a bank over- or underestimates its VaR. The Basel traffic light approach to backtesting, which maps the count of exceptions in the trailing year to a multiplicative penalty factor, can be viewed as a way to estimate the ‘recalibration factor’. We introduce techniques that provide a much more powerful inference on the recalibration factor than the Basel approach based on the count of exceptions. The notions ‘return on VaR (RoVaR)’ and ‘well-behaved forecast system’ are keys to linking the problem at hand to the established literature on the evaluation of density forecasts. We perform extensive bootstrapping analyses allowing (1) an assessment of the accuracy of our estimates of the recalibration factor and (2) a comparison of the estimation error of different scale and quantile estimators. Certain robust estimators turn out to outperform the more popular estimators used in the literature. Empirical results for the non-public data are compared to the corresponding results for hypothetical portfolios based on publicly available market data. While these comparisons have to be interpreted with care since the banks' P&L data tend to be more contaminated with errors than the major market indices, they shed light on the similarities and differences between banks' RoVaRs and market index returns.

Keywords:

• Banking supervision
• VaR
• Exploratory data analysis
• Backtesting

Acknowledgments

The first two authors point out that the views expressed herein should not be construed as being endorsed by the BaFin. We thank several anonymous referees for numerous suggestions as well as seminar participants from the Center of Financial Studies, Humboldt University, and the conference of the Swiss Society for Financial Market Research for helpful discussions. Special thanks go to Andreas Zapp for valuable pointers to literature, numerous discussions and support with regard to data and 'R'-coding.

Notes

†Note that ‘well behaved’ is not to be interpreted as ‘favourable from a supervisory point of view’. It rather stands for ‘tractable from a statistical point of view’.

†This is one of the important differences between the hypothetical and the economic P&L. While the mean of the economic P&L is highly significant and economically relevant, differences among banks in the mean of the hypothetical P&L are almost solely explained by differences in the treatment of theta.

‡It does not even have a first moment, like the Cauchy distribution.

†See Bühlmann (2002) for an overview of bootstrapping time series.