On the ranking consistency of systemic risk measures: empirical evidence*

Abstract

We empirically analyze the extent to which popular systemic risk measures (SRMs) yield comparable results regarding the systemic importance of a financial institution. More important, we also examine determinants of the degree of consistency in the classification according to the various SRMs. In general, rank correlations tend to be more associated with macroeconomic variables such as the unemployment rate than with bank-individual variables. Our results also reveal that rank correlations are particularly sensitive to the overall market conditions. During more volatile market phases, rank correlations are slightly larger than during less volatile phases. Furthermore, their association with bank-individual and macroeconomic variables changes with the market conditions. The less volatile the market, the more relevant the bank-individual variables become in explaining the rank correlations. Contrary, during less volatile market phases, the relevance of macroeconomic variables decreases. Overall, the analyses reveal a difficulty in detecting specific explanatory factors for the consistency in systemic risk rankings across settings.

KEYWORDS:

• Systemic risk
• risk rankings
• financial regulation

JEL Classifications:

• G01
• G21
• G28
• G32

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Several risk measures have already been shown to contain valuable information (see Benoit et al. (2017) for an overview). It is thus not advisable to focus on a single metric. However, it is difficult to draw conclusions based on systemic risk measures that assign systemic risk rankings inconsistently.

2 The work of Acharya et al. (2017) and Adrian and Brunnermeier (2016) has each been cited more than 2,000 times by December 2020 according to Google Scholar. To our knowledge, these are the most popular works regarding citations related to measuring systemic risk based on market data. The work of Brownlees and Engle (2016) has only been cited around 700 times, but this article exceeds any related works in terms of downloads on SSRN with more than 11,000 downloads as of December 2020.

3 In case of a merger or acquisition, banks that meet the specified criteria are included as long as they are independent.

4 The MSCI World index is used as a proxy for the market return, which is required for the computation of the various SRMs. We follow the approach of Acharya et al. (2017) by considering a non-financial market index. Interestingly, the systemic risk measures are quite sensitive towards the choice of a market index, which highlights the need to carefully assess and interpret SRM rankings.

5 Correlations are not winsorized as they are bounded by construction, such that it remains rather unclear how outliers could be determined.

6 This ambiguity is also found by Grundke (2019) in a theoretical simulation-based banking network model.

7 Although CBOE VIX is a measure of the volatility of the American S&P500 only, it is also understood as a general benchmark index for volatility on financial markets. For our sample period, the correlations between VIX, on the one hand, and on the other hand the corresponding European EUROSTOXX50 Volatility Index and the Canadian S&P/TSX60 Volatility Index is 0.9 and 0.85, respectively. That is why we advocate the use of the VIX to define more and less volatile periods in our work.

8 In further (unreported) tests, we repeat this exercise with stress periods defined as quarters with VIX values in the upper 75% quartile, and calm periods as quarters with VIX values in the lower 25% quartile. The results confirm that most correlations are larger during stress periods.

9 Results are available upon request.

10 Note that all standard deviations are computed based on the respective less or more volatile subsample.

11 See e.g. Danielsson et al. (2016a) and Löffler and Raupach (2018) for in-depth discussions of model risk concerns associated with systemic risk measurement.

12 As an alternative to Spearman’s rank correlation, Kendall’s rank correlation – also known as Kendall’s tau coefficient – can be computed. The basic idea behind this ranking is not to measure the distance between the ranking outcomes of pairs of ranks, but simply to measure whether the ranks of a series deviate in the same direction. This leads to a more robust procedure towards outliers. Kendall’s rank correlation coefficient is found to compute correlations that are more centered around zero than those of Spearman’s rank correlation coefficient. The subsequent analysis is repeated with Kendall’s tau instead of Spearman’s rho. Results are qualitatively similar; they are not shown here but are available upon request.

13 The results are in line with Benoit et al. (2013), who similarly find that correlations between ranks of MES and SRISK are considerably smaller than those between MES and (contribution-) ΔCoVaR. However, they do not test for rank correlations stemming from SRMs whose computations consider banks’ equity market value.

14 The results are confirmed when the start of the crisis period is altered to January or March 2020.

15 Detailed results are available upon request. The sample is reduced to 70 banks, because several banks have disappeared or merged in recent years. In addition, data availability does not allow for a comprehensive computation of SRISK. This analysis is thus based only on the four remaining SRMs.

16 Benoit et al. (2013) kindly provide their MATLAB code that they use to compute MES, SRISK and ΔCoVaR via their own open source project runmycode.org. Extensive technical details are provided in the appendix of their paper.

17 Recently, the discussion with respect to the relation between market concentration and systemic risk has gained momentum. Beck, Jonghe, and Mulier (2021) show that the unbundling of various facets of market concentration allows a more differentiated analysis of the different effects on systemic risk.

Additional information

Funding

This work was supported by Universitaetsgesellschaft Osnabrueck.

Notes on contributors

Michael Abendschein

Michael Abendschein is Junior-Economist in the Economics Department at the OECD. He was previously Research Associate and PhD Student at the Chair of Banking and Finance at Osnabrück University.

Peter Grundke

Peter Grundke is Professor at the Chair of Banking and Finance at Osnabrück University.