Getting the ROC into Sync

Abstract

Judging the conformity of binary events in macroeconomics and finance has often been done with indices that measure synchronization. In recent years, the use of Receiver Operating Characteristic (ROC) curve has become popular for this task. This article shows that the ROC and synchronization approaches are closely related, and each can be derived from a decision-making framework. Furthermore, the resulting global measures of the degree of conformity can be identified and estimated using the standard method of moments estimators. The impact of serial dependence in the underlying series upon inferences can therefore be allowed for. Such serial correlation is common in macroeconomic and financial data.

Keywords:

• Economic recession
• Kuipers score
• Receiver operating characteristic curve
• Serial dependence
• Synchronization

Supplementary Materials

The supplementary materials contain a detailed discussion on the asymptotic theory when the global accuracy measures are defined as integrals.

Acknowledgments

We thank Editor Atsushi Inoue, an associate editor, and two anonymous referees for many useful comments. However, any remaining errors are our responsibility. This article was originally circulated as CAMA Working Paper, Australian National University, 1/2022 (January): https://doi.org/10.2139/ssrn.4007695.

Disclosure Statement

The authors report there are no competing interests to declare.

Notes

1 See Berloco, Argiento, and Montagna (2022), Conti and Guzman (2022), Owyang, Piger, and Soques (2022), and Thorsrud (2020) for a sample of more recent applications of ROC in diverse empirical contexts.

2 There are other graphs. For example, Berge and Jordà (2011) used the correct classification frontier which plots $\widehat{H}(\tau )$ against $1-\widehat{F}(\tau )$. The same methods we describe later can be applied to this case.

3 Harding and Pagan (2006) showed that conclusions about the degree of conformity could be very different once dependence in data was allowed for.

4 This assumption is perhaps not as restrictive as it appears at first sight since one can always find a strictly monotonic transformation $\Psi (·)$ such that $\Psi (Y)\in [a,b]$. For instance, the probability integral transformation of a continuous random variable is a candidate for $\Psi (·)$. As well the ROC curve is invariant to any strictly monotonic transformation, meaning both Y and $\Psi (Y)$ share the same ROC curve (Krzanowski and Hand 2009).

5 For instance, Y could be generated from a first stage predictive model with estimated parameters. See Lieli and Hsu (2019) for an example.

6 As suggested by the associate editor, one may interpret $\omega (·)$ in two ways. First, it can be taken as a Bayesian prior or a posterior of τ. Alternatively, one can think of the forecaster as playing a mixed strategy in that (s)he draws τ randomly from the distribution $\omega (·)$ and the chosen action is $\mathbb{I}(Y\le \tau )$.

7 Note that $F({\tau }_0)=F(a)=0$ and $F({\tau }_M)=F(b)=1$ by definitions of τ₀ and τ_M .

8 Note that ${\omega }^{*}(·)$ in (16) is slightly different from $\omega (·)$ in (9). In order for $\omega (·)$ to be a valid weighting scheme, ${\sum }_{j=1}^M\omega ({\tau }_j)$ must be one. In contrast, ${\omega }^{*}(·)$ is directly derived from (11) and ${\sum }_{j=1}^M{\omega }^{*}({\tau }_j)=1-F({\tau }_1)/2\ne 1$. Nevertheless, ${\omega }^{*}(·)$ and $\omega (·)$ are related by observing that ${\omega }^{*}({\tau }_j)=\omega ({\tau }_j)(1-F({\tau }_1)/2)$ for $j=1,2,\dots ,M$. When τ₁ is close to a, the difference between ${\omega }^{*}(·)$ and $\omega (·)$ can be safely ignored.

9 For AUROC, Lieli and Hsu (2019) showed that the standard asymptotic normality results no long hold when $X_1,\dots ,X_l$ are independent of Z. This is caused by the overfitting problem as the first-stage regression of Y on X creates a spurious concordance between Z and $\widehat{Y}$ even if the true (yet unobserved) Y has no predictive power. In a particular case where $X_1,\dots ,X_l$ are Bernoulli random variables, they demonstrated that $\sqrt{T}(\widetilde{\text{AUROC}}-\text{AUROC})=\sqrt{T}(\widetilde{\text{AUROC}}-0.5)$ converges to a nonstandard distribution whose critical values have to be simulated. Besides, the asymptotic distribution of $\sqrt{T}(\widetilde{\text{AUROC}}-0.5)$ crucially depends on the distribution of $X_1,\dots ,X_l$ and the dimension l. However, there is no general results for any distribution of $X_1,\dots ,X_l$ yet.

10 Results for M > 100 are virtually the same as those in Figure 3.

11 $(1-{\widehat{\mu }}_Z)/{\widehat{\mu }}_Z=0.874/0.126=6.947$.

12 In principle, the weighting scheme of AUROC should vary with forecast horizons. However, as Z_t = 0 is the dominant event in our case comprising of 87% of the sample, the conditional distribution of the yield spread given Z_t = 0 is quite close to its marginal distribution, which obviously does not change much with forecast horizons. That is why only one benchmark line is shown in Figure 4(c).

13 The parameter estimates using the original sample 1952:II-1984:IV are much the same except that the downturn regime has a growth rate of –1.25% in the volatility-augmented model and –3.59% in the basic model. The estimated $\hspace{0.17em}\ln \hspace{0.17em}(\sigma )$ is –4.87 in the standard model while the two values in the volatility-augmented model are –4.662 and –4.926.

14 The parameter estimates for the standard MS model are the same as Hamilton reported except for the mean of growth rate in the upturn state. The data is available from Hamilton’s web page.

15 A plot (not reported) of the conditional density of Y given Z = 1 shows that a large amount of probability mass is assigned to values close to unity and the two conditional densities are largely nonoverlapping, indicating a good discriminatory power for the MS models (Lahiri and Yang 2013).