Evidential Calibration of Confidence Intervals

Figures & data

Figure 1: The RECOVERY trial (RECOVERY Collaborative Group 2021) found that dexamethasone treatment reduced mortality compared to usual care in hospitalized Covid-19 patients (estimated log hazard ratio $\widehat{\theta }=-0.19$ with standard error $\sigma =0.05$ and 95% confidence interval from $-0.29$ to $-0.07$). Assuming a normal likelihood $\widehat{\theta } | \theta \sim \mathrm{N}(\theta ,{\sigma }^2)$, the Bayes factor for contrasting ${\mathcal{H}}_0:\theta ={\theta }_0$ to ${\mathcal{H}}_1:\theta \ne {\theta }_0$ is shown as a function of the null value ${\theta }_0$. A unit-information normal distribution $\theta | {\mathcal{H}}_1\sim \mathrm{N}({\mu }_{\theta }=-0.22,{\sigma }_{\theta }^2=4)$ centered around the clinically relevant log hazard ratio is used as prior for $\theta$ under ${\mathcal{H}}_1$. Support intervals for different support levels k indicate the range of log hazard ratios supported by the data.

Table 1: Classifications of evidence for ${\mathcal{H}}_0$ provided by Bayes factors ${\text{BF}}_{01}=k$.

k	Jeffreys (1961)	k	Royall (1997)	k	Fisher (1956)
$>$ 100	Decisive	$>$ 64	Quite strong indeed	1/2 to 1	Good
30 to 100	Very strong	32 to 64	Quite strong	1/5 to 1/2	Fair
10 to 30	Strong	8 to 32	Strong	1/15 to 1/5	Poor
3 to 10	Substantial	4 to 8	Weak	$<$ 1/15	Open to grave suspicion
1 to 3	Bare mention

NOTE: The cutoffs from Jeffreys are slightly adjusted from powers of $\surd 10$, as suggested by Held and Ott (2018). Royall and Fisher defined their classifications only for likelihood ratios, that is, Bayes factors with simple hypotheses ${\mathcal{H}}_0:\theta ={\theta }_0$ versus ${\mathcal{H}}_1:\theta ={\theta }_1$. While Royall placed no restrictions on ${\theta }_1$, Fisher used the maximum likelihood estimate ${\theta }_1=\widehat{\theta }$. He named only the $k<1/15$ category.

Figure 2: Comparison of prior distributions for the parameter $\theta$ under the alternative ${\mathcal{H}}_1$ in terms of the resulting support interval width and the highest level for which it is nonempty. A data model $\widehat{\theta } | \theta \sim \mathrm{N}(\theta ,{\lambda }^2/n=4/n)$ is assumed in all cases. The prior scale/spread parameter is set to ${\sigma }_{\theta }=2$. The normal prior (correct mean) has a mean equal to the parameter estimate $\widehat{\theta }$, while the normal prior (wrong mean) has a mean one standard deviation $\lambda =2$ away from $\widehat{\theta }$.

Figure 3: Mapping between confidence level $(1-\alpha )100\%$ and minimum support level k for different types of minimum support intervals.

Figure 4: Different support intervals for the data from the RECOVERY trial. The normal prior is centered around ${\mu }_{\theta }=-0.22$ and has unit variance ${\sigma }_{\theta }^2=4$. The local normal prior also has unit variance ${\sigma }_{\theta }^2=4$. The spread parameter of the nonlocal normal moment prior is ${\sigma }_{\theta }=0.28$.

Table 2: Summary of confidence intervals (CI), support intervals (SI), and minimum support intervals (minSI) for an unknown parameter $\theta$ based on a parameter estimate $\widehat{\theta }$ with standard error $\sigma$.

Interval type	Standard error multiplier M
$(1-\alpha )100\%$ CI	${\Phi }^{-1}(1-\alpha /2)$
k SI (normal prior)	$\surd \{\hspace{0.17em}\log \hspace{0.17em}(1+{\sigma }_{\theta }^2/{\sigma }^2)+{(\widehat{\theta }-{\mu }_{\theta })}^2/({\sigma }^2+{\sigma }_{\theta }^2)-2\hspace{0.17em}\log \hspace{0.17em}k\}$
k SI (local normal prior)	$\surd [\{\hspace{0.17em}\log \hspace{0.17em}(1+{\sigma }_{\theta }^2/{\sigma }^2)-2\hspace{0.17em}\log \hspace{0.17em}k\}(1+{\sigma }^2/{\sigma }_{\theta }^2)]$
k SI (nonlocal normal moment prior)	$\surd ([2{\mathrm{W}}_0\{{(1+{\sigma }_{\theta }^2/{\sigma }^2)}^{3/2}/(2k{e}^{-1/2})\}-1]\{1+{\sigma }^2/{\sigma }_{\theta }^2\})$
k minSI (all priors)	$\surd (-2\hspace{0.17em}\log \hspace{0.17em}k)$
k minSI (local normal priors)	$\surd \{-{\mathrm{W}}_{-1}(-k^2/e)\}$
k minSI ($-e p \hspace{0.17em}\log \hspace{0.17em}p$)	${\Phi }^{-1}[1-\hspace{0.17em}\exp \hspace{0.17em}\{{\mathrm{W}}_{-1}(-k/e)\}/2]$

NOTE: All intervals are of the form $\widehat{\theta }\pm \sigma \times \mathrm{M}$. To transform an interval from type A to type B, first subtract $\widehat{\theta }$ from the boundaries of the interval, multiply by the ratio of the standard error multipliers ${\mathrm{M}}_{\mathrm{B}}/{\mathrm{M}}_{\mathrm{A}}$, and add again $\widehat{\theta }$ to the boundaries of the interval. The standard error multipliers M depend on either the confidence level $(1-\alpha )$ or the support level k. For the support intervals, the standard error multipliers M additionally depend on the parameters of the prior for $\theta$ under the alternative hypothesis: mean ${\mu }_{\theta }$ and variance ${\sigma }_{\theta }^2$ for the normal prior, variance ${\sigma }_{\theta }^2$ for the local normal prior, and spread ${\sigma }_{\theta }$ for the nonlocal normal moment prior. The quantile function of the standard normal distribution is denoted by ${\Phi }^{-1}(\cdot )$, ${\mathrm{W}}_0(\cdot )$ denotes the principal branch of the Lambert W function, and ${\mathrm{W}}_{-1}(\cdot )$ denotes the branch that satisfies $\mathrm{W}(y)<1$ for $y\in (-1/e,0)$. (Minimum) support intervals are only nonempty for support levels k for which the standard error multiplier is real-valued, that is, the term in the square root must be nonnegative and/or the argument for ${\mathrm{W}}_{-1}(\cdot )$ must be in $[-1/e,0)$. All interval types can be computed with the R package ciCalibrate (Appendix).

RECOVERY Collaborative Group. (2021), “Dexamethasone in Hospitalized Patients with Covid-19,” New England Journal of Medicine, 384, 693–704. DOI: 10.1056/nejmoa2021436.

 PubMed Web of Science ®Google Scholar

Jeffreys, H. (1961), Theory of Probability (3rd ed.), Oxford: Clarendon Press.

 Google Scholar

Royall, R. (1997), Statistical Evidence: A Likelihood Paradigm, London; New York: Chapman & Hall.

 Google Scholar

Fisher, R. A. (1956), Statistical Methods and Scientific Inference, Edinburgh: Oliver & Boyd.

 Google Scholar

Held, L., and Ott, M. (2018), “On p-values and Bayes Factors,” Annual Review of Statistics and Its Application, 5, 393–419. DOI: 10.1146/annurev-statistics-031017-100307.

 Web of Science ®Google Scholar

Data Availability Statement

The point estimate and 95% confidence interval of the adjusted log hazard ratio were extracted from the abstract of RECOVERY Collaborative Group (2021). All analyses were conducted in the R programming language version 4.3.0 (R Core Team 2023). Code and data for reproducing the results in this manuscript are available at https://github.com/SamCH93/ECoCI. A snapshot of the GitHub repository at the time of writing this article is archived at https://doi.org/10.5281/zenodo.6723249. An R package for calibration of confidence intervals to (minimum) support intervals is available at https://CRAN.R-project.org/package=ciCalibrate, see the Appendix for an illustration.