Comment on “The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations”

Figures & data

Table 1 Comparison of Bayes factor bound, $-{\hspace{0.17em}\text{log}\hspace{0.17em}}_{10}(p-\text{value})$, and replication probability calibration of p-values.

	p-value
Calibration method	0.10	0.05	0.01	0.001	0.0001	0.00001
BFB	1.6	2.5	8	53	399	3195
$-{\hspace{0.17em}\text{log}\hspace{0.17em}}_{10}(p-\text{value})$	1	1.3	2	3	4	5
$\widehat{\text{RP}}$ unadjusted	0.38	0.50	0.73	0.91	0.97	0.99
$\widehat{\text{RP}}$ adjusted for uncertainty	0.41	0.50	0.67	0.83	0.91	0.96
+ regression to the mean	0.23	0.35	0.57	0.77	0.89	0.94
+ heterogeneity	0.19	0.30	0.49	0.68	0.80	0.88

NOTE: Replication probabilities are either unadjusted or adjusted for uncertainty of original effect estimate, regression to the mean, between-study heterogeneity of the effect.

Table 2 Three examples with different original and replication studies.

	Example
	1	2	3
Original p-value	0.049	0.051	0.049
Replication p-value	0.049	0.001	0.001
Relative sample size	1	1	8
Harmonic mean p-value	0.003	0.0004	0.0004
BFB	6	129	133
Relative effect size	1	1.69	0.59
One-sided skeptical p-value	0.082	0.047	0.10

NOTE: Harmonic mean p-value, Bayes factor bound, relative effect size, and one-sided skeptical p-value are shown for each.