Comment on “The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations”

I thank Dr. Eric W. Gibson for the insightful overview of p-values and replication problems. I enjoyed the broad discussion on the proper understanding and interpretation of p-values based on his expertise and practice in randomized clinical trials. It is a great pleasure to discuss this article and to share some of my thoughts.

1 Problems on the Fixed Significance Level of 5%

As alerted in the ASA statement on “Statistical Significance and p-Values” (Wasserstein and Lazar 2016), null hypothesis statistical testing with the fixed significance level (denoted by α) is also prevailing in Japan. George Cobb’s post to an ASA discussion forum as noted in the preface of the ASA statement:

Q: Why do so many people still use p = 0.05?

A: Because that’s what they were taught in college or grad school.

is still true in Japan. This is partly because Japanese universities do not have any department of statistics; therefore, introductory level undergraduate statistics courses are taught by non-statisticians who are not familiar with statistical methods other than null hypothesis statistical testing, which is also understood superficially. Blind setting of α as 5% is routine in the biomedical community in Japan. Researchers are very pleased when they have the result of $p\le 0.05$ but are disappointed when $p>0.05$; they thus repeat experiments or enroll more subjects as a remedy. Akira Sakuma, the famous Japanese biostatistician, referred to this tendency as a serious disease, and gave it the scientific name of “significantosis” (Sakuma 1977).

Dr. Gibson stated “I do not believe there is a replication crisis” (Gibson 2020), and it’s true because it is the natural course when using the fixed α of 5%. In the example given in Gibson (2020, Figure 3), Dr. Gibson explained the situation that 1000 trials are designed with 80% power, that the prevalence of real effects is 10%, and that a threshold of $p\le 0.05$ is used for publication. It is easy to understand this as a screening test (Lash 2017). The upper section of Table 1 shows a rearrangement of Figure 3, and its false discovery proportion (FDP) is calculated as 36% (45/125). The lower section of Table 1 shows the expected counts when investigators would conduct replication trials for 125 significant results using the same 80% power and 5% threshold. Even when the true prevalence of real effects is set relatively high as 10%, the 5% α gives a replication proportion only 51% (64/125) (the observed replication proportion is 53% (66.75/125), but 2.25 trials are false positive results).

Table 1 Expected counts for 1000 trials with prevalence of a real effect as 10% using $p\le 0.05$ as the screening threshold (1 - specificity) with 80% power (sensitivity).

Original trial	Real effect	No effect	Total
Significant	80	45	125
Nonsignificant	20	855	875
Total	100	900	1000
Replication	Real effect	No effect	Total
Significant	64	2.25	66.25
Nonsignificant	16	42.75	58.75
Total	80	45	125

Figure 1 shows the FDPs and replication proportions in different prevalence of real effects. When the prevalence is 15%, the true replication proportion reaches 60% with 26% FDP, and goes down to 40% with 50% FDP, when the prevalence is 6%.

Fig. 1 False discovery proportion (FDP) and replication proportion, using $p\le 0.05$ as the threshold and power = 80%. Solid line: FDP, dotted line: observed replication proportion, dashed line: true replication probability.

2 Don’t Say “Statistically Significant”

Following the ASA statement, a Nature comment (Amrhein, Greenland, and McShane 2019) and an Editorial in the American Statistician (Wasserstein, Schirm, and Lazar 2019) have recommended to stop using the term “statistically significant.” Dr. Gibson has also avoided the use of “significant” (although I found one in his Acknowledgments), and has provided proper understanding and interpretation of p-values. As noted in a Clinical Trials Editorial (Cook et al. 2019), “(a) typical clinical trial will be premised upon a statistical hypothesis testing framework, both for the determination of the sample size and for its analysis.” It should be noted in a trial protocol, however I, as a trial statistician, do not need to say “statistically significant” in my clinical study reports or clinical trial articles. “When reporting the results of significance tests, precise p-values (e.g., “p = 0.034”) should be reported rather than making exclusive reference to critical values” (ICH E9 1998). It is important to “(w)atch out for pressure to embellish the abstract or the press release” (Wasserstein, Schirm, and Lazar 2019).

To understand whether the evidence is persuasive enough to support an affirmative decision, I do not think that proper interpretation of p-values is sufficient. According to the 4th and 6th principles in the ASA statement:

4. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold, and

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis,

we may consider other information such as strength of treatment effect, consistency of the reported effect, and clinical plausibility which are part of Hill’s nine aspects to consider when deciding whether an association is causal (Hill 1965).

When writing this comment, I asked myself whether I could stop conducting the PRAISE-2 trial. Based on Table 3 in Gibson (2020), the overall mortality result of the PRAISE-I trial gave a hazard ratio of 0.84 with a 95% confidence interval of 0.69 and 1.02, and in patients with nonischemic heart failure, the hazard ratio was 0.54 and the 95% confidence interval of 0.37 and 0.79. The 95% confidence interval for the overall result included no effect, but shifted to the amlodipine effective side. The promising result was a finding in one of eight subgroups; however, if, in the same situation, my clinical counterpart tells me that it is clinically plausible that amlodipine is effective for nonischemic heart failure, I would probably plan a new trial limited to nonischemic heart failure patients. I would like to ask Dr. Gibson the same question.

I would like to close my comment with another fixed 5% α level story in Japan (Sakuma 1999). Because Fisher (1925) suggested the 5% α level in his book “Statistical Methods for Research Workers,” once we had a very plausible rumor:

Since a research worker could conduct one experiment per year in the agricultural experiments, in which Fisher worked, research worker’s life was roughly 20 years. Fisher thought that only one failure in 20 experiments throughout one’s research life could be allowed.

When Fisher came to Japan, Motosaburo Masuyama, a Japanese mathematical statistician, asked Fisher whether the rumor was true. “I never thought like that,” Fisher answered.

ORCID

T. Shun Satohttp://orcid.org/0000-0001-9830-013X

Additional information

Funding

This research was supported by the Japan Agency for Medical Research and Development under grant 20lk0201061t0005.