Is the p-Value a Suitable Basis for the Construction of Measures of Evidence? Comment on “The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations”

Dr. Gibson has to be congratulated for having enriched the wealth of articles written in response to the ASA statement on p-values of 2016 by a valuable and thoughtful contribution. We particularly appreciate that a biostatistician being in charge of one of the worldwide leading pharmaceutical companies joins the fraction of expert statistical scientists who do not share the view that p-values ought to be deprived of their role as a major backbone of inference in the empirical sciences as a whole. Remarkably, among the colleagues taking a skeptical perspective regarding the inititative platformed on one of the flagship journals of the ASA are even top-ranking representatives of that highly respected scientific organization: Karen Kafadar (2019), who emphatically warned in an editorial to Amstat News of the potentially detrimental consequences of abandoning “our well-researched and theoretically sound statistical methodology,” was the President of the ASA when she wrote this! Fortunately, historical experience suggests that p-values and hypotheses tests are fairly robust against initiatives aiming to eliminate them from the practice of statistical data analysis. Actually, a p-value banning campaign was launched already in the 1980s by several influential epidemiologists. In that previous version, the campaign targeted at replacing hypothesis testing by confidence interval estimation as the standard methodology of inference for use in scientific data analysis (cf. Rothman 1978; Salsburg 1985; Gardner and Altman 1986). Some of its main proponents organized a symposium devoted to the topic at the 1986 Annual Conference of the Society for Epidemiologic Research (SER) and chaired it wearing t-shirts with “no p-value” logos. There is little, if any evidence, that the pro-confidence interval initiative had substantial impact on scientific practice and statistical education. As a striking proof of the contrary, a glance over a recent issue of the American Journal of Epidemiology might be sufficient: although p-values are strongly discouraged in the instruction to authors, numerous articles reporting results of significance tests are still published in that journal. One of the indicators suggesting that the recent campaign can be expected to share the same fate is that up to now no steps were taken to change the title of the magazine edited jointly by the ASA and the Royal Statistical Society for promoting the public perception of the statistical science. The title still reads “Significance,” and there are not a few additional facts making it very unlikely that methods of significance testing might really become obsolete in the near future.

In its more technical part, Gibson (2020) focuses on proposals for making the p-value a sensible measure of evidence through rescaling it by means of a suitable one-to-one transformation. We have serious doubts whether finding such a transformation is a worthwhile objective. Major reasons for this view are the following.

1. Distinguishing between good and poor measures of evidence requires that one is able to define this intrinsically elusive term without operationalizing it through providing a specific rule for its quantification. The definition offered by Gibson (“Evidence is information indicating the degree to which a proposition is valid.”) is nonoperational but fairly problematic from a logical point of view: a proposition of whatever kind is valid or not, implying that validity cannot be considered a degradable attribute.

2. In many areas of biopharmaceutical research and the medical sciences, one is regularly urged to take a decision on whether or not the “evidence” made available on a given problem is sufficient for considering it solved and setting the corresponding action(s) in effect. The p-value in its original form seems much better suited for decision making than a transform like $-{\hspace{0.17em}\text{log}\hspace{0.17em}}_{10}(p)$ not admitting a direct probabilistic interpretation. By experience, most people have a much more clear-cut sense of orders of magnitude for probabilities as compared to different scales. The high importance of decision making in the medical sciences was already cited by Fleiss (1986a, 1986b, 1986c) in his convincing refutation of the alleged inadequacy of significance testing as a basic tool of scientific inference.

3. In the simple example of the two-sample Gauss-test for differences between two normal means with unpaired samples of data of common known variance, the p-value and thus any transform of it, is a function of the normalized point estimate $\sqrt{n/2}\left|\widehat{\delta }\right|/\sigma$. The latter indicates how much larger the observed value of the natural estimator of the parameter of interest came out to be as compared to its standard error and seems at least as well suited for measuring the “evidence” for a deviation of the true parameter value from zero as are continuous monotone functions of this computationally straightforward quantity. Except for the factor $\sqrt{n/2}$, it coincides with the estimated “effect size” in the sense of the terminology popularized primarily among researchers in the social sciences by Cohen (1962).

An important issue with regard to which we fully agree with the author of the article under discussion, is the suitability claimed in a number of references cited there, of the so-called Bayes factor as an option for replacing the p-value by an improved measure of evidence. First, the term is misleading, since it refers to a quantity which is not intrinsically Bayesian but plays a prominent role in frequentist inference under the heading likelihood ratio. Second, the concept can easily made precise only in the practically rather irrelevant case of a couple of one-point hypotheses, since otherwise either H₀ or H₁ or both correspond to a whole, typically non-finite sub-family of densities of the outcome variable. Third and most important, we know of no convincing argument warranting the claim that, as a measure of “evidence,” the Bayes factor makes more sense than the p-value.