Null Hypothesis Significance Testing Interpreted and Calibrated by Estimating Probabilities of Sign Errors: A Bayes-Frequentist Continuum

Abstract

Hypothesis tests are conducted not only to determine whether a null hypothesis (H0) is true but also to determine the direction or sign of an effect. A simple estimate of the posterior probability of a sign error is PSE = (1 – PH0)p/2 + PH0, depending only on a two-sided p-value and PH0, an estimate of the posterior probability of H0. A convenient option for PH0 is the posterior probability derived from estimating the Bayes factor to be its e p ln $(1/p)$ lower bound. In that case, PSE depends only on p and an estimate of the prior probability of H0. PSE provides a continuum between significance testing and traditional Bayesian testing. The former effectively assumes the prior probability of H0 is 0, as some statisticians argue. In that case, PSE is equal to a one-sided p-value. (In that sense, PSE is a calibrated p-value.) In traditional Bayesian testing, on the other hand, the prior probability of H0 is at least 50%, which usually brings PSE close to PH0.

KEYWORDS:

• Calibrated effect size estimation
• Calibrated p-value
• Directional error
• Reproducibility crisis
• Sign error

Acknowledgments

I am grateful to two anonymous reviewers for comments leading to a more readable article.

Additional information

Funding

This research was partially supported by the Natural Sciences and Engineering Research Council of Canada (RGPIN/356018-2009). The revisions of the spring of 2020 were completed in the Ottawa office space generously provided by St. Paul Lutheran Church.