The widespread misinterpretation of p-values as error probabilities

Abstract

The anonymous mixing of Fisherian (p-values) and Neyman–Pearsonian (α levels) ideas about testing, distilled in the customary but misleading p < α criterion of statistical significance, has led researchers in the social and management sciences (and elsewhere) to commonly misinterpret the p-value as a ‘data-adjusted’ Type I error rate. Evidence substantiating this claim is provided from a number of fronts, including comments by statisticians, articles judging the value of significance testing, textbooks, surveys of scholars, and the statistical reporting behaviours of applied researchers. That many investigators do not know the difference between p’s and α’s indicates much bewilderment over what those most ardently sought research outcomes—statistically significant results—means. Statisticians can play a leading role in clearing this confusion. A good starting point would be to abolish the p < α criterion of statistical significance.

Keywords:

• α levels
• ‘data-adjusted’ type I errors
• Fisher
• Neyman–Pearson
• p-values
• p < α criterion
• significance test

Acknowledgements

This manuscript has benefited from comments by M.J. Bayarri. Any remaining errors are my responsibility.