Are Effect Sizes and Confidence Levels Problems For or Solutions To the Null Hypothesis Test?

Abstract

Some have proposed that the null hypothesis significance test, as usually conducted using the t test of the difference between means, is an impediment to progress in psychology. To improve its prospects, using Neyman-Pearson confidence intervals and Cohen's standardized effect sizes, d, is recommended. The purpose of these approaches is to enable us to understand what can appropriately be said about the distances between the means and their reliability. Others have written extensively that these recommended strategies are highly interrelated and use identical information. This essay was written to remind us that the t test, based on the sample—not the true—standard deviation, does not apply solely to distance between means. The t test pertains to a much more ambiguous specification: the difference between samples, including sampling variations of the standard deviation.