Brief Research Report: A User Friendly Effect Size—When Normality Matters

ABSTRACT

There is a need for effect sizes that are readily interpretable by a broad audience. One index that might fill this need is π, which represents the proportion of scores in one group that exceed the mean of another group. The robustness of estimates of π to violations of normality had not been explored. Using simulated data, three estimates of π (πˆ _direct, r, and r_robust) were studied under varying conditions of sample size, distribution shape, and group mean difference. This study demonstrated that r and r_robust were biased estimates of π when data were nonnormal. We recommend that neither be used in estimating π unless data are normally distributed.

KEYWORDS:

• Effect size
• robustness
• statistics
• simulation studies
• testing

Notes

1. Tilton advocated for the use of an alternative explanation of overlap as the “percentage of scores in one population that can be matched by scores in the other population” (1937, p. 659). Assuming the two groups are normally distributed, this index can be calculated based on the d statistic. Half of the d value is the point at which the two distributions intersect. Using half of d as a reference point, it is then a fairly simple process to calculate the area of the two distributions that is shared. Tilton's alternative explanation of overlap is the effect size that could most appropriately be described as a measure of overlap, as it is literally a measure of the degree to which two distributions share the same area. Although Tilton preferred this definition, his first definition corresponds to the index examined in this study.

2. r is based on the mean and standard deviation, in comparison to the direct estimate, which depends on the entire distribution of the scores. The mean and standard deviation should thus be more stable than the count. Baker (1949) discussed this point in more depth.

3. These skew and kurtosis values were for the underlying continuous distributions, not for the integer scores. The distributions were simulated as mixtures of normal distributions: 30% N(1.046 ,1.088), 70% N(−0.439,0.311) for skew = 1; 80% N(−0.288, 0.607) and 20% N(1.153,0.910) for skew = .5.

4. However, note that nonnormality in Group 1's distribution can slightly bias the estimate of its mean and standard deviation when the scores are integers. This effect is negligible.