Brief Research Report: Notes on a Nonparametric Estimate of Effect Size

Abstract

Researchers are encouraged to report effect size statistics to quantify treatment effects or effects due to group differences. However, estimates of effect sizes, most commonly Cohen’s d, make assumptions about the distribution of data that are not always true. An alternative nonparametric estimate of effect size, relying on the median absolute deviation, is proposed. Comparison of this estimate to Cohen’s d using (simulated) non-normally distributed data demonstrate that the nonparametric approach effect size may be a better estimate of effect size.

KEYWORDS:

• ANOVA/MANOVA
• effect size
• meta-analysis
• nonparametric
• simulation studies

Notes

1 We note that the definition of MAD_pooled is not equal to the mad of the union of the two samples. However, the former is more useful here than the latter.

2 Code for these simulations and others, are available at https://github.com/barneyricca/effect-sizes. Simulations whose results are referred to, but not displayed, here are also available at that repository. Additionally, some intermediate results are included there, particularly for simulations that required substantial time to compute.

3 We do note that Δ_MAD appears to be consistently biased by about 1% relative to d; exploration of this bias is left to a future study.

4 Occasional values of Cohen’s d would be negative; for any value of contamination, however, this was less than 1% of the total number of trials. Removing these from the calculations - Δ_MAD includes an absolute value in its formula – changed the values of d by less than 0.02 in every case.