Heterogeneity of variance and biased hypothesis tests

Abstract

This study examined the influence of heterogeneity of variance on Type I error rates and power of the independent-samples Student's t-test of equality of means on samples of scores from normal and 10 non-normal distributions. The same test of equality of means was performed on corresponding rank-transformed scores. For many non-normal distributions, both versions produced anomalous power functions, resulting partly from the fact that the hypothesis test was biased, so that under some conditions, the probability of rejecting H ₀ decreased as the difference between means increased. In all cases where bias occurred, the t-test on ranks exhibited substantially greater bias than the t-test on scores. This anomalous result was independent of the more familiar changes in Type I error rates and power attributable to unequal sample sizes combined with unequal variances.

Keywords:

• biased hypothesis test
• heterogeneity of variance
• heteroscedasticity
• unequal sample sizes
• t-test
• Wilcoxon–Mann–Whitney test
• rank transformation

Notes

The Mathematica code used in this study can be obtained by writing to the author.