EFFECT STRENGTH VS. STATISTICAL SIGNIFICANCE: A WARNING AGAINST THE DANGER OF SMALL SAMPLES

Abstract

In research on giftedness, small samples are the rule rather than the exception. Such samples require an especially meticulous research design. The absence of power‐analytical considerations (Le., defining the minimal sample sizes according to a‐ and β ‐error and expected effects) prior to data gathering can easily lead to false conclusions based on faulty data interpretation. Especially with smaller samples, it is strongly recommended that practical relevance (employing a suitable index of effect strength) be taken into consideration when interpreting observed group differences. This is better than relying on “statistical significance” alone. Group studies should not be conducted with extremely small samples. According to a proven “rule of thumb”, each subgroup tested should be comprised of at least 20 to 25 subjects. Because of the prevalence of small sample studies in research on giftedness (gifted individuals are by definition rare), it is important that statistical analyses be conducted in ways which are capable of detecting “significant” effects and which are not, for instance because of small sample size, predestined to conclude that no effects are present, even when they are (i.e., to accept the null hypothesis, even when it is false). The nature of the problem can be made more concrete by looking at a practical example taken from an article of Gefferth and Herskovits (1991).