Multiple Trials May Yield Exaggerated Effect Size Estimates

ABSTRACT

Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately, and thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this article describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items, but remain substantial in some cases. Some concepts, such as a P300 wave or a test score, are best defined as a composite of measures. Troubles may arise in more exploratory research where the interrelations among trials or items may not be well described.

Keywords:

• effect size
• multiple items
• regression/correlational designs
• repeated measures

AUTHOR NOTES

Andrew Brand is a software developer for the Department of Psychology at King's College in London. He is also the creator of iPsychExpts (http://www.ipsychexpts.com), a Web site that encourages and promotes the use of Web experiments for conducting psychological research. Michael T. Bradley is a professor of psychology at the University of New Brunswick. He has long been interested in statistical issues and has published papers examining the detection of deception. Lisa A. Best is an associate professor of psychology at the University of New Brunswick. Her primary research focuses on graphical perception and cognition and the history of data analytic techniques. George Stoica is the chair of Mathematical Sciences at the University of New Brunswick in Saint John. His research centres on mathematical finance, probability, and statistics.

Notes

1. The R scripts are available from the author on request.