Single group, pre- and post-test research designs: Some methodological concerns

Abstract

This article provides two illustrations of some of the factors that can influence findings from pre- and post-test research designs in evaluation studies, including regression to the mean (RTM), maturation, history and test effects. The first illustration involves a re-analysis of data from a study by Marsden (2004), in which pre-test scores are plotted against gain scores to demonstrate RTM effects. The second illustration is a methodological review of single group, pre- and post-test research designs (pre-experiments) that evaluate causal relationships between intervention and outcome. Re-analysis of Marsden’s prior data shows that learners with higher baseline scores consistently made smaller gains than those with lower baseline scores, demonstrating that RTM is clearly observable in single group, pre-post test designs. Our review found that 13% of the sample of 490 articles were evaluation studies. Of these evaluation studies, about half used an experimental design. However, a quarter used a single group, pre-post test design, and researchers using these designs did not mention possible RTM effects in their explanations, although other explanatory factors were mentioned. We conclude by describing how using experimental or quasi-experimental designs would have enabled researchers to explain their findings more accurately, and to draw more useful implications for pedagogy.

Keywords:

• education research
• pre-experiment
• regression to the mean
• control group
• research design
• research methods

View correction statement:

Corrigendum

Acknowledgements

We thank David Torgerson for his useful comments and suggestions on an earlier draft of the paper.

Notes

1. The data were, in fact, from a trial that used matched randomisation to an experimental and a comparison group, thereby controlling for RTM effects.

2. For reasons of space statistics are not provided, but the data can be found in Marsden, 2004.

3. The majority of studies that were NOT evaluation studies aimed to explore potential relationships, define constructs, or document processes.