Methodological challenges in education RCTs: reflections from England’s Education Endowment Foundation

ABSTRACT

Background: The Education Endowment Foundation (EEF) is an independent charity that was established in 2011 with the explicit aim of breaking the link between family income and educational achievement in England. Over the seven years since its inception, EEF has contributed to the existing evidence base by funding over one hundred randomised controlled trials (RCTs) in education.

Purpose: This article discusses five of the key challenges EEF experienced in running RCTs in English schools; namely: (1) ensuring interventions are ready for trial, (2) recruiting schools and retaining them in trials, (3) calculating the appropriate sample size and ensuring cost effectiveness, (4) carrying out robust implementation and process evaluation, and (5) choosing and delivering appropriate testing. It is hoped that this discussion may provide insights for others who may be engaged in funding and/or conducting trials in education.

Sources of evidence: The challenges in conducting RCTs in education are explored by drawing on EEF’s experience in designing and reporting on trials. Examples from some of the EEF trials and wider methodological literature are used to illustrate the points of discussion.

Main argument: The five challenges summarised here span the full process of running a trial in schools, from intervention design to the collection of outcome measures. These have had the biggest impact on the quality of trials and the confidence that EEF has in the results. The first lesson focuses on ensuring that the intervention is sufficiently developed and there is a level of assurance in both its scalability and feasibility prior to trialling. Key questions concerning an intervention’s readiness are described and ways to improve interventions through piloting are discussed. The second lesson focuses on the recruitment of schools to a trial. Issues include the importance of communicating roles to schools and the need for time to do this effectively. Thirdly, the sample sizes of early EEF trials were informed by the size of impact seen in previous research which led to an overestimation of the effects that EEF was likely to see. Current considerations that influence the size of the trials are described and there is a discussion of why, sometimes, even small effects can be important. Fourthly, EEF trials have developed a more integrated approach to implementation and process evaluation (IPE). This is discussed in the context of how it has led to a better understanding of programme delivery. Finally, testing is discussed in light of what EEF has learned about ensuring testing success in educational trials.

Conclusions: The main lessons that EEF has learned from the major challenges faced so far span a variety of facets of evaluation. Ultimately, though, these challenges can all be addressed at least partially, by two key approaches: namely by keeping schools central to research by considering their needs and by the early planning of trials to improve implementation, communication and delivery.

KEYWORDS:

• Randomised controlled trials
• education
• sample size
• implementation and process evaluation
• testing
• school recruitment

Acknowledgments

We would like to thank James Turner, Mathew Van Poortvliet and Professor Steve Higgins for reviewing earlier drafts of this paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The five features of trial security as defined by EEF are: design, minimum detectable effect size, attrition, balance and threats to internal validity.