Taming randomized controlled trials in education: exploring key claims, issues and debates

In his recent book, Keith Morrison argues that RCTs have garnered a privileged place in educational research, driving out other valuable approaches. Morrison acknowledges RCTs or similar designs can be useful in limited situations and warns against ‘unsupported whims, personal preferences, politically expedient, untried interventions’ (p. 24). Still, he presents RCTs as deeply flawed on technical and philosophical grounds and in need of ‘taming.’ In fact, many of his reproofs are aimed not just at RCTs, but the entire project of ‘evidence-based practice’ in education.

Morrison opposes ‘evidence’ as something externally verifiable. Instead, he shows it exists as part of an argument, evidence for something. RCTs pursue causality by ‘controlling out’ (p. 134) specific factors, but Morrison believes this is too simplistic. He appeals to ‘chaos’ and ‘complexity’ theories to highlight the consequences of minute changes to the educational situation. RCTs may miss critical ingredients in the ‘causal cake’ (p. 39). Morrison urges not only greater qualitative reporting of trials, but a clear focus on the causal processes in complex educational situations. Any attempt to reduce this complexity through research is presented as inherently political.

The book demonstrates a deep familiarity with RCT affordances and guidelines. For example, withholding an intervention from a control group may seem unfair, yet Morrison endorses the controlling principle of equipoise, a genuine uncertainty regarding the superior condition. Morrison spends a chapter addressing RCT analogies from healthcare. He shows educational trials need to consider intervention dosage and seek ongoing versus one-off consent. Unlike healthcare, blinding/masking is often impossible in educational trials, and context may matter more. According to Morrison, rather than ‘noise,’ contextual factors may be part of the ‘signal’ (p. 119) of ‘what works.’

Meta-analyses, which often rely on RCTs, are scrutinized extensively. Morrison shows the difficulty of comparing studies which have different sampling, assessments, and interventions (p. 185), are non-randomized, or use inappropriate analyses (See, 2018, cited in Morrison, p. 187). Morrison stresses the shortcomings of both statistical significance and effect size as summary measures. The impression is that, like RCTs, many meta-analyses are guided by rules not discernment and are ‘ … simply statistical processing with a cranked-out outcome’ (p. 188).

Morrison excels in promoting a critical view of educational trials. Rather than being straightforward measures of effectiveness, RCTs need careful interpretation. Even well-run RCTs may be unreplicable or ungeneralizable. Morrison also defends teachers’ rights to agency, rather than having all decisions dictated by ‘evidence.’ As a field, education needs to grapple with issues of values, such as the aims for education and what it means for education to improve. Morrison believes the field is side-stepping these to pursue ‘effectiveness,’ even with little translation of research to policy. He is not alone in raising such concerns, and he references voices on multiple sides of this debate. This book would make a good primer for educators engaging with research for the first time, who might adopt an oversimplified view of effectiveness or of research generally.

Morrison’s book is not without its faults, however, both in form and occasionally in substance. While expressing valid criticisms, his severity could stymie dialogue. Morrison criticizes early RCT advocates for their ‘ … Messianic fervour’ (p. 72), but his own zeal occasionally crosses the line:

… all the sparkling complexity of an intervention, its conditions and settings, its peoples, its multiple channels and directions of webs of causality and ‘causal cakes’, are reduced to a single figure; how demeaning; what a travesty of inclusive education … (p. 166)

In the same vein, the book closes with ‘fifty theses’ against RCTs, an intentional provocation to ‘crystallize’ (p. 212) the issues.

Morrison’s arguments would benefit from greater use of real instead of hypothetical examples, and he sometimes overlooks well-known counters to his criticisms. For example, when discussing attrition and contamination which spoil randomization, Morrison fails to consider intention-to-treat and sensitivity analyses as potential remedies, though he mentions both elsewhere (pp. 140, 34). He cites a worrying analysis that RCTs sponsored by major US and UK bodies are underpowered (Lortie-Forgues and Inglis, 2019, p. 162, as cited in Morrison, p. 150), but this post hoc analysis may not reflect the original researchers’ intents. And although Morrison repeatedly disparages ‘average’ effects (pp. 150–154), sometimes these might be the most applicable to mixed classrooms, where students belong to multiple subgroups.

Finally, it remains unclear what Morrison thinks should be the purposes of and models for educational research. Many of his criticisms apply even to qualitative designs, as he admits (p. 45). Complexity reduction is inherent to research, and even the richest case study selects and interprets the details. Focusing on individuals rather than groups may not always yield useful approaches for classrooms, schools, and state-level systems. Also, the still young evidence-based practice movement seeks meaningful engagement with stakeholders, not just ‘evidence.’ As RCTs and other trials continue, their benefits may be clearest within a planned cycle of research (Gorard, 2013, p. 14) alongside theoretical and interpretive research to uncover values issues, define questions, propose interventions, and understand processes at work. When read with the same criticality that Morrison encourages for RCTs, his book could be a valuable resource in this cycle.