Every Child Counts: testing policy effectiveness using a randomised controlled trial, designed, conducted and reported to CONSORT standards

Abstract

We report a randomised controlled trial evaluation of an intensive one-to-one numeracy programme – Numbers Count – which formed part of the previous government's numeracy policy intervention – Every Child Counts. We rigorously designed and conducted the trial to CONSORT guidelines. We used a pragmatic waiting list design to evaluate the intervention in real life settings in diverse geographical areas across England, to increase the ecological validity of the results. Children were randomly allocated within schools to either the intervention (Numbers Count in addition to normal classroom practice) or the control group (normal classroom practice alone). The primary outcome assessment was the Progress in Maths (PIM) 6 test from GL Assessment. Independent administration ensured that outcome ascertainment was undertaken blind to group allocation. The secondary outcome measure was the Sandwell test, which was not undertaken and marked blind to group allocation. At post-test the effect size (standardised mean difference between intervention and control group) on the PIM6 was d = 0.33 95% confidence intervals [0.12, 0.53], indicating strong evidence of a difference between the two groups. The effect size for the secondary outcome (Sandwell test) was d = 1.11 95% CI [0.91, 1.31]. Our results demonstrate a statistically significant effect of Numbers Count on our primary, independently marked, mathematics test. Like many trials, our study had both strengths and limitations. We feel, however, due to our a priori decision to report these in an explicit manner, as advocated by the CONSORT guidelines, that we could maximise rigour (e.g., by using blinded independent testing) and report potential problems (e.g., attrition rates). We have demonstrated that it is feasible to conduct an educational trial using the rigorous methodological techniques required by the CONSORT statement.

Keywords:

• experimental design
• mathematics education
• CONSORT guidelines

Acknowledgements

We acknowledge all members of the ECC evaluation team: Carole Torgerson, Andy Wiggins, David Torgerson, Hannah Ainsworth, Patrick Barmby, Catherine Hewitt, Karen Jones, Vivien Hendry, Mike Askew, Martin Bland, Rob Coe, Steve Higgins, Jeremy Hodgen, Charles Hulme and Peter Tymms. The trial described in this article was funded by the Department for Children, Schools and Families (DCFS). We also acknowledge Durham University and the University of York for additional funding to support the trial. The corresponding author was employed by the University of York when the field work for the trial was undertaken.