A new methodological approach for evaluating the impact of educational intervention implementation on learning outcomes

References

• Anders, J. D., C. Brown, M. Ehren, T. Greany, R. Nelson, J. Heal, A. Groot, M. Sanders, and R. Allen. 2017. Evaluation of Complex Whole-school Interventions: Methodological and Practical Considerations. London: Education Endowment Foundation.
   Google Scholar
• Anders, Y., H. G. Rossbach, S. Weinert, S. Ebert, S. Kuger, S. Lehrl, and J. von Maurice. 2012. “Home and Preschool Learning Environments and Their Relations to the Development of Early Numeracy Skills.” Early Childhood Research Quarterly 27 (2): 231–244.
   Web of Science ®Google Scholar
• Askell-Williams, H., K. L. Dix, M. J. Lawson, and P. T. Slee. 2013. “Quality of Implementation of a School Mental Health Initiative and Changes Over Time in Students’ Social and Emotional Competencies.” School Effectiveness and School Improvement 24 (3): 357–381.
   Web of Science ®Google Scholar
• Beach, R., and D. O’Brien. 2015. Using Apps for Learning Across the Curriculum. A Literacy-based Framework and Guide. New York: Routledge.
   Google Scholar
• Berkel, C., A. M. Mauricio, E. Schoenfelder, and I. N. Sandler. 2011. “Putting the Pieces Together: An Integrated Model of Program Implementation.” Prevention Science 12 (1): 23–33.
   PubMed Web of Science ®Google Scholar
• Biesta, G. J. 2010. “Why ‘What Works’ Still Won’t Work: From Evidence-based Education to Value-based Education.” Studies in Philosophy and Education 29 (5): 491–503.
   Web of Science ®Google Scholar
• Blackwell, C. K., A. R. Lauricella, and E. Wartella. 2014. “Factors Influencing Digital Technology Use in Early Childhood Education.” Computers & Education 77: 82–90.
   Web of Science ®Google Scholar
• Brantlinger, E., R. Jimenez, J. Klingner, M. Pugach, and V. Richardson. 2005. “Qualitative Studies in Special Education.” Exceptional Children 71 (2): 195–207.
   Web of Science ®Google Scholar
• Bronfenbrenner, U. 1979. The Ecology of Human Development. Massachusetts: Harvard University Press.
   Google Scholar
• Clarke, D. 2004. “Structured Judgement Methods – the Best of Both Worlds?” In Mixing Methods in Psychology: The Integration of Qualitative and Quantitative Practice, edited by Z. Todd, B. Nerlich, S. McKeown, and D. Clarke, 79–99. Hove: Psychology Press.
   Google Scholar
• Clarke, V., and V. Braun. 2014. “Thematic Analysis.” In Encyclopedia of Quality of Life and Well-being Research, edited by A. C. Michalos, 6626–6628. Dordrecht: Springer.
   Google Scholar
• Cohen, L., L. Manion, and K. Morrison. 2007. Research Methods in Education. New York: Routledge.
   Google Scholar
• Connolly, P., A. Biggart, S. Miller, L. O'Hare, and A. Thurston. 2017. Using Randomised Controlled Trials in Education. London: Sage.
   Google Scholar
• Connolly, P., C. Keenan, and K. Urbanska. 2018. “The Trials of Evidence-based Practice in Education: A Systematic Review of Randomised Controlled Trials in Education Research 1980–2016.” Educational Research 60 (3): 276–291.
   Web of Science ®Google Scholar
• Cook, B. G., and S. L. Odom. 2013. “Evidence-based Practices and Implementation Science in Special Education.” Exceptional Children 79 (2): 135–144.
   Web of Science ®Google Scholar
• Couse, L. J., and D. W. Chen. 2010. “A Tablet Computer for Young Children? Exploring its Viability for Early Childhood Education.” Journal of Research on Technology in Education 43 (1): 75–96.
   Google Scholar
• Cragg, L., S. Keeble, S. Richardson, H. E. Roome, and C. Gilmore. 2017. “Direct and Indirect Influences of Executive Functions on Mathematics Achievement.” Cognition 162: 12–26.
   PubMed Web of Science ®Google Scholar
• Creswell, J. W., and D. Creswell. 2018. Research Design. Qualitative, Quantitative, & Mixed Methods Approaches. London: Sage.
   Google Scholar
• Creswell, J. W., R. Shope, V. L. Plano Clark, and D. O. Green. 2006. “How Interpretive Qualitative Research Extends Mixed Methods Research.” Research in the Schools 13 (1): 1–11.
   Google Scholar
• Domitrovich, C. E., S. D. Gest, D. Jones, S. Gill, and R. M. S. DeRousie. 2010. “Implementation Quality: Lessons Learned in the Context of the Head Start REDI Trial.” Early Childhood Research Quarterly 25 (3): 284–298.
   PubMed Web of Science ®Google Scholar
• Durlak, J. A., and E. P. DuPre. 2008. “Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation.” American Journal of Community Psychology 4: 327–350.
   Google Scholar
• Elliott, J. 2001. “Making Evidence-based Practice Educational.” British Educational Research Journal 27 (5): 555–574.
   Web of Science ®Google Scholar
• Elliott, D. S., and S. Mihalic. 2004. “Issues in Disseminating and Replicating Effective Prevention Programs.” Prevention Science 5 (1): 47–53.
   PubMed Web of Science ®Google Scholar
• Evans, R., J. Scourfield, and S. Murphy. 2015. “Pragmatic, Formative Process Evaluations of Complex Interventions and Why We Need More of Them.” Journal of Epidemiology and Community Health 69: 925–926.
   PubMed Web of Science ®Google Scholar
• Forman, S. G., S. S. Olin, K. E. Hoagwood, M. Crowe, and N. Saka. 2009. “Evidence-based Interventions in Schools: Developers’ Views of Implementation Barriers and Facilitators.” School Mental Health 1 (1): 26–36.
   Google Scholar
• Gambaro, L., K. Stewart, and J. Waldfogel. 2015. “A Question of Quality: Do Children from Disadvantaged Backgrounds Receive Lower Quality Early Childhood Education and Care?” British Educational Research Journal 41 (4): 553–574.
   Web of Science ®Google Scholar
• Gathercole, S. E., and T. P. Alloway. 2006. “Practitioner Review: Short-term and Working Memory Impairments in Neurodevelopmental Disorders: Diagnosis and Remedial Support.” Journal of Child Psychology and Psychiatry 47 (1): 4–15.
   PubMed Web of Science ®Google Scholar
• Grant, A., E. Wood, A. Gottardo, M. A. Evans, L. Phillips, and R. Savage. 2012. “Assessing the Content and Quality of Commercially Available Reading Software Programs: Do They Have the Fundamental Structures to Promote the Development of Early Reading Skills in Children?” NHSA Dialog 15 (4): 319–342.
   Google Scholar
• Gray, P. 2015. Free to Learn: Why Unleashing the Instinct to Play Will Make our Children Happier, More Self-reliant, and Better Students for Life. New York: Basic Books.
   Google Scholar
• Green, C. S., D. Bavelier, A. F. Kramer, S. Vinogradov, U. Ansorge, K. K. Ball, U. Bingel, et al. 2019. “Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement.” Journal of Cognitive Enhancement 3 (1): 2–29.
   Google Scholar
• Hammersley, M. 2007. Educational Research and Evidence-based Practice. London: Sage.
   Google Scholar
• Hanley, P., B. Chambers, and J. Haslam. 2016. “Reassessing RCTs as the ‘Gold Standard’: Synergy Not Separatism in Evaluation Designs.” International Journal of Research & Method in Education 39 (3): 287–298.
   Web of Science ®Google Scholar
• Hansen, W. B. 2014. “Measuring Fidelity.” In Defining Prevention Science, edited by Z. Sloboda and H. Petras, 335–359. New York: Springer.
   Google Scholar
• Hansen, W. B., M. M. Pankratz, L. Dusenbury, S. M. Giles, D. C. Bishop, J. Albritton, L. P. Albritton, and J. Strack. 2013. “Styles of Adaptation: The Impact of Frequency and Valence of Adaptation on Preventing Substance Use.” Health Education 113 (4): 345–363.
   Google Scholar
• Haynes, L., O. Service, B. Goldacre, and D. Torgerson. 2012. Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. London: Cabinet Office.
   Google Scholar
• Herodotou, C. 2018. “Young Children and Tablets: A Systematic Review of Effects on Learning and Development.” Journal of Computer Assisted Learning 34 (1): 1–9.
   Web of Science ®Google Scholar
• Hilton, A. 2016. “Engaging Primary School Students in Mathematics: Can iPads Make a Difference?” International Journal of Science and Mathematics Education. doi:10.1007/s10763-016-9771-5.
   Web of Science ®Google Scholar
• Hirsh-Pasek, K., J. M. Zosh, R. M. Golinkoff, J. H. Gray, M. B. Robb, and J. Kaufman. 2015. “Putting Education in “Educational” Apps: Lessons from the Science of Learning.” Psychology Science 16 (1): 3–34.
   Web of Science ®Google Scholar
• Hodkinson, P., and J. Smith. 2004. “The Relationship Between Research, Policy and Practice.” In Evidence-based Practice in Education, edited by G. Thomas and R. Pring, 150–163. Maidenhead: Open University Press.
   Google Scholar
• Hoffmann, T. C., P. P. Glasziou, I. Boutron, R. Milne, R. Perera, D. Moher, D. G. Altman, et al. 2014. “Better Reporting of Interventions: Template for Intervention Description and Replication (TIDieR) Checklist and Guide.” BMJ 348: g1687.
   PubMed Web of Science ®Google Scholar
• Humphrey, N. 2013. Social and Emotional Learning: A Critical Appraisal. London: Sage.
   Google Scholar
• Humphrey, N., A. Lendrum, E. Ashworth, K. Frearson, R. Buck, and K. Kerr. 2016. Implementation and Process Evaluation (IPE) for Interventions in Educational Settings: A Synthesis of the Literature. London: Education Endowment Foundation.
   Google Scholar
• Irwin, D. M., and M. M. Bushnell. 1980. Observational Strategies for Child Study. New York: Holt, Rinehart, and Winston.
   Google Scholar
• Kalaycioğlu, D. B. 2015. “The Influence of Socioeconomic Status, Self-efficacy, and Anxiety on Mathematics Achievement in England, Greece, Hong Kong, the Netherlands, Turkey, and the USA.” Educational Sciences: Theory and Practice 15 (5): 1391–1401.
   Google Scholar
• Kirschner, P. A., J. Sweller, and R. E. Clark. 2006. “Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-based, Experiential, and Inquiry-based Teaching.” Educational Psychologist 41 (2): 75–86.
   Web of Science ®Google Scholar
• Kucian, K., U. Grond, S. Rotzer, B. Henzi, C. Schönmann, F. Plangger, M. Gälli, E. Martin, and M. von Aster. 2011. “Mental Number Line Training in Children with Developmental Dyscalculia.” Neuroimage 57 (3): 782–795.
   PubMed Web of Science ®Google Scholar
• Kucirkova, N. 2014. “iPads in Early Education: Separating Assumptions and Evidence.” Frontiers in Psychology 5: 715. doi:10.3389/fpsyg.2014.00715.
   PubMed Web of Science ®Google Scholar
• Langley, G. J., R. D. Moen, K. M. Nolan, T. W. Nolan, C. L. Norman, and L. P. Provost. 2009. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco: Jossey-Bass.
   Google Scholar
• Lendrum, A., N. Humphrey, and M. Greenberg. 2016. “Implementing for Success in School-based Mental Health Promotion: The Role of Quality in Resolving the Tension Between Fidelity and Adaptation.” In Mental Health and Wellbeing Through Schools: The Way Forward, edited by R. Shute and P. Slee, 53–63. London: Taylor and Francis.
   Google Scholar
• Lortie-Forgues, H., and M. Inglis. 2019. “Rigorous Large-scale Educational RCTs are Often Uninformative: Should We Be Concerned?” Educational Researcher 48 (3): 158–166.
   Web of Science ®Google Scholar
• Miles, M. B., and A. M. Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. London: Sage.
   Google Scholar
• Moore, J. E., B. K. Bumbarger, and B. R. Cooper. 2013. “Examining Adaptations of Evidence-based Programs in Natural Contexts.” The Journal of Primary Prevention 34 (3): 147–161.
   PubMed Web of Science ®Google Scholar
• Morse, J. M. 1991. “Approaches to Qualitative-quantitative Methodological Triangulation.” Nursing Research 40 (2): 120–123.
   PubMed Web of Science ®Google Scholar
• Naylor, P. J., L. Nettlefold, D. Race, C. Hoy, M. C. Ashe, J. W. Higgins, and H. A. McKay. 2015. “Implementation of School Based Physical Activity Interventions: A Systematic Review.” Preventive Medicine 72: 95–115.
   PubMed Web of Science ®Google Scholar
• Nilsen, P. 2015. “Making Sense of Implementation Theories, Models and Frameworks.” Implementation Science 10 (53): 1–13.
   PubMedGoogle Scholar
• Oakley, A., V. Strange, C. Bonell, E. Allen, and J. Stephenson. 2006. “Process Evaluation in Randomised Controlled Trials of Complex Interventions.” BMJ 332 (7538): 413–416.
   PubMed Web of Science ®Google Scholar
• O’Donnell, C. L. 2008. “Defining, Conceptualizing, and Measuring Fidelity of Implementation and its Relationship to Outcomes in K–12 Curriculum Intervention Research.” Review of Educational Research 78 (1): 33–84.
   Web of Science ®Google Scholar
• Ofsted. 2013. Annual Report 2012/2013: East Midlands Regional Report. London: Ofsted.
   Google Scholar
• Ofsted. n.d. Inspection Reports. Retrieved from https://reports.beta.ofsted.gov.uk for each participating school .
   Google Scholar
• Outhwaite, L. A., M. Faulder, A. Gulliford, and N. J. Pitchford. 2018. “Raising Early Achievement in Math with Interactive Apps: A Randomized Control Trial.” Journal of Educational Psychology 111 (2): 284–298.
   PubMed Web of Science ®Google Scholar
• Outhwaite, L. A., A. Gulliford, and N. J. Pitchford. 2017. “Closing the Gap: Efficacy of a Tablet Intervention to Support the Development of Early Mathematical Skills in UK Primary School Children.” Computers & Education 108: 43–58.
   Web of Science ®Google Scholar
• Pawson, R., and N. Tilley. 1997. “Realistic Evaluation.” In Encyclopedia of Evaluation, edited by S. Matthieson, 359–367. London: Sage.
   Google Scholar
• Peterson, A. 2016. “Getting ‘What Works’ Working: Building Blocks for the Integration of Experimental and Improvement Science.” International Journal of Research & Method in Education 39 (3): 299–313.
   Web of Science ®Google Scholar
• Pitchford, N. J. 2015. “Development of Early Mathematical Skills with a Tablet Intervention: A Randomized Control Trial in Malawi.” Frontiers in Psychology 6: 485. doi:10.3389/fpsyg.2015.00485.
   PubMed Web of Science ®Google Scholar
• Punch, K. F. 2013. Introduction to Social Research: Quantitative and Qualitative Approaches. London: Sage.
   Google Scholar
• Robson, C., and K. McCartan. 2011. Real World Research. Chichester: Wiley.
   Google Scholar
• Rogers, E. M. 2003. The Diffusion of Innovations. New York: Free Press.
   Google Scholar
• Saldaña, J. 2015. The Coding Manual for Qualitative Researchers. London: Sage.
   Google Scholar
• Schagen, S., S. Blenkinsop, I. Schagen, E. Scott, M. Eggers, I. Warwick, E. Chase, and P. Aggleton. 2005. “Evaluating the Impact of the National Healthy School Standard: Using National Datasets.” Health Education Research 20 (6): 688–696.
   PubMed Web of Science ®Google Scholar
• Shaffer, P. 2011. “Against Excessive Rhetoric in Impact Assessment: Overstating the Case for Randomised Controlled Experiments.” Journal of Development Studies 47 (11): 1619–1635.
   Web of Science ®Google Scholar
• Slavin, R. 2012. “Foreward.” In Handbook of Implementation Science for Psychology in Education, edited by B. Kelly, and D. F. Perkins, xv. Cambridge: Cambridge University Press.
   Google Scholar
• Strand, S., L. Malmberg, and J. Hall. 2015. English as an Additional Language (EAL) and Educational Achievement in England: An Analysis of the National Pupil Database. London: Education Endowment Foundation.
   Google Scholar
• Tashakkori, A., and C. Teddlie. 2010. “Putting the Human Back in ‘Human Research Methodology’: The Researcher in Mixed Methods Research.” Journal of Mixed Methods Research 4 (4): 271–277.
   Web of Science ®Google Scholar
• Thomas, G. 2016. “After the Gold Rush: Questioning the ‘Gold Standard’ and Reappraising the Status of Experiment and Randomized Controlled Trials in Education.” Harvard Educational Review 86 (3): 390–411.
   Web of Science ®Google Scholar
• Thomas, G., and R. Pring. 2004. Evidence-based Practice in Education. Maidenhead: Open University Press.
   Google Scholar
• Torgerson, D. J., and C. J. Torgerson. 2003. “Avoiding Bias in Randomised Controlled Trials in Educational Research.” British Journal of Educational Studies 51 (1): 36–45.
   Web of Science ®Google Scholar
• Vignoles, A., J. Jerrim, and R. Cowan. 2015. Mathematics Mastery: Primary Evaluation Report. London: Education Endowment Foundation.
   Google Scholar
• Wyse, D., and C. Torgerson. 2017. “Experimental Trials and ‘What Works?’ In Education: The Case of Grammar for Writing.” British Educational Research Journal 43 (6): 1019–1047.
   Web of Science ®Google Scholar