Evaluating the Impact of Implementation Planning: A Preregistered Meta-Analysis

REFERENCES

References marked with an asterisk indicate studies included in the meta-analysis.

• *Altschaefl, M. R. (2015). Promoting intervention fidelity of parent- and teacher-delivered math fluency interventions: An adult behavior change intervention (Publication No. 3715049) [Doctoral dissertation]. [ProQuest Dissertations & Theses Global]. University of Wisconsin-Madison.
   Google Scholar
• Barrett, C. A., Spear, S. E., Clinkscales, A., Wood, L. L., & Maki, K. E. (2023). What interventions are cost-effective? A systematic review of cost-effectiveness analyses of school-based programs from 2000 to 2020. School Psychology, Advance online publication. https://doi.org/10.1037/spq0000590
   Google Scholar
• Ben, T. A. (2023). A meta-analysis of implementation planning: The effects on teachers’ treatment adherence and quality when implementing behavioral interventions (Publication No. 982) [Master’s thesis]. University of Southern Mississippi, Aquila.
   Google Scholar
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effect models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. https://doi.org/10.1002/jrsm.12
   PubMed Web of Science ®Google Scholar
• *Boyle, A. (2019). Supporting paraeducators’ intervention fidelity to behavior support plans through implementation planning (Publication No. 2200) [Doctoral dissertation]. University of Connecticut. Retrieved from https://opencommons.uconn.edu/dissertations/2200
   Google Scholar
• Byron, J., Sanetti, L. M. H., & Charbonneau, S. (2021). Increasing teacher treatment fidelity to cover, copy, compare through consultation and computer-based implementation planning. International Journal of School & Educational Psychology, 9(sup1), S88–S102. https://doi.org/10.1080/21683603.2020.1753605
   Google Scholar
• *Byron, J. (2016). Implementation planning as a proactive approach to intervention fidelity maintenance of an academic intervention (Publication No. 1200). [Doctoral dissertation]. University of Connecticut. Retrieved from https://opencommons.uconn.edu/dissertations/1200
   Google Scholar
• Center for Open Science. (2022). Open Science Framework (OSF). Retrieved from https://www.cos.io/
   Google Scholar
• Chaffee, R. K., Briesch, A. M., Johnson, A. H., & Volpe, R. J. (2017). A meta-analysis of class-wide interventions for supporting student behavior. School Psychology Review, 46(2), 149–164. https://doi.org/10.17105/SPR-2017-0015.V46-2
   Web of Science ®Google Scholar
• Charlton, C. T., Rigby, D. M. G., Moulton, S. E., Sabey, C. V., & Richardson, M. J. (2023). Implementing behavior intervention plans in schools: A pilot study of the complex relationship between technical adequacy, treatment integrity, and student outcomes. Journal of Behavioral Education, 32(2), 382–403. https://doi.org/10.1007/s10864-021-09448-z
   Web of Science ®Google Scholar
• Cochrane, W. S., Sanetti, L. M. H., & Minster, M. C. (2019). School psychologists’ beliefs and practices about treatment integrity in 2008 and 2017. Psychology in the Schools, 56(3), 295–305. https://doi.org/10.1002/pits.22177
   Web of Science ®Google Scholar
• Codding, R. S. (2021). School Psychology vision for the future. School Psychology (Washington, D.C.), 36(1), 1–5. https://doi.org/10.1037/spq0000425
   PubMedGoogle Scholar
• Collier-Meek, M. A., Fallon, L. M., & Gould, K. (2018). How are treatment integrity data assessed? Reviewing the performance feedback literature. School Psychology Quarterly: The Official Journal of the Division of School Psychology, American Psychological Association, 33(4), 517–526. https://doi.org/10.1037/spq0000239
   PubMed Web of Science ®Google Scholar
• Collier-Meek, M. A., Sanetti, L. M., Fallon, L., & Chafouleas, S. (2020). Exploring the influences of assessment method, intervention steps, intervention sessions, and observation timing on treatment fidelity estimates. Assessment for Effective Intervention, 46(1), 3–13. https://doi.org/10.1177/1534508419857228
   Web of Science ®Google Scholar
• Cook, B. G., Johnson, A. H., Maggin, D. M., Therrien, W. J., Barton, E. E., Lloyd, J. W., Reichow, B., Talbott, E., & Travers, J. C. (2022). Open science and single-case design research. Remedial and Special Education, 43(5), 359–369. https://doi.org/10.1177/0741932521996452
   Web of Science ®Google Scholar
• Cook, B. G., Lloyd, J. W., Mellor, D., Nosek, B. A., & Therrien, W. J. (2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85(1), 104–118. https://doi.org/10.1177/0014402918793138
   Web of Science ®Google Scholar
• Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23–45. https://doi.org/10.1016/S0272-7358(97)00043-3
   PubMed Web of Science ®Google Scholar
• DiGennaro Reed, F. D., Reed, D. D., Baez, C. N., & Maguire, H. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44(3), 611–615. https://doi.org/10.1901/jaba.2011.44-611
   PubMed Web of Science ®Google Scholar
• Fallon, L. M., Collier-Meek, M. A., Famolare, G. M., DeFouw, E. R., & Gould, K. M. (2022). Electronically delivered support to promote intervention implementation fidelity: A research synthesis. School Psychology Review, 51(2), 221–236. https://doi.org/10.1080/2372966X.2020.1853485
   Web of Science ®Google Scholar
• Fallon, L. M., Collier-Meek, M. A., Maggin, D. M., Sanetti, L. M. H., & Johnson, A. H. (2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case research. Exceptional Children, 81(2), 227–246. https://doi.org/10.1177/0014402914551738
   Web of Science ®Google Scholar
• Haring, N. G., Lovitt, T. C., Eaton, M. D., & Hansen, C. L. (1978). The fourth R: Research in the classroom. Charles E. Merrill Publishing.
   Google Scholar
• Hasson, H., Blomberg, S., & Dunér, A. (2012). Fidelity and moderating factors in complex interventions: A case study of a continuum of care program for frail elderly people in health and social care. Implementation Science: IS, 7(1), 23. https://doi.org/10.1186/1748-5908-7-23
   PubMedGoogle Scholar
• Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504. https://doi.org/10.1037//1082-989X.3.4.486
   Web of Science ®Google Scholar
• Johnson, A. H., & Cook, B. G. (2019). Preregistration in single-case design research. Exceptional Children, 86(1), 95–112. https://doi.org/10.1177/0014402919868529
   Web of Science ®Google Scholar
• Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. https://doi.org/10.3102/0034654318759268
   Web of Science ®Google Scholar
• Kratochwill, T. R., Levin, J. R., & Horner, R. H. (2017). Negative results: Conceptual and methodological dimensions in single-case intervention research. Remedial and Special Education, 39(2), 67–76. https://doi.org/10.1177/0741932517741721
   Web of Science ®Google Scholar
• Levin, H. M., & Belfield, C. (2015). Guiding the development and use of cost-effectiveness analysis in education. Journal of Research on Educational Effectiveness, 8(3), 400–418. https://doi.org/10.1080/19345747.2014.915604
   Web of Science ®Google Scholar
• *Long, A. C., Sanetti, L. M. H., Lark, C. R., & Connolly, J. J. (2018). Examining behavioral consultation plus computer-based implementation planning on teachers’ intervention implementation in an alternative school. Remedial and Special Education, 39(2), 106–117. https://doi.org/10.1016/j.jsp.2015.12.002
   Web of Science ®Google Scholar
• Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021). Both questionable and open research practices are prevalent in education research. Educational Researcher, 50(8), 493–504. https://doi.org/10.3102/0013189X211001356
   Web of Science ®Google Scholar
• Merle, J. L., Thayer, A. J., Larson, M. F., Pauling, S., Cook, C. R., Rios, J. A., McGinnis, J. L., & Sullivan, M. M. (2022). Investigating strategies to increase general education teachers’ adherence to evidence-based social-emotional behavior practices: A meta-analysis of the single-case literature. Journal of School Psychology, 91, 1–26. https://doi.org/10.1016/j.jsp.2021.11.005
   PubMed Web of Science ®Google Scholar
• Metz, A., Albers, B., Burke, K., Bartley, L., Louison, L., Ward, C., & Farley, A. (2021). Implementation practice in human service systems: Understanding the principles and competencies of professionals who support implementation. Human Service Organizations: Management, Leadership, & Governance, 45(3), 238–259. https://doi.org/10.1080/23303131.2021.1895401
   Web of Science ®Google Scholar
• McClain, M. B., Callan, G. L., Harris, B., Floyd, R. G., Haverkamp, C. R., Golson, M. E., Longhurst, D. N., & Benallie, K. J. (2021). Methods for addressing publication bias in school psychology journals: A descriptive review of meta-analyses from 1980 to 2019. Journal of School Psychology, 84, 74–94. https://doi.org/10.1016/j.jsp.2020.11.002
   PubMed Web of Science ®Google Scholar
• Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097
   PubMed Web of Science ®Google Scholar
• Noell, G. H., Gansle, K. A., Mevers, J. L., Knox, R. M., Mintz, J. C., & Dahir, A. (2014). Improving treatment plan implementation in schools: A meta-analysis of single subject design studies. Journal of Behavioral Education, 23(1), 168–191. https://doi.org/10.1007/s10864-013-9177-1
   Google Scholar
• Noell, G. H., Gresham, F. M., & Gansle, K. A. (2002). Does treatment integrity matter? A preliminary investigation of instructional implementation and mathematics performance. Journal of Behavioral Education, 11(1), 51–67. https://doi.org/10.1023/A:1014385321849
   Google Scholar
• Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., Koenig, J. L., Resetar, J. L., & Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34(1), 87–106. https://doi.org/10.1080/02796015.2005.12086277
   Web of Science ®Google Scholar
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
   PubMed Web of Science ®Google Scholar
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 6251, aac4716. https://doi.org/10.1126/science.aac4716
   Web of Science ®Google Scholar
• Pierce. (2021, April 22). Understanding the impact of implementation planning on teacher treatment integrity: A meta-­analytic study. https://doi.org/10.17605/OSF.IO/38MXR
   Google Scholar
• Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7
   PubMed Web of Science ®Google Scholar
• Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20(3), 342–359. https://doi.org/10.1037/met0000019
   PubMed Web of Science ®Google Scholar
• Pustejovsky, J. E. (2018). Using response ratios for meta-analyzing single-case designs with behavioral outcomes. Journal of School Psychology, 68, 99–112. https://doi.org/10.1016/j.jsp.2018.02.003
   PubMed Web of Science ®Google Scholar
• Pustejovsky, J. E. (2020). clubSandwich: Cluster-Robust (Sandwich) variance estimators with small-sample corrections (0.4.2) [R package]. Retrieved from https://github.com/jepusto/clubSandwich
   Google Scholar
• Pustejovsky, J. E., & Tipton, E. (2022). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science: The Official Journal of the Society for Prevention Research, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3
   PubMed Web of Science ®Google Scholar
• R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
   Google Scholar
• Rhymer, K. N., Evans-Hampton, T. N., McCurdy, M., & Watson, T. S. (2002). Effects of varying levels of treatment integrity on toddler aggressive behavior. Special Services in the Schools, 18(1-2), 75–82. https://doi.org/10.1300/J008v18n01_05
   Google Scholar
• Rohatgi, A. (2019). WebPlotDigitizer (Version 4.2) [Computer software]. Retrieved from https://automeris.io/WebPlotDigitizer/
   Google Scholar
• Sanetti, L. M. H., Charbonneau, S., Knight, A., Cochrane, W. S., Kulcyk, M. C. M., & Kraus, K. E. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57(6), 901–922. https://doi.org/10.1002/pits.22364
   Web of Science ®Google Scholar
• Sanetti, L. M. H., & Collier-Meek, M. A. (2014). Increasing the rigor of procedural fidelity assessment: An empirical comparison of direct observation and permanent product review methods. Journal of Behavioral Education, 23(1), 60–88. https://doi.org/10.1007/s10864-013-9179-z
   Google Scholar
• Sanetti, L. M. H., & Collier-Meek, M. A. (2015). Data-driven delivery of implementation supports in a multi-tiered framework: A pilot study. Psychology in the Schools, 52(8), 815–828. https://doi.org/10.1002/pits.21861
   Web of Science ®Google Scholar
• Sanetti, L. M. H., & Collier-Meek, M. A. (2019). Supporting successful interventions in schools: Tools to plan, evaluate, and sustain effective implementation. Guilford Press.
   Google Scholar
• *Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C., Byron, J., & Kratochwill, T. R. (2015). Increasing teacher intervention fidelity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53(3), 209–229. https://doi.org/10.1016/j.jsp.2015.03.002
   PubMed Web of Science ®Google Scholar
• *Sanetti, L. M. H., Collier‐Meek, M. A., Long, A. C., Kim, J., & Kratochwill, T. R. (2014). Using implementation planning to increase teachers’ adherence and quality to behavior support plans. Psychology in the Schools, 51(8), 879–895. https://doi.org/10.1002/pits.21787
   Web of Science ®Google Scholar
• Sanetti, L. M. H., & Fallon, L. M. (2011). Intervention fidelity assessment: How estimates of adherence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21(3), 209–232. https://doi.org/10.1080/10474412.2011.595163
   Web of Science ®Google Scholar
• Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.
   Web of Science ®Google Scholar
• Sanetti, L. M. H., Kratochwill, T. R., Collier-Meek, M. A., Long, A. C. J. (2014). PRIME: Planning realistic implementation and maintenance by educators. University of Connecticut. Retrieved from https://implementationscience.uconn.edu
   Google Scholar
• Sanetti, L. M., Kratochwill, T. R., & Long, A. C. (2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly: The Official Journal of the Division of School Psychology, American Psychological Association, 28(1), 47–62. https://doi.org/10.1037/spq0000007
   PubMed Web of Science ®Google Scholar
• *Sanetti, L. M. H., Williamson, K. M., Long, A. C., & Kratochwill, T. R. (2018). Increasing in-service teacher implementation of classroom management practices through consultation, implementation planning, and participant modeling. Journal of Positive Behavior Interventions, 20(1), 43–59. https://doi.org/10.1177/1098300717722357
   Web of Science ®Google Scholar
• Schwarzer, R. (1992). Self-efficacy in the adoption and maintenance of health behaviors: Theoretical approaches and a new model. In R. Schwarzer (Ed.), Self-efficacy: Thought control of action (pp. 217–243) Hemisphere.
   Google Scholar
• Schwarzer, R., & Hamilton, K. (2020). Changing behavior using the health action process approach. In M. S. Hagger, L. D. Cameron, K. Hamilton, N. Hankonen, & T. Lintunen (Eds.), The handbook of behavior change (pp. 89–103). Cambridge University Press. https://doi.org/10.1017/9781108677318
   Google Scholar
• Scott, T. M., Gage, N. A., Hirn, R. G., Lingo, A. S., & Burt, J. (2019). An examination of the association between implementation fidelity measures and student outcomes. Preventing School Failure: Alternative Education for Children and Youth, 63(4), 308–316. https://doi.org/10.1080/1045988X.2019.1605971
   Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
   Google Scholar
• Shadish, W. R., Zelinsky, N. A. M., Vevea, J. L., & Kratochwill, T. R. (2016). A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, 49(3), 656–673. https://doi.org/10.1002/jaba.308
   PubMed Web of Science ®Google Scholar
• Solomon, B. G., Klein, S. A., & Politylo, B. C. (2012). The effect of performance feedback on teachers’ treatment integrity: A meta-analysis of the single-case literature. School Psychology Review, 41(2), 160–175. https://doi.org/10.1080/02796015.2012.12087518
   Web of Science ®Google Scholar
• Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
   Web of Science ®Google Scholar
• Wilder, D. A., Atwell, J., & Wine, B. (2006). The effects of varying levels of treatment integrity on child compliance during treatment with a three-step prompting procedure. Journal of Applied Behavior Analysis, 39(3), 369–373. https://doi.org/10.1901/jaba.2006.144-05
   PubMed Web of Science ®Google Scholar