An Exploratory Comparison of Three Treatment Fidelity Assessment Methods: Time Sampling, Event Recording, and Post-observation Checklist

References

• Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. doi:10.1007/s10488-010-0327-7
   PubMed Web of Science ®Google Scholar
• Barnett, D., Hawkins, R., McCoy, D., Wahl, E., Shier, A., Denune, H., & Kimener, L. (2014). Methods used to document procedural fidelity in school-based intervention research. Journal of Behavioral Education, 23, 89–107. doi:10.1007/s10864-013-9188-y
   Google Scholar
• Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., … Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH behavior change consortium. Health Psychology, 23(5), 443–451. doi:10.1037/0278-6133.23.5.443
   PubMed Web of Science ®Google Scholar
• Briesch, A. M., Chafouleas, S. M., & Riley-Tillman, T. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421.
   Web of Science ®Google Scholar
• Briesch, A. M., Volpe, R. J., & Floyd, R. G. (2018). School-based observation: A practical guide for assessing student behavior. New York, NY: Guilford Press.
   Google Scholar
• Chafouleas, S. M., Riley-Tillman, T. C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York, NY: Guilford Press.
   Google Scholar
• Cochrane, W., Sanetti, L. M. H., & Minster, M. (2018). School psychologists’ beliefs and practices about treatment integrity in 2008 and 2017. Psychology in the Schools. doi:10.1002/pits.22177
   PubMed Web of Science ®Google Scholar
• Cochrane, W. S., Sanetti, Lisa M. H.., & Minster, M. C. (2019). School psychologists’ beliefs and practices about treatment integrity in 2008 and 2017. Psychology in the Schools, 56, 295-306. doi:10.1002/pits.2019.56.issue-3
   Web of Science ®Google Scholar
• Codding, R. S., Feinberg, A. B., Dunn, E. K., & Pace, G. M. (2005). Effects of immediate performance feedback on implementation of behavior support plans. Journal of Applied Behavior Analysis, 38, 205–219. doi:10.1901/jaba.2005.98-04
   PubMed Web of Science ®Google Scholar
• Codding, R. S., Livanis, A., Pace, G. M., & Vaca, L. (2008). Using performance feedback to improve treatment integrity of class wide behavior plans: An investigation of observer reactivity. Journal of Applied Behavior Analysis, 41, 417–422. doi:10.1901/jaba.2008.41-417
   PubMed Web of Science ®Google Scholar
• Collier-Meek, M. A., Fallon, L. M., & DeFouw, E. R. (2017). Toward Feasible Implementation Support: E-Mailed Prompts to Promote Teachers' Treatment Integrity. School Psychology Review, 46, 379–394. DOI: 10.17105/SPR-2017-0028.V46-4 4
   Web of Science ®Google Scholar
• Collier-Meek, M. A., Fallon, L. M., & DeFouw, E. R. (2018). Assessing implementation of the good behavior game: Comparing estimates of adherence, quality, and exposure. Assessment for Effective Intervention. doi:10.1177/1534508418782620
   Web of Science ®Google Scholar
• Collier-Meek, M. A., Fallon, L. M., & Gould, K. (2018). How are treatment integrity data assessed? Reviewing the performance feedback literature. School Psychology Quarterly, 33, 517–526. doi:10.1037/spq0000239
   PubMed Web of Science ®Google Scholar
• Collier-Meek, M. A., Johnson, A. H., & Farrell, A. F. (2018). Development and initial evaluation of the measure of active supervision and interaction. Assessment for Effective Intervention, 43, 212–226. doi:10.1177/1534508417737516
   Web of Science ®Google Scholar
• Collier-Meek, M. A., Sanetti, L. M. H., & Boyle, A. M. (2019). Barriers to implementation of classroom management and behavior support plans: An exploratory investigation. Psychology in the Schools, 56, 5–17. doi:10.1002/pits.22127
   Web of Science ®Google Scholar
• Collier-Meek, M. A., Sanetti, L. M. H., & Fallon, L. M. (2017). Incorporating applied behavior analysis to assess and support educator’s treatment integrity. Psychology in the Schools, 54, 446–460. doi:10.1002/pits.22001
   Web of Science ®Google Scholar
• Collier-Meek, M. A., Sanetti, L. M. H., Fallon, L. M., & Chafouleas, S. M. (2019). Exploring the influences of assessment method, intervention steps, intervention sessions, and observation timing on treatment fidelity estimates. Assessment for Effective Intervention. Advance online publication. doi:10.1177/1534508419857228
   Google Scholar
• Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
   Google Scholar
• DiGennaro Reed, F. D., Reed, D. D., Baez, C. N., & Maguire, H. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44, 611–615. doi:10.1901/jaba.2011.44-611
   PubMed Web of Science ®Google Scholar
• Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0
   PubMed Web of Science ®Google Scholar
• Eames, C., Daley, D., Hutchings, J., Hughes, J. C., Jones, K., Martin, P., & Bywater, T. (2008). The leader observation tool: A process skills treatment fidelity measure for the incredible years parenting programme. Child: Care, Health and Development, 34, 391–400. doi:10.1111/j.1365-2214.2008.00828.x
   PubMed Web of Science ®Google Scholar
• Ferguson, T. D., Briesch, A. M., Volpe, R. J., & Daniels, B. (2012). The influence of observation length on the dependability of data. School Psychology Quarterly, 27, 187–197. doi:10.1037/spq0000005
   PubMed Web of Science ®Google Scholar
• Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37–50.
   Web of Science ®Google Scholar
• Gresham, F. M., Dart, E. H., & Collins, T. A. (2017). Generalizability of multiple measures of treatment integrity: Comparisons among direct observation, permanent products, and self-report. School Psychology Review, 46, 108–121. doi:10.17105/SPR46-1.108-121
   Web of Science ®Google Scholar
• Hemmeter, M. L., Snyder, P., Kinder, K., & Artman, K. (2011). Impact of performance feedback delivered via electronic mail on preschool teachers’ use of descriptive praise. Early Childhood Research Quarterly, 26, 96–101. doi:10.1016/j.ecresq.2010.05.004
   Web of Science ®Google Scholar
• Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507-519.
   Web of Science ®Google Scholar
• Hintze, J. M., & Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258–270.
   Web of Science ®Google Scholar
• Johnson, A. H., Chafouleas, S. M., & Briesch, A. M. (2017). Dependability of data derived from time sampling methods with multiple observation targets. School Psychology Quarterly, 32, 22–34. doi:10.1037/spq0000159
   PubMed Web of Science ®Google Scholar
• Johnson, A. H., Goldberg, T., Hinant, R., & Couch, L. (2019). Trends and practices in functional behavior assessments completed by school psychologists. Psychology in the Schools, 56, 360–377. doi:10.1002/pits.22191
   Web of Science ®Google Scholar
• Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. doi:10.1016/j.jcm.2016.02.012
   PubMed Web of Science ®Google Scholar
• Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from What Works Clearinghouse website http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
   Google Scholar
• Massachusetts Department of Education, 2017. School and District Report Cards. Retrieved from http://reportcards.doe.mass.edu
   Google Scholar
• McGill, R. J., & Busse, R. T. (2014). An evaluation of multiple single-case outcome indicators using convergent evidence scaling. Contemporary School Psychology, 18, 13–23. doi:10.1007/s40688-013-0004-4
   Google Scholar
• McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30
   Web of Science ®Google Scholar
• Michie, S., Van Stralen, M. M., & West, R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6, 42–53. doi:10.1186/1748-5908-6-42
   PubMed Web of Science ®Google Scholar
• Noell, G. H., & Gansle, K. A. (2014). Research examining the relationship between consutation procedures, treatment integrity, and outcomes. In W. P. Erchul & S. M. Sheridan (Eds.), Handbook of research in school consultation (pp. 386–408). New York, NY: Routledge.
   Google Scholar
• Noell, G. H., Volz, J. R., Henderson, M. Y., & Williams, K. L. (2017). Evaluating an integrated support model for increasing treatment plan implementation following consultation in schools. School Psychology Quarterly, 32, 525–538. doi:10.1037/spq0000195
   PubMed Web of Science ®Google Scholar
• Parker, R. I., & Vannest, K. J. (2009). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357–367. doi:10.1016/j.beth.2008.10.006
   PubMed Web of Science ®Google Scholar
• Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate difference for single-case research. Exceptional Children, 75, 135–150. doi:10.1177/001440290907500201
   Web of Science ®Google Scholar
• Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303–322. doi:10.1177/0145445511399147
   PubMed Web of Science ®Google Scholar
• Rathel, J. M., Drasgow, E., Brown, W. H., & Marshall, K. J. (2014). Increasing induction-level teachers’ positive-to-negative communication ratio and use of behavior-specific praise through e-mailed performance feedback and its effect on students’ task engagement. Journal of Positive Behavior Interventions, 16, 219–233. doi:10.1177/1098300713492856
   Web of Science ®Google Scholar
• Reinke, W. M., Lewis-Palmer, T., & Merrell, K. (2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315-332.
   PubMed Web of Science ®Google Scholar
• Reinke, W. M., Lewis-Palmer, T., & Merrell, K. (2008). The classroom check-up: A classwide teacher consultation model for increasing praise and decreasing disruptive behavior. School Psychology Review, 37, 315-332.
   PubMed Web of Science ®Google Scholar
• Sanetti, L. M. H., & Collier-Meek, M. A. (2014). Increasing the rigor of treatment integrity assessment: A comparison of direct observation and permanent product methods. Journal of Behavioral Education, 23, 60–88. doi:10.1007/s10864-013-9179-z
   Google Scholar
• Sanetti, L. M. H., & Collier-Meek, M. A. (2019). Supporting successful interventions in schools: Tools to plan, evaluate, and sustain effective implementation. New York, NY: Guilford Press.
   Google Scholar
• Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Byron, J. R., & Kratochwill, T. R. (2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229. doi:10.1016/j.jsp.2015.03.002
   PubMed Web of Science ®Google Scholar
• Sanetti, L. M. H., & DiGennaro Reed, F. M. (2012). Barriers to implementing treatment integrity procedures in school psychology research: Survey of treatment outcome researchers. Assessment for Effective Intervention, 37, 195–202. doi:10.1177/1534508411432466
   Google Scholar
• Sanetti, L. M. H., & Fallon, L. M. (2011). Treatment integrity assessment: How estimates of adherence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21, 209–232. doi:10.1080/10474412.2011.595163
   Web of Science ®Google Scholar
• Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459. Retrieved from http://www.nasponline.org/publications/spr/spr384index.aspx
   Web of Science ®Google Scholar
• Sanetti, L. M. H., Kratochwill, T. R., & Long, A. C. J. (2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47-62. doi:10.1037/spq0000007
   PubMed Web of Science ®Google Scholar
• Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of single-subject research: Methodology and validation. Remedial and Special Education, 8, 24–33. doi:10.1177/074193258700800206
   Web of Science ®Google Scholar
• Shapiro, E. S. (2013) Direct observation: Manual for the behavioral observation of students in schools (BOSS). Retrieved from https://images.pearsonclinical.com/images/Assets/BOSS/BOSS_UsersGuide.pdf
   Google Scholar
• Sheridan, S. M., & Kratochwill, T. R. (2008). Conjoint behavioral consultation: Promoting family-school connections and interventions. New York, NY: Springer.
   Google Scholar
• Sheridan, S. M., Swanger-Gagne, M., Welch, G. W., Kwon, K., & Garbacz, S. A. (2009). Fidelity measurement in consultation: Psychometric issues and preliminary examination. School Psychology Review, 38, 476–495.
   Web of Science ®Google Scholar
• Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Swanger-Gagné, M. S., Garbacz, S A.., & Sheridan, S. M. (2009). Intervention Implementation Integrity Within Conjoint Behavioral Consultation: Strategies for Working with Families. School Mental Health, 1, 131-142. DOI: 10.1007/s12310-009-9012-y 3
   Google Scholar
• Vollmer, T. R., Roane, H. S., Ringdahl, J. E., & Marcus, B. A. (1999). Evaluating treatment challenges with differential reinforcement of alternative behavior. Journal of Applied Behavior Analysis, 32, 9–23. doi:10.1901/jaba.1999.32-9
   Web of Science ®Google Scholar
• von der Embse, N. P., & Kilgus, S. P. (2018). Improving decision-making: Procedural recommendations for evidence-based assessment [Special issue]. School Psychology Review, 47(4), 329–332. doi:10.17105/SPR-2018-0059.V47-4
   Web of Science ®Google Scholar
• Wilder, D. A., Atwell, J., & Wine, B. (2006). The effects of varying levels of treatment integrity on child compliance during treatment with a three-step prompting procedure. Journal of Applied Behavior Analysis, 39(3), 369–373. doi:10.1901/jaba.2006.144-05
   PubMed Web of Science ®Google Scholar