Generalizability of Multiple Measures of Treatment Integrity: Comparisons Among Direct Observation, Permanent Products, and Self-Report

Abstract.

The concept of treatment integrity is an essential component to data-based decision making within a response-to-intervention model. Although treatment integrity is a topic receiving increased attention in the school-based intervention literature, relatively few studies have been conducted regarding the technical adequacy of treatment integrity assessment methods. In light of recent research utilizing generalizability theory (G theory) to assess the dependability of behavioral measurement in schools, the current study used G theory to examine the dependability of direct observation, permanent products, and self-report as measures of treatment integrity when six teachers implemented the Good Behavior Game across three study sites. Results indicated that direct observation yielded the most reliable treatment integrity data, followed by permanent products and self-report. Specifically, when assessment of treatment integrity is conducted twice per week, direct observation should provide a dependable estimate (φ = .821) after only four assessments. Implications, limitations, and future directions are discussed.

Notes

Additional information

Notes on contributors

Frank M. Gresham

Frank M. Gresham, PhD, is a professor in the school psychology program in the Department of Psychology at Louisiana State University. His research interests include data-based decision making within a problem-solving model of consultation, identification and intervention for students with emotional and behavioral disorders, social skills assessment and training, and implementation of response-to-intervention models in schools.

Evan H. Dart

Evan H. Dart, PhD, is an assistant professor in the school psychology program in the Department of Psychology at the University of Southern Mississippi. His research interests include peer-mediated behavioral interventions, the identification of school-based consultation strategies to promote teacher treatment adherence, and the integration of technology into evidence-based intervention strategies.

Tai A. Collins

Tai A. Collins, PhD, is an assistant professor in the school psychology program at the University of Cincinnati. He earned his doctorate in school psychology from Louisiana State University. His research interests include the implementation of peer management and group contingency interventions in schools and developing strategies to reduce the overrepresentation of African American students in special education and their underrepresentation in gifted education programs.