Tracking Evidence Based Practice with Youth: Validity of the MATCH and Standard Manual Consultation Records

Abstract

This study sought to evaluate the agreement between therapist report and coder observation of therapy practices. The study sampled session data from a community-based, randomized trial of treatment for youth ages 7 to 13. We used therapist report of session content and coverage gathered using formal Consultation Records and developed complimentary records for coders to use when watching or listening to therapy tape. We established initial reliability between coders and then conducted a random, stratified, and comprehensive sample of sessions across youth (N = 121), therapists (N = 57), conditions (MATCH and Standard Manuals), and study sites (Honolulu and Boston) to code and compare with therapist record reports. Intraclass correlation coefficients (ICCs) representing coder versus therapist agreement on manual content delivered ranged from .42 to 1.0 across conditions and problem areas. Analyses revealed marked variability in agreement regarding whether behavioral rehearsals took place (ICCs from −.01 to 1.0) but strong agreement on client comprehension of therapy content and homework assignments. Overall, the findings indicate that therapists can be accurate reporters of the therapeutic practices they deliver, although they may need more support in reporting subtle but valuable aspects of implementation such as types of behavioral rehearsals. Developing means to support accurate reporting is important to developing future clinical feedback methodology applicable to the implementation of evidence-based treatments in the real world.

Acknowledgments

The Research Network on Youth Mental Health: Charles Glisson, Children's Mental Health Services Research Center, University of Tennessee at Knoxville; Evelyn Polk Green, Attention Deficit Disorder Association, Wilmington, Delaware; Kimberly Eaton Hoagwood, Department of Mental Health Services and Policy Research, New York State Psychiatric Institute, Columbia University, New York; Kelly Kelleher, Research Institute of Nationwide Children's Hospital, and Ohio State University, Columbus; John Landsverk, Child and Adolescent Services Research Center, University of California, San Diego; and Stephen Mayberg, California Department of Mental Health, Sacramento.

Financial Disclosure: Modular Approach to Treatment of Children with Anxiety, Depression, or Conduct Problems was the precursor to a manual for which Drs Chorpita, Daleiden, and Weisz may receive income. Funding/Support: The study was supported by the MacArthur Foundation, the Norlien Foundation, and the National Institute of Mental Health. Role of the Sponsors: The funders/sponsors did not participate in the design and conduct of the study; in the collection, analysis, and interpretation of the data; or in the preparation, review, or approval of the manuscript.

Notes

Note: MATCH = Modular Approach to Therapy for Children; HW = Homework.

Note: Items for which intraclass correlation coefficients were not provided come from sessions wherein those types of content were not included. For example, there was neither opportunity for “rehearsal” nor any homework to assign in the Psychoeducation – P module.

^aKeep Thinking, Three-Step Plan, and Best Fit ACT & THINK were all prescribed by the manual to occur in a single session, though were distinct interventions. We examined their level of coverage, rehearsal, and receipt separately, but because final rehearsal and homework for this session represented all three interventions, we coded them together.

Note: Items for which intraclass correlation coefficients were not provided come from sessions wherein those types of content were not included. For example, there was neither opportunity for “rehearsal” nor any homework to assign in the Psychoeducation – P module.

Note: Items for which intraclass correlation coefficients were not provided come from sessions wherein those types of content were not included. For example, there was neither opportunity for “rehearsal” nor any homework to assign in the Psychoeducation – P module.

^aAttending to Compliance and Independent Play were prescribed by the manual to occur in a single session, though were distinct interventions. We examined their level of coverage, rehearsal, and receipt separately, but because final rehearsal and homework for this session represented all three interventions, we coded them together. The same was done for Effective Instructions and Compliance Training as well as the pair of Home Point System and Response Cost.