A Behaviorally-Anchored Rating System to Monitor Treatment Integrity for Community Clinicians Using the Adolescent Community Reinforcement Approach

Abstract

This study evaluated a process for training raters to reliably rate clinicians delivering the Adolescent Community Reinforcement Approach (A-CRA) in a national dissemination project. The unique A-CRA coding system uses specific behavioral anchors throughout its 73 procedure components. Five randomly selected raters each rated “passing” and “not passing” examples of the 19 A-CRA procedures. Ninety-four percent of the final intraclass correlation coefficients were at least “good” (≥.60) and 66.7% were “excellent” (≥.75), and 95% of the ratings exceeded the 60% or better agreement threshold between raters and the gold standard. Raters can be trained to provide reliable A-CRA feedback for large-scale dissemination projects.

Keywords:

• A-CRA
• coder training
• community reinforcement
• evidence-based treatment
• rater training
• treatment adherence

Acknowledgments

The authors thank the raters who participated in the study; the trainees whose clinical work was rated; and Christin Bair, Brandi Barnes, Jutta Butler, Mark D. Godley, Courtney Hupp, Stephanie Merkle, Robert J. Meyers, Randolph Muck, Laura Reichel, and Kelli Wright for their help and support of this project.

Notes

Note. “Passed” Samples = the therapist in the session had been rated previously as having passed the procedure; “Not Passed” Samples = the therapist in the session had been rated previously as not having passed the procedure. ICC = intraclass correlation coefficient.

^a ICC not computed because none of the randomly selected raters rated this procedure as having been attempted.

^b ICC could not be computed due to items having a negative average covariance.

^c ICC could not be computed due to items having zero variance.

Note. ICC = intraclass correlation coefficient.

^a Indicates the example was of a therapist who had been rated previously as having passed the procedure.

^b ICC could not be computed due to items having zero variance.

^c ICC could not be computed due to items having a negative average covariance.

^d Indicates the example was of a therapist who had been rated previously as not having passed the procedure.