Pay no attention to the man behind the curtain: The overlooked issue of rater selection in psychotherapy observer ratings

Abstract

Objective: Psychotherapy process research relies heavily upon trained raters to identify and code therapist and client behaviors. Raters are often selected out of convenience or availability with little research to inform what qualities are desirable for this role. In this study, we evaluated several rater characteristics as potential predictors of raters’ performance. Method: We provided training to 39 undergraduate students to serve as raters in a psychotherapy process study. We assessed baseline characteristics, personality characteristics, executive functioning, and interviewers’ assessment of students’ rating potential as predictors of rater agreement with gold standard ratings (i.e., consensus judgements from a panel of CBT-trained clinicians). We also assessed these variables as predictors of the risk of raters dropping out prior to completing assigned ratings. Results: Higher conscientiousness, higher neuroticism, and a lower year in college predicted greater agreement with the gold standard ratings. Lower year in college also predicted lower risk of dropout. Conclusion: These findings provide empirical support for key characteristics as predictors of rater performance. Such evidence raises the possibility of using these characteristics to select raters and thereby enhance the psychometric properties of psychotherapy ratings.

Keywords:

• rater reliability
• adherence
• cognitive behavioral therapy

Notes

1 Five of the 75 statements are used to assess validity of respondents’ answers and are not included in the two summary index scales.

2 Gold standard ratings were generated by one doctoral-level CBT expert (DRS) and two master-level CBT clinicians (IDE and BJP). The individuals on this panel had experience providing process ratings for CBT, as well as at least two years researching and providing CBT for depression.

3 Two post-baccalaureate students were recorded as 5th year students.

4 We entered both adherence measures as a repeated dependent variable as we were interested in predicted GS-deviations across both measures and did not have reason to expect differences across the two adherence scores. However, we also explored separate models for each adherence score. To do so, we used the same approach of testing a model for each of the four sets of predictors. In the models examining predictors of cognitive methods GS-deviations, only lower year in college was associated with ratings closer to the gold standard after correcting for multiple comparisons. In the models examining predictors of behavioral methods GS-deviations, female sex and having higher levels of conscientiousness, extraversion, and neuroticism were associated with ratings closer to the gold standard after correcting for multiple comparisons.