Brief Online Training with Standardised Vignettes Reduces Inflated Supervisor Ratings of Trainee Practitioner Competencies

Abstract

Objective

Supervisor assessments of trainee competence are integral to ensuring that clinical psychology trainees reach competency benchmarks. The commonly used Clinical Psychology Practicum Competencies Rating Scale (CΨPRS) has been shown to elicit inflated ratings of competency. Hence, the aim of this study is to examine whether brief supervisor training reduces ratings by providing objective criteria with which supervisors can assess trainee competency.

Method

The ratings included were of 124 psychology trainees from nine Australian university clinical programmes. Of 170 supervisors, 32 completed the online training immediately prior to commencing the CΨPRS. Training required supervisors to rate the competency level described in five standardised vignettes (Beginner through to Competent). Vignette ratings, as determined by a panel of expert supervisors, were provided as feedback. A sixth calibration vignette was also rated (no feedback provided). Firstly, CΨPRS ratings from the trained and untrained supervisors were compared. Secondly, the difference between supervisor and expert ratings of the calibration vignettes were compared across trained and untrained groups.

Results

Trained supervisors provided lower CΨPRS ratings than untrained supervisors. In addition, trained supervisors (vs untrained supervisors) provided ratings of the calibration vignette that more accurately matched the ratings provided by the expert panel.

Conclusions

Brief online training using standardised vignettes was associated with lower CΨPRS ratings. The standardised vignettes helped calibrate supervisors’ ratings and likely attuned supervisors to the skills and competency levels that are expected at particular developmental stages. As a consequence, training appeared to reduce ratings, arguably resulting in more accurate assessments of trainee performance.

Key words:

• competency assessment
• field placement evaluations
• online training
• psychology practicum assessment
• rater biases
• supervisor evaluations

The copyright line for this article was changed on 28 July 2018 after original online publication.

The copyright line for this article was changed on 28 July 2018 after original online publication.

Acknowledgements

Financial support for this publication has been provided by the Australian Government Office for Learning and Teaching. The views in this publication do not necessarily reflect the views of the Australian Government Office for Learning and Teaching. The authors thank the members of the research team, including site coordinators and project consultants.

Notes

The copyright line for this article was changed on 28 July 2018 after original online publication.

1. The GAF is used by mental health clinicians and physicians to measure psychological, social, and occupation functioning.

Additional information

Funding

Australian Government Office for Learning and Teaching