Developing a multisource feedback tool for postgraduate medical educational supervisors

Abstract

Purpose: Supervisors play a key role in the development of postgraduate medical trainees both in the oversight of their day-to-day clinical practice but also in the support of their learning experiences. In the UK, there has been a clear distinction made between these two activities. In this article, we report on the development of a web-based multisource feedback (MSF) tool for educational supervisors in the London Deanery, an organisation responsible for 20% of the UK's doctors and dentists in training.

Methods: A narrative review of the literature generated a question framework for a series of focus groups. Data were analysed using an interpretative thematic approach and the resulting instrument piloted online. Instrument performance was analysed using a variety of tools including factor analysis, generalisability theory and analysis of performance in the first year of implementation.

Results: Two factors were initially identified. Three questions performed inadequately and were subsequently discarded. Educational supervisors scored well, generally rating themselves lower than they were by their trainees. The instrument was launched in July 2010, requiring five respondents to generate a summated report, with further validity evidence collated over the first year if implementation.

Conclusions: Arising out of a robust development process, the London Deanery MSF instrument for educational supervisors is a tool that demonstrates considerable evidence of validity and can provide supervisors with useful evidence of their effectiveness.

Notes

Notes

1. The rationale for restricting the target users of the instrument to educational supervisors relates to the fact that this job role is more easily defined than that of ‘clinical supervisor’, which is often difficult to separate out from the ‘background radiation’ of clinical supervision provided by every clinician with whom a trainee comes into contact.

2. The square root of the measurement error (when estimated for varying numbers of assessors and/or events/items) constitutes the standard error of measurement; 95% CIs around the standard error are equal to the standard error of measurement multiplied by 1.96 and are added to and subtracted from an individual's mean rating.

3. Two changes were made during the first year in response to user feedback: (i) the threshold to receive a report was lowered with a report available to supervisors on request with just three trainee respondents accompanied by a warning about reliability. (ii) Additional validation checks were put in place to ensure that nominated raters identified the type of supervisory relationship they had with their supervisor. Those receiving purely clinical supervision were excluded from the process.