Educator appraisal—occurrence and experience of hospital consultants in North-West England

Abstract

Background: It is recommended all hospital consultants with educational responsibilities undergo appraisal of their educational role by their employer: a key component of revalidation. To support this process locally, the North Western Deanery developed guidance on educational appraisal. We wished to establish the frequency and perceived value of educator appraisal.

Methods: A web-based questionnaire of 605 hospital consultants with educational responsibilities in North-Western England was conducted to collect data.

Results: Only 17.7% had been appraised for their educational responsibilities in the previous 36 months despite 92.7% undergoing appraisal of clinical work. Educational leads were statistically more likely to have had educational appraisal than Educational supervisors (ES), similarly ES were more likely to have had appraisal than Clinical supervisors (CS). Clinical specialty does not appear to be a major determining factor. The majority found the supporting documentation helpful in guiding data collection and subsequent recording of evidence. All respondents felt appraisals were conducted in a supportive manner with 76.6% stating appraisal had reassured them they were fulfilling their educational roles and helped 78.5% to identify an education personal development plan.

Conclusions: The incidence of educator appraisal remains low but the experience is considered valuable and contributes effectively to educators’ professional development.

Introduction

Postgraduate medical education in the United Kingdom has changed dramatically in recent years. Since April 2010, the General Medical Council (GMC) has been the statutory body responsible for overseeing the postgraduate training of doctors in the UK. It means that, for the first time ever, one organisation sets standards for education and practice, overseeing medical education and training, operating the specialist register of doctors and ensuring they are competent and fit to practice.

The importance and value of developing trainers is increasingly recognised in many disciplines. In recent years this has gained a particularly high profile in the higher education sector. The ‘Dearing Report’ (Dearing 1997) made 93 recommendations including the need to accredit the professional development of lecturers, with the establishment of an Institute (now the Higher Education Authority) to aid this process. In undergraduate medicine the need for staff development programmes to include promotion of teaching and learning skills is recognised (GMC 2003); in the UK, undergraduate nursing operates a formal system of mentorship where all pre-registration students must be supported and assessed by mentors centrally registered by the Nursing and Midwifery Council (NMC 2008). Within British General Practice trainers must be accredited but concerns exist that training of educational supervisors has been and remains at a considerably less developed stage in secondary care postgraduate medical education (AoME 2009).

The GMC has published standards for trainers which state ‘Organisations providing specialty including GP training must ensure that trainers have adequate support and resources to undertake their training role’. A key aspect of such support is to provide appraisal of trainers (clinical supervisors, educational supervisors and those with more specific educational leadership roles). Appraisal will be the cornerstone of revalidation which is a new measure to be introduced in 2012 by the GMC for all doctors in the UK; revalidation involves each doctor demonstrating that they remain up to date and fit to practice (GMC 2010). It has been stated that the appraisal process must become more robust to support revalidation (Agius 2011). ‘Teaching and training’ is one of the key areas within the GMC's main guidance document to doctors, thus educational responsibilities will need to form a key component of both appraisal and revalidation (GMC 2006). It is imperative that consultants and their employers engage with a process which will be so integral to revalidation. Currently hospital consultants are required to have an annual appraisal of their clinical roles conducted by their National Health Service (NHS) employer (usually known as an NHS Trust), and guidance on educational appraisal for trainers and appraisers has been issued both nationally and regionally (North Western Deanery 2012).

Given the increasing emphasis on medical educator development in the UK, we wished to ascertain within the geographical area covered by the North Western Deanery the extent to which appraisal of those hospital consultants with an educational role (including clinical and educational supervisors) involves any aspect of their educational activity, or indeed whether separate appraisal of educational roles occurs.

Table

Practice points

Appraisal of educator roles remains uncommon but is valuable when it occurs Revalidation will include assessment of ‘teaching and training’. Appraisal will be a cornerstone of this process. Documentation to support appraisal is helpful to guide collection and recording of evidence, and formation of a personal development plan Appraisers are key to the process and should themselves be ‘selected, trained and supported’

Methods

A customised web-based self-completion questionnaire was devised, using a leading specialist online software platform. Questions were identified by a team comprising medical educators with particular expertise in faculty development and education research colleagues. These were based on the requirement to identify the extent of educational appraisal activity together with the use and perceived value of deanery guidance available at that time. The questionnaire included mainly a series of closed questions and a small number of questions inviting free text comments. The items contained in the questionnaire are listed in Table 1. The questionnaire was piloted in August 2009 with a random sample of hospital consultants; comments were invited on the form itself, survey content and the usage of the survey instrument. Refinements were made as a result of the pilot, with the survey subsequently commencing in early September 2009 and running until late October 2009.

Table 1.  Proportion having educational appraisal broken down by main educational role

Main educational role	Proportion having educational appraisal (%)
Clinical supervisors	19/199 = 9.5%
Education supervisors	59/353 = 16.7%
Educational leads	29/53 = 54.7%

Notes: Comparison of proportions having educational appraisal (Fisher's exact test, two-tailed).

Clinical supervisors vs Educational supervisors p = 0.0219

Clinical supervisors vs Educational leads p = 0.0001

Educational supervisors vs Educational leads p = 0.0004

The network of Trust Medical Education Managers based in each hospital within the deanery was utilized. Each Medical Education Manager was asked to forward an electronic letter of invitation, information sheet, and link to the questionnaire to each of the hospital consultants in their Trust. We calculated that 3000 hospital consultants would be contactable in this manner. A reminder email was circulated approximately 3 weeks after the initial administration of the survey.

The quantitative data were transferred into Microsoft Excel and statistically analyzed using Microsoft Excel and Graphpad (www.graphpad.com). For the purposes of statistical analysis three groups were created—Clinical Supervisors, Educational Supervisors and Education Leads (Directors of Medical Education, Training Programme Directors, Specialty tutors etc.) Thematic analysis was performed on free-text answers to draw out the significant themes. Statistical significance was set at the standard 5% level (p < 0.05).

Results

A total of 605 online questionnaires were completed giving an estimated response rate of 20.1%. Of these respondents 41.7% described their primary educational role as educational supervisor and 28.1% as clinical supervisor. The remaining responses were from those who described their primary educational role as a more specialist position such as department education lead or specialty tutor. Responses were obtained from consultants in all the main hospital specialty groups.

498 respondents (82.3%) reported that they had never had an appraisal for their educational role. 82 respondents (13.6%) had taken part in such an appraisal within the past year and 25 (4.1%) in the past three years. See Tables 1 and 2 for further analysis of these figures, broken down by respondents’ main educational role and specialty.

Table 2.  Proportion have educational appraisal broken down by specialty

Specialty (grouped)	Proportion having educational appraisal (%)
Anaesthetics & Intensive Care	14/107 (13.1%)
Emergency Medicine	5/32 (15.6%)
Laboratory based specialties	4/29 (13.8%)
Medical specialties	43/187 (23.0%)
Paediatrics	12/66 (18.2%)
Psychiatry	1/7 (14.3%)
Radiology	2/27 (7.4%)
Surgical specialties (incl. O&G)	26/150 (17.3%)

The only specialty with a statistically significantly higher proportion of those undergoing educational appraisal relative to the remaining specialties was Medicine (p = 0.0281, Fisher's exact test, two-tailed). No specialties had a statistically significant lower proportion.

Of the 498 respondents who had not had their educational role appraised, 454 (91.2%) had been in receipt of an NHS employer appraisal, whilst the remaining 44 (8.8%) had not. Table 3 breaks this down by respondents’ main educational role.

Table 3.  Comparison of the proportion having educational and clinical appraisal broken down by main educational role

Main educational role	Proportion having Educational Appraisal within the past 36 months	Proportion having Clinical (NHS) Appraisal within the past 36 months	Statistical significance of two groups (Fisher's exact test – two tailed) NS = not significant
All respondents	107/605 = 17.7%	561/605 = 92.7%	p = 0.0003
Clinical supervisors	19/199 = 9.5%	178/199 = 89.4%	p = 0.2309 (NS)
Education supervisors	59/353 = 16.7%	331/353 = 93.8%	p = 0.0334
Educational leads	29/53 = 54.7%	52/53 = 98.1%	p = 0.4528 (NS)

Of the 107 who had their education role appraised, in 61 (53.3%) it was as part of their NHS employer appraisal with the remainder taking part separately. The appraiser was most often the Clinical Director (45, 42.1%) or the Trust Director of Medical Education (17, 15.9%), but others included the Head of Department, Director of Human Resources and the Medical Director.

Of those respondents, 36 (33.6%) and 28 (26.2%) undergoing educational appraisal stated they used deanery guidance and appraisal documents, respectively. Respondents’ perceptions of the meeting and subsequent documentation are explored in Table 4.

Table 4.  Respondents’ perceptions of the meeting and subsequent documentation

How many respondents felt the meeting …	n	%
… was an appropriate length?	104	97.2
… allowed them to have their say?	105	98.1
… was conducted professionally?	106	99.1
… was conducted in a supportive manner?	107	100.0
How many respondents felt …
… they received appropriate feedback at the meeting	68	63.5
Was the feedback in subsequent documentation …
… appropriate	68	63.5
… partly appropriate	22	20.6
… not appropriate	17	15.9

Virtually all respondents (105, 98.1%) thought the process was helpful in identifying their views and achievements, with a similar proportion stating that it was definitely (73, 68.2%) or partly (33, 30.8%) helpful in identifying appropriate development goals. 82 (76.6%) thought the appraisal process had been helpful in reassuring them that they were fulfilling their educational role, with 22.4% of the remainder finding it partially reassuring. 78 (72.9%) felt that it helped them identify a suitable development plan for education in their specialty or Trust, whilst the remaining 29 (27.1%) felt it did not. 84 (78.5%) felt that it helped them identify a Personal Development Plan (PDP) for their role as an educator, whilst the remaining 23 (21.5%) felt it did not. Overall 51.4% rated it as moderately valuable, whilst 42% of respondents rated it as valuable or highly valuable.

Table 5 details the views of the 28 respondents who had made use of the deanery educational appraisal documentation available at that time.

Table 5.  Views of the respondents who had made use of the deanery educational appraisal documentation (n = 28)

Did you find the educational appraisal documentation … .	n	%
… helpful in deciding what evidence to collect?	27	96.4
… helpful for recording the evidence used?	24	85.7
… helpful in guiding your personal development plan?	24	85.7

Just over half (57, 53.3%) of respondents received formal documentary evidence of the appraisal and its outcomes. Of the 57 respondents who did receive such documentation, 94.7% of them thought it accurately captured the content and agreed actions from the meeting.

Free text comments about the whole educator appraisal process were received from 23 respondents. These were scrutinized for recurring themes using thematic analysis (Bryman 2008) and an illustrative selection of quotes is given in Box 1.

Box 1 Thematic analysis of free comments about the educator appraisal process

The importance of the role of the appraiser

‘Excellent appraiser’

‘Would have appreciated more constructive criticism, it was a pat on the back and box ticked rather than really helping me develop.’

Perceived value of the process in those who undertook it

‘This was the first and very useful, felt somewhat more empowered after it, appreciated and encouraged to develop new ideas.’

‘I thought it helpful and a good idea.’

‘Would hopefully be easier second time around now that we have things in place to collect the relevant info.’

Concerns regarding the process

‘It is worth considering the impact of further forms to fill on top of normal appraisal, which are completed in our own time at present.’

‘Need to have concrete planning, training and time for these roles.’

Discussion

Clearly, our most striking finding is the low rate of review of educational activity in those hospital consultants who have undergone appraisal. This may be due to the traditional placement of educational activities as low priority relative to clinical activity, or possibly due to oversight of both consultants and appraisers that training should be considered an area for development and therefore formally appraised. The strongly significant difference between the proportion having had educational and clinical (NHS) appraisal may lend weight to this theory. Used positively appraisal can be one step to raising the profile of educational activity.

Practical issues may be a key factor—paperwork may be outdated and in need of review to keep up to date with new developments and guidance in appraisal i.e. educational responsibilities may not be discussed if they do not feature in the documents. The introduction of guidance documents by the stakeholder bodies such as postgraduate deaneries (North Western Deanery 2012) and the National Association of Clinical Tutors (NACT 2009) have been designed to aid both trainers and appraisers with data collection and the appraisal process itself. Although relatively few of those responding to the survey had used the suggested paperwork it appeared to have high acceptability, and subsequently documentation has been revised and improved in response to the qualitative feedback received as part of this study.

In addition to the imminent need to demonstrate evidence of ‘Teaching and Training’ for revalidation, the GMC has recently held a consultation on the Recognition and Approval of Trainers (GMC 2012). In postgraduate education this relates to named Clinical and Educational Supervisors and proposes to use seven areas defined by the Academy of Medical Educators as a structure for such recognition (AoME 2010). If individual Supervisors cannot demonstrate evidence of adequate appraisal they may face hurdles not just with revalidation but also retaining their Clinical or Educational Supervisor status.

It is perhaps not surprising that with increasing educational responsibility the proportion having educational appraisal increases, with a statistically significant difference (p = 0.02) even between clinical supervisors (9.5% appraised) and educational supervisors (16.7%). Specialty does not appear to be a major determining factor in frequency of undergoing educational appraisal, with Medicine being the only specialty to have a statistically significantly higher proportion than the remaining specialties.

In our sample, of those who had had appraisal of their educational role, just under half had this at a different time from their NHS appraisal. It is likely that these results reflect the relatively high proportion of those with additional educational roles (e.g. specialty tutor) who completed the survey. Deanery guidance is that for clinical and educational supervisors the appraisal should be a routine part of their NHS appraisal and subsequent revalidation in keeping with the GMC guidance to minimize the burden and resources required. It is recognized that individuals may already have multiple appraisals for clinical, management and external roles. For educational leaders (most of whom will have additional dedicated job plan time for the role), it is strongly recommended that an enhanced appraisal of the education role is required in order to maximize its impact, usually at a separate time and by a different appraiser to their NHS appraisal. The most common appraiser was a consultant's clinical director, in keeping with typical NHS practice, although some NHS bodies allow consultants to choose their own appraiser. National guidance is that appraisers should themselves be ‘selected, trained and supported’ (NHS Clinical Governance Support Team 2007).

Qualitative data in the form of free text comments add a further source of useful information. Analysis of free text suggestions for improving the North Western Deanery's educational appraisal documents reveal suggestions for an online portfolio to accumulate data throughout the year, views that the paperwork should be simplified, and recommendations that guidance be included for the appraiser regarding the levels of training and development expected for the role.

Comments describing both positive and negative experiences of the appraiser (see Box 1) highlight the key role of a skilled and committed appraiser to the process—the framework and paperwork will only be of maximal benefit if complemented by an enthusiastic and skilled appraiser. The training of appraisers is intended to aid this process.

Clearly, the comments highlight that this is a relatively new process which must be honed in forthcoming years. Some responses complement the statistical data, indicating that education appraisal is valued when it happens, and that it should be perceived as the start of an ongoing process rather than a single event. Clearly, any additional intervention must demonstrate value and ‘teething’ problems are to be expected. Some respondents suggested that the appraisal entails extra work; for this reason it is crucial that appraisal emphasises the process as a bridge to change in practice and not just ‘a paper exercise’. Concerns regarding the increased time taken for educational appraisal must be acknowledged; on a leadership level ensuring that the process becomes embedded into the organisational culture with additional time allocated for data collection where necessary.

The results show that the process was largely valued by those who undertook appraisal of their educational activity. Among them, 93% found the overall process at least moderately valuable and over two-thirds felt the process was definitely helpful in identifying educational goals which must be a major help in pushing postgraduate education up the agenda on both an individual and organisational level. While paperwork was not used universally, over 85% of those using it found it helpful not only as a method of recording activity but also as a guide of what data to collect; most importantly over five in six respondents believed it had helped guide their personal development plan.

Introduction of paperwork is only one step in encouraging high levels of take-up. We have subsequently delivered workshops for lead educators to support development of educational appraisal within Trusts and specialities, and include regular monitoring of uptake and feedback using Deanery annual reporting and quality management visits.

The relatively low response rate means that the results must be interpreted with some caution, although the response rate was very similar to other surveys of trainers, in particular the national trainer survey conducted over a similar period, where the response rate was near identical (PMETB, 2009). As with the national survey a large limitation in acquiring a full response rate is the reliability of the flow of information, for example incomplete email lists, or failure of an administrator within a trust to forward emails.

Conclusions

When our data were collected in 2009, the occurrence of educator appraisal in the North Western Deanery was still low. There is a dearth of comparative evidence in the public domain from other regions in the UK. Subsequent regional and national work to raise the profile of educator appraisal within the medical educator community and NHS Trusts may well have increased the occurrence, which we plan to explore in future research. There is a need to identify strategies (including with regulatory bodies and NHS employers) to counter the status quo. Results suggest that, for those consultants whose educator role has been appraised, the experience has been valued and contributed effectively to their professional development. GMC trainer standards and the imminent introduction of revalidation bring added incentive to introduce educator appraisal for all consultants. Raising the profile of educator development should assist with the necessary culture change to recognise the importance of the educator role within the medical profession and the NHS.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.