DOPS assessment: A study to evaluate the experience and opinions of trainees and assessors

Abstract

Background: Workplace based assessments (WBAs) have been part of UK training for the last 3 years. Carrying out procedures efficiently and safely is of paramount importance in anaesthesia.

Aims: To explore opinions and experiences of Direct Observation of Procedural Skills (DOPS) assessments in a regional anaesthetic training programme.

Methods: 19 and 20-item questionnaires were distributed to trainees and consultants respectively.

Results: Questionnaire response rate was 76% (90/119) for trainees and 65% (129/199) for consultants. 43% of consultants and 33% of trainees were not trained in DOPS use. Assessments were usually not planned. 50% were ad hoc and the remainder mainly retrospective. Time spent on assessment was short with DOPS and feedback achieved in ≤15 minutes in the majority of cases with lack of suggestions for further improvement. Both trainees and consultants felt that DOPS was not a helpful learning tool (p = 0.001) or a reflection of trainee competency.

Conclusions: DOPS assessments are currently not valued as an educational tool. Training is essential in use of this WBA tool which needs to be planned and sufficient time allocated so as to address current negative attitudes.

Introduction

The medical profession has been under increased government and public scrutiny in the measures it takes to ensure that doctors completing their training are competent (Krnietowicz 2008). For specialties such as anaesthesia the ability to perform procedures efficiently and safely is of paramount importance. In addition, the reduction in working hours means there is even less time to master skills due to fewer learning opportunities both inside and outside normal working hours (Cooper & McClure 2008). As a result there has been a move away in recent years from ‘simply completing accredited posts for a set amount of time’ to a more ‘competency based curricula’ (Royal College of Anaesthetists 2007). Direct observation of procedural skills assessment (DOPS) was developed by the Royal College of Physicians (Norcini & McKinkley 2007), and now forms part of workplace based assessments (WBAs) for doctors in the foundation year (FY) and those in specialist training including anaesthesia (Royal College of Anaesthetists 2007).

A DOPS was specifically designed to assess procedural skills involving real patients in a single encounter. This is an important facet of DOPS which distinguishes it from other forms of assessment such as a supervisor's evaluation which relies on observation over a period of time. During the anaesthetic training programme, trainees are expected to undertake a high number of DOPS covering a range of procedures in anaesthesia and intensive care and need a minimum number of 75 DOPS in their 7 years of training (Birmingham School of Anaesthesia 2010a–c).

Table

Practice points

It is important that training is given in the use of DOPS to both trainers and trainees. DOPS should be viewed as a useful educational/training opportunity for trainees to improve performance in a skill. Collegial working between trainers and trainees is needed to ensure that DOPS are done in a timely manner. Timely, high quality feedback to trainees is an essential part of the assessment.

This study was designed to assess the opinions and experiences of the use of DOPS of both trainees and consultants across the Birmingham School of Anaesthesia, which consists of six hospitals, two tertiary referral and four district general hospitals.

Method

Each anaesthetic department within the Birmingham school of Anaesthesia was contacted via the respective College tutor and asked to distribute the appropriate questionnaires to all trainees and consultant staff. The questionnaire had a number of statements about which the respondents was asked to rate their response on a six-point Likert scale (1 = strongly disagree to 6 = strongly agree). The trainees’ questionnaire had 19 questions, while the one for consultants had 20 questions. Both questionnaires had a free text box for further comments. Data was entered into an Excel spreadsheet and analysed using SPSS 15.0. Free text comments were analysed using NVivo 8.

Results

A total of 76% (90/119) of trainees responded to the questionnaire. There were approximately equal numbers of male and female trainees (43 and 47 respectively). Most commonly trainees were, aged 31–35 years (55%, 49/90), worked full time (89%, 80/90) and were graduates from the UK (82%, 74/90) (Table 1). Consultant response to the questionnaire was 65% (129/199). Most consultants were female (71%, 91/129), had graduated from the UK (82%, 106/129), worked full time (95%, 122/129) and were aged between 31 and 40 years (41%, 53/129) (Table 1).

Table 1  Demographics

		Trainee n/90 (%)	Consultant n/129 (%)
Gender
Female		47 (52)	91 (71)
Male		43 (48)	38 (29)
Age
Trainee	Consultant
23–25 years	31–40 years	3 (3)	53 (41)
26–30 years	41–50 years	31 (34)	51 (39)
31–35 years	51–60 years	49 (55)	19 (15)
36–40 years	>60 years	7 (8)	6 (5)
Working pattern
Full time		80 (89)	122 (95)
Part time		9 (10)	7 (5)
Missing response		1 (1)
Place of graduation
UK		74 (82)	106 (82)
European Union		1 (1)	5 (4)
Outside the European Union		15 (17)	18 (14)
Grade of training
ST1		12 (13)
ST2		16 (18)
ST3		15 (17)
ST4		11 (12)
ST5		10 (11)
ST6		7 (8)
SpR 3		1 (1)
SpR 4		15 (17)
SpR 5		2 (2)
LAS		1 (1)

Reliability of the Likert questions completed by trainees and consultants was similar with Cronbach's alphas of 0.665 and 0.668, respectively. Both trainees and consultants were aware of how DOPS assessments should be conducted with mean Likert scores of 3.8 and 3.7 respectively. Whilst a third of trainees (30/90) had done no training in DOPS assessment, this was higher for consultants at 43% (56/129). The commonest training for consultants was a deanery course and training by the Royal College and for trainees Royal College or Trust was equally popular. E-learning packages were used by 3 trainees and 10 consultants.

All trainees had undertaken DOPS assessments (Table 2). Most commonly, consultants had done 1–5 DOPS assessments (37%, 48/129). DOPS were not felt to be reflective of a trainees’ capabilities either by trainees or consultants with mean Likert scores of 3.07 and 3.43, respectively. Most commonly trainees (50%, 45/90) and consultants (55%, 71/129) reported that assessments were being done ad hoc, i.e. on the job. Trainees reported that they found it easy to find an assessor and consultants agreed with this statement although less positively (Table 3). Both trainees and consultants reported that assessments were frequently done by consultants (mean score 5.33 and 5.00 respectively) as opposed to nursing staff (1.03 and 1.05 by trainees and consultants, respectively).

Table 2  DOPS – how assessments are performed

	Trainee n/90 (%)	Consultant n/129 (%)
Number of DOPS done
None	0 (0)	1 (1)
1–5	17 (19)	48 (37)
6–10	18 (20)	15 (12)
11–15	18 (20)	28 (22)
16–20	12 (13)	16 (12)
More than 20	25 (28)	21 (16)
How DOPS are planned
Pre planned	5 (6)	6 (5)
Retrospective	40 (44)	44 (34)
Ad hoc/on the job	45 (50)	71 (55)
Other	0 (0)	2 (2)
How soon feedback given after assessment
<5 min	61 (68)	73 (57)
6–10 min	20 (22)	34 (26)
11–20 min	3 (3)	7 (5)
21–30 min	1 (1)	6 (5)
31–60 min	3 (3)	4 (3)
1–2 h	2 (2)	4 (3)
Missing response		1 (1)
Time to do assessment and feedback
<5 min	34 (38)	35 (27)
6–10 min	23 (26)	36 (28)
11–15 min	20 (22)	19 (15)
16–20 min	10 (11)	18 (14)
21–25 min	1 (1)	11 (8)
26–30 min	1 (1)	6 (5)
>30mins	1 (1)	4 (3)
Paperwork/online form completed following assessment
Immediately	49 (55)	61 (47)
Within 2 h	15 (17)	30 (23)
Same day	12 (13)	13 (10)
Within 1 week	4 (4)	14 (11)
Within 1 month	3 (3)	5 (4)
Over 1 month	7 (8)	6 (5)

Table 3  Value of DOPS by trainees and consultants

	Mean score: Trainee*	95% Confidence interval for mean trainee score:	Mean score: Consultant*	95% Confidence interval for mean consultant score
Aware how DOPS are conducted	3.80	3.56–4.03	3.72	3.49–3.94
DOPS assessments are true reflection of capabilities	3.07	2.8–3.35	3.43	3.23–3.63
Easy to find assessor for DOPS assessment	4.15	3.86–4.44	3.80	3.58–4.02
Who does your DOPS	5.33	5.14–5.52	5.00	4.84–5.15
Consultant
SpR/ST6-8	2.75	2.39–3.11	2.65	2.37–2.94
ST4/5	1.97	1.66–2.28	1.81	1.59–2.03
ST2/3	1.48	1.26–1.70	1.39	1.23–1.54
Staff Grade	1.49	1.27–1.71	1.20	1.08–1.31
Sister	1.05	1.00–1.10	1.05	0.99–1.10
Nursing Staff	1.03	0.99–1.07	1.05	0.99–1.10
Box for future improvement completed	2.83	2.55–3.11	3.61	3.32–3.89
DOPS are useful for training	2.66	2.39–2.93	3.25	3.04–3.46
Educational supervisor reviews DOPS at appraisal meeting	4.16	3.87–4.45	3.26	2.85–3.98

Note: *Mean Likert scores 1 = disagree 6 = agree.

The majority of trainees (68%, 61/90) reported that they received feedback within 5 min of the assessment being performed. The time taken to complete both assessment and feedback was reported as 15 min or less in the majority of cases (77/90 (86%) trainees, 90/129 (70%) consultants). Paperwork/online assessment forms were completed promptly in the majority of cases (Table 2). One-fifth of consultants (25/129) reported they had refused to do a DOPS assessment with trainees reporting this less commonly (13%, 12/90). The most common reason stated by both groups was being too busy.

The box for future improvements following the DOPS assessment was often not completed (mean Likert scores of 2.83 by trainees and 3.61 by consultants p < 0.001). Both trainees and consultants did not find DOPS assessment a helpful tool for training (Table 3) (p = 0.001). DOPS assessments were reviewed by the trainees’ educational supervisor at the appraisal meetings with higher mean scores reported by trainees (4.16) in comparison to consultants (3.26). This was significant with a p value of <0.001.

The themes of the free text comments for both consultants and trainees can be seen in Table 4. The majority of comments were negative about the DOPS with only 12 being of a positive nature.

Table 4  The five themes generated by thematic analysis of free text comments and verbatim quotes

1. Useful method of assessment (12 comments)

• ‘Useful, should be encouraged. I find they help with learning and focusing’ (T)

• ‘It's a good method of assessment and a proof of your competency’ (T)

• ‘Very useful if actually used as a learning aid, and not a tick box exercise. But not everyone does this!’ (T)

• ‘Trainees must have paperwork available at all times!’(C)

2. Not helpful for training (14 comments)

• ‘Useless, waste of time, paper and energy for everybody involved’ (C)

• ‘Complete and utter waste of time. Completely meaningless’ (T)

• ‘I find them a ridiculous meaningless bureaucratic waste of time. I have been told candidates are rewarded for the number of DOPS etc completed rather than what they contain. If so failing doctors with dozens of poor performance DOPS will succeed at the expense of quality candidates completing a smaller number of assessments. Trainees are often embarrassed to ask for completion as they feel the same’(C)

• ‘In my opinion DOPS are a waste of time and effort. I would prefer a practical skills logbook containing a handbook of ‘top tips’ for safe performance of the skills and a record of procedures done +/− trainer's signature’ (C)

3. DOPS are a tick box exercise (11 comments)

• ‘Not useful at all. Paper exercise, burden for trainees and trainers with no benefit at all’(C)

• ‘Like most of the ever increasing paperwork, it is of limited use, the educational system is ‘lowering itself’ by providing such items’

• ‘Paper, tick box exercise. Trainees should give consultants warning and make sure that they can be truly assessed on the list they you want you to watch them on’ (C)

4. Not a proof of competency (17 comments)

• ‘Being asked to do these retrospectively is pointless in my view’(C)

• ‘Completed retrospectively hence have no educational value at the time. A poor trainee can look good on paper and vice versa’ (C)

• ‘Not a bad test but poorly used. Trainees always try to cram them in before appraisals – hence retrospective requests’ (C)

• ‘Poor assessment tool….Having a DOPS form completed does not indicate that a trainee is adequately trained in that skill’ (T)

• ‘DOPS assessment is a snapshot. It may reflect a very good day or a very bad day or your consistent performance’ (C)

5. Need for training in DOPs (7 comments)

• ‘Trainers need to be trained’ (T)

• ‘I would appreciate knowing more about what is expected. Despite the guidance I sometimes feel I’m too critical and not sure about measures of competence’ (C)

• ‘I find the whole thing incredibly frustrating. At my assessments (including ARCP) the focus is on number of DOPS rather than the quality or perceived progression. Few of the consultants take them seriously. A huge amount of time is spent on this process that would be spent on other more productive processes i.e. library study, 1 to 1 teaching, seeing patients’ (T)

Note: C = comment made by Consultant T = comment made by a trainees.

Discussion

The most significant finding of the study is that trainees and consultants felt the DOPS assessment was not a useful training tool within anaesthesia. DOPS assessments were viewed as a tick box exercise and not an educational opportunity or a true reflection of how well the trainee can perform the skill.

In our Deanery there has been an emphasis on Training the Trainers courses as the GMC requires all educational supervisors to be trained. It is therefore disappointing that 43% (56/129) of consultants had received no training in use of DOPS and that this was only marginally better for trainees. It will be important in future that the Birmingham School of Anaesthesia ensures that all trainees and Consultants using DOPS assessments receive training.

DOPS assessments are designed to be easily integrated into trainees and assessors normal routine and therefore considered highly feasible (Morris et al. 2006). For certain procedures, trainees have found opportunities to perform DOPS assessments only occurred outside normal working hours when assessors were not available (Morris et al. 2006). The ability of being able to decide when and who assesses them has been much criticised (Davies et al. 2005). However finding an assessor was not a reported difficulty in our study.

Most DOPS were completed by consultants, which is likely to be a reflection of the fact that anaesthetic trainees are well supported, with the majority of junior trainees having consultant supervision in theatre or within the theatre complex. Easy access to an assessor was found in a study of FY doctors who were able to achieve the six compulsory DOPS during their foundation year (Davies et al. 2009). The study finding that DOPS assessments were not completed by nursing staff was to be expected as many of the procedures performed in anaesthesia are usually only undertaken expertly by consultants.

The time taken to complete DOPS assessments for anaesthetic trainees is similar to findings for FY doctors (Davies et al. 2009). The time taken for a DOPS assessment for FY doctors was found to vary according to the procedure. Our study did not look at this variable and it would be interesting to repeat the study to see if consultant and trainee opinion of the merit of DOPS varied by complexity of procedure being performed.

There is little evidence in the literature about the educational impact of DOPS assessments. Supporters argue that the educational value lies in the process of immediate feedback after the assessment takes place including highlighting trainees’ strengths and weaknesses and formulation of an action plan to meet any learning needs (Boursicort et al. 2011). Feedback following workplace based assessments was found to be useful by a cohort of core medical trainees following workplace based assessments (Johnson et al. 2008). The Anaesthetic DOPS specifically requires the assessor to feedback to the trainee areas of good and bad practice and also identify a focus for future learning. This is an extremely important element of the assessment (Boursicort et al. 2011). It is therefore surprising that in around a third of cases that procedure and feedback was completed within 5 min. This either implies that procedures assessed were of low complexity and done perfectly or that not enough time was spent on feedback for trainees. In addition, a high proportion of DOPS were done retrospectively and there is evidence from other forms of assessment that reliability is poor when used retrospectively (Thompson et al. 1990).

Whilst both trainees and consultants may share negative attitudes towards DOPS there is a continued need for some sort of procedural assessment. Supervisor evaluations have been shown to be unreliable (Turnbull et al. 1998), while logbooks have two major limitations. The first is to know how many procedures need to be done before a trainee is deemed competent. This number will vary between trainees and also with the type of procedure being performed (De Oliveria Filho 2002). The second major limitation is that the logbook documents a number of procedures but there is no guarantee that procedures would have been performed competently.

Whilst in areas of feasibility DOPS assessment scores highly, in areas of face validity and educational impact it performs less well. Further research is necessary to investigate the use of DOPS in anaesthesia specialist training and improvements are needed to ensure that it is of educational benefit. This includes viewing it as a formative rather than summative tool and training for all those who are participating in DOPS assessments. There needs to be a greater emphasis on how DOPS are conducted especially planning and time allocated so as to address current negative attitudes and prevent continued detrimental effect on the use of DOPS as a workplace based assessment tool.

Competing interest: The authors declare no external funding or competing interests.