Do trainees value feedback in case-based discussion assessments?

Abstract

Background: Feedback is important in learning, including in workplace-based assessments.

Aim: To explore trainee's perceptions of the educational value of case-based discussions (CBDs) specifically focusing on feedback.

Methods: An online questionnaire and interviews obtaining detailed descriptions of paediatric trainees at UK specialist training levels 1 and 2 views and experiences were used. Qualitative data were analysed using a thematic framework analysis.

Results: Trainees viewed CBDs as educationally valuable, aiding reflective learning, improving decision making skills and effecting a change in practice. Opinions varied regarding how useful they found the feedback. Feedback was perceived as more valuable from assessors who had a positive attitude towards CBDs, understood the process and had experience in leading them. Time constraints and assessments performed in less suitable environments had a negative impact on feedback. Trainees felt the choice of case played an important role, with challenging cases resulting in more beneficial feedback.

Conclusions: CBD assessments provide a new opportunity for good quality learning and feedback, providing there is a commitment to the educational aspects of the process by both trainer and trainee. Trainers being aware of the qualities of the discussions that result in successful feedback, could significantly improve their educational value.

Introduction

Significant changes to junior doctors working patterns over recent years have lead to new challenges in providing good quality educational opportunities. There has been a reduction in working hours in the United Kingdom with the implementation of the European Working Time Directive and the New Deal document and a change in the structure of training with Modernising Medical Careers (Department of Health 1991, 2003, 2004, 2008). Reduced training time and contact with trainers has lead to a reported deterioration in the quality of training and less opportunity for reflection and feedback (Scallen 2003). The introduction of mandatory assessment tools, such as case-based discussions (CBDs), in specialist training (ST) programmes, may have a role in compensating for this effect (Carr 2006). The trainee is required to complete a minimum number of CBDs and other assessments (Mini-Clinical Evaluation Exercises and Direct Observed Procedures) in each year of training. Failure to do this, or evidence of poor performance in assessments may result in the year of training having to be repeated. Thus, although the assessments are in the main formative, they may have a summative component. The assessments are time consuming and so it is important that the educational benefits should be maximised.

The CBD assesses clinical decision making and the application of knowledge, and feedback is integral to the process. The trainee or trainer may choose the case. The value of feedback in learning has been demonstrated, including in workplace-based assessments (Hewson & Little 1998; Johnson et al. 2008) Studies describing feedback in the Mini-Clinical Evaluation Exercise and Multi-source feedback assessments have been published, however, there has been little work investigating feedback in CBDs (Hauer 2000; Kogan et al. 2002; Violato et al. 2003; Holemboe et al. 2004; Sargent et al. 2005). The aim of this study, therefore, was to explore trainee's perceptions of the educational value of CBDs specifically focusing on the feedback that they obtained as part of the process.

Table

Practice points

The supervisor and trainee must be committed to the educational aspect of the CBD process. CBDs should be carried out when there is enough time available in an appropriate environment. The case chosen should be suitable for the level of training of the trainee. Assessors should have specific training in providing feedback in CBDs.

Methods

The study was conducted from January to September 2009. All paediatric trainees at ST1 and ST2 levels in Mersey Deanery (32 trainees) were identified and contacted via the e-portfolio. Participation was voluntary and trainees were assured that all data would be anonymised and confidentiality maintained. Ethical approval was obtained from the NHS North West Strategic Health Authority R&D Committee.

Trainees completed an online questionnaire (Survey Monkey) with open and closed questions (Appendix 1). The questions were piloted with a group of specialty trainees not taking part in the study and modified after they made suggestions to improve readability and clarity. Responses were on a seven-point Likert rating scale, always/never descriptive scale and free text responses.

Trainees were then invited to participate in a recorded interview (Appendix 2), the structure of which was informed by responses from the questionnaire. Trainees who volunteered were opportunistically selected for interview. The point at which saturation was achieved determined the sample size (n = 9). Descriptive statistics were used to report responses from the Likert scales in the questionnaire. The recordings were transcribed and qualitative data from the questionnaires and interviews were analysed manually (by Fulya Mehta) using thematic framework analysis (Ritchie & Lewis 2003). Emerging themes were indexed and charted to develop a comprehensive matrix-based analysis of the comments. To ensure inter-rater reliability, a second researcher (Jeremy Brown) reviewed 30% of the transcripts to identify any discrepancies in the highlighted themes.

Results

A total of 32 trainees were invited to participate in the questionnaire and interviews; 26 (81%) completed the questionnaire having performed a median number of 4 (range 1–12) CBD assessments at ST level. The results from the specific questions in the questionnaire are presented in the table. Nine trainees were interviewed (from 12 volunteers). The main qualitative findings from the free text responses in the questionnaire and interviews are reported below under key themes identified.

Trainees experience and views of CBDs

In general trainees had a positive opinion of CBD assessments and described their experiences as constructive.

The last one I did … I learnt loads from it. I thought that whilst we were managing the baby I understood what was going on, but when I went through the CBD there was a lot more depth to it than I realised. (Trainee 9)

Trainees described CBDs as fair, providing a reasonably accurate picture of their competence, but suggested they could be subjective. Trainees placed greater value on the feedback to judge their progress, than the actual grading.

I think the impression they get of you as a doctor is a fair one … it's more about the what's said and what you learn from it than the actual marks you get (Trainee 8)

Some trainees commented on CBDs being the most useful of the compulsory assessments as a learning opportunity.

they all have their place … but the beauty of a CBD is really about the bigger picture, rather than just a snapshot of a practical procedure or examining a child it's a reflection of how you are progressing (Trainee 1)

Trainee's views on the educational value of feedback in CBDs

Opinions varied greatly regarding the usefulness of the feedback. However, overall the questionnaire and interviews results indicated that most trainees valued it.

It's usually very useful because it gives the consultant a chance to say what they think of you and your practice (Trainee 1)

The feedback was perceived as aiding reflective learning, decision making skills and affecting a change in practice.

I think that people quite often highlight other possible diagnosis that I may not have thought of but I will then think of them if I am faced with the same situation again (Trainee 2)

I think there is that chance of dialogue with somebody more senior to learn from their experiences it's really useful in that way (Trainee 1)

Other trainees described limited additional benefit from the feedback and the importance of the way it is delivered.

I think people struggle to find what was good and what was bad because you’ve gone through a lot of that discussion anyway and so the feedback isn’t a major aspect of it at all (Trainee 9)

If it's constructive criticism then it's useful, but if it's critical criticism with a degree of retribution attached to it then it's of no use at all. (Trainee 5)

Some trainees felt assessors used CBDs to give general feedback on their progress rather than specific case-related feedback, which they valued.

What I think is more useful is when you get feedback about how you’re performing as a trainee in general, which is what most supervisors are giving out when they’re giving feedback (in CBDs) (Trainee 6)

Factors influencing the quality of feedback

Four main themes emerged as influencing the quality of feedback in CBDs: the case, time, assessor and the environment.

Time was almost universally identified as a factor affecting the quality of the discussion and feedback. Trainees had difficulties completing CBDs during busy working days. When CBDs are rushed, trainees perceived the feedback to be of poor quality and limited educational value.

There's just not the opportunity to have 20 minutes of uninterrupted time on a busy unit. It certainly affects the feedback that you get … (Trainee 6)

Most trainees felt the choice of case played an important role, with challenging cases stimulating useful feedback.

I think the case is important. I chose something particularly difficult and took the opportunity to go through it because I really wanted to understand it. (Trainee 9)

Some trainees described choosing less complex cases to achieve a better assessment, but that this limited their learning from the process.

I rarely bring forward patients that I did badly with and so rarely get negative feedback. So it's useful but there's not a lot of constructive criticism to go on most of the time (Trainee 2)

All trainees identified the assessor as an important factor. Feedback was more valuable from assessors who understood the process, had experience in leading CBDs and a positive attitude towards the discussion.

If you choose someone who wants to teach and can teach, it makes a big difference. If they just fill the paperwork in then you’re not going to get anything from it (Trainee 9)

A lot of people don’t have the time or the motivation to do CBDs with you … the trainers are a bit fed up now because they have been assessing so many trainees and they think it's pointless (Trainee 4)

Several trainees raised the environment in which the discussion takes place as an issue. CBDs performed in a public place had a negative impact on feedback because of interruptions and both the assessor and trainee may feel unable to speak freely.

You have to do it wherever you find a computer … there are other trainees with you … and that's not the proper place to give feedback, especially if you want to say something negative (Trainee 4)

Specific learning points

The generation of specific learning points is part of the CBD and trainees indicated that this frequently did not occur (Table 1). Some felt it was an integral aspect that they found useful.

Some cases are … you need to read more or see more of this kind of case, or sit in with consultants during difficult discussions and those things are really important as well. So I think the learning points tick box should always be in there. (Trainee 1)

Other trainees found less additional value in formalising learning points, having already learnt from the discussion itself.

it can be very useful if it's something that reflects your needs. Unfortunately, sometimes … neither of you feel that there is anything else … to cover and you’re just putting something to fill the box. (Trainee 6)

The choice of case, assessor and time were identified as factors influencing why specific learning points were not generated more frequently.

I think it's quite hard to say in just a one off meeting exactly what you need to do and … sometimes assessors are reluctant to make learning points cause they don’t want it to seem like a negative judgement. (Trainee 7)

Table 1  Questionnaire results

Question	Always n (%)	Usually n (%)	Sometimes n (%)	Occasionally n (%)	Never n (%)
Do you feel the assessments gave an accurate picture of your competence?	2 (8)	10 (38)	10 (38)	3 (12)	1 (4)
How often did you feel specific learning points were generated?	0 (0)	10 (39)	11 (42)	5 (19)	0
How often did you agree an action plan with your supervisor?	3 (12)	12 (46)	7 (27)	4 (15)	0
How often have CBDs resulted in a change in your practice?	1 (4)	2 (8)	12 (46)	7 (27)	4 (15)
Have the action plans generated by the CBDs been revisited with your supervisor?	0 (0)	3 (12)	1 (4)	6 (24)	15 (60)
	Extremely not at all
	1n (%)	2n (%)	3n (%)	4n (%)	5n (%)	6n (%)	7n (%)
Overall, how involved did you feel in the discussion?	3 (12)	9 (34)	6 (23)	6 (23)	2 (8)	0	0
How valuable did you find the assessments from an educational point of view?	2 (8)	2 (8)	10 (38)	7 (27)	4 (15)	1 (4)	0
How useful did you find the feedback from the discussion?	1 (4)	4 (15)	9 (34)	8 (31)	3 (12)	1 (4)	0

Note: A total of 26 respondents – numbers of responses and % within parentheses.

Action plans

Overall, action plans were usually generated and valued.

it's useful in that it's written down … if your learning point is that you need to get to more clinics and in practice that's difficult … you can say this was clearly an action point and not getting to clinic has adversely affected my education (Trainee 1)

Other trainees found them less useful and rarely generated action plans.

Personally I learn things from the CBD its self, I don’t think an action plan needs to be formally documented. (Trainee 9)

Trainees rarely revisited action plans (Table 1) and there were few examples of doing so. This was described as only occurring with enthusiastic assessors who understood the process.

Yes I have done it once when the person actually doing the CBD with me understood the process and suggested a paper I should read … then at my end meeting someone could say did you do this and I had. But that's the only example I have. (Trainee 2)

Assessments tended to be performed towards the end of post, reducing the opportunity for revisiting action plans.

They don’t usually get followed up with the person doing the case with you, especially when your doing assessments towards the end of a job and you move on to somewhere else. (Trainee 3)

Discussion

In this study, we have investigated trainees perceptions of feedback during CBDs. We purposely only approached all ST 1 and 2 trainees to request participation, as more senior trainees may have been exposed to different training systems which did not include the use of workplace-based assessments thus making it inappropriate that they took part. The response rate of 81% of trainees approached makes their questionnaire responses likely to be representative of the whole group. It is possible that some trainees more senior than ST2, because of differing experiences, could have responded differently to the questionnaire. It was felt that saturation of themes was reached when the number of trainees interviewed reached nine, however, given this small number we accept that we should be cautious when generalising the results of our study as applying to other trainees.

In examining trainee's experiences of CBDs and the feedback they receive, we have gained insight in to how the educational potential of the process can be fulfilled. Trainees regard CBDs as a potentially useful learning experience through the case discussion and feedback process. CBDs appear to have a role in facilitating reflective learning, improving decision making skills and promoting positive changes in practice. Trainees valued feedback, which reflected their progress and competence. There was significant variation in trainee's attitudes towards the feedback with most finding it useful, while others perceived no additional benefit. It is possible that in the latter cases, if action plans had been followed up (which was rarely the case), increased benefit may have resulted. Trainees also described CBDs where they received little or no feedback, although there may be a disparity between the assessor and trainee's perception of what feedback is given, which has been described previously (Sender-Libermann et al. 2005; van de Ridder et al. 2008).

Several important factors emerged as influencing the quality of feedback. Feedback was perceived as more valuable from assessors who demonstrated understanding of the process, a positive attitude towards CBDs and experience in leading them. The quality of the assessor has previously been described as influencing the usefulness of workplace-based assessments (Johnson et al. 2008). Poor assessor quality may result in specific learning points and action plans not always being agreed upon as was often the case, indicated by the questionnaire results. The delivery of effective feedback is an essential step to improve performance. It is important that trainers are familiar with the role of feedback in learning and appropriately trained in delivering it. Time constraints and assessments performed in less suitable environments had a negative impact on feedback highlighting the value of undertaking CBDs in a planned manner and how important it is to make optimal use of contact with trainers when the opportunity does arise (Scallen 2003). The choice of case played an important role, with challenging cases resulting in more beneficial feedback. Trainees also discussed the dichotomy of using CBDs as a learning tool and an assessment, by choosing cases where they have performed well rather than those of most educational value.

Some trainees felt that assessors used CBDs as an opportunity to give general feedback on their professional development rather than focused areas for improvement. Although feedback is generally described as being more useful when it is specific, trainees valued this general feedback (van de Ridder 2008). This perhaps reflects the important all forms of feedback particularly as opportunities for regular feedback in the clinical setting have diminished with reduced working hours and contact with trainers (Scallen 2003).

Formative assessment and feedback has been shown to be a powerful tool in changing behaviour and improving practice in trainees (Norcini & Burch 2007). This is partly facilitated by the generation of specific learning points and action plans, which are a component of feedback in CBDs. Trainees varied in their attitudes and experience of this part of the assessment. Some perceived it as an essential aspect, describing experiences of generating learning points and action plans reflecting their learning needs and focusing clinical experience. The quality of the assessor and time factors were again highlighted as influencing this. Less valuable experiences related to developing generic learning points and action plans, and if they felt the assessor was reluctant to give useful advice to avoid a negative judgement. Action plans were rarely revisited formally resulting from assessments performed at the end of posts with lack of continuity of assessors, highlighting the need for timely assessments.

In summary, CBDs present an opportunity for good quality learning and feedback, providing there is a commitment to the process by both supervisor and trainee. The educational potential of CBDs should be emphasised to encourage trainees to bring forward suitable cases and promote a positive attitude in both trainees and assessors. An awareness of the key elements that facilitate constructive and valuable CBD assessments, and that generate successful discussions and feedback for trainees, will maximise their educational value. Attention should be paid to the practical issue of time constraints. Planning meetings in advance at regular intervals during each post may improve the educational outcome and opportunity for reflective learning, with time to put action plans into practice. It is also essential that a suitable environment is available for the discussion, to facilitate open and honest feedback. Trainees perceived the attitude and skill of the assessor played a major role in the educational usefulness of the CBD and feedback, and assessors might benefit from specific training on this. Future work examining assessor's experiences and views on CBDs and their perceived barriers to giving effective feedback will be useful in improving the educational value of the process.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Appendix 1

Questionnaire

The following questions all refer to case based discussions

Appendix 2

Interview schedule

Table

	Questions
1.	In general the results from the questionnaire have been positive towards the usefulness of CBDs.
(a)	How do feel CBDs compare with other assessments in terms of their educational value?
(b)	And as an assessment tool?
2.	How useful do you find the feedback from CBDs?
(a)	What factors influence the feedback?
(b)	How can it be improved?
3.	The questionnaire showed that specific learning points were either sometimes or usually demonstrated – what is your experience?
(a)	Is it useful to generate learning points?
(b)	Why do you think they are not generated more frequently?
4.	The questionnaire also showed that action plans were generated sometimes or usually.
(a)	Is it useful to generate an action plan?
(b)	Why do you think they are not generated more frequently?
(c)	They seem to be revisited very rarely. Why do you think this is?