A qualitative study on trainees’ and supervisors’ perceptions of assessment for learning in postgraduate medical education

Abstract

Introduction: Recent changes in postgraduate medical training curricula usually encompass a shift towards more formative assessment, or assessment for learning. However, though theoretically well suited to postgraduate training, evidence is emerging that engaging in formative assessment in daily clinical practice is complex.

Aim: We aimed to explore trainees’ and supervisors’ perceptions of what factors determine active engagement in formative assessment.

Methods: Focus group study with postgraduate trainees and supervisors in obstetrics and gynaecology.

Results: Three higher order themes emerged: individual perspectives on feedback, supportiveness of the learning environment and the credibility of feedback and/or feedback giver.

Conclusion: Engaging in formative assessment with a genuine impact on learning is complex and quite a challenge to both trainees and supervisors. Individual perspectives on feedback, a supportive learning environment and credibility of feedback are all important in this process. Every one of these should be taken into account when the utility of formative assessment in postgraduate medical training is evaluated.

Introduction

One of the most salient factors of recent changes in postgraduate assessment is probably the shift of emphasis from summative assessment in the form of certifying exams at the end of training towards formative assessment, aiming at steering and fostering trainee learning over the training period (NHS 2009; RCPSC 2009; ACGME 2010; Schuwirth & Van der Vleuten 2011). Driving forces behind this shift are the fragmentation of postgraduate training that it is the result of reduced working hours, increase part-time staff and subspecialisation (Kennedy et al. 2005; Watling et al. 2010) together with increased pressures for certification and professional regulation (Sutherland & Leatherman 2006; Irvine 2007). These changes have led to an appeal for more efficient postgraduate training and for transparent, credible assessment. Consequentially, the role of purely summative assessment, or assessment of learning, at the end of the training period is waning, and formative assessment, or assessment directed at steering and fostering learning of the trainee, is gaining ground, resulting in formative assessment being currently implemented in many postgraduate training curricula worldwide (Shepard 2000; Boud & Falchikov 2006; Irvine 2007).

Table

Practice points

Engaging trainees and supervisors in formative assessment is a challenge. Ownership and achievement goal orientation determine active engagement in assessment for learning. Scheduled assessment moments and clear standards, procedures and consequences facilitate assessment for learning. Long-term commitment between trainees and supervisors is a pre-requisite for a genuine impact of assessment for learning.

Though fairly new to medical training (Kogan et al. 2009), a growing body of evidence on the validity and the reliability of formative assessment instruments is emerging (Durning et al. 2002; Holmboe et al. 2003; Wilkinson et al. 2008; LeBlanc et al. 2009). However, whereas in summative assessment validity and reliability are seen as dominant determinants of utility, in formative assessment, utility, defined as learning that results from the assessment process, is much more dependent on how stakeholders (trainees and clinical supervisors) employ the instrument in practice (Swanson et al. 1995; van der Vleuten 1996; van der Vleuten & Schuwirth 2005). So far, few to none studies have addressed the issue of the effect of formative assessment on doctors learning and performance (Miller & Archer 2010; Satpathy & Kneebone 2011) Moreover, it is becoming increasingly clear that, even though formative assessment is theoretically well suited to postgraduate medical training, engaging both trainees and supervisors in meaningful formative assessment is quite complex (Dijksterhuis et al. 2011)

We set out to qualitatively explore trainees’ and supervisors’ perceptions on what factors determine active engagement in formative assessment. Both trainees’ and supervisors’ views were sought because engaging in meaningful formative assessment requires efforts from both feedback giver and feedback recipient (Cross et al. 2006).

Methods

To answer our research question we organised focus groups with trainees and focus groups with supervisors. A focus group approach was employed, rather than in-depth interviews or questionnaires, as we postulated that interaction between individuals would provide more information, make interconnections visible and perhaps even trigger the formulation of new ideas or theories on the subject.

Setting

This study was conducted in the Netherlands, where postgraduate training comprises a full-time training programme lasting six years. Training consists of clinical rotations in university and associated teaching hospitals, covering both in- and out-patients services supervised by faculty. This study was conducted at the start of the implementation of a major reorganisation of postgraduate training.

Ethical considerations

Although this study was exempt from ethical approval according to Dutch law, considerable effort was taken to protect the interest of participants: participants were informed: about the voluntary nature of participating in the study; about the aim of the study; that data would be tape recorded and that data would be analysed anonymously. Furthermore, participants were informed how to contact the researcher in the case of queries, concerns or when they wanted to withdraw from the study.

Participants

For this study we invited postgraduate Obstetrics and Gynaecology trainees and their clinical supervisors. To increase an optimally safe environment, focus group meetings with trainees and supervisors were held separately. Seven focus group sessions were held, three with supervisors and four with trainees, denoting four randomly chosen postgraduate training programmes. Both university and associated teaching hospitals were represented in the focus groups. Moderators had a background in obstetrics and gynaecology but neither moderators nor researchers were involved in the actual assessment or training of trainees participating in the focus groups, nor were they close colleagues. The group sizes of the focus groups with supervisors ranged from five to nine participants, 40% being female. For trainees, group sizes varied between five and eight participants, 70% being female. The disparity in female participants is a realistic representation as more female doctors enter postgraduate training, whereas the incumbent generation of medical specialists is still male dominated.

Procedure

At the start of each focus group session, all participants were once more informed on the purpose of the study and were guaranteed full confidentiality. Subsequently, informed consent to tape record the discussion was obtained from all participants. A moderator (PW, DB or FS) initiated the discussion using a pre-defined list of nine questions for guidance (Table 1). The first questions were meant to elucidate the current assessment structure, followed by questions on expectations and needs regarding assessment for learning. The moderators were instructed to elicit views from all participants.

Table 1.  Questions – focus group trainees/supervisors

	Questions – focus group trainees
	Questions – focus group supervisors
1	How is your progress currently assessed?
	How do you and your staff currently assess the progress of a trainee?
2	What do you think of the current assessment structure?
	What do you think of the current assessment structure?
3	How would you like to be assessed?
	How would you like to assess trainees?
4	How do you get feedback?
	What makes that you start giving feedback?
5	What is valuable feedback to you?
	What is valuable feedback to you?
6	What makes that you start to study?
	In your opinion, what makes a trainee start to study?
7	What stimulates you to excel?
	How do you stimulate a trainee to excel?
8	If all preconditions were optimal, what kind of assessment structure would you introduce?
9	Has anything been left unsaid that should have been mentioned?

Data analysis

Focus group recordings were transcribed verbatim and then coded using the qualitative data-analysis software Kwalitan, version 5.0. Every remark made in view of the research question was labelled and coded by MD and cross-checked by FS. Next, data were analysed by MD, FS and LS as described by Miles and Huberman, first by data reduction, than by data display, followed by conclusion drawing and verification (Miles & Huberman 1994).

Results

After extensive discussion of the codes, three higher order themes emerged explaining the level of active engagement in formative assessment by trainees and/or supervisors: individual perspectives on feedback, supportiveness of the learning environment and the credibility of feedback and/or feedback giver. We will present the results, interlaced with distinctive quotes, organised according to these themes. Quotes have been coded as follows: T denotes trainee, S denotes supervisor, 1–6 refer to training year, and A, B, C and D refer to different training programmes.

Individual perspectives on feedback

During the focus group discussions it became clear that the individual perspectives on feedback were largely determined by ownership and achievement orientation.

Ownership

Ownership can best be described as the belief that making the most of one's training period is a personal responsibility.

I strongly believe that it is my training, and thus my responsibility to ask for feedback. (T8-C)

Typically, feelings of ownership grew with years of experience; trainees in the first stages of specialist training were initially more focused on learning how to cope with the new working environment and were less actively involved in their own training pathway.

In the beginning you tend to have a more consuming attitude … . (T2-B)

However, once trainees progressed, their competence grew and transition to progressive independence was set in motion. At the same time, growing awareness of their personal responsibility to make the most of training, resulted in more self-reflection and active search for learning opportunities and feedback.

Just looking at myself, I do feel that my ability to self-reflect has grown during my training. It is only since last year, my 4th training year, that I’m much more aware that it is important to select appropriate learning goals and that I’m actively engaged in getting there. (T5-B)

It was also apparent that not actively engaging in seeking and obtaining feedback comes at a price, as it increases the probability of being confronted with unexpected feedback, usually prompted by negative incidents.

Unsolicited feedback is nearly always negative … . (T2-D)

When supervisors were asked what makes them give feedback they mostly referred to specific training situations like near incidents or when supervising practical skills. Little was revealed on their individual motivation to engage in feedback for learning. They did, however, comment on how important it is to them that trainees show ownership right from the start of training.

You would expect them to study spontaneously, just being involved in patient care should provide enough incentives to start looking things up. (S1-A)

Goal orientation

Both trainees and supervisors expressed varied assessment and feedback preferences. These seemed to be, at least partially, determined by their achievement goal orientation (performance-orientated or mastery-orientated). Performance-orientated trainees and supervisors indicated to prefer summative assessment. They see benefits from high-stakes summative assessment in which competence is assessed against a pre-defined standard, and in which failing has clear consequences.

I know it may sound a bit like going back to school, but I would like the introduction of good old knowledge exams: clear study materials, clear pass/fail standards and clear consequences. It helps me to start studying and in this way I know once I have mastered a subject. (T3 A)

I would like the introduction of a more rigorous assessment system with clear consequences. In my experience trainees study most when they get targeted assignments followed by assessment. They like it when they notice that they have actually mastered a subject. (S3-A)

Mastery-orientated trainees and supervisors are more pre-disposed towards self-assessment and/or formative assessment. They believe that learning is stimulated by feedback, self-reflection, coming back on issues and personal coaching.

To me, all feedback is valuable; I think you can use all information one way or another on your way to medical expertise. (T3-B)

To become a postgraduate trainee someone has to be very well motivated: first she has to get good grades in secondary school, second she needs to pass the undergraduate curriculum and next she needs to invest a lot of time and has to compete hard to acquire a training number. I sincerely wonder whether more summative assessment will increase performance for this group of motivated people. (S3-B)

Learning environment

The perception of the learning environment at large was another important feature when active engagement in assessment for learning was discussed. The term ‘learning environment’ in this study should be understood in its widest sense, encompassing encouraging supervisors, clear assessment procedures and a supportive learning climate.

Committed supervisors

Trainees frequently stressed the importance of committed supervisors, who are interested in teaching as well as in developing their teaching skills.

You can notice which supervisors are really teaching-minded: they tend to do teach the teacher courses, prepare themselves and give structured feedback. (T5-C)

When supervisors were additionally involved in in-training assessment, mentoring skills became another precious asset. It transpired that the ability to approach trainees with a genuine interest in their long-time progression, both in their career and private lives was sorely missed by the trainees.

I do not have the impression that my supervisor is well informed on how I’m progressing in my training. I find that a supervisor should be interested in his trainees and should be well informed on their progress and which competencies they have achieved. (T5-D)

Clear standards and consequences

Furthermore, both trainees and supervisors expressed the need for clearer standards and clarity on the consequences of substandard performance. Several trainees mentioned devaluation and/or disregard of feedback as a result of absence of these.

The only thing is, if you are being assessed with the purpose to stimulate learning and the result of the assessment is without consequences, the impact will be disappointing. (T6-C)

Acknowledgement

Additionally, acknowledgement of the importance of clinical teaching by hospital management, resulting in dedicated teaching time for trainees and supervisors, was stressed as an important enabler of formative assessment.

And you need time, we currently supervise the mini-CEX during our labor ward supervision, however, you are paged continuously, which is very disturbing, and then you have to go and do something else before you even had the opportunity to give feedback. (S7-C)

Credibility of feedback

Not all feedback automatically translated into learning. For this the credibility of both feedback content and feedback giver were of paramount importance. When these were judged not credible enough, feedback was often rejected and consequentially did not result in the intended learning.

The credibility of feedback content depended on issues like authenticity (does the feedback relate to a representative, directly observed doctor–patient encounter), and whether feedback can be judged against a clear, well accepted standard (e.g. guideline, latest research).

It is no problem to get some advice of a supervisor on a patient problem; however, usually I get a very directive answer, without him seeing the patient, while I really would like to get some structured feedback after being observed with the patient. (T3-A)

Apart from feedback content, personality traits and feedback strategies were other important determinants of the credibility of supervisors. Feedback from a supervisor who was perceived as a role model, well respected, enthusiastic about his chosen (sub) specialisation, encouraging to trainees, was valued very much. Especially, if this person was also able to provide structure during feedback sessions and remembered when and how to come back on issues.

… especially someone whom I personally regard as an exemplary doctor. If I see he is a professional in a way that I would like to be in the future. That's the person from whom I prefer to get feedback. (T3-B)

Discussion

The purpose of this study was to explore trainees’ and supervisors’ perception of what factors determine active engagement in assessment for learning. Central themes appear to be individual perspectives on feedback of trainees and supervisors, a supportive learning environment and credible feedback. We will discuss our findings in the light of existing literature and provide recommendations how active engagement in assessment for learning can be promoted.

Individual perspective on feedback

The individual perspective on feedback in this study is determined by both ownership and achievement goal orientation. Ownership can best be described as an internal drive to make the most of postgraduate training and act accordingly (Kusurkar et al. 2012). In our study, ownership of trainees plays a central role in both the motivation of a trainee to ask for feedback and the supervisor's willingness to start giving feedback. This finding is not unique, studies in other areas of higher education yield similar findings (Sadler 1989; Struyven et al. 2003; Veloski et al. 2006; Watling et al. 2008). In our study progress in training years and progressive independence increase the awareness of personal responsibility for and an active approach of trainees to their learning pathway. However, supervisors expect ownership right from the start of training. It is therefore essential that the importance of ownership is explicitly discussed with junior trainees right from the start of training.

Supervisors reveal little information about their motivation to get actively engaged in formative assessment. From the scant literature available on the subject, the picture arises that both a feeling of responsibility for the training of future doctors and beliefs arising from the supervisors’ achievement goal orientation play a role (Dahlstrom et al. 2005; Dudek 2005; Cleveland et al. 2007).

This is in accordance with our finding that goal orientation is a main determinant of the assessment preference of both trainees and supervisors. Whereas people with a mastery-goal orientation tend to focus on acquiring and developing competence, welcoming all feedback as an opportunity to improve their learning, the focus of people with a performance-orientation tends to be on demonstrating one's competence and outperforming others (VandeWalle et al. 2001; Wolters 2004; Senko et al. 2011); the latter usually valuing the clear standards and consequences of summative assessment (Entwistle & Tait 1990; Birenbaum 1997; Gijbels & Dochy 2006). Goal orientation partially depends on contextual factors and as such is not a given fact; nevertheless, instructional interventions that were designed to increase the adaption of a mastery orientation have been disappointing (Kember & Gow 1989; Black & Wiliam 1998). However, even though goal orientation itself may be difficult to modulate, awareness of both supervisors and trainees of their personal achievement goal orientation should be stimulated as this will aid in customising assessment and preventing frustrations on both sides.

Supportive learning environment

Participants of both focus groups point out that a supportive learning environment acts as an important facilitator in engaging in formative assessment. The need for dedicated assessment moments and teaching time can frequently be heard. The finding that time is important for formative assessment is supported by previous studies showing that time-pressures induce a surface learning approach (Gielen et al. 2003; Struyven et al. 2003).

Both trainees and supervisors express a need for clear standards of performance in the combination with unambiguous consequences for substandard performance. This in concordance with the literature where goal-setting is an important part of maximising learning (Sadler 1989). However, even though most people will agree what bad performance is for a doctor, the complex situation that a doctor encounters on the everyday work floor requires performances which are multi-dimensional (Yorke 2003), making it very difficult to explicate what good performance is, with the risk of getting lost in detailed, unrealistic lists full of desirable attitudes and skills (Southgate et al. 2001; Price et al. 2010; Hatem et al. 2011). More research should be dedicated to empowering supervisors in explicating what the minimal level of performance of a trainee should be and what are appropriate consequences of substandard performance. Meanwhile, there is a need for a continuing discussion on the work floor about professional values and what pertains to good clinical practice.

One can hypothesise that the trainees’ call for committed supervisors that we found is a direct consequence of the fragmentation of postgraduate training. Quite recently, data have started to emerge that, in order to increase the effectiveness of feedback and the subsequent impact on learning, trainees and supervisors need to engage in meaningful relationships over time, so that long-time follow up becomes a possibility (Carless 2006; Watling et al. 2010). Scheduling postgraduate rotations of sufficient length with a dedicated supervisor, together with emphasising the importance of the supervisor–trainee relationship in discussions with trainees and supervisors, will increase the effectiveness and credibility of assessment for learning.

Credibility of feedback and/or feedback giver

Our participants particularly emphasise the importance of credible feedback and feedback givers as perceived by trainees and the importance of authentic assessment as perceived by supervisors. Preferably, the gold standard in clinical practice is set by robust scientific evidence, nonetheless, the realm of daily clinical practice is full of complex situations for which no unambiguous evidence is available as yet. However, as van Ende already points out in 1983: without feedback young doctors tend to develop a system of internal validation that excludes validation from external sources (Ende 1983). In the meantime the evidence that physicians are rather poor at self-assessment is accumulating (Davis et al. 2006; Kruger & Dunning 2009; Nothnagle et al. 2011), making it imperative to develop a system of external evaluation of learner performance that participants trust and use.

At this point, it is important to stress that formative assessment is more than just giving feedback on a single occasion. For formative assessment to exert an effect on learning a plan of action, follow up and an opportunity to demonstrate improvement should be part of the process (Black & Wiliam 1998; Ericsson 2004) making it much harder to discard feedback or feedback giver as not credible enough. Furthermore, the ensuing discussions can form an important part of the learning process.

Strengths and weaknesses

Strength of our study is that by exploring trainees’ and supervisors’ perceptions of what factors determine active engagement in formative assessment we gained useful insights how active engagement with assessment for learning can be promoted. Combining the perspectives of both trainees and supervisors made it possible to explore the roles of both stakeholders in the process. Using the focus group technique has provided us with rich quality data and made the complexity and multi-dimensionality of formative assessment in daily clinical practice evident.

As participation in the focus groups was voluntarily, it is possible that participants were more than averagely engaged in postgraduate assessment and/or the holders of strong views. On the other hand, it can be argued that holders of strong views have a lot of information to share, which was exactly what we were looking for. Furthermore, some sort of group censoring cannot be ruled out, as participants of each focus group were connected to the same training programme.

Conclusion

The educational impact of formative assessment is multi-dimensional and actively engaging in assessment for learning is quite a challenge to both trainees and supervisors. Individual perspectives on feedback of trainees and supervisors, a supportive learning environment and credible feedback are all important determinants in this process. Every one of these factors should be taken into account when the utility of formative assessment in postgraduate medical training is assessed.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.