Abstract
Background: There is a paucity of research on the effects of interactive feedback methods and sustained assessment strategies in formative assessment of students in the workplace.
Aims: To investigate the outcome of long-term use of an assessment tool.
Methods: Retrospective analysis of summarised assessment tools from 3 years of 464 final-year students in general practice. Quantitative data were analysed using non-parametric tests and a multi-level approach. Qualitative data were subjected to content analysis.
Results: Students’ main deficiencies in the consultation were in the domains of working diagnoses and management plans; however, supervisors emphasised goals of patient-centred communication and structure of the medical interview. As a group, students underestimated their clinical performance, compared to supervisors’ judgement. Most students were supplied with specific goals, 58% with specific follow-up feedback. The majority of students and supervisors were satisfied with the assessment strategy. Long-term experience with the tool significantly increased the proportion of specific goals and feedback to students, supervisors’ stringency of the assessment, and their satisfaction with the tool.
Conclusions: The summarised assessment strategy proved feasible and acceptable with students and supervisors in a continuous attachment with assigned personal supervisors. However, there was room for improvement in supervisors’ provision of specific follow-up feedback.
Introduction
Feedback on students’ performance in clinical settings is a fundamental component in students’ development of clinical skills (Ende Citation1983; Cantillon & Sargeant Citation2008). Feedback should focus on action, not on the individual; be specific, and linked to personal goals, to help reinforce desirable performance and to correct poor performance (Hewson & Little Citation1998; Hattie & Timperley Citation2007; Cantillon & Sargeant Citation2008; Archer Citation2010). The qualities of effective feedback are supported by research (Hewson & Little Citation1998; Hattie & Timperley Citation2007), and are associated with students’ perceptions of high-quality teaching (Cantillon & Sargeant Citation2008).
Work-based assessment with feedback can have a powerful impact on the change of medical students’ behaviour (Norcini & Burch Citation2007). However, several studies suggest that direct observation and formative assessment of students in the workplace occur rather infrequently (Daelmans et al. Citation2004; Howley & Wilson Citation2004; Kogan & Hauer Citation2006; Pulito et al. Citation2006). Furthermore, the feedback provided is often of questionable value to students (Daelmans et al. Citation2006; Kogan & Shea Citation2008).
Poor supervisor contribution has been identified as the most important limiting factor in work-based assessment and feedback (Norcini & Burch Citation2007). By involving supervisors in the planning of the process, the frequency of assessment and feedback can increase (Daelmans et al. Citation2006), and by offering training to supervisors in rating students’ performance, accuracy can improve (Shumway & Harden Citation2003). To facilitate assessment of students and residents in the workplace, several unique structured tools have been developed and implemented (Kogan et al. Citation2009). Most tools include items on communication, history taking, physical examination, management and counselling.
The reliability of work-based assessment is a matter of appropriate sampling across content, patients and assessors, but in formative assessment a compromise on reliability in favour of educational impact is acceptable (van der Vleuten Citation1996). However, the educational impact of assessment and feedback on students’ performance is an issue that has only been sparsely researched (Norcini & Burch Citation2007). Students’ positive perceptions of the assessment process are reported in several studies (Lane & Gottlieb Citation2000; McKinley et al. Citation2000; Paukert et al. Citation2002; Burch et al. Citation2006), but results from students’ modifications of knowledge, attitudes and skills, self-assessed or assessed by their supervisors, are more seldom described (Kogan et al. Citation2009).
Interactive feedback methods, including students’ self-assessment and an action plan for improvement with follow-up, are critical components of effective feedback (Hattie & Timperley Citation2007), but underutilised in feedback to students (Fernando et al. Citation2008) and to residents (Holmboe et al. Citation2004). Written educational strategies for improvement (Hastings et al. Citation2006), and students’ self-assessment with feedback comments from supervisors and patients (Braend et al. Citation2010) have been reported. However, these studies did not include follow-up on feedback provided.
Long-term use of an assessment tool might allow supervisors to gradually get accustomed to the tool and improve in their assessment and feedback to students. However, most studies on workplace-based assessment of undergraduates are based on short interventions, and there is a paucity of research reports on sustainable assessment with feedback as a routine feature in the workplace (Norcini & Burch Citation2007).
The medical school curriculum of Lund University, Sweden, comprises five and a half years (11 semesters). At the time of the study, the intake was approximately 80 students twice a year. The programme of Community Medicine ran in the second half of students’ 10th semester, and included 16 days (4 days every second week) of practice in a health centre. Students’ end-of-clerkship evaluations and results from an investigation of the educational climate, measured by DREEM (Edgren et al. Citation2010), raised concerns about the feedback process during the general practice attachment. To improve the provision of feedback on students’ performance, we introduced an assessment and feedback tool, which included students’ self-assessment, mutually agreed goals, and follow-up feedback.
This article retrospectively analyses the results from the first 3 years’ use of the tool. The principal aim of this study was to investigate the outcome of long-term use of an assessment tool; specifically the content and nature of feedback provided, supervisors’ evaluations of students’ clinical performance and achievements, students’ self-assessment, and students’ and supervisors’ attitudes to the feedback strategy.
Methods
Setting
The objectives of the general practice attachment are that students will learn to manage common diseases and complaints in primary care, supervised mainly by personal general practice supervisors, but also by other general practitioners (GPs) at the practice. GPs in 50–60 health centres are engaged as supervisors. Students are expected to independently perform medical interviews and physical examinations, present working diagnoses and management plans, and conclude consultations under supervision. Supervisors are requested to frequently observe students during medical interviews and physical examinations, and are supposed to provide brief feedback daily as part of the flow of work. Moreover, summarised, formal feedback should be given twice during the attachment, at the midpoint and at the end.
In Sweden, medical students are allowed to work as locums in hospital after nine completed semesters in medical school. They work as ‘interns’, however closely supervised and with limited duties and responsibilities. Several students make use of this opportunity, particularly during summer holidays.
Assessment tool
We designed a tool for mandatory formative assessment and feedback to be applied in the two summarised feedback sessions. Prior to the study we had used evaluation forms, focusing mainly on students’ communication skills and with poor opportunities for assessment of other domains of the consultation, i.e. clinical reasoning, problem solving and management. The new tool was inspired by the modified version of the Leicester Assessment Package (LAP), developed for general practice, and previously found to be valid and reliable (McKinley et al. Citation2000). The modified LAP included five competency domains; however, we altered the instrument slightly, subdividing the three ‘double’ domains, resulting in eight domains of ‘equal value’ (). A seven-point scale was used with a rating span from 1–2 (unsatisfactory), 3 (borderline), 4–5 (satisfactory) to 6–7 (superior). The detailed competencies included in the LAP were not evaluated, but applied on the form to clarify the contents of each domain (Appendix). We briefed the GP supervisors on the tool in a regular supervisors’ meeting, and involved them in the modification of the form before implementation.
Box 1. Competency domains of the assessment form.
Identical assessment forms were used for students’ self-assessment and for supervisors’ formative assessment. Feedback sheets, attached to the forms, supplied detailed information on the assessment process and on how to differentiate between the scores of the rating form. The sheets also had space for narrative comments on agreed goals and feedback, and contained a few questions to students and to supervisors ().
Box 2. Questions to students and supervisors on the feedback sheets.
Assessment process
At the start of the programme, students were informed verbally of the assessment process. Although the assessment was formative in nature, students were required to self-assess and to be evaluated by their supervisors. Students were supplied with assessment forms and feedback sheets, all required to be completed and returned to the faculty office by the end of the course.
Before the first feedback session, supervisors and students should independently complete the forms by allocating suitable scores in each of the eight domains. The feedback should be provided as a ‘reflective feedback conversation’ (Cantillon & Sargeant Citation2008), goals and strategies for further training should be mutually agreed, and recorded on students’ and supervisors’ feedback sheets. At the end of the attachment students were supposed to be provided with follow-up feedback, documented on both feedback sheets. However, recording on only one of the sheets could be accepted.
As the assessment was formative, the scores of the assessment forms did not contribute to final examination results, but provided a framework for a dialogue between students and supervisors. The final summative examination consisted of a global, non-graded, pass/fail assessment of students’ performance and a case-based written test at the end of the semester.
There was no requirement of prior training for supervisors; however, we realised a need for training in two areas: the rating of students’ performance, and the provision of effective feedback. We offered a 1-day workshop on effective feedback on four different occasions during the first 2 years of the study. The workshops were based on video recorded role-plays, using written scenarios and standardised students, reviews of videotaped performances, and facilitated small group discussions. Written information with an overview of the basic tenets of effective feedback was also sent to supervisors each semester. Moreover, we included 2 h of rating practice, based on the tool, in our regular supervisors’ meetings twice a year, attended by approximately 70% of supervisors.
Ethics permission was at the time of the study not required for research based on data already collected for the purpose of assessment, if data were fully anonymised.
Data analysis
Assessment forms and feedback sheets from students and supervisors were collected during the first 3 years. Data from the first feedback session consisted of scores on the assessment forms and narrative agreed goals on the feedback sheets; from the second session, the narrative follow-up feedback on the feedback sheets.
All quantitative data, including students’ results on the final written test, were entered into SPSS, version 17.0. Students and supervisors were allocated code numbers to prevent identification of individuals. Non-parametric tests (Mann–Whitney U and Kruskal Wallis) were used for the comparison of groups. However, in calculations where groups of students, based on supervisors’ scores, were compared, we had to correct for the dependencies with repeated measures, as several students were ‘clustered’ into the same supervisors. We then used a multi-level approach, where a correction for dependence is built into the model (Maas & Snijders Citation2003). MlWin version 2.17 (Rasbash et al. Citation2009) was used for analysis, with residual (or restricted) maximum likelihood.
We estimated the internal consistency reliability of the tool using Cronbach's alpha for each semester and for the whole period of study. Spearman's correlation coefficient was calculated for associations between supervisors’ assessment of students’ average scores and students’ results on the written test. Since the maximum scores of the written test varied between semesters, students’ scores were ranked into deciles each semester. Supervisors’ assessment of students’ average scores were ranked into deciles for the whole period of study.
The narrative comments on the feedback sheets were written verbatim into a word processor by one of the authors, and then subjected to ‘basic’ content analysis (Crabtree & Miller Citation1999), involving a categorisation of words and meanings, using predefined templates. Mutually agreed goals in the first feedback session and follow-up feedback in the second session were identified as generalised or specific. Specific goals and specific feedback comments were mapped against the framework of the eight competency domains. The specific feedback was further evaluated for the possible occurrence of ‘feed forward’, advice for the future (Hattie & Timperley Citation2007).
Results
Students and supervisors
Analyses were based on the 464 students, who finished the course during the first six semesters (3 years). The number of students each semester varied between 71 and 83; 256 (55%) of students were female; median age was 26 years (range 24–45 years). Half of the students, with equal gender distribution, had worked as locums, and 281 students (61%) had a female supervisor.
The total number of supervisors engaged was 151, varying between 65 and 78 supervisors each semester; 87 (58%) were female. A few supervisors were responsible for more than one student each semester, but in alternate weeks. In the six semesters 49 supervisors were responsible for 4–12 students, 55 for 2–3 students and 47 for only 1 student. A total of 42 supervisors (28%) took part in one of the four workshops provided.
First feedback session: scores of the assessment form
The assessment form for judging students’ performance was used for 458 students; 99% (). Six students from the first two semesters were not evaluated, but were provided with free text summaries by their supervisors. Self-assessment data were missing from two students, and data on previous work as locums for eight students.
Figure 1. Flow chart of students between first and second feedback sessions.
‘Relationship with patients/time management’ and ‘Medical interview’ were awarded supervisors’ highest ratings of students’ performance, and ‘Working diagnoses’ and ‘Investigations and treatment’ the lowest (). Students’ highest self-ratings were assigned to ‘Medical interview’ and ‘History taking’, and the lowest to ‘Investigations and treatment’ and ‘Working diagnoses’.
Table 1. First feedback session: scores of the assessment form.
Box 3. Examples of generalised/specific goals (first feedback session) and generalised/specific feedback and ‘feed forward’ (second feedback session).
An ‘average’ score, averaging each student's results in the eight competency domains, was calculated (). Supervisors’ mean of students’ average scores was 5.3 (range 3–7), and students’ self-assessments showed a mean of the average scores of 4.5 (range 2.4–6.8).
Male students rated themselves significantly higher than female students in three competency domains (); however, in supervisors’ assessment there was no significant difference between genders.
Students with work experience as locums rated themselves significantly higher than students without work experience in three competency domains and in their average scores (). Supervisors’ ratings revealed significantly higher scores for the locum students, compared to the non-locum students in ‘Physical examination’, 5.5 vs. 5.2 (p = 0.003); ‘Investigations and treatment’, 5.1 vs. 5.0 (p = 0.02); and ‘Explanation and planning’, 5.4 vs. 5.2 (p = 0.02).
Most students were evaluated by their supervisors using a range of only two scores, but students themselves spread their scores in a wider range ().
Table 2. First feedback session: range of scores of the assessment form.
First feedback session: agreed goals
Narrative comments on students’ and supervisors’ feedback sheets were combined and categorised as specific goals (407 students; 88%; ), generalised goals (27 students; 6%; ) and no goals (30 students, 6%); ().
Specific goals were mapped against the framework of the eight competency domains (). There were 1084 goals recorded in total, with a mean of 2.7 and a range of 1–7 domains per student. The majority of specific goals related to ‘Medical interview’, particularly three detailed competencies: ‘Summarises’, ‘Negotiates agenda’ and ‘Elicits patient's perspective’. The domain of ‘Explanation and planning’ was also frequently addressed, with the emphasis on ‘Relates explanations to patient's perspective’ and ‘Checks patient's understanding’.
Table 3. First feedback session: specific goals; second feedback session: specific feedback.
Students, supervised by GPs who used a wider range of scores (3–5 scores, compared to 2 scores or 1 score), were significantly more likely to receive specific goals (p = 0.002). When students’ and supervisors’ gender was taken into account, female students with female supervisors were provided with specific goals significantly more often (95%); p = 0.004, compared to female students with male supervisors (85%), male students with female supervisors (85%) and male students with male supervisors (82%).
Second feedback session: follow-up feedback
The narrative comments of the follow-up feedback at the end of practice were categorised into specific feedback (266 students; 58%; ), generalised feedback (99 students; 21%; ) and no feedback (99 students; 21%); (). The category of specific feedback was further evaluated, and 86 of these 266 students (32%) were additionally provided with ‘feed forward’ (). Of the 266 students, 10 were given specific feedback despite no recorded specific goals ().
Specific feedback for the 256 students with previously documented specific goals was mapped against the framework of the eight competency domains (). There were 729 feedback comments in all, with a mean of 2.8 and a range of 1–6 domains per student. There was no significant difference between genders.
Students’ evaluation of the assessment strategy
The response rate for the students’ three questions on the quality of feedback varied between 89% and 93%. A total of 77% of respondents were of the opinion that their supervisors had correctly identified their strengths and weaknesses, 20% partially. Concerning specific advice on how to improve their consultations, 70% of respondents were completely satisfied, 26% partially. Finally, 77% of respondents thought they received effective follow-up feedback, 17% partially.
Students who were provided with specific goals were significantly more satisfied with their supervisors’ identification of their strengths and weaknesses (79%), compared to those with generalised goals (72%) and no goals (57%); p = 0.042. Students who received specific follow-up feedback were significantly more often completely satisfied (81%), compared to those with generalised feedback (78%) and no feedback (63%); p = 0.004.
Supervisors’ evaluation of the assessment strategy
Of the 151 supervisors, 147 (97%) answered the question whether the tool contributed to improved structure in the assessment of students’ performance. Several supervisors changed their views from partially to completely positive over the years. Finally, 111 (76% of respondents) were completely positive, 34 (23%) partially and 2 (1%) negative.
Of the 49 supervisors who had been responsible for 4–12 students during the six semesters, 86% were finally completely satisfied, compared to the 55 supervisors who had supervised 2–3 students (80%), and the 43 supervisors who only used the tool once (58%); p = 0.006. Of the 42 participants of the workshops 86% were completely satisfied.
Time and experience
Specific goals documented at the first feedback session increased significantly between the first three and last three semesters from 84% to 91% of students (p = 0.019). Specific feedback at the second session improved from 50% of students to 65% between the first three and last three semesters (p = 0.006).
A significantly increased stringency was found in the assessment, depending on supervisors’ experience with the tool. Students (n = 115) who were rated by experienced supervisors, who were using the tool for the 4th–6th semester, were assigned a significantly lower mean of average score (5.0); p = 0.001, compared to students (n = 187) who were supervised by GPs using the tool for the 2nd–3rd semester (5.3) and students (n = 156) whose supervisors were using the tool for the first time (5.4). However, students assigned to more experienced supervisors did not receive a significantly higher proportion of specific goals or specific feedback.
Internal consistency and correlations with written exam
We estimated internal consistency reliability using Cronbach's alpha for each semester separately (0.94–0.96) and for the whole study period (0.95). Correlations among specific competency domains ranged from 0.63 to 0.83.
The number of students who took the written test at the end of the course was 447; the remaining 17 students took the test on a later occasion. A significant but very weak correlation (Spearman rho = 0.16; p = 0.001) was found between supervisors’ ranked average scores of students’ clinical performance and students’ ranked written examination scores. However, this correlation was enhanced when only the 10% students (n = 44) with the lowest assessment scores were analysed (Spearman rho = 0.34; p = 0.026).
Discussion
This observational study on 3 years of work-based assessment, using a retrospective design, explored the whole feedback process. Almost all students were assessed and given feedback in the first session, most students were provided with specific goals, and a majority received follow-up feedback, aligned to their goals. Most students and supervisors were satisfied with the assessment strategy. Long-term use of the tool significantly increased the proportion of specific goals and specific feedback, supervisors’ stringency of the assessment, and their satisfaction with the tool.
Some interesting inconsistencies were found at the first feedback session between scores of the assessment forms and goals agreed for further training. The competency domains awarded the lowest scores by both students and supervisors were ‘Working diagnoses’ and ‘Investigations and treatment’, findings consistent with previous research on senior students’ performance in general practice (Hastings et al. Citation2006; Braend et al. Citation2010). Students probably perceived the generation and testing of hypotheses across the range of non-specific problems in general practice as problematic, and their previous experience from hospital clerkships in suggesting management plans was likely to be sparse. However, these domains, although regarded as students’ main deficiencies, only resulted in goals in a third of students. On the other hand, almost half of students were provided with goals in the two domains ‘Medical interview’ (awarded the highest scores of all domains) and ‘Explanation and planning’. Items of patient-centred communication and the structure of the medical interview were particularly emphasised. These goals focused on ‘process’ before ‘content’, and were probably perceived as more appropriate by supervisors. Others have also recommended a focus on ‘process’ teaching in direct observation of students’ clinical performance in general practice (Russell Citation2009).
As a group, students underestimated their clinical performance, compared to their supervisors’ judgement; a finding supported by previous research in general practice (Braend et al. Citation2010). A gender difference was noted, with male students rating their performance significantly higher than females in three competency domains. However, supervisors’ ratings did not suggest any gender difference. A trend for higher levels of confidence by males in their self-assessment was also found in the systematic review by Colthart et al. (Citation2008).
There was a significant difference between the locum students’ and the non-locum students’ self-assessment in three competency domains as well as in their average scores, demonstrating the locum students’ enhanced self-assurance. The differences between the locum and non-locum students were supported by supervisors’ assessment, particularly in physical examination, where the locum students’ performance was evaluated as significantly more skilled. Short periods of locum work evidently resulted in students being more accustomed to and more confident in physical examinations of patients.
As established, self-assessment on its own has clear limitations in medical students’ evaluations of their performance, and requires applicable external standards for comparison (Eva & Regehr Citation2005; Colthart et al. Citation2008). However, self-assessment as an elicitation of students’ perceptions of their performance, before receiving feedback, is a recommended strategy (Hewson & Little Citation1998; Cantillon & Sargeant Citation2008). The students in our study could use the feedback for reflection and integration with their self-appraisals, constituting an ‘informed self-assessment’ (Sargeant et al. Citation2010). In this process, the reflection, particularly if facilitated by the discussion with the supervisor, could play a key role in deciding whether to accept, assimilate and use the feedback or not (Sargeant et al. Citation2009).
Narrative comments by supervisors suggested that some supervisors were reluctant to provide honest corrective feedback to students they had worked closely with, which resulted in inflated scores or scores identical for all competency domains. Problems in transmitting unpleasant messages, fear of upsetting the student, or adversely affect their relationship with the student might contribute to supervisors’ reluctance. Students with supervisors who used a wide range of scores received significantly more specific goals, a finding supported by previous research (Fernando et al. Citation2008). Narrative comments by these supervisors indicated that they probably took the time to assess students accurately in the different domains and to discuss specific goals. Students also valued specific goals, designed to improve their performance, and these students were significantly more satisfied with their supervisors’ identification of their strengths and weaknesses. Moreover, female students with female supervisors were significantly more likely to receive specific goals, compared to remaining student–supervisor dyads. However, these findings could only permit incomplete conclusions. A previous study showed that female supervisors were more likely to give feedback to male than to female students, and that the gender dyad resulting in most feedback was male supervisor and male student (Carney et al. Citation2000).
The distribution of specific follow-up feedback in the second session almost exactly mirrored the distribution of specific goals; however, the proportion of specific feedback recorded could be increased. A fifth of students received only follow-up feedback of a generalised nature, often as nondescript praise. This kind of feedback does not contribute to students’ further development, and can be embarrassing to students (Ende Citation1983). The students’ evaluations of the second session also clearly supported specific follow-up feedback. In a fifth of cases, no follow-up feedback was documented, but two-thirds of these students were still satisfied, and have possibly received appropriate verbal feedback.
The proportion of specific goals and specific feedback increased significantly between the first three and last three semesters, although we could not show a significant association with supervisors’ length of experience of the tool. We interpreted these findings as an improved ‘feedback culture’ in practice, also among less experienced supervisors.
The face validity of the assessment was ascertained by involvement of supervisors in the adaptation of the assessment tool before implementation, and content validity was ensured by alignment to the goals of the attachment. In the estimation of concurrent validity, we found a low correlation with scores on the written exam; this was an expected finding, and in accordance with a recent systematic review of structured tools for direct observation (Kogan et al. Citation2009). However, the correlation increased if only students with the lowest performance scores were analysed, a result that could support the idea of supervisors’ ability to accurately identify students with an inadequate knowledge base. Internal consistency reliability of the tool was very high, suggesting more of a global assessment, and also in line with previous literature (Kogan et al. Citation2009). As for the educational impact of the formative assessment, we could confirm satisfaction with the assessment strategy among most students and supervisors. Furthermore, the narrative comments of the follow-up feedback, aligned to personal goals, could be interpreted as students’ development in clinical performance, albeit not objectively measured.
To the best of our knowledge, no previous study has reported on feedback, applied at two separate sessions in clinical training. Neither have we been able to find research designed to investigate the effects of long-term routine use of an assessment tool. Other strengths of our study are the face and content validity, the acceptability of the strategy, the application of interactive feedback methods and the attempt to form a conception of students’ achievements in practice.
Some limitations of the assessment strategy and the study should be considered. First, we were unable to examine the inter-rater reliability of the assessment, as the tool was not used in each of students’ patient encounters, in contrast to most research on work-based assessment. However, summarised, formal assessment twice during the attachment was a feasible strategy under the ideal conditions of continuity and a personal supervisor. As our main purpose for the assessment was the provision of feedback, we considered the sampling of cases and problems in general practice adequate for acceptable reliability in most cases, even if evaluations were more subjective (van der Vleuten & Schuwirth Citation2005). Second, despite recommendations on additional feedback on a day-to-day basis, some supervisors might have delayed the provision of feedback to a scheduled summarised session. It is widely acknowledged that for maximum effect informal feedback should take place shortly after an event, while the event is still fresh in mind (Cantillon & Sargeant Citation2008). However, in supporting students’ development of more advanced integrated professional skills, delayed feedback can be advantageous (Hattie & Timperley Citation2007; Archer Citation2010). Third, we were unable to perform an evaluation of supervisors’ benefits of the workshops on effective feedback, as participation was voluntary; less than a third of supervisors attended, and nearly half of these participated after one or two semesters’ experience with the tool.
The introduction of the tool emphasised the importance of direct observation, assessment and feedback in a clinical environment. After implementation of the tool, students’ end-of-clerkship evaluations confirmed increased satisfaction with the feedback process, and the results from a follow-up investigation of the educational environment, as measured by DREEM, were significantly improved with regard to supervisors’ provision of constructive criticism (Edgren et al. Citation2010).
Since the time of the study, the assessment tool has been slightly modified, and is now well established as documentary evidence for students’ portfolio reflections (Haffling et al. Citation2010). Accordingly, it is now possible to examine cases of poor feedback and deficient student–supervisor relationships, which we could not accomplish during the study, due to the retrospective design.
Conclusions
This study extended our knowledge about work-based assessment and feedback by presenting a feasible strategy, applied to a continuous attachment in general practice with personal supervisors. The strategy was approved by students and supervisors. Almost all students were assessed and provided with feedback in the first session, most students were given specific goals, and a majority received specific follow-up feedback, aligned to their goals. Long-term experience of the assessment tool could successfully contribute to an increased stringency of assessment and an enhanced ‘feedback culture’ among supervisors. However, there was room for further improvement, and additional supervisor training in the provision of effective follow-up feedback was required.
Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.
References
- Archer JC. State of the science in health professional education: Effective feedback. Med Educ 2010; 44: 101–108
- Braend AM, Frandsen Gran S, Frich JC, Lindbaek M. Medical students’ clinical performance in general practice: Triangulating assessments from patients, teachers and students. Med Teach 2010; 32: 333–339
- Burch VC, Seggie JL, Gary NE. Formative assessment promotes learning in undergraduate clinical clerkships. S Afr Med J 2006; 96: 430–433
- Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ 2008; 337: 1292–1294
- Carney PA, Dietrich AJ, Eliassen S, Pipas C, Donahue D. Differences in ambulatory teaching and learning by gender match of preceptors and students. Fam Med 2000; 32: 618–623
- Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, McKinstry B. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach 2008; 30: 124–145
- Crabtree BF, Miller WL, editors. Doing qualitative research. Sage Publishing, Thousand Oaks, CA 1999
- Daelmans HEM, Hoogenboom RJI, Donker AJM, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. Effectiveness of clinical rotations as a learning environment for achieving competences. Med Teach 2004; 26: 305–312
- Daelmans HEM, Overmeer RM, van der Hem-Stokroos HH, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. In-training assessment: Qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Med Educ 2006; 40: 51–58
- Edgren G, Haffling AC, Jakobsson U, McAleer S, Danielsen N. Comparing the educational environment (as measured by DREEM) at two different stages of curriculum reform. Med Teach 2010; 32: e233–e238
- Ende J. Feedback in clinical medical education. JAMA 1983; 250: 777–781
- Eva KW, Regehr G. Self assessment in the health professions: A reformulation and research agenda. Acad Med 2005; 80: S46–S54
- Fernando N, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ 2008; 42: 89–95
- Haffling AC, Beckman A, Pahlmblad A, Edgren G. Students’ reflections in a portfolio pilot: Highlighting professional issues. Med Teach 2010; 32: e532–e540
- Hastings AM, McKinley RK, Fraser RC. Strengths and weaknesses in the consultation skills of senior medical students: Identification, enhancement and curricular change. Med Educ 2006; 40: 437–443
- Hattie J, Timperley H. The power of feedback. Rev Educ Res 2007; 77: 81–112
- Hewson MG, Little ML. Giving feedback in medical education. J Gen Intern Med 1998; 13: 111–116
- Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. J Gen Intern Med 2004; 19: 558–561
- Howley LD, Wilson WG. Direct observation of students during clerkship rotations: A multiyear descriptive study. Acad Med 2004; 79: 276–280
- Kogan JR, Hauer KE. Brief report: Use of the mini-clinical evaluation exercise in internal medicine core clerkships. J Gen Intern Med 2006; 21: 501–502
- Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees. A systematic review. JAMA 2009; 302: 1316–1326
- Kogan JR, Shea JA. Implementing feedback cards in core clerkships. Med Educ 2008; 42: 1071–1079
- Lane JL, Gottlieb RP. Structured clinical observations: A method to teach clinical skills with limited time and financial resources. Pediatrics 2000; 105: 973–977
- Maas C, Snijders TAB. The multi-level approach to repeated measures for complete and incomplete data. Qual Quantity 2003; 37: 71–89
- McKinley RK, Fraser RC, van der Vleuten C, Hastings AM. Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the leicester assessment package. Med Educ 2000; 34: 573–579
- Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855–871
- Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg 2002; 183: 300–304
- Pulito AR, Donelly MB, Plymale M, Mentzer RM. What do faculty observe of medical students’ clinical performance?. Teach Learn Med 2006; 18: 99–104
- Rasbash J, Charlton C, Browne WJ, Healy M, Cameron B, 2009. MlWin, Centre for Multilevel Modelling, University of Bristol
- Russell G. Taking time to watch. Observation and learning in family practice. Can Fam Physician 2009; 55: 948–950
- Sargeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, Lockyer J, Loney E, Mann K, van der Vleuten C. The processes and dimensions of informed self-assessment: A conceptual model. Acad Med 2010; 85: 1212–1220
- Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. Reflection: A link between receiving and using assessment feedback. Adv Health Sci Educ 2009; 14: 399–410
- Shumway JM, Harden RM. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach 2003; 25: 569–584
- van der Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ 1996; 1: 41–67
- van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: From methods to programmes. Med Educ 2005; 39: 309–317