Abstract
Background: Many medical students in general practice clerkships experience lack of observation-based feedback. The StudentPEP project combined written feedback from patients, observing teachers and students.
Aim: This study analyzes the perceived usefulness of triangulated written feedback.
Methods: A total of 71 general practitioners and 79 medical students at the University of Oslo completed project evaluation forms after a 6-week clerkship. A principal component analysis was performed to find structures within the questionnaire. Regression analysis was performed regarding students’ answers to whether StudentPEP was worthwhile. Free-text answers were analyzed qualitatively.
Results: Student and teacher responses were mixed within six subscales, with highest agreement on ‘Teachers oral and written feedback’ and ‘Attitude to patient evaluation’. Fifty-four per cent of the students agreed that the triangulation gave concrete feedback on their weaknesses, and 59% valued the teachers’ feedback provided. Two statements regarding the teacher's attitudes towards StudentPEP were significantly associated with the student's perception of worthwhileness. Qualitative analysis showed that patient evaluations were encouraging or distrusted. Some students thought that StudentPEP ensured observation and feedback.
Conclusion: The patient evaluations increased the students’ awareness of the patient perspective. A majority of the students considered the triangulated written feedback beneficial, although time-consuming. The teacher's attitudes strongly influenced how the students perceived the usefulness of StudentPEP.
Introduction
Fifth year medical students at the University of Oslo participate in a 6-week clerkship period in general practice. The clerkship sites are characterized by great diversity in the students’ opportunities for independent consultations, clinical challenges, amount of teacher observation and the amount and quality of feedback (Baerheim & Thesen Citation2003; Gran et al. Citation2008).
Research shows that medical education in general provides little observation and feedback from clinical teachers (Kassebaum & Eaglen Citation1999; Norcini & Burch Citation2007). Good feedback connects assessment and instruction and is essential for learning. Most often, feedback is given orally and informally. A problem with oral feedback is that although the students value it, they do not necessarily recognize it or remember having received it, so that establishing a clear frame for the feedback is recommended (Dobbie & Tysinger Citation2005; Bienstock et al. Citation2007). Educational literature emphasizes the importance of individualized feedback which integrates the student's self-evaluation (Rushton Citation2005). The StudentPEP project () provided triangulation of written feedback from patients and teachers combined with students’ self-assessment (Braend et al. Citation2010).
Figure 1. The StudentPEP project. Note: This overview was used when informing students and teachers about the project.
The main goal of the project was to give the students more and better formative feedback during their general practice clerkship by providing a clear frame for feedback, partly after teacher observation. The intention was that learning more about how they perform in their role as an independent doctor would help them to perform better in the future. The patient feedback was intended to stimulate the students and teachers to reflect together on the patients’ views. Students, in general, emphasize that having an appropriate degree of responsibility for a group of one's own patients with adequate supervision provides high quality learning opportunities, but it has been shown that students more often have to learn by observation (Bowen & Irby Citation2002). Our pilot study indicated that students got more independent consultations and more teacher observation through the StudentPEP project, and this positive side effect might also reduce differences in learning opportunities among the clerkships (Braend et al. Citation2006). This study aimed at answering the following questions: (1) What were students’ and teachers’ perceptions of triangulated written evaluations in the StudentPEP project? (2) How did the students describe positive and negative experiences with StudentPEP? (3) Were there any predictors that could explain positive student perceptions?
Methods
The StudentPEP project was carried out at the University of Oslo in a general practice clerkship from 2007. This study reports data from autumn 2007, when the project had been mandatory for one semester. The project provided the students with a patient evaluation questionnaire, a modified version of the EUROPEP questionnaire (Grol et al. Citation2000), for 25 patients. Five questionnaires were to be filled out by patients after teacher-observed consultations, and 20 were meant for independent student consultations. The teachers gave the students feedback via a questionnaire with mainly open-ended questions about satisfactory performance and suggested improvements on four so-called SOAP elements: taking a history (‘Subjective’), clinical examination (‘Objective’), differential diagnostic judgment (‘Assessment’) and ‘Plan’ (Jones Citation2004). The students assessed their own consultations using the same open-ended questions. A complete description of the instrument and the data collected in the StudentPEP questionnaires in 2007 has been presented earlier (Braend et al. Citation2010).
The project evaluation questionnaire
The project evaluation questionnaire was based on focus group interviews with students in 2005 (Braend et al. Citation2006), regarding students’ perceptions of the feedback in the clerkship in general, and of patient evaluations and teacher observations in particular. We also included some questions from the evaluation questionnaire for the Danish EUROPEP project (Heje Citation2006). Thereafter, the project evaluation questionnaire was piloted with a group of teachers and final year medical students checking for ambiguity and content. It consisted of 29 statements regarding the students’ perceptions of the StudentPEP project (), and some questions regarding practical aspects of the clerkship, presented earlier (Gran et al. Citation2008). The questionnaire also encouraged reflection on the costs and benefits of this tool. The teachers completed a shorter project evaluation questionnaire with 12 statements; among them was the following item: ‘I agree with the total assessment of the student that the patients have given through the StudentPEP questionnaires’. The students handed in the project evaluation questionnaires at the university after their clerkship period.
Table 1. Responses from students and teachers on the StudentPEP project evaluation questionnaire, sorted by principal component analysis on student answers
Statistical analysis
Project evaluation forms from students and teachers were registered in SPSS 15.0. We conducted a principal component analysis to find underlying structures within the student form. The rotation method was oblimin with Kaiser normalization (Hair Citation1995). When the loadings were both negative and positive within one component, we computed inverted variables to match the group. Internal consistency was assessed with Cronbach's alpha for each component and for the whole questionnaire. We labelled each sub-scale after group consensus among the authors and calculated percentage of disagreement (scores 1 and 2), neutral answers (score 3) and agreement (scores 4 and 5) for each question and for the total component. Percentages were also calculated for 11 corresponding scores from the teachers. The correlations between teachers’ and students’ responses were calculated with Spearman's rho. A logistic regression analysis was performed using a dichotomous variable of the students’ answers to whether StudentPEP was worthwhile or not, as the dependent variable.
Qualitative analysis
The answers to the open-ended questions on the project evaluation questionnaires were analysed by a method for systematic text condensation described by Giorgi and modified by Malterud (Malterud Citation2001). We translated the illustrative quotes from Norwegian to English.
Results
Eighty-one students completed the clerkship. A total of 79 students (98%) and 71 teachers (88%) returned their project evaluation questionnaires to the university after the clerkship period.
The principal component analysis resulted in six components explaining 69.5% of the total variance. The components had loadings equal to or above 0.3 and an eigenvalue above 1, the largest being 10.6 (). The components were labelled ‘Opinions of effort versus benefit’, ‘Teacher's oral and written feedback’, ‘Attitude to patient evaluations’, ‘Usefulness of patient evaluations’, ‘Effect on teacher’ and ‘Helpful for reflection’ to reflect the underlying items (listed in ). Cronbach's alpha was between 0.7 and 0.9 for all six components, and above 0.9 for the whole project evaluation questionnaire.
Students’ opinions
The students’ opinions regarding the project differed on and within the six identified components, and there was little total agreement within the student group. For all items, the positive group was larger than the negative group, except for two items, ‘My teacher's feedback was more structured in the triangulated evaluations than generally’ and ‘I think StudentPEP helped me to ensure that I got the opportunity to have independent consultations’. One item regarding both the feedback given in the StudentPEP project and informally, ‘I feel satisfied with the total feedback I got in my clerkship’, stood out from the others with 80% positive answers.
The two components that showed the highest percentage of agreement were ‘Teacher's oral and written evaluation’ and ‘Attitude to patient evaluations’. These components included items concerning experiences and attitudes outside our project, and these items increased the mean for their subscale. If these items were excluded from the analysis, these two components still had the highest percentage of agreement.
Fifty-four per cent of the students agreed that the triangulated feedback sessions provided concrete feedback on their weaknesses, and 59% found it useful to get their teacher's assessments provided within the project. Forty-eight per cent of the students found it useful to evaluate themselves, and 58% of the teachers agreed that it was useful to read the students’ self-assessments.
Students’ versus teachers’ opinions
Fifty-six per cent of the teachers agreed with the patients’ total assessment of the students performance, 39% neither agreed nor disagreed and 5% disagreed. Teachers’ opinions about the project were also varied, again with larger positive groups on all items except one, ‘The teacher forms helped me to structure the assessment’. This item was one of the five items that showed significant correlation between teachers’ and students’ answers. The other correlated questions concerned information prior to the project, time spent, whether it was worthwhile and discussion concerning the feedback. The remaining six questions that were answered by both students and teachers did not show significant correlation. The logistic regression analysis () showed that only two teacher items were found to be significantly associated with whether the student thought that StudentPEP was worthwhile: whether StudentPEP was time-consuming or not and whether the teacher was interested in the patients’ views of the student. A number of factors were found not to be associated: the teacher's age and gender, the student's age and gender, whether the student had her own consultation room or the number of patients the student saw during an average day.
Table 2. Logistic regression analysis showing items associated with student agreement on the item ‘The StudentPEP project was worthwhile’, among 5th year Norwegian medical students in general practice clerkship
Qualitative results
The following result of the qualitative analysis is the condensation of the answers to the open-ended questions representing experiences and perceptions from students. Teachers’ responses in their free-text area for comments were scarce, but were also included in this analysis. Example quotes are given in .
Table 3. Examples of written statements on the StudentPEP project evaluation questionnaire from medical students
Positive patient feedback was found either encouraging or untrustworthy
Patient feedback gave some students insight into their strengths and weaknesses. Among the strengths were communication skills, good listening and thorough examination. Some reported that they needed to improve in giving information and taking care of the patient. Some students had learnt what aspects of doctor behaviour were important to the patients: taking time to listen and taking the patients seriously. Patients’ attitudes were surprisingly positive and increased some students’ self-confidence. Other students felt the patients were too positive, that they could not trust their feedback or that the lack of constructive criticism made them learn less from patients’ feedback.
Triangulation enhanced learning
Some students found it useful that the patients, the teachers and the students themselves had different perspectives when evaluating the consultations. Many emphasized that the feedback from their teacher had been especially valuable, when it was concrete and constructive. Some also found it useful to reflect on their own behaviour in the consultations.
Objections regarding methods and implementation
Many students thought the patients would be more honest if the patients believed that their answers were completely anonymous. Some wanted different feedback methods, different questions on the questionnaires, and some thought that the questions were unsuitable in many consultations. Some reported that the project was time consuming in a busy general practice setting, and some found it difficult to choose suitable patients for the observed consultations.
StudentPEP's overall impact on the clerkship
For some students, StudentPEP ensured them teacher observation and time for feedback and reflection. Some teachers and students preferred to observe and reflect without having to fill in forms; they found other informal feedback more useful. Other teachers and students thought that StudentPEP was a good method for feedback.
Discussion
Our study shows both through qualitative and quantitative analysis that the responses to our project with written triangulated evaluation were mixed, and that the positive effects for the student depended on the teacher's attitude. A majority of the students agreed that StudentPEP confirmed their belief that they could handle the doctor role. Students became aware of their own strengths and weaknesses as seen from the patients’ point of view. The StudentPEP project also ensured them teacher observations and helped them to reflect on their own patients’ views.
We wanted the students to learn more about their consultations in general practice. Such consultations have many aspects. The triangulation sessions may provide feedback on a variety of procedures, on interpersonal as well as technical skills, for students at different levels. It may be overambitious to attempt to make an educational tool broad and specific at the same time. It is also difficult to measure StudentPEP's contribution to the individual student's learning process.
When introducing a new tool, the costs should be weighed against its benefits. A third of the clinical teachers found the project time-consuming. Extra time spent on observation, writing and reflective discussion might be considered as a cost for the teacher, but a benefit for the student. Some teachers preferred ‘business as usual’. For the students, extra stress was added by making StudentPEP mandatory. The benefits measured consisted mainly of the written feedback and self-assessments, which ranged from remarkably good to quite disappointing (Braend et al. Citation2010). A third of the students thought that StudentPEP helped them to get better feedback in the clerkship, and they found the project worthwhile. Bowen and Irby report that many students in ambulatory settings have few consultations on their own (Bowen & Irby Citation2002). Some students in our pilot study had the same experience, and the StudentPEP project ensured them patients of their own (Braend et al. Citation2006). In this study, 20% of the students believed that StudentPEP helped them to get independent consultations (); 59% believed that StudentPEP ensured teacher observation, and many students found this aspect important.
Methodological aspects
The principal component analysis was used mainly to examine internal consistency of the project evaluation questionnaire. None of the subscales explained high levels of variance, and the subscales should therefore be interpreted with caution. We measured satisfaction with degrees of agreement from students and teachers, but the educational impact contains more than satisfaction. We also considered the quality of the feedback provided (Braend et al. Citation2010). Other outcomes are more elusive, as the goals with the StudentPEP project are broad and long-term. The quantitative data are supported and extended by the qualitative analysis, which strengthens our data material. The main findings regarding student satisfaction are to a large degree likely to be explained by the learning environment in each clerkship site or individual psychological characteristics, and these dimensions are difficult to capture.
Patient feedback
In our study, both teachers and students were interested in what patients think, as other studies have also indicated (Kvamme et al. Citation2000; Wensing et al. Citation2003). A widespread use of patient evaluations depends on them being considered feasible, legitimate and an advantage to doctors and their patients (Heje Citation2006). Elwyn et al. states that patients will increasingly play a part in assessing the quality of care, and that one of the purposes of collecting patient perspectives can be educational feedback (Elwyn et al. Citation2007). Previously, we have compared students’ scores on their own consultations with patients’ and teachers’ scores, and found significant differences. Patients and teachers gave approximately one point higher scores on a 5-point scale (Braend et al. Citation2010). Forty-four per cent of the students believed that the patients gave their honest opinion on the questionnaires, and this allowed some students to let the patient feedback increase their self-confidence. However, the effect of high patient scores seems to be two-directional; either the students adapt the positive feedback given from patients, or they discard it as unreliable.
Teachers’ feedback
Our subscale labelled ‘Teachers’ oral and written feedback’ had high levels of agreement. The qualitative material confirmed that the teachers’ feedback was valuable for the students, both the informal oral feedback, as well as the written one when the latter was concrete and constructive. A review on learning in clinical clerkships recommends written feedback notes, with ‘well-done’ and ‘improve’ markers (Dobbie & Tysinger Citation2005). The SOAP system for written feedback from teachers and for self-assessment provided useful written feedback on all four phases of the consultation. This one-size-fits-all method for feedback depends on the teachers’ mental checklist for both standard consultation behaviours and the specific clinical problem. This study shows that this method fits many of the teachers, but not all. It is possible that the new task of formulating feedback in writing made the teaching more challenging. It has not only been found that teachers lack self-confidence in making assessments and giving feedback, but also that the rapid pace of ambulatory clinics reduces the opportunities for teaching and learning (Bowen & Irby Citation2002; Hawthorne et al. Citation2006). Possibly, the StudentPEP project did not take the teachers’ busy work in general practice seriously enough. It takes time to give written feedback, and it might require another ‘state of mind’ to reflect on the students’ needs. Nevertheless, StudentPEP provided material for discussion and reflection, and some students showed gratitude towards the academic department for increasing the structure and demands on the clinical teachers.
The importance of how the teachers’ attitudes towards StudentPEP influenced the students’ attitudes was clearly demonstrated in the logistic regression analysis. This shows that the teachers must be well informed and motivated to carry out the StudentPEP project if the students are to benefit from it.
Student self-assessment
The ability to assess one's own performance is an explicit learning goal at the University of Oslo (Faculty of Medicine Citation1996). Studies have shown that the accuracy of students’ self-assessment of their own clinical performance is poor, but that it can be improved by receiving feedback (Srinivasan et al. Citation2007). Teachers in general practice have expressed that they felt positive about students identifying their weaknesses instead of hiding them during self-assessment sessions (Westberg & Jason Citation2001). Some students in our study found it useful to be stimulated to reflect on their own behaviour in the consultations, and a majority of the teachers found it useful to read the students’ written self-assessments.
Reflective discussions
Our project included a reflective discussion between students and teachers regarding the written material. On the question of whether the teachers and students actually discussed their collected feedback material, more students and teachers agreed than those who disagreed. If the students never summarized their quantitative data, StudentPEP results may have served like loose ends instead of being useful feedback. It has been found that facilitated reflection is helpful to assimilate and accept feedback given in multisource feedback sessions (Sargeant et al. Citation2007). We have not done a qualitative analysis of the content or techniques used to foster reflection in such discussions.
Conclusions
Our multi-method analysis of whether StudentPEP was successful in providing students more and better formative feedback and in increasing awareness on patient views by triangulation of feedback gave several conclusions. The patient evaluations have been shown either to increase the student's self-confidence, or to make students distrust patient opinions. Students and teachers have involved themselves in the patient perspective through the project, and many students report an increased awareness of patient expectations. The students have all been observed by their teachers, and many experienced that this was ensured by the StudentPEP project. The feedback from teachers was considered useful. Many students and teachers found the self-assessments useful. Some students appreciated the increased awareness of the three different perspectives in the triangulated evaluations. We believe that such clear frames for assessment and reflective discussions on the written material can meet the student's need for formative feedback, and that they may help them in their future clinical work.
Acknowledgements
Per Hjortdahl served as the scientific advisor. Aase Brinchmann-Hansen critically reviewed the manuscript. Magne Thoresen gave advice regarding statistical methods. This study was funded by Faculty of Medicine, University of Oslo and The Norwegian Medical Association's Foundation of Quality Assurance in Primary Healthcare.
Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.
References
- Baerheim A, Thesen J. Medical students’ evaluation of preceptorship in general practice in Vestlandet. Tidsskr Nor Laegeforen 2003; 123: 2271–2273
- Bienstock JL, Katz NT, Cox SM, Hueppchen N, Erickson S, Puscheck EE. To the point: Medical education reviews – providing feedback. Am J Obstet Gynecol 2007; 196: 508–513
- Bowen JL, Irby DM. Assessing quality and costs of education in the ambulatory setting: A review of the literature. Acad Med 2002; 77: 621–680
- Braend AM, Gran SF, Frich JC, Lindbaek M. Medical students’ clinical performance in general practice – triangulating assessments from patients, teachers and students. Med Teach 2010; 32(4)333–339
- Braend AM, Gran SF, Lindbaek M. Patients – a useful resource when evaluating medical students’ clinical practice?. Tidsskr Nor Laegeforen 2006; 126: 2122–2125
- Dobbie A, Tysinger JW. Evidence-based strategies that help office-based teachers give effective feedback. Fam Med 2005; 37: 617–619
- Elwyn G, Buetow S, Hibbard J, Wensing M. Measuring quality through performance. Respecting the subjective: Quality measurement from the patient's perspective. BMJ 2007; 335: 1021–1022
- Faculty of Medicine. 1996. Oslo-96: A new curriculum for the education of medical doctors, Oslo, University of Oslo.
- Gran SF, Braend AM, Lindbaek M. Clerkship in general practice – organized for learning?. Tidsskr Nor Laegeforen 2008; 128: 2443–2445
- Grol R, Wensing M, Mainz J, Jung HP, Ferreira P, Hearnshaw H, Hjortdahl P, Olesen F, Reis S, Ribacke M, Szecsenyi J. Patients in Europe evaluate general practice care: An international comparison. Br J Gen Pract 2000; 50: 882–887
- Hair JF, editor. Multivariate analysis: With readings, 4th ed. Prentice Hall, New York 1995
- Hawthorne K, Wood F, Hood K, Cannings-John R, Houston H. Learning to mark: A qualitative study of the experiences and concerns of medical markers. BMC Med Educ 2006; 6: 25
- Heje HN, 2006. Patient evaluation in general practice – methodological aspects, influence of patient and GP characteristics and the GPs’ experiences with the evaluation. Aarhus, University of Aarhus
- Jones R. Oxford textbook of primary medical care. Oxford University Press, Oxford 2004
- Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students’ clinical skills and behaviors in medical school. Acad Med 1999; 74: 842–849
- Kvamme OJ, Sandvik L, Hjortdahl P. Quality of general practice as experienced by patients. Tidsskr Nor Laegeforen 2000; 120: 2503–2506
- Malterud K. Qualitative research: Standards, challenges, and guidelines. Lancet 2001; 358: 483–488
- Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29: 855–871
- Rushton A. Formative assessment: A key to deep learning?. Med Teach 2005; 27: 509–513
- Sargeant J, Mann K, Sinclair D, Van Der Vleuten C, Metsemakers J. Challenges in multisource feedback: Intended and unintended outcomes. Med Educ 2007; 41: 583–591
- Srinivasan M, Hauer KE, Der-Martirosian C, Wilkes M, Gesundheit N. Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination. Med Educ 2007; 41: 857–865
- Wensing M, Vingerhoets E, Grol R. Feedback based on patient evaluations: A tool for quality improvement?. Patient Educ Couns 2003; 51: 149–153
- Westberg J, Jason H. Fostering reflection and providing feedback. Springer Publishing Company, New York 2001