Abstract
Background: Residents play a significant role in teaching, but formal training, feedback, and evaluation are needed.
Aims: Our aims were to assess resident teaching skills in the resident-as-teacher program, quantify correlations of faculty evaluations with resident self-evaluations, compare resident-as-teacher evaluations with clinical evaluations, and evaluate the resident-as-teacher program.
Method: The resident-as-teacher training program is a simulated, videotaped teaching encounter with a trained medical student and standardized teaching evaluation tool. Evaluations from the resident-as-teacher training program were compared to evaluations of resident teaching done by faculty, residents, and medical students from the clinical setting.
Results: Faculty evaluation of resident teaching skills in the resident-as-teacher program showed a mean total score of 4.5 ± 0.5 with statistically significant correlations between faculty assessment and resident self-evaluations (r = 0.47; p < 0.001). However, resident self-evaluation of teaching skill was lower than faculty evaluation (mean difference: 0.4; 95% CI 0.3–0.6). When compared to the clinical setting, resident-as-teacher evaluations were significantly correlated with faculty and resident evaluations, but not medical student evaluations. Evaluations from both the resident-as-teacher program and the clinical setting improved with duration of residency.
Conclusions: The resident-as-teacher program provides a method to train, give feedback, and evaluate resident teaching.
Introduction
Residents play a significant role in teaching medical students and fellowresidents and may eventually become faculty medical educators. Residents estimate that they spend up to 25% of their time teaching (Busari et al. Citation2002), and medical students attribute at least one-third of their knowledge to the teaching they receive from residents. (Bing-You & Sproul Citation1992). Most residents value their roles as teachers and desire to improve their teaching skills with formal resident-as-teacher programs. Such programs have been shown to improve residents’ self-confidence and self-assessed use of effective teaching behaviors and to improve their evaluations by students (Morrison & Haffler Citation2000, Hammoud et al. Citation2004, Rubak et al. Citation2008).
Formal resident-as-teacher curricula are becoming more common, but residency program directors continue to express the need for more types of training programs that improve teaching skills (Morrison et al. Citation2001). Resident-as-teacher programs described in the medical literature have included special electives for interested residents (Weissman et al. Citation2006), clinician educator tracks (Heflin et al. Citation2009), and retreats focused on teaching skills (Litzelman et al. Citation1994, Roberts et al. Citation1994). Teaching modalities include lectures, small-group discussions, case history teaching formats, role-plays, simulations, debriefing sessions, and the reviewing of films and videotapes of teaching performance. Some programs include instructor feedback on resident teaching performance (Jewett et al. Citation1982, Spickard et al. Citation1996).
The Accreditation Council for Graduate Medical Education (ACGME) includes resident teaching skills in several competencies (ACGME, Citation2007). For example, the Practice-based Learning and Improvement competency includes “facilitate the learning of others,” and the Interpersonal and Communication Skills competency includes “listening skills, respectful.” Standards for postgraduate medical education require that residency programs integrate modern educational methods into training, for both teaching skills and clinical skills. Evidence points toward improved resident learning and retention when the curriculum includes educational processing that comes after clinical experiences. By adding a formal interactive process of interpretation, construction of meaning, and reflection, there is growth and transformation, which serves to consolidate resident knowledge and training (Teunissen et al. Citation2007). Such curricula that enhance resident learning may also be used to enhance resident teaching skills in the same manner as clinical skills.
Self-assessment is increasingly a part of the process of physician lifelong learning. However, self-assessment has limitations, since achieving competency in both clinical and teaching skills is enhanced by guided feedback from experts that identifies areas for expanding knowledge and improving methods (Eva & Regehr Citation2005). Formal training, interpretation, reflection, feedback, and assessment can be used to follow simulated clinical teaching moments in order to help residents become better teachers. Such formal programs are lacking in most residency training programs.
Our specific aims were to (1) assess resident teaching skills with the resident-as-teacher program, (2) quantify the correlation of faculty evaluations with resident self-evaluations, (3) compare the resident-as-teacher assessment with those from the traditional clinical observational model, and (4) evaluate resident satisfaction and perceived effectiveness of the resident-as-teacher program. To our knowledge, this is the first resident-as-teacher program in the medical literature to use simulated teaching scenarios as a both a training method and a formal assessment method for resident teaching skills.
Methods
Description of the resident-as-teacher program
The resident-as-teacher training program in the Obstetrics and Gynecology Residency Program at Beth Israel Deaconess Medical Center, Harvard Medical School, is an annual simulated, videotaped teaching encounter with a trained medical student. There are five residents per year in the residency program (20 residents total), and residents are introduced to the program during first-year orientation. They are taught the principles of asking open-ended questions, use of body language, timing of questioning, and assessing the learner's background, and they are given an opportunity to practice teaching a trained medical student. During orientation they are introduced to the resident-as-teacher evaluation tool () and they conduct one resident-as-teacher exercise with a trained medical student. Although the residents view their performances on video, the orientation exercise has no evaluative component, as it is designed as an introduction to the program. Each resident then participates in the resident-as-teacher training program annually in each of the four years of residency. The program is meant to be instructional, as well as evaluative, and participation is mandatory.
Figure 1. Resident-as Teacher Evaluation Tool.
The protocol for the resident-as-teacher program is illustrated in . Medical students are trained on a teaching case, which they present to the resident as if they had just seen and evaluated the patient in an urgent care setting. Case topics are based upon core topics from the learning objectives for medical students written by the Association of Professors of Gynecology and Obstetrics, and the same topic is used for all residents in each academic year. During the three years of this study, the case topics included mastitis, vaginitis, and ectopic pregnancy.
Figure 2. Resident-as-Teacher Annual Protocol.
The resident is instructed to teach the student as they typically would in the clinical setting and to utilize their own style to teach the student the principles illustrated in the case. The encounter is allowed to last seven minutes and is videotaped. One obstetrics and gynecology faculty member observes each encounter from an observation booth with a one-way mirror and evaluates each resident using the evaluation tool.
The evaluation tool was created to focus on competencies the residents, students, and faculty educators had previously identified as areas for improvement when teaching in the clinical setting. There are several validated instruments (Irby 1987; Litzelman et al. Citation1998) used to measure resident teaching effectiveness, but the detailed nature of these scales and the diverse areas of teaching skills they cover made them impractical for the resident-as-teacher training program. The teaching evaluation tool utilizes a Likert scale (1 = poor to 5 = excellent) on the following six teaching skills: (1) creating a respectful, friendly, open climate; (2) eliciting learner's background knowledge or experience; (3) using body language that encourages openness, such as eye contact and leaning toward the learner; (4) using open-ended questions; (5) eliciting learner's thoughts before contributing teaching points; and (6) factual correctness. A total score is calculated by averaging these categories.
Immediately after the teaching encounter, the resident watches his/her video in private and completes a self-evaluation of his/her skills. In addition to the resident-as-teacher training program, two resident didactic lectures per year focus on principles of effective teaching and one annual departmental Grand Rounds topics focuses on medical education.
Evaluation of the resident-as-teacher program
In order to determine whether there was inter-observer variability in faculty evaluation within the resident-as-teacher program using the evaluation tool, a second faculty member who was not present at the encounter independently evaluated the videotapes from the first year of the program. Because the two faculty members’ perceptions of resident skills were highly correlated (r = 0.78; p = 0.04), a single faculty evaluator was used for all evaluations in the three years of the study. We used the scores from the one faculty member who participated in all of the resident-as-teacher assessments to determine how strongly the faculty evaluation correlated with the resident self-evaluation using the same evaluation tool.
Finally, we sought to compare the evaluative component of the resident-as-teacher training program faculty scores to the traditional clinical observational model of evaluation used in the clinical setting, which is done for each rotation by faculty, residents, and medical students as an assessment of each resident's teaching skills to see if they were similar. Attendings, fellow residents, and medical students evaluated resident teaching in the clinical setting using anonymous online surveys. Each resident was evaluated over the course of an academic year on various rotations by a range of 2–16 attendings, 4–15 residents, and 2–16 medical students. The teaching evaluations in the clinical setting use the same Likert scale as the resident-as-teacher program, and evaluators are asked to “rate the resident as teacher.” These evaluations were compared with the total score from the faculty evaluations in the resident-as-teacher training program.
In order to assess the residents’ perceived effectiveness of and satisfaction with the resident-as-teacher program, we conducted surveys of residents immediately following completion of each annual session. Residents used a Likert scale (1 = not at all helpful to 5 = extremely helpful) to rate the following aspects of the program in improving their teaching: (1) the resident-as-teacher program overall; (2) the videotaping and self-review; (3) receiving faculty feedback; and (4) recognizing the importance of teaching by having the program. They also evaluated their confidence in teaching (1 = not confident to 5 = extremely confident) in the following settings: (1) one-on-one teaching; (2) teaching in small groups; (3) giving a lecture; and (4) giving feedback.
Data analysis
All statistical tests were performed using SAS 9.2 (SAS institute Inc., Cary, NC). All tests were two sided, and p values <0.05 were considered statistically significant. Data are presented as mean with standard deviation (SD) or mean difference with 95% confidence interval (CI). Comparisons were made using the t tests for continuous variables. Spearman correlations were calculated to compare evaluation methods.
Mean faculty scores were calculated for each teaching skill and were compared with resident self-assessment. Correlations between the two faculty scores and between the faculty score and the self-evaluation were calculated.
In order to determine whether the faculty assessments of resident teaching skills in the simulated setting were comparable to assessments made in the clinical setting, all evaluations of resident teaching done by attendings, fellow residents, and medical students in the clinical setting were compared to faculty assessments from the resident-as-teacher program. In order to determine whether residents’ teaching improved over time as assessed by the resident-as-teacher evaluations as well as by evaluations in the clinical setting, trends in resident scores over four academic years were analyzed.
Results
Thirty residents participated in the three years studied (academic years 2007–2009), with each resident completing between one and three resident-as-teacher training sessions, for a total of 53 evaluations. As shown in , faculty evaluation of resident teaching skills shows a mean total score in all areas of 4.5 ± 0.5. The highest scores were given for use of body language that encourages openness, while the lowest scores were given for ability to elicit learners’ background knowledge. Resident self-perception of teaching skill was lower than faculty perception overall (mean difference: 0.4; 95% CI 0.3–0.6), and for all specific skills, especially in the area of factual correctness. There were moderately strong and statistically significant correlations between self-perception and faculty perception overall (r = 0.47; p < 0.001), and for all specific skills with the exception of creating a respectful, friendly, and open climate.
Table 1. Resident-as-teacher evaluation of teaching skills.
As shown in , resident-as-teacher evaluations were significantly correlated with faculty and resident evaluations in the clinical setting, while medical student evaluations were not. As shown in , teaching evaluation scores from the resident-as-teacher program and the clinical setting generally increased with duration of residency, with the exception of medical student evaluations, which declined.
Figure 3. Teaching Evaluations by Post Graduate Year displays the overall mean teaching evaluation scores by postgraduate year. The stars with the dotted line represent scores given by medical students; the closed circles with the dashed line represent scores given by residents; the closed triangles with the dotted line represent scores given by the attendings; the open squares represent scores given by the faculty in the RAT program; the closed squares represent resident self-evaluation scores in the RAT program.
Table 2. Correlations of resident-as-teacher evaluations with clinical setting evaluations.
In order to assess the effectiveness of the program in enhancing the skills and satisfaction of the residents, we administered a survey to evaluate resident perceptions of the program (N = 30). As shown in , residents felt the program in general was “very helpful” (4.1 ± 0.7). Residents rated receiving immediate feedback on their teaching strategies and recognizing the importance of teaching by having the program equally as the most helpful components of the program. Resident confidence in different teaching settings ranged from “somewhat confident” in lectures (2.8 ± 1.0) and small group settings (3.2 ± 0.8) to “very confident” with one-on-one encounters (3.8 ± 0.8).
Table 3. Resident perceptions of resident-as-teacher program.
Conclusions
This study describes a resident-as-teacher training program that utilizes videotaped, simulated teaching encounters using a standardized assessment tool, along with immediate faculty assessment and feedback, as a method for residents to practice teaching skills, as well as a structured and reproducible method to evaluate and provide feedback to residents on their teaching skills.
The faculty evaluations and resident self-evaluations from the resident-as-teacher program were moderately and significantly correlated. However, in nearly all categories, the residents’ assessments of their teaching skills were slightly lower than faculty assessments. The two faculty evaluations of resident teaching skills in the resident-as-teacher program were highly correlated, suggesting the methodology is reproducible among different faculty evaluators. Faculty evaluations of resident skills are highest in the areas of factual correctness and lowest in the area of eliciting the learner's background. Resident self-evaluations are lowest in the ability to elicit learner's thoughts before interrupting and highest in setting a positive teaching climate.
The literature on physician self-assessment versus observed measurements of competence is mixed. A systematic review of the literature comparing self and external assessment found that 13 studies demonstrated little, no, or an inverse relationship, while 7 studies demonstrated positive associations. The lowest accuracy in self-assessment is among physicians who are the least skilled and yet the most confident (Davis et al. Citation2006). Our findings are consistent with a meta-analysis of self-assessment versus external assessment, where correlations ranged from 0.05 to 0.82 with a mean of 0.39 (Falchikov & Boud Citation1989). Along with this meta-analysis, our correlation of 0.47 suggests that self-assessment of teaching abilities is only moderately correlated with observed measures from the resident-as-teacher program. Thus, self-assessment of teaching skills needs to be used cautiously.
The excellent correlation between resident-as-teacher assessments and traditional clinical assessments suggests that this methodology can be used reliably, perhaps in addition to observations in the clinical setting. It may be a more standardized and reproducible method of evaluation of teaching, since the resident-as-teacher scenarios are standardized and the assessment tool shows good correlation among evaluators, as well as with resident self-assessment. The assessment tool has the potential to be used across other disciplines in all types of residency programs. In addition to evaluation in a more consistent and reproducible fashion, this simulated resident-as-teacher training program also serves as a method for residents to practice and enhance their teaching skills. Finally, the resident-as-teacher program serves as a venue for self-reflection and self-evaluation, which are life-long learning skills.
Additional study is needed to examine how resident teaching skills improve over time with the resident-as-teacher program, and whether this program has a measurable effect on the experience of medical students. It is not clear why the medical student evaluations of resident teaching in the clinical setting decrease as residents become more experienced. Studies of medical student evaluations of teachers are inconsistent in the literature, with some studies showing that student evaluations of teachers are consistently lower than resident or peer evaluators (Irby et al. Citation1987), while others show that scores from medical students are higher than residents or peers (Williams 2001, Kommalage 2011). Others show they are well correlated with resident and faculty evaluations (Zabar et al. Citation2004), and still others show no correlation between medical student and academic staff perceptions (Haghdoost & Shakibi Citation2006). Teachers who are extensively involved with trainees have been rated significantly higher in overall teacher effectiveness than those who are only moderately or slightly involved (Irby et al. Citation1987). It is possible that junior residents who are closer in training level to medical students are more extensively involved with students, share more camaraderie, or perhaps teach an approach to management that is valued by medical students, and this accounts for the higher ratings by medical students of more junior residents compared to more senior residents. More study is needed to understand this unexpected trend. What is clear is that improved resident teaching skills should result in increased learning by medical students and may lead to improvements in medical student evaluations of their clerkship experiences and their residents as teachers. This methodology may be applied to fulfill and measure several important ACGME competencies related to teaching, including practice-based learning and improvement and interpersonal/communication skills. These competencies are difficult to objectively observe and measure in the clinical setting due to inter-observer variability and the difficulties of comparing varied clinical scenarios.
Acknowledgments
The authors wish to thank L. Renata Vicari, Beth Israel Deaconess Medical Center Harvard Medical School OBGYN medical student clerkship coordinator, for her support with setting up the resident-as-teacher simulations, data collection, and personal assistance.
Declaration of interest: The authors report no declarations of interest. The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.
References
- ACGME. 2007. The common program requirements document: http://www.acgme.org/outcome/comp/GeneralCompetenciesStandards21307.pdf.
- Bing-You RG, Sproul MS. Medical students’ perceptions of themselves and residents as teachers. Med Teach 1992; 14(2/3)133–138
- Busari JO, Prince KJ, Scherpbier AJ, Van Der Vleuten CP, Essed GG. How residents perceive their teaching role in the clinical setting: A qualitative study. Med Teach 2002; 24: 57–61
- Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence. A systematic review. JAMA 2006; 296: 1094–1102
- Eva KW, Regehr G. Self-assessment in the health professions: A reformulation and research agenda. Acad Med 2005; 80: S46
- Falchikov N, Boud D. Student self-assessment in higher education: A meta-analysis. Rev Ed Res 1989; 59: 395–430
- Haghdoost AA, Shakibi MR. Medical student and academic staff perceptions of role models: An analystical cross-sectional study. BMC Med Educ. 2006; 6: 1–6
- Hammoud MM, Haefner HK, Schigelone A, Gruppen LD. Teaching residents how to teach improves quality of clerkship. Am J Obstet Gynecol 2004; 191: 1741–1745
- Heflin MT, Pinheiro S, Kaminetzky CP, McNeill D. ‘So you want to be a clinician-educator … ’: Designing a clinician-educator curriculum for internal medicine residents. Med Teach 2009; Jun;31(6)e233–e240
- Irby DM, Gillmore GM, Ramsey PG. Factors affecting ratings of clinial teachers by medical students and residents. J Med Educ 1987; 62: 1–7
- Jewett LS, Greenberg LW, Goldberg RM. Teaching residents how to teach: A 1-year study. J Med Educ 1982; 57: 361–366
- Kommalage M, Gunawardena. Evaluation of physiology lectures conducted by students: Comparison evaluation by staff and students. Adv Physiol Educ 2011; 35: 48–52
- Litzelman DK, Stratos GA, Skeff KM. The effect of a clinical teaching retreat on residents’ teaching skills. Acad Med 1994; 69: 433–4, Med Educ 1980;55:1000–1005. 43
- Litzelman DK, Stratos GA, Marriot DJ, Skeff L. Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Acad Med 1998; 73: 688–695
- Morrison EH, Hafler JP. Yesterday a learner, today a teacher too: Residents as teachers 2000. Pediatr. 2000; Jan;105(1 Pt 3)238–241
- Morrison EH, Friedland JA, Boker J, Rucker L, Hollingshead J, Murata P. Residents-as-teachers training in U.S. residency programs and offices of graduate medical education. Acad Med 2001; Oct;76(10 Suppl)S1–S4
- Roberts KB, DeWitt TG, Goldberg RL, Scheiner AP. A program to develop residents as teachers. Arch Pediatr Adolesc Med 1994; 148: 405–410
- Rubak S, Mortensen L, Ringsted C, Malling B. A controlled study of the short- and long-term effects of a train-the-trainers course. Med Educ 2008; 42: 693–702
- Spickard A, Corbett EC, Schorling JB. Improving residents’ teaching skills and attitudes toward teaching. J Gen Intern Med 1996; 11: 475–80, Learn Med 1995;7:111–114
- Teunissen PW, Scheele F, Scherpbier AJ, Van Der Vleuten CPM, Boor K, Van Luijk SJ, Van Diemen-Steenvorrde JA. How residents learn: Qualitative evidence for the pivotal role of clinical activities. Med Educ 2007; 41: 763–770
- Weissman MA, Bensinger L, Koestler JL. Resident as teacher: Educating the educators. Mount Sinai J Med 2006; 73: 1165–1169
- Williams BC, Pillsbury MS, Stern DT, Grum CM. Comparison of resident and medical student evaluation of faculty teaching. Eval Health Prof 2001; 24: 53–60
- Zabar S, Hanley K, Stevens DL, Kalet A, Schwartz MD, Pearlman E, Brenner J, Kachur EK, Liplin M. Measuring the competence of residents as teachers. J Gen Intern Med. 2004; 19: 530–533