A randomised controlled study of role play in a faculty development programme

Abstract

Background: The Stanford Faculty Development Center at Stanford University has developed a teaching improvement course for medical teachers that has been widely disseminated using a train-the-trainer model.

Aims: This study was designed to investigate the relative impact of role playing as an instructional technique within that course for facilitating change in teaching behaviours.

Method: From January 2009 to April 2010, six faculty development courses were delivered at Uppsala University Hospital to 48 physicians from different departments. The standard course presentation includes a range of instructional methods including short lectures, small group discussion, review of video re-enactments, role-play exercises and personal goal setting. For this study, participants were randomised to participate in (1) a ‘standard’ course with role play or (2) an ‘alternative’ course with no role play. The effects of the course on teaching performance were assessed with retrospective pre- and post-course self-ratings of 29 specific teaching behaviours.

Results: Self-assessment ratings indicated significantly greater positive changes in teaching behaviour among faculty who attended the standard course (with role play) as compared to those in the alternative course (p = 0.015).

Conclusions: This study validates the commonly held view that role play is a useful instructional method for improving teaching.

Introduction

The Stanford Faculty Development Center (SFDC) for Medical Teachers at Stanford University has widely disseminated a teaching improvement course using a train-the-trainer model. Evaluation of the course, as delivered by trained facilitators to colleagues at their own institutions across the United States, has revealed positive effects on knowledge, skills and attitudes related to teaching (Skeff et al. 1992a). Also implemented internationally, it was found to be highly transportable to medical teachers in Sweden, producing positive results consistent with those found in the United States (Johansson et al. 2009).

An extensive review article on faculty development initiatives to improve teaching effectiveness in medical education concluded that one key feature of effective programmes is the inclusion of experiential learning activities, such as role play (Steinert et al. 2006). Also noted was the lack of research on what instructional components of faculty development interventions are most useful. The authors proposed that future research could compare different faculty development methods to discover which features of faculty development contribute to changes in teacher performance. One experiential method used in the SFDC's Clinical Teaching course is video-recorded role play in which course participants act as teachers, medical students and trainees of different educational levels. The role play is designed for course participants to demonstrate teaching behaviours, and then, through a video debriefing, analyse the effectiveness of their teaching behaviours and identify alternative future approaches.

Role play is generally acknowledged to be an effective teaching tool for skills development. However, it may not be incorporated in faculty development programmes for a variety of reasons, including that it is relatively time-consuming and faculty developers or participants may find role playing to be uncomfortable or are sceptical about its usefulness (Milroy 1982; Nestel & Tierney 2007). This randomised-controlled study was undertaken to investigate the additive impact of role play on medical teachers participating in the SFDC's course. We hypothesised that participation in a version of the course that included role play would result in greater self-reported improvement in teaching behaviours as compared to participation in a course format where role play was excluded. We based this hypothesis on the multiple potential benefits of a guided, video-recorded role-play review, including the usefulness of a facilitator to enhance the comfort of self-review, the power of video recording the role play for analysis of actual observed behaviours, the opportunity for guided application and practice of previously learned content, and discovery of the positive effects of changing one's teaching behaviours. These predicted benefits were based on Donald Schon's conceptual framework for reflective practice (Schon 1983) and by Bandura et al.'s (1974) work on behaviour and attitude change through participant modelling and self-directed practice.

Table

Practice points

The 14-hour Stanford faculty development programme resulted in positive changes in teaching behaviours as assessed by participants’ self-ratings. Participation in a course with role play led to higher ratings of the course's usefulness and to greater teaching improvement as compared to participation in the course without the role play. When designing courses for medical teachers, faculty developers should place role plays on their list of instructional methods.

Methods

Description of the SFDC's clinical teaching programme

The SFDC's Clinical Teaching course is thoroughly described elsewhere (Skeff et al. 1992a). Its transportability to other cultures has also previously been described (Wong & Agisheva 2004; Johansson et al. 2009). The general goals of the course are to (1) enhance teaching versatility and (2) foster participants’ ability to analyse and improve teaching using a 7-component educational framework that includes learning climate, control of session, communication of goals, promotion of understanding and retention, evaluation, feedback and promotion of self-directed learning. The course is delivered in seven 2-hour sessions, each addressing one educational category. The ‘standard’ presentation of the 2-hour sessions includes a brief ‘mini-lecture’ to enhance participants’ knowledge of the details of the category, discussions of pre-recorded video re-enactments of actual clinical teaching interactions to improve participants’ abilities to analyse teaching using the categorical framework, role-play exercises with video debriefing for participants to identify and practise desired teaching behaviours, and personal goal setting for future teaching.

In the role play, one of the participants acts the teacher's role while one to three others are given the roles of students, interns or residents. The teacher's only instructions are to interact in a manner that will increase the effectiveness of the specific educational category emphasised in that session. Other role-play participants receive specific written instructions outlining the characteristics they should manifest in their role. Remaining participants are observers and do not interact in the role play. After a short orientation by the facilitator, a 3-minute role play is enacted and video-recorded, followed by a facilitated debrief of the video focusing on the teacher's goals, the effects of teaching behaviours used, and the identification of alternative approaches with feedback from participants, observers and facilitator. A second role play is then conducted in which the teacher has the chance to implement alternative teaching behaviours to enhance teaching effectiveness derived from the first role-play debriefing. At the end of each session, all participants formulate personal goals for improving teaching behaviours related to the educational category. Subsequent sessions start with a discussion of participants’ experiences implementing their personal teaching goals. To stimulate further learning, optional readings related to the educational category are provided at the end of a session.

Intervention

From January 2009 to April 2010, six courses were conducted at Uppsala University Hospital, Sweden. Two courses, ‘standard’ and ‘alternative’, were given concurrently over a 7-week period and held directly after each other on Friday afternoons. The standard format conformed to the traditional SFDC course that included role play. At each standard session, one participant was randomly selected to play the teacher's role in the role play, but each participant was limited to playing the teacher's role in only one session of the entire course. The ‘alternative’ course format was identical to the standard course (mini-lecture, review of pre-recorded video vignettes and personal goal setting) with the exception that the role play was substituted by a 1-hour scheduled activity of independent reading related to the educational categories. Group assignment was performed by sealed envelope randomisation. The course was conducted in Swedish, but all handouts, readings and video re-enactments of clinical teaching were presented in the original English.

Participants

Course participants were recruited from several departments within the hospital through advertisement on the hospital's homepage, mail invitations and announcements at institutional and departmental meetings. Each semester, between 24 and 37 doctors applied to become one of the eight participants for each of the two courses to be offered. Sixteen doctors were randomised to either the standard group or the alternative group prior to the course. They were contacted approximately 3 weeks prior to the course, but were not informed of their group assignments. Informed consent was collected at the first session. Participation in at least four out of the seven sessions was considered mandatory for course completion.

Facilitator

A Swedish anaesthesiologist (JJ) at Uppsala University Hospital (Department of Surgical Sciences), Sweden, was trained in October 2004 at the 1-month SFDC facilitator-training programme at Stanford University to become a facilitator in clinical teaching. The facilitator had no other faculty development experience. Prior to conducting the courses for this study, the facilitator had delivered eight standard SFDC courses at Uppsala University Hospital during 2005–2008.

Evaluation of programme effectiveness

Using a post-training questionnaire, we collected two types of evaluation data 2–4 weeks after the seminars: participants’ ratings of (1) the usefulness of the course; and (2) the effects of the course on their teaching performance. To assess the course's usefulness, after the course participants rated both the anticipated (looking back prior to attending the course) and actual usefulness of the course, using a 5-point scale from definitely not useful to definitely useful. To assess effects on participants’ teaching behaviour, participants completed retrospective pre- and post-intervention ratings of 29 different teaching behaviours related to the seven educational categories covered in the course. These self-report ratings of teaching performance were completed between 2 and 4 weeks following the course (post-intervention). Participants simultaneously rated both their current teaching performance and their pre-course performance, using a 5-point scale (1 = strongly disagree, 5 = strongly agree). The retrospective pre- and post-intervention design was used because we believe this comparison provides a more sensitive and valid measure of the changes associated with this type of training than the traditional pre- and post-intervention comparison (Skeff et al. 1992b).

Data analysis

The primary outcome measure was the change in the aggregated means of participants’ retrospective pre- and post-course self-ratings. A sample size calculation suggested that 40 participants were needed to achieve 80% power for detecting a 7-point difference in total sum (i.e. one point difference/category) between the groups. Rating data were analysed using SPSS; p-values of <0.05 were considered statistically significant. Also, scores for each of the seven educational categories were calculated by averaging ratings across clusters of items related to each category. Two-tailed, paired t-tests were used to compare participants’ retrospective pre- and post-course mean ratings to assess within group changes across the seven different educational categories. To compare the two treatment groups, two-tailed, unpaired t-tests were calculated, using the difference scores from the retrospective pre- and post-course mean ratings of all 29 items.

Results

Forty-eight faculty members participated in this study. Participants’ characteristics are depicted in Table 1. The two treatment groups were similar on all characteristics except gender. The role-play group was relatively balanced in terms of gender, whereas the no role-play group had a higher proportion of male faculty (67%). Participants represented a broad spectrum of surgical and non-surgical departments. Seventeen (35%) had previously participated in faculty development activities. The vast majority of respondents (96%) indicated voluntary participation, whereas 4% indicated required course attendance. All participants completed the course, and all completed the post-course questionnaire.

Table 1.  Demographic data for standard and alternative course participants

	Standard (n = 24)	Alternative (n = 24)
Age (mean, SD)	45.2 (9.2)	45.4 (9.2)
Male (%)	46	67
Female (%)	54	33
Faculty (n)	20	17
Residents (n)	4	7
Years of teaching experience (mean, SD)	12.7 (7.0)	9.5 (7.4)
Course participation (mean, %)	5.8 (83%)	5.8 (82%)

Notes: SD, standard deviation.

Course participation is based on seven possible sessions.

Evaluation of seminar impact on teaching behaviour

Significant retrospective pre/post differences were found for all seven educational categories for both treatment groups as well as for aggregate mean self-ratings of the 29 items measuring teaching behaviours (p < 0.001) (Table 2). These differences reflected changes in a positive direction, that is, towards improved teaching performance.

Table 2.  Participants’ before vs. after course mean self-ratings of teaching performance by educational category

	Standard (n = 24)		Alternative (n = 24)
	Before M (SD)^a	After M (SD)	Before M (SD)	After M (SD)
Learning climate	3.36 (0.74)	4.19 (0.47)	3.72 (0.66)	4.29 (0.47)
Control of session	3.13 (0.84)	3.82 (0.76)	3.17 (0.73)	3.76 (0.63)
Communication of goals	2.30 (0.77)	3.79 (0.72)	2.47 (0.85)	3.66 (0.63)
Promotion of understanding and retention	3.25 (0.73)	3.77 (0.67)	3.34 (0.66)	3.74 (0.58)
Evaluation	2.69 (0.72)	3.70 (0.65)	2.83 (0.90)	3.58 (0.66)
Feedback	2.61 (0.66)	3.54 (0.70)	2.92 (0.87)	3.55 (0.74)
Promotion of self-directed learning	2.68 (0.86)	3.74 (0.93)	3.01 (0.91)	3.85 (0.71)

Notes: M = mean, SD = standard deviation; Scale: 1 = strongly disagree, 5 = strongly agree.

‘Before’ indicates retrospective pre-rating made post-course.

^aAll within group retrospective pre/post comparisons were statistically significant at p < 0.001, based on two-tailed paired t-test.

Comparison of retrospective pre/post difference scores for aggregate mean self-ratings between treatment groups significantly revealed greater improvements in teaching performance among the standard (role-play) group participants (retrospective-pre M = 2.86, SD = 0.61; post M = 3.79, SD = 0.58) as compared to participants in the alternative course (retrospective-pre M = 3.07, SD = 0.63); post M = 3.78, SD = 0.47; p = 0.015) (Table 2).

Evaluation of course usefulness

Following the course, the role-play group participants gave higher ratings to the course's usefulness (M = 4.92, SD = 0.28) than those in the no-role-play group (M = 4.50, SD = 0.51; Scale: 1 = definitely not, 5 = definitely yes; p < 0.001). Participants in both treatment groups perceived the course to be significantly more useful after participating than they had anticipated it would be (role-play group: anticipated usefulness M = 3.79, SD = 0.66 vs. actual usefulness M = 4.92, SD = 0.28; p < 0.001; no role-play group: anticipated usefulness M = 4.04, SD = 0.96 vs. actual usefulness M = 4.50, SD = 0.51; p = 0.024). However, the difference between anticipated and actual usefulness ratings was greater among role-play than no role-play group participants (p = 0.007). Participants in the role-play group indicated they would recommend the course to their colleagues (M = 4.96, SD = 0.20; Scale: 1 = definitely not, 5 = definitely yes) more highly than the no role-play group (M = 4.54, SD = 0.59; p = 0.002).

Participants’ ratings of the value of various educational methods are presented in Table 3. All instructional methods, except ‘Reading educational articles’ for the alternative group, were given higher ratings after the course by participants in both treatment groups. Regarding role plays, the role-play group indicated a greater increase in the value placed on them after the training. With respect to reading educational articles, the role-play group ratings indicated increased value, while the alternative group ratings decreased post-training.

Table 3.  Standard vs. alternative group participants’ mean ratings of usefulness of instructional methods for teaching improvement

	Standard (n = 24)		Alternative (n = 24)
	Before M (SD)^a	After M (SD)	Before M (SD)	After M (SD)	p*
Mini-lecture	3.54 (0.93)	4.21 (1.02)	3.67 (0.96)	4.33 (0.76)	1.00
Review of video vignettes	3.38 (1.06)	4.29 (0.91)	3.42 (0.93)	4.21 (0.83)	0.64
Role-play exercise	3.13 (1.33)	4.33 (1.17)	3.63 (1.17)	4.00 (1.22)	0.005
Setting personal goals for teaching behaviours	3.08 (1.18)	4.58 (0.65)	3.25 (0.90)	4.58 (0.58)	0.56
Reading medical education articles	2.88 (1.26)	3.63 (1.06)	2.96 (1.20)	2.88 (1.33)	0.006

Notes: M = mean, SD = standard deviation; Scale: 1 = low and 5 = high.

‘Before’ indicates retrospective pre-rating made post-course.

^ap-value for two-tailed, unpaired t-test calculated on group difference scores (after – before).

Discussion

The study results indicate that both variants of SFDC's course were highly useful and led to significant changes in self-reported teaching behaviours. However, participant ratings indicated greater teaching improvements with role play as compared to the course without role play, indicating the benefit of role play.

Steinert's review of faculty development research concluded that although it is laden with methodological limitations, key features characterising effective programmes can be identified (Steinert et al. 2006). These include experiential learning, feedback, collegial relationships and the use of diverse educational methods within single interventions that subscribe to principles of teaching and learning. These key features are incorporated in the SFDC course and may be relevant to its success (Skeff et al. 1992a, 1999; Stratos et al. 1997; Berbano et al. 2006; Johansson et al. 2009). The SFDC's programme includes several experiential components, including opportunities for guided practise with feedback. Role play has for decades been considered a cornerstone of the SFDC faculty development programme. However, its unique importance and additive impact had not been assessed previously. Steinert's review concluded that more rigorous research designs, possibly controlled trials, were warranted to identify features of faculty development programmes that contribute to changes in teacher performance. Accordingly, this study examined role play's effectiveness as a method for influencing the acquisition of teaching skills.

We believe that this is the first randomly assigned, experimentally designed study to document the impact of role play within faculty development programmes. Role play has been used for adult learning in medical education over several decades, and evaluated predominantly in settings of communication skills training with medical students rather than experienced physicians (Lane & Rollnick 2007). Its impact has not been investigated in terms of added value in multi-method faculty development programmes. Given the particular structure adopted for the SFDC role plays, this study supports the general usefulness of role play and provides evidence that corroborates published guidelines for effective role plays (Shaw et al. 1980; Simpson 1985; Steinert 1993) that suggest including a strict, few-minute time limit; the opportunity to perform more than one skill practice within the same session and a structured review of the role play with specific feedback.

The results may help faculty developers overcome sources of resistance to using role plays. Participation in role plays can certainly be stressful. Doubts about their helpfulness are not uncommon and practitioners may feel resistance to conduct role plays due to claims of artificiality and embarrassment of being observed by peers (Nestel & Tierney 2007). Video recording may possibly further increase these feelings. However, if role-play training sessions are carefully designed and facilitators well-trained, these challenges to role play participation can be addressed. Participants who experienced role playing rated it higher after the experience, suggesting that others who are more sceptical may similarly discover the method's value. Of note, participants who were given time to read education articles rated readings lower after the experience. This finding may indicate that certain educational experiences, such as readings, may require participants to be ‘primed’ with new knowledge and experience from an active experiential exercise such as role play as a prior motivating experience.

The use of retrospective pre-ratings revealed interesting findings. Lower retrospective pre-ratings appear to be a major reason for revealing significant difference between groups, suggesting the utility of using a retrospective pre/post comparison to detect change. It is possible that experience with self-examination during the role play inclined standard group participants to be more self-critical when looking back at their pre-intervention teaching performance.

Further strengths of this study are the 100% course completion and participant response rates. Additionally, the potential for generalisation of these findings to a wide spectrum of medical educators is supported by (1) the randomised selection of course applicants, (2) the random assignment of participants to treatment groups and (3) the broad cross-departmental representation of course participants.

This study has several limitations. First, the effectiveness data rely only on self-report and lack objective ratings of change in teaching behaviours or learning outcomes. However, earlier studies of the SFDC's teaching improvement course have demonstrated a consistency between teachers’ post course self-assessment and students’ ratings (Skeff et al. 1992a, 1992b), supporting the validity of these self-reports. When viewed through the lens of Kirkpatrick's (1994) commonly used training evaluation model, this study design precludes any conclusions about whether the course affects institutional or student performance levels (Kirkpatrick's level 4, ‘results’).

Second, the self-assessment in this study was performed relatively shortly after the course when participants may have been especially enthusiastic about the impact of the programme. Thus, the long-term effects on teaching behaviour among participants were not assessed. Although prior studies have demonstrated that participation in the SFDC course can result in sustained changes (Stratos et al. 1997; Wong et al. 2004), no conclusions can be drawn regarding if role plays will lead to substantial and long-term changes in participants’ teaching performance.

Third, the evaluation of role play was performed within a specific faculty development programme in which facilitators receive intensive training to debrief role plays. Thus, we cannot be certain about the generalisability to other programmes.

Fourth, participants were randomly selected from a cohort who voluntarily applied for a course on clinical teaching. Consequently, physicians who applied were most likely interested in teaching. Therefore, it is possible that they were more motivated than the average physician-teacher. However, one could also argue that their teaching abilities prior to attending the course were relatively high and that the potential beneficial effects of a teaching programme and a specific instructional method, like role play, could be even higher for teachers with less enthusiasm and skills in clinical teaching.

Fifth, in this study only two alternative formats of a faculty development programme were investigated and role play was not compared to another highly interactional group-learning activity. Although both groups had sessions of 2 h duration, their time together with the facilitator differed as did the type of learning activity. While the role-play group received 1 h of experiential learning via role play, the alternative group had 1 h of readings and possible discussion with the other participants, without the facilitator.

The effectiveness and satisfaction ratings of the role-play group may reflect the particular combination of educational approaches adopted by this method in the role-play exercises. First, the role plays were guided by a trained facilitator capable of providing psychological support as participants face the challenge of self-confrontation on video. Second, the use of an objective set of criteria as the basis for the guided role-play debrief may have reduced the stressful nature of the role play by making the focus of the analysis transparent. Third, the facilitator used a standardised debriefing format that encouraged an organised analysis of teaching behaviour and enhanced the process of reflection. This structured format guided participants to focus on (1) the intent behind their own teaching behaviours, (2) the effects of these behaviours on their learners and (3) the discovery of alternative behaviours to enhance their teaching. Thus, participants are not only prompted to reflect on their action (‘reflection on action’) (Schon 1983), but also they are guided to use a structured reflection about their actions, using criteria previously taught in the seminars. Fourth, the role play enabled the participants to experience the benefit of behavioural change by trying new behaviours and seeing the impacts of new behaviours on others. They also have the opportunity to see that a guided role-play exercise practice can lead to the acquisition of new behaviours. This direct observation of new skills development in oneself and others may generate enthusiasm for the method used to acquire those skills. This process of learning by doing is related to the theoretical concept of participant modelling advanced by Bandura, highlighting the benefit of the guided practice for skill mastery as a means to enhance both skill and self-efficacy (Bandura et al. 1974). Any combination of this complex set of characteristics could possibly have made the participants in the role-play group more positive about the course, more confident and skilled to change behaviour, and, consequently, led them to give higher ratings of their teaching improvement.

Conclusion

The primary aim of this study was to determine the impact of role play within the SFDC's Clinical Teaching course in relation to self-reported changes in teaching behaviours. Our results demonstrate that this faculty development programme was highly useful and led to significant changes in self-reported teaching behaviours – even for participants in the course without role play. However, participation in the course with role play led to higher ratings of the course's usefulness and to greater teaching improvement as compared to participation in the course without the role play, thus supporting the use of this technique for faculty development.

Acknowledgements

The authors express their gratitude to Patricia Madden, MPH for her help with the data analysis and Jane Mount for assisting with the literature search.

Declaration of interest: The authors report no conflicts of interest.