Development and pilot testing of a reflective learning guide for medical education

Abstract

Background: Reflection is increasingly incorporated into all levels of medical education but little is known about best practices for teaching and learning reflection.

Aims: To develop a literature-based reflective learning guide for medical education and conduct a pilot study to determine whether (1) guide use enhances medical students’ reflective writing skills and (2) reflective scores correlate with participant demographics and satisfaction.

Methods: Guide development consisted of literature review, needs assessment, single institution survey, and educational leader consensus. The pilot cohort study compared professionalism reflections written with and without the guide by third-year medical students on their core obstetrics and gynecology rotation. Reflections were scored using a previously validated rubric. A demographics and satisfaction survey examined effects of gender and satisfaction, as well as qualitative analysis of optional written comments. Analyses used independent t-tests and Pearson's correlations.

Results: We developed a two-page, literature-based guide in clinical Subjective-Objective-Assessment-Plan (SOAP) note format. There was a statistically significant difference, p < 0.001, in the reflection scores between groups, but no effects of gender or satisfaction. Student satisfaction with the guide varied widely.

Conclusions: A single exposure to a literature-based guide to reflective learning improved written reflections by third-year medical students.

Introduction

There is a growing mandate, nationally and internationally, to incorporate reflection into all levels of medical education (ABIM et al. 2002; GMC 2003; Frank 2005). Medical educators have argued that reflection is critical for training physicians who (1) respond creatively to complex health systems, clinical cases, and social situations, (2) participate collaboratively in teams, (3) behave professionally and compassionately in stressful work environments, and (4) have the motivation and skills to continuously improve their practice (Maudsley & Strivens 2000; Mann et al. 2007; Sandars 2009). Accrediting bodies and oversight organizations recognize reflection both as a requisite skill for the new generation of medical providers and as an important tool for assessing the greater array of competencies now acknowledged in medical education (Epstein & Hundert 2002; Accreditation Council for Graduate Medical Education 2009). As a result, educators and clinicians across the medical education spectrum are looking for guidance in the teaching and development of reflective ability.

The literature on reflective practice in education is significant and decades old. It begins with Dewey (1933), in the 1930s, who described the human mind as a meaning-making organ and stated that “all genuine education comes through experience” and reflection on that experience. Schön (1983), writing in 1980s, argued the importance of reflection as a tool for professionals coping with the complexities of practice. Kolb (1984) conceptualized the relationship between reflection and action as a cycle of experience, reflection, reframing, and experimentation. Mezirow (1991) described four levels of reflection: from habitual action to thoughtful action, then on to reflection, and ultimately to critical reflection. He defined critical reflection as transformative learning in which reflection on experience leads to a new understanding and a plan to modulate one's behavior in the future.

Despite these robust theoretical frameworks, we know relatively little about teaching and learning reflection. Most articles describing applications of reflective learning in medical education fall into one of the four categories: (1) analyses of reflections to better understand the learners’ experience (Brady et al. 2002; Dyrbye et al. 2007; Roche & Coote 2008); (2) assessment of the reflective skills of learners (Pee et al. 2002; Boenik et al. 2004; Plack et al. 2007); (3) comparison of educational formats such as oral vs. written reflection or peer vs. faculty feedback (Platzer et al. 2000; Baernstein & Fryer-Edwards 2003); and (4) evaluation of the impact of reflection on learning (Sobral 2005; Blatt et al. 2007; Mamede et al. 2008). Although the majority of these articles describe reflective exercises, the development and efficacy of such prompts are not the primary focus of the work, and there exists a remarkable diversity in both the focus and nature of such exercises (Branch et al. 1993; Johns 1994; DasGupta & Charon 2004; Toy et al. 2009; Wald et al. 2009). Few offer guidelines which incorporate recommendations from the current literature and no one approach has been widely adopted by teachers and learners with a range of reflective learning goals and in a multitude of educational settings.

Our experience at the University of California, San Francisco (UCSF) mirrors this larger reality. Many course, clerkship, and program directors have incorporated reflective exercises into their curricula despite the paucity of data on how best to teach reflection, little institutional coordination of efforts, no training of the teachers expected to respond to the learners’ reflections, and little contribution in the way of faculty role-modeling of reflective practices. Consequently, there was a pressing need to base curricular interventions in theory, devise best practices, and support educators in their efforts to develop learners’ reflective skills.

In this article, we (1) describe the development and components of an educational literature-based reflective learning guide for medical education; and (2) report on a pilot study to determine (a) whether use of this guide enhances reflective writing skills in medical students and (b) whether reflective scores correlate with participant demographics and satisfaction. We conclude with a discussion of limitations, lessons learned, and the relation of this work to the reflection literature.

Table

Practice points

A reflective learning guide based on theory improves learners’ written reflections. Reflective ability in medical students does not appear to correlate with gender or interest in reflection. Teaching reflective skill may require repeated exposure to reflection education and feedback. Additional research is needed to develop best practices for reflection in medical education.

Methods

Guide development

Our development process consisted of literature review, survey of current reflection activities in Undergraduate Medical Education (UME) at UCSF, an informal needs assessment of UCSF course and clerkship directors, and design and revision by group consensus of a reflective learning guide.

In the spring of 2008, we convened a group of UCSF medical school curriculum leaders representing seven major clinical specialties, members of the Office of Medical Education, and researchers with expertise in reflection. The Task Force met monthly for 6 months with the goal of using Kern's (1998) six-step model of curriculum development as a conceptual framework for tackling the challenge of incorporating reflection into our undergraduate medical student training. We began with a survey of all required courses and clerkship directors about reflection activities in the UCSF UME curriculum. Follow-up emails determined the context of these exercises, feedback procedures, and the directors’ perceptions of the challenges in teaching reflection. The latter served as an informal needs assessment. Coincident with the survey, we searched the PubMed and ERIC databases using the terms “reflection,” “curriculum,” “teaching,” “learning,” and multiple synonyms for each term. We also reviewed the bibliographies of relevant articles to find additional scholarship in this area. We then compiled the survey data and considered the reflection exercises in light of the literature on reflection.

We found 18 exercises distributed across departments and years 1–3 of the UCSF UME curriculum. Prompts for these exercises ranged from one line to half a page. Some consisted of short-answer questions specific to the assignment and requiring little reflection. A larger number asked about challenges and lessons learned. Many of the clinical assignments asked the learner to describe an experience which “surprised, moved or inspired” them. None provided information about reflection and few appeared likely to elicit even a minority of the critical elements of reflection described in the literature. In most settings, educators were not providing learners with the skills to reflect, consistently giving feedback on reflective exercises, or offering assessment of learners’ reflective skills. The course directors universally expressed a need for training and guidance in teaching reflection.

Based on our survey and literature review, we decided to develop a brief, stand-alone guide which would be instructive to both learners and faculty. We also agreed to pilot test the guide before implementing it widely in the medical school. Via a process of group discussion, we then developed a literature-based consensus definition for reflection in medical education: “The critical analysis of personal experience to enhance learning and improve future behavior and outcomes.”

Finally, we set the following parameters for the guide:

1. It should be brief enough for use in hour-long sessions.

2. For broad utility, it should focus on promoting reflective thinking rather than fulfilling a particular reflective agenda such as professional development or clinical reasoning.

3. We should proactively address common sources of faculty and student resistance including justification for adding to a full curriculum and the requirement for written reflection.

4. Because many learners and educators will not have had formal instruction in reflective learning, the guide should function adequately as a stand-alone tool.

5. The guide should make the steps for reflection sufficiently clear, so that they could be used to facilitate feedback on reflective skills from faculty and peers.

Pilot study

We conducted a cohort study during the 2008–2009 academic year to assess the impact of the Learning from your Experiences as a Professional (LEaP) guide on the written reflections of third-year medical students on their required obstetrics and gynecology rotation and to look for associations between reflection scores and demographic and satisfaction variables as assessed by a questionnaire.

Setting and participants

At UCSF, the third-year curriculum is divided into six blocks. During 2008–2009, 122 third-year medical students took the core obstetrics and gynecology clerkship during the six blocks of the study period. Although students take the clerkship at multiple sites, the reflection exercise is a required component of the clerkship. A total of 115 reflections were used in the study. Two students did not complete the rotation and five essays were partial or illegible. The UCSF IRB approved this study.

Intervention

All students completed a written reflection on professionalism using the prompt “Select a clinical situation during this rotation that taught you the most about demonstrating integrity, respect and responsiveness to the needs of the patient above your own.” Students in blocks 1 and 2 received only the prompt while students in blocks 3–6 also received the LEaP guide and completed a brief demographic and satisfaction questionnaire. Students in both cohorts were asked to write at least one page and turn in the reflection by an assigned date and time. The obstetrics and gynecology-related content of the reflections (and not how well the learners reflected) was discussed at mid-block in small groups with 1 faculty discussion leader and about 10 students. This feedback process was not part of the study. Additionally, while the original intent was to not use LEaP in block 6 to control for differences across the clerkship year, an administrative error led to the LEaP being used in that block as well.

Data collection

Two trained raters scored each reflection using a previously validated rubric which provides scores in increments of 0.5 from 0 (no reflection) to 6 (critical reflection) (Learman et al. 2008; O'sullivan et al. 2010). The rubric's step-wise progression is as follows: does not respond to the assignment; describes without reflecting; does not justify lessons learned; provides limited (personal) justification of lessons learned; includes evidence of lessons learned; analyzes factors from past experience; and integrates previous experience with current events and data to inform further action. The rubric provides brief and elaborated scoring criteria and examples for each step. Based on a previous generalizability study, we obtained a reliability of 0.89 for the reflective ability score when using two raters (O'Sullivan et al. 2010).

The demographic and satisfaction questionnaire requested information about gender and impressions of the LEaP guide. Specifically, students were asked to rate on a 5-point Likert scale ranging from strongly disagree (1) to strongly agree (5) whether the LEaP led to new insights, helped them formulate a learning plan, and helped them reflect well. It also asked about ease of use of the guide and whether students would recommend it to others. While the questionnaire did not specifically ask for written comments, half the students offered comments and one of us (Julie Lindow) solicited student recommendations following the small group discussions for making the guide more useful and user-friendly.

Analysis

We calculated descriptive statistics for average reflection score, gender, and all survey questions. We performed an independent t-test to compare the average reflection scores for pre-intervention and LEaP intervention groups and calculated an effect size (Hojat & Xu 2004). We grouped all LEaP intervention groups because we found no significant change in average reflection scores by block. We also calculated an independent t-test to compare the average reflection scores for male and female genders. Finally, we calculated Pearson correlations between average reflection scores and LEaP intervention group satisfaction survey data. Qualitative analysis of the satisfaction questionnaire written comments was done using inductive coding.

Results

Reflective learning guide

We developed the LEaP guide to reflective learning. The guide consists of a one-page document filling both front and back of the page: a structured approach to reflection (front) and an information sheet about reflection and strategies for successful reflection (back). The front page takes the learners through a five-step process based on theory and modeled on the format of clinical notes (Figure 1): Chief Complaint (Hatton & Smith 1995; Moon 2004), Subjective (Schon 1983; Boud et al. 1987; Mezirow 1991), Objective (Moon 2004), Assessment (Kolb 1984), and Plan (Mezirow 1998; Epstein 2007). The SOAP approach was selected for its familiarity to learners and to emphasize that the key steps in effective reflection are much like those in clinical reasoning, moving from gathering of subjective information, through listing of data to an assessment of all information and formulation of a plan. The back page begins with the definition of reflection in medical education developed by our Task Force. It then provides an explanation of the educational and professional benefits of reflection, lists factors associated with success in the development of reflective ability, and offers strategies for improving reflective writing. In essence, this page serves the critical role of explaining why the learners are being asked to reflect and how they can do so effectively.

Figure 1. UCSF LEaP: Learning from your Experiences as a Professional.

Pilot study

A total of 115 out of 122 third-year medical students took part in the pilot study, with 37 in the pre-intervention cohort, and 78 in the LEaP intervention group. Table 1 presents the effect of the LEaP on reflection scores. The reflection score was significantly higher with LEaP (p < 0.001) with a large effect size and scores indicating learners using LEaP included analysis of lessons learned and incorporated evidence into the reflection more often. There was no evidence of a maturation effect; in other words, scores in blocks 3 and 4 did not differ significantly from scores in blocks 5 and 6. This allowed for grouping of all LEaP scores. We found no difference in average reflection scores between male and female learners (males: mean = 3.6, SD = 0.75; females = 3.6, SD = 1.17, p = 0.74).

Table 1.  Impact of LEaP guide on student reflection scores

	LEaP (n = 78)			Control (n = 37)
	Mean	SD	Range	Mean	SD	Range	p-Value	Effect size
Reflection score^a	3.6	1.2	0.5–5.8	2.6	0.8	0–3.5	p < 0.001	1.25

Note: ^aScoring ranges from 0 (no reflection) to 6 (critical reflection).

We collected learner satisfaction as well as gender data on 34 of the 78 learners in the LEaP intervention group. Table 2 provides the survey results. On average, students had neutral perceptions of the LEaP as a learning tool, and we found no correlation between average reflection scores and any of the survey items on the satisfaction questionnaire. Learner comments included a variable set of responses. The responses fell into four categories: (1) great enthusiasm, (2) a perception that the guide was too constraining and distracted from the learner's ability to reflect, (3) complaint that the guide was too complicated, and (4) the sense that reflection should not be required since it takes time away from more important activities.

Table 2.  Results of the satisfaction survey

Item	Mean	SD
Gained insight	3.0	1.2
Helped formulate plan	2.8	1.1
Guide challenging	3.1	0.9
Helped reflect	2.9	0.9
Recommend LEaP	2.7	1.0

Note: 5-point Likert scale with 1 = strongly disagree and 5 = strongly agree.

Discussion

In this study, we developed and pilot tested a two-page literature-derived instructional guide to assist learners in reflection. We took this approach because literature review and a local needs assessment revealed marked variability in the application of theoretical frameworks in medical education reflection exercises and little teaching of learners about how to improve their reflective skill and maximize their reflective learning.

We learned many lessons from this guide development process and pilot testing which may be of use to others. Although our original intent was to produce a one-page document, creation of a stand-alone guide required the addition of a second page, which not all learners read. We chose the SOAP note framework both because of its familiarity and to emphasize the core elements of reflection but did not make the reason for this choice sufficiently clear on the page. In an effort to create a single guidance for both educators and learners, we included information of greater obvious utility to teachers. Some learners argued that the LEaP was too constraining and interfered with their ability to reflect, echoing a concern raised by Boud and Walker (1998) about “recipe following” in reflection without actually addressing the questions and significance of the experience being reflected upon. Others have noted, however, that students believe they are already reflecting and consequently do not see the need for structured exercises (Grant et al. 2006). This latter more accurately portrays our experience; a great majority of the unstructured “reflections” at UCSF consist of moving anecdotes, diatribes, or self-congratulatory tales with little evidence of learning.

Our pilot study demonstrated significant differences in average reflection scores of third-year medical students in favor of those using the LEaP guide. Without the guide, nearly all learners wrote reflections which scored below 3 and consisted exclusively of a description of an experience with little or no consideration of the event beyond their own vague impressions. Using the guide, significantly more learners earned higher scores because they incorporated more elements of reflection including reframing their experience by seeking feedback from others and going to the literature to clarify issues of clinical or professional uncertainty. Moreover, a subset of these higher scoring learners also analyzed their experience, integrated it with past learning, and/or formulated a learning plan based on the reflection. Prior to the intervention, no learners attempted any of these important steps and our analysis revealed that this was not an artifact of maturation effect. While those who reflected most effectively and earned the highest scores represent a minority of the sample, this was a single intervention. As with many skills, we expect that practice with this type of reflection would lead to even greater gains in reflective skill.

Our pilot also examined associations between reflection scores and demographic and satisfaction variables. Although preliminary data from another institution suggested that females might be better at reflection than males, we found no such difference (Dannefer & Bierer 2008). Our hypothesis that learners who perceived the guide as most useful or easy to use might be better reflectors was not supported. Possible explanations for this finding include greater impact of other forces on learners’ efforts in the reflective exercises such as desire to do well on the clerkship, individual attitudes toward reflection, whether or not learners read the entire guide, and the curricular context of the exercise. Notably, learners still scored significantly higher in this single intervention with the LEaP despite their lack of enthusiasm for it. Moreover, the variability of responses also might have resulted from many factors beyond the LEaP guide itself, including that such comments were not required and so may not be representative.

This study had several limitations. We conducted the pilot test at a single institution with learners at one stage of professional development. As a result, we do not know how the guide works with students in other learning contexts or in GME or CME settings. Learners had a single exposure to the guide; so, we were not able to test for the sorts of dose–response effects common to educational skill development. The comparison groups were of different sizes. The exercise, while required, was not graded and occurred in the middle of the rotation with no later follow-up in the clerkship. Motivational issues might have interfered with learner effort and guide use, and we could not study the impact of the reflective learning on the students’ knowledge, skills, or attitudes. Some might further dispute the scoring of reflections as reductionist. We acknowledge that concern but argue that if reflection is to become a key skill in medicine, we will need feasible, valid, and reliable tools to assess it. We used a theory-based validated rubric. Of course, the validity of the scoring system would be improved if we had evidence that high scores correlate with improved learning and clinical skill. To date, no studies of reflection have established behavioral or patient care outcomes.

As with many interventions that are tailored to meet specific objectives, an interaction between the intervention and the assessment is possible. It is possible that we have demonstrated that learners, when prompted, will incorporate items into their reflection that will earn them a higher reflection score. Two findings from this study suggest that we are in fact measuring reflective ability: (1) not all LEaP users followed the steps in the instructions, but their reflection scores were still higher than the control mean; and (2) individual control students achieved higher reflective scores without the structure. Collecting more assessments in a variety of situations would help to further clarify this issue. Of note, the rubric and the LEaP were developed independently, and both are based in the literature on what is needed for successful critical reflection. Equally important, the goal of the guide is to increase learner's reflective ability, and this study showed that providing structure helped students reflect in more depth. Indeed, it is possible that, having read the guide, the LEaP users may have been influenced by them, even if they did not follow the steps. Finally, other guidelines have been developed but not widely adopted and it would be useful to study the relative efficacy of the different approaches.

This study raises resource issues that are not unique to our approach but speak to concerns about written reflections generally in medical education. Two recent reviews of reflection in medical education note the critical role of feedback in reflective learning (Mann et al. 2007; Sandars 2009). But scoring reflections and providing feedback require trained raters with the time to read reflections. At our institution, we found few cases of individual responses to learners. More commonly at our institution, the content of reflections was thematically summarized for large groups of students. In some cases, the reflections were simply collected, and no feedback of any type was given. While some course directors used reflections to understand the student experience and evaluate distant clerkship sites, many educators cited lack of time and lack of training as the principal reasons for how they handled reflective essays. Schools and training programs will need to find resources to train faculty in reflection and compensate them for the time required to respond to written reflections.

Given the widespread incorporation of reflection into medical education, more work needs to be done to develop best practices for teaching and learning this fundamental skill. Our experience suggests marked educational benefits from a brief, structured, theory-based guide to reflection. While some will find this approach too restrictive, we suggest that the steps outlined in our guide do not preclude critical incident or narrative approaches to reflection but rather might be used to transform the stories produced by such techniques into explicit learning and planned professional development.

Acknowledgments

The authors acknowledge Martin Bogetz, Karen Hauer, David Irby, Cindy Lai, Helen Loeser, Kevin Mack, Jessica Muller, Steven Polevoi, Jason Satterfield, and Kevin Sousa, and aslo the UCSF Medical Education Research Fellowship.

Declaration of interest: We have no declarations of interest to report.