Competency-based residency training and the web log: modeling practice-based learning and enhancing medical knowledge^†

† This work has been presented as a poster at the Northwest Regional Society of General Internal Medicine Meeting in Seattle in February 2014 and the Society of General Internal Medicine's Annual Meeting in San Diego in April 2014.

Abstract

Background

By using web-based tools in medical education, there are opportunities to innovatively teach important principles from the general competencies of graduate medical education.

Objectives

Postulating that faculty transparency in learning from uncertainties in clinical work could help residents to incorporate the principles of practice-based learning and improvement (PBLI) in their professional development, faculty in this community-based residency program modeled the steps of PBLI on a weekly basis through the use of a web log.

Method

The program confidentially surveyed residents before and after this project about actions consistent with PBLI and knowledge acquired through reading the web log.

Results

The frequency that residents encountered clinical situations where they felt uncertain declined over the course of the 24 weeks of the project from a mean frequency of uncertainty of 36% to 28% (Wilcoxon signed rank test, p=0.008); however, the frequency with which residents sought answers when faced with uncertainty did not change (Wilcoxon signed rank test, p=0.39), remaining high at approximately 80%. Residents answered a mean of 52% of knowledge questions correct when tested prior to faculty posts to the blog, rising to a mean of 65% of questions correct when tested at the end of the project (paired t-test, p=0.001).

Conclusions

Faculty role modeling of PBLI behaviors and posting clinical questions and answers to a web log led to modest improvements in medical knowledge but did not alter behavior that was already taking place frequently among residents.

• graduate medical education
• core competencies
• blog
• practice-based learning

It has been over a decade since the Accreditation Council for Graduate Medical Education (ACGME) approved the six general competencies that serve as the organizational framework for graduate medical education (1). These competencies have also served as the framework for the Internal Medicine Milestone Project in the ACGME's Next Accreditation System (2). Practice-based learning and improvement (PBLI) and medical knowledge (MK) are two of the competencies delineated by the ACGME. Foundational steps in the competency of PBLI include the ability to monitor one's practice through reflection, recognize uncertainty and identify learning needs, then engage in learning and improvement while monitoring the impact on clinical practice (3). It has been concisely described as ‘how you get better at medicine’ (4). These steps have been incorporated into the Internal Medicine Milestones, which describe a resident who is ready for unsupervised practice as one whom, ‘routinely “slows down” to reconsider an approach to a problem, ask for help, or seek new information … [translating] new medical information needs into well-formed clinical questions’ (2).

Educational innovations have sought to teach PBLI either at its interface with another of the core competencies, system-based practice, in the form of quality improvement initiatives or as curricula in evidence-based practice (5). Less has been done to address how we teach the fundamental steps of ‘how you get better at medicine’. The literature suggests that role modeling is a nuanced, deliberate learning strategy (6, 7). For example, role modeling provides residents with templates for clinical decision-making that have both immediate and long-term relevance (7). Simultaneously, we know that learning in medical practice is embedded in clinical work.

We postulated that faculty transparency in how we learn from our uncertainties in clinical work could lead residents to embrace and incorporate the core principles in their professional development. Our faculty sought to model the core steps of PBLI detailed above on a weekly basis through the use of a web log by posting to the ‘blog’ a clinical scenario where he or she faced uncertainty about the best way to proceed, translated that into a clinical question, then pursued and shared an answer. Although resident-run web logs have been described, the impact of faculty blogging is unevaluated despite calls for the increased use of web-based tools in medical education (8).

Methods

We sought to understand the impact of this educational initiative by confidentially surveying residents before and after the project. Our residency is a university-affiliated, community-based, internal medicine program with seven residents per class. We have maintained a password-protected blog since 2006. The main use of the blog prior to this initiative was largely administrative (e.g., posting schedules).

Prior to the start of the project, using a web-based, secure survey yielding de-identified data, we asked residents about actions consistent with the fundamental steps of PBLI, including the frequency that they felt uncertain in clinical situations in the prior month, how often they searched for answers when faced with this uncertainty, and how often they successfully found answers (Fig. 1). We also surveyed residents about resources used to answer questions and purposes for using the blog. After this initial assessment, once each week for a period of 24 weeks, prompted by uncertainty in a clinical situation, one of the eight core faculty members wrote a summary of the situation, a detailed description of how he or she went about finding an answer, including search terms and resources used, and a discussion of knowledge learned. Prior to posting the summary to the blog, we distributed electronically a single pretest question to the residents to assess baseline knowledge in the area.

Fig. 1 Survey questions used to assess frequency of core behaviors of practice-based learning and improvement.

One month after the end of this 6-month project, we distributed a final survey to all residents, reassessing MK of topics covered in the faculty posts using the same 24 questions distributed initially as single pretest questions. Additionally, the survey reevaluated resident actions consistent with PBLI steps, repeating the same questions shown in Fig. 1, and reexamined resources residents used to answer questions. Lastly, the final survey gauged attitudes toward the practice of faculty blogging.

Using this pretest, posttest approach we measured changes in knowledge as well as changes in frequency of actions consistent with PBLI. We used PASW Statistics 18 Release 18.0.0 for all descriptive statistics and inference testing. To decide if there was a statistically significant change in MK, we used paired t-tests to compare percentage of correct answers before posting to percentage of correct answers at the end of the project. Given the non-parametric distribution of ordinal data about the frequency of PBLI actions, we used related-sample Wilcoxon signed rank test to decide if there was a statistically significant change in frequency.

Results

Examples of faculty postings included strategies to prevent urinary tract infections, etiologies of night sweats, and treatment for postherpetic neuralgia. All told, topics could be categorized as follows: five cardiovascular topics, four topics each in infectious disease and general medicine, three rheumatology topics, two topics each in hematology, gastroenterology, and neurology, one pulmonary topic, and one endocrinology topic. Postings averaged 651 words in length.

All 21 residents completed both the initial and final survey. Residents averaged completing 77% (range: 38–100%) of the pretest knowledge questions. The frequency that residents encountered clinical situations where they felt uncertain declined slightly but significantly over the course of the 24 weeks from a mean frequency of uncertainty of 36% to 28% (Wilcoxon signed rank test, p=0.008). Ten residents had no change in frequency of uncertainty, 10 residents experienced a decline in uncertainty, and just one resident had a slight increase in uncertainty (Fig. 2).

Fig. 2 Decile change from the beginning to the end of the faculty blogging project in the frequency of uncertainty residents experienced in the prior month. Values less than zero represent magnitude of declining uncertainty for individual residents, values greater than zero represent magnitude of increasing uncertainty for individual residents. The height of the bar is the number of residents recording this magnitude of change.

The frequency with which residents sought answers when faced with uncertainty prior to the project was high at a mean of 81% and did not change significantly, remaining at a mean of 78% (Wilcoxon signed rank test, p=0.39). Five residents did not change searching frequency at all whereas eight residents had a slight decline and two residents more substantial decline. Six residents increased searching frequency (Fig. 3). The range of the frequency of answer seeking was between 30 and 100% with three residents reporting on the final survey that they sought answers less than half the time whereas 15 residents sought answers at least 80% of the time. Success in finding answers to questions generated by clinical uncertainty also did not change significantly from before to after the project, remaining at a mean of approximately 75%. Though not reaching statistical significance, there appeared to be correlation (Pearson's correlation coefficient =0.377, p=0.092) between self-reported frequency of seeking answers when faced with uncertainty and percent of correct answers to MK questions on the final survey. Those engaged in less frequent searching appeared to answer fewer questions correctly.

Fig. 3 Decile change from the beginning to the end of the faculty blogging project in the frequency residents searched for answers in response to clinical uncertainty in the prior month. Values less than zero represent magnitude of declining frequency of searching for answers by individual residents, values greater than zero represent magnitude of increasing frequency of searching for answers by individual residents. The height of the bar is the number of residents recording this magnitude of change.

MK improved significantly. Prior to posting answers to clinical questions, residents got a mean of 52% of knowledge questions right. When reassessed 1 month after the end of the 6-month period, residents got a mean of 65% of these questions right (paired t-test, p=0.001, see Fig. 4). Prior to the project, the top five uses of the blog, in order, were accessing 1) schedules; 2) calendars; 3) the evaluation system; 4) our affiliated university's library of resources; and 5) clinical information. By the end of the project, residents continued to use the blog foremost to access schedules and calendars however accessing clinical content had climbed to the third most common reason to access the blog.

Fig. 4 Mean proportion of knowledge questions residents answered correctly prior to seeing the faculty post (prior to blogging) and at the end of the project (after blogging).

Finally, 18 of 21 residents considered the faculty blogging to be of high value and 95% of residents felt that faculty should continue to post clinical content to the blog. Representative comments on the final survey highlighted concerns about long entries and the relevance of content to immediate questions at hand, though clearly expressed enthusiasm. One resident wrote, ‘I like how the faculty is posting clinical questions and the answer, but I find it most helpful to see the discussion portion on how they reached their conclusion’. Another wrote, ‘I love the evidence-based weekly questions and blog posting. [It is] very helpful both clinically and as a resource for how to search’.

Discussion

Several findings from this educational innovation are worth highlighting. The statistically significant decline in uncertainty over the course of 6 months is consistent with expectations and supports the validity of our instrument relying on residents’ self-reporting of PBLI behaviors. However, we did not show improvement in residents’ frequency of searching for answers when faced with uncertainty in clinical situations. It is possible that our innovation had no impact on this behavior. However, it is notable that residents were already seeking answers frequently in response to clinical uncertainty. That this was happening around 8 out of every 10 times is not only reassuring but also leaves little room for demonstrating improvement.

Alternatively, resident fatigue as the academic year progressed may have been a countervailing force that kept our initiative from further improving PBLI behaviors. Unfortunately, lacking a control group, we could not describe the secular trend in behavior when faced with uncertainty. Further research into residents’ clinical uncertainty, its natural history, and the actions they take when faced with uncertainty would be valuable in deepening our understanding of how best to improve competency in the general domain of PBLI. Besides the limitation of not having a control arm, another important limitation was reliance on self-reported behaviors from surveying. Although recall bias may have affected responses, we believe our use of de-identified responses and 100% participation minimized overall biases present in these data.

Strategies residents used for finding answers to clinical questions did not seem to change substantially. Despite faculty role modeling using primary data found through PubMed searches, residents continued to most frequently use general resources when seeking answers to their questions. This suggests that traditional PBLI programs that teach the skills of evidence-based medicine in formal sessions, including structuring clinical questions and accessing most appropriate resources to answer those questions remain important.

Importantly, we did demonstrate that faculty posting to a blog led to modest improvements in MK. This was particularly noteworthy given that retesting of knowledge occurred up to 7 months after the initial posts of answers suggesting that knowledge learned was durable or, alternatively, that residents continued to access the blog to answer these clinical questions. Features likely contributing to the success of our blog for communicating MK included 1) its regular use by residents for administrative purposes prior to adopting it to post-clinical content; 2) posting by faculty well known to and trusted by residents; 3) the inherent structure of blogs which always put new content front and center on the web page; and 4) the use of keyword searching allowing residents to return quickly to clinical content they had seen previously. Notably, several reports over the past few years have described the use of blogs in medical education but have never proved effectiveness in enhancing MK (10–(14)). Our study is the first to show a blog as an efficacious tool for increasing MK and provides evidence that a blog can be an effective communication tool in medical education.

Notably, we found that some residents were less engaged in the fundamentals of PBLI. Although the substantial majority of residents sought answers frequently when faced with uncertainty, several residents engaged in this action much less frequently. Although underpowered, there was a suggestion of a correlation between answer seeking and MK with those less inclined toward the fundamental behaviors of PBLI also scoring lower on the MK assessment. This finding suggests that it may be important to identify these residents early as they may be less likely to succeed across multiple competency domains, and that efforts to encourage self-reflection, recognition of clinical uncertainty, and answer seeking are best targeted to this group. Finally, residents truly appreciated this intervention. As the blog became increasingly valued as a resource for clinical information, it even inspired some residents to do their own posting of MK content.

In summary, residents valued faculty role modeling of PBLI using a blog to highlight clinical uncertainties and then posting clinical questions and answers. Although this initiative led to modest improvements in MK, it did not alter behavior that was already taking place frequently. Benbassat notes that role modeling in medical education has had many definitions and that when defined as [learners] observing and reflecting on benefits and drawbacks of preceptors’ deliberate behavior and emulating important behaviors, then it is essential for professional development (9). Our purposeful modeling of acknowledging uncertainty, seeking answers, and sharing knowledge gained, fits this definition and is an educational innovation worth adopting.

Ethical approval

Institutional Review Board – Spokane deemed this project exempt, as it was part of program evaluation.

Conflicts of interest and funding

The author has not received any funding or benefits from industry or elsewhere to conduct this study.

Acknowledgements

The author acknowledges deeply the core faculty who supported this project and contributed regularly to the blog – Judy Benson, MD; John Frlan, MD; Jeremy Graham, DO; Mary Noble, MD; Samuel Palpant, MD; Darryl Potyk, MD; and Judy Swanson, MD. The author also acknowledges the wonderful residents in the program.

Notes

† This work has been presented as a poster at the Northwest Regional Society of General Internal Medicine Meeting in Seattle in February 2014 and the Society of General Internal Medicine's Annual Meeting in San Diego in April 2014.