Teaching evidence-based medicine: Impact on students’ literature use and inpatient clinical documentation

Abstract

Background: Effective teaching of evidence-based medicine (EBM) to medical students is important for lifelong self-directed learning.

Aims: We implemented a brief workshop designed to teach literature searching skills to third-year medical students. We assessed its impact on students’ utilization of EBM resources during their clinical rotation and the quality of EBM integration in inpatient notes.

Methods: We developed a physician-led, hands-on workshop to introduce EBM resources to all internal medicine clerks. Pre- and post-workshop measures included student's attitudes to EBM, citations of EBM resources in their clinical notes, and quality of the EBM component of the discussion in the note. Computer log analysis recorded students’ online search attempts.

Results: After the workshop, students reported improved comfort using EBM and increased utilization of EBM resources. EBM integration into the discussion component of the notes also showed significant improvement. Computer log analysis of students’ searches demonstrated increased utilization of EBM resources following the workshop.

Conclusions: We describe the successful implementation of a workshop designed to teach third-year medical students how to perform an efficient EBM literature search. We demonstrated improvements in students’ confidence regarding EBM, increased utilization of EBM resources, and improved integration of EBM into inpatient notes.

Introduction

Evidence based medicine (EBM) encourages utilization of evidence from clinical research to support clinical decision-making for individual patients (Sackett et al. 2000). According to Sackett, this involves five sequential steps: forming an answerable clinical question, searching the literature, critically appraising the studies available, applying the data to the individual patient and assessing the intervention's results (Sackett et al. 1996). In this manner, practitioners are able to deliver high-quality care by basing decisions on both the patient's wishes and the best available evidence found in the literature (Evidence-Based Medicine Working Group 1992). There is broad support to integrate EBM into undergraduate curricula (Hatala & Guyatt 2002). There are many reports of teaching steps one through three of Sackett's model; however, outcome measures in these studies have been limited to self-reported searching, improved quality of literature searches, and increased use of MEDLINE (Bennett et al. 1987; Landry et al. 1994; Rosenberg et al. 1998; Srinivasan et al. 2002; Cayley 2005; Gruppen et al. 2005; Schilling et al. 2006). It is still unknown whether or not these learned skills are applied to the care of patients in the clinical setting.

We used a physician-led, hands-on workshop to train third-year medical student clerks to design an answerable clinical question and perform a time-efficient medical literature search. Our primary outcome was evaluation of whether students’ utilization of EBM resources changed following our intervention as demonstrated by analysis of students’ inpatient note citations. We also measured students’ attitudes and knowledge about literature searching, as well as their ability to incorporate what they learned about EBM into write-ups they completed as part of their responsibilities of taking care of patients on the wards.

Table

Practice points

Effective methods to teach EBM skills to medical students are paramount. A physician-led hands-on workshop teaching literature searching sills was associated with increased comfort and confidence with EBM. Students demonstrated increased use of EBM and improved incorporation of EBM into their inpatient clinical notes following the workshop.

Methods

Population and setting

Third-year students on the inpatient portion of their internal medicine clerkship (12 weeks) at Vanderbilt Medical School are required to create written evaluations (“history and physical” notes) for two to three new patient admissions per week. Clerkship goals require students to read about their patients and discuss their findings from literature reviews in the assessment and plan section of the admission note. For over a decade, all student inpatient history and physical notes have been expected to include literature citations relevant to the patient's presentation. Students have access to a medical library that provides publicly available and proprietary evidence-based electronic databases and journals. All students received a single didactic session led by a medical librarian during their first year of medical school designed to teach MEDLINE skills, however, no further specific training program had been established to address literature searching and critical assessment skills.

Intervention

We implemented a single, physician-led, hands-on 3 h workshop to teach medical students clinical question formation and literature retrieval skills according to the first two steps of EBM as outlined by Sackett et al. (1996). In recognition of the increased availability and utilization of pre-appraised resources by busy practicing physicians (Montori & Guyatt 2008), we focused on teaching students how to locate and use these resources. This included guidelines such as the National Guidelines Clearinghouse, the United States Preventive Services Task Force guidelines, and the CDC as well as systematic reviews such as ACP Journal Club, Cochrane Database, and EBM. Selection of the EBM resources introduced to the students was based on the instructor's prior continuing medical education classes related to teaching EBM as well as personal use and experience. The instructor (EAS) has 6 years experience in EBM training. The session consisted of an interactive lecture and a hands-on practicum. The didactic component covered three topics: (1) formulating the clinical question; (2) devising a search strategy to answer clinical questions; and (3) introduction to the strengths and weakness of available pre-appraised resources, such as clinical guidelines and systematic reviews. Throughout this overview, the instructor led students through sample literature searches at their own individual computers.

The workshop was delivered during each of the four internal medicine clerkship blocks during the 2007–2008 academic year. Each clerkship block has approximately 30 students. In order to maintain a small group setting, we randomly assigned students to one of five small groups (4–6 students per group) and offered the workshop to one small group weekly for the first 5 weeks of the rotation. Therefore, the workshop was delivered both at different times during the rotation as well as during all four clerkship blocks. At the beginning of each clerkship block, we invited students to participate in the study. All third-year students, regardless of whether they agreed to be in the study, had the opportunity to participate in the workshop. The Institutional Review Board approved the study design.

Evaluation and outcome measures

Outcome measures included pre- and post-surveys, physician analysis of student write-ups, and computer log data of students’ searches. After reviewing the literature, we designed a 16-item multiple-choice and free response survey that assessed students’ attitudes toward literature searches, their prior use of EBM resources, and their knowledge of clinical question formation and searching techniques. Students completed the survey before the workshop (“pre-workshop survey”), at the conclusion of the workshop (“post-workshop survey”), and on the final day of the clerkship (“end of clerkship survey”) which occurred 8–12 weeks following the workshop depending on to which group the student was randomized. The pre-workshop survey had additional items regarding baseline characteristics of participants; the post-workshop survey had additional items to assess their satisfaction with the workshop.

All internal medicine clerks are expected to include evidence from the literature in their inpatient notes as it relates to care of their specific patients. However, prior to our workshop, students were not formally instructed on how to locate or utilize EBM resources because it had been assumed that this instruction would be part of the informal curriculum on the wards. Therefore, we evaluated changes in behavior following the intervention by analyzing students’ inpatient admission notes and using these notes as a surrogate marker of impact on patient care. At our institution, all notes written by students during their clerkship rotations are stored in the Learning Portfolio system, an online, secure, HIPAA-compliant web application (Spickard et al. 2008). Two trained physicians used a standardized scoring sheet to score students’ notes accessed through Learning Portfolio. Our standardized scoring sheet was based on our prior work (Spickard et al. 2008), which utilized a previously reported inpatient note evaluation instrument (Kogan & Shea 2005). Our scoring metric included a quantitative component to record the number of citations and the type of resource cited. Our scoring metric also included a quantitative component to record the number of citations and the type of resource cited. Additionally, elements from the Kogan and Shea (2005) evaluation instrument applicable to our study included prioritization of a problem list and formulation of a differential diagnosis. We further evaluated the overall quality of the discussion portion of the note by assessing the quality of the EBM citations, the application of the information from the citations back to the patient case, and an overall ability to demonstrate how the information from EBM citations might influence the patient's clinical care. These five qualitative components were scored on a five-point Likert scale.

Two physician raters evaluated student notes written immediately before and after the workshop. Given limitations of faculty and financial support available for this study, we randomly selected (using a random number generator) 51 of our participating students to be evaluated by the two raters. Raters scored the three admission notes written immediately prior to the workshop and the three admission notes written immediately following the workshop for each student evaluated. Admission notes were chosen consecutively based on their timing with respect to the workshop. The two raters were presented each of these 51 students’ notes via the Learning Portfolio system in a random order (using a random number generator) with respect to both student and timing. Raters were blinded to the date of the note and to whether the note was written before or after the workshop. To ensure similarity between raters, both raters jointly rated a test set of 10 documents, then independently rated 20 documents. Inter-rater reliability for all metrics on 20 admission notes before the start of the study was acceptable with a linearly weighted kappa of 0.51.

Finally, we analyzed computer log data that tracked student searches to various EBM resources using hyperlinks and search tools integrated within the electronic medical record system. Students are oriented to these tools as they begin their third-year medicine clerkship (prior to the EBM workshop). Since use of each EMR-linked resource is logged with a unique identifier for each student, we were able to characterize each resource use as before or after the specific date of each student's EBM workshop. This analysis did not include student access that may have occurred from home, as a result of directly entering in web addresses (i.e., URLs), or from computer systems outside the Vanderbilt system.

Data analysis

Data from all questionnaires and evaluation tools were collated into a Microsoft® Excel spreadsheet. Inpatient admission notes from both before and after the intervention were matched according to the student's study number. In this manner, each student's pre-intervention notes could serve as a control to be compared to post-intervention clinical documentation. Pre- and post-intervention note ratings were averaged for each student. Analyses used Fisher's Exact test for categorical data and two-tailed paired t-tests for the physician review of clinical notes. All analyses were two-tailed where applicable. Analyses of proportions used the binomial exact test (for binomial distributions) or a two-sample test of proportional equality (when comparing two proportions). Cohen's kappa was used to assess inter-rater agreement. All statistical analyses were performed using Stata, version 9.2 (College Station, TX).

Results

The study began in July 2007 and concluded in June 2008. All 115 medical students rotating through the internal medicine clerkship were eligible to participate in the study; however, only 100 students were able to attend the workshop and consented to the study. Of those participants, 98 students (85%) completed all three surveys (pre, post, and final). The mean age of participants was 25.6, 42% were female, and all participants had prior research experience. Ninety-seven percent of the students reported the workshop to be useful and >90% would recommend it to fellow students. In addition, 94% reported they would incorporate the skills learned into the clinical care of patients.

Table 1 demonstrates results of student's pre- and post-workshop attitudes toward EBM. Improvements in students’ confidence in using EBM and perceived value of the application of EBM in clinical medicine persisted through the end of the clerkship. After the workshop, fewer students believed that literature searches were excessively time consuming. In addition, less students believed that questions could be more quickly answered using textbooks or consultants rather than by performing literature searches. These changes persisted to the end of the clerkship.

Table 1.  Self-reported percentages of attitude outcomes among 98 students before and after a third-year EBM clerkship intervention

Survey question	Pre-workshop (percentage agree/ strongly agree)	Post-workshop (percentage agree/ strongly agree)	End of clerkship (percentage agree/ strongly agree)	p-Value
I feel comfortable using EBM to answer clinical questions	73	96	91	<0.001*
After performing a literature search, I feel confident I have found the best evidence to answer the question	30	89	75	<0.001*
EBM helps clinical decision making	89	99	94	0.010*
EBM is useful on a daily basis	80	95	91	0.031*
Literature searches are too time-consuming to do in the hospital or clinic	29	7	8	<0.001*
Questions can be more quickly answered with textbooks or consultants rather than literature searches	40	18	21	0.002*

Note: *p-Value < 0.05, Fisher's exact test.

Students also reported an overall shift in the type of information resource they used with increased utilization of evidence-based resources such as systematic reviews and reduced use of more general resources such as Google (Figure 1). Analysis of data within each of the four clerkship rotations in the academic year did not show any significant differences.

Figure 1. Survey data of average number of times per week a literature resource was reported utilized before and after the workshop.

Table 2 presents the data from the individual physician reviews of 289 clinical notes from 51 unique students participating in the study who served as a representative of all participants in our study. Analysis of admission notes showed the number of citations per inpatient note was generally low, and there was no significant difference in the number of citations before and after the workshop. Most (75%) citations in both pre- and post-analyses were to primary journal articles. There was a trend toward increased utilization of both clinical guidelines and systematic reviews following the workshop.

Table 2.  Number and type of EBM citations in clerkship notes

	Mean citations per note (95% CI)
	Pre-workshop	Post-workshop	p-Value
Total number of citations per note	2.29 (1.92–2.67)	2.15 (1.84–2.47)	0.57
Textbook citations per note	0.098 (0.04–0.16)	0.196 (0.10–0.29)	0.09
UpToDate citations per note	0.233 (0.071–0.395)	0.202 (0.105–0.301)	0.75
Journal citations per note	1.81 (1.48–2.13)	1.54 (1.27–1.81)	0.21
Guideline citations per note	0.105 (0.041–0.170)	0.123 (0.068–0.179)	0.68
Systematic Review citations per note	0.053 (0.009–0.096)	0.094 (0.041–0.148)	0.24

Notes: Numbers represent mean citations per note for 298 notes written by 51 unique students. p-Values were calculated using two-tailed t-tests.

Table 3 presents the data from the qualitative assessment of the student clinical notes (n = 270 from 46 students). Only 46 of our randomly selected 51 students had notes that were complete enough within the Learning Portfolio system to be able to be assessed using our scoring metric. Despite the lack of difference in number of citations, physician qualitative judgment of overall quality of the EBM incorporation into the discussion of the patient's problems improved following the intervention (3.62 vs. 3.91, p = 0.02, using a two-tailed paired t-test). While changes in other ratings of general assessment and plan quality that were not tied to EBM utilization were not significant, each trended toward improvement after the workshop (p = 0.07–0.15, using a two-tailed paired t-test). Inter-rater agreement for physician reviewer was good; the kappa ranged from 0.76 to 1.00 for numerical assessments; the linearly weighted kappa for the note quality assessment was 0.52.

Table 3.  Quality of EBM content in clerkship notes

	Mean quality score (95% CI)
	Pre-workshop	Post-workshop	Improvement (95% CI)	p-Value
Quality of EBM discussion and application to patient	3.62 (3.33–3.92)	3.91 (3.65–4.17)	0.29 (0.04 –0.53)	0.02*
Prioritizes problem list	4.07 (3.87–4.28)	4.25 (4.5–4.45)	0.17 (−0.05–0.41)	0.14
Formulates complete differential diagnosis	3.89 (3.66–4.11)	4.08 (3.84–4.32)	0.19 (−0.03–0.41)	0.09
Discusses likelihood of differential diagnosis and plan	3.97 (3.72–4.21)	4.17 (3.91–4.42)	0.20 (−0.02–0.41)	0.07
Overall quality	3.98 (3.74–4.22)	4.14 (3.91–4.38)	0.16 (−0.06–0.39)	0.15

Notes: Numbers represent mean physician ratings for each note's assessment and plan according to a 5-point Likert scale (1, poor; 5, excellent). p-Values were calculated using a two-tailed paired t-test. *p < 0.05.

Figure 2 presents the results of the computer log analysis. Usage for all electronic resources increased (odds ratio 6.82 [95% confidence interval, 6.00–7.75], p < 0.0001), with students clicking on 289 resources (0.12/week) before the workshop and 1775 resources (0.66/week) following the workshop. The proportion of systematic peer-reviewed databases (such as Cochrane Database, Clinical Guidelines, etc.) utilized in searches increased dramatically (odds ratio 5.57 [95% confidence interval, 3.78–8.47], p < 0.0001), accounting for only 13% of all resources accessed before the workshop but 59% of the resources accessed following the workshop.

Figure 2. Percent utilization of EBM resources and non-EBM resources as determined by computer log analysis before and after the workshop.

Discussion

EBM is a powerful tool that enables students to attain the skills and knowledge needed for lifelong self-directed learning. With the current rapid influx of new information in the biomedical literature, students must learn how to navigate the information thoroughly and efficiently to apply the best evidence available to their patients. Previous studies have demonstrated that third-year students are unable to construct efficient search strategies and more than three-quarters of those studied were unable to find search results adequate to affect clinical care (Burrows & Tylman 1999). Effective and feasible methods to teach EBM searching skills are paramount.

We demonstrate a positive impact of a short EBM workshop on student's attitudes about EBM, personal confidence in using EBM, student's utilization of EBM concepts in clinical notes, and student usage of EBM-based websites. Previous studies have been able to demonstrate positive changes in students’ attitudes toward EBM and resource utilization following similar interventions (Bennett et al. 1987; Landry et al. 1994; Srinivasan et al. 2002; Cayley 2005; Schilling et al. 2006). However, prior reports of studies teaching medical students how to form a clinical question and perform a literature search have had a different focus than our own in that they have been limited to preclinical students not involved in patient care (Rosenberg et al. 1998) or have involved resource intensive curricula on the order of eight sit down sessions (Dorsch et al. 2004), a 4–week elective (Gruppen et al. 2005), or a 4-year comprehensive EBM curriculum (Barnett et al. 2000). Other studies have focused more on outcome measures related to search analysis of hypothetical cases, rather than real time patient care situations (Rosenberg et al. 1998; Burrows & Tylman 1999; Dorsch et al. 2004; Gruppen et al. 2005). Schilling et al. (2006) showed that an online course was associated with improved ability of students to formulate a question, perform and effective search strategy, and use more MEDLINE resources during the clerkship. Landry, however, reported a curriculum focused on critical appraisal skills did not have an effect on the frequency of literature citations found in students’ inpatient write-ups (Landry et al. 1994). Others have not been able to show improved student attitudes about EBM, formation of an answerable question, or ability to answer the question efficiently (Burrows & Tylman 1999; Badgett et al. 2001) suggesting that there is still work to be done in finding the optimal methods to teach literature searching skills to medical students.

Our study demonstrated pre–post changes in attitudes about EBM, confidence in using EBM resources, and self-reported increase use of EBM resources that lasted for several weeks following an intervention designed to teach comprehensive literature searching skills. In addition, the self-reported increase in utilization of EBM resources was verified by our computer log analysis which demonstrated a greater increase in student use of EBM resources than of non-EBM resources. Our personal review of 289 student inpatient notes demonstrated no significant change in the overall number of citations following the intervention, as we expected; however, the quality of those citations did improve in as much as they reflected more EBM resource utilization.

Our study is unique in that our students demonstrated both a trend toward increased citation of peer-reviewed databases and clinical guidelines as well as improved quality of EBM incorporation into students’ notes discussing real-time patient issues following the workshop. Not only did the quality of the citations improve with increased utilization of pre-appraised EBM resources in the notes, but students were better able to apply the data back to their patient's case and use that information to formulate more evidence-based assessments and plans. Though, the number of citations per note is small and our sample size is limited, we consider that these trends in student behavior may suggest a possible association with the workshop. Perhaps, even more important is the demonstration of the increased utilization of EBM resources as documented by student log-ins. Although optimally, we would like to be able to measure the effect of our intervention on actual patient care outcomes, in practice this is difficult, and we believe it is likely that the demonstration of increased citation of quality EBM resources suggests increased utilization of these resources which may lead to increased implementation of evidence-based decisions in clinical care. In addition, instead of focusing solely on MEDLINE skills, we sought to expand students’ EBM toolkit to include systematic reviews and clinical guidelines. We believe that by focusing on pre-appraised EBM resources rather than original studies as has been done by others (Landry et al. 1994; Rosenberg et al. 1998; Burrows & Tylman 1999; Barnett et al. 2000; Dorsch et al. 2004; Cayley 2005; Gruppen et al. 2005), we have helped students to find quality information in a streamlined fashion. Of course, students still need to know how to search for original articles when previously evaluated resources are unavailable or do not directly address their patient's unique situations, however since busy physicians may not find the time for routine MEDLINE searches (Cook et al. 1997; Guyatt 2000), it is important to equip our students with the skills to use these pre-appraised resources.

Our study findings should be interpreted in the context of their limitations. First, our work at a single institution limits our ability to generalize our findings to other institutions. However, other schools could easily adopt a similar short-workshop approach. Second, without a control group, it is difficult to know whether the improvements demonstrated by students were due to the workshop, to repeated exposure to the survey, or other unexplained factors related to participating in the clerkship. The students were exposed to the same survey three times in the 12-week clerkship and this may have impacted our results. Additionally, all students may have improved knowing that learning about and using search strategies was an important part of their clerkship work. Analysis of the effect of time of workshop on pre–post outcomes showed ongoing improvements in students’ attitudes and reported used of EBM searches whether they had participated in the workshop early in the year versus late, or early during a particular block rotation versus later. This supports a workshop effect, for students who had several months of other clinic rotations or several weeks of internal medicine before the workshop still showed improvement in post-workshop analysis. Third, the analysis of the student notes showed fewer literature citations than expected. This limits the power to detect differences in integration of EBM resources into the note. Computer log data, however, demonstrated that students were performing more literature searches and preferentially selecting more evidence-based resources over summative resources such as UpToDate and Google. Our computer log data, however, is limited to those searches performed through the EMR and thus could represent better knowledge of the EMR specific search tool. In addition, while our computer log data showed a low number of searches per student, it is similar to other studies of students’ online searching and likely represent the limitation in current systems to capture all of a given students’ information retrieval efforts (Tannery et al. 2002; Schilling et al. 2006). Fourth, our inter-rater reliability is modest and this can be a source of bias, however, our percent agreement for subjective measures was greater than 80%. Finally, while we demonstrated improvements that lasted several weeks after the intervention, longer follow-up is needed to determine if the newly acquired changes in attitudes and behaviors can be maintained over time.

In summary, we demonstrate a feasible workshop to teach literature searching skills to third-year medicine clerks. Students enjoy the workshop and report more confidence in and utilization of EBM resources, used more evidence-based resources, and better utilized these in their notes. The workshop has easily been incorporated as an ongoing part of the third-year clerkship internal medicine rotation. Future studies will be needed to address step five of Sackett's model and further evaluate the impact of EBM curricula, such as ours, on actual patient outcomes.

Acknowledgments

This study is supported in part by a GIVME Medical Education grant provided through Vanderbilt University School of Medicine and a grant from the D.W. Reynolds Foundation.

Declaration of interest: The authors have no declarations of interest to report.