The impact of critical appraisal workshops on residents’ evidence based medicine skills and knowledge

Abstract

Objective

To assess the impact of four evidence based medicine (EBM) critical appraisal education workshops in improving residents’ EBM knowledge and skills.

Methods

The eligible participants in the workshops were 88 residents-in-training, postgraduate years one through four, rotating through the outpatient internal medicine clinic. Four EBM workshops, consisting of 3 days each (30 minutes daily), were taught by our faculty. Topics covered included critical appraisal of randomized controlled trials, case-control and cohort studies, diagnosis studies, and systematic reviews.

Results

As a program evaluation, anonymous pre-workshop and post-workshop tests were administered. Each of the four sets of tests showed improvement in scores: therapy from 58% to 77% (42% response rate), harm from 65% to 73% (38% response rate), diagnosis from 49% to 68% (49% response rate), and systematic review from 57% to 72% (30% response rate).

Conclusion

We found that teaching EBM in four short workshops improved EBM knowledge and critical appraisal skills related to the four topics.

Keywords:

• evidence based medicine
• medical education
• assessment methods
• graduate
• instructional design
• curriculum development
• curriculum evaluation

Introduction

Evidence based medicine (EBM) is defined as the integration of clinical expertise, best available evidence, and patient values into the decision-making process for patient care.¹

EBM skills are necessary for the graduating resident to provide up-to-date, high quality medical care to patients, but are lacking in the general internal medicine physician population.² Currently, there is no standard EBM curriculum that has been recommended for wide use. Nonetheless, the Accreditation Council for Graduate Medical Education (ACGME) requires all residency programs to integrate “practice-based learning and improvement” into their curriculum. Specifically, “residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to continuously improve patient care based on constant self-evaluation and lifelong learning.”³ Previous publications have described a paucity of EBM curricula.^4,5 In fact, a survey published in the Journal of General Internal Medicine in 2000, revealed that of 417 internal medicine residency programs in the United States queried regarding formal EBM teaching, only 37% of the responders offered a free-standing EBM curriculum.⁶ The scarcity of formal EBM curricula has also been published more recently among other subspecialties.^7,8

Various interventions have been evaluated including lectures, which varied in frequency and duration, online modules, and workshops, with the most common tool to deliver EBM skills in residency programs being a formal journal club.^4,5,7,8 Although valuable, many journal clubs lack consistent delivery of EBM skills.⁹ The workshop format as a teaching method has been shown in prior studies to be an efficient educational delivery method of EBM because of the small group structure that fosters frequent interaction between instructor and learner.^10–12

In many studies identified, assessment of skills demonstrated an improvement, however extensive time was required and gaps remained.¹³

This study aimed to assess the impact of four short EBM critical appraisal workshops, as a part of an EBM curriculum, in improving residents’ EBM knowledge and skills.

Methods

Our study was conducted at Beaumont Health, Royal Oak, Michigan, from July 2014 to June 2015 with exempt status from the Institution Review Board. The study was exempt under two categories: 1) instructional strategies in established educational settings, and 2) educational tests unlinkable to individuals and no risks from disclosure. The target population was residents rotating through the outpatient internal medicine resident clinic (total of 88 resident physicians) and included categorical internal medicine residents (total of 60 residents), internal medicine-pediatric residents (total of 16 residents), and transitional-year residents (total of 12 residents). The majority of participants were internal medicine residents, who rotate every fourth month through the ambulatory clinic for the entire calendar month. Resident level of training ranged from postgraduate year one to year four.

Starting in July 2013 a formal EBM curriculum was first implemented in the internal medicine residency program at our institution.¹⁴ Our curriculum integrated five specific interventions that aimed at covering all five steps of the EBM cycle: ask, acquire, appraise, apply, and assess. Educational lectures, journal club, senior morning report, chief rounds, and short interactive resident-led critical appraisal lectures were the teaching tools used in our curriculum at its onset. Based on the Kirkpatrick model of training evaluation, we simultaneously began our evaluation of our EBM program by assessing residents’ attitudes and beliefs.¹⁵ In this preliminary evaluation, the results of the implementation of the EBM curriculum were promising and suggested value to the residents as learners. Those results served as the impetus for the EBM faculty to further collaborate with residents in ongoing improvement of the EBM learning experiences with the goal of advancing the quality of patient care.

In our current study, we aimed to assess the residents’ EBM knowledge and skills acquired through an additional four educational workshops taught by EBM faculty, which focused on teaching critical appraisal of four topics: therapy, harm, diagnosis, and systematic review studies.

During the year, our EBM faculty taught 16 separate EBM educational workshops. The educational workshops were spread throughout the year to give the opportunity to every rotating resident to participate and were 90 minutes in length, delivered in 30-minute slots over 3 days. Each of the four workshops were part of the outpatient didactic curriculum and were repeated every month, for 4 months in a row. Each workshop was taught in a small group of 5–12 residents that promoted an interactive environment and allowed time for questions when topics were not completely understood. The four workshops reviewed critical appraisal of randomized controlled trials (therapy), case-control and cohort studies (harm), studies on diagnostic tests, and systematic reviews. Each 3-day workshop started with a patient case. The EBM faculty member selected an article relevant to the patient presentation. Throughout the 3 days the article was critically appraised via interactive discussion between the residents and the EBM faculty member. The JAMA Evidence¹ textbook was used to guide the curriculum, as were the associated critical appraisal worksheets. Each workshop addressed the validity of the paper, the results, and ended with a discussion of the applicability of the evidence to the specific patient scenario, as depicted in the workshop schematic in Figure 1. Residents were also taught several fundamental EBM concepts (Box 1). Workshop attendance was voluntary for residents rotating in the internal medicine clinic. The test was administered before and after each educational workshop and the participation was voluntary and anonymous. The workshop materials including introductory case scenarios and the multiple-choice tests used in the program evaluation can be found in the Supplementary materials.

Figure 1 Workshop schematic.

A multiple-choice test was developed for each of the four topics based on the intended learning outcomes for each workshop. Other previously validated tools for evaluating EBM knowledge and skills exist.¹⁶ However, the evaluation tool used in this study was synthesized de novo for two major reasons: first, for ease of administration and second, to allow the assessment tool to mirror the intended learning outcomes for each workshop. Some questions included in the tests were taken from the Berlin questionnaire.¹⁷ These were administered as pre-workshop and post-workshop tests to the resident participants to assess the impact of the workshops in improving EBM knowledge and critical appraisal skills. Specific aims of the assessment tool included the residents’ ability to appreciate the validity of each of the aforementioned study designs and to interpret the relevant statistics of those study types.

Box 1 Intended learning outcomes

Major topics:

• Critical appraisal of therapy studies

• Critical appraisal of harm studies

• Critical appraisal of diagnosis studies

• Critical appraisal of systematic reviews

Sub-topics:

• Concealment

• Intention to treat

• Relative risk

• Number needed to treat

• Surveillance bias

• Reference standard

• Likelihold ratios

• Publication bias

• Funnel plots

• Forest plots

• Heterogeneity

Descriptive data were collected from the pre- and post-test results for the entire study sample. Continuous variables were reported as median scores and interquartile range (IQR). Categorical data were reported as a percentage.

Results

Of 88 eligible residents, a total of 37, 33, 43, and 26 residents participated in the pre-test for therapy, harm, diagnosis, and systematic review workshops, which represented a participation rate of 42%, 38%, 49%, and 30% respectively. A total of 41, 35, 40, and 25 residents participated in the post-test for therapy, harm, diagnosis, and systematic review workshops, which represented a participation rate of 47%, 40%, 45%, and 28% respectively. Our study participants were mainly internal medicine residents (more than 70%), and mainly postgraduate year one (more than 40%) (Table 1).

Table 1 Participants’ demographics per workshop

Baseline characteristics	Therapy, n (%)		Harm, n (%)		Diagnosis, n (%)		Systematic review, n (%)
	Pre 37	Post 41	Pre 33	Post 34	Pre 43	Post 39	Pre 26	Post 25
PGY
1	− 19 (51)	− 22 (53)	− 14 (42)	− 14 (41)	− 18 (42)	− 20 (51)	− 18 (42)	− 13 (52)
2	− 7 (19)	− 8 (20)	− 9 (27)	− 12 (35)	− 11 (26)	− 11 (28)	− 11 (26)	− 6 (24)
3	− 11 (30)	− 11 (27)	− 9 (27)	− 7 (21)	− 12 (28)	− 8 (21)	− 12 (28)	− 5 (20)
4	− 0	− 0	− 1 (3)	− 1 (3)	− 2 (4)	− 0	− 2 (4)	− 1 (4)
Program
IM	− 33 (89)	− 39 (95)	− 31 (94)	− 28 (82)	− 40 (93)	− 34 (87)	− 20 (74)	− 18 (72)
Transitional	− 3 (8)	− 2 (5)	− 1 (3)	− 2 (6)	− 1 (2)	− 2 (5)	− 5 (19)	− 5 (20)
Med/Peds	− 1 (3)	− 0	− 1 (3)	− 4 (12)	− 2 (5)	− 7 (8)	− 2 (7)	− 2 (8)

Notes: Pre represents pre-workshop test; Post represents post-workshop test. The values in the column headings show how many residents answered that specific test.

Abbreviations: PGY, postgraduate year; IM, internal medicine; Med/Peds, internal medicine-pediatrics.

The therapy survey consisted of eight questions. The pre-workshop median score was 5 (IQR 3–6), and the post- workshop median score was 6 (IQR 6–7). Table 2 shows the following: improvement was seen in the following concepts: level of evidence (49% to 56%), concealment (35% to 83%), intention to treat principle (57% to 73%), relative risk (46% to 71%), and number needed to treat calculation (57% to 84%). The harm survey consisted of nine total questions. The pre-workshop median score was 6 (IQR 5–7). The post-workshop median score was 6 (IQR 6–7). Improvement was seen in the following categories: case-control design (21% to 33%), cohort design (23% to 30%), case-control validity (9% to 12%), surveillance bias (15% to 18%), and odds ratio (23% to 32%). The diagnosis survey consisted of ten total questions. The pre-workshop median score was 5 (IQR 4–6). The post-workshop median score was 7 (IQR 6–8). Improvement was seen in the following categories: pre-test probability (31% to 37%), design validity (23% to 29%), recognizing reference standard (16% to 25%), likelihood ratios (22% to 28%), and use of likelihood ratios (24% to 36%). The systematic review survey consisted of ten total questions. The pre-workshop median score was 6 (IQR 4–6). The post-workshop median score was 8 (IQR 6–8). Improvement was seen in all categories except the concept of reproducibility, most notably in the concept of meta-analysis content (48% to 72%), literature search (50% to 76%), funnel plots (15% to 64%), heterogeneity (63% to 88%), forest plots 44% to 76%), and confidence interval (89% to 100%).

Table 2 Median pre- and post-workshop test scores and individual question results

Test questions	Pre-workshop	Post-workshop
Therapy	N = 37	N = 41
Median (IQR)	5 (3–6)	6 (6–7)
Q1 Level of evidence	18 (49)	23 (56)
Q2 Design validity	35 (95)	38 (93)
Q3 Concealment	13 (35)	34 (83)
Q4 Intention to treat	21 (57)	30 (73)
Q5 Confidence interval	34 (92)	38 (93)
Q6 Relative risk	17 (46)	29 (71)
Q7 NNT calculation	21 (57)	35 (84)
Q8 NNT concept	13 (35)	26 (63)
Harm	N = 33	N = 35
Median (IQR)	6 (5–7)	6 (6–7)
Q1 Level of evidence	27 (82)	29 (83)
Q2 Case-control design	21 (64)	32 (91)
Q3 Cohort design	23 (68)	30 (86)
Q4 Cohort validity	33 (100)	34 (97)
Q5 Case-control validity	9 (27)	12 (34)
Q6 Surveillance bias	15 (45)	18 (51)
Q7 Odds ratio	23 (70)	32 (91)
Q8 Reasoning	24 (72)	28 (80)
Q9 Reasoning study design	19 (58)	16 (46)
Diagnosis	N = 43	N = 40
Median (IQR)	5 (4–6)	7 (6–8)
Q1 Pre-test probability	31 (72)	37 (95)
Q2 Concept pre-test probability	18 (42)	16 (41)
Q3 Screening	7 (16)	7 (18)
Q4 Design validity	22 (51)	29 (74)
Q5 Reference standard	16 (37)	25 (64)
Q6 Likelihood ratio (LR)	22 (51)	28 (72)
Q7 Conclusive LR	10 (23)	23 (59)
Q8 Inconclusive LR	29 (67)	30 (77)
Q9 Positive predictive value	30 (86)	34 (87)
Q10 Apply LR	24 (67)	36 (92)
Systematic review	N = 27	N = 25
Median (IQR)	6 (4–6)	8 (6–8)
Q1 Concept SR	19 (70)	19 (76)
Q2 Meta-analysis content	13 (48)	18 (72)
Q3 Sensible question	13 (48)	14 (56)
Q4 Literature search	16 (59)	19 (76)
Q5 Reproducibility	8 (30)	8 (32)
Q6 Validity	20 (74)	20 (80)
Q7 Funnel plots	4 (15)	16 (64)
Q8 Heterogeneity	17 (63)	22 (88)
Q9 Forest plots	12 (44)	19 (76)
Q10 Confidence interval	24 (89)	25 (100)

Note: Data shown as n (%) unless specified as median (interquartile range [IQR]).

Abbreviations: NNT, number needed to treat; Q, question number; SR, systematic review.

Conclusion and discussion

We were pleased to find that over 90% of the participants, before the workshops, were able to appreciate the validity of a therapy and cohort study design and were knowledgeable of the meaning and interpretation of confidence intervals. On the other hand, topics that the residents had low understanding of before the workshops were also found: concealment, case-control validity, screening, conclusive likelihood ratios, and funnel plot interpretation.

EBM topics that were well-explained and understood, as revealed in the improvement in post-test scores for those questions, include concealment and funnel plots. Concepts which remained poorly understood include case-control validity, screening, and conclusive likelihood ratios. This detailed analysis allowed us to adapt our curriculum and teaching methods accordingly. Retention of knowledge and implementation in practice will require further investigation.

Interestingly, we noticed that the pre-test median score for all four teaching workshops was at least five, implying that the residents had a solid base of EBM concepts knowledge before taking the critical appraisal workshops. This could be explained by previous EBM training during medical school or for the postgraduate year two and three, previous exposure to the EBM concepts in the first year when our EBM curriculum was implemented. This could also be explained by the fact that only the residents who were motivated and had some EBM knowledge and were comfortable with EBM concepts participated in the workshops; and possibly the residents who needed the training the most, who had very limited EBM knowledge and skills, were not surveyed.

Overall, the results suggest valuable implications for EBM faculty in all disciplines to further collaborate with residents to continue to improve the EBM learning experiences across residencies, in the hope of advancing the medical community’s quality of patient care. Our critical appraisal workshops meet the ACGME requirements and can be an adjunct to the more traditional EBM teaching tools, like journal club. The necessary time and the structure needed to learn all the EBM skills, and knowledge necessary to provide high-quality patient care remain unknown. We, however, were able to teach critical appraisal of a study design in a total of 90 minutes, including the pre-workshop and post-workshop tests. Our literature search revealed the most efficient workshop published to date required 2 hours a week for 7 consecutive weeks, showing an improvement in EBM skills as well.¹⁸ Our intervention also took into account that adult learners tend to retain information better if the content taught is interesting or relates to previous experience (this is why all our workshops started and ended with a patient case), and also if the teaching is done in a small group where interaction and questions are nurtured.^10–12

Our study strengths consist of a low risk of reporting bias as the tests were anonymous, as well as having a low risk of responder bias as everybody participating in the workshops took the pre- or post-tests. Our intervention is innovative and easily generalizable to any other residency program as it is very short, focused, and efficient. These 90-minute workshops can be implemented independently or as part of a wider EBM curriculum, as they do not rely on other parts of our curriculum to be complete. This program also allows for real-time feedback for the EBM instructors, allowing them to make changes to the curriculum to better address specific knowledge gaps.

Limitations in our study include small sample size, unpaired data, the lack of a control group as well as the use of an assessment tool that has not been validated. Survey bias might have had an impact on our pre-workshop scores as it is possible that the residents who participated in these workshops were self-selected from the residents who were most comfortable with the EBM skills and knowledge. Future iterations of this study will focus on improving the aforementioned design flaws.

To summarize, these EBM critical appraisal workshops, which are patient-centered and interactive, practical, and attractive to residents as learners, were found to increase the residents’ EBM skills and knowledge in this study. Beyond the short-term positive impact which was shown, we believe the virtues of our program include its: 1) structure, starting with a clinical case and ending with the applicability to the same patient case scenario, 2) the systematic teaching of the fundamental EBM concepts needed to determine the risk of bias of a certain study design, and 3) the small group teaching environment that promoted active interaction.^10–12 Undoubtedly, more research should be done to better determine the ideal EBM teaching model. We hope that with the publication of this workshop, other residency programs can continue to build upon the work that was initiated with this study.

Acknowledgments

We thank Judy Boura, M.S. biostatistician, for her feedback and assistance in preparing these data.

Disclosure

The authors report no conflicts of interest in this work.