A prospective mixed-methods study evaluating the integration of an evidence based medicine curriculum into an internal medicine residency program

Figures & data

Figure 1 EBM workflow.

Figure 2 Example daily lecture schedule, EBM weekly view for in-patient and out-patient perspective.

Figure 3 Structure of EBM curriculum and the level of resident participation.

Table 1 Evidence based medicine curriculum focus group summary

Overall EBM Learning Experience	Positive: “Some aspects are very helpful”; “Everybody is so involved and enthusiastic about it”; “It has helped”; EBM is a very good concept and it’s great that they pursued this type of curriculum for us”
	Negative: “Fragmented, which makes it difficult to follow”; “Should have more structure with clear objectives”; “Needs a little more organization”
Journal Club	Positive: “It did help me like break it down because I had no idea how to read an article before”; “Reviewing an article is still new and having that outline of the critical appraisal is helpful”
	Negative: “Boring; needs to be separated from EBM”; “there’s just not enough time to like talk about the topic of the article and then to talk about how to read the article itself”
Chief Rounds	“Not necessary, honestly, really, I’m not learning anything”; “I don’t really know exactly what the point of Chief Rounds is”
EBM workshops	“Very helpful for me”; “Best part of the curriculum”
Do you think you’re practicing EBM	“[I] Try to, I think it is very important … I have a case, I have a patient. I go look it up (Uptodate), so in theory I do practice EBM”
Suggestions	“Teach basic statistics: RR, NNT, set aside 6 months of EBM intern lectures to avoid redundancy”; “Create database of trials of what we base standard of care on, it will probably also help us remember”; “Journal Club should be educational for us to get up to date on the knowledge … and in the beginning, if someone could break it down for us really quickly that would be better”

Table 2 Comparing correct and incorrect responses for individual questions on therapy, diagnosis, SR and harm workshops

Therapy
Question	Prop.correct – Pretest (n=25)	Prop.correct – Posttest (n=25)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	1.00	0.96	1.0	0	(0,19)
Q2	0.92	0.96	1.0	∞	(0.05, ∞)
Q3	0.40	0.80	0.006	11	(1.87, 233)
Q4	0.76	0.88	0.38	4	(0.52, 97)
Q5	0.60	0.92	0.008	∞	(1.86, ∞)
Q6	0.64	0.68	1.0	1.25	(0.34, 4.93)
Q7	0.76	0.68	0.73	0.60	(0.12, 2.46)
Q8	0.88	0.88	1.0	1	(0.11, 9.24)
Q9	0.72	0.84	0.38	4	(0.52, 97)
Q10	0.64	0.72	0.77	1.4	(0.42, 4.52)
Diagnosis
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.38	0.75	0.03	∞	(1.49, ∞)
Q2	0.62	0.56	1.0	0.75	(0.15, 3.44)
Q3	0.69	0.81	0.62	3	(0.33, 77.5)
Q4	0.25	0.44	0.51	2	(0.43, 9.23)
Q5	0.19	0.50	0.12	6	(0.80, 136)
Q6	0.19	0.56	0.07	7	(1.0, 156)
Q7	0.81	1.0	0.25	∞	(0.58, ∞)
Q8	0.50	0.81	0.12	6	(0.80, 136)
Q9	0.88	0.88	1.0	1	(0.11, 9.24)
Q10	0.56	0.88	0.12	6	(0.80, 136)
Systematic review
Question	Prop.correct – Pretest (n=13)	Prop.correct – Posttest (n=13)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.92	1.00	1.00	∞	(0.05, ∞)
Q2	0.62	0.92	0.12	∞	(0.90, ∞)
Q3	0.38	0.31	1.0	0	(0, 19)
Q4	0.46	0.85	0.06	∞	(1, ∞)
Q5	0.62	0.62	1.0	1	(0.11, 9.24)
Q6	0.31	0.69	0.06	∞	(1, ∞)
Q7	0.54	1.0	0.03	∞	(1.49, ∞)
Q8	0.69	0.85	0.50	∞	(0.29, ∞)
Q9	0.62	0.92	0.12	∞	(0.90, ∞)
Harm
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q1	0.81	1.0	0.25	∞	(0.58, ∞)
Q2	0.50	1.0	0.008	∞	(1.86, ∞)
Q3	0.94	1.0	1.0	∞	(0.05, ∞)
Q4	0.88	1.0	0.50	∞	(0.29, ∞)
Q5	0.25	0.25	1.0	1	(0.24, 4.18)
Harm
Question	Prop.correct – Pretest (n=16)	Prop.correct – Posttest (n=16)	p-value (McNemar)	Odds ratio	95% CI for odds ratio
Q6	0.38	0.50	0.62	3	(0.33, 77.5)
Q7	0.81	0.94	0.50	∞	(0.29, ∞)
Q8	0.81	0.81	1.0	1	(0.11, 9.24)
Q9	0.50	0.56	1.0	1.5	(0.23, 12.1)

Note: ∞, infinity.

Abbreviation: Prop.correct, proportion correct.

Table 3 Short-term quantitative knowledge from the four EBM workshops

Workshop (n)	Total questions	Pretest- median score (IQR)	Posttest- median score (IQR)	Median Difference (IQR)	95% CI	p-value (signed-rank test)
Therapy (25)	10	8 (6–9)	8 (8–9)	1 (0–2)	0–2	0.006
Diagnosis (16)	10	6 (3–6)	7 (6–9)	1.5 (0.5–4)	0–4	0.006
Systematic Review (13)	9	4 (4–6)	7 (6–8)	2 (1–3)	1–3	0.002
Harm (16)	9	6 (5–7)	7 (7–8)	1 (0.5–2)	1–2	0.004

Figure 4 Median pre- and post- tests scores from the four short EBM workshops (therapy, diagnosis, systematic review (SR), and harm) Error bars represent inter-quartile range.

Table 4 Fresno test of competence in evidence based medicine: Pre-/Posttest scores

Questions 1–7: Subjective question from Fresno test with partial credit awarded*
Question Number	Maximum Points	Median (25th–75th) Pretest Score	Median (25th-75th)Posttest Score	p-value
1	24	18.5 (14.00–20.00	15.5 (10.00–23.00)	0.64
2	24	13 (6.00–20.00)	6 (4.00–18.00)	0.18
3	24	14 (8.00–18.00)	12 (12.00–15.00)	0.99
4	24	14 (8.00–16.00)	14 (12.00–15.00)	0.86
5	24	9.5 (5.00–10.00)	5 (5.00–10.00)	0.48
6	24	10 (5.00–18.00)	18 (18.00–24.00)	0.03
7	24	7 (5.00–10.00)	14 (9.00–18.00)	0.00
Questions 8–12: Objective questions from Fresno test with no partial credit awarded**
Question Number	Maximum Points	Correct responses (%) Pretest	Correct responses (%) Posttest	p-value
8a	4	13 (72.2%)	9 (50%)	0.17
8b	4	9 (50%)	8 (44.4%)	0.74
8c	4	9 (50%)	11 (61.1%)	0.50
8d	4	10 (55.6%)	8 (44.4%)	0.51
8e	4	5 (27.8%)	2 (11.1%)	0.02
9a	4	10 (55.6%)	14 (77.8%)	0.16
9b	4	7 (38.9%)	7 (38.9%)	1.00
9c	4	8 (44.4%)	10 (55,6%)	0.51
10	4	7 38.9%)	14 (77.8%)	0.02
11	4	3 (16.7%)	2 (16.7%)	1.00
12	4	12 66.7%)	15 (83.3%)	0.44
	Maximum Points	Median (25th–75th) Pretest Score	Median (25th–75th) Posttest Score
Overall Score	212	110.5 (96.00–124.00)	115 (100.00–130.00)	0.60

Notes: *Results displayed as median pretest and posttest score by question. Pretest and posttest were compared using Mann-Whitney U test. **Results displayed as number of residents who answered correctly and percentages. Pretest and posttests where compared using Chi-square or Fisher exact test.