Impact on house staff evaluation scores when changing from a Dreyfus- to a Milestone-based evaluation model: one internal medicine residency program's findings

Figures & data

Fig. 1.  Diagram showing the inclusion and exclusion criteria of faculty evaluations.

Table 1 Means (standard deviations) and significant pairwise comparisons from the ANOVA results for each competency score and total scores across the four evaluations (July–September, October–December, January–March, April–June) for PGY-1, PGY-2, and PGY-3 residents evaluated using the Dreyfus and Milestone models

Competency score	Evaluation	Model	PGY-1	PGY-2	PGY-3
Patient care	1 (n=102, 25, 36)	Dreyfus	2.60 (0.86)	3.88 (0.78)^a	4.39 (0.64)^a
	1 (n=123, 40, 37)	Milestone	1.76 (0.71)^b	4.00 (0.55)^a	4.92 (0.28)^a,c
	2 (n=100, 31, 37)	Dreyfus	3.14 (0.88)^d	4.16 (0.69)^a	4.35 (0.75)^a
	2 (n=95, 37, 20)	Milestone	2.41 (0.68)^b,d	3.92 (0.43)^a	4.95 (0.22)^a,c
	3 (n=90, 22, 23)	Dreyfus	3.39 (0.91)^d	4.27 (0.46)^a	4.52 (0.51)^a
	3 (n=83, 34, 15)	Milestone	2.82 (0.50)^b,d,e	4.15 (0.44)^a	4.80 (0.41)^a
	4 (n=103, 21, 27)	Dreyfus	3.54 (0.81)^e,f	4.14 (0.57)^a	4.26 (0.59)^a
	4 (n=59, 26, 14)	Milestone	3.24 (0.54)^d,e,f	4.23 (0.43)^a	5.00 (0.00)^a
Medical knowledge	1 (n=102, 25, 36)	Dreyfus	2.60 (0.85)	3.84 (0.75)^a	4.25 (0.60)^a
	1 (n=123, 40, 37)	Milestone	1.84 (0.73)^b	3.78 (0.62)^a	4.76 (0.43)^a,c
	2 (n=100, 31, 37)	Dreyfus	3.10 (0.83)^d	4.10 (0.70)^a	4.22 (0.75)^a
	2 (n=95, 31, 20)	Milestone	2.45 (0.65)^b,d	3.95 (0.33)^a	4.90 (0.31)^a,b,c
	3 (n=90, 22, 23)	Dreyfus	3.31 (0.86)	4.18 (0.50)^a	4.43 (0.51)^a
	3 (n=83, 34, 15)	Milestone	2.82 (0.52)^b,d,e	4.09 (0.45)^a	4.87 (0.35)^a,c
	4 (n=104, 21, 27)	Dreyfus	3.50 (0.72)^e,f	4.10 (0.70)^a	4.22 (0.64)^a
	4 (n=59, 26, 14)	Milestone	3.27 (0.49)^d,e,f	4.15 (0.37)^a	4.93 (0.27)^a,c
Practice-based learning and improvement	1 (n=102, 25, 36)	Dreyfus	2.71 (0.86)	4.00 (0.58)^a	4.31 (0.58)^a
	1 (n=123, 40, 37)	Milestone	1.80 (0.65)^b	3.90 (0.50)^a	4.84 (0.37)^a,c
	2 (n=98, 30, 37)	Dreyfus	3.19 (0.82)^d	4.17 (0.53)^a	4.24 (0.76)^a,b
	2 (n=95, 31, 20)	Milestone	2.37 (0.65)^b,d	3.92 (0.49)^a	4.90 (0.31)^a,c
	3 (n=90, 22, 22)	Dreyfus	3.41 (0.97)^d	4.18 (0.50)^a	4.55 (0.51)^a
	3 (n=83, 34, 15)	Milestone	2.72 (0.53)^b,d,f	4.09 (0.51)^a	4.80 (0.41)^a
	4 (n=104, 21, 27)	Dreyfus	3.54 (0.82)^d,e	4.14 (0.73)^a	4.15 (0.60)^a,b
	4 (n=59, 26, 14)	Milestone	3.10 (0.36)^b,d,e,f	4.19 (0.49)^a	4.93 (0.27)^a,c
Interpersonal and communication skills	1 (n=102, 25, 36)	Dreyfus	3.25 (1.03)	4.28 (0.61)^a	4.53 (0.56)^a
	1 (n=123, 40, 37)	Milestone	2.11 (0.82)^b	4.20 (0.56)^a	4.81 (0.40)^a
	2 (n=100, 30, 37)	Dreyfus	3.78 (0.95)^d	4.47 (0.57)^a	4.38 (0.76)^a
	2 (n=95, 37, 20)	Milestone	2.61 (0.79)^b,d	3.95 (0.57)^a	4.90 (0.31)^a,c
	3 (n=89, 22, 22)	Dreyfus	3.85 (1.08)^d	4.32 (0.48)	4.55 (0.51)^a
	3 (n=83, 34, 15)	Milestone	3.01 (0.65)^b,d,e	4.24 (0.50)^a	4.87 (0.35)^a
	4 (n=103, 21, 27)	Dreyfus	3.91 (0.86)^d	4.43 (0.60)	4.26 (0.59)
	4 (n=59, 26, 14)	Milestone	3.49 (0.68)^b,d,e,f	4.38 (0.50)^a	4.93 (0.27)^a
Systems-based practice	1 (n=102, 25, 36)	Dreyfus	2.73 (0.94)	3.92 (0.57)^a	4.36 (0.54)^a,c
	1 (n=123, 40, 37)	Milestone	1.93 (0.67)^b	3.93 (0.53)^a	4.65 (0.48)^a,c
	2 (n=100, 31, 37)	Dreyfus	3.18 (0.90)^d	4.16 (0.58)^a	4.27 (0.80)^a
	2 (n=95, 37, 20)	Milestone	2.47 (0.63)^b,d	3.84 (0.44)^a	4.80 (0.41)^a,c
	3 (n=89, 21, 23)	Dreyfus	3.44 (1.03)^d	4.29 (0.46)^a	4.57 (0.51)^a
	3 (n=83, 34, 15)	Milestone	2.75 (0.51)^b,d	4.09 (0.51)^a	4.73 (0.46)^a
	4 (n=104, 21, 27)	Dreyfus	3.57 (0.76)^d,e	4.19 (0.75)^a	4.22 (0.64)^a
	4 (n=59, 26, 14)	Milestone	3.15 (0.45)^b,d,e,f	4.19 (0.40)^a	4.93 (0.27)^a,c
Professionalism	1 (n=102, 25, 36)	Dreyfus	3.19 (0.54)	3.60 (0.50)	3.69 (0.47)^a
	1 (n=123, 40, 37)	Milestone	2.20 (0.90)^b	4.18 (0.59)^a,b	4.92 (0.28)^a,b,c
	2 (n=98, 31, 37)	Dreyfus	3.39 (0.59)	3.52 (0.57)	3.54 (0.60)
	2 (n=95, 37, 20)	Milestone	2.75 (0.77)^b,d	4.03 (0.60)^a,b	5.00 (0.00)^a,b,c
	3 (n=90, 22, 23)	Dreyfus	3.36 (0.68)	3.55 (0.51)	3.70 (0.47)
	3 (n=83, 34, 15)	Milestone	3.07 (0.66)^d,e	4.26 (0.51)^a,b	4.80 (0.41)^a,b
	4 (n=104, 21, 27)	Dreyfus	3.49 (0.52)	3.76 (0.44)	3.41 (0.64)
	4 (n=59, 26, 14)	Milestone	3.49 (0.63)^d,f	4.38 (0.50)^a,b	4.93 (0.27)^a,b
Overall	1 (n=102, 25, 36)	Dreyfus	2.64 (0.91)	3.88 (0.73)^a	4.42 (0.60)^a,c
	1 (n=123, 40, 37)	Milestone	1.88 (0.74)^b	4.00 (0.51)^a	4.86 (0.35)^a,c
	2 (n=98, 30, 37)	Dreyfus	3.16 (0.84)^d	4.19 (0.70)^a	4.35 (0.75)^a
	2 (n=95, 37, 20)	Milestone	2.52 (0.63)^b,d	3.92 (0.43)^a	4.95 (0.22)^a,c
	3 (n=88, 20, 22)	Dreyfus	3.45 (0.96)^d,e	4.25 (0.55)^a	4.57 (0.51)^a
	3 (n=83, 34, 15)	Milestone	2.86 (0.57)^b,d	4.18 (0.46)^a	4.80 (0.41)^a
	4 (n=103, 21, 27)	Dreyfus	3.59 (0.77)^b,d	4.14 (0.57)^a	4.27 (0.60)^a
	4 (n=59, 26, 14)	Milestone	3.19 (0.47)^b,d,e,f	4.23 (0.43)^a	5.00 (0.00)^a,c

^aWithin model, significantly different from PGY-1; ^bwithin PGY year and month of evaluation, significantly different from Dreyfus; ^cwithin model, significantly different from PGY-2; ^dwithin PGY year and model, significantly (p<0.001) different from evaluation 1; ^efrom evaluation 2; ^ffrom evaluation 3.

Table 2 MANOVA results for each of the competency and total scores

Scores	Model	PGY year	Evaluation	Model*PGY	Model*Evaluation	PGY*Evaluation	Model*PGY*Evaluation
Patient care	(1,948)=1.50, p=0.22	(2,948)=515.75, p<0.0001	(3,948)=22.31, p<0.0001	(2,948)=42.00, p<0.0001	(3,948)=1.88, p=0.13	(6,948)=18.03, p<0.0001	(6,948)=2.35, p=0.03
Medical knowledge	(1,949)=0.11, p=0.74	(2,949)=542.21, p<0.0001	(3,949)=28.35, p<0.0001	(2,949)=42.03, p<0.0001	(3,949)=2.17, p=0.09	(6,949)=13.49, p<0.0001	(6,949)=1.49, p=0.18
Practice-based learning and improvement	(1,945)=5.41, p=0.02	(2,945)=574.19, p<0.0001	(3,945)=19.76, p<0.0001	(2,945)=61.12, p<0.0001	(3,945)=3.19, p=0.02	(6,945)=13.97, p<0.0001	(6,945)=1.66, p=0.13
Interpersonal and communication skills	(1,945)=17.55, p<0.0001	(2,945)=290.32, p<0.0001	(3,945)=11.19, p<0.0001	(2,945)=51.75, p<0.0001	(3,945)=3.56, p=0.01	(6,945)=10.96, p<0.0001	(6,945)=1.81. p=0.09
System-based practice	(1,947)=9.11, p=0.0026	(2,947)=539.01, p<0.0001	(3,947)=22.30, p<0.0001	(2,947)=45.18. p<0.0001	(3,947)=2.95, p=0.03	(6,947)=10.30, p<0.0001	(6,947)=1.47, p=0.19
Professionalism	(1,947)=119.49, p<0.0001	(2,947)=246.54, p<0.0001	(3,947)=8.66, p<0.0001	(2,947)=135.35, p<0.0001	(3,947)=5.64, p=0.0008	(6,947)=11.42, p<0.0001	(6,947)=5.16, p<0.0001
Overall	(1,942) = 1.68, p=0.20	(2,942) = 501.56, p<0.0001	(3,942) = 21.13, p<0.0001	(2,942) = 40.37, p<0.0001	(3,942) = 1.84, p=0.14	(6,942) = 14.90, p<0.0001	(6,942) = 1.91, p=0.08

Fig. 2.  House staff evaluation scores done by full-time day faculty across the competencies patient care and medical knowledge and overall score across both evaluation models.

Fig. 3.  House staff evaluation scores done by full-time day faculty across the competencies practice-based learning and improvement, interpersonal and communication skills, systems-based practice and professionalism across both evaluation models.