Probing the effect of OSCE checklist length on inter-observer reliability and observer accuracy

Figures & data

Fig. 1.  Boxplots of observer accuracies and student grades.

Table 1 Distribution of observer accuracy and student grades for four OSCE stations as mean raw scores (% mean)

Station	20-item (accuracy^a)	40-item (accuracy^a)	20-item (grade^b)	40-item (grade^b on core)	40-item (grade^b on all)
Abdominal pain	17.4 (87%)	33.5 (84%)	12.3 (62%)	11.4 (57%)	22.7 (57%)
Jaundice	18.2 (91%)	35.8 (90%)	12.4 (62%)	11.8 (59%)	23 (57%)
Knee pain	18.6 (93%)	35.6 (89%)	13.8 (69%)	14.1 (70%)	24.2 (61%)
Vertigo	14.3 (72%)	30.3 (76%)	10.2 (51%)	9.5 (48%)	18.9 (47%)

^aAccuracy is a comparison of the raters’ scored items and the gold-standard, that is, how close the rater came to the “correct” grades.

^bGrade is the final grade on the OSCE that the student would have received from the rater.

Table 2 Inter-observer reliability, stratified by group and number of questions

	Abdominal pain [95% CI]	Jaundice [95% CI]	Knee [95% CI]	Vertigo [95% CI]
20 Questions	0.673 [0.54, 0.82]	0.758 [0.64, 0.87]	0.723 [0.6, 0.85]	0.542 [0.4, 0.72]
40 Questions – all	0.581 [0.48, 0.7]	0.713 [0.62, 0.81]	0.645 [0.55, 0.75]	0.432 [0.33, 0.56]
40 Questions – core	0.573 [0.43, 0.74]	0.761 [0.64, 0.87]	0.781 [0.67, 0.89]	0.436 [0.3, 0.63]
p-Value: 20 vs. 40 (all)	0.655	0.56	0.619	0.724
p-Value: 20 vs. 40 (core)	0.668	0.508	0.581	0.716
p-Value: 40 (all) vs. 40 (core)	0.515	0.566	0.695	0.509