Reliability of the Behaviorally Anchored Rating Scale (BARS) for assessing non-technical skills of medical students in simulated scenarios

Figures & data

Table 1. Demographic characteristics of 30 second-year medical students participating in pediatric emergency simulation curriculum

Characteristic
Age, years: mean (SD)	24.4 (1.6)
Gender, %
Female	56.7
Male	43.3
Race, %
White	60.0
Asian	16.7
Black	3.3
White and Asian	10.0
White and Hispanic	3.3
Other	6.7
Intended Career Field, %
Anesthesiology	3.3
Emergency Medicine	3.3
Internal Medicine	13.3
Obstetrics and Gynecology	3.3
Pediatrics	16.7
Surgery	10.0
Undecided	50.0
Witnessed a code or resuscitation, %
Yes	30.0
No	70.0
Training prior to medical school, %
EMT	13.3
EMT and CPR Instructor	3.3
Nurse	3.3
Nurse Aid	3.3
PhD	3.3
None	73.3

Abbreviations: SD, Standard Deviation; EMT, Emergency Medical Technician; CPR, Cardiopulmonary Resuscitation; PhD, Doctor of Philosophy

Table 2. Summary of rated items on a scale from 1 to 9 on the Behaviorally Anchored Rating Scale (BARS) evaluating second-year medical students in simulated pediatric emergencies

BARS Domain	Mean (SD)	Median [IQR]	Median [Range]
Individual Situational Awareness	2.5 (1.2)	2.0 [2.0, 3.0]	2.0 [1.0, 7.0]
Individual Decision Making	2.4 (1.2)	2.0 [1.0, 3.0]	2.0 [1.0, 7.0]
Individual Communication	2.5 (1.2)	2.0 [2.0, 3.0]	2.0 [1.0, 7.0]
Individual Teamwork	2.4 (1.1)	2.0 [2.0, 3.0]	2.0 [1.0, 7.0]
Team Situational Awareness	2.7 (1.2)	3.0 [2.0, 3.0]	3.0 [1.0, 7.0]
Team Decision Making	2.6 (1.2)	2.0 [2.0, 3.0]	2.0 [1.0, 7.0]
Team Communication	2.7 (1.1)	3.0 [2.0, 3.0]	3.0 [1.0, 7.0]
Team Teamwork	2.6 (1.1)	2.0 [2.0, 3.0]	2.0 [1.0, 7.0]

Abbreviations: SD, Standard Deviation; IQR, Interquartile Range

Table 3. Inter- and intra-rater reliability of the Behaviorally Anchored Rating Scale (BARS) evaluating second-year medical students in simulated pediatric emergencies

BARS Domain	Inter-Rater Reliability (Krippendorff’s Alpha)	Intra-Rater Reliability (Krippendorff’s Alpha)
Individual Situational Awareness	0.488	0.638
Individual Decision Making	0.529	0.691
Individual Communication	0.347	0.473
Individual Teamwork	0.414	0.466
Team Situational Awareness	0.450	0.593
Team Decision Making	0.423	0.703
Team Communication	0.256	0.517
Team Teamwork	0.415	0.490

Table

Leader	Situational Awareness									Decision Making									Communication									Teamwork
	Poor			Medium			Excellent			Poor			Medium			Excellent			Poor			Medium			Excellent			Poor			Medium			Excellent
	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9
Team	Situational Awareness									Decision Making									Communication									Teamwork
	Poor			Medium			Excellent			Poor			Medium			Excellent			Poor			Medium			Excellent			Poor			Medium			Excellent
	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9	1	2	3	4	5	6	7	8	9