Reliability of the Behaviorally Anchored Rating Scale (BARS) for assessing non-technical skills of medical students in simulated scenarios

ABSTRACT

Purpose

Caring for critically ill patients requires non-technical skills such as teamwork, communication, and task management. The Behaviorally Anchored Rating Scale (BARS) is a brief tool used to assess non-technical skills. The investigators determined inter- and intra-rater reliability of the BARS when used to assess medical students in simulated scenarios.

Method

The investigators created simulation scenarios for medical students during their pediatric clerkship. Content experts reviewed video recordings of the simulations and assigned BARS scores for four performance components (Situational Awareness, Decision-Making, Communication, and Teamwork) for the leader and for the team as a whole. Krippendorff’s alpha with ordinal difference was calculated to measure inter- and intra-rater reliability.

Results

Thirty medical students had recordings available for review. Inter- and intra-rater reliability for performance components were, respectively, Individual Situational Awareness (0.488, 0.638), Individual Decision-Making (0.529, 0.691), Individual Communication (0.347, 0.473), Individual Teamwork (0.414, 0.466), Team Situational Awareness (0.450, 0.593), Team Decision Making (0.423, 0.703), Team Communication (0.256, 0.517), and Team Teamwork (0.415, 0.490).

Conclusions

The BARS demonstrated limited reliability when assessing medical students during their pediatric clerkship. Given the unique needs of this population, a modified or new objective scoring system for assessing non-technical skills may be needed for medical students.

KEYWORDS:

• Behaviorally Anchored Rating Scale
• crisis resource management
• medical student education
• non-technical skills
• simulation

Acknowledgments

The authors would like to acknowledge the Vanderbilt Center for Experiential Learning and Assessment (CELA) and its director, Dr. Arna Banerjee, for their support of this project. The authors thank the medical students at Vanderbilt University School of Medicine for their enthusiasm in supporting research into further medical education.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Ethical review

The Vanderbilt University Medical Center Institutional Review Board deemed this as exempt from oversight as it was not considered human subjects research (IRB #190,652, 5/1/2019).

Additional information

Funding

This work was supported by the Vanderbilt Institute for Clinical and Translational Research under grant VR54652 and the National Center for Advancing Translational Sciences CTSA award No. UL1 TR002243. Its contents are solely the responsibility of the authors and do not necessarily represent official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.