Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score

ABSTRACT

Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency. Background: Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment. Approach: Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated. Results: Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72–1.00; p < .001; percentage agreement =90%–100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees. Conclusions: A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.

KEYWORDS:

• medical education
• competency-based design
• assessment
• surgery

Acknowledgments

We thank the residents and faculty of the Division of Orthopaedic Surgery at the University of Ottawa for their participation.

Disclosure statement

No potential conflict of interest was reported by the authors.