Impact of tablet-scoring and immediate score sheet review on validity and educational impact in an internal medicine residency Objective Structured Clinical Exam (OSCE)

Abstract

Introduction: The Objective Structured Clinical Examination (OSCE) is used globally for formative and summative purposes. The objective of this study was to examine the impact of tablet-scoring on sources of validity evidence for an Internal Medicine residency OSCE.

Methods: We compared paper-scored OSCEs from 2014 to tablet-scored OSCEs in 2015 for missing data, amount of comments, and time to pass/fail decision. We then examined in 2016 the impact on learning of showing residents their score sheets and asking them to write an action plan immediately after an OSCE.

Results: Tablet-scoring significantly reduced stations with missing data from 1.8 to 0.2%, and stations without comments from 42 to 28% with an increase in word count per comment. Time to official results reduced from 3 weeks to 12 h with tablet-scoring. Residents who wrote a learning plan after reviewing their OSCE score sheets were more likely (with medium to large effect sizes) to pursue further studying and/or change their behavior (e.g. history taking or physical examination) in the clinical environment.

Conclusions: OSCE tablet-scoring improved many sources of validity evidence, especially educational impact with timeliness of feedback supporting a change in behavior, a hard to achieve goal of educational interventions.

Acknowledgments

Dr. Daniels would like to acknowledge the Alberta Academic Medicine and Health Services Program for its financial support.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Validity: Is a unitary concept, with construct validity as the whole of validity. Assessments are not valid or invalid, rather assessment scores have more (or less) validity evidence to support the proposed interpretations. Validity requires multiple sources of evidence to support or refute meaningful score interpretation. Validity is always approached as hypothesis. Validation research uses theory, data and logic to argue for or against specific score interpretations.

Definition is taken from the Key Learning Points in the following article:

Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–837.

Additional information

Funding

This work was supported by the University of Alberta’s Teaching and Learning Enhancement Fund with Dr. Lai as Principal Investigator and Drs. Daniels and Hillier as Co-Investigators.

Notes on contributors

Vijay J. Daniels

Vijay Daniels, MD MHPE, is an Associate Professor in the Department of Medicine at the University of Alberta. He is the Assistant Dean of Assessment for the MD Program and the Associate Program Director of Assessment for the Internal Medicine Residency Program.

Alicia C. Strand

Alicia Strand, MD, is a fourth year resident in the General Internal Medicine subspecialty residency program at the University of Alberta.

Hollis Lai

Hollis Lai, PhD, is an Associate Professor in the School of Dentistry at the University of Alberta. He is the Assistant Dean, Education Quality and Accreditation for the Faculty of Medicine and Dentistry.

Tracey Hillier

Tracey Hillier, MD MEd, is an Assistant Professor in the Department of Radiology and Diagnostic Imaging at the University of Alberta. She is the Associate Dean of Undergraduate Medical Education.