Use of an objective structured clinical exam (OSCE) for early identification of communication skills deficits in interns

Abstract

Introduction: There is limited information about whether OSCE during GME orientation can identify trainee communication deficits before these become evident via clinical performance evaluations.

Methods: Ninety-seven interns matriculating to eight residency programs in six specialties at four hospitals participated in a nine-station communication skills OSCE. Ratings were based on the “Kalamazoo, adapted” communication skills checklist. Possible association with intern performance evaluations was assessed by repeated-measures logistic regression and ROC curves were generated.

Results: The mean OSCE score was 4.08 ± 0.27 with a range of 3.3–4.6. Baseline OSCE scores were associated with subsequent communication concerns recorded by faculty, based on 1591 evaluations. A 0.1-unit decrease in the OSCE communication score was associated with an 18% higher odds of being identified with a communication concern by faculty evaluation (odds ratio 1.18, 95% CI 1.01–1.36, p = 0.034). ROC curves did not demonstrate a “cut-off” score (AUC= 0.558). Non-faculty evaluators were 3–5 times more likely than faculty evaluators to identify communication deficits, based on 1900 evaluations.

Conclusions: Lower OSCE performance was associated with faculty communication concerns on performance evaluations; however, a “cut-off” score was not demonstrated that could identify trainees for potential early intervention. Multi-source evaluation also identified trainees with communication skills deficits.

Acknowledgements

Susan E. Farrell, MD, EdM, for development of the OSCE cases and Anne Rigg, Partners HealthCare, GME Project Manager for evaluation management.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Funding

The study was funded by The Risk Management Foundation of the Harvard Medical Institutions, Inc. (CRICO/RMF). This work was conducted with support from Harvard Catalyst | The Harvard Clinical and Translational Science Center (National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health Award UL1 TR001102) and financial contributions from Harvard University and its affiliated academic healthcare centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic healthcare centers, or the National Institutes of Health.

Notes on contributors

Dr Mary Ellen J. Goldhamer, MD, MPH, is Education Specialist, Partners Office of Graduate Medical Education, Partners HealthCare System and Instructor of Medicine, Division of General Internal Medicine at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.

Ms Amy Cohen, EdM, is Assistant Fellowship Director, Harvard Medical School Academy Fellowship in Medical Education Research and Director of Data Analytics and Instructor, Department of Health Policy and Management at the Harvard T.H. Chan School of Public Health, Boston, Massachusetts.

Ms Michelle Brooks, MS, RD, was Research Computing Core Administrative Manager, EDC Programmer and Clinical Consultant, Enterprise Research IS Group, Partners HealthCare, Boston, Massachusetts. At the time of this submission, Ms. Brooks is Senior Clinical Data Manager, Alnylam Pharmaceuticals, Cambridge, Massachusetts.

Dr Eric A. Macklin, PhD, is Instructor in Medicine, Massachusetts General Hospital Biostatistics Center and Harvard Medical School, Boston, Massachusetts.

Dr John Patrick T. Co, MD, MPH, is Director for Graduate Medical Education at the Partners HealthCare System and Associate Professor of Pediatrics at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.

Dr Debra Weinstein, MD, is Vice President for Graduate Medical Education at the Partners HealthCare System, and Associate Professor of Medicine at Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts.