Interrater reliability in neurology objective structured clinical examination across specialties

Abstract

Purpose

To assess interrater reliability and examiners’ characteristics, especially specialty, associated with scoring of neurology objective structured clinical examination (OSCE).

Material and methods

During a neurology mock OSCE, five randomly chosen students volunteers were filmed while performing 1 of the 5 stations. Video recordings were scored by physicians from the Lyon and Clermont-Ferrand university teaching hospitals to assess students performance using both a checklist scoring and a global rating scale. Interrater reliability between examiners were assessed using intraclass coefficient correlation. Multivariable linear regression models including video recording as random effect dependent variable were performed to detect factors associated with scoring.

Results

Thirty examiners including 15 (50%) neurologists participated. The intraclass correlation coefficient of checklist scores and global ratings between examiners were 0.71 (CI95% [0.45-0.95]) and 0.54 (CI95% [0.28-0.91]), respectively. In multivariable analyses, no factor was associated with checklist scores, while male gender of examiner was associated with lower global rating (β coefficient = –0.37; CI 95% [–0.62–0.11]).

Conclusions

Our study demonstrated through a video-based scoring method that agreement among examiners was good using checklist scoring while moderate using global rating scale in neurology OSCE. Examiner’s specialty did not affect scoring whereas gender was associated with global rating scale.

Keywords:

• educational measurement
• objective structured clinical examination
• neurology

Acknowledgments

Quentin Barrier, Pierre-Aurélien Beuriat, Lucie Bihel, Sébastien Boulogne, Hélène Catenoix, Tae-Hee Cho, Guillaume Economos, Roger Erivan, Julia Fontaine, Florent Gobert, Julie Haesebaert, Bastien Joubert, Pierre Lardeux, André Lecoanet, Alexandra Montavont, Norbert Nighoghossian, Elodie Ong, Cécile Plumereau, Matteo Pozzi, Thomas Ritzenthaler, Nellie Della-Schiava, and Julie Traclet for their participation to the mock OSCE and/or videos scoring, Annaëlle Testud for reviewing article, the CLESS (Centre Lyonnais d‘Enseignement par la Simulation en Santé) for equipment, and hospital direction (Caroline Mons, Bertrand Cazelles) for the provision of premises.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Laura Mechtouff

Laura Mechtouff, MD, PhD, Stroke Department, Hospices Civils de Lyon, Lyon, France.

Baptiste Balanca

Baptiste Balanca, MD, PhD, Anesthesiology and intensive care medicine, Hospices Civils de Lyon, Lyon, France.

Julien Jung

Julien Jung, MD, PhD, Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France.

Julie Bourgeois-Vionnet

Julie Bourgeois-Vionnet, MD, Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France.

Chloé Dumot

Chloé Dumot, MD, Department of Vascular and Tumoral Neurosurgery, Hospices Civils de Lyon, Lyon, France.

Déborah Guery

Déborah Guery, MD, Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France.

Thiébaud Picart

Thiébaud Picart, MD, Department of neurosurgery, Hospices Civils de Lyon, France.

Lionel Bapteste

Lionel Bapteste, MD, Anesthesiology and intensive care medicine, Hospices Civils de Lyon, Lyon, France.

Geneviève Demarquay

Geneviève Demarquay, MD, PhD, Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France.

Alexandre Bani-Sadr

Alexandre Bani-Sadr, MD, Neuroradiology Department, Hospices Civils de Lyon, France.

Lucie Rascle

Lucie Rascle, MD, Stroke Department, Hospices Civils de Lyon, Lyon, France.

Yves Berthezène

Yves Berthezène, MD, PhD, Neuroradiology Department, Hospices Civils de Lyon, France.

Timothée Jacquesson

Timothée Jacquesson, MD, Department of Neurosurgery B, Hospices Civils de Lyon, Lyon, France.

Camille Amaz

Camille Amaz, MSc, Clinical Investigation Center, INSERM 1407, Hospices Civils de Lyon, Lyon, France.

Juliette Macabrey

Juliette Macabrey, MD, Research on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard Lyon 1, Lyon, France.

Inès Ramos

Inès Ramos, MSc, Research on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard Lyon 1, Lyon, France.

Marie Viprey

Marie Viprey, MD, PhD, Research on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard Lyon 1, Lyon, France.

Gilles Rode

Gilles Rode, MD, PhD, Physical medicine and rehabilitation department, Henry Gabrielle Hospital, Hospices Civils de Lyon, Saint-Genis-Laval, France.

Marion Cortet

Marion Cortet, MD, PhD, Gynecology and Obstetrics Department, Croix Rousse Hospital, Hospices Civils de Lyon, Lyon, France.