Numerical versus narrative: A comparison between methods to measure medical student performance during clinical clerkships

Abstract

Background: Medical school evaluations typically rely on both language-based narrative descriptions and psychometrically converted numeric scores to convey performance to the grading committee. We evaluated inter-rater reliability and correlation of numeric versus narrative evaluations for students on their Neurology Clerkship.

Design/Methods: 50 Neurology Clerkship in-training evaluation reports completed by their residents and faculty members at the University of Rochester School of Medicine were dissected into narrative and numeric components. 5 Clerkship grading committee members retrospectively gave new narrative scores (NNS) while blinded to original numeric scores (ONS). We calculated intra-class correlation coefficients (ICC) and their associated confidence intervals for the ONS and the NNS. In addition, we calculated the correlation between ONS and NNS.

Results: The ICC was greater for the NNS (ICC = .88 (95% CI = .70–.94)) than the ONS (ICC = .62 (95% CI = .40–.77)) Pearson correlation coefficient showed that the ONS and NNS were highly correlated (r = .81).

Conclusions: Narrative evaluations converted by a small group of experienced graders are at least as reliable as numeric scoring by individual evaluators. We could allow evaluators to focus their efforts on creating richer narrative of greater value to trainees.

Disclosure statement

None of the authors have any conflicts of interest or financial disclosures influencing the contents of this paper.

Glossary

Lexical anchor: Specific term shared within a discipline that carries specific meaning to individuals within the discipline.

Notes on contributors

Josef Bartels, MD, MPH, is a Resident Physician in the Family Medicine Residency of Idaho and is interested in doctor-patient-computer communication, continuity, social medicine, and clinical performance evaluation. He envisions a practice enmeshed in free flows of clinical information via remote consultation and bundled payment models. Aside from medicine, he runs, bicycles, and plays the violin.

Christopher Mooney, PhD, MPH, is Director of Assessment and Instructor at the University of Rochester School of Medicine and Dentistry. His research centers on medical student learner assessment and the intersection of psychological, behavioral, and social factors with older adult’s health. He is a father of three and an avid cyclist and Nordic skier.

Robert Thompson Stone, MD, is Assistant Professor of Neurology and Pediatrics at the University of Rochester, co-director of the Neurology clerkship, program director of the Child Neurology Residency program. His research focuses on assessment and teaching of clinical skills in graduate and undergraduate medical education as well as overall medical student clerkship assessment.