Exploring the relationship between examiners’ memories for performances, domain separation and score variability

Abstract

Background: OSCE examiners’ scores are variable and may discriminate domains of performance poorly. Examiners must hold their observations of OSCE performances in “episodic memory” until performances end. We investigated whether examiners vary in their recollection of performances; and whether this relates to their score variability or ability to separate disparate performance domains.

Methods: Secondary analysis was performed on data where examiners had: 1/scored videos of OSCE performances showing disparate student ability in different domains; and 2/performed a measure of recollection for an OSCE performance. We calculated measures of “overall-score variance” (the degree individual examiners’ overall scores varied from the group mean) and “domain separation” (the degree to which examiners separated different performance domains). We related these variables to the measure of examiners’ recollection.

Results: Examiners varied considerably in their recollection accuracy (recognition beyond chance −5% to +75% for different examiners). Examiners’ recollection accuracy was weakly inversely related to their overall score accuracy (R = −0.17, p < 0.001) and related to their ability to separate domains of performance (R = 0.25, p < 0.001).

Conclusions: Examiners vary substantially in their memories for students’ performances which may offer a useful point of difference to study processing and integration phases of judgment. Findings could have implication for the utility of feedback.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Episodic memory: A component of long-term memory that that enables human beings to remember past experiences. Storing information in episodic memory involves attention, encoding and processing and then later retrieval to relive the memories. A variety of processes can interfere with these processes, so episodic memory is incompletely accurate (Tulving 2002).

Additional information

Funding

PY received a “Starter Grant for New Lecturers” from the Academy of Medical Sciences in 2013 which supported the data collection for Yeates et al. (2017).

Notes on contributors

Natasha Cleaton

Natasha Cleaton, is a Year 4 medical student at Keele University School of Medicine. She is interested in assessment of Medical Students competence and practical methods to improve the reliability of this process.

Peter Yeates

Peter Yeates, MRCP, PhD, is a lecturer in medical education research at Keele University School of Medicine, and a consultant in Acute and Respiratory Medicine at Pennine Acute Hospitals NHS Trust. He is interested in assessor cognition and innovative ways of improving the dependability of OSCEs.

Gareth McCray

Gareth McCray, PhD, is a medical statistician at the Institute for Primary Care and Health Sciences, Keele University, and a fellow of the Royal Statistical Society. His main interests are the measurement of child development, psychometric modeling of judgement data, and creation of short-form PROMS.