Managing extremes of assessor judgment within the OSCE

Abstract

Context: There is a growing body of research investigating assessor judgments in complex performance environments such as OSCE examinations. Post hoc analysis can be employed to identify some elements of “unwanted” assessor variance. However, the impact of individual, apparently “extreme” assessors on OSCE quality, assessment outcomes and pass/fail decisions has not been previously explored. This paper uses a range of “case studies” as examples to illustrate the impact that “extreme” examiners can have in OSCEs, and gives pragmatic suggestions to successfully alleviating problems.

Method and results: We used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behavior are considered and discussed in detail, highlighting both the impact of individual examiner behavior and station design on subsequent judgments.

Conclusions: In complex assessment environments, institutions have a duty to maximize the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behavior is automatically just that.

Glossary

Sequential testing In a sequential testing format, all candidates sit an initial “screening” test and only those candidates who fail to demonstrate sufficient competence on this part are required to sit a supplementary test, usually of a similar size to the first part (Pell et al. 2013). Pass/fail decisions for this weaker group are made using performance across both parts of the assessment.

Pell G, Fuller R, Homer M, Roberts T. 2013. Advancing the objective structured clinical examination: sequential testing in theory and practice. Med Educ. 47(6):569–577.

Standard error of measurement All assessments are subject to measurement error (i.e. to error in the test scores). The standard error of measurement is an estimate of the size of this error, and is related to the reliability of the assessment (so small standard errors of measurement correspond to high levels of reliability and vice versa) (Streiner & Norman 2008, p. 190–193).

Streiner D, Norman G. 2008. Health measurement scales: a practical guide to their development and use. 4th ed. OUP Oxford.

Acknowledgments

Ethical approval for this study was not required – no participants were involved, and all assessment data were anonymized in advance of the analysis.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Funding

No external funding was received.

Notes on contributors

Richard Fuller, MA, FRCP, FAcadMed, is a Physician and Director of Medical Education Programmes at Leeds Institute of Medical Education. He heads the Leeds Assessment Research Group and his main research interests focus on the development and personalization of assessment, including the impact of sequential testing, workplace assessment and the use of mobile technology to facilitate learner and assessment methodology development.

Matthew Homer, BSc, MSc, PhD, CStat, is the Principal Educational Statistician at Leeds Institute of Medical Education working in both Schools of Medicine and Education. He is a key member of the Leeds Assessment Research Group with current work focusing on standard setting, borderline judgment and analysis of the introduction of sequential testing methods

Godfrey Pell, BEng, MSc, CStat, CSci, is a Principal Research Fellow Emeritus in Statistics. His particular interests center on the quality and methodology of undergraduate assessment and is a member of the Leeds Assessment Research Group.

Jennifer Hallam, BSc, MSc, PhD, is an Educational Statistician at the University of Leeds, working in both the Schools of Medicine and Dentistry. She is a Psychologist by background and her main research interests relate to individual differences in assessment and assessment for learning formats in workplace settings.