Assuring the quality of high-stakes undergraduate assessments of clinical competence

Abstract

In the UK, and in many Commonwealth countries, a university degree is accepted by registration bodies as an indication of competence to practice as a PRHO or intern. Concerns have been raised that the quality of university examinations may not always be sufficient for such high-stakes decision-making. Assessments of clinical competence are subject to many potential sources of error. The search for standardization, and high validity and reliability, demands the identification and reduction of measurement errors and biases due to poor test design or variation in test items, judges, patients or examination procedures. Generalizability and other research studies have identified where the likely sources of error might arise and have been taken into account in the development of published guidelines on international best practice, which institutions should strive to follow. The purpose of this paper is to describe the development of the integrated final-year assessment of clinical competence at the University of Sheffield. The aim was to introduce a range of strategies to ensure the examination met the best practice guidelines. These included blueprinting the assessment to achieve a high degree of content validity; lengthening the examination by adding a written component to the OSCE component to ensure an adequate level of reliability; providing training and feedback for examiners and simulated patients; paying attention to item development; and providing statistical information to assist the examination committee in standard setting and decision-making. This evidence-based approach should be readily achievable by all medical schools.

Additional information

Notes on contributors

Chris Roberts

CHRIS ROBERTS is Associate Professor of Medical Education and Director of the Office of Teaching and Learning in Medicine at the University of Sydney, Australia.

David Newble

DAVID NEWBLE is Emeritus Professor of Medical Education at the University of Sheffield. He returned to Australia in 2004 where he is a professor of Medical Education at Flinders University and works as a consultant in Medical Education.

Brian Jolly

BRIAN JOLLY is Director of the Centre for Medical and Health Sciences Education, Faculty of Medicine, Nursing & Health Sciences, Monash University, Australia.

Malcolm Reed

MALCOLM REED is Professor of Surgery in the Department of Surgery in the Sheffield Teaching Hospitals Trust. He is Chair of the Assessment Committee responsible for the finals examination.

Kingsley Hampton

KINGSLEY HAMPTON is Senior Clinical Lecturer Division of Genomic Medicine and Consultant Haematologist at the Sheffield Teaching Hospitals Trust. He is the current Director of Studies responsible for the finals examination.