Ensuring reliability in UK written tests of general practice: the MRCGP examination 1998–2003

Abstract

Reliability in written examinations is taken very seriously by examination boards and candidates alike. Within general education many factors influence reliability including variations between markers, within markers, within candidates and within teachers. Mechanisms designed to overcome, or at least minimize, the impact of such variables are detailed. Methods of establishing reliability are also explored in the context of a range of assessment situations. In written tests of general practice within the Membership of the Royal College of General Practitioner (MRCGP) examination considerable effort has put been put into achieving acceptable levels of reliability. Current mechanisms designed to ensure high reliability are described and related to the evolution of the written component of the examination. In addition to description of marker selection and training, question development including construct a detailed example of specific and generic marking schedules is provided. Examination results for the Written Paper of the MRCGP from 1998 to 2003 are reported including Cronbach's alpha coefficients and standard error of measurements, mean scores (and SD) and pass rates. In addition individual discrimination scores for each question in the October 2002 paper are shown. Consistent high reliability of the written component of the MRCGP examination provides valuable lessons in terms of selection, training and monitoring of markers as well as practical methods of moderating factors affecting candidate variability. The challenge for examination developers is to carry these important lessons forward into a modernized assessment structure of UK general practice.