National inter-rater agreement of standardised simulated-patient-based assessments

Abstract

Purpose

The forthcoming UK Medical Licensing Assessment will require all medical schools in the UK to ensure that their students pass an appropriately designed Clinical and Professional Skills Assessment (CPSA) prior to graduation and registration with a licence to practice medicine. The requirements for the CPSA will be set by the General Medical Council, but individual medical schools will be responsible for implementing their own assessments. It is therefore important that assessors from different medical schools across the UK agree on what standard of performance constitutes a fail, pass or good grade.

Methods

We used an experimental video-based, single-blinded, randomised, internet-based design. We created videos of simulated student performances of a clinical examination at four scripted standards: clear fail (CF), borderline (BD), clear pass (CPX) and good (GD). Assessors from ten regions across the UK were randomly assigned to watch five videos in 12 different combinations and asked to give competence domain scores and an overall global grade for each simulated candidate. The inter-rater agreement as measured by the intraclass correlation coefficient (ICC) based on a two-way random-effects model for absolute agreement was calculated for the total domain scores.

Results

120 assessors enrolled in the study, with 98 eligible for analysis. The ICC was 0.93 (95% CI 0.81–0.99). The mean percentage agreement with the scripted global grade was 74.4% (range 40.8–96.9%).

Conclusions

The inter-rater agreement amongst assessors across the UK when rating simulated candidates performing at scripted levels is excellent. The level of agreement for the overall global performance level for simulated candidates is also high. These findings suggest that assessors from across the UK viewing the same simulated performances show high levels of agreement of the standards expected of students at a ‘clear fail,’ ‘borderline,’ ‘clear pass’ and ‘good’ level.

Keywords:

• Undergraduate
• assessment
• inter-rater reliability

Acknowledgements

The authors are grateful to all UK medical school assessment leads for their help in recruiting assessors. The authors are also grateful to the Medical Schools Council for administrative support with the study.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Additional information

Funding

MG is supported by the National Institute for Health Research (NIHR) Cambridge Biomedical Research Centre. CAB is supported by the NIHR Applied Research Collaboration (ARC) West Midlands. The Medical Schools Council funded the recruitment of the simulated candidates, simulated patient and sourcing of the recording equipment for this study.

Notes on contributors

Amir H. Sam

Amir H. Sam, BSc, MBBS, PhD, FRCP, SFHEA, is head of Imperial College School of Medicine and consultant physician and endocrinologist at Imperial College Healthcare NHS Trust.

Michael D. Reid

Michael D. Reid, MA, MB, BChir, MAcadMEd, MRCP, was a Clinical Education & Research Fellow at Imperial College London and is now a trainee in Geriatric Medicine at Kingston Hospital.

Viral Thakerar

Viral Thakerar, MBBS, MRCP, MRCGP, is the lead for year 1 and 2 clinical placements at Imperial College School of Medicine and a practising general practitioner.

Mark Gurnell

Mark Gurnell, PhD, MA(MEd), FHEA, FAcadMEd, FRCP, is Clinical SubDean at the University of Cambridge School of Clinical Medicine and Professor of Clinical Endocrinology at Institute of Metabolic Science & Department of Medicine.

Rachel Westacott

Rachel Westacott, MB, ChB, is a Senior Lecturer in Medical Education at Birmingham Medical School and an acute medicine consultant at University Hospitals of Leicester NHS Trust (CCT in Nephrology).

Malcolm W. R. Reed

Malcolm W. R. Reed, MD, BMedSci, MBCHB, FRCS, is a breast cancer surgeon who has been Dean of Brighton and Sussex Medical School since 2014 having moved from Sheffield University Medical School where he was head of Undergraduate Assessment for medicine. He is currently Co-Chair of Medical Schools Council and Chair of the education subcommittee.

Celia A. Brown

Celia A. Brown, PhD, SFHEA, is an Associate Professor in Quantitative Methods at Warwick Medical School. She has research interests in selection and assessment and teaches quantitative methods at all levels in Higher Education.