Improving postgraduate clinical assessment tools: The introduction of video recordings to assess decision making

Abstract

Background: Competency in the management of acutely unwell patients has not previously been formally assessed as part of an objective structured clinical examination (OSCE).

Aim: The reliability of the paediatric postgraduate OSCE was calculated. An objective structured video examination was designed to assess candidates’ clinical decision making ability when managing acutely unwell children.

Methods: The performance of 3522 postgraduate paediatric trainees was assessed (2006–2008). OSCE reliability was determined using Cronbach's alpha and mean inter-item correlation. Performance in the video station was compared with overall performance (not including video station; Mann–Whitney U) and video station scores correlated with individual station scores (Spearman's Rho correlation coefficient).

Results: Clinical examination pass rates for the 684 UK graduates, 1608 overseas candidates training in the UK and 1104 overseas candidates training overseas were 69.7%, 28% and 22.3%, respectively (graduation information not available for 126 candidates). Cronbach's alpha was 0.62. Mean inter-item correlation was 0.15. Candidates who passed the OSCE overall had significantly higher scores on the video station (t (3520) = 14.48); p < 0.001). There was significant positive correlation between scores on the video station, individual stations and overall total score (r's = 0.300; p = 0.001).

Conclusions: The postgraduate paediatric OSCE provides a sound and valid means of assessing clinical skills at the postgraduate level. The video station provides an important new method of assessment. Its use in other postgraduate clinical examinations should be explored.

Acknowledgements

We thank all of the examiners, trainees and RCPCH staff involved in setting up and running the OSCE and video station. We are particularly grateful to children and their families who allowed us to use this material, collected at times when their children were acutely unwell.

Declaration of interest: There are no competing interests.