The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method

Abstract

Background: In Objective Structured Clinical Examinations (OSCEs), the use of simulated patients (SPs) at many stations is a key aspect of the assessment. Often the SPs are asked to provide formal feedback (ratings) of their experience with the students under examination.

Aims: This study analyses whether and how exactly SP data can be best used to enhance the robustness of the formal standard setting process.

Methods: A retrospective statistical investigation into the relationship between SP ratings and those provided by the clinical assessors (criterion-based checklist scores and overall grades for each station) is presented. In addition, the paper also includes a study into the impact of the inclusion of the SP ratings in the formal standard setting process for OSCEs, particularly when pass marks are calculated using the borderline regression method.

Results: The general results of the analysis, including discussion of two distinct methods for the combining of the SP ratings and assessor judgements, are presented, and demonstrate that the inclusion of this additional data can have important effects on individual student results.

Conclusion: It is possible for the overall quality of the OSCE assessment process to be improved, with increased reliability, by combining assessor checklist scores and SP ratings.

Notes

Notes

1. That is, the fourth year (out of five) of an undergraduate medical degree (MBChB).

2. Pearson's correlation was used here in spite of the ordinal nature of both of the grades, since this early analysis is exploratory only.

3. Generalizablity theory is not employed in this study for measuring reliability because of limitations in the nature of the OSCE data available. There is insufficient crossing of items (stations) with assessors, meaning that only the simplest model of interactions (items clossed with subjects) could be employed and then the G-coefficient is equivalent to (the non-standardised) Cronbach's alpha and secondly, the OSCE stations have checklist scores that vary in their maxima, and in their mean (expected) student score–under such conditions, g-coefficients are likely to be depressed in value, and do not necessarily give an accurate measure of reliability.

4. The assessor marks are on a longer scale (generally of the order 0 to 30) and have typically higher numerical values compared to the SP ratings. Hence, the analysis presented here is relatively conservative in the sense that the weighting in the combined “mark” of the SP ratings is comparatively low. A different weighting could be chosen that would give more emphasis to the SP ratings and this might be appropriate for future research.

5. Note here that this statistic is relevant to the full set of stations, and not only to those where SPs were employed.

6. The borderline grade at the stations that do no have SP ratings would remain as 1.

7. A number of medical schools currently use SP ratings as formative assessments, counselling those students who receive a relatively high number of poor ratings across the OSCE as a whole.

Additional information

Notes on contributors

Matthew Homer

Dr MATTHEW HOMER is a Research Fellow at the University of Leeds, working in the both the Schools of Medicine and Education. He works on a range of research projects and provides general statistical support to colleagues. His research interests include the statistical side of assessment, particularly related to OSCEs.

Godfrey Pell

GODFREY PELL is a senior statistician who has a strong background in management. Before joining the University of Leeds he was with the Centre for Higher Education Practice at the Open University. Current research includes standard setting for practical assessment in higher education, and the value of short term interventionist programmes in literacy.