Do role-players affect the outcome of a high-stakes postgraduate OSCE, in terms of candidate sex or ethnicity? Results from an analysis of the 52,702 anonymised case scores from one year of the MRCGP clinical skills assessment

Abstract

The MRCGP clinical skills assessment (CSA), a high-stakes OSCE examination whereby GPs exit their training, uses trained role-players to depict patients (though not to mark candidates). Just as subgroups of examiners can potentially affect the results of subgroups of candidates through biases expressed in discriminatory marking, so possibly could subgroups of role-players alter the difficulty for different candidate groups. This study looks at the contribution of role-players to such possible systematic unfairness in the assessment. Using multiple linear regression, we examined the data from all 52,702 case scores from the MRCGP CSA for the academic year 2012–2013. Candidates were dichotomised by sex, by ethnicity and by source of primary medical qualification (PMQ); role-players were dichotomised by sex and binary ethnicity; and the transaction of candidate/role-player encounters were classified as ‘same’ or ‘different’ in terms of the two parties’ sex and of their ethnicity. Neither examiner nor role-player characteristics were found to predict any statistically significant portion of case score variance, where the significant (p < .001) predictors were source of PMQ (UK or elsewhere: 11% of case score variance), candidates’ ethnicity (1%), candidates’ sex (0.6%) and ‘transactional’ sex (0.1%). We did not therefore find any substantial degree of support for the proposition that role-player subgroups systematically influence candidate subgroups’ scores.

Keywords:

• Role-players
• simulated patients
• postgraduate assessment
• family medicine
• OSCE
• candidates
• fairness

Acknowledgements

We thank the RCGP for encouraging this work and permitting the analyses. We are grateful to the reviewers whose comments have helped develop this report.

Disclosure statement

MLD and RW have been involved in the development and Quality Assurance of the MRCGP CSA since its introduction.