A collaborative comparison of objective structured clinical examination (OSCE) standard setting methods at Australian medical schools

Abstract

Background: A key issue underpinning the usefulness of the OSCE assessment to medical education is standard setting, but the majority of standard-setting methods remain challenging for performance assessment because they produce varying passing marks. Several studies have compared standard-setting methods; however, most of these studies are limited by their experimental scope, or use data on examinee performance at a single OSCE station or from a single medical school. This collaborative study between 10 Australian medical schools investigated the effect of standard-setting methods on OSCE cut scores and failure rates.

Methods: This research used 5256 examinee scores from seven shared OSCE stations to calculate cut scores and failure rates using two different compromise standard-setting methods, namely the Borderline Regression and Cohen’s methods.

Results: The results of this study indicate that Cohen’s method yields similar outcomes to the Borderline Regression method, particularly for large examinee cohort sizes. However, with lower examinee numbers on a station, the Borderline Regression method resulted in higher cut scores and larger difference margins in the failure rates.

Conclusion: Cohen’s method yields similar outcomes as the Borderline Regression method and its application for benchmarking purposes and in resource-limited settings is justifiable, particularly with large examinee numbers.

Acknowledgements

The authors would like to thank all participating ACCLAiM medical schools and their administrative and academic staff, who supported the organization and implementation of the examinations.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Ethical approval

All participating schools obtained ethics approval from their local Ethics Committee. All information was de-identified before data analysis.

Glossary

Standard Setting: Standard setting is the process of defining or judging the level of knowledge and skill required to meet a typical level of performance and then identifying a score on the examination score scale that corresponds to that performance standard

Relative Standards: Standards that are established based on a comparison of those who take the assessment to each other are relative standards.

Absolute Standards: Standards set by determining the amount of test material that must be answered (or performed) correctly in order to pass are absolute standards

Reference: McKinley, D.W., and Norcini, J.J. “How to set standards on performance-based examinations: AMEE Guide No. 85.” Medical Teacher 2014. 36,2: 97–110.

Notes on contributors

Bunmi Malau-Aduli, BSc, MSc, PhD, is a Senior Lecturer in Medical Education and the Academic Lead for Assessment and Evaluation at the College of Medicine and Dentistry, James Cook University.

Peta-Ann Teague, MBChB, DRCOG, MRCGP, Dip Med Ed, FRACGP, is an Associate Professor and the Director of the Generalist Medical Training (GMT) Program at James Cook University.

Karen D’souza, MBBS(Hons), is a Senior Lecturer in Medical Education (Clinical Skills) and the Coordinator, Doctor and Patient Theme for the School of Medicine at Deakin University.

Clare Heal, MBChB DRACOG, FRACGP, MPHandTM, Dip GU Med, PhD, is a Professor of General Practice and Rural Medicine for James Cook University in Mackay.

Richard Turner, MBBS, BMedSc, FRACS, is a Professor of Surgery and the Director of the Hobart Clinical School at the School of Medicine, University of Tasmania.

David Garne, MBChB, MSC, MPhil, is an Associate Professor and the Associate Dean of Community, Primary, Remote and Rural Health at the School of Medicine at the University of Wollongong.

Cees van der Vleuten, MA, PhD, is the Scientific Director of the Graduate School of Health Professions Education at Maastricht University.