Conjunctive standards in OSCEs: The why and the how of number of stations passed criteria

Abstract

Introduction

Many institutions require candidates to achieve a minimum number of OSCE stations passed (MNSP) in addition to the aggregate pass mark. The stated rationale is usually that this conjunctive standard prevents excessive degrees of compensation across an assessment. However, there is a lack of consideration and discussion of this practice in the medical education literature.

Methods

We consider the motivations for the adoption of the MNSP from the assessment designer perspective, outlining potential concerns about the complexity of what the OSCE is trying to achieve, particularly around the blueprinting process and the limitations of scoring instruments. We also introduce four potential methods for setting an examinee-centred MNSP standard, and highlight briefly the theoretical advantages and disadvantages of these approaches.

Discussion and conclusion

There are psychometric arguments for and against the limiting of compensation in OSCEs, but it is clear that many stakeholders value the application of an MNSP standard. This paper adds to the limited literature on this important topic and notes that current MNSP practices are often problematic in high stakes settings. More empirical work is needed to develop understanding of the impact on pass/fail decision-making of the proposed standard setting methods developed in this paper.

Keywords:

• OSCE
• assessment
• standard setting

Glossary

Conjunctive standard: In a summative assessment such as an OSCE, candidates are usually required to achieve an aggregate score in order to pass. If this is the sole passing requirement, then students can (fully) compensate between elements of the assessment – for example, they can fail individual stations and still pass. A conjunctive standard is an additional requirement, such as passing a minimum number of stations or passing separate domains (e.g. management, communication skills, clinical skills) (Ben-David 2000). Careful consideration of impacts on pass rates, and the degree of potential error in decision-making, is needed when implementing such conjunctive standards.

Ben-David, M.F. 2000. AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher. 22(2), pp.120–130.

Acknowledgments

The authors would like to thank David B. Swanson from the American Board of Medical Specialties for discussions on early aspects of some of this work, particularly his initial outline of method 3. They would also like to thank Godfrey Pell for various discussions about the underlying philosophical aspects of this work. Anything erroneous remains, of course, the authors’ responsibility.

Ethics

No data was generated or analysed in this study. Hence, ethical approval was not required.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Matt Homer

Matt Homer, BSc, MSc, PhD, PGCE, CStat, is an Associate Professor in the Schools of Education and Medicine at the University of Leeds. Within medical education, he has a research interest in assessment design, standard setting methodologies and psychometrics analysis. He also advises the UK General Medical Council on a range of assessment issues.

Jen Russell

Jen Russell, MBCHB, FCEM, MRCS (Eng), Med, FHEA, is a Consultant in Emergency Medicine at Leeds Teaching Hospitals NHS Trust and Honorary Senior Lecturer in the School of Medicine at the University of Leeds. Her research interests centre on developing frameworks for personalising feedback in OSCEs.