Using item response theory to appraise key feature examinations for clinical reasoning

Abstract

Background

Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach.

Methods

Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach’s alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function.

Results

According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges.

Conclusions

Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.

Keywords:

• Medical education
• clinical reasoning
• key feature
• item response theory
• test theory

Acknowledgements

The authors thank all medical students who devoted their time to this study.

Ethics approval and consent to participate

This study was approved by the local Ethics Committee (application number 16/2/19), and all participants provided written consent.

Disclosure statement

None of the authors have any conflict of interest to declare.

Glossary

Item-response theory: Item response models can be used to fit exam data to a mathematical model, which estimates the probability of a correct answer dependent on the ability level of the examinee (latent variable) and characteristics of each item

Edwards MC: An Introduction to Item Response Theory Using the Need for Cognition Scale. Social and Personality Psychology Compass. 2009; 3: 507–529.

Additional information

Funding

This work was supported by the Federal Ministry of Education and Research [project number: 01GK1801A]. The funding body played no role in the design of the study and in collection, analysis, and interpretation of data and in writing the manuscript.

Notes on contributors

Simon Zegota

Simon Zegota, B.Sc., is a data analyst with a specific focus on business psychology who has also worked on student selection for medical schools.

Tim Becker

Tim Becker, M.A., has obtained a degree in education, medical psychology and sport science. He has done work on interprofessional education as well as key feature examinations. He is currently enrolled in a medical education master’s degree programme.

York Hagmayer

York Hagmayer, PhD, is a professor of psychology; his research is focused on causal chains, decision-making in the clinical context as well as Bayesian statistics.

Tobias Raupach

Tobias Raupach, MD, MME, is a cardiologist and head of the Department of Medical Education at Bonn University Medical Centre. His current research focuses on test-enhanced learning, assessment formats and evaluation.