Abstract
Background
In high-stakes assessments in medical education, the decision to let a particular participant pass or fail has far-reaching consequences. Reliability coefficients are usually used to support the trustworthiness of assessments and their accompanying decisions. However, coefficients such as Cronbach’s Alpha do not indicate the precision with which an individual’s performance was measured.
Objective
Since estimates of precision need to be aligned with the level on which inferences are made, we illustrate how to adequately report the precision of pass-fail decisions for single individuals.
Method
We show how to calculate the precision of individual pass-fail decisions using Item Response Theory and illustrate that approach using a real exam. In total, 70 students sat this exam (110 items). Reliability coefficients were above recommendations for high stakes test (> 0.80). At the same time, pass-fail decisions around the cut score were expected to show low accuracy.
Conclusions
Our results illustrate that the most important decisions–i.e. those based on scores near the pass-fail cut-score–are often ambiguous, and that reporting a traditional reliability coefficient is not an adequate description of the uncertainty encountered on an individual level.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes
1 In the present article, we used the Rasch model for our illustrations. Thus, all considerations specifically apply to this model. However, the utilized concept of measurement precision is universal and applies to all IRT models.
Additional information
Notes on contributors
Stefan K. Schauber
Stefan K. Schauber, Dr. rer. medic., is an Associate Professor at the Centre for Educational Measurement at the University of Oslo (CEMO) and Centre for Health Sciences Education, University of Oslo, Oslo, Norway.
Martin Hecht
Martin Hecht, Dr. (PhD equivalent), university diploma in psychology, Humboldt-Universität zu Berlin, Department of Psychology, Berlin, Germany.