Development of a modified Cohen method of standard setting

Abstract

Background: A new ‘Cohen’ approach to standard setting was recently described where the pass mark is calculated as 60% of the score of the student at the 95th percentile, after correcting for guessing.

Aim: This article considers how two potential criticisms of the Cohen method can be addressed and proposes a modified version, with the assumptions tested using local data.

Methods: The modified version removes the correction for guessing and uses the score of the 90th, rather than the 95th percentile student as the reference point, based on the cumulative density functions for 32 modules from one medical school; and incorporates an indirect criterion-referenced passing standard by changing the 60% multiplier to the ratio of the cut score to the score of the student at the 90th percentile on exams that have been standard set using modified Angoff.

Results: The assumption that the performance of the 90th percentile student is consistent over time holds for multiple choice questions. Applying the modified Cohen method to the 32 modules generally reduced the variation in failure rate across modules, compared to a fixed pass mark of 50%.

Conclusion: The results suggest that the modified Cohen method holds much promise as an economical approach to standard setting.