Turning words into numbers: establishing an empirical cut score for a letter graded examination

Abstract

Background: High stakes postgraduate specialist certification examinations have considerable implications for candidates’ future careers. The cut score i.e. pass/fail mark of such examinations needs to be determined in a defensible and credible manner. A number of methods, suitable for use with numeric scoring methods, have been described. Determining the cut score of letter-graded examinations, is however, not described in the literature.

Aim: The aim of this study was to determine a defensible and credible method for deriving the cut score of a letter-graded examination.

Method: The cut score of the Fellowship examination of the College of Physicians of South Africa was estimated using a novel method. This method was validated by comparing the results obtained to those obtained using the contrasting groups method.

Results: By using the examiners’ decision as the ‘gold standard’ we found that a cut score of 50% best approximated the cutpoint of this letter-graded examination, achieving a sensitivity and specificity of 83.7% and 82.8% respectively.

Conclusion: This paper describes a useful strategy for estimating the cut score of letter-graded examinations.

Additional information

Notes on contributors

Vanessa C. Burch

VANESSA BURCH is Professor and Chair of Clinical Medicine at the University of Cape Town, Cape Town, South Africa.

G. R. Norman

GEOFFREY NORMAN is Director of the Programme for Educational Research and Development, McMaster University, Hamilton, Ontario, Canada