Modifying the Hofstee method may overcome problems

Dear Sir

Our colleagues Tavakol & Dennick (2014) convincingly argue that the conventional-Hofstee (CH) standard setting method lacks transparency; they also demonstrate that large changes in the upper-limits (UL) and lower-limits (LL) of pass-marks and fail-rates generate negligible changes in cut-scores (1–2%).

We agree with Tavakol & Dennick’s conclusions and we have developed a substantially modified-Hofstee (MH) method that obviates subjective judgements. In the MH protocol, the UL and LL for fail-rates are set at 100% and 0% of the cohort, respectively, i.e. potentially all candidates could fail or pass, avoiding arbitrary pass/fail quotas. The LL for the pass-mark is also set at 0%, avoiding an arbitrary boundary for this value. The UL for the pass-mark is standard set an absolute percentage below the median percentage mark of the cohort (based on trials on historical data). The rationale for this is that cohort performance is an objective measure of exam difficulty when the cohort is sufficiently large and its selection is consistent year-on-year (as is likely for selection of medical students). (Others have similarly argued for using cohort data for setting borderlines (Cohen-Schotanus & van der Vleuten, 2010). A diagonal is drawn for the UL pass-mark on the X-axis to the UL of fails (100%) on the Y-axis; the intersection with the cumulative frequency curve of students’ marks gives the actual pass-mark.

We applied the MH protocol to historical data of 49 summative assessments sat by cohorts of >240 year 1–3 medical students over five years. The range of pass-marks generated by MH was 47%–60% (median 57%). We further examined the effect on 15 assessments of raising the UL of the pass-mark by 10%; this increased the cut-scores by 5.4%–8.5% (median 6.6%), which is substantially greater than reported by Tavakol & Dennick for CH. Furthermore, if the diagonal does not intersect with the frequency curve, this indicates that all students pass.

Whilst Anghoff/Ebel are appropriate for “high-stakes”, broad-based clinical exams, they may be impractical for multiple, subject-specific exams, where assembling and ensuring consistent operation of multiple panels of experts is not feasible. In such circumstances, the MH protocol may provide a viable and robust alternative for credible standard setting.

Declaration of interest: The authors report no conflicts of interest.