Simpler Standards for Local Performance Examinations: The Yes/No Angoff and Whole-Test Ebel

Abstract

Background: The whole-test Ebel and Yes/No Angoff methods offer the possibility of faster and easier standard-setting exercises for local, medium-stakes performance exams. Purpose: We sought to establish if these less demanding methods might be substituted for the traditional but complex case-level Ebel. Methods: Six faculty judges each used all three methods to set standards for six standardized-patient checklists used for 178 fourth-year medical students. We compared the cut scores, passing rates, reliability, ease of use, and credibility of the three methods. Results: The three methods produced roughly equivalent cut scores. Generalizability of judgments was .94 and .96 for the case-level and whole-test Ebel, and .76 for the Yes/No Angoff. Judges found the simplified methods more credible than the case-level Ebel. The Yes/No Angoff method was preferred by five of six judges. Conclusions: The whole-test Ebel and the Yes/No Angoff may be simple and realistic options for setting standards for local performance exams.

Acknowledgments

We gratefully acknowledge the efforts of our faculty standard-setters, Drs. Blitzstein, Braun, Falck, Khan, Loy, and Riddle.

Notes

^a Yes = 1, No = 0.

^a Item relevance based on judges' prior categorization of items.

^b Item difficulty based on student performance data.

^c Judges estimate the probability of a borderline student getting the type of item in each cell correct. In the Case Method, judges provide cell estimates independently for each case. In the Whole-test method, judges provide a single set of cell estimates, which are used across all cases.

^a Twenty-three students who saw the “headache” backup case instead of one of the other cases were excluded from the exam-level analysis.

^b The far right column values are median standard deviations.

^c SIS1 = standard deviation of student scores/standard deviation of cut score judgments; the far right column values are median SIS1.

^d The far right column values are median phi.

^e The far right column values are median RMSE.