The Objective Borderline Method: a probabilistic method for standard setting

References

• Bandaranayake, R. 2008. “Setting and Maintaining Standards in Multiple Choice Examinations: Amee Guide No. 37.” Medical Teacher 30 (9–10): 836–845. doi:10.1080/01421590802402247.10.1080/01421590802402247
   PubMed Web of Science ®Google Scholar
• Barker, Chris, Nancy Pistrang, and Robert Elliott. 2005. “Foundations of Quantitative Measurement.” In Research Methods in Clinical Psychology, edited by C. Barker, N. Pistrang, and R. Elliott, 49–71. Chichester: John Wiley.10.1002/0470013435
   Google Scholar
• Barman, A. 2008. “Standard Setting in Student Assessment: Is a Defensible Method Yet to Come.” Annals of the Academy of Medicine, Singapore 37: 957–963.
   PubMed Web of Science ®Google Scholar
• Ben-David, M. 2000. “Amee Guide No. 18: Standard Setting in Student Assessment.” Medical Teacher 22 (2): 120–130.10.1080/01421590078526
   Web of Science ®Google Scholar
• Boursicot, K., T. Roberts, and G. Pell. 2007. “Using Borderline Methods to Compare Passing Standards for Osces at Graduation across Three Medical Schools.” Medical Education 41 (11): 1024–1031. doi:10.1111/j.1365-2923.2007.02857.x.10.1111/med.2007.41.issue-11
   PubMed Web of Science ®Google Scholar
• Brand, H. S., and M. Schoonheim-Klein. 2009. “Is the Osce More Stressful? Examination Anxiety and Its Consequences in Different Assessment Methods in Dental Education.” European Journal of Dental Education 13 (3): 147–153. doi:10.1111/j.1600-0579.2008.00554.x.10.1111/j.1600-0579.2008.00554.x
   PubMed Web of Science ®Google Scholar
• Brannick, Michael T., H. Tugba Erol-Korkmaz, and Matthew Prewett. 2011. “A Systematic Review of the Reliability of Objective Structured Clinical Examination Scores.” Medical Education 45 (12): 1181–1189. doi:10.1111/j.1365-2923.2011.04075.x.10.1111/med.2011.45.issue-12
   PubMed Web of Science ®Google Scholar
• Buckendahl, C., R. Smith, J. Impara, and B. Plake. 2002. “A Comparison of Angoff and Bookmark Standard Setting Methods.” Journal of Educational Measurement 39 (3): 253–263. doi:10.1111/j.1745-3984.2002.tb01177.x.10.1111/jedm.2002.39.issue-3
   Web of Science ®Google Scholar
• Chang, L., C. Dziuban, M. Hynes, and A. Olson. 1996. “Does a Standard Reflect Minimal Competency of Examinees or Judge Competency?” Applied Measurement in Education 9 (2): 161.10.1207/s15324818ame0902_5
   Web of Science ®Google Scholar
• Cizek, G. 1996. “Standard-setting Guidelines.” Educational Measurement: Issues and Practice 15 (1): 13–22.
   Google Scholar
• Cizek, G. 2001. “Conjectures on the Rise and Call of Standard-Setting: An Introduction to Context and Practice.” In Setting Performance Standards: Concepts, Methods, and Perspectives, edited by G. Cizek, 3–17. Mahwah, NJ: Erlbaum.
   Google Scholar
• Cizek, G. 2012. Setting Performance Standards: Foundations, Methods, and Innovations. 2nd ed. London: Routledge.
   Google Scholar
• Cizek, G., and M. Bunch. 2007a. “The Contrasting Groups and Borderline Group Methods.” In Standard Setting, edited by G. Cizek and M. Bunch, 105–117. London: Sage.
   Google Scholar
• Cizek, G., and M. Bunch. 2007b. Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. London: Sage.
   Google Scholar
• Crossley, J., H. T. O. Davies, G. Humphris, and B. Jolly. 2002. “Generalisability: A Key to Unlock Professional Assessment.” Medical Education 36 (10): 972–978.10.1046/j.1365-2923.2002.01320.x
   PubMed Web of Science ®Google Scholar
• Cusimano, M., and A. Rothman. 2003. “The Effect of Incorporating Normative Data into a Criterion-Referenced Standard Setting in Medical Education.” Academic Medicine 78 (10): S88–S90.10.1097/00001888-200310001-00028
   Google Scholar
• Dwoning, S., and T. Haladyna. 1997. “Test Item Development: Validity Evidence from Quality Assurance Procedures.” Applied Measurement in Education 10 (1): 61–82.10.1207/s15324818ame1001_4
   Web of Science ®Google Scholar
• Ferdous, A., and B. Plake. 2008. “Item Response Theory-based Approaches for Computing Minimum Passing Scores from an Angoff-based Standard-Setting Study.” Educational and Psychological Measurement 68 (5): 778–796. doi:10.1177/0013164407312605.10.1177/0013164407312605
   Web of Science ®Google Scholar
• George, S., S. Haque, and F. Oyebode. 2006. “Standard Setting: Comparison of Two Methods.” BMC Medical Education 6 (1): 1–6.10.1186/1472-6920-6-46
   PubMedGoogle Scholar
• Hambleton, R., R. Jaeger, B. S. Plake, and C. Mills. 2000. “Setting Performance Standards on Complex Educational Assessments.” Applied Psychological Measurement 24 (4): 355–366. doi:10.1177/01466210022031804.10.1177/01466210022031804
   Web of Science ®Google Scholar
• Hambleton, R., and S. Powell. 1983. “A Framework for Viewing the Process of Standard Setting.” Evaluation & the Health Professions 6 (1): 3–24. doi:10.1177/016327878300600101.
   PubMedGoogle Scholar
• Hays, R., T. Gupta, and J. Veitch. 2008. “The Practical Value of the Standard Error of Measurement in Borderline Passfail Decisions.” Medical Education 42 (8): 810–815.10.1111/med.2008.42.issue-8
   PubMed Web of Science ®Google Scholar
• Hurtz, Gregory M., and Meredith A. Auerbach. 2003. “A Meta-analysis of the Effects of Modifications to the Angoff Method on Cutoff Scores and Judgment Consensus.” Educational and Psychological Measurement 63 (4): 584–601. doi:10.1177/0013164403251284.10.1177/0013164403251284
   Web of Science ®Google Scholar
• Huxham, M., F. Campbell, and J. Westwood. 2010. “Oral versus Written Assessments: A Test of Student Performance and Attitudes.” Assessment & Evaluation in Higher Education 37 (1): 125–136. doi:10.1080/02602938.2010.515012.
   Web of Science ®Google Scholar
• Jalili, Mohammad, Sara M. Hejri, and John J. Norcini. 2011. “Comparison of Two Methods of Standard Setting: The Performance of the Three-Level Angoff Method.” Medical Education 45 (12): 1199–1208. doi:10.1111/j.1365-2923.2011.04073.x.10.1111/med.2011.45.issue-12
   PubMed Web of Science ®Google Scholar
• Kane, M. 2013. “So Much Remains the Same: Conception and Status of Validation in Setting Standards.” In Standard Setting: Concepts, Methods, and Perspectives, edited by G. Cizek, 53–88. London: Routledge.
   Google Scholar
• Kaufman, D., K. Mann, A. Muijtjens, and C. van der Vleuten. 2000. “A Comparison of Standard-Setting Procedures for an Osce in Undergraduate Medical Education.” Academic Medicine 75 (3): 267–271.10.1097/00001888-200003000-00018
   PubMed Web of Science ®Google Scholar
• Kramer, Anneke, Arno Muijtjens, Koos Jansen, Herman Düsman, Lisa Tan, and C. van der Vleuten. 2003. “Comparison of a Rational and an Empirical Standard Setting Procedure for an Osce.” Medical Education 37 (2): 132–139. doi:10.1046/j.1365-2923.2003.01429.x.10.1046/j.1365-2923.2003.01429.x
   PubMed Web of Science ®Google Scholar
• MacCann, R. G., and G. Stanley. 2006. “The Use of Rasch Modeling to Improve Standard Setting.” Practical Assessment Research and Evaluation 11 (2): 1–17.
   Google Scholar
• Muijtjens, A., A. Kramer, D. Kaufman, and C. Van der Vleuten. 2003. “Using Resampling to Estimate the Precision of an Empirical Standard-setting Method.” Applied Measurement in Education 16 (3): 245–256. doi:10.1207/s15324818ame1603_5.10.1207/S15324818AME1603_5
   Web of Science ®Google Scholar
• Pant, H., A. Rupp, S. Tiffin-Richards, and O. Köller. 2009. Validity Issues in Standard-setting Studies. Studies in Educational Evaluation 35 (2–3): 95–101. doi: http://dx.doi.org/10.1016/j.stueduc.2009.10.008
   Google Scholar
• Peterson, C., E. Matthew Schulz, and George Engelhard Jr. 2011. “Reliability and Validity of Bookmark-based Methods for Standard Setting: Comparisons to Angoff-based Methods in the National Assessment of Educational Progress.” Educational Measurement: Issues and Practice 30 (2): 3–14. doi:10.1111/j.1745-3992.2011.00200.x.10.1111/emip.2011.30.issue-2
   Google Scholar
• Raykov, T., and G. Marcoulides. 2011. Introduction to Psychometric Theory. London: Routledge.
   Google Scholar
• Rees, Charlotte E., Lynn V. Knight, and Jennifer A. Cleland. 2009. “Medical Educators’ Metaphoric Talk about their Assessment Relationships with Students: ‘You don’t Want to Sort of Be the One Who Sticks the Knife in them’.” Assessment & Evaluation in Higher Education 34 (4): 455–467. doi:10.1080/02602930802071098.
   Web of Science ®Google Scholar
• Schmitt, N., J. Schneider, and S. Cohen. 1990. “Factors Affecting Validity of a Regionally Administered Assessment Center.” Personnel Psychology 43: 1–12.10.1111/peps.1990.43.issue-1
   Web of Science ®Google Scholar
• Schoonheim-Klein, M., A. Muijtjens, L. Habets, M. Manogue, C. van der Vleuten, and U. van der Velden. 2009. “Who Will Pass the Dental Osce? Comparison of the Angoff and the Borderline Regression Standard Setting Methods.” European Journal of Dental Education 13 (3): 162–171.10.1111/j.1600-0579.2008.00568.x
   PubMed Web of Science ®Google Scholar
• Shulruf, B., R. Turner, P. Poole, and T. Wilkinson. 2013. “The Objective Borderline Method (Obm): A Probability-based Model for Setting up an Objective Pass/Fail Cut-off Score for Borderline Grades in Medical Education Programmes.” Advances in Health Sciences Education 18 (2): 231–244. doi:10.1007/s10459-012-9367-y.10.1007/s10459-012-9367-y
   PubMed Web of Science ®Google Scholar
• Sireci, S., R. Hambleton, and M. Pitoniak. 2004. “Setting Passing Scores on Licensure Exams Using Direct Consensus.” CLEAR Exam Review 15: 21–25.
   Google Scholar
• Smee, S. 2001. “Setting Standards for an Objective Structured Clinical Examination: The Borderline Group Method Gains Ground on Angoff.” Medical Education 35: 1009–1010.10.1046/j.1365-2923.2001.01047.x
   PubMed Web of Science ®Google Scholar
• Stone, G., S. Beltyukova, and C. Fox. 2008. “Objective Standard Setting for Judge-Mediated Examinations.” International Journal of Testing 8 (2): 180–196. doi:10.1080/15305050802007083.10.1080/15305050802007083
   Google Scholar
• Stone, G., K. Koskey, and T. Sondergeld. 2011. “Comparing Construct Definition in the Angoff and Objective Standard Setting Models: Playing in a House of Cards without a Full Deck.” Educational and Psychological Measurement 71 (6): 942–962. doi:10.1177/0013164410394338.10.1177/0013164410394338
   Web of Science ®Google Scholar
• Sturmberg, J., and J. Hinchy. 2010. “Borderline Competence – From a Complexity Perspective: Conceptualization and Implementation for Certifying Examinations.” Journal of Evaluation in Clinical Practice 16 (4): 867–872. doi:10.1111/j.1365-2753.2010.01501.x.10.1111/jep.2010.16.issue-4
   PubMed Web of Science ®Google Scholar
• Sutcliffe, J. P. 1958. “Error of Measurement and the Sensitivity of a Test of Significance.” Psychometrika 23 (1): 9–17. doi:10.1007/BF02288974.10.1007/BF02288974
   Web of Science ®Google Scholar
• Tavakol, M., and R. Dennick. 2012. “Standard Setting: The Application of the Receiver Operating Characteristic Method.” International Journal of Medical Education 3: 198–200.10.5116/ijme.506 f.1aaa
   Google Scholar
• Tweed, M., Mark Thompson-Fawcett, and T. Wilkinson. 2013. “Decision-making Bias in Assessment: The Effect of Aggregating Objective Information and Anecdote.” Medical Teacher 35 (10): 832–837. doi:10.3109/0142159X.2013.803062.10.3109/0142159X.2013.803062
   PubMed Web of Science ®Google Scholar
• Verheggen, M., A. Muijtjens, J. Van Os, and L. Schuwirth. 2008. “Is an Angoff Standard an Indication of Minimal Competence of Examinees or of Judges?” Advances in Health Sciences Education 13 (2): 203–211. doi:10.1007/s10459-006-9035-1.10.1007/s10459-006-9035-1
   PubMed Web of Science ®Google Scholar
• Viswanathan, M. 2005. What Causes Measurement Error? New York: Sage.
   Google Scholar
• Wang, N. 2003. “Use of the Rasch IRT Model in Standard Setting: An Item-mapping Method.” Journal of Educational Measurement 40 (3): 231–253. doi:10.1111/j.1745-3984.2003.tb01106.x.10.1111/jedm.2003.40.issue-3
   Web of Science ®Google Scholar
• Wass, V., C. van der Vleuten, John Shatzer, and Roger Jones. 2001. “Assessment of Clinical Competence.” The Lancet 357 (9260): 945–949.10.1016/S0140-6736(00)04221-5
   PubMed Web of Science ®Google Scholar
• Wayne, D. B., Elaine Cohen, Gregory Makoul, and William C. McGaghie. 2008. “The Impact of Judge Selection on Standard Setting for a Patient Survey of Physician Communication Skills.” Academic Medicine 83 (10): S17–S20. doi:10.1097/ACM.1090b1013e318183e318187bd.10.1097/ACM.0b013e318183e7bd
   Google Scholar
• Wayne, D., M. Fudala, J. Butter, V. Siddall, J. Feinglass, L. Wade, and W. McGaghie. 2005. “Comparison of Two Standard-setting Methods for Advanced Cardiac Life Support Training.” Academic Medicine 80 (10): S63–S66.10.1097/00001888-200510001-00018
   Google Scholar
• Wilkinson, T., C. Frampton, M. Thompson-Fawcett, and T. Egan. 2003. “Objectivity in Objective Structured Clinical Examinations: Checklists are No Substitute for Examiner Commitment.” Academic Medicine 78 (2): 219–223.10.1097/00001888-200302000-00021
   PubMed Web of Science ®Google Scholar
• Woehr, David J., Winfred Arthur, and Melinda L. Fehrmann. 1991. “An Empirical Comparison of Cutoff Score Methods for Content-related and Criterion-related Validity Settings.” Educational and Psychological Measurement 51 (4): 1029–1039. doi:10.1177/001316449105100423.10.1177/001316449105100423
   Web of Science ®Google Scholar
• Wood, T., S. Humphrey-Murto, and G. Norman. 2006. “Standard Setting in a Small Scale Osce: A Comparison of the Modified Borderline-group Method and the Borderline Regression Method.” Advances in Health Sciences Education 11 (2): 115–122. doi:10.1007/s10459-005-7853-1.10.1007/s10459-005-7853-1
   PubMed Web of Science ®Google Scholar
• Wyse, A., and M. Reckase. 2012. “Examining Rounding Rules in Angoff-type Standard-setting Methods.” Educational and Psychological Measurement 72 (2): 224–244. doi:10.1177/0013164411413572.10.1177/0013164411413572
   Web of Science ®Google Scholar
• Zieky, M. J. 2012. “So Much has Changed.” In Setting Performance Standards: Foundations, Methods, and Innovations. 2nd ed., edited by G. Cizek, 15–32. London: Routledge.
   Google Scholar