Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch

References

• Albano AD. 2013. Multilevel modelling of item position effects. J Educ Meas. 50:408–426.
   Web of Science ®Google Scholar
• Andrich D, Sheridan B, Luo G. 2002. RUMM2020: rasch unidimensional models for measurement. Perth, Western Australia: RUMM Laboratory.
   Google Scholar
• Bhakta B, Tennant A, Horton M, Lawton G, Andrich D. 2005. Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education. BMC Med Educ. 5:9.
   PubMedGoogle Scholar
• Bond TG, Fox CM. 2007. Applying the rasch model: fundamental measurement in the human sciences. 2nd ed. Mahwah, NJ: Psychology Press.
   Google Scholar
• Case S, Swanson D. 2001. Constructing Written Test Questions for the Basic and Clinical Sciences online [Internet]. Philadelphia: NBME; [cited 2013 Jul 3]. Available from: http://www.heacademy.ac.uk/resources/detail/resource_database/SNAS/Constructing_Written_Test_Questions_for_the_Basic_and_Clinical_Sciences.
   Google Scholar
• Cizek GJ, Bunch MB. 2007. Standard setting a guide to establishing and evaluating performance standards on tests online [Internet]. Thousand Oaks, Calif: Sage Publications; [cited 2013 Aug 21]. Available from: http://SRMO.sagepub.com/view/standard-setting/SAGE.xml.
   Google Scholar
• Clauser BE, Harik P, Margolis MJ, McManus IC, Mollon J, Chis L, Williams S. 2008. An empirical examination of the impact of group discussion and examinee performance information on judgments made in the angoff standard-setting procedure. Appl Meas Educ. 22:1–21.
   Web of Science ®Google Scholar
• Clauser BE, Mee J, Baldwin SG, Margolis MJ, Dillon GF. 2009. Judges’ use of examinee performance data in an angoff standard-setting exercise for a medical licensing examination: an experimental study. J Educ Meas. 46:390–407.
   Web of Science ®Google Scholar
• Clauser JC, Margolis MJ, Clauser BE. 2014. An examination of the replicability of angoff standard setting results within a generalizability theory framework. J Educ Meas. 51:127–140.
   Web of Science ®Google Scholar
• Cohen-Schotanus J, van der Vleuten CPM. 2010. A standard setting method with the best performing students as point of reference: practical and affordable. Med Teach. 32:154–160.
   PubMed Web of Science ®Google Scholar
• Cook DA. 2014. When I say… validity. Med Educ. 48:948–949.
   PubMed Web of Science ®Google Scholar
• Cumming G. 2011. Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
   Google Scholar
• Cusimano MD. 1996. Standard setting in medical education. Acad Med. 71:S112–S120. p
   PubMed Web of Science ®Google Scholar
• Downing SM. 2003. Item response theory: applications of modern test theory in medical education. Med Educ. 37:739–745.
   PubMed Web of Science ®Google Scholar
• Downing SM, Lieska NG, Raible MD. 2003. Establishing passing standards for classroom achievement tests in medical education: a comparative study of four methods. Acad Med. 78:S85–S87.
   PubMed Web of Science ®Google Scholar
• Ebel RL. 1972. Essentials of educational measurement online [Internet]. xii, p. 370. Englewood Cliff, N.J., US: Prentice-Hall, Inc; [cited 2010 Jan 15]. Available from: http://openlibrary.org/b/OL17728612M/Essentials_of_educational_measurement.
   Google Scholar
• Fuller R, Homer M, Pell G, Hallam J. Managing extremes of assessor judgement within the OSCE. Med Teach. 2016. [Epub ahead of print]. (in press).
   Google Scholar
• General Medical Council. 2015. GMC Council approves development of UK medical licencing assessment. [cited 2016 Jul 18]. Available from: http://www.gmc-uk.org/news/26549.asp.
   Google Scholar
• Hays R, Gupta TS, Veitch J. 2008. The practical value of the standard error of measurement in borderline pass/fail decisions. Med Educ. 42:810–815.
   PubMed Web of Science ®Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35:564–568.
   PubMed Web of Science ®Google Scholar
• Homer M, Darling J, Pell G. 2012. Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality. Assess Eval High Educ. 37:787–804.
   Web of Science ®Google Scholar
• Homer M, Pell G, Fuller R, Patterson J. 2016. Quantifying error in OSCE standard setting for varying cohort sizes: a resampling approach to measuring assessment quality. Med Teach. 38:181–188.
   PubMed Web of Science ®Google Scholar
• Jaeger RM. 1991. Selection of judges for standard-setting. Educ Meas. 10:3–14.
   Google Scholar
• Jalili M, Hejri SM, Norcini JJ. 2011. Comparison of two methods of standard setting: the performance of the three-level Angoff method . Med Educ. 45:1199–1208. p
   PubMed Web of Science ®Google Scholar
• Kane M. 2001. So much remains the same: conception and status of validation in setting standards. In: Cizek GJ, editor. Setting performance standards: concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates. p. 53–58.
   Google Scholar
• Kane MT. 2013. Validating the interpretations and uses of test scores. J Educ Meas. 50:1–73.
   Web of Science ®Google Scholar
• Kolen MJ, Brennan RL. 2014. Test equating, scaling, and linking: methods and practices. New York, NY: Springer Science & Business Media.
   Google Scholar
• Livingston SA, Zieky MJ. 1982. Passing scores: a manual for setting standards of performance on educational and occupational tests. [cited 2015 March 19]. Available from: http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED227113.
   Google Scholar
• Maccann RG. 2009. Standard setting with dichotomous and constructed response items: some rasch model approaches. J Appl Meas. 10:438–454.
   PubMedGoogle Scholar
• Margolis MJ, Clauser BE. 2014. The impact of examinee performance information on judges’ cut scores in modified angoff standard-setting exercises. Educ Meas. 33:15–22.
   Web of Science ®Google Scholar
• Margolis MJ, Mee J, Clauser BE, Winward M, Clauser JC. 2016. Effect of content knowledge on angoff-style standard setting judgments. Educ Meas online [Internet]. [cited 2016 February 15]. Available from: http://onlinelibrary.wiley.com/doi/10.1111/emip.12104/abstract.
   Google Scholar
• McManus IC. 2012. The misinterpretation of the standard error of measurement in medical education: a primer on the problems, pitfalls and peculiarities of the three different standard errors of measurement. Med Teach. 34:569–576.
   PubMed Web of Science ®Google Scholar
• Mee J, Clauser BE, Margolis MJ. 2013. The impact of process instructions on judges’ use of examinee performance data in angoff standard setting exercises. Educ Meas. 32:27–35.
   Web of Science ®Google Scholar
• Norcini JJ. 2003. Setting standards on educational tests. Med Educ. 37:464–469.
   PubMed Web of Science ®Google Scholar
• Panayides P, Robinson C, Tymms P. 2010. The assessment revolution that has passed England by: Rasch measurement. Br Edu Res J. 36:611–626.
   Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts T. 2013. Advancing the objective structured clinical examination: sequential testing in theory and practice. Med Educ. 47:569–577.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts T. 2010. How to measure the quality of the OSCE: a review of metrics – AMEE guide no. 49. Med Teach. 32:802–811.
   PubMed Web of Science ®Google Scholar
• Pell G, Homer M, Fuller R. 2015. Investigating disparity between global grades and checklist scores in OSCEs. Med Teach. 37:1106–1113.
   PubMed Web of Science ®Google Scholar
• Ricketts C. 2009. A plea for the proper use of criterion-referenced tests in medical assessment. Med Educ. 43:1141–1146.
   PubMed Web of Science ®Google Scholar
• Schneid SD, Armour C, Park YS, Yudkowsky R, Bordage G. 2014. Reducing the number of options on multiple-choice questions: response time, psychometrics and standard setting. Med Educ. 48:1020–1027.
   PubMed Web of Science ®Google Scholar
• Schuwirth LWT, Vleuten CPM. 2006. A plea for new psychometric models in educational assessment. Med Educ. 40:296–300.
   PubMed Web of Science ®Google Scholar
• Shulruf B, Wilkinson T, Weller J, Jones P, Poole P. 2016. Insights into the angoff method: results from a simulation study. BMC Med Educ. 16:134.
   PubMed Web of Science ®Google Scholar
• Skakun EN, Kling S. 1980. Comparability of methods for setting standards. J Educ Meas. 17:229–235.
   Web of Science ®Google Scholar
• Taube KT. 1997. The incorporation of empirical item difficulty data into the angoff standard-setting procedure. Eval Health Prof. 20:479–498.
   PubMed Web of Science ®Google Scholar
• Tavakol. 2013. Psychometric evaluation of a knowledge based examination using Rasch analysis: an illustrative guide: AMEE Guide No. 72. Med Teach. 35:74–84.
   Web of Science ®Google Scholar
• Taylor CA. 2011. Development of a modified Cohen method of standard setting. Med Teach. 33:e678–e682.
   PubMed Web of Science ®Google Scholar
• Tennant A, Conaghan PG. 2007. The Rasch measurement model in rheumatology: what is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum. 57:1358–1362.
   PubMed Web of Science ®Google Scholar