Shining a spotlight on scoring in the OSCE: Checklists and item weighting

References

• American Educational Research Association. 2014. Standards for educational and psychological testing. Washington (DC): American Educational Research Association.
   Google Scholar
• Chandler P, Sweller J. 1991. Cognitive load theory and the format of instruction. Cogn Instr. 8(4):293–332.
   Web of Science ®Google Scholar
• Cizek GJ, Bunch MB. 2007. Standard setting: a guide to establishing and evaluating performance standards on tests. 1st ed. Thousand Oaks (CA): SAGE Publications, Inc.
   Google Scholar
• Cook DA, Brydges R, Ginsburg S, Hatala R. 2015. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med Educ. 49(6):560–575.
   PubMed Web of Science ®Google Scholar
• Daniels VJ, Bordage G, Gierl MJ, Yudkowsky R. 2014. Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE. Adv Health Sci Educ Theory Pract. 19(4):497–506.
   PubMed Web of Science ®Google Scholar
• Farmer EA, Page G. 2005. A practical guide to assessing clinical decision-making skills using the key features approach. Med Educ. 39(12):1188–1194.
   PubMed Web of Science ®Google Scholar
• Fuller R, Homer M, Pell G. 2013. Longitudinal interrelationships of OSCE station level analyses, quality improvement and overall reliability. Med Teach. 35(6):515–517.
   PubMed Web of Science ®Google Scholar
• Fuller R, Homer M, Pell G, Hallam J. 2017. Managing extremes of assessor judgment within the OSCE. Med Teach. 39(1):58–66.
   PubMed Web of Science ®Google Scholar
• General Medical Council. 2018. Outcomes for graduates 2018. London: General Medical Council.
   Google Scholar
• Harden R, Lilley P, Patricio M. 2015. The definitive guide to the OSCE: the objective structured clinical examination as a performance assessment. 1st ed. Edinburgh; New York: Churchill Livingstone.
   Google Scholar
• Harden RM, Stevenson M, Downie WW, Wilson GM. 1975. Assessment of clinical competence using objective structured examination. Br Med J. 1(5955):447–451.
   PubMed Web of Science ®Google Scholar
• Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad Med. 74(10):1129–1134.
   PubMed Web of Science ®Google Scholar
• Homer M, Fuller R, Pell G. 2017. The benefits of sequential testing: improved diagnostic accuracy and better outcomes for failing students. Med Teach. 40(3):275–284.
   PubMed Web of Science ®Google Scholar
• Homer M, Pell G, Fuller R. 2017. Problematizing the concept of the “borderline” group in performance assessments. Med Teach. 39(5):469–475.
   PubMed Web of Science ®Google Scholar
• Ilgen JS, Ma IWY, Hatala R, Cook DA. 2015. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 49(2):161–173.
   PubMed Web of Science ®Google Scholar
• Kahraman N, Clauser BE, Margolis MJ. 2008. A comparison of alternative item weighting strategies on the data gathering component of a clinical skills performance assessment. Acad Med. 83(10 Suppl):S72–S75.
   PubMed Web of Science ®Google Scholar
• Kane MT. 2006. Validation. In: Brennan RL, editor. Educational measurement. Westport (CT): Praeger Publishers; p. 17–64.
   Google Scholar
• Khan KZ, Gaunt K, Ramachandran S, Pushkar P. 2013. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Med Teach. 35(9):e1447–e1463.
   PubMed Web of Science ®Google Scholar
• Khan KZ, Ramachandran S, Gaunt K, Pushkar P. 2013. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 35(9):e1437.
   PubMed Web of Science ®Google Scholar
• Lafleur A, Côté L, Leppink J. 2015. Influences of OSCE design on students' diagnostic reasoning. Med Educ. 49(2):203–214.
   PubMed Web of Science ®Google Scholar
• Ma IWY, Zalunardo N, Pachev G, Beran T, Brown M, Hatala R, McLaughlin K. 2012. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ. 17(4):457–470.
   PubMed Web of Science ®Google Scholar
• McKinley DW, Norcini JJ. 2014. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach. 36(2):97–110.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts T, 2010. How to measure the quality of the OSCE: a review of metrics – AMEE Guide No. 49. Med Teach. 32(10):802–811.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts T. 2013. Advancing the objective structured clinical examination: sequential testing in theory and practice. Med Educ. 47(6):569–577.
   PubMed Web of Science ®Google Scholar
• Pell G, Homer M, Fuller R. 2015. Investigating disparity between global grades and checklist scores in OSCEs. Med Teach. 37(12):1106–1113.
   PubMed Web of Science ®Google Scholar
• Pugh D, Halman S, Desjardins I, Humphrey-Murto S, Wood TJ. 2016. Done or almost done? Improving OSCE checklists to better capture performance in progress tests. Teach Learn Med. 28(4):406–414.
   PubMed Web of Science ®Google Scholar
• Regehr G, Freeman R, Robb A, Missiha N, Heisey R. 1999. OSCE performance evaluations made by standardized patients: comparing checklist and global rating scores. Acad Med. 74(10 Suppl):S135–S137.
   PubMed Web of Science ®Google Scholar
• Regehr G, MacRae H, Reznick RK, Szalay D. 1998. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med J Assoc Am Med Coll. 73(9):993–997.
   Web of Science ®Google Scholar
• Revelle W, Zinbarg RE. 2009. Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma. Psychometrika. 74(1):145–154.
   Web of Science ®Google Scholar
• Ringsted C, Østergaard D, Ravn L, Pedersen JA, Berlac PA, Vleuten C. 2003. A feasibility study comparing checklists and global rating forms to assess resident performance in clinical skills. Med Teach. 25(6):654–658.
   PubMed Web of Science ®Google Scholar
• Sadler DR. 2009. Indeterminacy in the use of preset criteria for assessment and grading. Assess Eval High Educ. 34(2):159–179.
   Web of Science ®Google Scholar
• Sandilands DD, Gotzmann A, Roy M, Zumbo BD, De Champlain A. 2014. Weighting checklist items and station components on a large-scale OSCE: is it worth the effort? Med Teach. 36(7):585–590.
   PubMed Web of Science ®Google Scholar
• Schuwirth LWT, van der Vleuten C. 2012. Programmatic assessment and Kane's validity perspective. Med Educ. 46(1):38–48.
   PubMed Web of Science ®Google Scholar
• Streiner DL, Norman GR. 2008. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford; New York: Oxford University Press.
   Google Scholar
• Tavares W, Eva KW. 2013. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract. 18(2):291–303.
   PubMed Web of Science ®Google Scholar
• Wainer H. 1976. Estimating coefficients in linear models: it don’t make no nevermind. Psychol Bull. 83(2):213–217.
   Web of Science ®Google Scholar
• Wilkinson TJ, Frampton CM, Thompson-Fawcett M, Egan T. 2003. Objectivity in objective structured clinical examinations: checklists are no substitute for examiner commitment. Acad Med. 78(2):219–223.
   PubMed Web of Science ®Google Scholar
• Wood TJ, Pugh D. 2019. Are rating scales really better than checklists for measuring increasing levels of expertise? Med Teach. 0(0):1–6.
   Google Scholar
• Yudkowsky R, Park YS, Riddle J, Palladino C, Bordage G. 2014. Clinically discriminating checklists versus thoroughness checklists: improving the validity of performance test scores. Acad Med J Assoc Am Med Coll. 89(7):1057–1062.
   Web of Science ®Google Scholar