Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests

References

• Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education 1979;13:41–54.
   PubMed Web of Science ®Google Scholar
• Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The objective structured clinical examination (OSCE): AMEE Guide no. 81. Part I: An historical and theoretical perspective. Medical Teacher 2013;35:e1437–46.
   PubMed Web of Science ®Google Scholar
• Pugh D, Touchie C, Wood TJ, Humphrey Murto S. Progress testing: Is there a role for the OSCE? Medical Education 2014;6:623–31.
   Web of Science ®Google Scholar
• Gold J, DeMuth R, Mavis B, Wagner D. Progress testing 2.0: Clinical skills meets necessary science. Medical Education Online 2015;20. Available at: http://med-ed-online.net/index.php/meo/article/view/27769. Accessed April 28, 2016.
   PubMed Web of Science ®Google Scholar
• Joiner AB, Abbott S, Longson D. Is performance-based progress testing in psychiatry feasible? Academic Psychiatry. Published Online: December 14, 2015, 2015. (doi:10.1007/s40596-015-0464-0).
   Google Scholar
• Pugh D, Smee S. Guidelines for the development of objective structured clinical examination (OSCE) cases. Medical Council of Canada. Available at: http://mcc.ca/wp-content/uploads/osce-booklet-2014.pdf. Accessed November 23, 2015.
   Google Scholar
• Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine 1998;73:993–7.
   PubMed Web of Science ®Google Scholar
• Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Academic Medicine 1999;74:1129–34.
   PubMed Web of Science ®Google Scholar
• Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Medical Education 2003;37:1012–6.
   PubMed Web of Science ®Google Scholar
• Ilgen JS, Ma IWY, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education 2015;49:161–73.
   PubMed Web of Science ®Google Scholar
• Williams R, Klamen D, McGaghie W. Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine 2003;15:270–92.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KE. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Academic Medicine 2011;86:S1–7.
   PubMed Web of Science ®Google Scholar
• McManus IC, Thompson M, Mollon J. Assessment of examiner leniency and stringency (‘hawk-dove effect’) in the MRCP(UK) clinical examination (PACES) using multi-facet rasch modelling. BMC Medical Education 2006;6:42.
   PubMedGoogle Scholar
• Bartman I, Smee S, Roy M. A method for identifying extreme OSCE examiners. The Clinical Teacher 2013;10:27–31.
   PubMedGoogle Scholar
• Gorman CA, Rentsch JR. Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology 2009;94:1336–44.
   PubMed Web of Science ®Google Scholar
• Cook D, Dupras D, Beckman T, Thomas K, Pankratz V. Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. Journal of General Internal Medicine 2009;24:74–9.
   PubMed Web of Science ®Google Scholar
• Castorr AH, Thompson KO, Ryan JW, Phillips CY, Prescott PA, Soeken KL. The process of rater training for observational instruments: Implications for interrater reliability. Research in Nursing and Health 1990;13:311–8.
   PubMed Web of Science ®Google Scholar
• Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: Rater performance and behavior when considering multiple competencies. Teaching and Learning in Medicine 2016;28:41–51.
   PubMed Web of Science ®Google Scholar
• Gofton WT, Dudek NL, Wood TJ, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A tool to assess surgical competence. Academic Medicine 2012;87:1401–7.
   PubMed Web of Science ®Google Scholar
• Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Medical Education 2011;45:560–9.
   PubMed Web of Science ®Google Scholar
• Sandilands DD, Gotzmaa A, Roy M, Zumbo BD, De Champlain A. Weighting checklist items and station components on a large-scale OSCE: Is it worth the effort. Medical Teacher 2014;36:585–90.
   PubMed Web of Science ®Google Scholar
• LaRochelle J, Durning SJ, Boulet JR, van der Vleuten C, van Merrienboer J, Donkers J. Beyond standard checklist assessment: Question sequence may impact student performance. Perspectives on Medical Education 2016;5:95–102.
   PubMedGoogle Scholar
• Daniels VJ, Bordage G, Gierl MJ, Yudkowsky R. Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE. Advances in Health Sciences Education 2014;19:497–506.
   PubMed Web of Science ®Google Scholar
• Bartman I, Smee S, Fraser J. Traditional Objective Structured Clinical Exams (OSCE) checklist versus key features. Medical Education 2010;44(Suppl 2):47.
   Google Scholar
• Kogan J, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education 2011;45:1048–60.
   PubMed Web of Science ®Google Scholar
• Berendonk C, Stalmeijer R, Schuwirth LWT. Expertise in performance assessment: Assessors' perspectives. Advances in Health Sciences Education 2013;18:559–71.
   PubMed Web of Science ®Google Scholar
• Smee S, Bartman I, Sebok S. Two-Point OSCE Checklist Items: Do they better capture examiner judgments? Available at: https://www.proreg.ca/events/ccme/archive_2014/schedule_abs.php?id=100937. Accessed April 28, 2016.
   Google Scholar
• Sebok S. Understanding the shades of grey between “done” and “not done” in Objective Structured Clinical Examination (OSCE) assessments. Unpublished manuscript.
   Google Scholar
• Pugh D, Touchie C, Humphrey-Murto S, Wood TJ. The OSCE progress test—Measuring clinical skill development over residency training. Medical Teacher 2015;38:168–73.
   PubMed Web of Science ®Google Scholar
• Pugh D, Bhanji F, Cole G, Dupre J, Hatala R, Humphrey-Murto S, et al. Do OSCE progress test scores predict performance on a national high-stakes examination? Medical Education 2016;50:351–8.
   PubMed Web of Science ®Google Scholar
• Holmboe E, Sherbino J, Long D, Swing S, Frank J. The role of assessment in competency-based medical education. Medical Teacher 2010;32:676–82.
   PubMed Web of Science ®Google Scholar
• Royal College of Physicians and Surgeons of Canada. Objectives of training in Internal Medicine, Version 1.0. 2011. Available at: http://www.royalcollege.ca/cs/groups/public/documents/document/y2vk/mdaw/∼edisp/tztest3rcpsced000910.pdf. Accessed June 9, 2016.
   Google Scholar
• Wood T, Humphrey-Murto S, Norman G. Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Advances in Health Sciences Education 2006;11:115.
   PubMed Web of Science ®Google Scholar
• Feldt LS, Woodruff DJ, Salih FA. Statistical inference for coefficient alpha. Applied Psychological Measurement 1987;11:93–103.
   Web of Science ®Google Scholar
• Sergeant J, Armson H, Chesluk B, Dornan T, Eva K, Holmboe E, et al. The processes and dimensions of informed self-assessment: A conceptual model. Academic Medicine 2010;85:1212–20.
   PubMed Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association, 2014.
   Google Scholar
• Downing SM, Haladyna T. Validity and its threats. In SM Downing & R Yudkowsky (Eds.), Assessment in health professions education (pp. 21–55). New York, NY: Routledge, 2009.
   Google Scholar