Interrater reliability in neurology objective structured clinical examination across specialties

References

• Blood AD, Park YS, Lukas RV, Brorson JR. 2015. Neurology objective structured clinical examination reliability using generalizability theory. Neurology [Internet]. 85(18):1623–1629. doi: 10.1212/WNL.0000000000002053.
   PubMed Web of Science ®Google Scholar
• Donohoe CL, Reilly F, Donnelly S, Cahill RA. 2020. Is there variability in scoring of student surgical OSCE performance based on examiner experience and expertise? J Surg Educ. 77(5):1202–1210. doi: 10.1016/j.jsurg.2020.03.009.
   PubMed Web of Science ®Google Scholar
• Downing SM. 2005. Threats to the validity of clinical teaching assessments: what about rater error? Med Educ. 39(4):353–355. doi: 10.1111/j.1365-2929.2005.02138.x.
   PubMed Web of Science ®Google Scholar
• Floreck LM, De Champlain AF. 2001. Assessing sources of score variability in a multisite medical performance assessment: an application of hierarchial linear modeling. Acad Med. 76(Supplement):S93–S95. doi: 10.1097/00001888-200110001-00031.
   PubMedGoogle Scholar
• Govaerts MJB, van der Vleuten CPM, Schuwirth LWT. 2002. Optimising the reproducibility of a performance-based assessment test in midwifery education. Adv Health Sci Educ Theory Pract. 7(2):133–145.
   PubMed Web of Science ®Google Scholar
• Harasym PH, Woloschuk W, Cunning L. 2008. Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs. Adv Health Sci Educ Theory Pract. 13(5):617–632. doi: 10.1007/s10459-007-9068-0.
   PubMed Web of Science ®Google Scholar
• Harden RM, Stevenson M, Downie WW, Wilson GM. 1975. Assessment of clinical competence using objective structured examination. BMJ [Internet]. 1(5955):447–451. [accessed 2021 Sep 5]. doi: 10.1136/bmj.1.5955.447.
   PubMed Web of Science ®Google Scholar
• Hodges B, McIlroy JH. 2003. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 37(11):1012–1016. doi: 10.1046/j.1365-2923.2003.01674.x.
   PubMed Web of Science ®Google Scholar
• Homer M. 2022. Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes. Adv Health Sci Educ Theory Pract. 27(2):457–473. doi: 10.1007/s10459-022-10096-9.
   PubMed Web of Science ®Google Scholar
• Hope D, Cameron H. 2015. Examiners are most lenient at the start of a two-day OSCE. Med Teach. 37(1):81–85. doi: 10.3109/0142159X.2014.947934.
   PubMed Web of Science ®Google Scholar
• Humphrey-Murto S, Smee S, Touchie C, Wood TJ, Blackmore DE. 2005. A comparison of physician examiners and trained assessors in a high-stakes OSCE setting. Acad Med [Internet]. 80(Supplement):S59–S62. doi: 10.1097/00001888-200510001-00017.
   PubMedGoogle Scholar
• Koo TK, Li MY. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 15(2):155–163. doi: 10.1016/j.jcm.2016.02.012.
   PubMed Web of Science ®Google Scholar
• Landy FJ, Farr JL. 1980. Performance rating. Psychol Bull [Internet]. 87(1):72–107. doi: 10.1037/0033-2909.87.1.72.
   Web of Science ®Google Scholar
• Ma IWY, Zalunardo N, Pachev G, Beran T, Brown M, Hatala R, McLaughlin K. 2012. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract. 17(4):457–470. doi: 10.1007/s10459-011-9322-3.
   PubMed Web of Science ®Google Scholar
• McManus I, Thompson M, Mollon J. 2006. Assessment of examiner leniency and stringency ('hawk-dove effect’) in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Med Educ. 6(1):42. doi: 10.1186/1472-6920-6-42.
   PubMedGoogle Scholar
• Nicholl DJ, Appleton JP. 2015. Clinical neurology: why this still matters in the 21st century. J Neurol Neurosurg Psychiatry. 86(2):229–233. [accessed 2021 Sep 5]. doi: 10.1136/jnnp-2013-306881.
   PubMed Web of Science ®Google Scholar
• Pell G, Homer M, Fuller R. 2015. Investigating disparity between global grades and checklist scores in OSCEs. Med Teach. 37(12):1106–1113. doi: 10.3109/0142159X.2015.1009425.
   PubMed Web of Science ®Google Scholar
• Pell G, Homer MS, Roberts TE. 2008. Assessor training: its effects on criterion‐based assessment in a medical context. Int J Res Method in Educ [Internet]. 31(2):143–154. doi: 10.1080/17437270802124525.
   Google Scholar
• Regehr G, MacRae H, Reznick RK, Szalay D. 1998. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination: academic Medicine [Internet]. Acad Med. 73(9):993–997. doi: 10.1097/00001888-199809000-00020.
   PubMed Web of Science ®Google Scholar
• Sebok SS, Roy M, Klinger DA, De Champlain AF. 2015. Examiners and content and site: oh My! A national organization’s investigation of score variation in large-scale performance assessments. Adv Health Sci Educ Theory Pract. 20(3):581–594. doi: 10.1007/s10459-014-9547-z.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Frampton CM, Thompson-Fawcett M, Egan T. 2003. Objectivity in objective structured clinical examinations: Checklists are no substitute for examiner commitment. Acad Med. 78(2):219–223. doi: 10.1097/00001888-200302000-00021.
   PubMed Web of Science ®Google Scholar
• Wilson GM, Lever R, Harden RM, Robertson JIS, Macritchie J. 1969. Examination of clinical examiners. The Lancet [Internet]. 293(7584):37–40. doi: 10.1016/S0140-6736(69)90998-2.
   Google Scholar
• Wong ML, Fones CSL, Aw M, Tan CH, Low PS, Amin Z, Wong PS, Goh PS, Wai C-T, Ong B, et al. 2007. Should non-expert clinician examiners be used in objective structured assessment of communication skills among final year medical undergraduates? Med Teach. 29(9):927–932. [accessed 2021 Sep 5]. doi: 10.1080/01421590701601535.
   PubMed Web of Science ®Google Scholar
• Yeates P, Cope N, Hawarden A, Bradshaw H, McCray G, Homer M. 2019. Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs. Med Educ. 53(3):250–263. doi: 10.1111/medu.13783.
   PubMed Web of Science ®Google Scholar
• Yeates P, Moult A, Cope N, McCray G, Xilas E, Lovelock T, Vaughan N, Daw D, Fuller R, McKinley RK. (Bob). 2021. Measuring the effect of examiner variability in a multiple-circuit objective structured clinical examination (OSCE). Academic Medicine [Internet]. 96(8):1189–1196. doi: 10.1097/ACM.0000000000004028.
   PubMed Web of Science ®Google Scholar
• Yeates P, Sebok-Syer SS. 2017. Hawks, Doves and Rasch decisions: understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling. Med Teach. 39(1):92–99. doi: 10.1080/0142159X.2017.1248916.
   PubMed Web of Science ®Google Scholar