Inter-school variations in the standard of examiners’ graduation-level OSCE judgements

References

• Norcini J, Anderson MB, Bollela V, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102–1109. doi: 10.1080/0142159X.2018.1500016
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Kogan JR. Will any road get you there? Examining warranted and unwarranted variation in medical education. Acad Med. 2022;97(8):1128–1136. doi: 10.1097/ACM.0000000000004667
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti LN, Iobst WF, et al. Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med. 2014;89(5):721–727. doi: 10.1097/ACM.0000000000000221
   PubMed Web of Science ®Google Scholar
• Hernandez CA, Daroowalla F, Larochelle JS, et al. Determining grades in the internal medicine clerkship: results of a national survey of clerkship directors. Acad Med. 2021;96(2):249–255. doi: 10.1097/ACM.0000000000003815
   PubMed Web of Science ®Google Scholar
• MacDougall M. Variation in assessment and standard setting practices across UK undergraduate medicine and the need for a benchmark. Int J Med Educ. 2015;6:125–135. doi: 10.5116/ijme.560e.c964
   PubMed Web of Science ®Google Scholar
• Taylor CA, Gurnell M, Melville CR, et al. Variation in passing standards for graduation-level knowledge items at UK medical schools. Med Educ. 2017;51(6):612–620. doi: 10.1111/medu.13240
   PubMed Web of Science ®Google Scholar
• Ward H, Chiavaroli N, Fraser J, et al. Standard setting in Australian medical schools. BMC Med Educ. 2018;18(1):80. doi: 10.1186/s12909-018-1190-6
   PubMedGoogle Scholar
• Yeates P, Cope N, Luksaite E, et al. Exploring differences in individual and group judgements in standard setting. Med Educ. 2019;53(9):941–952. doi: 10.1111/medu.13915
   PubMed Web of Science ®Google Scholar
• Hope D, Kluth D, Homer M, et al. Variation in performance on common content items at UK medical schools. BMC Med Educ. 2021;21(1):323. doi: 10.1186/s12909-021-02761-1
   PubMed Web of Science ®Google Scholar
• Norcini JJ, Boulet JR, Opalek A, et al. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89(8):1157–1162. doi: 10.1097/ACM.0000000000000310
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Cate O. T, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–645. doi: 10.3109/0142159X.2010.501190
   PubMed Web of Science ®Google Scholar
• Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: are we addressing the concerns and challenges? Med Educ. 2015;49(11):1086–1102. (doi: 10.1111/medu.12831
   PubMed Web of Science ®Google Scholar
• Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004;38(2):199–203. doi: 10.1046/j.1365-2923.2004.01755.x
   PubMed Web of Science ®Google Scholar
• Boursicot K, Kemp S, Wilkinson T, et al. Performance assessment: consensus statement and recommendations from the 2020 Ottawa Conference. Med Teach. 2021;43(1):58–67. doi: 10.1080/0142159X.2020.1830052
   PubMed Web of Science ®Google Scholar
• Sebok SS, Roy M, Klinger D. A, et al. Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments. Adv Health Sci Educ Theory Pract. 2015;20(3):581–594. doi: 10.1007/s10459-014-9547-z
   PubMed Web of Science ®Google Scholar
• Floreck LM, Champlain AF. Assessing sources of score variability in a multi-site medical performance assessment: an application of hierarchical linear modeling. Acad Med. 2001;76(10 Suppl):S93–S95. doi: 10.1097/00001888-200110001-00031
   PubMed Web of Science ®Google Scholar
• Tamblyn RM, Klass DJ, Schnabl GK, et al. Sources of unreliability and bias in standardized‐patient rating. Teach Learn Med. 1991;3(2):74–85. doi: 10.1080/10401339109539486
   Google Scholar
• Yeates P, Cope N, Hawarden A, et al. Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs. Med Educ. 2019;53(3):250–263. doi: 10.1111/medu.13783
   PubMed Web of Science ®Google Scholar
• Yeates P, Moult A, Cope N, et al. Measuring the effect of examiner variability in a multiple-circuit objective structured clinical examination (OSCE). Acad Med. 2021;96(8):1189–1196. doi: 10.1097/ACM.0000000000004028
   PubMed Web of Science ®Google Scholar
• Yeates P, Maluf A, Cope N, et al. Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE). BMC Med Educ. 2023;23(1):803. doi: 10.1186/s12909-023-04774-4
   PubMed Web of Science ®Google Scholar
• Yeates P, Moult A, Lefroy J, et al. Understanding and developing procedures for video-based assessment in medical education. Med Teach. 2020;42(11):1250–1260. doi: 10.1080/0142159X.2020.1801997
   PubMed Web of Science ®Google Scholar
• Yeates P, McCray G, Moult A, et al. Determining the influence of different linking patterns on the stability of students’ score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA). BMC Med Educ. 2022;22(1):41. doi: 10.1186/s12909-022-03115-1
   PubMed Web of Science ®Google Scholar
• Yeates P, Moult A, Cope N, et al. Determining influence, interaction and causality of contrast and sequence effects in objective structured clinical exams. Med Educ. 2022;56(3):292–302. doi: 10.1111/medu.14713
   PubMed Web of Science ®Google Scholar
• Yeates P, McCray G. n.d. Investigating the accuracy of adjusting for examiner differences in multi-centre Objective Structured Clinical Exams (OSCEs). A simulation study of Video-based Examiner Score Comparison and Adjustment (VESCA). In submission with BMC Medical Education – pre-print available at: https://www.researchsquare.com/article/rs-4151118/v1.
   Google Scholar
• Yeates P, Sebok-Syer SS. Hawks, Doves and Rasch decisions: understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling. Med Teach. 2017;39(1):92–99. doi: 10.1080/0142159X.2017.1248916
   PubMed Web of Science ®Google Scholar
• Yeates P, Maluf A, Kinston R, et al. Enhancing Authenticity, Diagnosticity and Equivalence (AD-Equiv) in multi-centre OSCE exams in Health Professionals Education. Protocol for a Complex Intervention Study. BMJ Open. 2022;12(12):e064387. doi: 10.1136/bmjopen-2022-064387
   PubMed Web of Science ®Google Scholar
• Yeates P, Maluf A, Kinston R, et al. A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness. Med Teach. 2024:1–9. doi: 10.1080/0142159X.2024.2339413
   PubMed Web of Science ®Google Scholar
• Lefroy J, Gay SP, Gibson S, et al. Development and face validation of an instrument to assess and improve clinical consultation skills. Int J Clin Skills. 2011;5(2):115–125.
   Google Scholar
• IBM Corporation. 2019. IBM SPSS Statistics for Windows (26).
   Google Scholar
• Bates D, Mächler M, Bolker B, et al. Fitting linear mixed-effects models using lme4. J Stat Soft. 2015;67(1):1–48. doi: 10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Core Team R. 2021. R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/.
   Google Scholar
• Fox J. 2015. Applied regression analysis and generalized linear models. 3rd Edition. SAGE Publications, Inc.
   Google Scholar
• Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education procedures for establishing defensible absolute passing scores on performan. Teach Learn Med. 2006;18(1):50–57. doi: 10.1207/s15328015tlm1801
   PubMed Web of Science ®Google Scholar
• Tighe J, McManus IC, Dewhurst NG, et al. The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations. BMC Med Educ. 2010;10(1):40. doi: 10.1186/1472-6920-10-40
   PubMedGoogle Scholar
• Homer M. Towards a more nuanced conceptualisation of differential examiner stringency in OSCEs. Adv in Health Sci Educ. 2023; doi: 10.1007/s10459-023-10289-w
   PubMed Web of Science ®Google Scholar
• GMC. n.d. Requirements for the MLA clinical and professional skills assessment. [accessed 2023 December 20]. https://www.gmc-uk.org/education/medical-licensing-assessment/uk-medical-schools-guide-to-the-mla/clinical-and-professional-skills-assessment-cpsa.
   Google Scholar
• Bleakley A. Broadening conceptions of learning in medical education: the message from teamworking. Med Educ. 2006;40(2):150–157. doi: 10.1111/j.1365-2929.2005.02371.x
   PubMed Web of Science ®Google Scholar
• Chandratilake M, Mcaleer S, Gibson J. Cultural similarities and differences in medical professionalism: a multi-region study. Med Educ. 2012;46(3):257–266. doi: 10.1111/j.1365-2923.2011.04153.x
   PubMed Web of Science ®Google Scholar
• Beck C, Brown C. Could the UK Foundation Programme training post allocation process result in regional variations in the knowledge and skills of Foundation doctors? A cross-sectional study. Health Sci Rep. 2020;3(4):e201. doi: 10.1002/hsr2.201
   PubMedGoogle Scholar
• McManus IC, Harborne AC, Horsfall HL, et al. Exploring UK medical school differences: the MedDifs study of selection, teaching, student and F1 perceptions, postgraduate outcomes and fitness to practise. BMC Med. 2020;18(1):136. doi: 10.1186/s12916-020-01572-3
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment : the missing link in competency-based medical education. Acad Med. 2011;86(4):460–467. doi: 10.1097/ACM.0b013e31820cb2a7
   PubMed Web of Science ®Google Scholar
• Cook D. A, Dupras DM, Beckman TJ, et al. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Gen Intern Med. 2009;24(1):74–79. doi: 10.1007/s11606-008-0842-3
   PubMed Web of Science ®Google Scholar
• Kogan JR, Dine CJ, Conforti LN, et al. Can rater training improve the quality and accuracy of workplace-based assessment narrative comments and entrustment ratings? A randomized controlled trial. Acad Med. 2023;98(2):237–247. doi: 10.1097/ACM.0000000000004819
   PubMed Web of Science ®Google Scholar