Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

References

• Akinola L, Khan L. 2022. RCOG Differential Attainment Report 2021. https://www.rcog.org.uk/media/ctalnj3f/tef-2021-report-differential-attainment.pdf.
   Google Scholar
• Albanese M, Case SM. 2016. Progress testing: critical analysis and suggested practices. Adv Health Sci Educ Theory Pract. 21(1):221–234. doi: 10.1007/s10459-015-9587-z.
   PubMed Web of Science ®Google Scholar
• American Board of Anesthesiology. 2022. Policy Book. Raleigh (NC): American Board of Anesthesiology.
   Google Scholar
• American Board of Internal Medicine. 2022. ABIM MOC Requirements. Philadelphia (PA): NEJM. https://knowledgeplus.nejm.org/board-review/abim-moc-requirements/.
   Google Scholar
• American Board of Medical Specialties. 2014. Standards for the ABMS program for maintenance of Certification (MOC). Chicago (IL): American Board of Medical Specialties.
   Google Scholar
• American Board of Surgery. 2022. Continuous Certification. https://www.absurgery.org/default.jsp?exam-moc.
   Google Scholar
• American Society of Anesthesiologists. 2022. Anesthesiology ITE. American Society of Anesthesiologists. https://www.asahq.org/education-and-career/asa-resident-component/study-resources-and-education/anesthesiology-ite.
   Google Scholar
• ANZCA [@ANZCA]. 2020. March 6). UPDATE TO MEDICAL VIVAS: Due to concerns over #COVID19 @ANZCA has cancelled the Medical Vivas due to be held in Australian and NZ hospitals on Saturday 14 March. All candidates have been notified. More info at: http://bit.ly/2wtitip. [Tweet]. Twitter. https://twitter.com/ANZCA/status/1235722850259972098.
   Google Scholar
• Archer J, Lynn N, Coombes L, Roberts M, Gale T, Price T, Regan de Bere S. 2016. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC Med Edu. 16(1):. 212. 10.1186/s12909-016-0729-7.
   PubMed Web of Science ®Google Scholar
• Baker K. 2011. Determining resident clinical performance. Getting beyond the noise. Anesthesiology. 115(4):862–878. doi: 10.1097/ALN.0b013e318229a27d.
   PubMed Web of Science ®Google Scholar
• Baker K, Sun H, Harman A, Poon KT, Rathmell JP. 2016. Clinical performance scores are independently associated with the American Board of Anesthesiology certification examination scores. Anesth Analg. 122(6):1992–1999. doi: 10.1213/ANE.0000000000001288.
   PubMed Web of Science ®Google Scholar
• Bates J, Schrewe B, Ellaway RH, Teunissen PW, Watling C. 2019. Embracing standardisation and contextualisation in medical education. Med Educ. 53(1):15–24. doi: 10.1111/medu.13740.
   PubMed Web of Science ®Google Scholar
• Bricker S. 2017. The Anaesthesia Science Viva Book. 3rd ed. Cambridge: Cambridge University Press.
   Google Scholar
• Castanelli DJ, Moonen-van Loon JMW, Jolly B, Weller JM. 2019. The reliability of a portfolio of workplace-based assessments in anesthesia training. Can J Anaesth. 66(2):193–200. doi: 10.1007/s12630-018-1251-7.
   PubMed Web of Science ®Google Scholar
• Castanelli DJ, Wickramaarachchi SA, Wallis S. 2017. Burnout and the learning environment of anaesthetic trainees. Anaesth Intensive Care. 45(6):744–751. doi: 10.1177/0310057X1704500615.
   PubMed Web of Science ®Google Scholar
• Chen Y, Henning M, Yielde J, Jones R, Wearn A, Weller J. 2015. Progress testing in the medical curriculum: students” approaches to learning and perceived stress. BMC Med Educ. 15:147. doi: 10.1186/s12909-015-0426-y.
   PubMed Web of Science ®Google Scholar
• Chin M, Lagasse RS. 2017. Assessment of competence: developing trends and ethical considerations. Curr Opin Anaesthesiol. 30(2):236–241. doi: 10.1097/ACO.0000000000000431.
   PubMed Web of Science ®Google Scholar
• Cizek GJ. 2001. More unintended consequences of high-stakes testing. EdM: IPractice. 20(4):19–27. doi: 10.1111/j.1745-3992.2001.tb00072.x.
   Google Scholar
• Colson YL, Putnam JB, Yang SC, Fann JI, Vaporciyan AA, Dearani JA, Jones DR, Allen MS, Meyers BF, Wright CD, et al. 2019. American board of thoracic surgery 10-year maintenance of certification exam improves and validates knowledge acquisition. Ann Thorac Surg. 108(6):1895–1900. doi: 10.1016/j.athoracsur.2019.05.074.
   PubMed Web of Science ®Google Scholar
• Cook DA, Blachman MJ, West CP, Wittich CM. 2016. Physician attitudes about maintenance of certification: a cross-specialty national survey. Mayo Clin Proc. 91(10):1336–1345. doi: 10.1016/j.mayocp.2016.07.004.
   PubMed Web of Science ®Google Scholar
• Craig S, Braitberg G, Nicolas C, White G, Egerton-Warburton D. 2010. Assessment and feedback in emergency medicine training: views of Australasian emergency trainees: assessment and feedback in emergency medicine. Emerg Med Australas. 22(6):537–547. doi: 10.1111/j.1742-6723.2010.01353.x.
   PubMed Web of Science ®Google Scholar
• Dave S, Chakravorty I, Menon G, Sidhu K, Bamrah J, Mehta R. 2020. Differential attainment in summative assessments within postgraduate medical education & training: 2020 thematic series on tackling differential attainment in healthcare professions. Sushruta J Health Policy Opinion. 13(3):1–7 doi: 10.38192/13.3.15.
   Google Scholar
• De Champlain AF. 2010. A primer on classical test theory and item response theory for assessments in medical education: classical test theory and item response theory. Med Educ. 44(1):109–117. doi: 10.1111/j.1365-2923.2009.03425.x.
   PubMed Web of Science ®Google Scholar
• Devakumar D, Mandeville K. 2011. Should UK membership exams be held overseas? No. BMJ. 343:d4560. doi: 10.1136/bmj.d4560.
   PubMedGoogle Scholar
• Dijksterhuis MGK, Schuwirth LWT, Braat DDM, Scheele F. 2013. An exploratory study into the impact and acceptability of formatively used progress testing in postgraduate obstetrics and gynaecology. Perspect Med Educ. 2(3):126–141. doi: 10.1007/s40037-013-0063-2.
   PubMedGoogle Scholar
• Dwyer T, Takahashi SG, Hynes MK, Herold J, Wasserstein D, Nousiainen M, Ferguson P, Wadey V, Murnaghan ML, Leroux T, et al. 2014. How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: use of an objective structured clinical examination (OSCE). Can J Surg. 57(4):230–236. doi: 10.1503/cjs.018813.
   PubMed Web of Science ®Google Scholar
• Ellis R, Brennan PA, Lee AJ, Scrimgeour DSG, Cleland J. 2022. Differential attainment at MRCS according to gender, ethnicity, age and socioeconomic factors: a retrospective cohort study. J R Soc Med. 115(early online access). (7):257–272. doi: 10.1177/01410768221079018.
   PubMed Web of Science ®Google Scholar
• Ellis R, Cleland J, Scrimgeour D, Lee A, Brennan P. 2022a. The impact of disability on performance in a high-stakes postgraduate surgical examination: a retrospective cohort study. J R Soc Med. 115(2):58–68. doi: 10.1177/01410768211032573.
   PubMed Web of Science ®Google Scholar
• Ellis R, Cleland J, Scrimgeour DSG, Lee AJ, Brennan PA. 2022b. A cross-sectional study examining the association between MRCS performance and surgeons receiving sanctions against their medical registration. Surgeon. 20(4):211–215. doi: 10.1016/j.surge.2021.04.003.
   PubMed Web of Science ®Google Scholar
• Esmail A, Roberts C. 2013. Academic performance of ethnic minority candidates and discrimination in the MRCGP examinations between 2010 and 2012: analysis of data. BMJ. 347(2):f5662. doi: 10.1136/bmj.f5662.
   PubMedGoogle Scholar
• Eva K. 2003. On the generality of specificity. Med Educ. 37(7):587–588. doi: 10.1046/j.1365-2923.2003.01563.x.
   PubMed Web of Science ®Google Scholar
• Federation of the Royal Colleges of Physicians. 2022. Specialty Certificate Examinations. Federation of the Royal Colleges of Physicians. https://www.mrcpuk.org/mrcpuk-examinations/specialty-certificate-examinations.
   Google Scholar
• Fisher WG, Schloss EJ. 2016. Medical specialty certification in the United States—A false idol? J Interv Card Electrophysiol. 47(1):37–43. doi: 10.1007/s10840-016-0119-4.
   PubMed Web of Science ®Google Scholar
• Flum E, Maagaard R, Godycki-Cwirko M, Scarborough N, Scherpbier N, Ledig T, Roos M, Steinhäuser J. 2015. Assessing family medicine trainees – what can we learn from the European neighbours? GMS Z Med Ausbild. 32(2):Doc21.
   PubMedGoogle Scholar
• General Medical Council. 2017. Excellence by design: standards for postgraduate curricula. London: General Medical Council.
   Google Scholar
• General Medical Council. 2022. National professional examinations: the expectations. https://www.gmc-uk.org/education/standards-guidance-and-curricula/guidance/national-professional-examinations/the-expectations.
   Google Scholar
• Grant MJ, Booth A. 2009. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 26(2):91–108. doi: 10.1111/j.1471-1842.2009.00848.x.
   PubMed Web of Science ®Google Scholar
• Hall AK, Chaplin T, McColl T, Petrosoniak A, Caners K, Rocca N, Gardner C, Bhanji F, Woods R. 2020. Harnessing the power of simulation for assessment: consensus recommendations for the use of simulation-based assessment in emergency medicine. CJEM. 22(2):194–203. doi: 10.1017/cem.2019.488.
   PubMedGoogle Scholar
• Halman S, Fu AYN, Pugh D. 2020. Entrustment within an objective structured clinical examination (OSCE) progress test: bridging the gap towards competency-based medical education. Med Teach. 42(11):1283–1288. doi: 10.1080/0142159X.2020.1803251.
   PubMed Web of Science ®Google Scholar
• Hatala R, Ainslie M, Kassen BO, Mackie I, Roberts JM. 2006. Assessing the mini-clinical evaluation exercise in comparison to a national specialty examination. Med Educ. 40(10):950–956. doi: 10.1111/j.1365-2929.2006.02566.x.
   PubMed Web of Science ®Google Scholar
• Heubert JP, Hauser RM. 1999. High stakes: testing for tracking, promotion, and graduation. Washington (DC): National Academy Press.
   Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568. doi: 10.3109/0142159X.2013.789134.
   PubMed Web of Science ®Google Scholar
• Howard-McNatt M, Sabel M, Agnese D, Amersi F, Ammori J, Arciero C, Berman R, Dominici L, Fleming RY, Han D, et al. 2019. Maintenance of certification and continuing medical education: are they still required? Ann Surg Oncol. 26(12):3820–3823. doi: 10.1245/s10434-019-07702-3.
   PubMed Web of Science ®Google Scholar
• Hoyt DB. 2017. Looking forward. BullAm Coll Surg. 102(8):10–11.
   PubMedGoogle Scholar
• Hutchinson L, Aitken P, Hayes T. 2002. Are medical postgraduate certification processes valid? A systematic review of the published evidence: are medical postgraduate certification processes valid? Med Educ. 36(1):73–91. doi: 10.1046/j.1365-2923.2002.01120.x.
   PubMed Web of Science ®Google Scholar
• Jaques H. 2013. Royal College of Psychiatrists offers free CPD in response to fee surplus. BMJ. 346:f1161. doi: 10.1136/bmj.f1161.
   Google Scholar
• Jefferies A, Simmons B, Tabak D, Mcilroy JH, Lee K-S, Roukema H, Skidmore M. 2007. Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training. Med Teach. 29(2-3):183–191. doi: 10.1080/01421590701302290.
   PubMed Web of Science ®Google Scholar
• Jones AT, Biester TW, Buyske J, Lewis FR, Malangoni MA. 2014. Using the American board of surgery in-training examination to predict board certification: a cautionary study. J Surg Educ. 71(6):e144–e148. doi: 10.1016/j.jsurg.2014.04.004.
   PubMedGoogle Scholar
• Kahn J, Goodman CR, Albert A, Agarwal A, Jeans E, Tye K, Campbell SR, Marcrom S, Colbert LE. 2020. Top concerns of radiation oncology trainees in 2019: job market, board examinations, and residency expansion. Int J Radiat Oncol Biol Phys. 106(1):19–25. doi: 10.1016/j.ijrobp.2019.07.006.
   PubMed Web of Science ®Google Scholar
• Karay Y, Schauber SK. 2018. A validity argument for progress testing: examining the relation between growth trajectories obtained by progress tests and national licensing examinations using a latent growth curve approach. Med Teach. 40(11):1123–1129. doi: 10.1080/0142159X.2018.1472370.
   PubMed Web of Science ®Google Scholar
• Kearney R, Sullivan P, Skakun E. 2000. Performance on ABA-ASA in-training examination predicts success for RCPSC certification. American board of anesthesiology-American society of anesthesiologists. Royal College of physicians and surgeons of Canada. Can J Anaesth. 47(9):914–918. doi: 10.1007/BF03019676.
   PubMed Web of Science ®Google Scholar
• Kempen PM. 2014. Maintenance of certification: ethics and conflicts of interest. J Am Phys Surg. 19(3):85–89
   Google Scholar
• Kim PY, Wallace DA, Allbritton DW, Altose MD. 2012. Predictors of success on the written anesthesiology board certification examination. Int J Med Educ. 3:225–235 doi: 10.5116/ijme.5096.5d66.
   Google Scholar
• Langham R. 2022. Australian Medical Council statement on the consideration of the Royal Australian and New Zealand College of Psychiatrists Alternate Assessment Pathway. Australian Medical Council. https://www.amc.org.au/wp-content/uploads/2022/03/2022-03-02-AMC-statement-consideration-of-RANZCP-AAP.pdf.
   Google Scholar
• Lee AG, Oetting TA, Blomquist PH, Bradford G, Culican SM, Kloek C, Krishnan C, Lauer AK, Levi L, Naseri A, et al. 2012. A Multicenter analysis of the ophthalmic knowledge assessment program and American board of ophthalmology written qualifying examination Performance. Ophthalmology. 119(10):1949–1953. doi: 10.1016/j.ophtha.2012.06.010.
   PubMed Web of Science ®Google Scholar
• Looseley A, Wainwright E, Cook TM, Bell V, Hoskins S, O”Connor M, Taylor G, Mouton R. 2019. Stress, burnout, depression and work satisfaction among UK anaesthetic trainees; a quantitative analysis of the satisfaction and wellbeing in anaesthetic Training study. Anaesthesia. 74(10):1231–1239., & doi: 10.1111/anae.14681.
   PubMed Web of Science ®Google Scholar
• Luton OW, Mellor K, Robinson DBT, Barber Z, James OP, Powell AGMT, Hemington-Gorse S, Walsh L, Egan RJ, Lewis WG. 2023. Differential attainment in higher surgical training: scoping pan-specialty spectra. Postgrad Med J. 99(1174):849–854. postgradmedj-2022-141638. doi: 10.1136/postgradmedj-2022-141638.
   PubMed Web of Science ®Google Scholar
• Maudsley RF. 1990. Medical licensure: let’s not lose sight of the objective. Can Med Assoc J. 143(2):98–100.
   Web of Science ®Google Scholar
• McAlpine L, Selamaj E, Shannon C, Chis L, Dacre J, Elder A. 2014. UK postgraduate medicine examinations: opportunities for international candidates. Clin Med. 14(5):500–505. doi: 10.7861/clinmedicine.14-5-500.
   PubMed Web of Science ®Google Scholar
• McClintock JC, Gravlee GP. 2010. Predicting success on the certification examinations of the American Board of anesthesiology. Anesthesiology. 112(1):212–219. doi: 10.1097/ALN.0b013e3181c62e2f.
   PubMed Web of Science ®Google Scholar
• McCrary HC, Colbert-Getz JM, Poss WB, Smith BK. 2021. A Systematic review of the relationship between in-training examination scores and specialty board examination scores. j Grad Med Educ. 13(1):43–57. doi: 10.4300/JGME-D-20-00111.1.
   PubMedGoogle Scholar
• McDonald FS, Duhigg LM, Arnold GK, Hafer RM, Lipner RS. 2018. The American Board of Internal Medicine Maintenance of Certification Examination and state medical board disciplinary actions: a population cohort study. J Gen Intern Med. 33(8):1292–1298. doi: 10.1007/s11606-018-4376-z.
   PubMed Web of Science ®Google Scholar
• McManus I, Ludka K. 2012. Resitting a high-stakes postgraduate medical examination on multiple occasions: nonlinear multilevel modelling of performance in the MRCP (UK) examinations. BMC Med. 10(1):60. doi: 10.1186/1741-7015-10-60.
   PubMedGoogle Scholar
• Molleman E, Rink F. 2015. The antecedents and consequences of a strong professional identity among medical specialists. Soc Theory Health. 13(1):46–61. doi: 10.1057/sth.2014.16.
   Web of Science ®Google Scholar
• Moonen-van Loon JMW, Overeem K, Donkers HHLM, Van Der Vleuten CPM, Driessen EW. 2013. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 18(5):1087–1102. doi: 10.1007/s10459-013-9450-z.
   PubMed Web of Science ®Google Scholar
• Muir G, Thacker J. 2011. Should UK membership exams be held overseas? Yes. BMJ. 343(jul28 2):d4566–d4566. doi: 10.1136/bmj.d4566.
   PubMedGoogle Scholar
• Nelson M, Butterworth JF. IV 2018. Maintenance of certification. Has MOC gone amok? Anesthesiology. 129(4):631–633. doi: 10.1097/ALN.0000000000002387.
   PubMed Web of Science ®Google Scholar
• Noel K, Archibald D, Brailovsky C, Mautbur A. 2016. Progress testing in family medicine—A novel use for simulated office oral exams. Med Teach. 38(4):364–368. doi: 10.3109/0142159X.2015.1031733.
   PubMed Web of Science ®Google Scholar
• Pugh D, Bhanji F, Cole G, Dupre J, Hatala R, Humphrey-Murto S, Touchie C, Wood TJ. 2016. Do OSCE progress test scores predict performance in a national high-stakes examination? Med Educ. 50(3):351–358. doi: 10.1111/medu.12942.
   PubMed Web of Science ®Google Scholar
• Rademakers J, ten Cate O, Bär PR. 2005. Progress testing with short answer questions. Med Teach. 27(7):578–582. doi: 10.1080/01421590500062749.
   PubMed Web of Science ®Google Scholar
• Rich JV, Fostaty Young S, Donnelly C, Hall AK, Dagnone JD, Weersink K, Caudle J, Van Melle E, Klinger DA. 2020. Competency‐based education calls for programmatic assessment: but what does this look like in practice? J Eval Clin Pract. 26(4):1087–1095. doi: 10.1111/jep.13328.
   PubMed Web of Science ®Google Scholar
• Ringsted C, Østergaard D, Scherpbier A. 2003. Embracing the new paradigm of assessment in residency training: an assessment programme for first-year residency training in anaesthesiology. Med Teach. 25(1):54–62. doi: 10.1080/0142159021000061431.
   PubMed Web of Science ®Google Scholar
• Ringsted C, Østergaard D, van der Vleuten CPM. 2003. Implementation of a formal in-training assessment programme in anaesthesiology and preliminary results of acceptability. Acta Anaesthesiol Scand. 47(10):1196–1203. doi: 10.1046/j.1399-6576.2003.00255.x.
   PubMed Web of Science ®Google Scholar
• Robson V. 2022. FFICM Chair of Examiners Report – October 2021 FFICM. The Faculty of Intensive Care Medicine. https://www.ficm.ac.uk/sites/ficm/files/documents/2022-01/FFICM%20Chair%27s%20Report%20Oct%202021.pdf.
   Google Scholar
• Royal Australasian College of Physicians. 2022. Examinations. Royal Australasian College of Physicians. https://www.racp.edu.au/trainees/examinations.
   Google Scholar
• Royal Colleges of Physicians of the United Kingdom. 2019a. PACES 2019 annual performance report. https://www.mrcpuk.org/document/paces-2019-annual-performance-report.
   Google Scholar
• Royal Colleges of Physicians of the United Kingdom. 2019b. Part 1 2019 annual performance report. https://www.mrcpuk.org/document/part-1-2019-annual-performance-report.
   Google Scholar
• Royal Colleges of Physicians of the United Kingdom. 2019c. Part 2 2019 annual performance report. https://www.mrcpuk.org/document/part-2-2019-annual-performance-report.
   Google Scholar
• Rutgers DR, van Raamt F, van Lankeren W, Ravesloot CJ, van der Gijp A, ten Cate TJ, van Schaik JPJ. 2018. Fourteen years of progress testing in radiology residency training: experiences from The Netherlands. Eur Radiol. 28(5):2208–2215. doi: 10.1007/s00330-017-5138-8.
   PubMed Web of Science ®Google Scholar
• Rutgers DR, van Schaik JPJ, Kruitwagen CLJJ, Haaring C, van Lankeren W, van Raamt AF, ten Cate O. 2020. Introducing summative progress testing in radiology residency: little change in residents” test results after transitioning from formative progress testing. Med Sci Educ. 30(2):943–953. doi: 10.1007/s40670-020-00977-2.
   PubMedGoogle Scholar
• Schut S, Driessen E, Van Tartwijk J, Van Der Vleuten C, Heeneman S. 2018. Stakes in the eye of the beholder: an international study of learners” perceptions within programmatic assessment. Med Educ. 52(6):654–663. doi: 10.1111/medu.13532.
   PubMed Web of Science ®Google Scholar
• Scrimgeour DSG, Brennan PA, Griffiths G, Lee AJ, Smith FCT, Cleland J. 2018. Does the intercollegiate membership of the Royal College of Surgeons (MRCS) examination predict “on-the-job” performance during UK higher specialty surgical training? Ann R Coll Surg Engl. 100(8):1–7. 65. doi: 10.1308/rcsann.2018.0153.
   PubMed Web of Science ®Google Scholar
• Scrimgeour DSG, Cleland J, Lee AJ, Brennan PA. 2019. Prediction of success at UK specialty board examinations using the mandatory postgraduate UK surgical examination. BJS Open. 3(6):865–871. doi: 10.1002/bjs5.50212.
   PubMed Web of Science ®Google Scholar
• Scrimgeour DSG, Cleland J, Lee AJ, Griffiths G, McKinley AJ, Marx C, Brennan PA. 2017. Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training: performance in a postgraduate surgical examination and selection into specialty training. BJS Open. 1(3):67–74. doi: 10.1002/bjs5.7.
   PubMed Web of Science ®Google Scholar
• Sklar DP. 2016. What happens after medical school? Current controversies about licensure, maintenance of ertification, and continuing professional development. Acad Med. 91(1):1–3. doi: 10.1097/ACM.0000000000001011.
   PubMed Web of Science ®Google Scholar
• Specialist Education Accreditation Committee. 2015. Standards for Assessment and Accreditation of Specialist Medical Education Programs and Professional Development Programs by the Australian Medical Council. Kingston (Australia): Australian Medical Council Limited.
   Google Scholar
• ten Cate O, Regehr G. 2019. The power of subjectivity in the assessment of medical trainees. Acad Med. 94(3):333–337. doi: 10.1097/ACM.0000000000002495.
   PubMed Web of Science ®Google Scholar
• Thoma B, Monteiro S, Pardhan A, Waters H, Chan T. 2022. Replacing high-stakes summative examinations with graduated medical licensure in Canada. CMAJ. 194(5):E168–E170. doi: 10.1503/cmaj.211816.
   PubMed Web of Science ®Google Scholar
• Tiffin PA, Paton LW. 2021. Differential attainment in the MRCPsych according to ethnicity and place of qualification between 2013 and 2018: a UK cohort study. Postgrad Med J. 97(1154):764–776. doi: 10.1136/postgradmedj-2020-137913.
   PubMed Web of Science ®Google Scholar
• Toale C, Morris M, O”Keeffe D, Boland F, Ryan DM, Nally DM, Kavanagh DO. 2023. Assessing operative competence in core surgical training: a reliability analysis. Am J Surg. doi: 10.1016/j.amjsurg.2023.06.020.
   PubMed Web of Science ®Google Scholar
• Wanderer JP, de Oliveira Filho GR, Rothman BS, Sandberg WS, McEvoy MD. 2018. Implementation and evaluation of the Z-score system for normalizing residency evaluations. Anesthesiology. 128(1):144–158. doi: 10.1097/ALN.0000000000001919.
   PubMed Web of Science ®Google Scholar
• Weller JM, Henning M, Butler R, Thompson A. 2014. The Impact of the Australian and New Zealand College of Anaesthetists” specialist examinations on trainee learning and wellbeing: A qualitative study. Anaesth Intensive Care. 42(6):736–744. doi: 10.1177/0310057X1404200609.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Tweed MJ. 2018. Deconstructing programmatic assessment. Adv Med Educ Pract. 9:191–197. doi: 10.2147/AMEP.S144449.
   PubMed Web of Science ®Google Scholar
• Woolf K. 2020. Differential attainment in medical education and training. BMJ. 368:m339. doi: 10.1136/bmj.m339.
   PubMed Web of Science ®Google Scholar
• Xu T, Mehta A, Park A, Makary MA, Price DW. 2019. Association between board certification, maintenance of certification, and surgical complications in the United States. Am J Med Qual. 34(6):545–552. doi: 10.1177/1062860618822752.
   PubMed Web of Science ®Google Scholar
• Yamamoto S, Tanaka P, Madsen MV, Macario A. 2017. Comparing anesthesiology residency training structure and requirements in seven different countries on three continents. Cureus. 9(2):e1060. doi: 10.7759/cureus.1060.
   PubMed Web of Science ®Google Scholar
• Yeates P, Woolf K, Benbow E, Davies B, Boohan M, Eva K. 2017. A randomised trial of the influence of racial stereotype bias on examiners” scores, feedback and recollections in undergraduate clinical exams. BMC Med. 15(1):179. doi: 10.1186/s12916-017-0943-0.
   PubMed Web of Science ®Google Scholar
• Zhou Y, Sun H, Macario A, Keegan MT, Patterson AJ, Minhaj MM, Wang T, Harman AE, Warner DO. 2018. Association between performance in a maintenance of certification program and disciplinary actions against the medical licenses of anesthesiologists. Anesthesiology. 129(4):812–820. doi: 10.1097/ALN.0000000000002326.
   PubMed Web of Science ®Google Scholar
• Zhou Y, Sun H, Macario A, Keegan MT, Patterson AJ, Minhaj MM, Wang T, Harman AE, Warner DO. 2019. Association between participation and performance in MOCA minute and actions against the medical licenses of anesthesiologists. Anesth Analg. 129(5):1401–1407. doi: 10.1213/ANE.0000000000004268.
   PubMed Web of Science ®Google Scholar