Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference

References

• AERA, APA, NCME. 2014. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Archer E, Prinsloo P. 2020. Speaking the unspoken in learning analytics: troubling the defaults. Assess Eval High Educ. 45(6):888–813.
   Web of Science ®Google Scholar
• Barrett A, Galvin R, Scherpbier A, Teunissen P, O'Shaughnessy A, Horgan M. 2017. Is the learning value of workplace-based assessment being realised? A qualitative study of trainer and trainee perceptions and experiences. Postgrad Med J. 93(1097):138–142.
   PubMed Web of Science ®Google Scholar
• Barrett A, Galvin R, Steinert Y, Scherpbier A, O'Shaughnessy A, Horgan M, Horsley T. 2016. A BEME (Best Evidence in Medical Education) review of the use of workplace-based assessment in identifying and remediating underperformance among postgraduate medical trainees: BEME Guide No. 43. Med Teach. 38(12):1188–1198.
   PubMed Web of Science ®Google Scholar
• Biggs J. 1996. Enhancing teaching through constructive alignment. High Educ. 32(3):347–364.
   Web of Science ®Google Scholar
• Bindal T, Wall D, Goodyear HM. 2011. Trainee doctors' views on workplace-based assessments: are they just a tick box exercise? Med Teach. 33(11):919–927.
   PubMed Web of Science ®Google Scholar
• Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. 2017. Feedback for learners in medical education: what is known? A scoping review. Acad Med. 92(9):1346–1354.
   PubMed Web of Science ®Google Scholar
• Bing-You R, Ramani S, Ramesh S, Hayes V, Varaklis K, Ward DC, Blanco M. 2019. The interplay between residency program culture and feedback culture: a cross-sectional study exploring perceptions of residents at three institutions. Med Educ Online. 24(1):1611296.
   PubMed Web of Science ®Google Scholar
• Boursicot K, Etheridge L, Setna Z, Sturrock A, Ker J, Smee S, Sambandam E. 2011. Performance in assessment: consensus statement and recommendations from the Ottawa conference. Med Teach. 33(5):370–383.
   PubMedGoogle Scholar
• Boursicot K, Roberts T, Burdick W. 2018. Structured assessments of clinical competence. Understanding medical education. London: Wiley; p. 335–345.
   Google Scholar
• Chahine S, Holmes B, Kowalewski Z. 2016. In the minds of OSCE examiners: uncovering hidden assumptions. Adv Health Sci Educ Theory Pract. 21(3):609–625.
   PubMed Web of Science ®Google Scholar
• Cianciolo AT, Kegg JA. 2013. Behavioral specification of the entrustment process. J Grad Med Educ. 5(1):10–12.
   PubMedGoogle Scholar
• Coderre S, Woloschuk W, McLaughlin K. 2009. Twelve tips for blueprinting. Med Teach. 31(4):322–324.
   PubMed Web of Science ®Google Scholar
• Cohen R, Rothman AI, Ross J, Poldre P. 1993. Impact of repeated use of objective structured clinical examination stations. Acad Med. 68(10 Suppl):S73–S75.
   PubMed Web of Science ®Google Scholar
• Cook DA, Brydges R, Ginsburg S, Hatala R. 2015. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med Educ. 49(6):560–575.
   PubMed Web of Science ®Google Scholar
• Cook DA, Kuper A, Hatala R, Ginsburg S. 2016. When assessment data are words: validity evidence for qualitative educational assessments. Acad Med. 91(10):1359–1369.
   PubMed Web of Science ®Google Scholar
• Crossley J, Jolly B. 2012. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ. 46(1):28–37.
   PubMed Web of Science ®Google Scholar
• Daniels VJ, Pugh D. 2018. Twelve tips for developing an OSCE that measures what you want. Med Teach. 40(12):1208–1213.
   PubMed Web of Science ®Google Scholar
• Dijkstra J, Van der Vleuten CP, Schuwirth LW. 2010. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 15(3):379–393.
   PubMed Web of Science ®Google Scholar
• Downing SM. 2003. Validity: on meaningful interpretation of assessment data. Med Educ. 37(9):830–837.
   PubMed Web of Science ®Google Scholar
• Downing SM, Haladyna TM. 2004. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ. 38(3):327–333.
   PubMed Web of Science ®Google Scholar
• Duijn C, Welink LS, Bok HGJ, ten Cate OTJ. 2018. When to trust our learners? Clinical teachers' perceptions of decision variables in the entrustment process. Perspect Med Educ. 7(3):192–199.
   PubMed Web of Science ®Google Scholar
• Eva KW, Bordage G, Campbell C, Galbraith R, Ginsburg S, Holmboe E, Regehr G. 2016. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 21(4):897–913.
   PubMed Web of Science ®Google Scholar
• Foster C, Francis P. 2019. A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes. Assess Eval High Educ. 22:1–20.
   Google Scholar
• Fuller R, Homer M, Pell G. 2013. Longitudinal interrelationships of OSCE station level analyses, quality improvement and overall reliability. Med Teach. 35(6):515–517.
   PubMed Web of Science ®Google Scholar
• Fuller R, Homer M, Pell G, Hallam J. 2017. Managing extremes of assessor judgment within the OSCE. Med Teach. 39(1):58–66.
   PubMed Web of Science ®Google Scholar
• Gauthier G, St-Onge C, Tavares W. 2016. Rater cognition: review and integration of research findings. Med Educ. 50(5):511–522.
   PubMed Web of Science ®Google Scholar
• Ghouri A, Boachie C, McDowall S, Parle J, Ditchfield CA, McConnachie A, Walters MR, Ghouri N. 2018. Gaining an advantage by sitting an OSCE after your peers: a retrospective study. Med Teach. 40(11):1136–1142.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Kogan J, Yeates P, Govaerts MJ, Holmboe E. 2014. Seeing the 'black box' differently: assessor cognition from three research perspectives. Med Educ. 48(11):1055–1068.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Ramlo SE, van der Vleuten CPM, Eva KW, Regehr G. 2017. Inter-rater variability as mutual disagreement: identifying raters' divergent points of view. Adv Health Sci Educ Theory Pract. 22(4):819–838.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KW. 2011. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 86(10 Suppl):S1–S7.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, van der Vleuten CP, Eva KW, Lingard L. 2017. Cracking the code: residents' interpretations of written assessment comments. Med Educ. 51(4):401–410.
   PubMed Web of Science ®Google Scholar
• Gotzmann A, De Champlain A, Homayra F, Fotheringham A, de Vries I, Forgie M, Pugh D. 2017. Cheating in OSCEs: the impact of simulated security breaches on OSCE performance. Teach Learn Med. 29(1):52–58.
   PubMed Web of Science ®Google Scholar
• Govaerts MJ, Schuwirth LW, van der Vleuten CPM, Muijtjens AM. 2011. Workplace-based assessment: effects of rater expertise. Adv Health Sci Educ Theory Pract. 16(2):151–165.
   PubMed Web of Science ®Google Scholar
• Govaerts MJ, van der Vleuten CP. 2013. Validity in work-based assessment: expanding our horizons. Med Educ. 47(12):1164–1174.
   PubMed Web of Science ®Google Scholar
• Govaerts MJ, Van der Wiel MW, Schuwirth LW, van der Vleuten CPM, Muijtjens AM. 2013. Workplace-based assessment: raters' performance theories and constructs. Adv Health Sci Educ Theory Pract. 18(3):375–396.
   PubMed Web of Science ®Google Scholar
• Harden RM. 2016. Revisiting 'assessment of clinical competence using an objective structured clinical examination (OSCE)'). Med Educ. 50(4):376–379.
   PubMed Web of Science ®Google Scholar
• Harden RM, Gleeson FA. 1979. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 13(1):41–54.
   PubMed Web of Science ®Google Scholar
• Hatala R, Ginsburg S, Hauer KE, Gingerich A. 2019. Entrustment ratings in internal medicine training: capturing meaningful supervision decisions or just another rating? J Gen Intern Med. 34(5):740–743.
   PubMed Web of Science ®Google Scholar
• Hattie J, Clarke S. 2018. Visible learning: feedback. London: Routledge.
   Google Scholar
• Hattie J, Timperley H. 2007. The power of feedback. Revi Educ Res. 77(1):81–112.
   Web of Science ®Google Scholar
• Hejri SM, Jalili M, Muijtjens AMM, Van Der Vleuten CPM. 2013. Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination. J Res Med Sci. 18(10):887–891.
   PubMed Web of Science ®Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568.
   PubMed Web of Science ®Google Scholar
• Hodges B, McIlroy JH. 2003. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 37(11):1012–1016.
   PubMed Web of Science ®Google Scholar
• Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad Med. 74(10):1129–1134.
   PubMed Web of Science ®Google Scholar
• Homer M, Fuller R, Hallam J, Pell G. 2020. Setting defensible standards in small cohort OSCEs: understanding better when borderline regression can ‘work’. Med Teach. 42(3):306–315.
   PubMed Web of Science ®Google Scholar
• Homer M, Pell G, Fuller R, Patterson J. 2016. Quantifying error in OSCE standard setting for varying cohort sizes: a resampling approach to measuring assessment quality. Med Teach. 38(2):181–188.
   PubMed Web of Science ®Google Scholar
• Humphrey-Murto S, Mihok M, Pugh D, Touchie C, Halman S, Wood TJ. 2016. Feedback in the OSCE: what do residents remember? Teach Learn Med. 28(1):52–60.
   PubMed Web of Science ®Google Scholar
• Ilgen JS, Ma IWY, Hatala R, Cook DA. 2015. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 49(2):161–173.
   PubMed Web of Science ®Google Scholar
• Issenberg SB. 2011. Ottawa 2010 Conference – consensus statements and recommendations. Med Teach. 33(3):181–182.
   PubMed Web of Science ®Google Scholar
• Joynes V, Fuller R. 2016. Legitimisation, personalisation and maturation: using the experiences of a compulsory mobile curriculum to reconceptualise mobile learning. Med Teach. 38(6):621–627.
   PubMed Web of Science ®Google Scholar
• JRCPTB. 2020. Workplace-based assessment: assessment methods. Joint Royal Colleges of Physicians Training Board; [accessed July 12]. https://www.jrcptb.org.uk/assessment/workplace-based-assessment.
   Google Scholar
• Kane MT. 2013a. Validating the interpretations and uses of test scores. J Educ Meas. 50(1):1–73.
   Web of Science ®Google Scholar
• Kane MT. 2013b. Validation as a pragmatic, scientific activity. J Educ Meas. 50(1):115–122.
   Web of Science ®Google Scholar
• Kawasumi Y, Ernst P, Abrahamowicz M, Tamblyn R. 2011. Association between physician competence at licensure and the quality of asthma management among patients with out-of-control asthma. Arch Intern Med. 171(14):1292–1294.
   PubMedGoogle Scholar
• Kelleher M, Kinnear B, Wong SEP, O'Toole J, Warm E. 2020. Linking workplace-based assessment to ACGME milestones: a comparison of mapping strategies in two specialties. Teach Learn Med. 32(2):194–203.
   PubMed Web of Science ®Google Scholar
• Khan KZ, Ramachandran S, Gaunt K, Pushkar P. 2013. The objective structured clinical examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 35(9):e1437–e1446.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. 2011. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 45(10):1048–1060.
   PubMed Web of Science ®Google Scholar
• LaDonna KA, Ginsburg S, Watling C. 2018. "Rising to the level of your incompetence": what physicians' self-assessment of their performance reveals about the imposter syndrome in medicine. Acad Med. 93(5):763–768.
   PubMed Web of Science ®Google Scholar
• LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. 2017. Staging a performance: learners' perceptions about direct observation during residency. Med Educ. 51(5):498–510.
   PubMed Web of Science ®Google Scholar
• Lockyer J, Carraccio C, Chan M, Hart D, Smee S, Touchie C, Holmboe E, Frank JM. 2017. Core principles of assessment in competency-based medical education. Med Teach. 39(6):609–616.
   PubMed Web of Science ®Google Scholar
• Lörwald A, Lahner F, Mooser B, Perrig M, Widmer M, Greif R, Huwendiek S. 2019. Influences on the implementation of Mini-CEX and DOPS for postgraduate medical trainees’ learning: a grounded theory study. Med Teach. 41(4):448–449.
   PubMed Web of Science ®Google Scholar
• Massie J, Ali JM. 2016. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract. 21(2):455–473.
   PubMed Web of Science ®Google Scholar
• McKinley DW, Norcini JJ. 2014. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach. 36(2):97–110.
   PubMed Web of Science ®Google Scholar
• Miller GE. 1990. The assessment of clinical skills/competence/performance [invited review]. Acad Med. 65(9):S63–S67.
   PubMed Web of Science ®Google Scholar
• Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. 2013. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 18(5):1087–1102.
   PubMed Web of Science ®Google Scholar
• Niehaus AH, DaRosa DA, Markwell SJ, Folse R. 1996. Is test security a concern when OSCE stations are repeated across clerkship rotations? Acad Med. 71(3):287–289.
   PubMed Web of Science ®Google Scholar
• Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay M, Roberts T, Swanson DB. 2018. 2018 Consensus framework for good assessment. Med Teach. 40(11):1102–1109.
   PubMed Web of Science ®Google Scholar
• Ossenberg C, Henderson A, Mitchell M. 2019. What attributes guide best practice for effective feedback? A scoping review. Adv Health Sci Educ Theory Pract. 24(2):383–401.
   PubMed Web of Science ®Google Scholar
• Oudkerk Pool A, Govaerts MJ, Jaarsma D, Driessen EW. 2018. From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Adv Health Sci Educ Theory Pract. 23(2):275–287.
   PubMed Web of Science ®Google Scholar
• Pearce J. 2020. In defence of constructivist, utility-driven psychometrics for the 'post-psychometric era'. Med Educ. 54(2):99–102.
   PubMed Web of Science ®Google Scholar
• Pelgrim EAM, Kramer AWM, Mokkink HGA, van den Elsen L, Grol R, van der Vleuten CPM. 2011. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract. 16(1):131–142.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, Roberts TE. 2010. How to measure the quality of the OSCE: a review of metrics – AMEE Guide No. 49. Med Teach. 32(10):802–811.
   PubMed Web of Science ®Google Scholar
• Pell G, Homer MS, Roberts TE. 2008. Assessor training: its effects on criterion‐based assessment in a medical context. Int J Res Method Educ. 31(2):143–154.
   Google Scholar
• Pugh D, Bhanji F, Cole G, Dupre J, Hatala R, Humphrey-Murto S, Touchie C, Wood TJ. 2016. Do OSCE progress test scores predict performance in a national high-stakes examination? Med Educ. 50(3):351–358.
   PubMed Web of Science ®Google Scholar
• Pugh D, Desjardins I, Eva K. 2018. How do formative objective structured clinical examinations drive learning? Analysis of residents' perceptions. Med Teach. 40(1):45–52.
   PubMed Web of Science ®Google Scholar
• Pugh D, Halman S, Desjardins I, Humphrey-Murto S, Wood TJ. 2016. Done or almost done? Improving OSCE checklists to better capture performance in progress tests. Teach Learn Med. 28(4):406–414.
   PubMed Web of Science ®Google Scholar
• Ramani S, Könings K, Ginsburg S, van der Vleuten CPM. 2019. Twelve tips to promote a feedback culture with a growth mind-set: swinging the feedback pendulum from recipes to relationships. Med Teach. 41(6):625–631.
   PubMed Web of Science ®Google Scholar
• Ramani S, Krackov SK. 2012. Twelve tips for giving feedback effectively in the clinical environment. Med Teach. 34(10):787–791.
   PubMed Web of Science ®Google Scholar
• Raymond MR, Grande JP. 2019. A practical guide to test blueprinting. Med Teach. 41(8):854–861.
   PubMed Web of Science ®Google Scholar
• RCGP. 2020. MRCGP workplace based assessment (WPBA). Royal College of General Practitioners; [accessed 2020 July 12]. https://www.jrcptb.org.uk/assessment/workplace-based-assessmentrcgp.org.uk/training-exams/training/mrcgp-workplace-based-assessment-wpba.aspx.
   Google Scholar
• RCOG. 2020. Workplace-based assessments (WPBAs). Royal College of Obstetricians and Gynaecologists; [accessed 2020 July 12]. https://www.rcog.org.uk/en/careers-training/about-specialty-training-in-og/assessment-and-progression-through-training/workplace-based-assessments/.
   Google Scholar
• Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. 2016. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med. 91(2):186–190.
   PubMed Web of Science ®Google Scholar
• Sabey A, Harris M. 2011. Training in hospitals: what do GP specialist trainees think of workplace-based assessments? Educ Prim Care. 22(2):90–99.
   PubMedGoogle Scholar
• Sales D, Sturrock A, Boursicot K, Dacre J. 2010. Blueprinting for clinical performance deficiencies-lessons and principles from the General Medical Council's fitness to practise procedures. Med Teach. 32(3):e111–e114.
   PubMed Web of Science ®Google Scholar
• Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF. 2009. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. 14(3):399–410.
   PubMed Web of Science ®Google Scholar
• Scarff CE, Bearman M, Chiavaroli N, Trumble S. 2019. Keeping mum in clinical supervision: private thoughts and public judgements. Med Educ. 53(2):133–142.
   PubMed Web of Science ®Google Scholar
• Schüttpelz-Brauns K, Nühse K, Strohmer R, Kaden JJ. 2019. Training OSCE examiners: minimal effort with far-reaching results. Med Educ. 53(11):1153–1154.
   PubMed Web of Science ®Google Scholar
• Schuwirth LWT, Van der Vleuten CPM. 2011. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 33(6):478–485.
   PubMed Web of Science ®Google Scholar
• Schuwirth LWT, van der Vleuten CPM. 2012. Programmatic assessment and Kane's validity perspective. Med Educ. 46(1):38–48.
   PubMed Web of Science ®Google Scholar
• Simon SR, Volkan K, Hamann C, Duffey C, Fletcher SW. 2002. The relationship between second-year medical students' OSCE scores and USMLE Step 1 scores. Med Teach. 24(5):535–539.
   PubMed Web of Science ®Google Scholar
• Soleas E, Dagnone D, Stockley D, Garton K, van Wylick R. 2020. Developing academic advisors and competence committees members: a community approach to developing CBME faculty leaders. Can Med Educ J. 11(1):e46–e56.
   PubMedGoogle Scholar
• Swanson DB, Clauser BE, Case SM. 1999. Clinical skills assessment with standardized patients in high-stakes tests: a framework for thinking about score precision, equating, and security. Adv Health Sci Educ Theory Pract. 4(1):67–106.
   PubMed Web of Science ®Google Scholar
• Swanson DB, van der Vleuten CP. 2013. Assessment of clinical skills with standardized patients: state of the art revisited. Teach Learn Med. 25(Suppl 1):S17–S25.
   PubMed Web of Science ®Google Scholar
• Tamblyn R, Abrahamowicz M, Dauphinee WD, Hanley JA, Norcini J, Girard N, Grand'Maison P, Brailovsky C. 2002. Association between licensure examination scores and practice in primary care. JAMA. 288(23):3019–3026.
   PubMed Web of Science ®Google Scholar
• Tannenbaum RJ, Kane MT. 2019. Stakes in testing: not a simple dichotomy but a profile of consequences that guides needed evidence of measurement quality. ETS Res Rep Ser. 2019(1):1–16.
   Google Scholar
• Tekian A, Watling C, Roberts TE, Steinert Y, Norcini J. 2017. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 39(12):1245–1249.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2005. Entrustability of professional activities and competency-based training. Med Educ. 39(12):1176–1177.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2020. When I say … entrustability. Med Educ. 54(2):103–104.
   PubMed Web of Science ®Google Scholar
• Thoma B, Bandi V, Carey R, Mondal D, Woods R, Martin L, Chan TF. 2020. Developing a dashboard to meet Competence Committee needs: a design-based research project. Can Med Educ J. 11(1):e16–e34.
   PubMedGoogle Scholar
• Torre DM, Schuwirth LWT, Van der Vleuten CPM. 2020. Theoretical considerations on programmatic assessment. Med Teach. 42(2):213–220.
   PubMed Web of Science ®Google Scholar
• van der Leeuw RM, Teunissen PW, van der Vleuten CPM. 2018. Broadening the scope of feedback to promote its relevance to workplace learning. Acad Med. 93(4):556–559.
   PubMed Web of Science ®Google Scholar
• van der Schaaf M, Donkers J, Slof B, Moonen-van Loon JM, van Tartwijk J, Driessen E, Badii A, Serban O, ten Cate O. 2017. Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Education Tech Research Dev. 65(2):359–380.
   Web of Science ®Google Scholar
• van der Vleuten CPM. 2016. A programmatic approach to assessment. Med Sci Educ. 26(S1):9–10.
   Google Scholar
• Voyer S, Cuncic C, Butler DL, MacNeil K, Watling C, Hatala R. 2016. Investigating conditions for meaningful feedback in the context of an evidence-based feedback programme. Med Educ. 50(9):943–954.
   PubMed Web of Science ®Google Scholar
• Watling C, Driessen E, van der Vleuten CP, Lingard L. 2014. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 48(7):713–723.
   PubMed Web of Science ®Google Scholar
• Watling C, Ginsburg S. 2019. Assessment, feedback and the alchemy of learning. Med Educ. 53(1):76–85.
   PubMed Web of Science ®Google Scholar
• Weller JM, Jolly B, Misur MP, Merry AF, Jones A, Crossley JG, Pedersen K, Smith K. 2009. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth. 102(5):633–641.
   PubMed Web of Science ®Google Scholar
• Whelan GP, Boulet JR, McKinley DW, Norcini JJ, van Zanten M, Hambleton RK, Burdick WP, Peitzman SJ. 2005. Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA®). Med Teach. 27(3):200–206.
   PubMed Web of Science ®Google Scholar
• Wilkinson T, Frampton C. 2004. Comprehensive undergraduate medical assessments improve prediction of clinical performance. Med Educ. 38(10):1111–1116.
   PubMed Web of Science ®Google Scholar
• Wood TJ, Pugh D. 2020. Are rating scales really better than checklists for measuring increasing levels of expertise? Med Teach. 42(1):46–51.
   PubMed Web of Science ®Google Scholar
• Yang M, Carless D. 2013. The feedback triangle and the enhancement of dialogic feedback processes. Teach High Educ. 18(3):285–297.
   Web of Science ®Google Scholar
• Yeates P, Cardell J, Byrne G, Eva KW. 2015. Relatively speaking: contrast effects influence assessors’ scores and narrative feedback. Med Educ. 49(9):909–919.
   PubMed Web of Science ®Google Scholar
• Yeates P, Cope N, Hawarden A, Bradshaw H, McCray G, Homer M. 2019. Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs. Med Educ. 53(3):250–263.
   PubMed Web of Science ®Google Scholar
• Yepes-Rios M, Dudek N, Duboyce R, Curtis J, Allard RJ, Varpio L. 2016. The failure to fail underperforming trainees in health professions education: a BEME systematic review: BEME Guide No. 42. Med Teach. 38(11):1092–1099.
   PubMed Web of Science ®Google Scholar
• Yousuf N, Violato C, Zuberi RW. 2015. Standard setting methods for pass/fail decisions on high-stakes Objective Structured Clinical Examinations: a validity study. Teach Learn Med. 27(3):280–291.
   PubMed Web of Science ®Google Scholar
• Yudkowsky R, Park YS, Riddle J, Palladino C, Bordage G. 2014. Clinically discriminating checklists versus thoroughness checklists: Improving the valididty of performance test scores. Acad Med. 89(7):1057–1062.
   PubMed Web of Science ®Google Scholar