On the validity of summative entrustment decisions

References

• American Educational Research Association, American Psychological Association, National Council on Measurement in Education [AERA]. 2014. Standards for educational and psychological testing. Washington (DC): American Educational Research Association.
   Google Scholar
• Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. 2009. Evaluating obstetrical residency programs using patient outcomes. JAMA. 302(12):1277–1283.
   PubMed Web of Science ®Google Scholar
• Bansal N, Simmons KD, Epstein AJ, Morris JB, Kelz RR. 2016. Using patient outcomes to evaluate general surgery residency program performance. JAMA Surg. 151(2):111–119.
   PubMed Web of Science ®Google Scholar
• Bok HGJ, de Jong LH, O’Neill T, Maxey C, Hecker KG. 2018. Validity evidence for programmatic assessment in competency-based education. Perspect Med Educ. 7(6):362–372.
   PubMed Web of Science ®Google Scholar
• Calaman S, Hepps JH, Bismilla Z, Carraccio C, Englander R, Feraco A, Landrigan CP, Lopreiato JO, Sectish TC, Starmer AJ. 2016. The creation of standard-setting videos to support faculty observations of learner performance and entrustment decisions. Acad Med. 91(2):204–209.
   PubMed Web of Science ®Google Scholar
• Chahine S, Cristancho S, Padgett J, Lingard L. 2017. How do small groups make decisions? A theoretical framework to inform the implementation and study of clinical competency committees. Perspect Med Educ. 6(3):192–198.
   PubMed Web of Science ®Google Scholar
• Chahine S, Kulasegaram KM, Wright S, Monteiro S, Grierson LE, Barber C, Sebok-Syer S, McConnell M, De Champlain YW, et al. 2018. A call to investigate the relationship between education and health outcomes using big data. Acad Med. 93(6):829–832.
   PubMed Web of Science ®Google Scholar
• Chen C, Petterson S, Phillips R, Bazemore A, Mullan F. 2014. Spending patterns in region of residency training and subsequent expenditures for care provided by practicing physicians for Medicare beneficiaries. JAMA. 312(22):2385–2393.
   PubMed Web of Science ®Google Scholar
• Colbert CY, Dannefer EF, French JC. 2015. Clinical competency committees and assessment: changing the conversation in graduate medical education. J Grad Med Educ. 7(2):162–165.
   PubMedGoogle Scholar
• Cook DA, Brydges R, Ginsburg S, Hatala R. 2015. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 49(6):560–575.
   PubMed Web of Science ®Google Scholar
• Cook DA, Hatala R. 2016. Validation of educational assessments: a primer for simulation and beyond. Adv Simul (Lond). 1:31.
   PubMedGoogle Scholar
• Cook DA, Lineberry M. 2016. Consequences validity evidence: evaluating the impact of educational assessments. Acad Med. 91(6):785–795.
   PubMed Web of Science ®Google Scholar
• Crossley J, Johnson G, Booth J, Wade W. 2011. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ. 45(6):560–569.
   PubMed Web of Science ®Google Scholar
• Dai P. 2013. The conceptual model of influencing factors and influencing mechanism on team decision-making quality mediated by information sharing. IB. 05(04):119–125.
   Google Scholar
• Dauphinee WD, Boulet JR, Norcini JJ. 2019. Considerations that will determine if competency-based assessment is a sustainable innovation. Adv Health Sci Educ Theory Pract. 24(2):413–421.
   PubMed Web of Science ®Google Scholar
• De Champlain AF, Gotzmann A, Qin S. 2016. Assessing the reliability of performance assessment scores: some considerations in selecting an appropriate framework. J Grad Med Educ. 8(4):504–506.
   PubMedGoogle Scholar
• Dickey CC, Thomas C, Feroze U, Nakshabandi F, Cannon B. 2017. Cognitive demands and bias: challenges facing clinical competency committees. J Grad Med Educ. 9(2):162–164.
   PubMedGoogle Scholar
• Downing SM. 2003. Validity: on the meaningful interpretation of assessment data. Med Educ. 37(9):830–837.
   PubMed Web of Science ®Google Scholar
• Duijn CC, van Dijk EJ, Mandoki M, Bok HG, ten Cate O. 2019. Assessment tools for feedback and entrustment decisions in the clinical workplace: a systematic review. J Vet Med Educ. 46(3):340–352.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach. 32(8):638–645.
   PubMed Web of Science ®Google Scholar
• Gruppen LD, Ten CO, Lingard LA, Teunissen PW, Kogan JR. 2018. Enhanced requirements for assessment in a competency-based, time-variable medical education system. Acad Med. 93(3S):S17–S21.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Cate OT, Boscardin CK, Iobst W, Holmboe ES, Chesluk B, Baron RB, O’Sullivan PS. 2016. Ensuring resident competence: a narrative review of the literature on group decision making to inform the work of clinical competency committees. J Grad Med Educ. 8(2):156–164.
   PubMedGoogle Scholar
• Hauer KE, ten Cate O, Boscardin C, Irby DM, Iobst W, O’Sullivan PS. 2014. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Pract. 19(3):435–456.
   PubMed Web of Science ®Google Scholar
• Hodge S. 2007. The origins of competency-based training. Aust J Adult Learn. 47(2):179–209.
   Google Scholar
• Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. 2017. A call to action: the controversy of and rationale for competency-based medical education. Med Teach. 39(6):574–581.
   PubMed Web of Science ®Google Scholar
• Janis IL. 1971. Groupthink. Psychol Today. 5:84–90.
   Google Scholar
• Jonker G, Ochtman A, Marty A, Kalkman C, ten Cate O, Hoff R. 2020. Would you trust your loved ones to this resident? Certification decisions in postgraduate anesthesiology training. Brit J Anaesth. 125(5):E408–E410.
   PubMed Web of Science ®Google Scholar
• Kane MT. 2001. Current concerns in validity theory. J Educ Meas. 38(4):319–342.
   Web of Science ®Google Scholar
• Kane MT. 1992. An argument-based approach to validity. Psychol Bull. 112(3):527–535.
   Web of Science ®Google Scholar
• Kelleher M, Kinnear B, Sall D, Schumacher D, Schauer DP, Warm EJ. 2020. A reliability analysis of entrutment-derived workplace-based assessments. Acad Med. 95(4):616–622.
   PubMed Web of Science ®Google Scholar
• Kinnear B, Kelleher M, Sall D, Schumacher DJ, Warm EJ. 2020. When I say… the wisdom of crowds. Med Educ. 54(6):502–503.
   PubMed Web of Science ®Google Scholar
• Kinnear B, Warm EJ, Hauer KE. 2018. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 40(11):1–6.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti LN, Bernabeo E, Iosbt W, Holmboe E. 2015. How faculty members experience workplace-based assessment rater training: a qualitative study. Med Educ. 49(7):692–708.
   PubMed Web of Science ®Google Scholar
• Lineberry M. 2019. Validity and quality. In: Yudkowsky R, Park YS, Downing SM, editors. Assessment in health professions education. New York (NY): Routledge; p. 17–32.
   Google Scholar
• Lorenz J, Rauhut H, Schweitzer F, Helbing D. 2011. How social influence can undermine the widom of crowd effect. Proc Natl Acad Sci USA. 108(22):9020–9025.
   PubMed Web of Science ®Google Scholar
• Lurie SJ, Mooney CJ, Lyness JM. 2011. Pitfalls in assessment of competency-based educational objectives. Acad Med. 86(4):412–414.
   PubMed Web of Science ®Google Scholar
• Marceau M, Gallagher F, Young M, St-Onge C. 2018. Validity as a social imperative for assessment in health professions education: a concept analysis. Med Educ. 52(6):641–653.
   PubMed Web of Science ®Google Scholar
• Messick S. 1989. Validity. In: R. Linn, editor. Educational measurement. New York (NY): Macmillan Publishing; p. 13–104.
   Google Scholar
• Norman J, Norcini J, Bordage G. 2014. Competency-based education: milestones or millstones. J Grad Med Educ. 6(1):1–6.
   PubMedGoogle Scholar
• O’Dowd E, Lydon S, O’Connor P, Madden C, Byrne D. 2019. A systematic review of 7 years of research on entrustable professional activities in graduate medical education, 2011–2018. Med Educ. 53(3):234–249.
   PubMed Web of Science ®Google Scholar
• Oudkerk PA, Govaerts MJB, Jaarsma D, Driessen EW. 2018. From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Adv in Health Sci Educ. 23(2):275–287.
   PubMed Web of Science ®Google Scholar
• Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. 2016. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med. 91(2):186–190.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Dornoff E, Carraccio C, Busari J, van der Vleuten C, Kinnear B, Kelleher M, Sall DR, Warm E, Martini A. 2020c. The power of contribution and attribution in assessing educational outcomes for individuals, teams, and programs. Acad Med. 95(7):1014–1019.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Holmboe E, Carraccio C, Martini A, van der Vleuten C, Busari J, Sobolewski B, Byczkowski TL. 2020a. Resident-sensitive quality measures in the pediatric emergency department: exploring relationships with supervisor entrustment and patient acuity and complexity. Acad Med. 95(8):1256–1264.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, West DC, Schwartz A, Li S-T, Millstein L, Griego EC, Turner T, Herman BE, Englander R, Hemond J, et al. 2020b. Longitudinal assessment of resident performance using entrustable professional activities. JAMA Netw Open. 3(1):e1919316–e1919316.
   Web of Science ®Google Scholar
• Schuwirth L, Ash J. 2013. Assessing tomorrow’s learners: in competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Med Teach. 35(7):555–559.
   PubMed Web of Science ®Google Scholar
• Shorey S, Lau TC, Lau ST, Ang E. 2019. Entrustable professional activities in health care education: a scoping review. Med Educ. 53(8):766–777.
   PubMed Web of Science ®Google Scholar
• Smirnova A, Sebok-Syer SS, Chahine S, Kalet AL, Tamblyn R, Lombarts KM, van der Vleuten CPM, Schumacher DJ. 2019. Defining and adopting clinical performance measures in graduate medical education: where are we now and where are we going? Acad Med. 94(5):671–677.
   PubMed Web of Science ®Google Scholar
• Smit MP, de Hoog M, Brackel HJ, Ten Cate O, Gemke RJ. 2019. A national process to enhance the validity of entrustment decisions for Dutch pediatric residents. J Grad Med Educ. 11(4s):158–164.
   PubMedGoogle Scholar
• St-Onge C, Young M, Eva KW, Hodges B. 2017. Validity: one word with a plurality of meanings. Adv Health Sci Educ Theory Pract. 22(4):853–867.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, Holmboe E, Iobst W, Lovell E, Snell LS, et al. 2016. Entrustment decision making in clinical training. Acad Med. 91(2):191–198.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Regher G. 2019. The power of subjectivity in the assessment of medical trainees. Acad Med. 94(3):333–337.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Schwartz A, Chen HC. 2020. Assessing trainees and making entrustment decisions: on the nature and use of entrustment-supervision scales. Acad Med. 95(11):1662–1669.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2017. Entrustment decisions: bringing the patient into the assessment equation. Acad Med. 92(6):736–738.
   PubMed Web of Science ®Google Scholar
• ten Cate O. 2006. Trust, competence, and the supervisor’s role in postgraduate training. BMJ. 333(7571):748–751.
   PubMed Web of Science ®Google Scholar
• Triola MM, Hawkins RE, Skochelak SE. 2018. The time is now: using graduates’ practice data to drive medical education reform. Acad Med. 93(6):826–828.
   PubMed Web of Science ®Google Scholar
• Uijtdehaage S, Schuwirth LW. 2018. Assuring the quality of programmatic assessment: moving beyond psychometrics. Perspect Med Educ. 7(6):350–351.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• van Eemeren FH, Grootendorst R, Johnson RH, Plantin C, Willard CA. 2013. Fundamentals of argumentation theory: a handbook of historical backgrounds and contemporary developments. New York (NY): Routledge.
   Google Scholar
• van Enk A, ten Cate O. 2020. “Languaging” tacit judgment in formal postgraduate assessment: the documentation of ad hoc and summative entrustment decisions. Perspect Med Educ. 9(6):373–378.
   PubMed Web of Science ®Google Scholar
• Warm EJ, Mathis BR. 2019. ambulatory education: time to move from process to outcome. J Grad Med Educ. 11(2):143–145.
   PubMedGoogle Scholar
• Weinstein DF. 2017. Optimizing GME by measuring its outcomes. N Engl J Med. 377(21):2007–2009.
   PubMed Web of Science ®Google Scholar
• Wijnen-Meijer M, Van der Schaaf M, Booij E, Harendza S, Boscardin S, Van Wijngaarden J, ten Cate TJ. 2013. An argument-based approach to the validation of UHTRUST: Can we measure how recent graduates can be trusted with unfamiliar tasks? Adv in Health Sci Educ Theory Pract. 18(5):1009–1027.
   PubMed Web of Science ®Google Scholar