Capturing outcomes of competency-based medical education: The call and the challenge

References

• Andrews JS, Bale JF, Jr, Soep JB, Long M, Carraccio C, Englander R, Powell D, EPAC Study Group. 2018. Education in pediatrics across the continuum (EPAC): first steps toward realizing the dream of competency-based education. Acad Med. 93(3):414–420.
   PubMed Web of Science ®Google Scholar
• Baxter SK, Blank L, Woods HB, Payne N, Rimmer M, Goyder E. 2014. Using logic model methods in systematic review synthesis: describing complex pathways in referral management interventions. BMC Med Res Methodol. 14(1):62.
   PubMed Web of Science ®Google Scholar
• Berliner DC. 2002. Comment: educational research: the hardest science of all. Educ Res. 31(8):18–20.
   Google Scholar
• Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. 2002. Shifting paradigms: from Flexner to competencies. Acad Med. 77(5):361–367.
   PubMed Web of Science ®Google Scholar
• Century J, Rudnick M, Freeman C. 2010. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 31(2):199–218.
   Web of Science ®Google Scholar
• Challis M. 2000. AMEE medical education guide no. 19: personal learning plans. Med Teach. 22(3):225–236.
   Web of Science ®Google Scholar
• Chan TM, Paterson QS, Hall AK, Zaver F, Woods RA, Hamstra SJ, Stefan A, Ting DK, Thoma B. 2020. Outcomes in the age of competency-based medical education: recommendations for emergency medicine training in Canada from the 2019 Symposium of Academic Emergency Physicians. Can J Emerg Med. 22(2):204–214.
   Web of Science ®Google Scholar
• Cuban L. 2013. Inside the black box of classroom practice: change without reform in American education. Cambridge (MA): Harvard Education Press.
   Google Scholar
• Earl S, Carden F, Smutylo T. 2001. Outcome mapping: building learning and reflection into development programs. Ottawa (ON): International Development Research Centre.
   Google Scholar
• Fédération des Médicins Résidents du Québec. 2020. Year 3 of implementation of competence by design: negative impact still outweighs theoretical benefits. Montreal (QC): Fédération des Médicins Résidents du Québec.
   Google Scholar
• Frank JR, Danoff D. 2007. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 29(7):642–647.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach. 32(8):638–645.
   PubMed Web of Science ®Google Scholar
• Frye AW, Hemmer PA. 2012. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 34(5):e288–e299.
   PubMed Web of Science ®Google Scholar
• Funnell SC, Rogers PJ. 2011. Purposeful program theory: effective use of theories of change and logic models. San Francisco (CA): John Wiley & Sons.
   Google Scholar
• General Medical Council. 2013. Good medical practice. London: General Medical Council.
   Google Scholar
• Griffiths J, Dalgarno N, Schultz K, Han H, Van Melle E. 2019. Competency-based medical education implementation: are we transforming the culture of assessment? Med Teach. 41(7):811–818.
   PubMed Web of Science ®Google Scholar
• Haan CK, Edwards FH, Poole B, Godley M, Genuardi FJ, Zenni EA. 2008. A model to begin to use clinical outcomes in medical education. Acad Med. 83(6):574–580.
   PubMed Web of Science ®Google Scholar
• Haji FA, Dubrowski A, Drake J, De Ribaupierre S. 2013. Needs assessment for simulation training in neuroendoscopy: a Canadian national survey. J Neurosurg. 118(2):250–257.
   PubMed Web of Science ®Google Scholar
• Hall AK, Rich J, Dagnone JD, Weersink K, Caudle J, Sherbino J, Frank JR, Bandiera G, Van Melle E. 2020. It’s a marathon, not a sprint: rapid evaluation of competency-based medical education program implementation. Acad Med. 95(5):786–793.
   PubMed Web of Science ®Google Scholar
• Hall AK, Schumacker DJ, Thoma B, Caretta-Weyer H, Kinnear B, Gruppen L, Cooke LJ, Frank JR, Van Melle E. 2021. Outcomes of competency-based medical education: a taxonomy for shared language. Med Teach. (in press).
   Web of Science ®Google Scholar
• Hall AK, Woods R, Frank JR. 2019. Changing the culture of residency training through faculty development. Can J Emerg Med. 21(4):446–448.
   Web of Science ®Google Scholar
• Hamstra SJ, Yamazaki K, Barton MA, Santen SA, Beeson MS, Holmboe ES. 2019. A national study of longitudinal consistency in ACGME milestone ratings by clinical competency committees: exploring an aspect of validity in the assessment of Residents’ competence. Acad Med. 94(10):1522–1531.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Chesluk B, Iobst W, Holmboe E, Baron RB, Boscardin CK, Cate OT, O'Sullivan PS. 2015. Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment. Acad Med. 90(8):1084–1092.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Clauser J, Lipner RS, Holmboe ES, Caverzagie K, Hamstra SJ, Hood S, Iobst W, Warm E, McDonald FS. 2016. The internal medicine reporting milestones: cross-sectional description of initial implementation in U.S. residency programs. Ann Intern Med. 165(5):356–362.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Vandergrift J, Hess B, Lipner RS, Holmboe ES, Hood S, Iobst W, Hamstra SJ, McDonald FS. 2016. Correlations between ratings on the resident annual evaluation summary and the internal medicine milestones and association with ABIM certification examination scores among US internal medicine residents, 2013–2014. JAMA. 316(21):2253–2262.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Vandergrift J, Lipner RS, Holmboe ES, Hood S, McDonald FS. 2018. National internal medicine milestone ratings: validity evidence from longitudinal three-year follow-up. Acad Med. 93(8):1189–1204.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR, ICBME Collaborators. 2017. A call to action: the controversy of and rationale for competency-based medical education. Med Teach. 39(6):574–581.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Yamazaki K, Nasca TJ, Hamstra SJ. 2020. Using longitudinal milestones data and learning analytics to facilitate the professional development of residents: early lessons from three specialties. Acad Med. 95(1):97–103.
   PubMed Web of Science ®Google Scholar
• Holmboe ES. 2017. The journey to competency-based medical education-implementing milestones. Marshall J Med. 3(1):3.
   Google Scholar
• Kerdijk W, Snoek JW, van Hell EA, Cohen-Schotanus J. 2013. The effect of implementing undergraduate competency-based medical education on students' knowledge acquisition, clinical performance and perceived preparedness for practice: a comparative study. BMC Med Educ. 13(1):76.
   PubMed Web of Science ®Google Scholar
• Klamen DL, Williams RG, Roberts N, Cianciolo AT. 2016. Competencies, milestones, and EPAs - are those who ignore the past condemned to repeat it? Med Teach. 38(9):904–910.
   PubMed Web of Science ®Google Scholar
• Litwack K, Brower AM. 2018. The University of Wisconsin-Milwaukee flexible option for bachelor of science in nursing degree completion. Acad Med. 93:S37–S41.
   PubMed Web of Science ®Google Scholar
• Marcotte L, Egan R, Soleas E, Dalgarno N, Norris M, Smith C. 2019. Assessing the quality of feedback to general internal medicine residents in a competency-based environment. Can Med Ed J. 10(4):e32–e47.
   PubMedGoogle Scholar
• McGaghie WC. 1978. Competency-based curriculum development in medical education. An introduction. Public Health Paper. no. 68. Geneva (Switzerland): World Health Organization.
   Google Scholar
• Mejicano GC, Bumsted TN. 2018. Describing the journey and lessons learned implementing a competency-based, time-variable undergraduate medical education curriculum. Acad Med. 93:S42–S48.
   PubMed Web of Science ®Google Scholar
• Morcke AM, Dornan T, Eika B. 2013. Outcome (competency) based education: an exploration of its origins, theoretical basis, and empirical evidence. Adv Health Sci Educ Theory Pract. 18(4):851–863.
   PubMed Web of Science ®Google Scholar
• Mowbray CT, Holter MC, Teague GB, Bybee D. 2003. Fidelity criteria: development, measurement, and validation. Am J Eval. 24(3):315–340.
   Web of Science ®Google Scholar
• Norman G, Norcini J, Bordage G. 2014. Competency-based education: milestones or millstones? J Grad Med Educ. 6(1):1–6.
   PubMedGoogle Scholar
• Patton MQ. 2010. Developmental evaluation: applying complexity concepts to enhance innovation and use. New York (NY): Guilford Press.
   Google Scholar
• Patton MQ. 2016. What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. Am J Eval. 37(2):250–265.
   Web of Science ®Google Scholar
• Pawson R, Greenhalgh T, Harvey G, Walshe K. 2005. Realist review — a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 10(1):21–34.
   PubMedGoogle Scholar
• Raaum SE, Lappe K, Colbert-Getz JM, Milne CK. 2019. Milestone implementation’s impact on narrative comments and perception of feedback for internal medicine residents: a mixed methods study. J Gen Intern Med. 34(6):929–935.
   PubMed Web of Science ®Google Scholar
• Regehr G. 2010. It’s NOT rocket science: rethinking our metaphors for research in health professions education. Med Educ. 44(1):31–39.
   PubMed Web of Science ®Google Scholar
• Rich JV, Fostaty Young S, Donnelly C, Hall AK, Dagnone JD, Weersink K, Caudle J, Van Melle E, Klinger DA. 2020. Competency-based education calls for programmatic assessment: but what does this look like in practice? J Eval Clin Pract. 26(4):1087–1095.
   PubMed Web of Science ®Google Scholar
• Royal Australasian College of Surgeons 2020. Nine RACS competencies. [accessed 2020 May 30]. https://www.surgeons.org/en/Trainees/the-set-program/racs-competencies.
   Google Scholar
• Rycroft-Malone J, McCormack B, Hutchinson AM, DeCorby K, Bucknall TK, Kent B, Schultz A, Snelgrove-Clarke E, Stetler CB, Titler M, et al. 2012. Realist synthesis: illustrating the method for implementation research. Implement Sci. 7(1):33.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Dornoff E, Carraccio C, Busari J, van der Vleuten C, Kinnear B, Kelleher M, Sall DR, Warm E, Martini A, et al. 2020. The power of contribution and attribution in assessing educational outcomes for individuals, teams, and programs. Acad Med. 95(7):1014–1019.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Holmboe ES, van der Vleuten C, Busari JO, Carraccio C. 2018. Developing resident-sensitive quality measures: a model from pediatric emergency medicine. Acad Med. 93(7):1071–1078.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman MA, Ludwig S, Hicks PJ, Guralnick S, Englander R, et al. 2013. The pediatrics milestones: initial evidence for their use as learning road maps for residents. Acad Pediatr. 13(1):40–47.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, Martini A, Holmboe E, Carraccio C, van der Vleuten C, Sobolewski B, Busari J, Byczkowski TL. 2020. Initial implementation of resident-sensitive quality measures in the pediatric emergency department: a wide range of performance. Acad Med. 95(8):1248–1255.
   PubMed Web of Science ®Google Scholar
• Schumacher DJ, West DC, Schwartz A, Li ST, Millstein L, Griego EC, Turner T, Herman BE, Englander R, Hemond J, et al. 2020. Longitudinal assessment of resident performance using entrustable professional activities. JAMA Netw Open. 3(1):e1919316.
   PubMed Web of Science ®Google Scholar
• Sebok-Syer SS, Chahine S, Watling CJ, Goldszmidt M, Cristancho S, Lingard L. 2018. Considering the interdependence of clinical performance: implications for assessment and entrustment. Med Educ. 52(9):970–980.
   PubMed Web of Science ®Google Scholar
• Simpson JG, Furnace J, Crosby J, Cumming AD, Evans PA, Friedman Ben David M, Harden RM, Lloyd D, McKenzie H, McLachlan JC, et al. 2002. The Scottish doctor-learning outcomes for the medical undergraduate in Scotland: a foundation for competent and reflective practitioners. Med Teach. 24(2):136–143.
   PubMed Web of Science ®Google Scholar
• Sirianni G, Glover Takahashi S, Myers J. 2020. Taking stock of what is known about faculty development in competency-based medical education: a scoping review paper. Med Teach. 42(8):909–915.
   PubMed Web of Science ®Google Scholar
• Smirnova A, Sebok-Syer SS, Chahine S, Kalet AL, Tamblyn R, Lombarts K, van der Vleuten CPM, Schumacher DJ. 2019. Defining and adopting clinical performance measures in graduate medical education: Where are we now and where are we going? Acad Med. 94(5):671–677.
   PubMed Web of Science ®Google Scholar
• Swing SR. 2007. The ACGME outcome project: retrospective and prospective. Med Teach. 29(7):648–654.
   PubMed Web of Science ®Google Scholar
• Tamblyn R. 1999. Outcomes in medical education: what is the standard and outcome of care delivered by our graduates? Adv Health Sci Educ Theory Pract. 4(1):9–25.
   PubMed Web of Science ®Google Scholar
• Tomiak A, Braund H, Egan R, Dalgarno N, Emack J, Reid MA, Hammad N. 2020. Exploring how the new entrustable professional activity assessment tools affect the quality of feedback given to medical oncology residents. J Cancer Educ. 35(1):165–177.
   PubMed Web of Science ®Google Scholar
• Triola MM, Pusic MV. 2012. The education data warehouse: a transformative tool for health education research. J Grad Med Educ. 4(1):113–115.
   PubMedGoogle Scholar
• van der Aa JE, Aabakke AJM, Ristorp Andersen B, Settnes A, Hornnes P, Teunissen PW, Goverde AJ, Scheele F. 2020. From prescription to guidance: a European framework for generic competencies. Adv Health Sci Educ Theory Pract. 25(1):173–187.
   PubMed Web of Science ®Google Scholar
• Van Melle E, Frank JR, Holmboe ES, Dagnone D, Stockley D, Sherbino J, International Competency-based Medical Education Collaborators. 2019. A core components framework for evaluating implementation of competency-based medical education programs. Acad Med. 94(7):1002–1009.
   PubMed Web of Science ®Google Scholar
• Van Melle E, Gruppen L, Holmboe ES, Flynn L, Oandasan I, Frank JR, International Competency-Based Medical Education Collaborators. 2017. Using contribution analysis to evaluate competency-based medical education programs: it’s all about rigor in thinking. Acad Med. 92(6):752–758.
   PubMed Web of Science ®Google Scholar
• Van Melle E. 2016. Using a logic model to assist in the planning, implementation, and evaluation of educational programs. Acad Med. 91(10):1464.
   PubMed Web of Science ®Google Scholar
• van Rossum TR, Scheele F, Sluiter HE, Bosman PJ, Rijksen L, Heyligers IC. 2018. Flexible competency based medical education: more time efficient, higher costs. Med Teach. 40(3):315–317.
   PubMed Web of Science ®Google Scholar
• van Rossum TR, Scheele F, Sluiter HE, Paternotte E, Heyligers IC. 2018. Effects of implementing time-variable postgraduate training programmes on the organization of teaching hospital departments. Med Teach. 40(10):1036–1041.
   PubMed Web of Science ®Google Scholar
• Whitehead CR, Kuper A. 2017. Faith-based medical education. Adv Health Sci Educ Theory Pract. 22(1):1–3.
   PubMed Web of Science ®Google Scholar
• Witteles RM, Verghese A. 2016. Accreditation council for graduate medical education (ACGME) milestones-time for a revolt? JAMA Intern Med. 176(11):1599–1600.
   PubMed Web of Science ®Google Scholar
• Zafar MA, Diers T, Schauer DP, Warm EJ. 2014. Connecting resident education to patient outcomes: the evolution of a quality improvement curriculum in an internal medicine residency. Acad Med. 89(10):1341–1347.
   PubMed Web of Science ®Google Scholar