Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

References

• American Educational Research Association. 1999. Standards for educational and psychological testing. Washington (DC): American Educational Research Association.
   Google Scholar
• Angus SV, Vu TR, Willett LL, Call S, Halvorsen AJ, Chaudhry S. 2017. Internal medicine residency program directors’ views of the core entrustable professional activities for entering residency: an opportunity to enhance communication of competency along the continuum. Acad Med. 92(6):785–791.
   PubMed Web of Science ®Google Scholar
• Association of American Medical Colleges (AAMC). 2014. Core entrustable professional activities for entering residency: curriculum developers' guide. AAMC iCollaborative, Washington, DC. [accessed 2 Nov 2019]. https://store.aamc.org/downloadable/download/sample/sample_id/63/%20
   Google Scholar
• Boursicot K, Etheridge L, Setna Z, Sturrock A, Ker J, Smee S, Sambandam E. 2011. Performance in assessment: consensus statement and recommendations from the Ottawa conference. Med Teach. 33(5):370–383.
   PubMedGoogle Scholar
• Brown DGM, Lupi C, Obeso V, Hyderi A, Cutrer B, Warren J, Carney P, Phillipi C. 2017a. Supervisory language task force executive summary.
   Google Scholar
• Brown DR, Warren JB, Hyderi A, Drusin RE, Moeller J, Rosenfeld M, Orlander PR, Yingling S, Call S, Terhune K, et al. 2017b. Finding a path to entrustment in undergraduate medical education: a progress report from the AAMC core entrustable professional activities for entering residency entrustment concept group. Acad Med. 92(6):774–779.
   PubMed Web of Science ®Google Scholar
• Carraccio CL, Englander R. 2013. From flexner to competencies: reflections on a decade and the journey ahead. Acad Med. 88(8):1067–1073.
   PubMed Web of Science ®Google Scholar
• Chen HC, van den Broek WE, ten Cate O. 2015. The case for use of entrustable professional activities in undergraduate medical education. Acad Med. 90(4):431–436.
   PubMed Web of Science ®Google Scholar
• Crossley J, Johnson G, Booth J, Wade W. 2011. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ. 45(6):560–569.
   PubMed Web of Science ®Google Scholar
• Crossley J, Jolly B. 2012. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ. 46(1):28–37.
   PubMed Web of Science ®Google Scholar
• Cutrer WB, Miller B, Pusic MV, Mejicano G, Mangrulkar RS, Gruppen LD, Hawkins RE, Skochelak SE, Moore DE. Jr. 2017. Fostering the development of master adaptive learners: a conceptual model to guide skill acquisition in medical education. Acad Med. 92(1):70–75.
   PubMed Web of Science ®Google Scholar
• Dolan BM, O’Brien CL, Green MM. 2017. Including entrustment language in an assessment form may improve constructive feedback for student clinical skills. Med Sci Educ. 27(3):461–464.
   Google Scholar
• Eliasz KL, Ark TK, Nick MW, Ng GM, Zabar S, Kalet AL. 2018. Capturing entrustment: using an end-of-training simulated workplace to assess the entrustment of near-graduating medical students from multiple perspectives. Med Sci Educ. 28(4):739.
   Google Scholar
• Englander R, Flynn T, Call S, Carraccio C, Cleary L, Fulton TB, Garrity MJ, Lieberman SA, Lindeman B, Lypson ML, et al. 2016. Toward defining the foundation of the MD degree: core entrustable professional activities for entering residency. Acad Med. 91(10):1352–1358.
   PubMed Web of Science ®Google Scholar
• Englander R, Frank JR, Carraccio C, Sherbino J, Ross S, Snell L, Collaborators I. 2017. Toward a shared language for competency-based medical education. Med Teach. 39(6):582–587.
   PubMed Web of Science ®Google Scholar
• Favreau MA, Tewksbury L, Lupi C, Cutrer WB, Jokela JA, Yarris LM, 2017. Constructing a shared mental model for faculty development for the core entrustable professional activities for entering residency. Acad Med. 92(6):759–764.
   PubMed Web of Science ®Google Scholar
• Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. 2012. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med. 87(10):1401–1407.
   PubMed Web of Science ®Google Scholar
• Halman S, Rekman J, Wood T, Baird A, Gofton W, Dudek N. 2018. Avoid reinventing the wheel: implementation of the Ottawa Clinic Assessment Tool (OCAT) in internal medicine. BMC Med Educ. 18(1):218.
   PubMed Web of Science ®Google Scholar
• Hoang NS, Lau JN. 2018. A call for mixed methods in competency-based medical education: how we can prevent the overfitting of curriculum and assessment. Acad Med. 93(7):996–1001.
   PubMed Web of Science ®Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568.
   PubMed Web of Science ®Google Scholar
• Kane MT. 2013. Validating the interpretations and uses of test scores. J Educ Measure. 50(1):1–73.
   Web of Science ®Google Scholar
• Khan K, Ramachandran S. 2012. Conceptual framework for performance assessment: competency, competence and performance in the context of assessments in healthcare–deciphering the terminology. Med Teach. 34(11):920–928.
   PubMed Web of Science ®Google Scholar
• Kreiter C. 2015. When I say… response process validity. Med Educ. 49(3):247–248.
   PubMed Web of Science ®Google Scholar
• Linn R, Baker E, Dunbar S. 1991. Complex, performance-based assessment: expectations and validation criteria. Educ Res. 20(8):15–21.
   Google Scholar
• Lomis KD, Russell RG, Davidson MA, Fleming AE, Pettepher CC, Cutrer WB, Fleming GM, Miller BM. 2017. Competency milestones for medical students: design, implementation, and analysis at one medical school. Med Teach. 39(5):494–504.
   PubMed Web of Science ®Google Scholar
• Lomis K, Amiel JM, Ryan MS, Esposito K, Green M, Stagnaro-Green A, Bull J, Mejicano GC. 2017. Implementing an entrustable professional activities framework in undergraduate medical education: early lessons from the AAMC core entrustable professional activities for entering residency pilot. Acad Med. 92(6):765–770.
   PubMed Web of Science ®Google Scholar
• Lucey CR, Thibault GE, ten Cate O. 2018. Competency-based, time-variable education in the health professions: crossroads. Acad Med. 93:S1–S5.
   PubMed Web of Science ®Google Scholar
• Messick S. 1989. Validity. In Linn R, editor. Educational measurement (3rd ed.). New York (NY): Macmillian Publishing Company.
   Google Scholar
• Messick S. 1994. The interplay of evidence and consequences in the validation of performance assessments. Educ Res. 23(2):13–23.
   Google Scholar
• Natesan P, Batley NJ, Bakhti R, El-Doueihi PZ. 2018. Challenges in measuring ACGME competencies: considerations for milestones. Int J Emerg Med. 11(1):1–6.
   PubMedGoogle Scholar
• Padilla JL, Benitez I. 2014. Validity evidence based on response processes. Psicothema. 26(1):136–144.
   PubMed Web of Science ®Google Scholar
• Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. 2016. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med. 91(2):186–190.
   PubMed Web of Science ®Google Scholar
• Rubin HJ, Rubin IS. 2004. Qualitative interviewing: the art of hearing data. Thousand Oaks (CA): Sage Publications, Inc.
   Google Scholar
• Sagasser MH, Fluit C, van Weel C, van der Vleuten CPM, Kramer AWM. 2017. How entrustment is informed by holistic judgments across time in a family medicine residency program: an ethnographic nonparticipant observational study. Acad Med. 92(6):792–799.
   PubMed Web of Science ®Google Scholar
• Saldana J. 2009. The coding manual for qualitative researchers. Thousand Oaks (CA): Sage Publications, Inc.
   Google Scholar
• Swing SR. 2007. The ACGME outcome project: retrospective and prospective. Med Teach. 29(7):648–654.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Regehr G. 2018. The power of subjectivity in the assessment of medical trainees. Acad Med. 94(3):1.
   Web of Science ®Google Scholar
• ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. 2015. Curriculum development for the workplace using Entrustable Professional Activities (EPAs): AMEE guide no. 99. Med Teach. 37(11):983–1002.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CP. 2014. When I say … context specificity. Med Educ. 48(3):234–235.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CP, Schuwirth LW. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.
   PubMed Web of Science ®Google Scholar
• Warm EJ, Kinnear B, Kelleher M, Sall D, Holmboe E. 2018. Transforming resident assessment: an analysis using Deming’s system of profound knowledge. Acad Med. 94(2):195–201.
   Web of Science ®Google Scholar
• Whitehead CR, Kuper A, Hodges B, Ellaway R. 2015. Conceptual and practical challenges in the assessment of physician competencies. Med Teach. 37(3):245–251.
   PubMed Web of Science ®Google Scholar