Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales

Abstract

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.

Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017–18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.

Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.

Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.

Glossary

McNemar test is a statistical procedure used to evaluate differences between paired ordinal-level scores. In medicine, it is often used to make comparisons between findings of different diagnostic procedures. https://journals.sagepub.com/doi/abs/10.1177/0962280214541852

Entrustable Professional Activities (EPAs) are specific units of work common in medical practice. EPAs provide a competency-based framework for assessing level of entrustment (or alternatively, supervision) appropriate for learners in clinical workplaces. AAMC (2014) Association of American Medical Colleges Core entrustable professional activities for entering residency: curriculum developers' guide. AAMC iCollaborative, Washington, DC.

Acting Internships (AIs) are intensive, inpatient experiences designed to provide medical students with increased responsibilities for the assessment and management of patients. Students are integrated into the health care team and call schedule. During their AI, students assume ownership for the overall care of their assigned patients for the entire continuum of admission to discharge including transitions of care. In a closely supervised setting, the Acting Intern will experience the roles and responsibilities of an intern on the team. Acting Internships, Vanderbilt School of Medicine, https://medschool.vanderbilt.edu/ume/acting-internships/

Acknowledgments

The Educational Design and Informatics team developed a portfolio system to facilitate competency-based assessment. Medical students, faculty raters, and educational staff contributed their time and expertise to provide feedback and continuously improve the assessment system. Graduate students Yao Sang and Yasi Wang created integrated datasets to support quantitative analysis. Local assessment experts and colleagues in the AAMC national EPA pilot shared invaluable experiences, interpretations, and guidance.

Disclosure statement

The authors report no conflict of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

William B. Cutrer

William B. Cutrer, MD, MEd, is an associate professor of pediatrics, critical care medicine, and associate dean for undergraduate medical education at the Vanderbilt University School of Medicine, Nashville, TN, USA.

Regina G. Russell

Regina G. Russell, PhD, MA, MEd, is director of learning system outcomes and assistant in medical education and administration at Vanderbilt University School of Medicine, Nashville, TN, USA.

Mario Davidson

Mario Davidson, PhD, MS, MAS, MA, is an assistant professor in biostatistics at Vanderbilt University School of Medicine, Nashville, TN, USA.

Kimberly D. Lomis

Kimberly D. Lomis, MD, is vice president for UME innovations at the American Medical Association, Chicago, IL, USA.