Capturing outcomes of competency-based medical education: The call and the challenge

Abstract

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

Keywords:

• Curriculum
• evaluation
• outcome-based
• learning outcomes
• general
• patient management
• management
• medical education research

Glossary

Complex Service Intervention is defined as: An intervention with several interacting components.

The dimensions of complexity include:

• Number of and interactions between components within the experimental and control interventions.

• Number and difficulty of behaviors required by those delivering or receiving the intervention.

• Number of groups or organizational levels targeted by the intervention.

• Number and variability of outcomes.

• Degree of flexibility or tailoring of the intervention permitted.

Source: Medical Research Council. Developing and Evaluating Complex Service Interventions, 2006.

Program Evaluation is defined as: The systematic investigation of the quality of programs for the purpose of decision-making.

Yarborough D, Shulha L, Hopson R, Caruthers F. 2001. The program evaluation standards: a guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage Publications Inc.

Logic Model is defined as: A common visual tool that illustrates how the program components should work together to produce intended outcomes.

McLaughlin J, Jordan G. 2004. Using logic models. In: Wholey KJS, Hatry HP, Newcomer KE, editors. Handbook of practical program evaluation. 2nd ed. Hoboken, NJ: Jossey-Bass.

Competency-based Medical Education is defined as: An approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It de-emphasizes time-based training and promises greater accountability, flexibility, and learner-centeredness.

Frank et al. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631–637.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article. Resources and secretariat support for this project were provided by the Royal College of Physicians and Surgeons of Canada.

Additional information

Notes on contributors

Elaine Van Melle

Elaine Van Melle, PhD, is a senior education scientist at the Royal College of Physicians and Surgeons of Canada and an adjunct faculty in the Department of Family Medicine, Queen’s University.

Andrew K. Hall

Andrew K. Hall, MD, FRCPC, MMEd, is an associate professor and CBME lead in the Department of Emergency Medicine, Queen’s University, and a clinician educator at the Royal College of Physicians and Surgeons of Canada.

Daniel J. Schumacher

Daniel J. Schumacher, MD, PhD, is an associate professor and Education Research Unit director in the Division of Emergency Medicine, Cincinnati Children’s Hospital Medical Center, and the Department of Pediatrics, University of Cincinnati College of Medicine.

Benjamin Kinnear

Benjamin Kinnear, MD, MEd, is an associate professor and assistant residency program director in the Department of Pediatrics, University of Cincinnati College of Medicine.

Larry Gruppen

Larry Gruppen, PhD, is a professor in the Department of Learning Health Sciences and director of the Master of Health Professions Education Program, University of Michigan.

Brent Thoma

Brent Thoma, MD, MA, MSc, is an associate professor in the Department of Emergency Medicine, University of Saskatchewan, and a clinician educator at the Royal College of Physicians and Surgeons of Canada.

Holly Caretta-Weyer

Holly Caretta-Weyer, MD, is a clinical assistant professor and assistant residency program director in the Department of Emergency Medicine, Stanford University School of Medicine.

Lara J. Cooke

Lara J. Cooke, MD, MSc, (MedEd), FRCPC is an associate professor in the Division of Neurology, Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, and a clinician educator at the Royal College of Physicians and Surgeons of Canada.

Jason R. Frank

Jason R. Frank, MD, MA(Ed), FRCPC, is the director of specialty education at the Royal College of Physicians and Surgeons of Canada and an associate professor and director of educational research & development in the Department of Emergency Medicine, University of Ottawa.