How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study

Abstract

Purpose: This study examines how Clinical Competency Committees (CCCs) synthesize assessment data to make judgments about residents’ clinical performances.

Methods: Between 2014 and 2015, after four six-month reporting periods to the Accreditation Council for Graduate Medical Education (ACGME), 7 of 16 CCC faculty at Rush University Medical Center completed questionnaires focused on their perspectives about rating residents on their achievement of the milestones and participated in a focus group. Qualitative data were analyzed using grounded theory. Milestones ratings for two six-month ACGME reporting cycles (n = 100 categorical residents) were also analyzed.

Results: CCC members weighted resident rotation ratings highest (weight = 37%), followed by faculty rotation comments (weight = 27%) and personal experience with residents (weight = 14%) for making judgments about learner’s milestone levels. Three assessment issues were identified from qualitative analyses: (1) “design issues” (e.g. problems with available data or lack thereof); (2) “synthesis issues” (e.g. factors influencing ratings and decision-making processes) and (3) “impact issues” (e.g. how CCC generated milestones ratings are used).

Conclusions: Identifying factors that affect assessment at all stages of the CCC process can contribute to improving assessment systems, including support for faculty development for CCCs. Recognizing challenges in synthesizing first and second-hand assessment data is an important step in understanding the CCC decision-making process.

Acknowledgments

The authors would like to thank the RUMC internal medicine residency program coordinators (Emelina Quintana and Laney McDougal) for their support of the project. In addition, we are grateful to our study participants, the RUMC internal medicine residents from whom we continue to learn, and the RUMC internal medicine department leadership for their support via the academic career development program.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Funding

This study received funding from the Department of Internal Medicine at RUMC. There was no external funding

Glossary

Clinical Competency Committee A “required body comprising three or more members of the active teaching faculty who is advisory to the program director and reviews the progress of all the residents in the program.”

ACGME Glossary of Terms July 1, 2013 accessed April 9, 2017 http://acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/abACGMEglossary.pdf

Notes on contributors

Dr. Ekpenyong, MD, MHPE, is an Associate Professor in the Department of Internal Medicine at Rush University Medical Center, Chicago, IL.

Dr. Baker, MD, MHPE, is and Associate Professor in the Department of Internal Medicine at Rush University Medical Center, Chicago, IL.

Dr. Harris, PhD, is a Professor and Head of the Department of Medical Education at the University of Illinois College of Medicine at Chicago, Chicago, IL.

Dr. Tekian, PhD, MHPE, is a Professor in the Department of Medical Education at the University of Illinois College of Medicine at Chicago, Chicago, IL

Dr. Abrams, MD, is an Associate Professor in the Department of Internal Medicine at Rush University Medical Center, Chicago, IL.

Dr. Reddy, MD, MHPE, is a Professor of medicine in the Department of Internal Medicine at the University of Chicago Pritzker School of Medicine and an associate program director in the Department of Internal Medicine at Mercy Hospital and Medical Center in Chicago, IL.

Dr. Park, PhD, is an Assistant Professor in the Department of Medical Education at the University of Illinois College of Medicine at Chicago, Chicago, IL.