A novel workplace-based assessment for competency-based decisions and learner feedback

Abstract

Background: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace.

Methods: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement.

Results: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4–8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process.

Conclusions: Outcomes – including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment – provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

Ethical approval

This study was reviewed and approved by each participating residency program’s Institutional Review Board and by the Institutional Review Board of the University of Illinois at Chicago.

Acknowledgements

The following members of the PMAC Module 1 Study Group also meet the criteria for authorship of this paper and should be so indexed: Beatrice Boateng, Ann Burke, Su-Ting T. Li, Julia Shelburne, Teri L. Turner. Additional members of the group who should be indexed as collaborators on this work include Dorene Balmer, Vasu Bhavaraju, Kim Boland, Alan Chin, Sophia Goslings, Hilary Haftel, Nicola Orlov, Amanda Osta, Sara Multerer, Jeanine Ronan, Sahar Rooholamini, Rebecca Tenney-Soeiro, Rebecca Wallihan, and Anna Weiss.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Glossary

Competency-based decisions: Decisions to advance learners, justify learner or practitioner readiness or privilege to care for patients (i.e. take on a level of responsibility with a particular level of supervision or independence), or support claims about an individual practitioner/learner (who may need additional training, for example) based on a competency or performance framework.

Additional information

Funding

This study was funded by the National Board of Medical Examiners and the American Board of Pediatrics. Authors Margolis and Clauser are employees of the National Board of Medical Examiners. Author Carraccio is an employee of the American Board of Pediatrics. Authors Hicks and Schwartz were supported in part by contracts between the funders and their respective institutions.

Notes on contributors

Patricia J. Hicks

Patricia J. Hicks, MD, MHPE, is Professor of Clinical Pediatrics in the Perelman School of Medicine at the University of Pennsylvania and Director, Pediatrics Milestones Assessment Collaborative, a joint project of the American Board of Pediatrics, the Association of Pediatric Program Directors and the National Board of Medical Examiners^©.

Melissa J. Margolis

Melissa J. Margolis, PhD, is Senior Measurement Scientist at the National Board of Medical Examiners (NBME). Her work over the past 20 years has focused on workplace-based assessment, assessment and instrument design, validity, automated scoring of complex performance tasks, and standard setting.

Carol L. Carraccio

Carol L. Carraccio, MD, MA, is Vice President of Competency-based Assessment at the American Board of Pediatrics where she leads many national research projects in assessment.

Brian E. Clauser

Brian E. Clauser, EdD, is Vice-President for the Center for Advanced Assessment at the National Board of Medical Examiners. His research interests include automated scoring of simulations, validity theory, standard setting, applications of generalizability theory, and workplace-based assessment.

Kathleen Donnelly

Kathleen Donnelly, MD, is a critical care attending and Director of the Inova Children’s Hospital Pediatric Residency Program.

H. Barrett Fromme

H. Barrett Fromme, MD, MHPE, is an Associate Professor of Pediatrics at the University of Chicago Pritzker School of Medicine where she is the Associate Pediatric Residency Program Director and the Director of Faculty Development in Medical Education.

Kimberly A. Gifford

Kimberly A. Gifford, MD, is assistant professor of pediatrics and primary care faculty at Geisel School of Medicine at Dartmouth and Director of Competency-based Education at the Children’s Hospital at Dartmouth-Hitchcock.

Sue E. Poynter

Sue E. Poynter, MD, MEd, is Associate Professor of Pediatrics and Director of the Pediatric Residency Program at Cincinnati Children’s/University of Cincinnati where she is also a critical care attending and Medical Director, Division of Respiratory Care.

Daniel J. Schumacher

Daniel J. Schumacher, MD, MEd, is Assistant Professor of Pediatrics and Pediatric Emergency Medicine physician at Cincinnati Children’s Hospital/University of Cincinnati and PhD Candidate at Maastricht University School of Health Professions Education in Maastricht, The Netherlands. His research efforts focus on entrustment and the association between learner and patient outcomes.

Alan Schwartz

Alan Schwartz, PhD, is the Michael Reese Endowed Professor of Medical Education, Associate Head and Director of Research in the Department of Medical Education at the University of Illinois at Chicago. He is also Director of the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network.