A multi-source feedback tool for measuring a subset of Pediatrics Milestones

Abstract

Background: The Pediatrics Milestones Assessment Pilot employed a new multisource feedback (MSF) instrument to assess nine Pediatrics Milestones among interns and subinterns in the inpatient context.

Objective: To report validity evidence for the MSF tool for informing milestone classification decisions.

Methods: We obtained MSF instruments by different raters per learner per rotation. We present evidence for validity based on the unified validity framework.

Results: One hundred and ninety two interns and 41 subinterns at 18 Pediatrics residency programs received a total of 1084 MSF forms from faculty (40%), senior residents (34%), nurses (22%), and other staff (4%). Variance in ratings was associated primarily with rater (32%) and learner (22%). The milestone factor structure fit data better than simpler structures. In domains except professionalism, ratings by nurses were significantly lower than those by faculty and ratings by other staff were significantly higher. Ratings were higher when the rater observed the learner for longer periods and had a positive global opinion of the learner. Ratings of interns and subinterns did not differ, except for ratings by senior residents. MSF-based scales correlated with summative milestone scores.

Conclusion: We obtain moderately reliable MSF ratings of interns and subinterns in the inpatient context to inform some milestone assignments.

Acknowledgments

Authors/Group Information

Writing group: Alan Schwartz, Melissa Margolis, Sara Multerer, Hilary Haftel, Daniel Schumacher

Group member sites: Personnel are listed as (I) for investigator or (C) for coordinator. University of Michigan: Hilary Haftel (I), Kristen Wright, MD (I), Rosalind Moffitt (C). Children's Hospital of Philadelphia: Patricia J. Hicks, MD (I); Jeanine Ronan (I); Rebecca Tenney-Soeiro (I); Dawn Young (C). Baylor College of Medicine: Teri Turner, MD (I); Melodie Allison, C-TAGME (I). University of Virginia: Linda Waggoner-Fountain, MD (I), Lisa Morris (C). Phoenix Children's Hospital: Grace Caputo, MD (I); Sandra Barker (I); Vasu Bhavaraju, MD (I); Ana Velazquez (C). Cincinnati Children's Hospital: Javier Gonzales del Rey (I); Sue Poynter, MD (I); Jackie Berry (C). Vermont Children's Hospital: Ann Guillot, MD (I); Karen Leonard, MD (I). University of Louisville: Sara Multerer, MD (I); Kimberly Boland, MD (I); Olivia Mittel, MD (I). University of Illinois at Chicago: Amanda Osta, MD (I); Michelle Barnes, MD (I); Emri Tas, MD (I); Jen McDonnell, MD (I). University of Florida: Cynthia Powell (I), Nicole Paradise Black, MD (I); Lilly Chang, MD (I). Winthrop University Hospital: Jill Leavens-Maurer, MD (I); Ulka Kothari, MD (I); Robert Lee, MD (I). Boston University: Daniel J. Schumacher, MD, MEd (I). Emory University: Susie Buchter, MD (I). Inova Fairfax Hospital for Children: Meredith Carter, MD (I). State University of New York at Stony Brook: Robyn Blair, MD (I). University of Texas Health Sciences Center: Sandra Arnold, MD (I); Mark C. Bugnitz, MD (I). Wright State University: Ann Burke, MD (I). Children's National Medical Center: Aisha Barber Davis, MD (I).

National Board of Medical Examiners: Stephen G. Clyman, MD; Thomas Rebbecchi, MD; Colleen Canavan, MS; Christa Chaffinch, MA; Melissa J. Margolis, PhD; Margaret Richmond, MS; Kathleen M. Rose; Leta Rose; Yelena Spector, MPH.

Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network: Alan Schwartz, PhD; Patricia J. Hicks, MD, MHPE.

Note to Medline indexers: Authors meeting ICMJE criteria who are not listed individually but included in the group author “APPD LEARN - NBME Pediatrics Milestones Assessment Group” are: Patricia J. Hicks, Stephen G. Clyman, and member site personnel listed as (I) above. All others above are collaborators.

Disclosure statement

The authors have no additional interests to disclose.

Funding information

This study was funded in part by the Association of Pediatric Program Directors (APPD). The ABP Foundation funded, in part, APPD’s support of this study. Schwartz is supported as APPD LEARN Director through a contract from APPD to the University of Illinois at Chicago. Hicks was also supported by APPD for during part of the project through a contract from APPD to the Children’s Hospital of Philadelphia.

Ethical approval

This study was reviewed and approved by each above listed PMAG member site’s Institutional Review Board and approved by the Institutional Review Board of the University of Illinois at Chicago.

Notes on contributors

Alan Schwartz, PhD, is the Michael Reese Endowed Professor of Medical Education and Associate Head, UIC Department of Medical Education, Research Professor, UIC Department of Pediatrics, and Director, APPD LEARN.

Melissa Margolis, PhD, is Senior Measurement Scientist, National Board of Medical Examiners, Philadelphia, Pennsylvania.

Sara Multerer, MD, is Associate Professor, Department of Pediatrics, and Associate Director, Pediatric Residency Program, University of Louisville.

Hilary Haftel, MD, MHPE, is Professor of Pediatrics, Internal Medicine, and Medical Education, and Associate Chair and Director of Pediatric Education, Department of Pediatrics, University of Michigan.

Daniel J. Schumacher, MD, MEd, is Assistant Professor, Emergency Medicine, Cincinnati Children’s Hospital Medical Center.