Evaluating medical residents as managers of care: a critical appraisal of assessment methods

Figures & data

Figure 1 Flowchart illustrating the various stages of the inclusion process.

Table 1 Summary of categories 1 and 2

Study	Method	Target group (number)	Comments	Statements about assessment tool
Essex and Jackson^Citation22	Pre- and post-test 65-item knowledge questionnaire (true/false/do not know).	N=40 34 trainees and 6 trainers	Highly significant increase in knowledge.	None
Junker et al^Citation25	Pre- and post-test evaluation of perceived knowledge of PM (0–5 scale). Standardized clerkship evaluation of PM module.	N=16	74% increase in perceived insurance issues knowledge; 27.7% increase in perceived managed care knowledge. 65% increase in perceived financial management knowledge; 41.4% increase in perceived laws and regulations knowledge. Curriculum was rated excellent in every area.	None
Kochar et al^Citation17	Development of a 360° assessment tool based on ACGME-competencies. Needs-assessment.	N=685 residents and fellows in 84 programs	Continuing redesigning residency trainings environment.	None
Bayard et al^Citation21	Pre- and post-test evaluation. 13-item self-assessment of knowledge/comfort of practice evaluation management curriculum (Likert scale 0–4). Evaluation of curriculum.	Not reported	(Strongly) agreed that the course was beneficial; increased interest and knowledge in PM. Mock practices were the glue that kept the didactic sessions together; increased interest in PM.	None
Higgins et al^Citation18	Developed a (45-item and three open-ended questions) 360° evaluation tool based on ACGME-competencies.	N=6 residents	Offers a comprehensive picture of an individual’s performance, facilitates self-awareness. Combination of tools gives possibly most accurate and reliable view.	None
Babitch and Chinsky^Citation20	Pre- and post-test (MCQ) executive leadership in health care-management.	Not reported	Slight improvement in post-test.	None
Babitch^Citation19	Pre- and post-test five questions about comprehension of the lecture content. ACGME and AAP-based curriculum. Standard satisfaction surveys for curriculum content.	Not reported	20%–40% improvement in comprehension.	None
Hemmer et al^Citation23	Pretest 20-item closed book and post-test 22- or 50-item open-book/note knowledge about key topics. Evaluation of curriculum content (scale 1–5).	N=16	Short-term understanding of leadership and management concepts seems to improve.	Validity was not measured
Jones et al^Citation24	Surgical coding compliance. Evaluation for each of the PM lectures (Likert scale 1–5).	Not reported	Surgical compliance increased from 36%–90% over a 12-month period. Evaluation scores ranged from 4.1–4.8 (average 4.3) (1= information not very pertinent – 5= information very pertinent).	None
LoPresti et al^Citation26	Pre- and post-test 40-item (25 MCQ + 15 Pick N type questions) knowledge test (constructed according to NBME-guidelines) to assess application based on AAFP and ACGME.	N=28 tests by 17 residents Pre-test N=17 Post-test N=11	Overall scores about PM knowledge improved in 7/14 areas in the intervention group. The comparison group improved more than the intervention group in the taxation topic.	Judged to have content validity No construct validity
Frohna et al^Citation27	OSCE, patient satisfaction, 360° evaluation, fishbowl evaluation. Clinical supervisor evaluation, SPs, written examinations. Computer-based PM problems, portfolios.	Not reported	Features a multifaceted assessment system that includes elements of each of Miller’s four levels of competence. Critical for obtaining an accurate picture of learners’ competency and may allow finer gradations of competency. However, individual residency programs need to shape their own assessment system in a way that addresses the local needs and resources best.	None

Abbreviations: AAFP, American Academy of Family Physicians; ACGME, Accreditation Council of Graduate Medical Education; MCQ, multiple-choice question; NBME, National Board of Medical Examiners; OSCE, objective structured clinical examination; PM, practice management; SP, standardized patient encounters; AAP, American Academy of Pediatrics.

Table 2 Summary of categories 3 and 4

Study	Assessment tool/method	Target group (number)	Comments
Reisdorff et al^Citation38	Modified current evaluation instrument based on ACGME-competencies.	Not reported	A practical method was proposed for modifying an existing evaluation tool to use in assessing residents’ GCs.
Swing^Citation39	Development and implementations of assessment methods based on ACGME framework; ratings, checklists, 360° evaluations, structural oral examinations, structured case discussion, simulators, models, simulations, and portfolios.	Not reported	Assessment approaches that tend to have a higher reliability are OSCEs, SP exams, and checklists. OSCE and SP exams seem to be the best method for conducting assessment for high-stakes decisions, advised to complement with snapshot assessment methods. 360° evaluation and portfolios may provide unique insights and should be further developed and tested.
Reisdorff et al^Citation35	Modified a 61-item global assessment device. Ordinal scale 1–9.	N=150 emergency medicine residents	Valid evaluation items for assessing GCs were developed by using a structured process. It is challenging to measure objective dynamics in addition to more objective, behavior-driven competencies.
Silber et al^Citation30	Modified global rating form assessment of 23 items based on ACGME competencies. (1–5 scale).	N=1,295 residents	Global rating forms were the most often used tool to assess residents’ GCs, may not be adequate to distinguish among the six ACGME GCs.
Weigelt et al^Citation34	Developed a 23-item 360° evaluation based on the ACGME-competencies. Likert scale 1–9.	N=10 residents	Compared with traditional ratings, 360° evaluation adds limited new information.
Roark et al^Citation37	Online assessment system for surgical post-graduate ACGME-based education to support implementation of various instruments; mini-clinical examination exercise, 360° assessment, oral presentation of research projects, written examination, in-service examination, and documentation of surgical experience.	N=1,336 evaluations residents otolaryngology	This study revealed significant difference among evaluators in total (all GCs) average score. Further conclusions are hard to draw and further experiments and research are needed.
Jefferies et al^Citation32	Developed a 10-station OSCE based on the CanMEDS competencies.	N=24 candidates, 13 faculty members	The OSCE may, despite the resource intensity and high costs, be useful as a reliable, valid assessment method to simultaneously assess multiple GCs.
Chou et al^Citation28	Two-question survey to measure use and satisfaction of: ITER, MCQ, SAQ, essay, oral examination, OSCE, simulation, and logbook. Likert scale 1–5.	N=280 program directors	ITER was the tool used mostly to assess management competency. Program directors concerned about assessment of management role in their programs. Satisfaction with evaluation management role (1= very dissatisfied), mean (SD): 2.95 (0.84).
Lee et al^Citation31	Report experience using implementation matrix of ACGME competencies. Tools used: OSCE, 360° evaluation, and portfolio.	Not reported	Provided specific practical solutions for overcoming the identified barriers.
Lurie et al^Citation33	A systematic review of assessment tools for ACGME competencies. Global rating form, 360° evaluation, direct observation, and portfolio.	N=56 articles	No evidence found that current assessment tools can assess the ACGME GCs independently of one another.
Varney et al^Citation36	Designed a developmental criterion-referenced assessment (CoBRA). Likert scale 1–9. Descriptive anchors. Compared ABIM score.	N=2,941 CoBRA N=3,648 ABIM form	Labor-/time-intensive project, but with positive returns: ongoing faculty development, focus on evaluation as a part of teaching, and the explanatory effect on goals and objectives within our department.
Garstang et al^Citation29	OSCE 9-station based on ACGME-competencies developed by OSCE committee (authors and department member). Compared with ABPMR scores.	N=9 residents	OSCE management scores had a (not significant) positive correlation with ABPMR part 1 scores. OSCE management scores had a (not significant) negative correlation with ABPMR part 2 scores.
Dyne et al^Citation42	SBP assessment tools; direct observation, global rating, 360° evaluation, portfolios, standardized oral examinations, written MCQs, chart-stimulated recall oral examination, OSCE, and patient survey. ACGME-based curriculum.	Not reported	Primary assessment methodologies: direct observation, global ratings, 360° evaluation, portfolio assessment. Testing: oral and written.
Kolva et al^Citation40	A systematic review of ACGME-competencies in PM curriculum: ITER, MCQs, SAQs, essays, oral examinations, OSCEs, simulations, and logbooks.	N=33 articles	Few studies on long-term outcomes of PM-related curricula. New PM curricular material developed should be evaluated and results should be published.

Abbreviations: ABIM, American Board of Internal Medicine; ABPMR, American Board of Physical Medicine and Rehabilitation; ACGME, Accreditation Council for Graduate Medical Education; CanMEDS, Canadian Medical Education Directives for Specialists; CoBRA, competency-based resident assessment; GC, general competencies; ITER, in-training evaluation report; MCQ, multiple-choice question; OSCE, objective structured clinical examination; PM, practice management; SAQ, short-answer question; SBP, systems-based practice; SD, standard deviation; SP, standardized patient encounters.

EssexBJacksonRNMoneymed: a game to develop management skills in general practiceJ R Coll Gen Pract1981312337357397338867

 PubMedGoogle Scholar

JunkerJAMillerTDavisMSPractice management: a third-year clerkship experienceFam Med2002342878911874029

 PubMed Web of Science ®Google Scholar

KocharMSSimpsonDEBrownDGraduate medical education at the Medical College of Wisconsin: new initiatives to respond to the changing residency training environmentWMJ20031022384212754907

 PubMedGoogle Scholar

BayardMPeeplesCRHoltJDavidDJAn interactive approach to teaching practice management to family practice residentsFam Med200335962262414523656

 PubMed Web of Science ®Google Scholar

HigginsRSBridgesJBurkeJMO’DonnellMACohenNMWilkesSBImplementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedbackAnn Thorac Surg2004771121714726026

 PubMed Web of Science ®Google Scholar

BabitchLAChinskyDJLeadership education for physicians. Refine your focus and communicate your goals when developing physician leadership programsHealthc Exec2005201495115656230

 PubMedGoogle Scholar

BabitchLATeaching practice management skills to pediatric residentsClin Pediatr (Phila)200645984684917041173

 PubMed Web of Science ®Google Scholar

HemmerPRKaronBSHernandezJSCuthbertCFidlerMETazelaarHDLeadership and management training for residents and fellows: a curriculum for future medical directorsArch Pathol Lab Med2007131461061417425393

 PubMed Web of Science ®Google Scholar

JonesKLebronRAMangramADunnEPractice management education during surgical residencyAm J Surg20081966878881 discussion 881–87219095103

 PubMed Web of Science ®Google Scholar

LoPrestiLGinnPTreatRUsing a simulated practice to improve practice management learningFam Med200941964064519816827

 PubMed Web of Science ®Google Scholar

FrohnaJGKaletAKachurEAssessing residents’ competency in care management: report of a consensus conferenceTeach Learn Med2004161778414987180

 PubMed Web of Science ®Google Scholar

ReisdorffEJHayesOWCarlsonDJWalkerGLAssessing the new general competencies for resident education: a model from an emergency medicine programAcad Med200176775375711448836

 PubMed Web of Science ®Google Scholar

SwingSRAssessing the ACGME general competencies: general considerations and assessment methodsAcad Emerg Med20029111278128812414482

 PubMed Web of Science ®Google Scholar

ReisdorffEJCarlsonDJReevesMWalkerGHayesOWReynoldsBQuantitative validation of a general competency composite assessment evaluationAcad Emerg Med200411888188415289197

 PubMed Web of Science ®Google Scholar

SilberCGNascaTJPaskinDLEigerGRobesonMVeloskiJJDo global rating forms enable program directors to assess the ACGME competencies?Acad Med200479654955615165974

 PubMed Web of Science ®Google Scholar

WeigeltJABraselKJBraggDSimpsonDThe 360-degree evaluation: increased work with little return?Curr Surg2004616616626 discussion 627–61815590037

 PubMedGoogle Scholar

RoarkRMSchaeferSDYuGPBranovanDIPetersonSJLeeWNAssessing and documenting general competencies in otolaryngology resident training programsLaryngoscope2006116568269516652072

 PubMed Web of Science ®Google Scholar

JefferiesASimmonsBTabakDUsing an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate trainingMed Teach2007292–318319117701631

 PubMed Web of Science ®Google Scholar

ChouSColeGMcLaughlinKLockyerJCanMEDS evaluation in Canadian postgraduate training programmes: tools used and programme director satisfactionMed Educ200842987988618715485

 PubMed Web of Science ®Google Scholar

LeeAGOettingTBeaverHACarterKThe ACGME Outcome Project in ophthalmology: practical recommendations for overcoming the barriers to local implementation of the national mandateSurv Ophthalmol200954450751719539838

 PubMed Web of Science ®Google Scholar

LurieSJMooneyCJLynessJMMeasurement of the general competencies of the accreditation council for graduate medical education: a systematic reviewAcad Med200984330130919240434

 PubMed Web of Science ®Google Scholar

VarneyAToddCHingleSClarkMDescription of a developmental criterion-referenced assessment for promoting competence in internal medicine residentsJ Grad Med Educ200911738121975710

 PubMedGoogle Scholar

GarstangSAltschulerELJainSDelisaJADesigning the objective structured clinical examination to cover all major areas of physical medicine and rehabilitation over 3 yrsAm J Phys Med Rehabil201291651952722469878

 PubMed Web of Science ®Google Scholar

DynePLStraussRWRinnertSSystems-based practice: the sixth core competencyAcad Emerg Med20029111270127712414481

 PubMed Web of Science ®Google Scholar

KolvaDEBarzeeKAMorleyCPPractice management residency curricula: a systematic literature reviewFam Med200941641141919492188

 PubMed Web of Science ®Google Scholar