The construct and criterion validity of the multi-source feedback process to assess physician performance: a meta-analysis

Figures & data

Figure 1 Selection of studies for the meta-analysis.

Table 1 Characteristics of MSF studies with construct and criterion (concurrent/predictive) validity effect size measures

Study source	Group	Contrast†	MSF domain*	Effect size difference (dUWM^‡)
Archer et al^Citation20 Sample size, 112 pediatrics (20 specialist registrars, 92 senior house officers) Total forms =921	A	SPRS (MC)/SHO (MC)	2, and 5	1.22
Brinkman et al^Citation19 Sample size, 36 pediatric residents (16 with feedback and 16 with no feedback) Total forms =1,263	A	Feedback (MC)/No-feedback (MC)	1, 2, and 3	1.8
Massagli and Carline^Citation21 Sample size, 56 rehabilitation residents (nine PGY2, nine PGY3, nine PGY4) Total forms =930	A	PGY2/PGY3 PGY2/PGY4 PGY3/PGY4	1, 2, 4, and 5 1, 2, 4, and 5 1, 2, 4, and 5	0.05 0.17 0.23
Archer et al^Citation8 Sample size, 553 multiple specialties residents (219 Foundation year 1,334 Foundation year 2) Total forms =5,544	A	Foundation year 1 (MC)/Foundation year 2 (MC)	2, and 5	0.34
Archer et al^Citation15 Sample size, 577 pediatric (343 SPRS year 2, 201 SPRS year 4, 10 pediatricians in years 1, 3, 5, 6) Total forms =4,770	A	SPRS year 2 (MC)/SPRS year 4 (MC)	2, and 5	0.29
Wood et al^Citation18 Sample size, 67 obstetrics and gynecology residents Total forms =578	B	ObGyn time 1/ObGyn time 2	4, and 5	2.41
Lockyer et al^Citation22 Sample size, 250 family physicians Total forms =500	B	Phys time 1/Phys time 2 (Self)	1, 2, 3, and 4	0.46
Brinkman et al^Citation19 Sample size, 36 pediatric residents Total forms =1,263	B	Nurse time 1 (CW)/Nurse time 2 (CW) (Parents) time 1/(Parents) time 2	1, and 2 1, and 2	1.31 2.00
Violato et al^Citation9 Sample size, 250 family physicians Total forms =20,500	B	Phys time 1/Phys time 2 (MC) Phys time 1/Phys time 2 (CW) Phys time 1/Phys time 2 (Patients)	1, 2, and 5 1, and 3 1, 3, and 4	0.66 0.22 0.01
Risucci et al^Citation7 Sample size, 32 surgical residents Total forms =1,024	C	Self/Peer (MC) Self/Supervisors (MC) Peer (MC)/Supervisors (MC)	1, 2, and 5 1, 2, and 5 1, 2, and 5	0.56 0.21 0.25
Wenrich et al^Citation41 Sample size, 318 internal medicine physicians Total forms =1,877	C	Nurse (CW)/Phys (MC) medical knowledge Nurse (CW)/Phys (MC) humanistic	2, and 5 2, and 5	0.51 −0.46
Lelliott et al^Citation42 Sample size, 347 psychiatrists Total forms =11,426	C	Self/MC Patients/MC	2, 3, and 5 2, 3, and 5	0.47 0.85
Violato et al^Citation43 Sample size, 28 family physicians Total forms =170	C	Self/MC Self/Patients Self/CW	1, 2, 4, and 5 1, 2, 4, and 5 1, 2, 3, and 5	0.58 0.95 0.77
Hall et al^Citation3 Sample size, 295 multiple specialties Physicians Total forms =11,665	C	Self/Patients Self/MC Self/Consultant (MC) Self-Referring physicians (MC) Self/CW Consultant (MC)/MC Consultant (MC)/CW	1, 2, 3, 4, and 5 1, 2, and 5 1, 2, and 5 1, 2, and 5 1, 2, 3, and 5 1, 2, and 5 1, 2, 3, and 5	1.30 0.37 0.80 1.18 0.76 0.46 0.18
Thomas et al^Citation44 Sample size, 16 internal medicine residents Total forms =177	C	MC (Intern)/MC MC (Intern)/CW MC/CW	2, and 5 2, and 5 2, and 5	0.41 1.06 0.65
Lipner et al^Citation45 Sample size, 356 internal medicine physicians Total forms =12,460	C	MC/Patients	1, 2, and 3	2.60
Violato et al^Citation5 Sample size, 252 surgeons Total forms =7,237	C	Self/MC Self/CW Self/Patients MC/CW MC/Patients CW/Patients	1, 2, 3, and 5 1, 2, 3, and 5 1, 2, 3, 4, and 5 1, 2, 3, and 5 1, 2, 3, 4, and 5 3, 4, and 5	0.62 0.61 0.58 0.00 0.00 0.00
Wood et al^Citation27 Sample size, 7 radiology residents Total forms =57	C	Patients/MC Patients/CW MC/CW	1, and 3 1, and 3 1, and 3	0.98 1.31 0.04
Joshi et al^Citation46 Sample size, 8 obstetrics/gynecology residents Total forms =512	C	MC/CW MC/Patients CW/Patients	3, and 5 3, and 5 3, and 5	1.34 0.43 0.97
Lockyer et al^Citation47 Sample size, 197 anesthesiology physicians Total forms =5,957	C	MC/Patients	1, 2, and 3	0.06
Violato et al^Citation48 Sample size, 100 pediatric physicians Total forms =3,963	C	Self/MC Self/CW Self/Patients MC/CW MC/Patients CW/Patients	1, 2, and 3 1, 2, 3, and 5 1, 2, 3, and 4 1, 2, 3, and 5 1, 2, 3, and 4 1, 3, 4, and 5	0.04 0.18 0.07 0.97 0.79 0.26
Violato et al^Citation32 Sample size, 101 psychiatry physicians Total forms =4,069	C	Self/MC Self/CW Self/Patients MC/CW MC/Patients CW/Patients	1, 2, and 4 1, 2, 3, 4, and 5 1, 2, 3, and 4 1, 2, 3, and 5 1, 2, 3, and 4 1, 3, 4, and 5	0.83 1.52 1.13 0.68 0.28 0.40
Archer et al^Citation8 Sample size, 553 multiple specialties residents Total forms =5,544	C	(Consultant) MC/(Resident) MC	2, and 5	0.37
Pollock et al^Citation14 Sample size, 6 plastic surgery residents Total forms =240	C	CW/MC	1, 2, 3, 4, and 5	0.87
Davies et al^Citation40 Sample size, 92 histopathology residents Total forms =1,012	C	Consultant (MC)/CW	2, and 4	0.98
Campbell et al^Citation33 Sample size, 291 multiple specialties physicians Total forms =18,023	C	Patients/MC	1, 2, 3, and 5	0.19
Meng et al^Citation34 Sample size, 15 anesthesiology residents Total forms =429	C	Nurse (CW)/Secretaries (CW) Nurse (CW)/Nurse aids (CW) Nurse (CW)/Technicians (CW) Secretaries (CW)/Nurse aids (CW) Secretaries (CW)/Technicians (CW) Nurse aids (CW)/Technicians (CW)	1, 3, and 5 1, 3, and 5 1, 3, and 5 1, 3, and 5 1, 3, and 5 1, 3, and 5	0.16 0.64 0.65 0.16 0.46 0.00
Lockyer et al^Citation35 Samples size, 101 pathologists/laboratory physicians Total forms =808	C	Self/MC Self/Referring physicians (MC) Self/CW MC/Referring physicians (MC) MC/CW Referring physicians (MC)/CW	1, 2, and 5 1, 2, 4, and 5 1, 2, 3, and 5 1, 2, 4, and 5 1, 2, 3, and 5 1, 2, 3, and 4	0.22 0.58 0.18 0.38 0.03 0.40
Lockyer et al^Citation36 Sample size, 187 emergency medicine physicians Total forms =6,889	C	Self/MC Self/CW Self/Patients MC/CW MCPatients CW/Patients	1, 2, and 4 1, 2, 4, and 5 1, 2, 3, 4, and 5 1, 2, 4, and 5 1, 2, 3, 4, and 5 1, 2, 3, and 5	0.78 0.93 1.13 0.43 0.63 0.17
Archer et al^Citation15 Sample size, 577 pediatric residents Total forms =4,770	C	Consultant (MC)/Resident (MC)	2, and 5	0.64
Chandler et al^Citation16 Sample size, 66 pediatrics residents Total forms =823	C	Self/Attending (MC) Self/CW Self/Patients Attending (MC)/CW Attending (MC)/Patients CW/Patients	3, and 5 3, and 5 3, and 5 3, and 5 3, and 5 3, and 5	0.87 1.10 0.08 0.26 0.30 0.45
Campbell et al^Citation17 Sample size, 179 family physicians Total forms =10,895	C	Patients/MC	1, 2, 3, and 5	0.02
Archer and McAvoy^Citation37 Sample size, 68 different specialties physicians Total forms =2,365	C	Patients/MC Assessor nominated by physicians/ assessors nominated by referring body	2, and 5 2, and 5	1.90 1.91
Overeem et al^Citation38 Sample size, 146 multiple specialties Physicians Total forms =3,648	C	MC/Patients MC/CW CW/Patients	1, 2, 3, 4, and 5 1, 2, 3, and 4 1, 2, 3, and 5	0.44 0.75 0.45
Lockyer et al^Citation39 Sample size, 216 surgeons Total forms =9,072	C	Self/MC Self/CW Self/Patients MC/CW MC/Patients CW/Patients	1, 2, 3, and 4 1, 2, and 3 1, 2, 3, 4, and 5 1, 2, 3, and 4 1, 2, 3, 4, and 5 3, 4, and 5	1.11 0.86 1.00 0.44 0.30 0.21
Qu et al^Citation23 Sample size, 258 multiple specialties residents Total forms =4,128	C	Self/Attending (MC) Self/MC Self/CW Self/Patients Self/Office staff (CW) Attending (MC)/MC Attending (MC)/CW Attending (MC)/Patients Attending (MC)/Office staff (CW) Patients/Office staff (CW) Patients/MC Patients/CW	1, and 3 1, and 3 1, and 3 1, 2, 3, 4, and 5 1, and 3 1, and 3 1, and 3 1, 2, 3, 4, and 5 1, and 3 1, 2, 3, 4, and 5 1, 2, 3, 4, and 5 1, 2, 3, 4, and 5	0.30 0.13 −0.55 0.19 1.78 0.08 0.82 0.38 2.31 1.87 0.37 0.42
Lockyer et al^Citation49 Sample size, 37 general practice physicians Total forms =1,130	C	Self/MC Self/CW Self/Patients MC/CW MC/Patients CW/Patients	1, and 2 1, 2, and 3 1, 2, 3, and 4 1, 2, and 3 1, 2, 3, and 4 1, 3, and 4	0.22 0.05 0.04 0.22 0.21 0.00
Risucci et al^Citation7 Sample size, 32 surgical residents Total forms =1,024	D	MSF/ABSITE	1, 2, and 5	1.45
Wood et al^Citation27 Sample size, 7 radiology residents Total forms =57	D	MSF (PT)/global examination MSF (MC)/global examination MSF (CW)/global examination	1, and 3 1, and 3 1, and 3	1.96 1.02 1.60
Davies et al^Citation40 Sample size, 92 histopathology residents Total forms =1,012	D	MSF (PATH-SPRAT)/OSPE	2, and 3	1.09
Yang et al^Citation24 Sample size, 245 multiple specialties residents Total forms =1,053	D	MSF/small scale OSCE MSF/small scale OSCE + DOPS	1, 2, and 3 1, 2, and 3	0.79 2.07

Notes:

† A, predictive validity (physicians in different years level); B, predictive validity (physicians performance on MSF in two occasions separated with time); C, concurrent validity (differences in personnel ratings); D, construct validity (comparing MSF with standardized measures).

* MSF domains consist of the following: 1= professionalism, covering psychosocial skills, psychosocial management, humanistic qualities, compassion, attitude, professional development, teaching, and professional responsibilities and professional managements; 2= clinical competence covering clinical care, good medical practice, patient care, safe practice, clinical performance, knowledge, critical thinking, diagnosis, and management of complex problem; 3= communication, covering communication with staff and interpersonal communication skills; 4= management, covering reporting, self-management, administrative skills, office personal, access to doctor, practice process, physical office, and physical space; and 5= interpersonal relationships, covering relationships with patients, colleagues, family members, collegiality, collaboration, patient education, information provision, and patient interaction. Two of the authors (AA, TD) agreed on the names of the main five domains and agreed on the items included. d_UWM^‡ refers to the unweighted mean effect size difference as defined by Cohen’s d.

Abbreviations: CW, coworkers; MC, medical colleagues; MSF, multi-source feedback; PGY, postgraduate year; SPRS, specialist registrar; Phys, family physician; ObGyn, obstetrics and gynecology; CW, coworkers; ABSITE, American Board Of Surgery In-Training Examination; PATH-SPRAT, Pathology-Sheffield Peer Review Assessment Tool; OSPE, Objective Structured Practical Examination; OSCE, Objective Structured Clinical Examination; DOPS, Direct Observation of Procedural Skills; SHO, senior house officer; PT, patients.

Table 2 Random effects model (Cohen’s d) of the MSF domains with different physician years (group A)/different physician performance in two occasions (group B)

MSF domain measure	Studies included (number of outcomes)	Sample size	MSF with different physician years*	Studies included (number of outcomes)	Sample size	Difference between physicians’ performance on two occasions**
Professional	2 (4)	126	0.56 (0.39–1.59)	3 (6)	1,054	0.65 (0.30–1.00)
Clinical competence	5 (7)	1,335	0.62 (0.25–1.00)	3 (4)	554	0.99 (0.53–1.45)
Communication	1 (1)	72	1.78 (1.22–2.34)	2 (3)	750	0.23 (0.02–048)
Manager	1 (3)	54	0.14 (0.40–0.69)	3 (3)	567	0.92 (0.01–1.84)
Interpersonal relationships	4 (6)	1,263	0.42 (0.16–0.67)	2 (2)	317	1.50 (0.19–3.22)

Notes:

* Effect sizes combined for physicians in different year levels (different PGY level, eg, year 1, year 2, senior house officer, specialist registrar);^8,15,19–21

** effect sizes combined for physicians’ performance on two occasions separated by time (eg, 5 years, 7 months, 7 years).^9,18,19,22

Abbreviations: MSF, multi-source feedback; PGY, post graduate year.

Figure 2 Random and fixed effects model forrest plots for the MSF “personnel rating differences” for professional measures.

Notes: *The effect size values are taken from the raw data reported for the outcomes in studies group C. The Cochran Q-test for heterogeneity shows significant overall heterogeneity between studies.
Abbreviations: MSF, multi-source feedback; Pt, patients; MC, medical colleagues; Const, consultant; CW, co-workers; RefPhys, referring physicians; Nu, nursing; NuA, nursing aid; Sec, secretary; Tech, technicians; Officstaff, office staff; Attend, attending.

Table 3 Random effects model (Cohen’s d) of the MSF domains with personnel ratings/academic performance (groups C and D)

MSF domain measure	Studies included (number of outcomes)	Sample size	Personnel rating differences*	Studies included (number of outcomes)	Sample size	MSF with different global measurement**
Professional	19 (82)	12,415	0.56 (0.44–0.67)	3 (6)	543	1.42 (0.72–2.12)
Clinical competence	24 (75)	12,720	0.60 (0.49–0.72)	3 (4)	614	1.34 (0.65–2.05)
Communication	20 (76)	11,280	0.56 (0.42–0.67)	3 (6)	603	1.35 (0.71–1.99)
Manager	13 (38)	6,089	0.60 (0.45–0.74)	–	–
Interpersonal relationships	23 (74)	11,660	0.54 (0.44–0.64)	1 (1)	32	1.43 (0.87–2.00)

Notes:

* Effect size combined between differences in personnel ratings (ie, resident versus faculty, specialist versus consultant);^{3,5,7,8,14–17,23,27,32–39,41–49}

** effect sizes combined between MSF with standardized measures (eg, global ratings, OSPE).^7,24,27,40

Abbreviations: MSF, multi-source feedback; OSPE, Objective Structured Practical Examination.

ArcherJCNorciniJDaviesHAUse of SPRAT for peer review of paediatricians in trainingBMJ20053301251125315883137

 PubMed Web of Science ®Google Scholar

BrinkmanWBGeraghtySRLanpherBPEffect of multisource feedback on resident communication skills and professionalismArch Pediatr Adolesc Med2007161444917199066

 PubMedGoogle Scholar

MassagliTLCarlineJDReliability of a 360-degree evaluation to assess resident competenceAm J Phys Med Rehabil20078684585217885319

 PubMed Web of Science ®Google Scholar

ArcherJNorciniJSouthgateLHeardSDaviesHMini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK?Adv Health Sci Educ Theory Pract20081318119217036157

 PubMed Web of Science ®Google Scholar

ArcherJMcGrawMDaviesHAssuring validity of multisource feedback in a national programmePostgrad Med J20108652653120841329

 PubMed Web of Science ®Google Scholar

WoodLWallDBullockAHassellAWhitehouseACampbellI‘Team observation’: a six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gynecology training in the UKMed Teach200628e177e18417594543

 PubMedGoogle Scholar

LockyerJViolatoCFidlerHWhat multisource feedback factors influence physicians’ self-assessments? A five-year longitudinal studyAcad Med2007827780

 PubMed Web of Science ®Google Scholar

ViolatoCLockyerJFidlerHChange in performance: a 5-year longitudinal study of participants in a multi-source feedback programmeMed Educ2008421007101318823520

 PubMed Web of Science ®Google Scholar

RisucciDATortolaniAJWardRJRatings of surgical residents by self, supervisors and peersSurg Gynecol Obstet19891695195262814768

 PubMedGoogle Scholar

WenrichMDCarlineJDGilesLMRamseyPGRatings of the performances of practicing internists by hospital-based registered nursesAcad Med1993686806878397633

 PubMed Web of Science ®Google Scholar

LelliottPWilliamsRMearsAQuestionnaires for 360-degree assessment of consultant psychiatrists: development and psychometric propertiesBr J Psychiatry200819315616018670003

 PubMed Web of Science ®Google Scholar

ViolatoCMariniATowesJFeasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physiciansAcad Med1997728284

 PubMed Web of Science ®Google Scholar

HallWViolatoCLewkoniaRAssessment of physician performance in Alberta: the Physician Achievement ReviewCMAJ1999161525710420867

 PubMed Web of Science ®Google Scholar

ThomasPAGeboKAHellmannDBA pilot study of peer review in residency trainingJ Gen Intern Med19991455155410491244

 PubMed Web of Science ®Google Scholar

LipnerRSBlankLLLeasBFFortnaGSThe value of patient and peer ratings in recertificationAcad Med200277Suppl 106466

 PubMed Web of Science ®Google Scholar

ViolatoCLockyerJFidlerHMultisource feedback: a method of assessing surgical practiceBMJ200332654654812623920

 PubMed Web of Science ®Google Scholar

WoodJCollinsJBurnsideESPatient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skillsAcad Radiol20041193193915288041

 PubMed Web of Science ®Google Scholar

JoshiRLingFWJaegerJAssessment of a 360-degree instrument to evaluate residents’ competency in interpersonal and communication skillsAcad Med20047945846315107286

 PubMed Web of Science ®Google Scholar

LockyerJViolatoCFidlerHA multi-source feedback program for anesthesiologyCan J Anesth200653333916371607

 PubMed Web of Science ®Google Scholar

ViolatoCLockyerJFidlerHAssessment of pediatricians by a regulatory authorityPediatrics200611779680216510660

 PubMed Web of Science ®Google Scholar

ViolatoCLockyerJFidlerHAssessment of psychiatrists in practice through multisource feedbackCan J Psychiatry20085352553318801214

 PubMed Web of Science ®Google Scholar

PollockRADonnellyMBPlymaleMAStewartDHVasconezHC360-degree evaluations of plastic surgery resident accreditation council for graduate medical education competencies: experience using a short formPlast Reconstr Surg200812263964918626387

 PubMed Web of Science ®Google Scholar

DaviesHArcherJBatemanASpecialty-specific multi-source feedback: assuring validity, information trainingMed Educ2008421014102018823521

 PubMed Web of Science ®Google Scholar

CampbellJLRichardsSHDickensAGrecoMNarayananABrearleySAssessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnairesQual Saf Health Care20081718719318519625

 PubMedGoogle Scholar

MengLMetroDGPatelRMEvaluating professionalism and interpersonal and communication skills: implementing a 360-degree evaluation instrument in an anesthesiology residency programJ Grad Med Educ20091021622021975981

 PubMedGoogle Scholar

LockyerJViolatoCFidlerHAlakijaPThe assessment of pathologists/laboratory medicine physicians through a multisource feedback toolArch Pathol Lab Med20091331301130819653730

 PubMed Web of Science ®Google Scholar

LockyerJViolatoCFidlerHThe assessment of emergency physicians by a regulatory authorityAcad Emerg Med2006131296130317099191

 PubMed Web of Science ®Google Scholar

ChandlerNHendersonGParkBByerleyJBrownWDSteinerMJUse of a 360-degree evaluation in the outpatient settings: the usefulness of nurse, faculty, patient/family, and resident self-evaluationJ Grad Med Educ20101043043421976094

 PubMedGoogle Scholar

CampbellJNarayananABurfordBGrecoMValidation of a multi-source feedback tool for use in general practiceEduc Prim Care20102116517920515545

 PubMedGoogle Scholar

ArcherJMcAvoyPFactors that might undermine the validity of patient and multi-source feedbackMed Educ20114588689321848716

 PubMed Web of Science ®Google Scholar

OvereemKWollersheimHCArahOACruijsbergJKGrolRPLombartsKMEvaluation of physicians’ professional performance: an iterative development and validation study of multisource feedback instrumentsBMC Health Serv Res20121211122214259

 PubMed Web of Science ®Google Scholar

LockyerJViolatoCWrightBFidlerHChanRLong-term outcomes for surgeons from 3- and 4-year medical school curriculaCan J Surg20125515

 Web of Science ®Google Scholar

QuBZhaoYHSunBZAssessment of resident physicians in professionalism, interpersonal and communication skills: a multisource feedbackInt J Med Sci2012922823622577337

 PubMed Web of Science ®Google Scholar

LockyerJLBlackmoreDFidlerHA study of a multi-source feedback system for international medical graduates holding defined licencesMed Educ20064034034716573670

 PubMed Web of Science ®Google Scholar

YangYYLeeFYHsuHCAssessment of first-year post-graduate residents: usefulness of multiple toolsJ Chin Med Assoc20117453153822196467

 PubMed Web of Science ®Google Scholar