Simulation-based assessments in health professional education: a systematic review

Figures & data

Figure 1 Flow diagram of search.

Table 1 Adapted critical appraisal tool

	Yes (=1)	No (=0)	Not addressed (=0)	Not applicable (=0)
1. Was the study purpose clearly stated?
2. Was relevant background literature reviewed?
3. Was the sample described in detail?
4. Was sample size justified?
5. Were the outcome measures reliable?
6. Were the outcome measures valid?
7. Was intervention described in detail?
8. Was contamination avoided?
9. Was cointervention avoided?
10. Were results reported in terms of statistical significance?
11. If multiple outcomes, was that taken into account in the statistical analysis?
12. Were the analysis methods appropriate?
13. Was educational importance reported?
14. Were dropouts reported?
15. Were the conclusions appropriate given study methods and results?

Note: Data from Law et al.²⁰

Table 2 Summary of included articles

Author (year)	Sample size/discipline	Primary outcome	Simulation type	Assessment type	Main findings
Asprey et al (2007)^Citation67	101 Medical students, 71 physician assistant students	Comparison of third year medical student scores with physician assistant scores on SP assessments - clinical skill checklist	SP	Checklists – completed by the SP	SP-based assessments were not able to distinguish differences between medical and physician assistant performance and are therefore measuring clinical experience as the two groups had comparable clinical experience.
Bick et al (2013)^Citation55	26 Anesthetists	Comparison of expert vs novice anesthetist performance on a simulated transesophageal echocardiography examination – time and accuracy checklist	Virtual reality	Checklists – completed via video by blinded (voice masking technology) experts	Simulated transesophageal echocardiography examination is able to discriminate between expert and novice performers.
Boulet et al (2003)^Citation59	24 Medical students, 13 medical residents	Reliability and validity of simulation assessment for final year medical students and residents	High-fidelity patient simulator	Checklists – scored via video by four raters (two unaware of the training levels of participants)	Simulation can validly and reliably assess acute care skills, but multiple encounters are required to predict performance.
Burns et al (2013)^Citation62	26 Medical doctors (16 interns, five PGY-2, two PGY-3, two chief residents, and three pediatric hematology/oncology fellows)	Performance scores of interns (<1-year experience in pediatrics) vs the rest of the residents and fellows (>1-year experience)	High-fidelity pediatric patient simulator	Checklists and global rating scale – scored via video by two raters	The manikin- based assessment provided reliable and valid measures of participants’ performance.
Edelstein et al (2000)^Citation68	147 Medical students	Comparison of three measures of medical student performances - SP examinations, computer- based case simulations, and traditional performance indicators (GPAs)	SP and computer- based simulations	Unclear how they were assessed on the SP examinations – rated by both the SP as well as the author	Differences were found between the SP assessments and computer simulations and traditional assessments. A multipronged assessment approach is supported.
Fehr et al (2011)^Citation61	27 Anesthetic residents and eight anesthetic fellows	Reliability and validity of simulation assessments for anesthesia residents and pediatric fellows	High-fidelity patient simulator	Checklists – rated by lead investigator and two experts	The multiple-scenario assessment could reliably assess pediatric residents and determine the skill level of participants. Further validation required, including comparing of clinical performance.
Gimpel et al (2003)^Citation71	121 Osteopathy students	Reliability and validity of simulation assessment the Comprehensive Osteopathic Medical Licensing Examination – USA performance-based clinical skills examination (COMPLEX-USA-PE)	SP	Checklists – rated by SPs on all stations and experts on the stations that the participants may have chosen to perform a manipulation technique as part of their treatment	Acceptable reliability and validity of the assessment. Use of the Complex-USA-PE is supported for future use to assess the readiness of osteopathic medical students for clinical practice.
Grantcharov et al (2005)^Citation75	Ten medical students, ten medical residents, and eight medical doctors	Validity of virtual reality simulator for assessment of gastrointestinal endoscopy skills. Comparison of time and accuracy scores in three groups (experienced, residents, and medical students)	Virtual reality	Parameters were calculated and recorded by the computer	Virtual reality simulator was able to distinguish between the three levels of experience and therefore possessed acceptable construct validity.
Hawkins et al (2004)^Citation70	54 Medical doctors	Evaluation of computer- based case simulations and SP methods for patient management skills via checklists	Computer-based case simulations and SP	Computer-based scoring and checklists and a patient perception questionnaire for the SP station – rated by the SP	Computer-based case simulations and SP examinations were unable to distinguish between different experience levels; it appears that they may be useful in a multipronged assessment approach.
Iyer et al (2013)^Citation78	16 Medical residents	Validity of objective structured assessment of technical skills for neonatal lumbar puncture via competency-based scoring tool	Task trainer	Checklist and global rating scale – rated by six raters via video	Reasonable evidence of validity of this assessment tool. Adoption of this tool in clinical environments is recommended for providing formative and summative feedback to improve resident skills.
Lammers et al (2009)^Citation76	212 Paramedics	Identifying performance of paramedics in simulated pediatric emergencies using prevalidated scoring checklists	Low-, medium-, and high-fidelity patient simulators	Checklist – rated by one evaluator at the time and an author via video later, who amended scores if required and made the final decision	Scores from manikin- based simulations objectively identified multiple performance deficiencies in paramedics.
Lipner et al (2010)^Citation56	115 Medical cardiology fellows and cardiologists	Evaluation of simulation- based assessment of technical and cognitive skills of physicians during coronary interventions by determining whether assessment differentiates performances of novice, skilled, and expert	Virtual reality	Computer-based scoring system based on the consensus of a committee rating that included rating of the potential risk of taking that action	This assessment approach was able to identify poor- performance physicians who are unlikely to be providing appropriate patient care.
McBride et al (2011)^Citation58	29 Medical residents (13 interns, nine second year residents, six third year residents, and one chief resident)	Multiple-scenario assessment (n=20) of residents’ acute pediatric management skills using action checklists and global rating scales	High-fidelity pediatric patient simulator	Checklists and global rating scale – rated by two raters: one at the time, and one via video later	Residents’ scores in the assessments were found to be reliable and valid measures of their ability as long as multiple scenarios were used. Scores were able to discriminate between different skill levels.
Murray et al (2007)^Citation60	64 Medical residents and 35 anesthesiologists	Evaluation of anesthesia resident performances in a simulation- based intraoperative environment using an item score checklist	High-fidelity patient simulator	Checklists and time taken to perform the key actions – rated by two blinded raters via video; where there were large discrepancies, a third rater scored the performance	The simulation-based assessment was found to be a reliable and valid measure of performance and was able to distinguish more experienced from early training participants.
Nagoshi et al (2004)^Citation69	39 Medical students, 49 medical residents, and eleven medical fellows	Evaluation of a simulation- based standardized assessment of geriatric management skills by examining reliability and validity of the tool and scored using checklists	SP	Checklists – rated by the SP	The assessment was found to be reliable but did not discriminate between levels of training of participants.
Nunnink et al (2010)^Citation57	45 Medical doctors (ICU trainees)	Validity of simulation- based assessments of intensive care trainees procedural skills by comparing with written exams and oral viva formats	Low-high fidelity patient simulators and SP’s	Templates for written responses and checklists for all other – one rater (also rated the written questions on the same topic)	There was a lack of correlation between exam formats suggesting a multi- modal approach to assessment is favorable; and that simulation may be more useful in assessment of procedural skills.
Panzarella and Manyon (2008)^Citation66	34 Physical therapy students	Evaluation of an Integrated Standardized Patient Examination for Physical therapy students	SP	Checklists for the subjective and objective assessment, four-point rubric for the integration question – rated by four raters: two in the room (the SP and an expert rater), one via video (the participant) and a “criterion” rater (one of the authors blinded to the others’ score)	Student scores need comparison to other forms of assessment of performance. Interrater reliability was found to vary between the raters training for raters needs investigation. Good content validity reported by participants and expert raters. Needs further testing.
Penprase et al (2012)^Citation65	70 Registered nurses	Evaluation of simulation- based assessment of applicants to Nurse Anesthesia programs by comparing scores to candidate interview scores	High-fidelity patient simulator	Checklists – rated by one of two trained raters concealed behind a one-way mirror	Simulation-based assessment may provide a useful adjunct to the admission process.
von Wyl et al (2009)^Citation77	30 Paramedics	Determine the interrater reliability of assessing technical and nontechnical skills of paramedics during simulated emergency scenarios using previously validated checklists	Medium-fidelity patient simulator	Checklists – rated by two raters (an experienced emergency physician and a psychologist) via video	The assessment of technical and nontechnical skills using simulation-based assessment was shown to be feasible and reliable. There was a significant positive correlation between both skill types.
Waldrop et al (2009)^Citation63	12 Medical interns and 44 medical residents	Evaluate the reliability and validity of simulation- based assessment in the management of intraoperative equipment-related errors in anesthesia	High-fidelity patient simulator	Checklists – rated initially by two raters for the first four participants; as scores were 100% in agreement, only one rater was used for the remaining participants	The assessment was found to be an effective, reliable, and valid method to determine individual performance. The assessment was also able to discriminate between the experience levels of participants.
Weller et al (2005)^Citation64	21 Medical doctors	Evaluation of the psychometric properties of a simulation- based assessment for anesthetists	High-fidelity patient simulator	Global rating scale - rated by four blinded raters via video	The results show that 12–15 cases were required for acceptable reliability in this assessment modality. At lower level of performance, trainees overrated performance compared to those of high-performance levels, suggesting self- assessment may not be reliable.

Abbreviations: GPA, grade point average, ICU, intensive care unit; PGY, postgraduate year; SP, standardized patient.

LawMStewartDPollockNLettsLBoschJWestmorlandMCritical Review Form – Quatitative StudiesMcMaster University Occupational Therapy Evidence-Based Practice Research GroupHamilton, Ontario1998

 Google Scholar

AspreyDPHegmannTEBergusGRComparison of medical student and physician assistant student performance on standardized-patient assessmentsJ Physician Assist Educ20071841619

 Google Scholar

BickJSDeMariaSJrKennedyJDComparison of expert and novice performance of a simulated transesophageal echocardiography examinationSimul Healthc20138532933424030477

 PubMedGoogle Scholar

BouletJRMurrayDKrasJWoodhouseJMcAllisterJZivAReliability and validity of a simulation-based acute care skills assessment for medical students and residentsAnesthesiology20039961270128014639138

 PubMed Web of Science ®Google Scholar

BurnsTLDeBaunMRBouletJRMurrayGMMurrayDJFehrJJAcute care of pediatric patients with sickle cell disease: a simulation performance assessmentPediatr Blood Cancer20136091492149823633232

 PubMed Web of Science ®Google Scholar

EdelsteinRAReidHMUsatineRWilkesMSA comparative study of measures to evaluate medical students’ performancesAcad Med200075882583310965862

 PubMed Web of Science ®Google Scholar

FehrJJBouletJRWaldropWBSniderRBrockelMMurrayDJSimulation-based assessment of pediatric anesthesia skillsAnesthesiology201111561308131522037637

 PubMed Web of Science ®Google Scholar

GimpelJRBouletJRErrichettiAMEvaluating the clinical skills of osteopathic medical studentsJ Am Osteopath Assoc2003103626712834100

 PubMedGoogle Scholar

GrantcharovTPCarstensenLSchulzeSObjective assessment of gastrointestinal endoscopy skills using a virtual reality simulatorJSLS20059213013315984697

 PubMedGoogle Scholar

HawkinsRGaglioneMMLaDucaTAssessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patientsMed Educ200438995896815327677

 PubMed Web of Science ®Google Scholar

IyerMSSantenSANypaverMAccreditation Council for Graduate Medical Education Committee, Emergency Medicine and Pediatric Residency Review CommitteeAssessing the validity evidence of an objective structured assessment tool of technical skills for neonatal lumbar puncturesAcad Emerg Med201320332132423517267

 PubMed Web of Science ®Google Scholar

LammersRLByrwaMJFalesWDHaleRASimulation-based assessment of paramedic pediatric resuscitation skillsPrehosp Emerg Care200913334535619499472

 PubMed Web of Science ®Google Scholar

LipnerRSMessengerJCKangilaskiRA technical and cognitive skills evaluation of performance in interventional cardiology procedures using medical simulationSimul Healthc201052657420661006

 PubMedGoogle Scholar

McBrideMEWaldropWBFehrJJBouletJRMurrayDJSimulation in pediatrics: the reliability and validity of a multiscenario assessmentPediatrics2011128233534321746717

 PubMed Web of Science ®Google Scholar

MurrayDJBouletJRAvidanMPerformance of residents and anesthesiologists in a simulation-based skill assessmentAnesthesiology2007107570571318073544

 PubMed Web of Science ®Google Scholar

NagoshiMWilliamsSKasuyaRSakaiDMasakiKBlanchettePLUsing standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellowsAcad Med200479769870215234924

 PubMed Web of Science ®Google Scholar

NunninkLVenkateshBKrishnanAVidhaniKUdyAA prospective comparison between written examination and either simulation-based or oral viva examination of intensive care trainees’ procedural skillsAnaesth Intensive Care201038587688220865872

 PubMed Web of Science ®Google Scholar

PanzarellaKJManyonATUsing the integrated standardized patient examination to assess clinical competence in physical therapist studentsJ Phys Ther Educ20082232432

 Google Scholar

PenpraseBMiletoLBittingerAThe use of high-fidelity simulation in the admissions process: one nurse anesthesia program’s experienceAANA J2012801434822474804

 PubMedGoogle Scholar

von WylTZuercherMAmslerFWalterBUmmenhoferWTechnical and non-technical skills can be reliably assessed during paramedic simulation trainingActa Anaesthesiol Scand200953112112719032564

 PubMed Web of Science ®Google Scholar

WaldropWBMurrayDJBouletJRKrasJFManagement of anesthesia equipment failure: a simulation-based resident skill assessmentAnesth Analg2009109242643319608813

 PubMed Web of Science ®Google Scholar

WellerJMRobinsonBJJollyBPsychometric characteristics of simulation-based assessment in anaesthesia and accuracy of self-assessed scoresAnaesthesia200560324525015710009

 PubMed Web of Science ®Google Scholar