Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis

Abstract

Background: Although technology-enhanced simulation is increasingly used in health professions education, features of effective simulation-based instructional design remain uncertain.

Aims: Evaluate the effectiveness of instructional design features through a systematic review of studies comparing different simulation-based interventions.

Methods: We systematically searched MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, Scopus, key journals, and previous review bibliographies through May 2011. We included original research studies that compared one simulation intervention with another and involved health professions learners. Working in duplicate, we evaluated study quality and abstracted information on learners, outcomes, and instructional design features. We pooled results using random effects meta-analysis.

Results: From a pool of 10 903 articles we identified 289 eligible studies enrolling 18 971 trainees, including 208 randomized trials. Inconsistency was usually large (I² > 50%). For skills outcomes, pooled effect sizes (positive numbers favoring the instructional design feature) were 0.68 for range of difficulty (20 studies; p < 0.001), 0.68 for repetitive practice (7 studies; p = 0.06), 0.66 for distributed practice (6 studies; p = 0.03), 0.65 for interactivity (89 studies; p < 0.001), 0.62 for multiple learning strategies (70 studies; p < 0.001), 0.52 for individualized learning (59 studies; p < 0.001), 0.45 for mastery learning (3 studies; p = 0.57), 0.44 for feedback (80 studies; p < 0.001), 0.34 for longer time (23 studies; p = 0.005), 0.20 for clinical variation (16 studies; p = 0.24), and −0.22 for group training (8 studies; p = 0.09).

Conclusions: These results confirm quantitatively the effectiveness of several instructional design features in simulation-based education.

Introduction

Technology-enhanced simulation permits educators to create learner experiences that encourage learning in an environment that does not compromise patient safety. We define technology-enhanced simulation as an educational tool or device with which the learner physically interacts to mimic an aspect of clinical care for the purpose of teaching or assessment. Previous reviews have confirmed that technology-enhanced simulation, in comparison with no intervention, is associated with large positive effects (Cook et al. 2011; McGaghie et al. 2011). However, the relative merits of different simulation interventions remain unknown. Since the advantages of one simulator over another are context-specific (i.e. a given simulator may be more or less effective depending on the instructional objectives and educational context), it makes sense to focus on the instructional design features that define effective simulation training—the active ingredients or mechanisms. A comprehensive synthesis of evidence would be timely and useful to educators.

One systematic review identified 10 key features based on prevalence in the literature, but did not examine the impact of these features on educational outcomes (Issenberg et al., 2005). Other reviews have found an association between longer training time and improved outcomes (McGaghie et al. 2006) and that simulation with deliberate practice has consistently positive effects (McGaghie et al. 2011). In a review of simulation in comparison with no intervention (Cook et al. 2011), subgroup meta-analyses provided weak evidence suggesting better outcomes when learning activities were distributed over >1 day and when learners were required to demonstrate mastery of the task. When comparing simulation with non-simulation instruction (Cook et al. 2012), subgroup meta-analyses suggested better outcomes when extraneous cognitive load was low, when learners worked in groups, and when feedback and learning time were greater.

However, such subgroup analyses represent an inefficient method of exploring the effectiveness of design features because they evaluate differences between studies, and between-study variation in learners, contexts, clinical topics, and outcome measures introduces error and confounds interpretations. Most of the subgroup interactions evaluated in these reviews (Cook et al. 2011, 2012) varied from outcome to outcome and most were statistically non-significant. The direct comparison of two instructional variations in a single study offers a less problematic approach, as it capitalizes on within-study (rather than between-study) design differences. For example, meta-analysis of head-to-head comparisons has been used to identify effective instructional design features in Internet-based instruction. (Cook et al. 2010b)

A comprehensive review of head-to-head comparisons of different simulation-based instructional interventions (i.e. comparative effectiveness studies) would fulfill two important needs in health professions education. First, a quantitative synthesis of evidence regarding specific instructional design features would immediately inform educational practice. Second, a thematic summary of the comparisons made and research questions addressed would inform future research by providing a list of common comparisons (indicating themes felt to be important and likely worthy of further study) and by highlighting evidence gaps. We sought to address both of these needs through a systematic review.

Methods

This review was planned, conducted, and reported in adherence to PRISMA standards of quality for reporting meta-analyses (Moher et al. 2009).

Questions

We sought to answer: what instructional design features are associated with improved outcomes in studies directly comparing one technology-enhanced simulation training approach with another, and what themes have been addressed in such comparisons? To answer the first question we selected eight instructional design features identified in the review by Issenberg et al. (2005) and additional features of cognitive interactivity, distributing training across multiple sessions, group vs independent practice, and time spent learning (see Box 1 for definitions). We hypothesized that outcomes would be higher with more of each feature.

Box 1 Definitions of terms*

Participants

Health professions learner: a student, postgraduate trainee, or practitioner in a profession directly related to human or animal health, including physicians, dentists, nurses, veterinarians, physical, occupational, and respiratory therapists, and emergency medical technicians and other first responders.

Instructional design key features

Clinical variation: Variation in the clinical context, for example multiple different patient scenarios (absent if no clinically relevant context was stated).

Cognitive interactivity: Training that promotes learners’ cognitive engagement using strategies such as multiple repetitions, feedback, task variation, or intentional task sequencing.

Curricular integration: Incorporation of the simulation intervention as an integral part (required or formal element) of the curriculum or training program.

Distributed practice: Training spread over a period of time. For this review, we counted this as present for interventions that involved >1 day of simulation training.

Feedback: Information on performance provided to the learner by the instructor, a peer, or a computer, either during or after the simulation activity.

Group (vs independent) practice: Training activities involving two or more learners (as compared with training alone).

Individualized learning: Training responsive to individual learner needs (i.e. tailored or adapted depending on performance).

Mastery learning: Training model in which learners must attain a clearly-defined standard of performance before qualifying or advancing to the next task.

Multiple learning strategies: The number of different instructional strategies used to facilitate learning, such as patient case, worked example, discussion, feedback, intentional sequencing, or task variation.

Range of task difficulty: Variation in the difficulty or complexity of the task (explicitly stated).

Repetitive practice: The opportunity for more than one task performance.

Outcomes

Satisfaction. Learners’ reported satisfaction with the course.

Knowledge: Subjective (e.g. learner self-report) or objective (e.g. multiple-choice question knowledge test) assessments of factual or conceptual understanding.

Skills: Subjective (e.g. learner self-report) or objective (e.g. faculty ratings, or objective tests of clinical skills such as computer-scored technique in a virtual reality surgery simulator, or number of masses detected when examining a breast model) assessments of learners’ ability to demonstrate a procedure or technique in an educational setting (typically a simulation task). We further classified skills as measures of time (how long it takes a learner to complete the task), process (e.g. global rating scales, efficiency, or minor errors), and product (successful completion of the task, evaluation of the finished product, or major errors that would impact a real patient's well-being). For purposes of meta-analysis we combined process and product skills into a single outcome, non-time skills.

Behaviors and patient effects: Subjective (e.g. learner or patient self-report) or objective (e.g. chart audit or faculty ratings) assessments of behaviors in practice (such as test ordering) or effects on patients (such as medical errors). We used a classification system similar to that used for Skills, with time and process measures being counted as behaviors (e.g. procedure time, test ordering, or interviewing technique with real patients) and products being counted as patient effects (e.g. complications, patient discomfort, or procedure completion rates).

Note: *Some of these definitions were published as an online appendix to a previous publication (Cook et al. 2011).

Study eligibility

We included studies published in any language that investigated use of technology-enhanced simulation to teach health professions learners at any stage in training or practice, in comparison with another technology-enhanced simulation design or a variation or augmentation of the first, using outcomes (Kirkpatrick 1996) of reaction (satisfaction), learning (knowledge or skills in a test setting), behaviors (in practice), or effects on patients (see Box 1). Technology-enhanced simulation encompasses diverse products including computer-based virtual reality simulators, high fidelity and static mannequins, plastic models, live animals, inert animal products, and human cadavers. Because they have been the subject of recent reviews, we excluded studies in which the only simulation activities involved computer-based virtual patients (Cook & Triola 2009; Cook et al. 2010a) and human patient actors (standardized patients) (Bokken et al. 2008; May et al. 2009).

Study identification

We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, and Scopus using a search strategy developed by an experienced research librarian (PJE). The search included terms for the intervention (including simulator, simulation, manikin, cadaver, MIST, Harvey, and many others), topic (surgery, endoscopy, anesthesia, trauma, colonoscopy, etc.), and learners (education medical, education nursing, education professional, internship and residency, etc.). We used no beginning date cutoff, and the last date of search was May 11, 2011. In addition, we added all articles published in two journals devoted to health professions simulation (Simulation in Healthcare and Clinical Simulation in Nursing) since their inception, and the entire reference list from several published reviews of health professions simulation. Finally, we searched for additional studies in the reference lists of all included articles published before 1990 and a random sample of 160 included articles published in or after 1990. Our complete search strategy has been published previously (Cook et al. 2011).

Study selection

Working independently and in duplicate, we screened all titles and abstracts for inclusion. In the event of disagreement or insufficient information in the abstract we reviewed the full text of potential articles, again independently and in duplicate, resolving conflicts by consensus. Chance-adjusted interrater agreement for study inclusion, determined using intraclass correlation coefficient (ICC), was 0.69. Non-English articles were translated in full.

Data extraction

Using a data abstraction form we abstracted data independently and in duplicate for all variables where reviewer judgment was required, resolving conflicts by consensus. Interrater agreement was fair (ICC 0.2–0.4) or moderate (0.4–0.6) for most variables (Landis & Koch 1977). We identified the main theme of each comparison (research question, study hypothesis) using an inductive, iterative approach. We abstracted information on the training level of learners, clinical topic, training location (simulation center or clinical environment), study design, method of group assignment, outcomes, and methodological quality. We planned to abstract information on simulation fidelity but dropped this variable due to difficulty operationalizing it with acceptable reliability. We coded simulation features (see Box 1) of:

• clinical variation (present/absent; ICC, 0.46),

• cognitive interactivity (high/low; ICC, 0.35),

• curriculum integration (present/absent; ICC, 0.49),

• distributed practice (training on 1 or >1 day; ICC, 0.73),

• feedback (high/low; ICC, 0.46),

• group vs independent practice (ICC, 0.71),

• individualized learning (present/absent; ICC, 0.25, with raw agreement 85%),

• mastery learning (Issenberg's “defined outcomes,” i.e. training to a predefined level of proficiency, present/absent; ICC, 0.53),

• multiple learning strategies (high/low; ICC, 0.49),

• range of task difficulty (present/absent; ICC, 0.30, with raw agreement 82%),

• repetitive practice (number of repetitions; ICC, 0.60), and

• time spent learning (ICC, 0.72).

Table 1.  Description of included studies

Study Characteristic	Level	No. studies	No. participants*
All studies		289	18 971
Group allocation	Randomized	208	12 227
Study design	Posttest-only	162	12 167
	Pretest-posttest	127	6804
Location	Simulation center	269	16 785
	Clinical environment	13	1421
	Both simulation center and clinical	7	765
Participants^†	Medical students	111	4894
	Physicians postgraduate training	87	2408
	Physicians in practice	39	1684
	Nurses and nursing students	42	4177
	Emergency medical technicians and students	18	824
	Dentists and dental students	22	1348
	Veterinarians and veterinary students	6	276
	Chiropractors and chiropractic students	2	104
	Other/ambiguous/mixed	43	3256
Clinical topics^‡	Minimally invasive surgery	79	3000
	Resuscitation/trauma training	51	6630
	Other surgery	34	1530
	Dental	21	1416
	Intubation	20	1922
	Physical examination	18	933
	Endoscopy and ureteroscopy	18	827
	Vascular access	15	605
	Communication and team skills	11	485
	Anesthesia	8	260
	Obstetrics	6	641
Key features	Clinical variation^§, present	64	4826
	Cognitive interactivity, high	193	13 339
	Curriculum integration^§, present	52	5395
	Distributed practice, >1 day	98	4745
	Feedback^§, high	75	6245
	Group practice	72	6317
	Individualized learning^§, high	18	1545
	Mastery learning, present^§	23	1050
	Multiple learning strategies^§, high	34	3102
	Range of task difficulty^§, present	40	1593
	Repetitive practice^§, present	233	15 521
Outcomes^†	Satisfaction	56	3042
	Knowledge	34	2687
	Skill: Time	100	4198
	Skill: Process	197	9983
	Skill: Product	56	2976
	Behavior: Time	1	28
	Behavior: Process	9	275
	Patient effects	8	422
Quality	Newcastle-Ottawa ≥4 points	160	10 888
	MERSQI ≥12.5 points	173	11 459

Notes: See Appendix 1 for details on individual studies.

*Numbers reflect the number enrolled, except for Outcomes which reflect the number of participants who provided observations for analysis.

^†The number of studies and learners in some subgroups may sum to more than the number for all studies, and percentages may total more than 100%, because several studies included >1 learner group, fit within >1 clinical topic, or reported multiple outcomes.

^‡Selected listing of the most frequently addressed topics (numerous other topics were addressed, with lower frequency). Some interventions addressed >1 clinical topic.

^§Features identified by Issenberg et al. (Issenberg et al., 2005) Mastery learning is similar to Issenberg's “defined outcomes.”

Methodological quality was graded using the Medical Education Research Study Quality Instrument (Reed et al. 2007) and an adaptation of the Newcastle-Ottawa scale for cohort studies (Wells et al. 2007; Cook et al. 2008b) that evaluates representativeness of the intervention group (ICC, 0.68), selection of the comparison group (ICC, 0.26 with raw agreement 86%), comparability of cohorts (statistical adjustment for baseline characteristics in nonrandomized studies [ICC, 0.88], or randomization [ICC, 0.84] and allocation concealment for randomized studies [ICC, 0.63]), blinding of outcome assessment (ICC, 0.58), and completeness of follow-up (ICC, 0.36 with raw agreement 80%).

Since the results associated with simulation training may vary for different outcomes, we distinguished outcomes using Kirkpatrick's classification (Kirkpatrick 1996) and abstracted information separately for satisfaction, learning (knowledge and skills, with skills further classified as time to complete the task, process, and product [see Box 1 for definitions]), behaviors with patients (time and process), and results (patient effects). Authors frequently reported multiple measures of a single outcome (e.g. multiple measures of process skill), in which case we selected, in order of priority, (1) the author-defined primary outcome, (2) a global or summary measure of effect, (3) the most clinically relevant measure, or (4) the average of the measures reported. We also prioritized skill outcomes assessed in a different setting (e.g. different simulator or clinical setting) over those assessed in the simulator used for training.

Data synthesis

For each reported outcome we calculated the standardized mean difference (Hedges’ g effect size) between each group using standard techniques (Borenstein 2009; Morris & DeShon 2002; Curtin et al. 2002; Hunter & Schmidt 2004) as we have detailed previously. (Cook et al. 2011) For studies reporting neither p values nor any measure of variance, we used the average standard deviation from all other studies reporting that outcome. If we could not calculate an effect size using reported data we requested additional information from authors via e-mail.

We used the I² statistic (Higgins et al. 2003) to quantify inconsistency (heterogeneity) across studies. I² estimates the percentage of variability across studies not due to chance, and values >50% indicate large inconsistency. Large inconsistency weakens the inferences that can be drawn, but does not preclude the pooling of studies sharing a common conceptual link.

We planned meta-analyses to evaluate the effectiveness of each instructional design feature, pooling the results of all studies for which that feature varied between two simulation-based interventions. For example, if one study group received high feedback and the other low feedback, this study would be included in the “Feedback” meta-analysis. If feedback were equal in both arms, it would be excluded from that analysis. To increase the power of these analyses, we merged process and product skills into a single outcome of “non-time skills,” and we also combined behaviors and patient effects. Because we found large inconsistency in most analyses, we used random effects models to pool weighted effect sizes. Many studies appear in >1 analysis (i.e. both feedback and repetitive practice) but no study appeared more than once per analysis. For studies with >2 groups (for example, three different simulation instructional designs), we selected for the main analysis the designs with the greatest between-group difference, and then performed sensitivity analyses substituting the other design(s). We also performed sensitivity analyses excluding low-quality studies (those with NOS and MERSQI scores below the median) and studies with imprecise effect size estimation (p value upper limits or imputed standard deviations).

We used SAS 9.2 (SAS Institute, Cary, NC) for all analyses. Statistical significance was defined by a two-sided alpha of 0.05. Determinations of educational significance emphasized Cohen's effect size classifications (<0.2 = negligible; 0.2–0.49 = small; 0.5–0.8 = moderate) (Cohen, 1988).

Results

Trial flow

We identified 10 297 articles using our search strategy and 606 from our review of reference lists and journal indices. From these we identified 295 studies comparing two or more simulation training interventions (Figure 1) of which 290 reported eligible outcomes. Two articles reported the same data and we selected the most detailed one for inclusion. We obtained additional outcomes data for 1 study from study authors. Ultimately, we included 289 studies enrolling 18 971 trainees. Twenty-six of these 289 were multi-arm studies that included a comparison with no-intervention, and the no-intervention results were reported previously (Cook et al. 2011). Table 1 summarizes key study characteristics and Appendix 1 provides a complete listing of articles with additional information.

Figure 1. Trial flow.

Study characteristics

Studies in our sample used technology-enhanced simulations to teach topics such as minimally invasive surgery, dentistry, intubation, physical examination, and teamwork. Nearly half the articles (N = 139) were published in or after 2008, and five were published in a language other than English. Learners included student and practicing physicians, nurses, emergency medicine technicians, dentists, chiropractors, and veterinarians, among others. Table 1 summarizes the prevalence of instructional design features such as feedback (75 studies), repetitive practice (233 studies), and distributed practice (98 studies). Most studies reported learner skills, including 100 time, 197 process, and 56 product skill outcomes. Fifty-six studies reported satisfaction, 34 reported knowledge outcomes, 1 reported time behavior, 9 reported process behavior, and 8 reported patient effects.

Study quality

Table 2 summarizes the methodological quality of included studies. The number of participants providing outcomes ranged from 4 to 817 with a median of 30 (interquartile range 20–53). Groups were randomly assigned in 208 studies (72%). Studies lost more than 25% of participants from time of enrollment or failed to report follow-up for 13 of 56 satisfaction outcomes (23%), 5 of 34 knowledge (15%), 31 of 100 time skill (31%), 62 of 197 process skill (31%), 18 of 56 product skill (32%), and 1 of 9 process behavior (11%) (time behavior and patient effect outcomes had complete follow-up). Assessors were blinded to group assignment for 309 of 461 outcome measures (67%). Most outcomes reflect objective measures (e.g. computer scoring, objective key, or human rater). All knowledge and time behavior outcomes were determined objectively, while trainee self-assessments comprised five process skill outcomes and one each of time skill, product skill, process behavior, and patient effect outcomes. The mean (SD) quality scores averaged 3.5 (1.3) for the Newcastle-Ottawa Scale (6 points indicating highest quality) and 12.3 (1.8) for the Medical Education Research Study Quality Instrument (maximum 18 points).

Table 2.  Quality of included studies

Scale Item	Subscale (points if present)	No. (%) present; N = 289
Medical Education Research Study Quality Instrument (MERSQI)*
Study design (maximum 3)	Observational 2-group (2)	81 (28)
	Randomized 2-group (3)	208 (72)
Sampling: No. institutions (maximum 1.5)	1 (0.5)	255 (88)
	2 (1)	10 (4)
	>2 (1.5)	24 (8)
Sampling: Follow-up (maximum 1.5)	<50% or not reported (0.5)	81 (28)
	50–74% (1)	9 (3)
	≥75% (1.5)	199 (69)
Type of data: Outcome assessment (maximum 3)	Subjective (1)	38 (13)
	Objective (3)	251 (87)
Validity evidence (maximum 3)	Content (1)	83 (29)
	Internal structure (1)	78 (27)
	Relations to other variables (1)	27 (9)
Data analysis: appropriate (maximum 1)	Appropriate (1)	260 (90)
Data analysis: sophistication (maximum 2)	Descriptive (1)	13 (5)
	Beyond descriptive analysis (2)	276 (95)
Highest outcome type (maximum 3)	Satisfaction (1)	25 (9)
	Knowledge, skills (1.5)	249 (86)
	Behaviors (2)	7 (2)
	Patient/health care outcomes (3)	8 (3)
Newcastle-Ottawa Scale (modified)^†
Representativeness of sample	Present (1)	53 (18)
Comparison group from same community	Present (1)	268 (93)
Comparability of comparison cohort, criterion A^‡	Present (1)	213 (74)
Comparability of comparison cohort, criterion B^‡	Present (1)	106 (37)
Blinded outcome assessment	Present (1)	175 (61)
Follow-up high^‡	Present (1)	208 (72)

Notes: *MERSQI total score (maximum 18): mean 12.3 (SD 1.8), median 12.5 (range 6.5–16).

^†NOS total score (maximum 6): mean 3.5 (SD 1.3), median 4 (range 1–6).

^‡Comparability of cohorts criterion A was present if the study (a) was randomized, or (b) controlled for a baseline learning outcome; criterion B was present if (a) a randomized study concealed allocation, or (b) an observational study controlled for another baseline learner characteristic. Follow-up was high if ≥75% of those enrolled provided outcome data, or if authors described those lost to follow-up.

Table 3.  Research themes (comparisons) addressed by studies

Theme, No. (%)*	Theme definition	Sub-theme	No. studies (% of theme)*
Group composition, N = 2 (1%)	Compare different approaches to grouping learners.	Interdisciplinary vs single-discipline group	1 (50%)
		Independent vs group	1 (50%)
Instructional design, N = 133 (39%)	Compare different design features to enhance instructional effectiveness.	Feedback	47 (35.3%)
		Sequence	24 (18%)
		Teach cognitive or mental imagery techniques	12 (9%)
		Repetition	10 (7.5%)
		Task variability	9 (6.8%)
		Clinical scenario	5 (3.8%)
		Hands on practice	5 (3.8%)
		Timing	4 (3%)
		Instructions	3 (2.3%)
		Stress	3 (2.3%)
		Additional practice	2 (1.5%)
		Blending simulation and non-simulation instruction	2 (1.5%)
		Interactivity	2 (1.5%)
		Mastery	2 (1.5%)
		Testing effect	2 (1.5%)
		Add reminder	1 (0.8%)
Instructor, N = 27 (8%)	Compare different levels of instructor training or presence.	Self-instruction	17 (63%)
		Instructor intensity	5 (18.5%)
		Instructor training/experience	3 (11.1%)
		Distance supervision	2 (7.4%)
Modality added, N = 37 (11%)	Evaluate the addition of one or more other modalities (eg, lecture, CAI, or other simulation) to baseline simulation training	CAI, add or compare	9 (24.3%)
		Simulator, add	9 (24.3%)
		Patient/SP experience, add or compare	7 (18.9%)
		Discussion (excluding debriefing), add or compare	3 (8.1%)
		External support, add	2 (5.4%)
		Robot assistance, add	2 (5.4%)
		Team training, add	2 (5.4%)
		Lecture, add or compare	2 (5.4%)
		Tutor, add	1 (2.7%)
Sensory augmentation, N = 19 (6%)	Evaluate the addition of a feature or effect to enhance sensory experience.	Tactile (includes haptics)	11 (57.9%)
		Visual	7 (36.8%)
		Olfactory	1 (5.3%)
Simulation modalities compared, N = 119 (35%)	Compare two technology-enhanced simulation modalities	VR vs model	40 (33.6%)
		Mannequin vs mannequin	16 (13.4%)
		Model vs model	11 (9.2%)
		Mannequin vs model	9 (7.6%)
		Model vs animal	9 (7.6%)
		VR vs VR	8 (6.7%)
		Mannequin vs cadaver	5 (4.2%)
		Model vs cadaver	4 (3.4%)
		VR vs animal	4 (3.4%)
		Mannequin vs animal	3 (2.5%)
		Synthetic vs human/animal products	3 (2.5%)
		VR vs cadaver	2 (1.7%)
		VR vs mannequin	2 (1.7%)
		Animal living vs dead	1 (0.8%)
		Animal products vs animal products	1 (0.8%)
		Cadaver vs animal	1 (0.8%)

Notes: *The 289 studies addressed a total of 337 research themes (38 studies had 2 or 3 arms). Percentages are for sub-themes within a theme.

Abbreviations: CAI = computer-assisted instruction; SP = standardized patient; VR = virtual reality.

Meta-analysis

For meta-analysis we merged process and product skills into a single outcome of “non-time skills,” and we likewise merged behaviors and patient effects. Figure 2 shows the pooled effect size for each instructional design feature, organized by outcome (panels A–E). For non-time skills we confirmed small to moderate positive effects favoring the presence of each proposed feature of effective simulation except group training, and most (7 of 11) effects were statistically significant. Results for other outcomes nearly always (35 of 38) favored the proposed feature, but results were usually not statistically significant.

Figure 2. Random effects meta-analysis Comparisons of simulation interventions varying in the specified key feature; positive numbers favor the intervention with more of that feature. If a feature is not present in the analysis for a given outcome, it is because no studies reporting that outcome had interventions that varied for that particular feature. Some feature comparisons had only 1 relevant study; these are included in the figures but we note that the effect size reflects only that 1 study (ie, not pooled).

For example, for the non-time skill outcomes (Figure 2, panel D), 20 studies reported a comparison in which one simulation design included tasks reflecting a range of difficulty and the other did not. Among these studies, designs offering a range of difficulty were associated with better outcomes than those of uniform difficulty, with pooled effect size (ES) 0.68 (95% confidence interval [CI], 0.30–1.06, p < 0.001). This difference is statistically significant, and moderate in magnitude using Cohen's classification. Further differences of small to moderate magnitude were found for instructional designs incorporating clinical variation (0.20), more interactivity (0.65), training over >1 day (0.66), more feedback (0.44), individualization (0.52), mastery learning (0.45), more learning strategies (0.62), repetition (ES 0.68), and longer time (0.34). Findings for knowledge, time, and behavior-patient effect outcomes were similarly favorable, but with smaller and usually statistically non-significant effects (see Figure 2). Inconsistency was large (I² > 50%) in most analyses.

The exception to the predicted pattern was group training, which showed a small negative association with non-time skills (ES −0.22 [95% CI, −0.48 to 0.03], p = 0.09). Knowledge and time outcomes (one study each) showed similar results.

Several studies reporting non-time skills and behavior-patient effects had three simulation arms. Since we could only compare two groups at once, we first included the groups with the greatest between-design difference and then performed sensitivity analyses substituting the third group (see Appendix 2). Results changed almost imperceptibly for non-time skills: pooled effect sizes varied by <0.08 in all analyses, and statistical significance changed in only one instance (the group practice analysis was now statistically significant, p = 0.02). For behavior–patient effects the pooled ES for feedback dropped to 0.18.

Additional sensitivity analyses excluded low-quality studies. The direction of effect reversed only rarely (5 of 153 analyses), namely: mastery learning (non-time skill outcomes) when excluding imprecise effect size estimation or low NOS score (N = 4 studies remaining for each analysis); feedback (satisfaction outcomes) when excluding low MERSQI or NOS score (N = 2 for each); and interactivity (behavior–patient effects) when excluding low MERSQI scores (N = 2).

Research themes

Through an iterative process we identified six main research themes, with a much larger number of sub-themes (Table 3). Thirty-eight studies had two or three arms each, resulting in a total of 337 comparisons. The most prevalent main theme involved comparisons of different instructional design features such as the amount or method of feedback, the sequence of training activities, task variability, or repetition. Second most common were studies that compared two technology-enhanced simulation modalities, such as mannequin vs part-task model, mannequin vs virtual reality, or two different mannequins. Several studies evaluated the addition of another instructional modality (e.g. a lecture, computer-assisted instruction, or another simulation modality) to the standard simulation training. The remaining themes focused on the role of the instructor, sensory augmentation including haptics, and group composition. We initially identified a main theme of “fidelity,” but upon further reflection realized that all of the studies thus classified could be more appropriately classified using another theme, most often that of “modality comparison.”

Discussion

In 2005, Issenberg et al. proposed 10 features of effective simulation based on prevalence in the literature. Our synthesis of research provides empiric support for nearly all of these features and several others. Although the pooled effect sizes were often small and not statistically significant, and between-study inconsistency was high, the consistent direction of effect across outcomes suggests that the benefits are real. To put these findings in perspective, the effect size of 0.68 observed for “range of difficulty” using non-time skill outcomes would translate to a 5% improvement (out of 100%) on a typical skill assessment. Effect sizes were generally larger for skills than for knowledge, a tendency we also observed among studies comparing simulation with non-simulation instruction (Cook et al. 2012). Of the twelve features evaluated, only group instruction failed to demonstrate consistently positive effects. Interestingly, in the previous meta-analysis (Cook et al. 2012) of studies comparing simulation with non-simulation instruction, group instruction was associated with improved outcomes; this incongruity merits further study.

We also classified the research themes for 337 simulation–simulation comparisons. The most prevalent theme involved evaluating key features of instructional design. These studies, along with those exploring instructor roles and group composition, typically allowed generalizable conclusions. By contrast, another one-third of the themes focused on comparing different simulation modalities. While modality comparisons initially appear useful, we noted that the results varied widely as technologies changed and evolved, the educational context varied, and different implementations of the same technology employed different instructional designs. As a result, we suspect the findings from modality comparisons will have limited generalizability.

One design feature from Issenberg et al.'s review that we did not code was fidelity. We found fidelity difficult to code, during both the quantitative data abstraction and the thematic analysis. We found that “fidelity” encompasses a number of different facets related to the simulation activity, including the characteristics of the simulator that mediate sensory impressions (visual, auditory, olfactory, and tactile/haptic), the nature of the learning objectives and task demands, the environment, and other factors that might affect learner engagement and suspension of disbelief. Labeling a simulation as “high fidelity” conveys such diverse potential meanings that the term loses nearly all usefulness. Based on our experiences during this review, we suggest that researchers and educators employ more specific terminology when discussing the physical and contextual attributes of simulation training.

Limitations and strengths

In order to present a comprehensive thematic overview of the field and achieve adequate statistical power for meta-analyses, we used intentionally broad inclusion criteria. However, in so doing we included studies reflecting diverse training topics, instructional designs, and outcome measures. These differences likely contributed to the large between-study inconsistency. This inconsistency tempers our inferences, but does not preclude meta-analytic pooling (Montori et al. 2003; Cook 2012b). Future original research and research syntheses might further clarify the importance of these instructional design features for specific topics, such as technical and non-technical tasks.

Literature reviews are necessarily constrained by the quantity and quality of available evidence. Among the included studies, sample sizes were relatively small, sample representativeness was rarely addressed, outcome validity evidence was infrequently presented, and many reports failed to clearly describe key features of the context, instructional design, or outcomes. Although we found numerous studies reporting skill outcomes and several reporting satisfaction and knowledge, we found few studies reporting higher-order outcomes of behavior and patient effects. However, more than 70% of the studies used randomization, and MERSQI scores were substantially higher than those found in previous reviews of medical education research (Reed et al. 2007, 2008).

Coding reproducibility was suboptimal for some instructional design features, likely due to both poor reporting and difficulty operationalizing coding criteria. However, we reached consensus on all codes prior to meta-analysis.

To increase statistical power and to reduce the number of independent meta-analyses, we combined process and product outcomes for assessments in both an education setting (skills) and with real patients (behaviors and patient effects). Analyzing these separately might have led to slightly different conclusions.

Our review has several additional strengths, including an extensive literature search led by a skilled librarian; no restriction based on time or language of publication; explicit inclusion criteria encompassing a broad range of learners, outcomes, and study designs; duplicate, independent, and reproducible data abstraction; rigorous coding of methodological quality; and hypothesis-driven analyses.

Comparison with previous reviews

The present review complements our recent meta-analysis showing that simulation training is associated with large positive effects in comparison with no intervention (Cook et al. 2011). Having established that simulation can be effective, the next step is to understand what makes it effective. Although several other reviews have addressed simulation in general (Issenberg et al. 2005; McGaghie et al. 2010) or in comparison with no intervention, (Gurusamy et al. 2008; McGaghie et al. 2011), we are not aware of previous reviews focused on comparisons of different technology-enhanced simulation interventions or instructional designs. By confirming the effectiveness of the design features proposed by Issenberg et al. (2005), our comprehensive and quantitative synthesis represents a novel and important contribution to the field.

Our findings of small to moderate effects favoring theory-predicted instructional design features parallel the findings of a review of Internet-based instruction (Cook et al. 2008b). An association between longer time in practice and improved outcomes was also reported in a previous review of simulation-based education (McGaghie et al. 2006).

Implications

The features proposed by Issenberg et al. (2005) as central to effective simulation appear to work, as do the additional features we identified. We recommend that these be considered the current “best practices” for the field. In order of pooled effect size, these are: range of difficulty, repetitive practice, distributed practice, cognitive interactivity, multiple learning strategies, individualized learning, mastery learning, feedback, longer time, and clinical variation.

However, we simultaneously highlight the need for further research elucidating what works, for whom, under what circumstances. The large inconsistency observed in nearly all analyses indicates that the effect varies from study to study, and the relative contribution of multiple potentially influential variables (learners, environment, operational definition of interventions, outcomes, and other study methods) remains unclear. Besides meta-analysis, other synthesis methods such as realist review (Pawson et al. 2005) will help interpret existing evidence.

Going forward, we believe that a fundamental change in the conception and design of new research is required. To-date, the number of studies attempting to clarify the use of simulation by directly comparing different simulation-based interventions is small (N = 289) relative to the number of studies comparing simulation with no intervention or non-simulation instruction (N = 690 [see Figure 1]) and studies without any comparison (N = 864). Studies comparing simulation with simulation will do far more to advance the field than comparisons of simulation with non-simulation approaches. (Cook 2010) Yet not all simulation–simulation comparisons are equally useful, and studies evaluating modalities or instructional designs without a conceptual or theoretical rationale have limited generalizability.

The field thus needs research that goes beyond simple comparisons of presence/absence of key features (Weinger, 2010). For example, it appears that feedback improves outcomes—but we expect that much could yet be learned about the basis, timing, and delivery of feedback. This research will require progressively refined theories and conceptual frameworks that programmatically study carefully constructed questions (Bordage 2009; McGaghie et al. 2010). The themes identified in this review (see Table 3) provide a starting point for such research programs. It will also be important to systematically account the costs of alternate instructional approaches (Levin 2001), and explore how costs can inform design decisions (Zendejas 2012).

Table A1.  List of all included studies

	Participants				Research comparisons			Quality
Citation (sorted by year then author)	Trainee	N	RCT	Topic	Features	Theme	Outcomes	MERSQI	NOS
Wittrock JW, et al. Use of different sized models in teaching restorative dentistry. J Dent Educ. 1970; 34:76–82.	D	86		Dentistry		SA:Visual	SPd	10.5	2
Johnson GH, et al. Teaching pelvic examination to second-year medical students using programmed patients. Am J Obstet Gynecol. 1975; 121:714–717.	MS	140		Physical exam	ClinV, CogI, FB, Indiv, MLS	MA:Patient/SP experience, add	Sa	9	2
Salvendy G, et al. A second generation training simulator for acquisition of psychomotor skills in cavity preparation. J Dent Educ. 1975; 39:466–471.	D	30	RCT	Dentistry	CogI, FB, GrpP, Indiv	CM:VR vs model	ST, SPc	13.5	3
Holzman GB, et al. Initial pelvic examination instruction: The effectiveness of three contemporary approaches. Am J Obstet Gynecol. 1977; 129:124–129.	MS	28		Physical exam	Time	MA:Tutor, add	K, SPc	12.5	3
Lefcoe DL, et al. Simulated models: a mode for instruction in root planing procedures. Educ Dir Dent Aux. 1979; 3:20–4.	O	12	RCT	Dental cleaning		SA:Visual	P	13.5	5
Herrin TJ, et al. Modular approach to CPR training. South Med J. 1980; 73:742–744.	MS, RN	303		Resuscitation (BLS,ACLS,ATLS)	FB, Indiv	In:Self-instruction	K	11.5	3
Salvendy G, et al. The development and validation of an analytical training program for medical suturing. Hum Factors. 1980; 22:153–170.	MS	18	RCT	Open surgery/suturing		ID:Teach cognition	ST, SPc	13.5	4
LaTurno SA, et al. An evaluation of a teaching aid in endodontics. J Endod. 1984; 10:507–511.	D	150		Dentistry	MLS	SA:Visual	SPd	10.5	2
Stewart RD, et al. Effect of varied training techniques on field endotracheal intubation success rates. Ann Emerg Med. 1984; 13:1032–1036.	EMT	94		Intubation	ClinV, CogI, Indiv, Mast, MLS, RangeD	MA:Patient/SP experience, add	P	12	4
Weiner S, et al. An evaluation of sequential models in the preclinical laboratory. J Dent Educ. 1985; 49:109–110.	D	142		Dentistry		ID:Sequence	SPd	10.5	2
Champagne MT, et al. Use of a heart sound simulator in teaching cardiac auscultation. Focus Crit Care. 1989; 16:448–456.	RN	37	RCT	Physical exam		SA:Tactile	SPc	15.5	3
Carpenter LG, et al. A comparison of surgical training with live anesthetized dogs and cadavers. Veterinary Surgery. 1991; 20:373–378.	V	12	RCT	Veterinary surgery		CM:Animal living vs dead	ST, SPc, SPd	12.5	6
Stratton SJ, et al. Prospective study of manikin-only versus manikin and human subject endotracheal intubation training of paramedics. Ann Emerg Med. 1991; 20:1314–1318.	EMT	60	RCT	Intubation	ClinV, Mast, RangeD	MA:Simulator, add	P	11	3
Trooskin SZ, et al. Teaching endotracheal intubation using animals and cadavers. Prehospital Disaster Med. 1992; 7:179–182.	EMT	19	RCT	Intubation	CogI, Indiv	CM:Manikin vs animal	SPd, P	14.5	4
Bhat BV, et al. Undergraduate training in neonatal resuscitation – a modified approach. Indian J Matern Child Health. 1993; 4:87–88.	MS	110		Resuscitation (BLS,ACLS,ATLS)	DistP	ID:Timing	K	11.5	4
Holmberg DL, et al. Use of a dog abdominal surrogate for teaching surgery. Journal of Veterinary Medical Education. 1993; 20:61–62.	V	116X		Veterinary surgery	ClinV	CM:Model vs animal	Sa	7.5	2
Mazzuca SA, et al. Improved training of house officers in a rheumatology consult service. Arthritis Care Res. 1993; 6:59–63.	PG	11		Physical exam	MLS	ID:Add reminder	BP	12	2
Campbell HS, et al. Teaching medical students how to perform a clinical breast examination. Acad Med. 1994; 69:993–995.	MS	54	RCT	Physical exam	CogI, FB, Indiv, MLS, Time	In:Instructor training/experience	SPd	11.5	3
From RP, et al. Assessment of an interactive learning system with “sensorized” manikin head for airway management instruction. Anesth Analg. 1994; 79:136–142.	MS	97	RCT	Intubation	MLS	In:Self-instruction	K, BP	13	6
Reader CM, et al. Anatomical artificial teeth for teaching preclinical endodontics. Journal Dental Education. 1994; 58:229–232.	D	29X		Dentistry		CM:Model vs cadaver	Sa, SPd	7.5	2
Greenfield CL, et al. Comparison of surgical skills of veterinary students trained using models or live animals. J Am Vet Med Assoc. 1995; 206:1840–1845.	V	36	RCT	Open surgery/suturing	ClinV	CM:Model vs animal	SPc	11.5	4
van Stralen DW, et al. Retrograde intubation training using a mannequin. Am J Emerg Med. 1995; 13:50–52.	RN, EMT, O	88		Intubation	FB, Indiv	In:Self-instruction	ST	11.5	2
Olsen D, et al. Evaluation of a hemostasis model for teaching basic surgical skills. Vet Surg. 1996; 25:49–58.	V	40	RCT	Open surgery/suturing	RepP	CM:Model vs animal	SPc, SPd	13.5	5
Thomas WE, et al. A preliminary evaluation of an innovative synthetic soft tissue simulation module (‘Skilltray’) for use in basic surgical skills workshops. Ann R Coll Surg Engl. 1996; 78:268–271.	PG	9X		Open surgery/suturing		CM:Synthetic vs human/animal products	Sa	7	2
Long NK, et al. A comparison of two teaching simulations in preclinical operative dentistry. Oper Dent. 1997; 22:133–137.	D	87	RCT	Dentistry		CM:Synthetic vs human/animal products	ST, SPd	14.5	6
Noordergraaf GJ, et al. Learning cardiopulmonary resuscitation skills: does the type of mannequin make a difference? Eur J Emerg Med. 1997; 4:204–209.	MS	104		Resuscitation (BLS,ACLS,ATLS)	FB, GrpP	CM:Manikin vs manikin	SPc, SPd	12.5	4
Roberts I, et al. Airway management training using the laryngeal mask airway: a comparison of two different training programmes. Resuscitation. 1997; 33:211–214.	RN	52	RCT	Intubation	ClinV, CogI, MLS, RangeD	MA:Patient/SP experience, compare	SPd	12.5	3
Ali J, et al. Effect of the Advanced Trauma Life Support program on medical students’ performance in simulated trauma patient management. Journal of Trauma: Injury Infection & Critical Care. 1998; 44:588–591.	MS	44		Resuscitation (BLS,ACLS,ATLS)	CogI, FB, Indiv, MLS, RepP	ID:Hands on	SPc	10.5	2
Christenson J, et al. A comparison of multimedia and standard advanced cardiac life support learning. Acad Emerg Med. 1998; 5:702–708.	MS	113		Resuscitation (BLS,ACLS,ATLS)	ClinV, CogI, FB, Indiv	MA:CAI, compare	K, SPc	13.5	4
Kaczorowski J, et al. Retention of neonatal resuscitation skills and knowledge: a randomized controlled trial. Fam Med. 1998; 30:705–711.	PG	27	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB, Indiv, MLS, Time	In:Self-instruction	K, SPc	12	6
Rogers DA, et al. Computer-assisted learning versus a lecture and feedback seminar for teaching a basic surgical technical skill. Am J Surg. 1998; 175:508–510.	MS	82	RCT	Open surgery/suturing	CogI, FB, MLS	MA:CAI, compare	ST, SPc, SPd	13.5	5
Todd KH, et al. Randomized, controlled trial of video self-instruction versus traditional CPR training. Ann Emerg Med. 1998; 31:364–369.	MS	89	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB, GrpP, Indiv, Time	In:Self-instruction	K, SPc	13.5	6
Ali J, et al. Comparison of performance of interns completing the old (1993) and new interactive (1997) Advanced Trauma Life Support courses. Journal of Trauma-Injury Infection & Critical Care. 1999; 46:80–86.	PG	32	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, MLS	ID:Interactivity	Sa, K, SPc	12.5	4
Anastakis DJ, et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg. 1999; 177:167–170.	PG	23X	RCT	Open surgery/suturing		CM:Model vs cadaver	SPc	12.5	5
Burdea G, et al. Virtual reality-based training for the diagnosis of prostate cancer. IEEE Trans Biomed Eng. 1999; 46:1253–1260.	PG	8		Physical exam	CogI, FB	CM:VR vs model	SPd	6.5	1
Gallagher AG, et al. Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST VR). Endoscopy. 1999; 31:310–313.	O	16	RCT	Min. invasive surg.	CogI, DistP, FB, Indiv, RangeD	CM:VR vs model	SPc	12.5	3
Griffon DJ, et al. Evaluation of a hemostasis model for teaching ovariohysterectomy in veterinary surgery. Vet Surg. 2000; 29:309–316.	V	40	RCT	Open surgery/suturing	RepP	CM:Model vs cadaver	K, ST, SPc	13.5	6
Jordan J-A, et al. A comparison between randomly alternating imaging, normal laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition. Am J Surg. 2000; 180:208–211.	O	16	RCT	Min. invasive surg.	CogI, FB, Indiv, RangeD	CM:VR vs model	SPc	10.5	3
Keyser EJ, et al. A simplified simulator for the training and evaluation of laparoscopic skills. Surg Endosc. 2000; 14:149–153.	PG	22X	RCT	Min. invasive surg.		CM:Model vs model	SPc	12.5	4
Kovacs G, et al. A randomized controlled trial on the effect of educational interventions in promoting airway management skill maintenance. Ann Emerg Med. 2000; 36:301–309.	MS, D, O	53	RCT	Intubation	CogI, FB, Indiv	ID:Repetition	SPc	14.5	5
Jordan JA, et al. Virtual reality training leads to faster adaptation to the novel psychomotor restrictions encountered by laparoscopic surgeons. Surg Endosc. 2001; 15:1080–1084.	MS, PG, O	24	RCT	Min. invasive surg.	CogI, FB, Indiv, RangeD	CM:VR vs model	SPc	13.5	4
Pugh CM, et al. The effect of simulator use on learning and self-assessment: The case of Stanford University's E-Pelvis simulator. Stud Health Technol Inform. 2001; 81:396–400.	MS	59	RCT	Physical exam	CogI	ID:Feedback	SPc	11.5	3
Risucci D, et al. The Effects of Practice and Instruction on Speed and Accuracy during Resident Acquisition of Simulated Laparoscopic Skills. Curr Surg. 2001; 58:230–235.	PG	14	RCT	Min. invasive surg.	CogI, Indiv, MLS	In:Instructor intensity	ST, SPc	11.5	3
Torkington J, et al. Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc. 2001; 15:1076–1079.	MS	20	RCT	Min. invasive surg.	CogI, FB, Indiv	CM:VR vs model	ST, SPc	12.5	4
Agazio JB, et al. Evaluation of a virtual reality simulator in sustainment training. Mil Med. 2002; 167:893–897.	MD, RN, EMT	51	RCT	Venous access	ClinV, CogI, FB, Indiv	CM:VR vs model	Sa, ST, SPc	11.5	3
Block EF, et al. Use of a human patient simulator for the advanced trauma life support course. Am Surg. 2002; 68:648–651.	O	14X		Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs animal	Sa	7	2
Byrne AJ, et al. Effect of videotape feedback on anaesthetists’ performance while managing simulated anaesthetic crises: a multicentre study. Anaesthesia. 2002; 57:176–179.	PG	32	RCT	Anesthesia	CogI, Indiv, MLS	ID:Feedback	ST, SPc	11.5	4
Chang KK, et al. Learning intravenous cannulation: a comparison of the conventional method and the CathSim Intravenous Training System. J Clin Nurs. 2002; 11:73–78.	RN, RN	28	RCT	Venous access	FB	CM:VR vs model	BP, P	16	4
Clancy JM, et al. A comparison of student performance in a simulation clinic and a traditional laboratory environment: three-year results. J Dent Educ. 2002; 66:1331–1337.	D	186X		Dentistry		CM:Manikin vs model	SPd	12.5	4
Hamilton EC, et al. Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc. 2002; 16:406–411.	PG	19	RCT	Min. invasive surg.	CogI, FB, Indiv, RangeD	CM:VR vs model	Sa, ST, SPc, BP	14	5
Harold KL, et al. Prospective randomized evaluation of surgical resident proficiency with laparoscopic suturing after course instruction. Surg Endosc. 2002; 16:1729–1731.	PG	17	RCT	Min. invasive surg.	CogI, FB, MLS	In:Self-instruction	ST, SPc	11.5	3
Kothari SN, et al. Training in laparoscopic suturing skills using a new computer-based virtual reality simulator (MIST-VR) provides results comparable to those with an established pelvic trainer system. J Laparoendosc Adv Surg Tech. 2002; 12:167–173.	MS	24	RCT	Min. invasive surg.	CogI, FB, Indiv, RangeD	CM:VR vs model	ST	12.5	4
Matsumoto ED, et al. The effect of bench model fidelity on endourological skills: a randomized controlled study. J Urol. 2002; 167:1243–1247.	MS	33	RCT	Endoscopy (GI,Urology,Bronch.)		CM:Manikin vs model	ST, SPc, SPd	14.5	5
Scaringe JG, et al. The effects of augmented sensory feedback precision on the acquisition and retention of a simulated chiropractic task. J Manipulative Physiol Ther. 2002; 25:34–41.	C	71	RCT	Physical exam	FB	ID:Feedback	SPc	11.5	4
Allen J, et al. A teaching tool in spinal anesthesia. AANA J. 2003; 71:29–36.	RN	26X	RCT	Percutaneous proc.	CogI, FB	ID:Feedback	ST, SPd	12.5	3
Ameur S, et al. Learning bronchoscopy in simulator improved dexterity rather than judgement [Swedish]. Lakartidningen. 2003; 100:2694–2699.	MS	20		Endoscopy (GI,Urology,Bronch.)	CogI	ID:Clinical scenario	Sa, SPc	10.5	1
Engum SA, et al. Intravenous catheter training system: computer-based education versus traditional learning methods. Am J Surg. 2003; 186:67–74.	MS, RN	163		Venous access	CogI, FB	CM:VR vs model	Sa, K, SPc	11.5	4
Gerling GJ, et al. Effect of augmented visual performance feedback on the effectiveness of clinical breast examination training with a dynamically configurable breast model. Conf Proc IEEE Int Conf Syst Man Cybern. 2003; 3:2095–2100.	MS	6		Physical exam	CogI, FB	ID:Feedback	SPc	11	2
Gerling GJ, et al. Effectiveness of a dynamic breast examination training model to improve clinical breast examination (CBE) skills. Cancer Detect Prev. 2003; 27:451–456.	MS	48	RCT	Physical exam	ClinV, CogI, MLS, RangeD	ID:Task variability	SPd	12.5	4
Gormley GJ, et al. A randomised study of two training programmes for general practitioners in the techniques of shoulder injection. Ann Rheum Dis. 2003; 62:1006–1009.	MD	38	RCT	Percutaneous proc.	ClinV, CogI, MLS, RangeD, Time	MA:Patient/SP experience, add	SPc, BP	12	4
Katz R, et al. Cadaveric versus porcine models in urological laparoscopic training. Urulogia Internationalis. 2003; 71:310–315.	PG, MD	16		Min. invasive surg.		CM:Cadaver vs animal	Sa	7	1
Lee SK, et al. Trauma assessment training with a patient simulator: a prospective, randomized study. Journal of Trauma: Injury Infection & Critical Care. 2003; 55:651–657.	PG	60	RCT	Resuscitation (BLS,ACLS,ATLS)		MA:Patient/SP experience, compare	SPc	14	3
Quinn F, et al. A pilot study comparing the effectiveness of conventional training and virtual reality simulation in the skills acquisition of junior dental students. Eur J Dent Educ. 2003; 7:13–19.	D	32	RCT	Dentistry	ClinV, MLS	CM:VR vs model	SPd	14.5	5
Quinn F, et al. A study comparing the effectiveness of conventional training and virtual reality simulation in the skills acquisition of junior dental students. Eur J Dent Educ. 2003; 7:164–9.	D	20	RCT	Dentistry	FB	In:Self-instruction	SPd	13.5	3
Buchanan JA. Experience with virtual reality-based technology in teaching restorative dental procedures. J Dent Educ. 2004; 68:1258–1265.	D	44	RCT	Dentistry	FB, Time	CM:VR vs model	SPd	12.5	5
Grober ED, et al. The educational impact of bench model fidelity on the acquisition of technical skill: the use of clinically relevant outcome measures. Ann Surg. 2004; 240:374–381.	PG	40	RCT	Microsurgery/Ophthalmology		CM:Synthetic vs human/animal products	ST, SPc, SPd	13.5	5
Hochberger J, et al. The compact Erlangen Active Simulator for Interventional Endoscopy: a prospective comparison in structured team-training courses on ‘endoscopic hemostasis’ for doctors and nurses. Scand J Gastroenterol. 2004; 39:895–902.	PG, MD, RN, O	207		Endoscopy (GI,Urology,Bronch.)		CM:Model vs model	Sa	10	2
Jasinevicius TR, et al. An evaluation of two dental simulation systems: virtual reality versus contemporary non-computer-assisted. J Dent Educ. 2004; 68:1151–1162.	D	28	RCT	Dentistry	ClinV, FB	CM:VR vs model	SPd	13.5	4
Kim HK, et al. Virtual-reality-based laparoscopic surgical training: The role of simulation fidelity in haptic feedback. Comput Aided Surg. 2004; 9:227–234.	O	16		Min. invasive surg.		SA:Tactile	SPc	11.5	2
Martin KM, et al. Effective nonanatomical endoscopy training produces clinical airway endoscopy proficiency. Anesth Analg. 2004; 99:938–44.	PG, MD, O	40	RCT	Endoscopy (GI,Urology,Bronch.)		CM:Model vs model	ST, SPc	14.5	5
Munz Y, et al. Laparoscopic virtual reality and box trainers: is one superior to the other? Surg Endosc. 2004; 18:485–494.	MS	16	RCT	Min. invasive surg.		CM:VR vs model	ST, SPc	12.5	5
Nishida M, et al. Training in tooth preparation utilizing a support system. J Oral Rehabil. 2004; 31:149–154.	D	10	RCT	Dentistry		MA:External support, add	SPd	10.5	4
Rumball C, et al. Endotracheal intubation and esophageal tracheal Combitube insertion by regular ambulance attendants: A comparative trial. Prehosp Emerg Care. 2004; 8:15–22.	EMT	61		Intubation	DistP, Indiv, Time	ID:Repetition	P	14	2
St. Pierre M, et al. Simulator-based modular human factor training in anesthesiology. Concept and results of the module “Communication and Team Cooperation” [German]. Anaesthesist. 2004; 53:144–152.	MD	34	RCT	Critical thinking	CogI, MLS	MA:Discussion, add	Sa, SPc	12.5	3
Backstein D, et al. Effectiveness of repeated video feedback in the acquisition of a surgical technical skill. Can J Surg. 2005; 48:195–200.	PG	26	RCT	Open surgery/suturing	CogI, FB, MLS	ID:Feedback	SPc	15.5	3
Bathalon S, et al. Cognitive skills analysis, kinesiology, and mental imagery in the acquisition of surgical skills. J Otolaryngol. 2005; 34:328–332.	MS	31	RCT	Resuscitation (BLS,ACLS,ATLS),Percutaneous proc.	CogI, FB, MLS, RepP	ID:Teach cognition	SPc	12.5	4
Bowyer CM, et al. Validation of SimPL – a simulator for diagnostic peritoneal lavage training. Stud Health Technol Inform. 2005; 111:64–67.	MS	40	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:VR vs animal	K, SPc	12.5	4
Bowyer MW, et al. Teaching intravenous cannulation to medical students: comparative analysis of two simulators and two traditional educational approaches. Stud Health Technol Inform. 2005; 111:57–63.	MS	14	RCT	Venous access	ClinV, RangeD	CM:VR vs VR	SPc	13.5	4
Donnon T, et al. Impact of cognitive imaging and sex differences on the development of laparoscopic suturing skills. Can J Surg. 2005; 48:387–393.	MS	42	RCT	Min. invasive surg.	CogI, MLS	ID:Teach cognition	ST	12.5	3
Dubrowski A, et al. The influence of practice schedules in the learning of a complex bone-plating surgical task. Am J Surg. 2005; 190:359–363.	MS	19	RCT	Open surgery/suturing		ID:Sequence	SPc, SPd	12.5	3
Enebo B, et al. Experience and practice organization in learning a simulated high-velocity low-amplitude task. J Manipulative Physiol Ther. 2005; 28:33–43.	C	33X		Spinal manipulation		ID:Sequence	SPc	12.5	2
Gerling GJ, et al. Augmented, pulsating tactile feedback facilitates simulator training of clinical breast examinations. Hum Factors. 2005; 47:670–681.	MS	48	RCT	Physical exam	Indiv, MLS, RangeD	ID:Task variability	SPd	13.5	3
Kiesslich R, et al. Combined simulation training: a new concept and workshop is useful for crisis management in gastrointestinal endoscopy [German]. Z Gastroenterol. 2005; 43:1031–1039.	MD	100X		Endoscopy (GI,Urology,Bronch.),Team training	CogI, FB, GrpP, MLS, RepP	CM:VR vs animal	Sa	11	3
Kirlum HJ, et al. Advanced paediatric laparoscopic surgery: repetitive training in a rabbit model provides superior skills for live operations. Eur J Pediatr Surg. 2005; 15:149–152.	PG	12	RCT	Min. invasive surg.		CM:Model vs animal	ST	9.5	2
Lehmann KS, et al. A prospective randomized study to test the transfer of basic psychomotor skills from virtual reality to physical reality in a comparable training setting. Ann Surg. 2005; 241:442–449.	MS, MD	32	RCT	Min. invasive surg.		CM:VR vs model	ST, SPc	12.5	2
Lentz GM, et al. A six-year study of surgical teaching and skills evaluation for obstetric/gynecologic residents in porcine and inanimate surgical models. Am J Obstet Gynecol. 2005; 193:2056–2061.	PG	27		Min. invasive surg.,Open surgery/suturing		ID:Timing	K, SPc	14.5	4
Madan AK, et al. Participants’ opinions of laparoscopic training devices after a basic laparoscopic training course. Am J Surg. 2005; 189:758–61.	MS	18X		Min. invasive surg.		CM:VR vs model	Sa	8	1
Monsieurs KG, et al. Improved basic life support performance by ward nurses using the CAREvent® Public Access Resuscitator (PAR) in a simulated setting. Resuscitation. 2005; 67:45–50.	RN	152	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB	MA:External support, add	SPc	12.5	5
Tani Botticelli A, et al. The effectiveness of video support in the teaching of manual skills related to initial periodontal therapy tested on phantoms. Int J Comput Dent. 2005; 8:117–127.	O	84	RCT	Dentistry	CogI	MA:CAI, compare	SPc	12.5	3
Uchal M, et al. Validation of a six-task simulation model in minimally invasive surgery. Surg Endosc. 2005; 19:109–16.	PG	17	RCT	Min. invasive surg.		ID:Sequence	SPc	15.5	6
Wierinck E, et al. Effect of augmented visual feedback from a virtual reality simulation system on manual dexterity training. Eur J Dent Educ. 2005; 9:10–16.	D	24	RCT	Dentistry	FB, Indiv	ID:Feedback	ST, SPd	12.5	4
Youngblood PL, et al. Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance. J Am Coll Surg. 2005; 200:546–551.	MS	33	RCT	Min. invasive surg.	CogI, FB	CM:VR vs model	ST, SPc	13.5	4
Aggarwal R, et al. A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg. 2006; 191:128–133.	MS	20	RCT	Min. invasive surg.	RangeD, Time	ID:Sequence	ST	12.5	5
Berkenstadt H, et al. Evaluation of the Trauma-Man® simulator for training in chest drain insertion. European Journal of Trauma. 2006; 32:523–526.	PG	42X	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs animal	Sa	8	4
Bond WF, et al. Cognitive versus technical debriefing after simulation training. Acad Emerg Med. 2006; 13:276–283.	PG	62	RCT	Critical thinking		ID:Teach cognition	Sa	10	4
Chandrasekera SK, et al. Basic laparoscopic surgical training: examination of a low-cost alternative. Eur Urol. 2006; 50:1285–1291.	MS	36	RCT	Min. invasive surg.		CM:Model vs model	ST, SPc, SPd	12.5	5
Chou DS, et al. Comparison of results of virtual-reality simulator and training model for basic ureteroscopy training. J Endourol. 2006; 20:266–271.	MS	16	RCT	Endoscopy (GI,Urology,Bronch.)	RangeD	CM:VR vs model	SPc	12.5	4
Crofts JF, et al. Training for shoulder dystocia: a trial of simulation using low-fidelity and high-fidelity mannequins. Obstet Gynecol. 2006; 108:1477–1485.	PG, MD, O	110	RCT	Obstetrics	CogI, FB, Indiv, MLS	CM:Manikin vs manikin	ST, SPc, SPd	13.5	5
Cummings AJ, et al. Evaluation of a novel animal model for teaching intubation. Teach Learn Med. 2006; 18:316–319.	MS, PG, MD, RN, EMT, O	42X	RCT	Intubation		CM:Model vs animal	Sa, ST	8.5	2
Dubrowski A, et al. Randomised, controlled study investigating the optimal instructor: student ratios for teaching suturing skills. Med Educ. 2006; 40:59–63.	MS	72	RCT	Open surgery/suturing	CogI, FB	In:Instructor intensity	ST, SPc	12.5	4
Halvorsen FH, et al. Virtual reality simulator training equals mechanical robotic training in improving robot-assisted basic suturing skills. Surg Endosc. 2006; 20:1565–1569.	MS	26	RCT	Robotic surg.		CM:VR vs model	Sa, SPc	11.5	2
Heinrich M, et al. Comparison of different training models for laparoscopic surgery in neonates and small infants. Surg Endosc. 2006; 20:641–644.	PG	12	RCT	Min. invasive surg.		CM:Model vs animal	SPc	12.5	2
Jamison RJ, et al. A pilot study assessing simulation using two simulation methods for teaching intravenous cannulation. Clinical Simulation in Nursing Education. 2006; 2:e9–e12.	RN	18	RCT	Venous access		CM:VR vs model	K	14.5	3
Kimura T, et al. Usefulness of a virtual reality simulator or training box for endoscopic surgery training. Surg Endosc. 2006; 20:656–659.	MS	12		Min. invasive surg.		CM:VR vs model	ST, SPc	11.5	2
Mazilu D, et al. Synthetic torso for training in and evaluation of urologic laparoscopic skills. J Endourol. 2006; 20:340–345.	MS, PG, MD, O	25	RCT	Min. invasive surg.		CM:Model vs model	Sa	8	3
Ocel JJ, et al. Formal procedural skills training using a fresh frozen cadaver model: a pilot study. Clin Anat. 2006; 19:142–146.	MS	7X		Venous access,Percutaneous proc.,Natural orifice proc.		CM:Manikin vs cadaver	Sa	7	2
Owen H, et al. Comparison of three simulation-based training methods for management of medical emergencies. Resuscitation. 2006; 71:204–211.	PG	50	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, Indiv, MLS	CM:Manikin vs manikin	SPc	13	5
Panait L, et al. Telementoring versus on-site mentoring in virtual reality-based surgical training. Surg Endosc. 2006; 20:113–118.	MS	20	RCT	Min. invasive surg.		In:Distance supervision	ST, SPc	13	4
Rosenthal ME, et al. Achieving housestaff competence in emergency airway management using scenario based simulation training: comparison of attending vs housestaff trainers. Chest. 2006; 129:1453–1458.	PG	49	RCT	Resuscitation (BLS,ACLS,ATLS)		In:Instructor training/experience	SPc	12.5	4
Rosser JC, Jr., et al. The use of a “hybrid” trainer in an established laparoscopic skills program. J Soc Laparoendosc Surg. 2006; 10:4–10.	PG, MD	817		Min. invasive surg.	FB, Indiv, MLS	ID:Feedback	ST	10.5	1
Savoldelli GL, et al. Value of debriefing during simulated crisis management: oral versus video-assisted oral feedback. Anesthesiology. 2006; 105:279–285.	PG	28	RCT	Anesthesia,Team training	CogI, FB, Indiv, MLS	ID:Feedback	SPc	14.5	5
Scerbo MW, et al. Comparison of a virtual reality simulator and simulated limbs for phlebotomy training. J Infus Nurs. 2006; 29:214–224.	MS	16		Venous access	ClinV, RangeD, Time	CM:VR vs model	SPc, BP	12	2
Scerbo MW, et al. The efficacy of a medical virtual reality simulator for training phlebotomy. Hum Factors. 2006; 48:72–84.	MS	20	RCT	Venous access	ClinV, RangeD	CM:VR vs model	SPc	12.5	3
Stefanidis D, et al. Proficiency maintenance: impact of ongoing simulator training on laparoscopic skill retention. J Am Coll Surg. 2006; 202:599–603.	MS	18	RCT	Min. invasive surg.	CogI, Time	ID:Repetition	SPc	11.5	3
Strom P, et al. Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc. 2006; 20:1383–1388.	PG	19X	RCT	Min. invasive surg.		SA:Tactile	SPc	11.5	3
Ti L-K, et al. The impact of experiential learning on NUS medical students: our experience with task trainers and human-patient simulation. Ann Acad Med Singapore. 2006; 35:619–623.	MS	54		Anesthesia	CogI	ID:Hands on	SPc	11.5	3
Verdaasdonk EGG, et al. Validation of a new basic virtual reality simulator for training of basic endoscopic skills: The SIMENDO. Surg Endosc. 2006; 20:511–518.	MS	16	RCT	Min. invasive surg.		CM:VR vs model	ST, SPc	13.5	4
Wierinck E, et al. Effect of reducing frequency of augmented feedback on manual dexterity training and its retention. J Dent. 2006; 34:641–647.	D	24	RCT	Dentistry	FB, MLS	ID:Feedback	ST, SPd	12.5	4
Wierinck E, et al. Effect of tutorial input in addition to augmented feedback on manual dexterity training and its retention. Eur J Dent Educ. 2006; 10:24–31.	D	24	RCT	Dentistry	FB, MLS	In:Self-instruction	ST, SPd	12.5	4
Adermann J, et al. The impact of force feedback on training of surgical skills in virtual neuroendoscopy. Int J Comput Assist Radiol Surg. 2007; 2(Suppl 1):S198–S200.	MS, MD	60	RCT	Endoscopy (GI,Urology,Bronch.)	FB	SA:Tactile	ST, SPc	11.5	3
Berry M, et al. Porcine transfer study: virtual reality simulator training compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol. 2007; 30:455–461.	MD	12X	RCT	Endovascular proc.		CM:VR vs animal	Sa, SPc	12.5	2
Birch L, et al. Obstetric skills drills: evaluation of teaching methods. Nurse Educ Today. 2007; 27:915–922.	PG, O	4	RCT	Obstetrics	MLS	ID:Blending	SPc	13.5	5
Bruynzeel H, et al. Desktop simulator: key to universal training? Surg Endosc. 2007; 21:1637–1640.	MS	20	RCT	Min. invasive surg.		CM:Model vs model	ST, SPd	12.5	4
Chang J-Y, et al. Effectiveness of two forms of feedback on training of a joint mobilization skill by using a joint translation simulator. Phys Ther. 2007; 87:418–430.	O	24	RCT	Physical therapy	CogI, FB, Indiv, MLS	ID:Feedback	SPc	12.5	5
Cherry RA, et al. The effectiveness of a human patient simulator in the ATLS shock skills station. J Surg Res. 2007; 139:229–235.	PG	44	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs model	K, SPc	13.5	4
Crofts JF, et al. Change in knowledge of midwives and obstetricians following obstetric emergency training: a randomised controlled trial of local hospital, simulation centre and teamwork training. BJOG. 2007; 114:1534–1541.	PG, MD, O	133	RCT	Obstetrics	CogI, FB, Indiv, MLS	CM:Manikin vs manikin	K	14.5	5
Davis DP, et al. The effectiveness of a novel, algorithm-based difficult airway curriculum for air medical crews using human patient simulators. Prehosp Emerg Care. 2007; 11:72–79.	RN, EMT	120X		Intubation	ClinV, MLS, Time	MA:Simulator, add	SPc, P	14	2
Dubrowski A, et al. A comparison of single- and multiple-stage approaches to teaching laparoscopic suturing. Am J Surg. 2007; 193:269–273.	PG	24	RCT	Min. invasive surg.	MLS, RangeD	ID:Task variability	ST, SPc, SPd	13.5	3
Gutierrez F, et al. The effect of degree of immersion upon learning performance in virtual reality simulations for medical education. Stud Health Technol Inform. 2007; 125:155–160.	MS	25	RCT	Resuscitation (BLS,ACLS,ATLS),Physical exam		CM:VR vs VR	K	13.5	5
Heinrichs WL, et al. Criterion-based training with surgical simulators: proficiency of experienced surgeons. J Soc Laparoendosc Surg. 2007; 11:273–302.	MD	17X	RCT	Min. invasive surg.		CM:VR vs VR	Sa	8	4
Iglesias-Vazquez JA, et al. Cost-efficiency assessment of Advanced Life Support (ALS) courses based on the comparison of advanced simulators with conventional manikins. BMC Emergency Medicine. 2007; 7:18.	MD, RN	250	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	SPc	12	2
Immenroth M, et al. Mental training in surgical education: a randomized controlled trial. Ann Surg. 2007; 245:385–391.	MD	66	RCT	Min. invasive surg.	CogI, MLS	ID:Teach cognition	SPc	13.5	6
Kahol K, et al. Augmented virtual reality for laparoscopic surgical tool training. Lect Notes Comput Sci. 2007; 4553 LNCS:459–467.	PG	8		Min. invasive surg.	CogI, FB	ID:Feedback	SPc	11.5	2
Lazarski MP, et al. How do feedback and instructions affect the performance of a simulated surgical task? J Oral Maxillofac Surg. 2007; 65:1155–61.	MS, D	61		Open surgery/suturing	MLS	ID:Instructions	SPd	11.5	3
LeFlore JL, et al. Comparison of self-directed learning versus instructor-modeled learning during a simulated clinical experience. Simul Healthc. 2007; 2:170–177.	RN	11		Resuscitation (BLS,ACLS,ATLS),Team training	CogI, FB, Indiv	In:Self-instruction	K, ST, SPc	13.5	5
Lee JC, et al. Randomized controlled trial of an instructional DVD for clinical skills teaching. Emerg Med Australas. 2007; 19:241–245.	MS, PG, MD, RN	36	RCT	Venous access	CogI, FB, Indiv	MA:Lecture, compare	Sa, SPc	12.5	5
Madan AK, et al. Prospective randomized controlled trial of laparoscopic trainers for basic laparoscopic skills acquisition. Surg Endosc. 2007; 21:209–213.	MS	32	RCT	Min. invasive surg.		MA:Simulator, add	ST, SPc	12.5	5
Neequaye SK, et al. Identification of skills common to renal and iliac endovascular procedures performed on a virtual reality simulator. Eur J Vasc Endovasc Surg. 2007; 33:525–532.	PG	20	RCT	Endovascular proc.	CogI	ID:Task variability	ST, SPc	11.5	4
Schlosser K, et al. Training of laparoscopic skills with virtual reality simulator: a critical reappraisal of the learning curve. European Surgery – Acta Chirurgica Austriaca. 2007; 39:180–184.	MS	14	RCT	Min. invasive surg.		ID:Task variability	ST, SPc	12.5	4
Sidhu RS, et al. Laboratory-based vascular anastomosis training: a randomized controlled trial evaluating the effects of bench model fidelity and level of training on skill acquisition. J Vasc Surg. 2007; 45:343–349.	PG	27	RCT	Open surgery/suturing		CM:Model vs cadaver	ST, SPc, SPd	12.5	4
Spooner BB, et al. An evaluation of objective feedback in basic life support (BLS) training. Resuscitation. 2007; 73:417–424.	O	98	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB	ID:Feedback	SPc, SPd	13.5	5
Stefanidis D, et al. Closing the gap in operative performance between novices and experts: does harder mean better for laparoscopic simulator training? J Am Coll Surg. 2007; 205:307–313.	MS	25	RCT	Min. invasive surg.	CogI, Time	ID:Stress	SPc	12.5	4
Stefanidis D, et al. Construct and face validity and task workload for laparoscopic camera navigation: virtual reality versus videotrainer systems at the SAGES Learning Center. Surg Endosc. 2007; 21:1158–1164.	PG, MD, O	90X		Min. invasive surg.		CM:VR vs model	Sa, SPc	12.5	3
Stefanidis D, et al. Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery. 2007; 142:202–206.	MS	22		Min. invasive surg.	FB, Indiv	ID:Feedback	SPc	11.5	1
Thomas EJ, et al. Teaching teamwork during the Neonatal Resuscitation Program: a randomized trial. J Perinatol. 2007; 27:409–414.	PG	32	RCT	Team training	MLS	MA:Team training, add	SPc	13.5	5
Torgerson C, et al. Low fidelity simulation of temporal bone drilling leads to improved but suboptimal outcomes. Stud Health Technol Inform. 2007; 125:470–472.	MS	24		Open surgery/suturing		ID:Task variability	SPd	9.5	1
Van Sickle KR, et al. The effect of escalating feedback on the acquisition of psychomotor skills for laparoscopy. Surg Endosc. 2007; 21:220–224.	MS, O	16	RCT	Min. invasive surg.		ID:Feedback	SPc	13.5	4
Verdaasdonk EGG, et al. The influence of different training schedules on the learning of psychomotor skills for endoscopic surgery. Surg Endosc. 2007; 21:214–219.	MS, O	20	RCT	Min. invasive surg.	DistP	ID:Timing	ST, SPc	11.5	3
Welk A, 21et al. Mental training in dentistry. Quintessence Int. 2007; 38:489–497.	D	41	RCT	Dentistry	MLS	ID:Teach cognition	K, SPc, SPd	13.5	5
Xeroulis GJ, et al. Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery. 2007; 141:442–449.	MS	30	RCT	Open surgery/suturing	CogI, FB, Indiv, MLS	MA:CAI, compare	SPc	13.5	4
Baranauskas MB, et al. Simulation of ultrasound-guided peripheral nerve block: learning curve of CET-SMA/HSL Anesthesiology residents. Rev Bras Anestesiol. 2008; 58:106–111.	PG	6	RCT	Anesthesia,Radiology/other noninvasive dx	Time	ID:Repetition	ST, SPc	9.5	4
Bingener J, et al. Randomized double-blinded trial investigating the impact of a curriculum focused on error recognition on laparoscopic suturing training. Am J Surg. 2008; 195:179–182.	MS	30	RCT	Min. invasive surg.	MLS	ID:Feedback	ST, SPc	15.5	4
Botden SMBI, et al. The importance of haptic feedback in laparoscopic suturing training and the additive value of virtual reality simulation. Surg Endosc. 2008; 22:1214–1222.	PG, MD	45X	RCT	Min. invasive surg.		ID:Sequence	Sa, SPc, SPd	12	4
Chandra DB, et al. Fiberoptic oral intubation: the effect of model fidelity on training for transfer to patient care. Anesthesiology. 2008; 109:1007–1013.	O	28	RCT	Endoscopy (GI,Urology,Bronch.)		CM:VR vs model	BT, BP, P	16	5
Chang S, et al. Verbal communication improves laparoscopic team performance. Surgical Innovation. 2008; 15:143–147.	MS, PG, O	24	RCT	Min. invasive surg.	CogI, FB, Indiv	ID:Feedback	ST, SPc	12.5	2
Chmarra MK, et al. Force feedback and basic laparoscopic skills. Surg Endosc. 2008; 22:2140–2148.	PG	19	RCT	Min. invasive surg.		SA:Tactile	ST, SPc	11.5	4
Cho J, et al. Comparison of manikin versus porcine models in cricothyrotomy procedure training. Emerg Med J. 2008; 25:732–734.	PG, MD, RN, EMT	49X	RCT	Open surgery/suturing,Intubation		CM:Model vs animal	Sa	9	2
Crofts JF, et al. Patient-actor perception of care: a comparison of obstetric emergency training using manikins and patient-actors. Qual Saf Health Care. 2008; 17:20–24.	MD, O	64	RCT	Obstetrics	CogI, FB, Indiv	CM:Manikin vs manikin	SPc	13.5	4
De Regge M, et al. Basic life support refresher training of nurses: individual training and group training are equally effective. Resuscitation. 2008; 79:283–287.	RN	103	RCT	Resuscitation (BLS,ACLS,ATLS)	FB, GrpP, MLS	GC:Solo vs group	SPc	12.5	6
Dine CJ, et al. Improving cardiopulmonary resuscitation quality and resuscitation training by combining audiovisual feedback and debriefing. Crit Care Med. 2008; 36:2817–2822.	RN	65	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB, Indiv, MLS	ID:Feedback	K, SPc	12.5	5
Ellis D, et al. Hospital, simulation center, and teamwork training for eclampsia management: a randomized controlled trial. Obstet Gynecol. 2008; 111:723–731.	MD, O	24	RCT	Obstetrics		CM:Manikin vs manikin	ST	14.5	5
Friedman Z, et al. Teaching lifesaving procedures: the impact of model fidelity on acquisition and transfer of cricothyrotomy skills to performance on cadavers. Anesth Analg. 2008; 107:1663–1669.	PG	22	RCT	Intubation		CM:Manikin vs model	ST, SPc	12.5	4
Grady JL, et al. Learning nursing procedures: the influence of simulator fidelity and student gender on teaching effectiveness. J Nurs Educ. 2008; 47:403–408.	RN	39X		Natural orifice proc.		CM:Manikin vs model	Sa, SPc	13.5	3
Grodin MH, et al. Ophthalmic surgical training: a curriculum to enhance surgical simulation. Retina. 2008; 28:1509–1514.	PG, MD	45	RCT	Microsurgery/Ophthalmology		MA:CAI, compare	SPc	13.5	4
Isbye DL, et al. Voice advisory manikin versus instructor facilitated training in cardiopulmonary resuscitation. Resuscitation. 2008; 79:73–81.	MS	43	RCT	Resuscitation (BLS,ACLS,ATLS)		In:Self-instruction	SPc	13.5	5
Jiang C, et al. A training model for laparoscopic urethrovesical anastomosis. J Endourol. 2008; 22:1541–1545.	PG	40	RCT	Min. invasive surg.,Endoscopy (GI,Urology,Bronch.)		CM:Animal products vs animal products	ST, SPc	12.5	4
Johnson KB, et al. Part Task and variable priority training in first-year anesthesia resident education: a combined didactic and simulation-based approach to improve management of adverse airway and respiratory events. Anesthesiology. 2008; 108:831–840.	PG	21	RCT	Anesthesia	CogI, FB, MLS	ID:Sequence	K, SPc, SPd	14.5	4
Kanumuri P, et al. Virtual reality and computer-enhanced training devices equally improve laparoscopic surgical skill in novices. J Soc Laparoendosc Surg. 2008; 12:219–226.	MS	16	RCT	Min. invasive surg.		CM:VR vs VR	ST, SPc, SPd	13.5	4
Lammers RL. Learning and retention rates after training in posterior epistaxis management. Acad Emerg Med. 2008; 15:1181–9.	PG	28	RCT	Epistaxis management	FB, MLS, Time	ID:Sequence	ST, SPc	14.5	5
LeFlore JL, et al. Effectiveness of 2 methods to teach and evaluate new content to neonatal transport personnel using high-fidelity simulation. J Perinat Neonatal Nurs. 2008; 22:319–328.	RN, EMT, O	24		Resuscitation (BLS,ACLS,ATLS)	CogI, FB, Indiv	ID:Hands on	Sa, SPc	13.5	2
Low D, et al. The use of the BERCI DCI® Video Laryngoscope for teaching novices direct laryngoscopy and tracheal intubation. Anaesthesia. 2008; 63:195–201.	MS, EMT	42	RCT	Intubation	CogI, FB	SA:Visual	ST, SPd	12.5	5
Miotto HC, et al. Advanced Cardiac Life Support Courses: live actors do not improve training results compared with conventional manikins. Resuscitation. 2008; 76:244–248.	MD, RN, O	225	RCT	Resuscitation (BLS,ACLS,ATLS)	MLS	MA:Patient/SP experience, add	K	12	5
Murphy MA, et al. Should we train the trainers? Results of a randomized trial. J Am Coll Surg. 2008; 207:185–190.	MS	30	RCT	Venous access	CogI, Indiv, MLS	In:Instructor training/experience	ST, SPc	14.5	5
Nousiainen M, et al. Comparison of expert instruction and computer-based video training in teaching fundamental surgical skills to medical students. Surgery. 2008; 143:539–544.	MS	16	RCT	Open surgery/suturing	CogI, FB, Indiv, MLS	ID:Interactivity	ST, SPc	13.5	4
O’Connor A, et al. How much feedback is necessary for learning to suture? Surg Endosc. 2008; 22:1614–1619.	MS	6	RCT	Min. invasive surg.	CogI, FB, Indiv, MLS	ID:Feedback	SPd	11.5	2
Pierce J, et al. Comparative usability studies of full vs. partial immersive virtual reality simulation for medical education and training. Stud Health Technol Inform. 2008; 132:372–377.	MS	25	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:VR vs VR	Sa	10	4
Rafiq A, et al. Objective assessment of training surgical skills using simulated tissue interface with real-time feedback. J Surg Educ. 2008; 65:270–274.	MS	12	RCT	Open surgery/suturing	CogI, FB, Indiv, MLS	ID:Feedback	SPc	11.5	2
Reyes SD, et al. Implementation and evaluation of a virtual simulator system: teaching intravenous skills. Clinical Simulation in Nursing Education. 2008; 4:e43–e49.	RN	28	RCT	Venous access	CogI, DistP	CM:VR vs model	Sa, K, SPc	11.5	2
Rissanen MJ, et al. Asynchronous teaching of psychomotor skills through VR annotations: evaluation in digital rectal examination. Stud Health Technol Inform. 2008; 132:411–416.	MS	8		Physical exam	CogI, FB, Indiv, MLS	ID:Feedback	SPc	10.5	2
Roberson DW, et al. Improving wound care simulation with the addition of odor: A descriptive, quasi-experimental study. Ostomy Wound Management. 2008; 54:36–43.	RN	99		Physical exam,wound care		SA:Olfactory	Sa	9	2
Sanders CW, et al. Learning basic surgical skills with mental imagery: using the simulation centre in the mind. Med Educ. 2008; 42:607–612.	MS	63	RCT	Open surgery/suturing	CogI	ID:Teach cognition	SPc	14.5	3
Tanoue K, et al. Effectiveness of endoscopic surgery training for medical students using a virtual reality simulator versus a box trainer: a randomized controlled trial. Surg Endosc. 2008; 22:985–990.	MS	40	RCT	Min. invasive surg.	RangeD	CM:VR vs model	ST	12.5	4
Tzafestas CS, et al. Pilot evaluation study of a virtual paracentesis simulator for skill training and assessment: the beneficial effect of haptic display. Presence: Teleoperators and Virtual Environments. 2008; 17:212–229.	MD, RN	20	RCT	Venous access	CogI, Indiv	SA:Tactile	ST, SPc	12.5	3
Van Herzeele I, et al. Cognitive training improves clinically relevant outcomes during simulated endovascular procedures. J Vasc Surg. 2008; 48:1223–1230.	PG	20		Endovascular proc.	MLS, Time	MA:Lecture, add	ST, SPd	10.5	2
Vankipuram M, et al. Virtual reality based training to resolve visio-motor conflicts in surgical environments. HAVE 2008 – IEEE International Workshop on Haptic Audio Visual Environments and Games Proceedings. 2008; Article number 4685290:7–12.	PG	10		Min. invasive surg.	RangeD	SA:Tactile	ST, SPc, SPd	11.5	2
Wang XP, et al. Effect of emergency care simulator combined with problem-based learning in teaching of cardiopulmonary resuscitation [Chinese]. Chung Hua I Hsueh Tsa Chih. 2008; 88:1651–1653.	MS	42	RCT	Resuscitation (BLS,ACLS,ATLS),Team training	ClinV, CogI	MA:Simulator, add	SPc	12.5	4
Wheeler DW, et al. Retention of drug administration skills after intensive teaching. Anaesthesia. 2008; 63:379–384.	O	72		Physiology: pharmacology	MLS	MA:CAI, add	SPc	11.5	4
Youngquist ST, et al. Paramedic self-efficacy and skill retention in pediatric airway management. Acad Emerg Med. 2008; 15:1295–303.	EMT	135		Intubation	CogI, FB	In:Self-instruction	SPd	13.5	3
de Vries W, et al. Self–training in the use of automated external defibrillators: the same results for less money. Resuscitation. 2008; 76:76–82.	RN	30	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB, GrpP, Indiv, MLS	In:Self-instruction	SPc	10.5	2
Butler KW, et al. Implementation of active learning pedagogy comparing low-fidelity simulation versus high-fidelity simulation in pediatric nursing education. Clinical Simulation in Nursing. 2009; 5:e129–e136.	RN	30	RCT	Physiology:fluid and electrolyte		CM:Manikin vs manikin	Sa	12	3
Campbell DM, et al. High-fidelity simulation in neonatal resuscitation. Paediatrics and Child Health. 2009; 14:19–23.	PG	15	RCT	Resuscitation (BLS,ACLS,ATLS)	Indiv	CM:Manikin vs manikin	Sa, K, ST	13.5	4
Carter YM, et al. Open lobectomy simulator is an effective tool for teaching thoracic surgical skills. Ann Thorac Surg. 2009; 87:1546–1550.	MS	18		Open surgery/suturing	CogI, DistP, FB, Indiv, MLS, RepP, Time	ID:Repetition	K, ST, SPc	11.5	2
Day T, et al. Effect of performance feedback on tracheal suctioning knowledge and skills: randomized controlled trial. J Adv Nurs. 2009; 65:1423–1431.	RN, EMT	38	RCT	Natural orifice proc.,tracheal suctioning		ID:Feedback	K, SPc	15	4
Deladisma AM, et al. A pilot study to integrate an immersive virtual patient with a breast complaint and breast examination simulator into a surgery clerkship. Am J Surg. 2009; 197:102–106.	MS	21	RCT	Physical exam	DistP, MLS, Time	MA:CAI, add	SPc	10	4
Deutsch ES, et al. Multimodality education for airway endoscopy skill development. Ann Otol Rhinol Laryngol. 2009; 118:81–86.	PG	36X		Endoscopy (GI,Urology,Bronch.)		CM:VR vs manikin	SPc	9.5	1
Domuracki KJ, et al. Learning on a simulator does transfer to clinical practice. Resuscitation. 2009; 80:346–349.	MS, RN, RN	101	RCT	Intubation	FB, Indiv, Mast, MLS	ID:Feedback	SPc	13.5	5
Donoghue AJ, et al. Effect of high-fidelity simulation on Pediatric Advanced Life Support training in pediatric house staff: a randomized trial. Pediatr Emerg Care. 2009; 25:139–144.	PG	51	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	SPc	14.5	4
Friedman Z, et al. Clinical impact of epidural anesthesia simulation on short- and long-term learning curve: High- versus low-fidelity model training. Reg Anesth Pain Med. 2009; 34:229–232.	PG	24	RCT	Anesthesia,Percutaneous proc.,epidural	ClinV	CM:VR vs model	BP	15	5
Gerling GJ, et al. The design and evaluation of a computerized and physical simulator for training clinical prostate exams. IEEE Transactions on Systems, Man, and Cybernetics Part A, Systems and Humans. 2009; 39:388–403.	MS, RN	26	RCT	Physical exam	ClinV, FB, Indiv, RangeD	CM:Model vs model	SPc, SPd	11.5	2
Girzadas DV, Jr., et al. Hybrid simulation combining a high fidelity scenario with a pelvic ultrasound task trainer enhances the training and evaluation of endovaginal ultrasound skills. Acad Emerg Med. 2009; 16:429–435.	PG	45	RCT	Critical thinking,Obstetrics		ID:Sequence	Sa, ST	9	4
Helmy S, et al. Development of laparoscopic skills using a new inexpensive webcam trainer. Journal of Biological Sciences. 2009; 9:766–771.	PG	12X	RCT	Min. invasive surg.		CM:Model vs model	ST	13	3
Hoadley TA. Learning advanced cardiac life support: a comparison study of the effects of low- and high-fidelity simulation. Nurs Educ Perspect. 2009; 30:91–95.	MD, RN, EMT, O	53	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	Sa, K, SPc	14	2
Jayaraman S, et al. Novel hands-free pointer improves instruction efficiency in laparoscopic surgery. Surgical Innovation. 2009; 16:73–77.	MS, PG, MD	10X		Min. invasive surg.	CogI, FB	ID:Instructions	ST	11.5	2
Jensen AR, et al. Acquiring basic surgical skills: Is a faculty mentor really needed? Am J Surg. 2009; 197:82–88.	PG	44	RCT	Open surgery/suturing	CogI, FB	In:Self-instruction	ST, SPc, SPd	12.5	6
Kahol K, et al. Cognitive simulators for medical education and training. Journal of Biomedical Informatics. 2009; 42:593–604.	PG	10		Min. invasive surg.	CogI, RangeD	ID:Additional practice	SPc	12.5	2
Kardong-Edgren S, et al. VitalSim® versus SimMan®: a comparison of BSN student test scores, knowledge retention, and satisfaction. Clinical Simulation in Nursing. 2009; 5:e105–e111.	RN	76	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	Sa, K	14.5	4
Kishore TA, et al. Task deconstruction facilitates acquisition of transurethral resection of prostate skills on a virtual reality trainer. J Endourol. 2009; 23:665–668.	MS	18	RCT	Endoscopy (GI,Urology,Bronch.)	RangeD	ID:Sequence	Sa, SPc	11.5	3
Kromann CB, et al. The effect of testing on skills learning. Med Educ. 2009; 43:21–27.	MS	81	RCT	Resuscitation (BLS,ACLS,ATLS)	Indiv	ID:Testing effect	SPc	13.5	5
LeFlore JL, et al. Alternative educational models for interdisciplinary student teams. Simul Healthc. 2009; 4:135–142.	RN, EMT	13	RCT	Resuscitation (BLS,ACLS,ATLS),Team training	CogI, FB, Indiv, RepP	ID:Hands on	Sa, ST, SPc	14.5	4
McDougall EM, et al. Preliminary study of virtual reality and model simulation for learning laparoscopic suturing skills. J Urol. 2009; 182:1018–1025.	MS	20	RCT	Min. invasive surg.		CM:VR vs model	Sa, ST, SPc	15.5	4
Moulton C-A, et al. Teaching communication skills using the integrated procedural performance instrument (IPPI): a randomized controlled trial. Am J Surg. 2009; 197:113–118.	MS, PG	30	RCT	Open surgery/suturing,Communication skill	CogI, FB	ID:Feedback	SPc	13.5	4
Muller MP, et al. Excellence in performance and stress reduction during two different full scale simulator training courses: a pilot study. Resuscitation. 2009; 80:919–924.	MD	29	RCT	Resuscitation (BLS,ACLS,ATLS),Team training		ID:Teach cognition	SPc	14.5	4
Panait L, et al. The role of haptic feedback in laparoscopic simulation training. J Surg Res. 2009; 156:312–316.	MS	10X		Min. invasive surg.	CogI, MLS	SA:Tactile	ST, SPc	11.5	3
Rodgers DL, et al. The effect of high-fidelity simulation on educational outcomes in an Advanced Cardiovascular Life Support course. Simul Healthc. 2009; 4:200–206.	RN	34		Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	K, SPc	13.5	3
Rodriguez Garcia JI, et al. Does the incorporation of a virtual simulator improve abilities in endoscopic surgery acquired with an inanimate simulator? Cir Esp. 2009; 86:167–70.	PG	17	RCT	Min. invasive surg.	CogI, FB, Time	MA:Simulator, add	SPc	10.5	2
Rosenthal ME, et al. Pretraining on Southwestern stations decreases training time and cost for proficiency-based fundamentals of laparoscopic surgery training. J Am Coll Surg. 2009; 209:626–631.	MS	20		Min. invasive surg.	Time	MA:Simulator, add	ST, SPc	11.5	1
Sullivan-Mann J, et al. The effects of simulation on nursing students’ critical thinking scores: a quantitative study. Newborn and Infant Nursing Reviews. 2009; 9:111–116.	RN	53	RCT	Critical thinking	ClinV	ID:Repetition	K	14.5	4
Szafranski C, et al. Distractions and surgical proficiency: an educational perspective. Am J Surg. 2009; 198:804–810.	PG	14		Min. invasive surg.	Indiv	ID:Stress	ST, SPc	12.5	2
Ti LK, et al. Experiential learning improves the learning and retention of endotracheal intubation. Med Educ. 2009; 43:654–660.	MS	210	RCT	Intubation	CogI, FB, Indiv	In:Instructor intensity	SPc, SPd	12.5	6
Walsh CM, et al. Concurrent versus terminal feedback: it may be better to wait. Acad Med. 2009; 84 (10 Suppl):S54–S57.	MS	30	RCT	Endoscopy (GI,Urology,Bronch.)		ID:Feedback	ST, SPc	13.5	4
Welke TM, et al. Personalized oral debriefing versus standardized multimedia instruction after patient crisis simulation. Anesth Analg. 2009; 109:183–189.	PG	30	RCT	Resuscitation (BLS,ACLS,ATLS),Team training	CogI, FB, Indiv	ID:Feedback	SPc	14.5	5
Yasukawa Y. The effectiveness of cavity preparation training using a virtual reality simulation system with or without feedback. [Japanese]. Kokubyo Gakkai Zasshi. 2009; 76:73–80.	D	39	RCT	Dentistry	CogI, FB, Indiv, MLS	ID:Feedback	ST, SPc	12.5	5
Zausig YA, et al. Inefficacy of simulator-based training on anaesthesiologists’ non-technical skills. Acta Anaesthesiol Scand. 2009; 53:611–619.	MD	42	RCT	Anesthesia	FB, Time	ID:Teach cognition	SPc	15.5	6
Acton RD, et al. Synthesis versus imitation: evaluation of a medical student simulation curriculum via Objective Structured Assessment of Technical Skill. J Surg Educ. 2010; 67:173–178.	MS	189		Open surgery/suturing	CogI, MLS, Time	ID:Sequence	ST, SPc	12.5	1
Ahmad I, et al. Evaluation of Real-time Visio-haptic deformable Bovine Rectal Palpation Simulator. Proceedings 2010 International Symposium on Information Technology – Visual Informatics, ITSim’10 2010; 1:Art. No.: 5561364.	V	20		Physical exam		SA:Visual	SPc	11.5	2
Blum CA, et al. High-fidelity nursing simulation: impact on student self-confidence and clinical competence. International Journal of Nursing Education Scholarship. 2010; 7:Article 18.	RN	53		Communication skill,Critical thinking,Nursing health assessment		CM:Manikin vs model	SPc	14.5	3
Brydges R, et al. Comparing self-guided learning and educator-guided learning formats for simulation-based clinical training. J Adv Nurs. 2010; 66:1832–1844.	RN	30	RCT	Venous access	Indiv, Mast, Time	ID:Sequence	SPc	14.5	5
Brydges R, et al. Coordinating progressive levels of simulation fidelity to maximize educational benefit. Acad Med. 2010; 85:806–812.	MS	30	RCT	Venous access	ClinV, Indiv	ID:Sequence	Sa, SPc	14.5	5
Buzink SN, et al. Do basic psychomotor skills transfer between different image-based procedures? World J Surg. 2010; 34:933–940.	MS	29X	RCT	Min. invasive surg.,Endoscopy (GI,Urology,Bronch.)	ClinV	CM:VR vs VR	ST	11.5	3
Cason CL, et al. Improving learning of airway management with case-based computer microsimulations. Clinical Simulation in Nursing. 2010; 6:e15–e23.	RN	76	RCT	Critical thinking		MA:CAI, compare	K	11.5	5
Chandra V, et al. A comparison of laparoscopic and robotic assisted suturing performance by experts and novices. Surgery. 2010; 147:830–839.	PG	20	RCT	Min. invasive surg.,Robotic surg.		MA:Robot assistance, add	ST, SPc	13.5	4
Da Cruz JAS, et al. Does training laparoscopic skills in a virtual reality simulator improve surgical performance? J Endourol. 2010; 24:1845–1849.	MS	10	RCT	Min. invasive surg.	MLS	ID:Sequence	ST, SPc	11.5	4
Dantas AK, et al. Assessment of preclinical learning on oral surgery using three instructional strategies. J Dent Educ. 2010; 74:1230–1236.	D	20		Dentistry	CogI, FB, Indiv, MLS	MA:Discussion, compare	SPc	10.5	2
Davoudi M, et al. Comparative effectiveness of low- and high-fidelity bronchoscopy simulation for training in conventional transbronchial needle aspiration and user preferences. Respiration. 2010; 80:327–334.	MD	44X	RCT	Endoscopy (GI,Urology,Bronch.)		CM:VR vs model	Sa	9	3
DeMaria Jr S, et al. Adding emotional stressors to training in simulated cardiopulmonary arrest enhances participant performance. Med Educ. 2010; 44:1006–1015.	MS	25	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, MLS	ID:Stress	K, SPc	13.5	4
Debes AJ, et al. A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. Am J Surg. 2010; 199:840–845.	MS	38X	RCT	Min. invasive surg.	RangeD	CM:VR vs model	ST, SPc	11.5	5
Donoghue AJ, et al. Perception of realism during mock resuscitations by pediatric housestaff: the impact of simulated physical features. Simul Healthc. 2010; 5:16–20.	PG	51	RCT	Resuscitation (BLS,ACLS,ATLS)		CM:Manikin vs manikin	Sa	11	3
Dunnican WJ, et al. Reverse alignment “Mirror Image” visualization as a laparoscopic training tool improves task performance. Surgical Innovation. 2010; 17:108–113.	MS, PG	21	RCT	Min. invasive surg.		ID:Clinical scenario	SPc	12	4
Gauger PG, et al. Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg. 2010; 199:72–80.	PG	14	RCT	Min. invasive surg.	Mast, MLS	ID:Mastery	SPc, BP	13	3
Grant JS, et al. Using video-facilitated feedback to improve student performance following high-fidelity simulation. Clinical Simulation in Nursing. 2010; 6:e177–e184.	RN	40	RCT	Communication skill,Team training		ID:Feedback	SPc	14.5	5
Hein C, et al. A training program for novice paramedics provides initial laryngeal mask airway insertion skill and improves skill retention at 6 months. Simul Healthc. 2010; 5:33–39.	EMT	50	RCT	Intubation		ID:Repetition	ST, SPc	13.5	4
Hunziker S, et al. Brief leadership instructions improve cardiopulmonary resuscitation in a high-fidelity simulation: a randomized controlled trial.[Erratum appears in Crit Care Med. 2010 Jun;38(6):1510]. Crit Care Med. 2010; 38:1086–1091.	MS	63	RCT	Resuscitation (BLS,ACLS,ATLS)		ID:Teach cognition	ST, SPc	13.5	6
Kardong-Edgren SE, et al. Comparison of two instructional modalities for nursing student CPR skill acquisition. Resuscitation. 2010; 81:1019–1024.	RN	604		Resuscitation (BLS,ACLS,ATLS)	GrpP, Indiv, MLS	In:Self-instruction	SPc	12.5	2
Kent DJ. Effects of a just-in-time educational intervention placed on wound dressing packages: a multicenter randomized controlled trial. Journal of Wound, Ostomy, & Continence Nursing. 2010; 37:609–614.	RN	139	RCT	Dressing change	CogI	ID:Instructions	Sa, SPd	14.5	4
Kromann CB, et al. The testing effect on skills learning might last 6 months. Adv Health Sci Educ Theory Pract. 2010; 15:395–401.	MS	89	RCT	Resuscitation (BLS,ACLS,ATLS)	MLS	ID:Testing effect	SPc	13.5	6
Kruglikova I, et al. The impact of constructive feedback on training in gastrointestinal endoscopy using high-fidelity virtual-reality simulation: a randomised controlled trial. Gut. 2010; 59:181–185.	PG	21	RCT	Endoscopy (GI,Urology,Bronch.)	CogI, FB, Indiv, MLS	ID:Feedback	ST, SPc	12.5	5
Lauscher JC, et al. A new surgical trainer (BOPT) improves skill transfer for anastomotic techniques in gastrointestinal surgery into the operating room: a prospective randomized trial. World J Surg. 2010; 34:2017–2025.	PG	35	RCT	Open surgery/suturing		CM:Manikin vs model	ST, SPc	12.5	4
Leblanc F, et al. A comparison of human cadaver and augmented reality simulator models for straight laparoscopic colorectal skills acquisition training. J Am Coll Surg. 2010; 211:250–255.	MD	31		Min. invasive surg.		CM:VR vs cadaver	Sa, SPc, SPd	12.5	1
Leblanc F, et al. Hand-assisted laparoscopic sigmoid colectomy skills acquisition: augmented reality simulator versus human cadaver training models. J Surg Educ. 2010; 67:200–204.	MD	26		Min. invasive surg.	Indiv	CM:VR vs cadaver	Sa, SPc, SPd	11.5	1
Lee CC, et al. Comparison of traditional advanced cardiac life support (ACLS) course instruction vs. a scenario-based, performance oriented team instruction (SPOTI) method for Korean paramedic students. J Emerg Med. 2010; 38:89–92.	EMT	30	RCT	Resuscitation (BLS,ACLS,ATLS)	FB, Indiv, MLS	ID:Blending	ST, SPc	10.5	4
Lerner MA, et al. Does training on a virtual reality robotic simulator improve performance on the da Vinci® surgical system? J Endourol. 2010; 24:467–472.	MS, PG	22		Robotic surg.	Curr, FB	CM:VR vs model	ST	11.5	2
Martínez AM, et al. Adaptation to a dynamic visual perspective in laparoscopy through training in the cutting task. Surg Endosc. 2010; 24:1341–1346.	PG	26X		Min. invasive surg.		SA:Visual	ST, SPc	10.5	2
McCormick MJ, et al. Case scenarios and simulations as techniques to facilitate asthma education and team building among health care students. Journal of Asthma and Allergy Educators. 2010; 1:18–22.	RN, O	34		Physiology:Asthma		GC:Interdiscplinary vs single disc.	Sa	9	1
Mishra S, et al. Percutaneous renal access training: Content validation comparison between a live porcine and a virtual reality (VR) simulation model. BJU International. 2010; 106:1753–1756.	MD	24X		Endovascular proc.		CM:VR vs animal	Sa	7	2
Mohammadi Y, et al. Comparison of laparoscopy training using the box trainer versus the virtual trainer. J Soc Laparoendosc Surg. 2010; 14:205–212.	MS	43		Min. invasive surg.		CM:VR vs model	ST, SPc	11.5	3
Molinas CR, et al. Defining a structured training program for acquiring basic and advanced laparoscopic psychomotor skills in a simulator. Gynecological Surgery. 2010; 7:427–435.	PG, MD	40	RCT	Min. invasive surg.	RangeD	ID:Sequence	SPc	11.5	3
Morandeira Rivas A, et al. Low cost simulator for acquiring basic laparoscopic skills. Cir Esp. 2010; 87:26–32.	MS	16	RCT	Min. invasive surg.		CM:Model vs model	ST	13.5	4
Muresan C, III, et al. Transfer of training in the development of intracorporeal suturing skill in medical student novices: a prospective randomized trial. Am J Surg. 2010; 200:537–541.	MS	20	RCT	Min. invasive surg.		ID:Additional practice	ST, SPc	12.5	3
Oermann MH, et al. HeartCode BLS with voice assisted manikin for teaching nursing students: preliminary results. Nurs Educ Perspect. 2010; 31:303–308.	RN	603	RCT	Resuscitation (BLS,ACLS,ATLS)	GrpP, Indiv, MLS	ID:Feedback	SPc	12.5	4
Okrainec A, et al. Telesimulation: an effective method for teaching the fundamentals of laparoscopic surgery in resource-restricted countries. Surg Endosc. 2010; 24:417–422.	PG, MD	16		Min. invasive surg.	CogI, FB, Indiv, MLS	In:Distance supervision	SPc	12	1
Orde S, et al. A randomised trial comparing a 4-stage to 2-stage teaching technique for laryngeal mask insertion. Resuscitation. 2010; 81:1687–1691.	MS, RN, RN	120	RCT	Intubation	MLS	ID:Sequence	ST, SPc, SPd	13.5	5
Perkins GD, et al. The effect of pre-course e-learning prior to advanced life support training: A randomised controlled trial. Resuscitation. 2010; 81:877–881.	MD, RN, O	551	RCT	Resuscitation (BLS,ACLS,ATLS)	FB, MLS	MA:CAI, add	K, SPc	15.5	5
Persoon MC, et al. The effect of a low-fidelity model on cystoscopic skill training: a single-blinded randomized controlled trial. Simul Healthc. 2010; 5:213–218.	MS	32	RCT	Endoscopy (GI,Urology,Bronch.)		MA:Simulator, add	Sa, ST, SPc, SPd	13.5	4
Salkini MW, et al. The role of haptic feedback in laparoscopic training using the LapMentor II. J Endourol. 2010; 24:99–102.	MS	20		Min. invasive surg.	FB	SA:Tactile	SPc	10.5	2
Scavone BM, et al. A randomized controlled trial of the impact of simulation-based training on resident performance during a simulated obstetric anesthesia emergency. Simul Healthc. 2010; 5:320–324.	PG	32	RCT	Anesthesia		ID:Clinical scenario	Sa, ST, SPc	14.5	4
Stefanidis D, et al. Initial laparoscopic basic skills training shortens the learning curve of laparoscopic suturing and is cost-effective. J Am Coll Surg. 2010; 210:436–440.	MS	18	RCT	Min. invasive surg.	Time	MA:Simulator, add	ST, SPc	14.5	5
Stefanidis D, et al. Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload. Surg Endosc. 2010; 24:377–382.	MS	24X		Min. invasive surg.,Robotic surg.		MA:Robot assistance, add	ST, SPc	11.5	1
Suebnukarn S, et al. Augmented kinematic feedback from haptic virtual reality for dental skill acquisition. J Dent Educ. 2010; 74:1357–1366.	D	16	RCT	Dentistry	CogI, FB, MLS	ID:Feedback	ST, SPc	12.5	4
Thomas EJ, et al. Team training in the neonatal resuscitation program for interns: teamwork and quality of resuscitations. Pediatrics. 2010; 125:539–546.	PG	67	RCT	Resuscitation (BLS,ACLS,ATLS)		MA:Team training, add	ST, SPc	14.5	6
Van Heukelom JN, et al. Comparison of postsimulation debriefing versus in-simulation debriefing in medical simulation. Simul Healthc. 2010; 5:91–97.	MS	161	RCT	Resuscitation (BLS,ACLS,ATLS)		ID:Feedback	Sa	11	5
Wandell HF. Using a virtual reality simulator in phlebotomy training. Laboratory Medicine. 2010; 41:463–466.	O	25		Venous access	ClinV, FB	CM:VR vs model	SPc	11.5	2
Wong W, et al. The effect of cross-training with adjustable airway model anatomies on laryngoscopy skill transfer. Anesth Analg. 2010; Available online October 21, 2010; in press.	MS, EMT	47		Intubation	CogI, MLS, RangeD	ID:Task variability	SPd	10.5	2
Yang JH, et al. Comparison of four manikins and fresh frozen cadaver models for direct laryngoscopic orotracheal intubation training. Emerg Med J. 2010; 27:13–16.	MD, RN	56X	RCT	Intubation		CM:Manikin vs cadaver	Sa	9	3
Zendejas B, et al. Teaching first or teaching last: Does the timing matter in simulation-based surgical scenarios? J Surg Educ. 2010; 67:432–438.	MS, PG	49X		Various surgical topics		ID:Sequence	K	12.5	5
Alfes CM. Evaluating the use of simulation with beginning nursing students. J Nurs Educ. 2011; 50:89–93.	RN	63		Pain management	CogI	ID:Hands on	Sa, SPc	9.5	2
Arora S, et al. Mental practice enhances surgical technical skills: A randomized controlled study. Ann Surg. 2011; 253:265–270.	PG	18	RCT	Min. invasive surg.	CogI, Indiv, MLS	ID:Teach cognition	SPc	14.5	5
Auerbach M, et al. Repetitive pediatric simulation resuscitation training. Pediatr Emerg Care. 2011; 27:29–31.	PG	151		Resuscitation (BLS,ACLS,ATLS)	CogI, MLS, RepP	ID:Repetition	Sa	9	1
Bath J, et al. Standardization is superior to traditional methods of teaching open vascular simulation. J Vasc Surg. 2011; 53:229–223.e2	PG	18	RCT	Open surgery/suturing	MLS	ID:Sequence	Sa, K, SPc	12.5	5
Fraser K, et al. Simulation training improves diagnostic performance on a real patient with similar clinical findings. Chest. 2011; 139:376–381.	MS	57	RCT	Physical exam		ID:Clinical scenario	SPd	11.5	4
Guhde J. Nursing students’ perceptions of the effect on critical thinking, assessment, and learner satisfaction in simple versus complex high-fidelity simulation scenarios. J Nurs Educ. 2011; 50:73–78.	RN	133X		Nursing tasks	FB, GrpP, MLS	ID:Task variability	Sa	8	3
Maggio MP, et al. The effect of magnification loupes on the performance of preclinical dental students. Quintessence Int. 2011; 42:45–55.	D	232		Dentistry	MLS	SA:Visual	ST, SPc	11.5	3
Naughton PA, et al. Skills training after night shift work enables acquisition of endovascular technical skills on a virtual reality simulator. J Vasc Surg. 2011; 53:858–866.	PG	20		Endovascular proc.		ID:Timing	ST, SPc	10.5	2
Oermann MH, et al. Effects of monthly practice on nursing students’ CPR psychomotor skill performance. Resuscitation. 2011; 82:447–453	RN	495	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, DistP, FB, Indiv, MLS, RepP, Time	ID:Repetition	SPc	13.5	5
Snyder CW, et al. Effects of Virtual Reality Simulator Training Method and Observational Learning on Surgical Performance. World J Surg. 2011; 35:245–252.	MS	32	RCT	Min. invasive surg.,Endoscopy (GI,Urology,Bronch.)	FB, Indiv, MLS	ID:Sequence	ST	13.5	4
Sutton RM, et al. “Booster” training: Evaluation of instructor-led bedside cardiopulmonary resuscitation skill training and automated corrective feedback to improve cardiopulmonary resuscitation... Pediatr Crit Care Med. 2011; Online 2010 Jul 9; in press.	PG, RN	46	RCT	Resuscitation (BLS,ACLS,ATLS)	CogI, FB	ID:Feedback	SPc	12.5	4
Swanson EA, et al. Comparison of selected teaching strategies incorporating simulation and student outcomes. Clinical Simulation in Nursing. 2011; 7(3):e81–e90.	RN	96	RCT	Critical thinking	FB	In:Instructor intensity	SPc	14.5	4
Thompson JR, et al. Limited value of haptics in virtual reality laparoscopic cholecystectomy training. Surg Endosc. 2011; 25:1107–1114.	MS, O	8	RCT	Min. invasive surg.		SA:Tactile	ST, SPc	12	3
Uccelli J, et al. The validity of take-home surgical simulators to enhance resident technical skill proficiency. Am J Surg. 2011; 201:315–319.	PG	14	RCT	Min. invasive surg.	CogI, FB, Indiv, MLS, Time	ID:Sequence	ST, SPc	11.5	3
Zhao YC, et al. Can virtual reality simulator be used as a training aid to improve cadaver temporal bone dissection? Results of a randomized blinded control trial. Laryngoscope. 2011; 121:831–837.	PG	20	RCT	Endoscopy (GI,Urology,Bronch.)	FB, GrpP, Indiv, MLS	CM:VR vs model	SPc, SPd	13.5	4

Trainees: MS = medical student, PG = postgraduate physician trainee, MD = practicing physician, RN = nurse or nursing student, EMT = emergency medical technician/paramedic/first responder or EMT student, D = dentist or dental student, V = veterinarian or veterinary student, C = chiropractor or student, O = other/mixed.

N: Number of outcome observations; this is usually the number of trainees, but in some cases reflects the number of teams observed or the number of patient observations. X = Crossover.

RCT = randomized controlled trial.

Feature: Key features that varied between interventions (i.e. included in meta-analyses), see main text for definitions. Features that were coded the same for both groups (eg, both present or both absent) are not listed. ClinV = clinical variation; CogI = cognitive interactivity; Curr = curriculum integration; DistP = distributed practice; FB = feedback; GrpP = group practice; Indiv = individualization; Mast = mastery learning; MLS = multiple learning strategies; RangeD = range of difficulty; RepP = repetitive practice; Time = time learning.

Theme: The overall research theme (research question). Main themes are indicated first, with sub-themes following the colon. MA = modality added; GC = group composition; ID = instructional design; In = Instructor; CM = compare sim modality; SA = sensory augmentation.

Outcomes: Sa = Satisfaction, K = knowledge, ST = skill-time, SPc = skill-process, SPd = skill-product, BT = behavior-time, BP = behavior-process, P = patient effects.

Quality: MERSQI = Medical Education Research Study Quality Instrument (maximum score 18); NOS = modified Newcastle-Ottawa scale (maximum score 6).

Of course, such research will not be possible without adequate funding. Health professions education research is underfunded (Reed et al. 2005), even though funding is associated with higher quality work (Reed et al. 2007). Those responsible for funding decisions must recognize the importance of theory-building research that clarifies (Cook et al. 2008a) the modalities and features of simulation-based education that improve learner and patient outcomes with greatest effectiveness and at lowest cost.

Finally, we note that the effect sizes for these comparisons are much lower than those observed for comparisons with no intervention. This is not unexpected, as comparing training vs no training ought to result in greater improvement than comparing two active instructional interventions (Cook 2012a). However, we caution investigators that the small samples that proved sufficient to identify statistically significant differences in no-intervention-comparison studies will be inadequate for simulation–simulation research. Advance calculation of sample size, clear justification of educational significance, and use of confidence intervals when interpreting results will be essential. These, together with other research methods that minimize confounding, will facilitate studies that truly advance our understanding of how to improve healthcare through simulation-based education.

Practice points

• Evidence supports the following as best practices for simulation-based education: range of difficulty, repetitive practice, distributed practice, cognitive interactivity, multiple learning strategies, individualized learning, mastery learning, feedback, longer time, and clinical variation.

• Future research should clarify the mechanisms of effective simulation-based education: what works, for whom, in what contexts?

• Direct comparisons of alternate simulation-based education instructional designs can clarify these mechanisms.

Acknowledgements

Portions of this work were presented at the 2012 International Meeting on Simulation in Healthcare, in San Diego, California.

Declaration of interest: This work was supported by intramural funds, including an award from the Division of General Internal Medicine, Mayo Clinic. The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Appendix

Table A2.  Results of sensitivity analyses

Feature	No. studies	Pooled ES (95% CI)	p	I^2	Main analysis pooled ES
Non-time Skills
Clinical variation	16	0.18 (−0.14, 0.51)	0.27	83	0.20
Cog. interactivity	89	0.59 (0.44, 0.75)	0.00	88	0.65
Feedback	80	0.36 (0.21, 0.50)	0.00	87	0.44
Group practice	8	−0.30 (−0.56, −0.05)	0.02	75	−0.22
Individualization	59	0.46 (0.28, 0.64)	0.00	88	0.52
Mult. learning strat.	70	0.54 (0.40, 0.68)	0.00	85	0.62
Range of difficulty	20	0.64 (0.26, 1.03)	0.00	87	0.68
Repetitive practice	7	0.68 (−0.03, 1.38)	0.06	89	0.68
Behaviors and Patient Effects
Feedback	2	0.18 (−0.23, 0.58)	0.39	0	0.32

Notes: Several studies reporting non-time skills had 3 simulation arms. Since we could only compare 2 groups at once, we first included the interventions with the greatest between-group difference and then performed sensitivity analyses substituting the third intervention. The table above summarizes all results influenced by the alternate analyses. The main analysis pooled effect size (ES) is provided to aid in comparison.