151
Views
27
CrossRef citations to date
0
Altmetric
Review

How valid are commercially available medical simulators?

, , , , &
Pages 385-395 | Published online: 14 Oct 2014

Abstract

Background

Since simulators offer important advantages, they are increasingly used in medical education and medical skills training that require physical actions. A wide variety of simulators have become commercially available. It is of high importance that evidence is provided that training on these simulators can actually improve clinical performance on live patients. Therefore, the aim of this review is to determine the availability of different types of simulators and the evidence of their validation, to offer insight regarding which simulators are suitable to use in the clinical setting as a training modality.

Summary

Four hundred and thirty-three commercially available simulators were found, from which 405 (94%) were physical models. One hundred and thirty validation studies evaluated 35 (8%) commercially available medical simulators for levels of validity ranging from face to predictive validity. Solely simulators that are used for surgical skills training were validated for the highest validity level (predictive validity). Twenty-four (37%) simulators that give objective feedback had been validated. Studies that tested more powerful levels of validity (concurrent and predictive validity) were methodologically stronger than studies that tested more elementary levels of validity (face, content, and construct validity).

Conclusion

Ninety-three point five percent of the commercially available simulators are not known to be tested for validity. Although the importance of (a high level of) validation depends on the difficulty level of skills training and possible consequences when skills are insufficient, it is advisable for medical professionals, trainees, medical educators, and companies who manufacture medical simulators to critically judge the available medical simulators for proper validation. This way adequate, safe, and affordable medical psychomotor skills training can be achieved.

Introduction

Simulators for medical training have been used for centuries. More primitive forms of physical models were used before plastic mannequins and virtual systems (VS) were available.Citation1 Since then, simulation in medical education has been deployed for a variety of actions, such as assessment skills, injections, trauma and cardiac life support, anesthesia, intubation, and surgical skills (SuS).Citation2,Citation3 These actions require psychomotor skills, physical movements that are associated with cognitive processes.Citation4,Citation5 Among these psychomotor skills are skills that require (hand–eye) coordination, manipulation, dexterity, grace, strength, and speed. Studies show that medical skills training which requires physical actions can be optimally performed by actual practice in performing these actions, eg, instrument handling.Citation6 This is explained by the fact that when learning psychomotor skills, the brain and body co-adapt to improve the manual (instrument) handling. This way, the trainee learns which actions are correct and which are not.Citation5

Four main reasons to use simulators instead of traditional training in the operating room have been described.Citation6 Firstly, improved educational experience; when simulators are placed in an easily accessible location, they are available continuously. This overcomes the problem of dependency on the availability of an actual patient case. Simulators also allow easy access to a wide variety of clinical scenarios, eg, complications.Citation6 Secondly, patient safety; simulators allow the trainee to make mistakes, which can equip the resident with a basic skills level that would not compromise patient safety when continuing training in the operating room.Citation7Citation14 Thirdly, cost efficiency; the costs of setting up a simulation center are in the end often less than the costs of instructors’ training time, and resources required as part of the training.Citation6 Moreover, the increased efficiency of trainees when performing a procedure adds to the return on investment achieved by medical simulators, as Frost and Sullivan demonstrated.Citation15 Lastly, simulators offer the opportunity to measure performance and training progress objectively by integrated sensors that can measure, eg, task time, path length, and forces.Citation6,Citation9,Citation16Citation18

With the increased developments and experiences in research settings, a wide variety of simulators have become commercially available. The pressing question is whether improvements in performance on medical simulators actually translates into improved clinical performance on live patients. Commercially available simulators in other industries, such as aerospace, the military, business management, transportation, and nuclear power, have been demonstrated to be valuable for performance in real life situations.Citation19Citation23 Similarly, it is of high importance that medical simulators allow for the correct training of medical skills to improve real life performances. Lack of proper validation could imply that the simulator at hand does not improve skills or worse, could cause incorrect skills training.Citation24,Citation25

Since validation of a simulator is required to guarantee proper simulator training, the aim of this review is to determine the availability of medical simulators and whether they are validated or not, and to discuss their appropriateness. This review is distinctive as it categorizes simulators based on simulator type and validation level. In this way, it provides a complete overview of all sorts of available simulators and their degree of validation. This will offer hospitals and medical educators, who are considering the implementation of simulation training in their curriculum, guidelines on the suitability of various simulators to fulfil their needs and demands.

Methods

The approach to achieve the study goal was set as follows. Firstly, an inventory was made of all commercially available simulators that allow medical psychomotor skills training. Secondly, categories that represent medical psychomotor skills were identified and each simulator was placed in one of those categories. Each category will be discussed and illustrated with some representative simulators. Thirdly, validity levels for all available simulators were determined. Lastly, study designs of the validity studies were evaluated in order to determine the reliability of the results of the validity studies.

Inventory of medical simulators

The inventory of commercially available medical simulators was performed by searching the Internet using search engines Google and Yahoo, and the websites of professional associations of medical education (). The search terms were split up in categories to find relevant synonyms (). Combinations of these categorized keywords were used as search strategy. For each Internet search engine, a large number of “hits” were found. Relevant websites were selected using the following inclusion criteria: the website needs to be from the company that actually manufactures and sells the product; the simulator should be intended for psychomotor skills training in the medical field (this implies that the models, mannequins or software packages that only offer knowledge training or visualization were excluded); if the company’s website provided additional medical simulators, all products that fulfil the criteria were included separately; the website should have had its latest “update” after January 2009, so that it can be expected that the company is still actively involved in commercial activities in the field of medical simulators.

Table 1 List of societies and associations concerning medical education and simulation

Table 2 Search terms

Categorization of simulator type

For our study purpose, medical simulators were categorized based on their distinct characteristics: VS and physical mannequins with or without sensors ().Citation14,Citation26 VS are software based simulators. The software simulates the clinical environment that allows practicing individual clinical psychomotor skills. Most of these simulators have a physical interface and provide objective feedback to the user about their performance with task time as the most commonly used performance parameter.Citation26 The physical mannequins are mostly plastic phantoms simulating (parts of) the human body. The advantage of physical models is that the sense of touch is inherently present, which can provide a very realistic training environment. Most models do not provide integrated sensors and real-time feedback. These models require an experienced professional supervising the skills training. Some physical models have integrated sensors and computer software which allow for an objective performance assessment.Citation14,Citation27,Citation28 As these simulators take over part of the assessment of training progress, it might be expected that they are validated in a different manner. Therefore, a distinction was made between simulators that provide feedback and simulators that do not.

Figure 1 Schematic overview of the number of simulators per skills category (in brackets) and the number of simulators per simulator type (in brackets).

Abbreviations: VS, virtual systems; PM, physical model; MES, manual patient examination skills; IPIS, injections, needle punctures, and intravenous catheterization skills; BLSS, basic life support skills; SuS, surgical skills.
Figure 1 Schematic overview of the number of simulators per skills category (in brackets) and the number of simulators per simulator type (in brackets).

Categorization of medical psychomotor skills

Skills were categorized in the following categories as they are the most distinct psychomotor skills medical professionals will learn during their education starting at BSc level: 1) manual patient examination skills (MES): an evaluation of the human body and its functions that requires direct physical contact between physician and patient; 2) injections, needle punctures, and intravenous catheterization (peripheral and central) skills (IPIS): the manual process of insertion of a needle into human skin tissue for different purposes such as taking blood samples, lumbar or epidural punctures, injections or vaccinations, or the insertion of a catheter into a vein; 3) basic life support skills (BLSS).Citation29,Citation30 BLSS refers to maintaining an open airway and supporting breathing and circulation, which can be further divided into the following psychomotor skills: continued circulation, executed by chest compression and cardiac massage; opening the airway, executed by manually tilting the head and lifting the chin; continued breathing, executed by closing the nose, removal of visible obstructions, mouth-to-mouth ventilation, and feeling for breathing;Citation31 4) SuS: indirect tissue manipulation for diagnostic or therapeutic treatment by means of medical instruments, eg, scalpels, forceps, clamps, and scissors. Surgical procedures can cause broken skin, contact with mucosa or internal body cavities beyond a natural or artificial body orifice, and are subdivided into minimally invasive and open procedures.

Inventory of validation and study design quality assessment

The brand name of all retrieved simulators added to the keyword “simulator” was used to search PubMed for scientific evidence on validity of that particular simulator. After scanning the abstract, validation studies were included and the level of validation of that particular simulator was noted.Citation32 Studies were scored for face validityCitation24,Citation32,Citation33 (the most elementary level), construct validity,Citation33 concurrent validity,Citation24,Citation32 and the most powerful level, predictive validity.Citation24,Citation32,Citation33

The validation studies were evaluated for their study design using Issenberg’s guidelines for educational studies involving simulators ().Citation34 Each study was scored for several aspects concerning the research question, participants, methodology, outcome measures, and manner of scoring (). An outcome measure is considered appropriate when it is clearly defined and measured objectively.

Table 3 Checklist for the evaluation of validation study, using Issenberg’s guidelines for educational studies involving simulators

Table 4 Outcome measures to test the efficacy of the simulator (the parameters indicate psychomotor skills performance)

The validation studies demonstrated substantial heterogeneity in study design, therefore, analysis of the data was performed qualitatively and trends were highlighted.

Results

Inventory and categorization of medical simulators

In total, 433 commercially available simulators were found (see Supplementary material), offered by 24 different companies. From these simulators, 405 (93.5%) are physical models and 28 (6.5%) are virtual simulators (). An almost equal distribution of simulators is available for each of the four defined skills categories (), with the SuS category containing the noticeably highest portion of virtual reality simulators (86%). Objective feedback was provided by the simulator itself in 65 cases (15%).

Simulators for patient examination (MES) training provide the possibility for physical care training, eg, respiratory gas exchange, intubation, and anesthesia delivery.Citation28,Citation35Citation38 The typical simulators in this category predominantly consist of (full body) mannequins that have anatomical structures and simulate physiological functions such as respiration, and peripheral pulses (eg, Supplementary material: simulators 3 and 21).

IPIS simulators provide training on needle punctures and catheterization. Such simulators usually consist of a mimicked body part, eg, an arm or a torso. An example is the Lumbar Puncture simulator (Kyoto Kagaku Co., Kitanekoya-cho Fushimi-ku Kyoto, Japan).Citation39,Citation40 This simulator consists of a life-like lower torso with a removable “skin” that does not show the marks caused by previous needle punctures. Integral to the simulator is a replaceable “puncture block”, which can represent different types of patients (eg, “normal”, “obese”, “elderly”), and which is inserted under the “skin”.Citation41

BLSS simulators allow for emergency care skills training, such as correct head tilt and chin lift, application of cervical collars, splints, and traction or application to spine board.Citation42 These simulators predominantly consist of full body mannequins having primary features such as anatomically correct landmarks, articulated body parts to manipulate the full range of motion, removable mouthpieces and airways, permitting the performance of chest compressions, oral or nasal intubation, and simulated carotid pulse (eg, Supplementary material: simulators 243, 245, and 270). SuS simulators are used for skills training required when performing open or minimally invasive surgery, like knot tying, suturing, instrument and tissue handling, dissection, simple and complex wound closure. Both physical and virtual simulators form part of this category. A representative example of a physical simulator is the life-sized human torso with thoracic and abdominal cavities and neck/trachea. Such a model is suited to provide training on a whole open surgery procedure, including preparing the operative area, (local) anesthesia, tube insertion, and closure (eg, Supplementary material: 355 and 357). The torso is covered with a polymer that mimics the skin and contains red fluid that mimics blood. Virtual reality systems start to take an important place in minimally invasive surgical procedure training, especially for hand–eye co-ordination training. The VS provide instrument handles with or without a phantom limb and a computer screen in which a virtual scene is presented (eg, the Symbionix simulators 316–322 [Simbionix, Cleveland, OH, USA] and the Simendo simulators 425–426 [Simendo B.V., Rotterdam, the Netherlands] in the Supplementary material). Software provides a ‘‘plug-and-play’’ connection to a personal computer via a USB port.Citation43,Citation44

Inventory of validation and study design quality assessment

One hundred and thirty validation studies evaluated 35 commercially available medical simulators for levels of validity ranging from face to predictive validity (). From these 35 simulators, two (5.7%) simulators were tested for face validity, four (11.4%) for content validity, seven (20%) for construct validity, 14 (40%) for concurrent validity and 8 (22.9%) for predictive validity (). References of the validated simulators are shown in the Supplementary material (between brackets). Twenty-four (37%) simulators that provide objective feedback have been validated, from which six occurred in MES, one in IPIS and 17 in SuS ().

Figure 2 The number of validated simulators.

Notes: Arrangement is based on skills category, level of validation, and whether the simulator gives feedback or not. Nine MES simulators, three IPIS simulators, one BLSS simulator and 22 SuS simulators are validated.
Abbreviations: MES, manual patient examination skills; IPIS, injections, needle punctures, and intravenous catheterization skills; BLSS, basic life support skills; SuS, surgical skills.
Figure 2 The number of validated simulators.

The numbers of validated simulators per category were substantially different, as was the level of validity (). SuS simulators were most validated (62.9%), and most frequently for the highest validity level (, predictive validity). MES simulators were primarily tested for content and concurrent validity. The proportion of validated IPIS and BLSS simulators was small ().

The quality of the study designs was verified for ten important aspects. Although all studies clearly described the researched question, study population, and outcome measures, few studies met all other criteria on the checklist. Most studies did not perform a power analysis to guarantee a correct number of participants before inclusion. Twelve percent of the 130 studies used a standardized assessment system or performed blind assessment. The majority of the studies (111) performed a correct selection of subjects: either based on experience level or with a randomly selected control group. However, 20 studies did not select their control group randomly or had no control group at all () (37 studies tested face or content validity).Citation45Citation48

Each study used proper outcome measures to test the efficacy of the simulator, which indicated psychomotor skills performance. The most commonly used performance measures are depicted in . To assess performance data objectively the following standardized scoring methods were used: team leadership-interpersonal skills (TLIS) and emergency clinical care scales (ECCS),Citation49 objective structural clinical examination (OSCE),Citation27,Citation50 objective structured assessment of technical skills (OSAT),Citation39,Citation51Citation55 and global rating scale (GRS).Citation56Citation59 All other studies used assessment methods that were developed specifically for that study.

Methodologically speaking, the studies that tested concurrent and predictive validity outperformed the studies that tested face, content, and construct validity.

Discussion

This study reviewed the availability of medical simulators, their validity level, and the reliability of the study designs. Four hundred and thirty-three commercially available simulators were found, of which 405 (94%) were physical models. Evidence of validation was found for 35 (6.5%) simulators (). Mainly in category two and three, the number of validated simulators was marginal. Solely SuS simulators were validated for the highest validity level. Sixty-three percent of the 65 simulators that provide feedback on performance have not been validated, which is remarkable as these simulators take over part of the supervisors’ judgment. Studies that tested more powerful levels of validity (concurrent and predictive validity) were methodologically stronger than studies that tested more elementary levels of validity (face, content, and construct validity).

These findings can partly be explained: the necessity of a high level validation and the extent to which simulators need to mimic reality is firstly dependent on the type of skills training, and secondly on the possible consequences for patients when medical psychomotor skills are insufficient. This could especially be the case for SuS skills, because minimally invasive SuS are presumably most distinct from daily use of psychomotor skills, and as a result not well developed. In addition, when these skills are taught incorrectly, it can have serious consequences for the patient, eg, if a large hemorrhage occurs as a result of an incorrect incision. To guarantee patient safety, it is important that simulators designed for this type of training demonstrate high levels of validity.Citation60,Citation61 For other types of skills, such as patient examination, a lower validity level can be acceptable, because these skills are closer related to everyday use of psychomotor skills, and solely require a basic level of training on a simulator, which can be quickly adapted in a real-life situation.Citation45,Citation46 Moreover, it requires less extensive methodology to determine face validity than to determine predictive validity.

Certain factors made it difficult to score all validity studies on equal terms; substantial heterogeneity exists among the studies. However, in general, it can be stated that a substantial part of the validation studies showed methodological flaws. For example, many studies did not describe a power analysis, so it was difficult to judge whether these studies included the correct number of participants. Furthermore, only 15 of 130 studies used standardized assessment methods and blinded assessors. Unvalidated assessment methods and unblinded ratings are less objective, which affects reliability and validity of the test.Citation26 This raises the question whether the presented studies were adequate enough to determine the validity level of a certain simulator. Future validity studies should focus on a proper study design, in order to increase the reliability of the results.

There are several limitations to our study. Firstly, our inventory of commercially available medical simulators was performed solely by searching the Internet. We did not complement our search by contacting manufacturers or by visiting conferences. This might implicate that our list of available simulators is not complete. Secondly, the available level of validity for the simulators was also determined by searching public scientific databases. Quite possibly, manufacturers have performed validity tests with a small group of experts, but refrained from publishing the results. It is also possible that studies have been rejected for publication or have not been published yet. Therefore, the total number of simulators and number of validated simulators that was found, might be underestimated. However, this does not undermine the fact that few simulators were validated. Especially high levels of validation are scanty.

Our results should firstly make medical trainers aware of the fact that a low number of simulators are actually tested, while validation is truly important. Although it is possible that unvalidated simulators provide proper training, validity of a device is a condition to guarantee proper acquisition of psychomotor skillsCitation1,Citation6,Citation7Citation9,Citation18 and lack of validity brings the risk of acquisition of improper skills.Citation1,Citation35 Secondly, a simulator that provides feedback independent of a professional supervisor, should have been validated to guarantee that the provided feedback is adequate and appropriate in real-life settings.Citation1,Citation63 Thirdly, for reliable results of validity studies, proper study design is required. Well conducted studies have shown to be limited so far. Lastly, it is necessary to determine the type of skills educators will offer to their trainees with a simulator and the level of validity that is required to guarantee adequate training.

Our plea is for researchers to collaborate with manufacturers to develop questionnaires and protocols to test newly developed simulators. Simulators from the same category can be tested simultaneously with a large group of relevant participants.Citation43 When objective evidence for basic levels of validity is obtained, it is important to publish the results so that this information is at the disposal of medical trainers. Before introducing a simulator in the training curriculum, it is recommended to first consider which skills training is needed, and the complexity and possible clinical consequences of executing those skills incorrectly. Subsequently, the minimum required level of validity should be determined for the simulator that allows for that type of skills training. The qualitative results support the concept that the level of validation depends on the difficulty level of skills training and the unforeseen consequences when skills are insufficient or lead to erroneous actions. This combination of selection criteria should guide medical trainers in the proper selection of a simulator for safe and adequate training.

Conclusion

For correct medical psychomotor skills training and to provide objective and correct feedback it is essential to have a realistic training environment. Scientific testing of simulators is an important way to prove and validate the training method. This review shows that 93.5% of the commercially available simulators are not known to be tested for validity, which implies that no evidence is available that they actually improve individual medical psychomotor skills. From the validity studies that were done for 35 simulators, many show some methodological flaws, which weaken the reliability of the results. It is also advisable for companies that manufacture medical simulators to validate their products and provide scientific evidence to their customers. This way, a quality system becomes available, which contributes to providing adequate, safe, and affordable medical psychomotor skills training.

Disclosure

This research was funded by the Marti-Keuning Eckhart Foundation, Lunteren, the Netherlands. The authors report no conflict of interest.

References

  • WilfongDNFalsettiDJMcKinnonJLDanielLHWanQCThe effects of virtual intravenous and patient simulator training compared to the traditional approach of teaching nurses: a research project on peripheral iv catheter insertionJ Infus Nurs2011341556221239952
  • GormanPJMeierAHKrummelTMSimulation and virtual reality in surgical education: real or unreal?Arch Surg1999134111203120810555634
  • AhmedRNaqviZWolfhagenIPsychomotor skills for the undergraduate medical curriculum in a developing country – PakistanEduc Health (Abingdon)200518151315804641
  • WolpertDMGhahramaniZComputational principles of movement neuroscienceNat Neurosci20053Suppl1212121711127840
  • WolpertDMGhahramaniZFlanaganJRPerspectives and problems in motor learningTrends Cogn Sci200151148749411684481
  • KunklerKThe role of medical simulation: an overviewInt J Med Robot20062320321017520633
  • CannonWDEckhoffDGGarrettWEJrHunterRESweeneyHJReport of a group developing a virtual reality simulator for arthroscopic surgery of the knee jointClin Orthop Relat Res2006442212916394734
  • HengPAChengCYWongTTVirtual reality techniques. Application to anatomic visualization and orthopaedics trainingClin Orthop Relat Res200644251216394732
  • HowellsNRGillHSCarrAJPriceAJReesJLTransferring simulated arthroscopic skills to the operating theatre: a randomised blinded studyJ Bone Joint Surg Br200890449449918378926
  • McCarthyADMoodyLWaterworthARBickerstaffDRPassive haptics in a knee arthroscopy simulator: is it valid for core skills training?Clin Orthop Relat Res2006442132016394733
  • MichelsonJDSimulation in orthopaedic education: an overview of theory and practiceJ Bone Joint Surg Am20068861405141116757778
  • PossRMabreyJDGilloglySDKasserJRDevelopment of a virtual reality arthroscopic knee simulatorJ Bone Joint Surg Am200082-A101495149911057478
  • SafirODubrowskiAMirksyCBacksteinDCarnahanHWhat skills should simulation training in arthroscopy teach residents?Int J CARS20083433437
  • TuijthofGJvan SterkenburgMNSiereveltINvanOJVan DijkCNKerkhoffsGMFirst validation of the PASSPORT training environment for arthroscopic skillsKnee Surg Sports Traumatol Arthrosc201018221822419629441
  • Frost and SullivanReturn on Investment Study for Medical Simulation Training: Immersion Medical, Inc. Laparoscopy AccutouchÚ System2012 Available from: http://www.healthleadersmedia.com/content/138774.pdfAccessed June 23, 2014
  • InselACarofinoBLegerRArcieroRMazzoccaADThe development of an objective model to assess arthroscopic performanceJ Bone Joint Surg Am20099192287229519724008
  • TuijthofGJVisserPSiereveltINvan DijkCNKerkhoffsGMDoes perception of usefulness of arthroscopic simulators differ with levels of experience?Clin Orthop Relat Res201146961701170821290203
  • HoremanTRodriguesSPJansenFWDankelmanJvan den DobbelsteenJJForce measurement platform for training and assessment of laparoscopic skillsSurg Endosc201024123102310820464416
  • IssenbergSBMcGaghieWCHartIRSimulation technology for health care professional skills training and assessmentJAMA1999282986186610478693
  • GoodmanWThe world of civil simulatorsFlight International Magazine197818435
  • StreufertSPogashRPiaseckiMSimulation-based assessment of managerial competence: reliability and validityPersonnel Psychology1988413537557
  • RolfeJMStaplesKJFlight simulationCambridge University Press1988
  • KeysBWolfeJThe Role of Management Games and Simulations in Education and ResearchJournal of Management1990162307336
  • SchijvenMPJakimowiczJJValidation of virtual reality simulators: Key to the successful integration of a novel teaching technology into minimal access surgeryMinim Invasive Ther Allied Technol200514424424616754170
  • ChmarraMKKleinSde WinterJCJansenFWDankelmanJObjective classification of residents based on their psychomotor laparoscopic skillsSurg Endosc20102451031103919915915
  • van HovePDTuijthofGJVerdaasdonkEGStassenLPDankelmanJObjective assessment of technical surgical skillsBr J Surg201097797298720632260
  • AlinierGHuntWBGordonRDetermining the value of simulation in nurse education: study design and initial resultsNurse Educ Pract20044320020719038158
  • DonoghueAJDurbinDRNadelFMStryjewskiGRKostSINadkarniVMEffect of high-fidelity simulation on Pediatric Advanced Life Support training in pediatric house staff: a randomized trialPediatr Emerg Care200925313914419262421
  • MonsieursKGDeRMSchelfoutSEfficacy of a self-learning station for basic life support refresher training in a hospital: a randomized controlled trialEur J Emerg Med201219421421921897264
  • European Society for Emergency Medicine [homepage on the Internet]Emergency Medicine2012 Available from: http://www.eusem.org/whatisem/Accessed June 23, 2014
  • HandleyAJBasic life supportBr J Anaesth19977921511589349125
  • OropesaISanchez-GonzalezPLamataPMethods and tools for objective assessment of psychomotor skills in laparoscopic surgeryJ Surg Res20111711e81e9521924741
  • CarterFJSchijvenMPAggarwalRConsensus guidelines for validation of virtual reality surgical simulatorsSurg Endosc200519121523153216252077
  • IssenbergSBMcGaghieWCWaughRAComputers and evaluation of clinical competenceAnn Intern Med1999130324424510049216
  • MartinJTRedaHDorityJSZwischenbergerJBHassanZUSurgical resident training using real-time simulation of cardiopulmonary bypass physiology with echocardiographyJ Surg Educ201168654254622000542
  • TanGMTiLKSureshSHoBSLeeTLTeaching first-year medical students physiology: does the human patient simulator allow for more effective teaching?Singapore Med J200243523824212188075
  • NehringWMLashleyFRCurrent use and opinions regarding human patient simulators in nursing education: an international surveyNurs Educ Perspect200425524424815508564
  • FernandezGLLeePCPageDWD’AmourEMWaitRBSeymourNEImplementation of full patient simulation training in surgical residencyJ Surg Educ201067639339921156297
  • BathJLawrencePChandraAStandardization is superior to traditional methods of teaching open vascular simulationJ Vasc Surg201153122923421115317
  • UppalVKearnsRJMcGradyEMEvaluation of M43B Lumbar puncture simulator-II as a training tool for identification of the epidural space and lumbar punctureAnaesthesia201166649349621568983
  • UppalJKVarshneyRHazariPPChuttaniKKaushikNKMishraAKBiological evaluation of avidin-based tumor pretargeting with DOTA-Triazole-Biotin constructed via versatile Cu(I) catalyzed click chemistryJ Drug Target201119641842620678008
  • Laerdal, helping saves lives2012 Available from: http://www.laerdal.com/nl/Accessed August 8, 2014
  • VerdaasdonkEGStassenLPMontenyLJDankelmanJStandardization is superior to traditional methods of teaching open vascular simulationSurg Endosc200620351151816437275
  • VerdaasdonkEGStassenLPSchijvenMPDankelmanJConstruct validity and assessment of the learning curve for the SIMENDO endoscopic simulatorSurg Endosc20072181406141217653815
  • VermaABhattHBootonPKneeboneRThe Ventriloscope(R) as an innovative tool for assessing clinical examination skills: appraisal of a novel method of simulating auscultatory findingsMed Teach2011337e388e39621696273
  • WilsonMShepherdIKellyCPitznerJAssessment of a low-fidelity human patient simulator for the acquisition of nursing skillsNurse Educ Today2005251566715607248
  • KrugerAGillmannBHardtCDoringRBeckersSKRossaintRVermittlung von „soft skills“ für Belastungssituationen. [Teaching non-technical skills for critical incidents: Crisis resource management training for medical students]Anaesthesist2009586582588 German19189061
  • ShuklaAKlineDCherianAA simulation course on lifesaving techniques for third-year medical studentsSimul Healthc200721111519088603
  • PascualJLHolenaDNVellaMAShort simulation training improves objective skills in established advanced practitioners managing emergencies on the ward and surgical intensive care unitJ Trauma201171233033721825935
  • KarnathBThorntonWFryeAWTeaching and testing physical examination skills without the use of patientsAcad Med200277775312114177
  • NadlerILileyHGSandersonPMClinicians can accurately assign Apgar scores to video recordings of simulated neonatal resuscitationsSimul Healthc20105420421221330798
  • ChouDSAbdelshehidCClaymanRVMcDougallEMComparison of results of virtual-reality simulator and training model for basic ureteroscopy trainingJ Endourol200620426627116646655
  • LucasSTuncelABensalahKVirtual reality training improves simulated laparoscopic surgery performance in laparoscopy naive medical studentsJ Endourol20082251047105118643722
  • KoretsRMuesACGraversenJAValidating the use of the Mimic dV-trainer for robotic surgery skill acquisition among urology residentsUrology20117861326133022001096
  • AdlerMDTrainorJLSiddallVJMcGaghieWCDevelopment and evaluation of high fidelity simulation case scenarios for pediatric resident educationAmbul Pediatr20077218218617368414
  • SchoutBMAnaniasHJBemelmansBLTransfer of cysto-urethroscopy skills from a virtual-reality simulator to the operating room: a randomized controlled trialBJU Int2010106222623119912184
  • SchoutBMMuijtjensAMAcquisition of flexible cystoscopy skills on a virtual reality simulator by experts and novicesBJU Int2010105223423919583729
  • KnollTTrojanLHaeckerAAlkenPMichelMSValidation of computer-based training in ureterorenoscopyBJU Int20059591276127915892816
  • ClaymanRVAssessment of basic endoscopic performance using a virtual reality simulatorJ Urol20031702 Pt 169214601606
  • Inspectie voor de Gezondheidszorg [homepage on the Internet]Rapport ‘Risico’s minimaal invasieve chirurgie onderschat, kwaliteitssysteem voor laparoscopische operaties ontbreekt’ [Healthcare Inspectorate. Risks minimally invasive surgery underestimated.]2012 Available from: http://www.igz.nl/zoeken/document.aspx?doc=Rapport+'Risico's+minimaal+invasieve+chirurgie+onderschat%2C+kwaliteitssysteem+voor+laparoscopische+operaties+ontbreekt'&docid=475Accessed June 23, 2014
  • IssenbergSBMcGaghieWCPetrusaERLeeGDScaleseRJFeatures and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic reviewMed Teach2005271102816147767
  • RosenthalRGantertWAHamelCAssessment of construct validity of a virtual reality laparoscopy simulatorJ Laparoendosc Adv Surg Tech A200717440741317705717
  • KuduvalliPMJervisATigheSQRobinNMConstruct validity and assessment of the learning curve for the SIMENDO endoscopic simulatorAnaesthesia200863436436918336486
  • OddoneEZWaughRASamsaGCoreyRFeussnerJRTeaching cardiovascular examination skills: results from a randomized controlled trialAm J Med19939543893968213871
  • JonesJSHuntSJCarlsonSASeamonJPAssessing bedside cardiologic examination skills using “Harvey,” a cardiology patient simulatorAcad Emerg Med19974109809859332631
  • Giovanni deGDRobertsTNormanGRelative effectiveness of high-versus low-fidelity simulation in learning heart soundsMed Educ200943766166819573189
  • EndeAZopfYKonturekPStrategies for training in diagnostic upper endoscopy: a prospective, randomized trialGastrointest Endosc201275225426022153875
  • FerlitschAGlauningerPGupperAEvaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopyEndoscopy200234969870212195326
  • RitterEMMcCluskyDAIIILedermanABGallagherAGSmithCDObjective psychomotor skills assessment of experienced and novice flexible endoscopists with a virtual reality simulatorJ Gastrointest Surg20037787187714592660
  • EversbuschAGrantcharovTPLearning curves and impact of psychomotor training on performance in simulated colonoscopy: a randomized trial using a virtual reality endoscopy trainerSurg Endosc200418101514151815791380
  • FelsherJJOlesevichMFarresHValidation of a flexible endoscopy simulatorAm J Surg2005189449750015820469
  • GrantcharovTPCarstensenLSchulzeSObjective assessment of gastrointestinal endoscopy skills using a virtual reality simulatorJSLS20059213013315984697
  • KochADBuzinkSNHeemskerkJExpert and construct validity of the Simbionix GI Mentor II endoscopy simulator for colonoscopySurg Endosc200822115816217516114
  • ShiraiYYoshidaTShiraishiRProspective randomized study on the use of a computer-based endoscopic simulator for training in esophagogastroduodenoscopyJ Gastroenterol Hepatol2008237 Pt 11046105018554236
  • BuzinkSNGoossensRHSchoonEJdeRHJakimowiczJJDo basic psychomotor skills transfer between different image-based procedures?World J Surg201034593394020151134
  • FayezRFeldmanLSKanevaPFriedGMTesting the construct validity of the Simbionix GI Mentor II virtual reality colonoscopy simulator metrics: module mattersSurg Endosc20102451060106519911225
  • Van SickleKRBuckLWillisRA multicenter, simulation-based skills training collaborative using shared GI Mentor II systems: results from the Texas Association of Surgical Skills Laboratories (TASSL) flexible endoscopy curriculumSurg Endosc20112592980298621487880
  • GettmanMTLeCQRangelLJSlezakJMBergstralhEJKrambeckAEAnalysis of a computer based simulator as an educational tool for cystoscopy: subjective and objective resultsJ Urol2008179126727118001785
  • OganKJacomidesLShulmanMJRoehrbornCGCadedduJAPearleMSVirtual ureteroscopy predicts ureteroscopic proficiency of medical students on a cadaverJ Urol2004172266767115247757
  • ShahJMontgomeryBLangleySDarziAValidation of a flexible cystoscopy courseBJU Int200290983388512460341
  • ShahJDarziAVirtual reality flexible cystoscopy: a validation studyBJU Int200290982883212460340
  • MishraSKurienAPatelRValidation of virtual reality simulation for percutaneous renal access trainingJ Endourol201024463564020218892
  • Vitish-SharmaPKnowlesJPatelBAcquisition of fundamental laparoscopic skills: is a box really as good as a virtual reality trainer?Int J Surg20119865966121964217
  • ZhangAHunerbeinMDaiYSchlagPMBellerSConstruct validity testing of a laparoscopic surgery simulator (Lap Mentor): evaluation of surgical skill with a virtual laparoscopic training simulatorSurg Endosc20082261440144417972134
  • YamaguchiSKonishiKYasunagaTConstruct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulatorSurg Endosc200721122253225717479319
  • AyodejiIDSchijvenMJakimowiczJGreveJWFace validation of the Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculumSurg Endosc20072191641164917356944
  • AndreattaPBWoodrumDTBirkmeyerJDLaparoscopic skills are improved with LapMentor training: results of a randomized, double-blinded studyAnn Surg2006243685486016772789
  • HallABRandomized objective comparison of live tissue training versus simulators for emergency proceduresAm Surg201177556156521679588
  • DayanABZivABerkenstadtHMunzYA simple, low-cost platform for basic laparoscopic skills trainingSurg Innov200815213614218492732
  • BoonJRSalasNAvilaDBooneTBLipshultzLILinkREConstruct validity of the pig intestine model in the simulation of laparoscopic urethrovesical anastomosis: tools for objective evaluationJ Endourol200822122713271619099517
  • YoungbloodPLSrivastavaSCuretMHeinrichsWLDevPWrenSMComparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performanceJ Am Coll Surg2005200454655115804468
  • TaffinderNSuttonCFishwickRJMcManusICDarziAValidation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: results from randomised controlled studies using the MIST VR laparoscopic simulatorStud Health Technol Inform19985012413010180527
  • JordanJAGallagherAGMcGuiganJMcClureNVirtual reality training leads to faster adaptation to the novel psychomotor restrictions encountered by laparoscopic surgeonsSurg Endosc200115101080108411727074
  • McNattSSSmithCDA computer-based laparoscopic skills assessment device differentiates experienced from novice laparoscopic surgeonsSurg Endosc200115101085108911727075
  • KothariSNKaplanBJDeMariaEJBroderickTJMerrellRCTraining in laparoscopic suturing skills using a new computer-based virtual reality simulator (MIST-VR) provides results comparable to those with an established pelvic trainer systemJ Laparoendosc Adv Surg Tech A200212316717312184901
  • SeymourNEGallagherAGRomanSAVirtual reality training improves operating room performance: results of a randomized, double-blinded studyAnn Surg2002236445846312368674
  • GrantcharovTPKristiansenVBBendixJBardramLRosenbergJFunch-JensenPRandomized clinical trial of virtual reality simulation for laparoscopic skills trainingBr J Surg200491214615014760660
  • MaithelSSierraRKorndorfferJConstruct and face validity of MIST-VR, Endotower, and CELTS: are we ready for skills assessment using simulators?Surg Endosc200620110411216333535
  • HackethalAImmenrothMBurgerTEvaluation of target scores and benchmarks for the traversal task scenario of the Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR) laparoscopy simulatorSurg Endosc200620464565016424991
  • TanoueKIeiriSKonishiKEffectiveness of endoscopic surgery training for medical students using a virtual reality simulator versus a box trainer: a randomized controlled trialSurg Endosc200822498599017710487
  • DebesAJAggarwalRBalasundaramIJacobsenMBA tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skillsAm J Surg2010199684084520079480
  • BotdenSMde HinghIHJakimowiczJJMeaningful assessment method for laparoscopic suturing training in augmented realitySurg Endosc200923102221222819118427
  • SrivastavaSYoungbloodPLRawnCHaririSHeinrichsWLLaddALInitial evaluation of a shoulder arthroscopy simulator: establishing construct validityJ Shoulder Elbow Surg200413219620514997099
  • BrewinJNedasTChallacombeBElhageOKeisuJDasguptaPFace, content and construct validation of the first virtual reality laparoscopic nephrectomy simulatorBJU Int2010106685085420128776
  • TorkingtonJSmithSGReesBDarziAThe role of the basic surgical skills course in the acquisition and retention of laparoscopic skillSurg Endosc200115101071107511727072
  • SchijvenMPJakimowiczJThe learning curve on the Xitact LS 500 laparoscopy simulator: profiles of performanceSurg Endosc200418112112714625738
  • SchijvenMJakimowiczJFace-, expert, and referent validity of the Xitact LS500 laparoscopy simulatorSurg Endosc200216121764177012098029
  • SchijvenMJakimowiczJConstruct validity: experts and novices performing on the Xitact LS500 laparoscopy simulatorSurg Endosc200317580381012582752
  • BajkaMTuchschmidSFinkDSzekelyGHardersMEstablishing construct validity of a virtual-reality training simulator for hysteroscopy via a multimetric scoring systemSurg Endosc2010241798819551434
  • BajkaMTuchschmidSStreichMFinkDSzekelyGHardersMEvaluation of a new virtual-reality training simulator for hysteroscopySurg Endosc20092392026203318437471
  • BuzinkSNBotdenSMHeemskerkJGoossensRHdeRHJakimowiczJJCamera navigation and tissue manipulation; are these laparoscopic skills related?Surg Endosc200923475075718626705
  • BuzinkSNGoossensRHdeRHJakimowiczJJTraining of basic laparoscopy skills on SimSurgery SEPMinim Invasive Ther Allied Technol2010191354120095896
  • van der MeijdenOABroedersIASchijvenMPThe SEP “robot”: a valid virtual reality robotic simulator for the Da Vinci Surgical System?Surg Technol Int201019515820437345
  • WohaibiEMBushRWEarleDBSeymourNESurgical resident performance on a virtual reality simulator correlates with operating room performanceJ Surg Res20101601677219261297
  • BayonaSFernández-ArroyoJMMartínIBayonaPAssessment study of insight ARTHRO VR arthroscopy virtual training simulator: face, content, and construct validitiesJournal of Robotic Surgery200823151158
  • HassonHMKumariNVEekhoutJTraining simulator for developing laparoscopic skillsJSLS20015325525611548833
  • FicheraAPrachandVKivesSLevineRHassonHPhysical reality simulation for training of laparoscopists in the 21st century. A multi-specialty, multi-institutional studyJSLS20059212512915984696
  • HassonHMSimulation training in laparoscopy using a computerized physical reality simulatorJSLS200812436336719275849
  • HungAJZehnderPPatilMBFace, content and construct validity of a novel robotic surgery simulatorJ Urol201118631019102421784469
  • LernerMAAyalewMPeineWJSundaramCPDoes training on a virtual reality robotic simulator improve performance on the da Vinci surgical system?J Endourol201024346747220334558
  • KenneyPAWszolekMFGouldJJLibertinoJAMoinzadehAFace, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgeryUrology20097361288129219362352
  • IwataNFujiwaraMKoderaYConstruct validity of the LapVR virtual-reality surgical simulatorSurg Endosc201125242342820585960
  • SethiASPeineWJMohammadiYSundaramCPValidation of a novel virtual reality robotic simulatorJ Endourol200923350350819265469
  • MahmoodTDarziAA study to validate the colonoscopy simulatorSurg Endosc200317101583158912915972
  • AhlbergGHultcrantzRJaramilloELindblomAArvidssonDVirtual reality colonoscopy simulation: a compulsory practice for the future colonoscopist?Endoscopy200537121198120416329017
  • RoweRCohenRAAn evaluation of a virtual reality airway simulatorAnesth Analg2002951626612088944
  • BroeDRidgwayPFJohnsonSTierneySConlonKCConstruct validation of a novel hybrid surgical simulatorSurg Endosc200620690090416738979
  • Van SickleKRMcCluskyDAIIIGallagherAGSmithCDConstruct validation of the ProMIS simulator using a novel laparoscopic suturing taskSurg Endosc20051991227123116025195
  • PellenMGHorganLFBartonJRAttwoodSEConstruct validity of the ProMIS laparoscopic simulatorSurg Endosc200923113013918648875
  • BotdenSMBerlageJTSchijvenMPJakimowiczJJFace validity study of the ProMIS augmented reality laparoscopic suturing simulatorSurg Technol Int200817263218802880
  • JonssonMNMahmoodMAskerudTProMIS can serve as a da Vinci(R) simulator – a construct validity studyJ Endourol201125234535021114413
  • ChandraVNehraDParentRA comparison of laparoscopic and robotic assisted suturing performance by experts and novicesSurgery2010147683083920045162
  • FeiferADelisleJAnidjarMHybrid augmented reality simulator: preliminary construct validation of laparoscopic smoothness in a urology residency programJ Urol200818041455145918710760
  • CesanekPUchalMUranuesSDo hybrid simulator-generated metrics correlate with content-valid outcome measures?Surg Endosc200822102178218318622566
  • NearyPCBoyleEDelaneyCPSenagoreAJKeaneFBGallagherAGConstruct validation of a novel hybrid virtual-reality simulator for training and assessing laparoscopic colectomy; results from the first course for experienced senior laparoscopic surgeonsSurg Endosc200822102301230918553207
  • RitterEMKindelanTWMichaelCPimentelEABowyerMWConcurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS)Surg Endosc20072181441144517593461
  • WoodrumDTAndreattaPBYellamanchilliRKFeryusLGaugerPGMinterRMConstruct validity of the LapSim laparoscopic surgical simulatorAm J Surg20061911283216399102
  • DuffyAJHogleNJMcCarthyHConstruct validity for the LAPSIM laparoscopic surgical simulatorSurg Endosc200519340140515624062
  • VerdaasdonkEGDankelmanJLangeJFStassenLPTransfer validity of laparoscopic knot-tying training on a VR simulator to a realistic environment: a randomized controlled trialSurg Endosc20082271636164218027030
  • DayalRFariesPLLinSCComputer simulation as a component of catheter-based trainingJ Vasc Surg20044061112111715622364
  • NicholsonWJCatesCUPatelADFace and content validation of virtual reality simulation for carotid angiography: results from the first 100 physicians attending the Emory NeuroAnatomy Carotid Training (ENACT) programSimul Healthc20061314715019088583
  • WilloteauxSLionsCDuhamelARéalité virtuelle en radiologie vasculaire interventionnelle: évaluation des performances. [Virtual interventional radiology: evaluation of performances as a function of experience]J Radiol2009903741 French19182712
  • Van HerzeeleIAggarwalRNeequayeSDarziAVermassenFCheshireNJCognitive training improves clinically relevant outcomes during simulated endovascular proceduresJ Vasc Surg20084851223123018771880
  • Van HerzeeleIAggarwalRChoongABrightwellRVermassenFECheshireNJVirtual reality simulation objectively differentiates level of carotid stent experience in experienced interventionalistsJ Vasc Surg200746585586317980270
  • PatelADGallagherAGNicholsonWJCatesCULearning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiographyJ Am Coll Cardiol20064791796180216682303
  • BerryMReznickRLystigTLonnLThe use of virtual reality for training in carotid artery stenting: a construct validation studyActa Radiol200849780180518608009
  • TedescoMMPakJJHarrisEJJrKrummelTMDalmanRLLeeJTSimulation-based endovascular skills assessment: the future of credentialing?J Vasc Surg20084751008111118372149
  • SottoJAAyusteECBowyerMWExporting simulation technology to the Philippines: a comparative study of traditional versus simulation methods for teaching intravenous cannulationStud Health Technol Inform200914234635119377182
  • BrittRCNovoselTJBrittLDSullivanMThe impact of central line simulation before the ICU experienceAm J Surg2009197453353619249739
  • CavaleiroAPGuimaraesHCalheirosFTraining neonatal skills with simulators?Acta Paediatr200998463663919120041
  • St ClairEWOddoneEZWaughRACoreyGRFeussnerJRAssessing housestaff diagnostic skills using a cardiology patient simulatorAnn Intern Med199211797517561416578