Assessment of Simulated Emergency Scenarios: Are Trained Observers Necessary?

References

• Chopra V, Gesink BJ, de Jong J, Bovill JG, Spierdijk J, Brand R. Does training on an anaesthesia simulator lead to improvement in performance? Br J Anaesth. 1994;73(3):293–7.
   PubMed Web of Science ®Google Scholar
• Delasobera BE, Goodwin TL, Strehlow M, et al. Evaluating the efficacy of simulators and multimedia for refreshing ACLS skills in India. Resuscitation. 2010;81(2):217–23.
   PubMed Web of Science ®Google Scholar
• Petscavage JM, Wang CL, Schopp JG, Paladin AM, Richardson ML, Bush WH, Jr. Cost analysis and feasibility of high-fidelity simulation based radiology contrast reaction curriculum. Acad Radiol. 2011;18(1):107–12.
   PubMed Web of Science ®Google Scholar
• Cook DA, Brydges R, Hamstra SJ, et al. Hatala R. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc. 2012;7(5):308–20.
   PubMed Web of Science ®Google Scholar
• Bhanji F, Mancini ME, Sinz E, et al. Part 16: education, implementation, and teams: 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. 2010;122(18 Suppl 3):S920–33.
   PubMed Web of Science ®Google Scholar
• Mancini ME, Soar J, Bhanji F, et al. Education I, Teams Chapter C. Part 12: Education, implementation, and teams: 2010 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations. Circulation. 2010;122(16 Suppl 2):S539–81.
   PubMed Web of Science ®Google Scholar
• Chopra V. Simulators in anaesthesia. Curr Anaesth Crit Care. 1998;9187–192.
   Google Scholar
• DeVita MA, Schaefer J, Lutz J, Wang H, Dongilli T. Improving medical emergency team (MET) performance using a novel curriculum and a computerized human patient simulator. Qual Saf Health Care. 2005;14(5):326–31.
   PubMedGoogle Scholar
• Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27(1):10–28.
   PubMed Web of Science ®Google Scholar
• Boulet JR, Murray D. Review article: assessment in anesthesiology education. Can J Anaesth. 2012;59(2):182–92.
   PubMed Web of Science ®Google Scholar
• Scalese RJ. Competency assessment. In: Levine AI (ed), The Comprehensive Textbook of Healthcare Simulation. New York: Springer Science+Business Media, 2013, pp 135–60.
   Google Scholar
• Morgan PJ, Cleave-Hogg DM, Guest CB, Herold J. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anaesth. 2001;48(3):225–33.
   PubMed Web of Science ®Google Scholar
• Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90(5):580–8.
   PubMed Web of Science ®Google Scholar
• Marsch SC, Muller C, Marquardt K, Conrad G, Tschan F, Hunziker PR. Human factors affect the quality of cardiopulmonary resuscitation in simulated cardiac arrests. Resuscitation. 2004;60(1):51–6.
   PubMed Web of Science ®Google Scholar
• Murray DJ, Boulet JR, Kras JF, McAllister JD, Cox TE. A simulation-based acute skills performance assessment for anesthesia training. Anesth Analg. 2005;101(4):1127–34.
   PubMed Web of Science ®Google Scholar
• Murray DJ, Boulet JR, Kras JF, Woodhouse JA, Cox T, McAllister JD. Acute care skills in anesthesia practice: a simulation-based resident performance assessment. Anesthesiology. 2004;101(5):1084–95.
   PubMed Web of Science ®Google Scholar
• Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology. 2001;95(1):36–42.
   PubMed Web of Science ®Google Scholar
• Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability and validity of a simulation-based acute care skills assessment for medical students and residents. Anesthesiology. 2003;99(6):1270–80.
   PubMed Web of Science ®Google Scholar
• Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013;88(6):872–83.
   PubMed Web of Science ®Google Scholar
• Ker J, Hogg C. Cost-effective simulation. In: Walsh K (ed), Cost Effectvenesss in Medical Education. Oxon: Radcliffe, 2010, pp 61–71.
   Google Scholar
• Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical education: a systematic review. J Gen Intern Med. 2013;28(8):1078–89.
   PubMed Web of Science ®Google Scholar
• Nishisaki A, Keren R, Nadkarni V. Does simulation improve patient safety? Self-efficacy, competence, operational performance, and patient safety. Anesthesiol Clin. 2007;25(2):225–36.
   PubMedGoogle Scholar
• Fletcher JD, Wind AP. Cost considerations in using simulations for medical training. Mil Med. 2013;178(10 Suppl):37–46.
   PubMed Web of Science ®Google Scholar
• Boet S, Bould MD, Sharma B, Revees S, Naik VN, Triby E, Grantcharov T. Within-team debriefing versus instructor-led debriefing for simulation-based education: a randomized controlled trial. Ann Surg. 2013;258(1):53–8.
   PubMed Web of Science ®Google Scholar
• Dochy F. The use of self-, peer, and co-assessment in higher education: a review. Stud High Educ. 1999;24(3):331–50.
   Web of Science ®Google Scholar
• Boud D. Enhancing Learning through Self-assessment. London and Philadelphia: Kogan Page, 1995.
   Google Scholar
• Boet S, Bould MD, Bruppacher HR, Desjardins F, Chandra DB, Naik VN. Looking in the mirror: self-debriefing versus instructor debriefing for simulated crises. Crit Care Med. 2011;39(6):1377–81.
   PubMed Web of Science ®Google Scholar
• Somerwell H. Issues in assessment, enterprise and higher education: the case forself-, peer and collaborative assessment. Assess Eval High Educ. 1993;18221–33.
   Google Scholar
• Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–102.
   PubMed Web of Science ®Google Scholar
• Riem N, Boet S, Bould MD, Tavares W, Naik VN. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth. 2012;109(5):723–8.
   PubMed Web of Science ®Google Scholar
• Murray WB, Foster PA. Crisis resource management among strangers: principles of organizing a multidisciplinary group for crisis resource management. J Clin Anesth. 2000;12(8):633–8.
   PubMed Web of Science ®Google Scholar
• Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology. 1998;89(1):8–18.
   PubMed Web of Science ®Google Scholar
• Fletcher GC, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88(3):418–429.
   PubMed Web of Science ®Google Scholar
• Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology. 2005;103(2):241–8.
   PubMed Web of Science ®Google Scholar
• von Wyl T, Zuercher M, Amsler F, Walter B, Ummenhofer W. Technical and non-technical skills can be reliably assessed during paramedic simulation training. Acta Anaesthesiol Scand. 2009;53(1):121–7.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–26.
   PubMed Web of Science ®Google Scholar
• Guidelines 2000 for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Part 6: advanced cardiovascular life support: 7C: a guide to the International ACLS algorithms. The American Heart Association in collaboration with the International Liaison Committee on Resuscitation. Circulation. 2000;102(8 Suppl):I142–57.
   PubMedGoogle Scholar
• National Association of Emergency Medical Technicians (NAEMT). PHTLS: Prehospital Trauma Life Support. 7th ed. Jones and Bartlett, 2011.
   Google Scholar
• Kortbeek JB, Al Turki SA, Ali J, et al. Advanced Trauma Life Support. 8th ed. The evidence for change. J Trauma. 2008;64(6):1638–50.
   PubMedGoogle Scholar
• Rosner B. Fundamentals of Biostatistics. Belmont, CA: Wadsworth, 1995.
   Google Scholar
• Khanduja PK, Bould MD, Naik VN, Hladkowicz E, Boet S. The role of simulation in continuing medical education for acute care physicians: a systematic review. Crit Care Med. 2015;43(1):186–93.
   PubMed Web of Science ®Google Scholar
• Mudumbai SC, Gaba DM, Boulet JR, Howard SK, Davies MF. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7(2):73–80.
   PubMedGoogle Scholar
• Harris T, Ellis DY, Foster L, Lockey D. Cricoid pressure and laryngeal manipulation in 402 pre-hospital emergency anaesthetics: essential safety measure or a hindrance to rapid safe intubation? Resuscitation. 2010;81(7):810–6.
   PubMed Web of Science ®Google Scholar
• Dunham CM, Barraco RD, Clark DE, et al. Group EPMGW. Guidelines for emergency tracheal intubation immediately after traumatic injury. J Trauma. 2003;55(1):162–79.
   PubMedGoogle Scholar
• Rosenstock C, Moller J, Hauberg A. Complaints related to respiratory events in anaesthesia and intensive care medicine from 1994 to 1998 in Denmark. Acta Anaesthesiol Scand. 2001;45(1):53–8.
   PubMed Web of Science ®Google Scholar
• Cook TM, MacDougall-Davis SR. Complications and failure of airway management. Br J Anaesth. 2012;109(Suppl 1):i68–85.
   PubMed Web of Science ®Google Scholar
• Gjeraa K, Jepsen RM, Rewers M, Ostergaard D, Dieckmann P. Exploring the relationship between anaesthesiologists' non-technical and technical skills. Acta Anaesthesiol Scand. 2015;6036–47.
   Web of Science ®Google Scholar
• Devitt JH, Kurrek MM, Cohen MM, et al. Testing the raters: inter-rater reliability of standardized anaesthesia simulator performance. Can J Anaesth. 1997;44(9):924–8.
   PubMed Web of Science ®Google Scholar
• Devitt JH, Kurrek MM, Cohen MM, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg. 1998;86(6):1160–4.
   PubMed Web of Science ®Google Scholar
• Norcini JJ. Peer assessment of competence. Med Educ. 2003;37(6):539–43.
   PubMed Web of Science ®Google Scholar
• Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269(13):1655–60.
   PubMed Web of Science ®Google Scholar
• Hall W, Violato C, Lewkonia R, et al. Assessment of physician performance in Alberta: the physician achievement review. CMAJ. 1999;161(1):52–7.
   PubMed Web of Science ®Google Scholar
• Ramsey PG, Carline JD, Blank LL, Wenrich MD. Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Acad Med. 1996;71(4): 364–70.
   PubMed Web of Science ®Google Scholar
• Petranek CF, Corey S, Black R. Three levels of learning in simulations: participating, debriefing, and journal writing. Simul Gaming. 1992;23174–85.
   Google Scholar
• Steinwachs B. How to facilitate a debriefing. Simul Gaming. 1992;23186–95.
   Google Scholar
• Porter T. Beyond metaphor: applying a new paradigm of change to experential debriefing. J Exp Educ. 1999;2285–90.
   Google Scholar
• Pugh DM, Wood TJ, Boulet JR. Assessing procedural competence: validity considerations. Simul Healthc. 2015;10(5):288–94.
   PubMedGoogle Scholar
• Cunnington JP, Neville AJ, Norman GR. The risks of thoroughness: Reliability and validity of global ratings and checklists in an OSCE. Adv Health Sci Educ Theory Pract. 1996;1(3): 227–33.
   PubMed Web of Science ®Google Scholar
• Streiner DL NG. Reliability. In: Streiner DL NG (ed), Health Meassurement Scales: A practical Guide to their Development and Use. Oxford: Oxford University Press, 1995, pp 104–27.
   Google Scholar
• Govaerts MJ, Schuwirth LW, Van der Vleuten CP, Muijtjens AM. Workplace-based assessment: effects of rater expertise. Adv Health Sci Educ Theory Pract. 2011;16(2):151–65.
   PubMed Web of Science ®Google Scholar
• Govaerts MJ, Van de Wiel MW, Schuwirth LW, Van der Vleuten CP, Muijtjens AM. Workplace-based assessment: raters' performance theories and constructs. Adv Health Sci Educ Theory Pract. 2013;18(3):375–96.
   PubMed Web of Science ®Google Scholar
• Boulet JR, Murray DJ. Simulation-based assessment in anesthesiology: requirements for practical implementation. Anesthesiology. 2010;112(4):1041–52.
   PubMed Web of Science ®Google Scholar