ABSTRACT
Objective: Paramedics and emergency medical technicians (EMTs) triage pediatric disaster victims infrequently. The objective of this study was to measure the effect of a multiple-patient, multiple-simulation curriculum on accuracy of pediatric disaster triage (PDT). Methods: Paramedics, paramedic students, and EMTs from three sites were enrolled. Triage accuracy was measured three times (Time 0, Time 1 [two weeks later], and Time 2 [6 months later]) during a disaster simulation, in which high and low fidelity manikins and actors portrayed 10 victims. Accuracy was determined by participant triage decision concordance with predetermined expected triage level (RED [Immediate], YELLOW [Delayed], GREEN [Ambulatory], BLACK [Deceased]) for each victim. Between Time 0 and Time 1, participants completed an interactive online module, and after each simulation there was an individual debriefing. Associations between participant level of training, years of experience, and enrollment site were determined, as were instances of the most dangerous mistriage, when RED and YELLOW victims were triaged BLACK. Results: The study enrolled 331 participants, and the analysis included 261 (78.9%) participants who completed the study, 123 from the Connecticut site, 83 from Rhode Island, and 55 from Massachusetts. Triage accuracy improved significantly from Time 0 to Time 1, after the educational interventions (first simulation with debriefing, and an interactive online module), with a median 10% overall improvement (p < 0.001). Subgroup analyses showed between Time 0 and Time 1, paramedics and paramedic students improved more than EMTs (p = 0.002). Analysis of triage accuracy showed greatest improvement in overall accuracy for YELLOW triage patients (Time 0 50% accurate, Time1 100%), followed by RED patients (Time 0 80%, Time 1 100%). There was no significant difference in accuracy between Time 1 and Time 2 (p = 0.073). Conclusion: This study shows that the multiple-victim, multiple-simulation curriculum yields a durable 10% improvement in simulated triage accuracy. Future iterations of the curriculum can target greater improvements in EMT triage accuracy.
Introduction
Disasters are one of the greatest challenges emergency medical services (EMS) providers face. Whether they are acts of violence, natural calamities, or the result of overcrowded motorways, these events strain or overwhelm available resources, and require deviation from protocols or degradations to the standard of care.Citation1–3 Triage in disasters requires fast, accurate decisions about the severity of a victim's illness or injury, the likelihood of survival given the available resources, nature of the disaster event, and the total number of victims requiring care.Citation2,4–7
Performing disaster triage when pediatric patients are among the victims may be more difficult for paramedics and emergency medical technicians (EMTs).Citation8–10 Indeed, prehospital care providers care for pediatric patients, yet pediatric patients represent only 10% of the patients they treat,Citation11 and few of these young patients are critically ill or injured.Citation12,13 In addition to providers infrequently caring for pediatric patients, pediatric training is often a small component of EMS education and core curriculum refreshers.Citation14,15
Mass casualty events are infrequent, yet the stakes are high.Citation16 Additionally, EMS curricula are replete with crucial topics, and training time for pediatric MCIs is limited.Citation17,18 To be effective, EMS educators and disaster management planners must consider the frequency, educational strategy and methods, specific curricula, and duration of pediatric disaster triage (PDT) training.Citation19 There have been previous efforts to establish a national pediatric disaster curriculum, with limited success.Citation20
In this investigation, we aimed to evaluate the effect of a simulation-based curriculum on the accuracy of prehospital care provider PDT.Citation21 Furthermore, we measured learner retention of PDT knowledge and skills six months after completing the initial educational intervention.
Methods
Design
We conducted a longitudinal, prospective study of the effect of a simulation-based mass casualty curriculum on the PDT knowledge and skills of EMS personnel.
Population and Setting
Participants were EMS providers in Rhode Island, Massachusetts, and Connecticut. Participants were recruited via e-mail and in-person invitations and were enrolled in a PDT curriculum. The participants included EMTs, paramedic students (most of whom were already EMTs), and practicing paramedics.
Participants completed a pre-participation survey, which assessed their prior pediatric and disaster training, including Pediatric Advanced Life Support (PALS), and Basic and Advanced Disaster Life Support (BDLS and ADLS) courses. Years of experience and highest level of training were also recorded.
The institutional review boards of the three academic institutions that conducted this project reviewed and approved the study. The need for written informed consent was waived due to the study occurring in an educational context.
Curriculum
We used a modified Delphi method with nine subject matter experts from across North America to create three high-fidelity simulations and corresponding evaluation tools for the participants who completed the simulations. The development of the simulations has been previously described.Citation21 The three simulations involved 10 patients with a corresponding evaluation tool. The simulations included a multi-family house fire, a school shooting, and a school bus crash. The selected scenarios were chosen because such events are common when compared to other types of multiple patient incidents. Furthermore, the frequency of these events is not likely to be affected by local factors, such as proximity to a coast or peculiarities of climate.
The initial plan was to alternate the scenarios thusly:
Year 1: House Fire, School Shooting, School Bus Rollover
Year 2: School Bus Rollover, School Shooting, House Fire
Year 3: School Shooting, House Fire, School Bus Rollover
This design was intended to control for any inherent difference in difficulty among the three scenarios. The scenarios were designed to represent the same 10 kinds of injuries and proportions of triage levels (e.g., each scenario with an unconscious patient with a head injury, one with tachypnea due to penetrating chest trauma, a patient with special healthcare needs and minimal injuries, etc.)
This plan was altered after the events of December 14, 2012, when there was a mass shooting at the Sandy Hook Elementary School in Newtown, Connecticut. Due to potential adverse mental health consequences if participants experienced a school shooting simulation <1 month after a local mass shooting, the order of simulations was altered to match the Year 2 schedule.
In the various simulations, each of the 10 patients had a different illness or injury. Across the simulation scenarios, the same presentations and equivalent injuries were represented. Furthermore, we represented the same sounds (e.g., sirens), odors (e.g., smoke), and injuries (e.g., moulaged wounds) at each site. Examples include a patient who was unable to walk due to a back injury and an uninjured child with special health care needs. shows the triage domains assessed for each patient and the simulation modality used to represent the patient.Citation21
Table 1. Simulated disaster victims with learner domains for triage assessment and management
Using a modified Delphi method,Citation21 subject matter experts from across North America assigned expected triage levels for each of the patients: RED (Emergent), YELLOW (Delayed), GREEN (Ambulatory), and BLACK (Deceased). For all three PDT strategies, RED is defined as patients with life-threatening illness or injury who might survive if treated immediately, YELLOW is a non-ambulatory patient without life-threatening injuries, GREEN is an ambulatory patient, and BLACK is a patient who is dead or expected to die given the disaster situation and current resources. Evaluation instruments were designed alongside the simulations during the modified Delphi process.
Each site used one PDT strategy, according to local policy. Smart TriageCitation22 was used in Connecticut, clinical decision-making with no formal algorithm in Rhode Island,Citation23 and JumpSTART/STARTCitation24 Triage in Massachusetts. In Connecticut, the Smart algorithm and length-based triage tape were available to participants. In Massachusetts, the JumpSTART algorithm was available to participants. In all three states, participants were instructed to use the local PDT strategy, by name.
The curriculum was five hours in total, including prebriefings, simulations, individual debriefings, and an interactive e-learning module. The e-learning module included general disaster management principles, pediatric specific content, common features of disaster triage strategies, and customized content for JumpSTART, Smart, and clinical decision-making without a formal algorithm. The educational interventions included the initial simulation, the e-learning module, and individual, structured debriefings following each simulation. During the debriefing, expected triage levels for each patient were compared to triage levels assigned by the participant.
Evaluation of Triage Performance
Individual learners were assessed three times. Performances in each simulation, prior to (Time 0), two weeks after (Time 1), and 6 months after (Time 2) educational intervention were rated using a checklist-based tool.Citation21 The timing of the simulations and evaluations in this study are shown in the project flow diagram ().
Figure 1. Study time flow with curricular elements and evaluations.
An onsite evaluator used a standardized checklist to evaluate participants in real time, and there was no time limit for completion of the simulations. Video recordings were made to ensure future questions about participant performance could be assessed. Participants were video-recorded using a single handheld video camera and a wearable microphone by a single videographer as they completed the disaster triage simulations. Video recordings were de-identified, and stored in a password-protected online file repository.
Triage accuracy was defined as participant triage agreement with the predetermined Delphi gold standard, and was agnostic to any single triage strategy.
A final analysis was performed, which assessed the number of instances when patients who should have been triaged RED or YELLOW were actually triaged BLACK. We evaluated instances of this mistriage as they are the most dangerous of triage errors.
Statistical Analysis
All data were manually recorded during the simulation and then entered into Microsoft Excel version 14.0 (Microsoft, Redmond, WA) and transferred into SPSS (v. 22.0, IBM Corp., Armonk, NY), with which all statistical analyses were performed. We examined differences in triage accuracy by each factor (certification, years of experience, training, and site), using Wilcoxon-Mann-Whitney U tests. Differences in repeated measures were assessed with Wilcoxon Signed-rank tests.
We used a generalized linear multivariable mixed-effects model with change in triage score accuracy from Time 0 to Time 1 and Time 1 to Time 2 as the dependent variables with a robust covariance estimator to account for inter-site variability. Importantly, we included a categorical variable to represent the order in which the participant was assigned to the simulations at each time point (e.g., house fire, school shooting, bus crash; school shooting, house fire, bus crash… etc.) in order to control for a possible ordering effect on the change in triage accuracy. Other variables in the model were certification, years of experience, PALS training, and BDLS training.
Results
The study enrolled 337 participants, and the analysis included 261 (77.4%) participants who completed the study: 123 from Connecticut, 83 from Rhode Island, and 55 from Massachusetts. Participant demographics and prior disaster education and training are shown in . Of note, the Massachusetts site had a large proportion of paramedic students (71.2%).
Table 2. Participant demographics, prior disaster training and experience
For the Connecticut group, there was a single training site. The disaster triage curriculum was offered in the final six months of the paramedicine curriculum, and the learners had had their pediatrics module. In Rhode Island and in Massachusetts, there was heterogeneity among the learners regarding their advancement within the paramedicine program, with some learners being near-completion of their coursework, and others at the onset of the curriculum. There were 3 paramedicine training sites represented in Massachusetts and 5 sites represented in Rhode Island.
Acquisition of PDT skills
Triage accuracy improved significantly from Time 0 to Time 1, after the educational interventions (first simulation with debriefing and interactive online module), with a median 10% overall improvement (p < 0.001, ). This represents a clinical improvement of one additional patient being triaged accurately after the intervention. The 76 participants who did not complete the study were compared to the 261 who did. Of the group who completed only one simulation, the Median Time 0 score (IQR) was 90% (80, 90). For the group who completed two simulations, the Median Time 0 score (IQR) was 80% (70, 90), which was significantly lower than the group that did not complete the second simulation (p = 0.005). The Median Time 1 score (IQR) for those that completed the study was 90% (80,100).
Figure 2. Improvement in triage accuracy over time.
Subgroup analyses were performed to evaluate the association between participant characteristics and magnitude of improvement in triage accuracy. From Time 0 to Time 1, paramedics and paramedic students improved more than EMTs (p = 0.002, ). Additional factors associated with improved triage accuracy during the intervention included prior completion of the PALS course (p < 0.001), being from the Massachusetts site (p = 0.017), and greater number of years' experience (p = 0.047).
Table 3. Learner improvements in and retention of triage accuracy
Retention of PDT skills over time
There was no significant difference in PDT six months after the onset of the study (Time 2) compared with two weeks after the study (Time 1) (p = 0.073), but the effect was durable and significant when comparing triage accuracy between Time 0 and Time 2 (p = 0.001, ).
Unadjusted associations between participant characteristics and retention of improvement in triage accuracy at Time 2 compared to Time 1 revealed that while all groups had the same median score (90%) at Time 2, EMTs showed an increase in their triage accuracy. By contrast, paramedics and paramedic students showed no change in accuracy (p = 0.001, ). Prior PALS training, training site, and years of experience were not associated with a difference in triage accuracy at Time 2.
A multivariable model controlling for inter-site variability and a possible ordering effect to model improvements in triage accuracy from Times 0 to 1 and from 1 to 2 revealed significant differences not shown in the bivariate analyses (). Notably, students and paramedics showed a higher improvement from Time 0 to Time 1 compared to EMTs (β = 8.6, 95% confidence interval [CI]: 0.1, 17.2 and β = 13.2, 95% CI: 4.9, 21.4, respectively). However, the improvement was diminished at Time 2, as students and paramedics showed modest declines in triage accuracy from Times 1 to 2 vs. EMTs (β = −5.8, 95% CI: −13.8, 2.1 and β = −8.8, 95% CI: −16.0, −1.6, respectively). There was also a decline from Times 0 to 1 in participants with BDLS training vs. no BDLS training (β = −7.5, 95% CI: −14.5, −0.5) and in Times 1 to 2 in participants with PALS training vs. no PALS training (β = −6.6, 95% CI: −12.3, −0.9). Notably, there was an ordering effect observed in the multivariable model from Times 1 to 2.
Table 4. Multivariable analyses of changes between scores
Analysis of triage improvement by triage level showed greatest improvement in overall triage accuracy for YELLOW triage patients (Time 0 50% accurate, Time 1 100%, ), followed by RED patients (Time 0 80%, Time 1 100%). There were not significant changes in accuracy in GREEN and BLACK patients, as baseline accuracy was high. Instances of mistriage when salvageable RED and YELLOW patients were triaged BLACK are also shown in .
Table 5. Triage accuracy by predetermined patient triage level
Discussion
This study demonstrates that our simulation-based curriculum confers significant, durable 10% improvement in pediatric disaster triage accuracy in a population of EMS providers. A key strength of the intervention is the retention in improvement over six months, with no systemic additional training from Time 1 to Time 2.
Considering the participant characteristics associated with improvements in triage accuracy yields several important observations. First, although baseline accuracy was the same for all groups, paramedic students and paramedics outperformed EMTs in their improvement (Time 1) in triage accuracy. This may be that paramedics and paramedic students have had more experience with simulation as an education modality than EMTs, that paramedics and students are more attuned to continuing education opportunities because they have more such opportunities than EMTs, or that EMTs have less overall EMS experience as a group than other participants.Citation25 It was surprising that at Time 2, the EMTs had gained in their performance, and their triage accuracy matched that of paramedics and students. This could be related to improved familiarity with simulation after the first two sessions.
Next, both PALS course completion and being from the Massachusetts site were associated with more gains in triage accuracy. Given that the completion of a PALS course is more likely for paramedics, and the Massachusetts group had a large proportion of paramedic students, we controlled for both of these factors in the multivariable analyses, which revealed that PALS training predicted a slight decrease in triage accuracy in Time 1 to Time 2. This could be explained by the fact that participants with PALS training had a higher baseline triage accuracy vs. participants without (median accuracy at Time 0 85% vs. 80%, p = 0.031). In other words, those without PALS training had a greater knowledge gap and triage accuracy at the Time 0 than those with PALS training. Likewise, the association between the large proportion of paramedic students in the Massachusetts and greater triage accuracy may be attributable to more recent disaster triage learning compared to the other sites.
The association between years of experience and improvement in triage accuracy was significant, if narrowly so. This finding bears consideration. Multiple casualty incidents and larger disasters occur infrequently, and EMS providers with decades of experience may have only a modest advantage of disaster-specific experience compared to their younger colleagues. Other learner factors in the multivariate analysis were not associated with improvement in triage accuracy: BDLS training was associated with a decrease in accuracy between Time 0 and Time 1, and, as previously noted, PALS training with a decrease in accuracy between Time 1 and Time 2. These findings may be due to lack of triage content in either course, or due to timespan since completion of BDLS and/or PALS.
It is noteworthy that the gains in triage accuracy were more significant for YELLOW and RED patients than for GREEN and BLACK patients (). Beyond the lower baseline accuracy for RED and YELLOW patients, characterizing a patient RED vs. YELLOW is potentially more difficult in primary triage than the GREEN or BLACK categories. GREEN patients are able to ambulate and generally quickly identified, and BLACK patients are immobile, apneic, and unresponsive to airway maneuvers and bag-valve-mask ventilation. A potential target for future interventions is to further improve the safety and efficacy of triage, is decreasing the proportion of patients who should be triaged RED or YELLOW, but were mistriaged as BLACK.
Future directions for this work include applying the results of this study to the iterative development of our curriculum. Presently, paramedics who complete the curriculum show more improvement at Time 1 than EMTs. For the next iteration we intend to enhance the simulation orientation that is provided to EMTs and enhance the focus on simulated patients who are intended to be RED and YELLOW.
Within the last two years, START/JumpSTART remained the dominant triage system in use across the United States.Citation26 Other systems may result in more accurate triage and ultimately better patient outcomes than START/JumpSTART. Finally, understanding that the Model Uniform Core Criteria (MUCC) for MCI triage,Citation27,28 and the attendant Sort-Assess-Lifesaving Treatment triage systemCitation29 are promoted by the United States federal government and being adopted in an increasing number of states and municipalities, future directions for this work can include a MUCC compliant version of the curriculum.
Limitations
This study is subject to a number of limitations. First, the participant groups at each of the three sites were not homogenous, with the Massachusetts group having a larger proportion of paramedic students than the other two groups. This limitation is more important for comparisons of the three triage strategies,Citation30 where data across the three sites were pooled for the analysis, and the multivariable analysis controlled for variability between sites. A second limitation of the study is that although long-term durability of disaster triage accuracy was assessed at Time 2, six months after the initial educational interventions, disasters occur with unpredictable infrequency. Consequently, an EMS provider who completes the simulation-based curriculum may not encounter an actual mass casualty incident until years after the training, a timeframe we did not assess. A final limitation of the study is generalizability due to cost. To conduct three simulations with 10 patients three times over six months and ensure that a cadre of busy EMS providers is able to participate in all simulations is expensive and resource-intensive. We acknowledge that, in the simulations, the mistriage of RED and YELLOW patients as BLACK may have been due to simulation limitations rather than participant knowledge. If the simulated patients were all portrayed with the same model of manikin, standardization would have controlled for the limitations inherent in modern simulators. Alternate educational strategies, including serious video games, are being investigated to facilitate education for larger numbers of EMS providers.
Conclusion
This study demonstrates that our simulation-based PDT curriculum leads to significant, durable improvement in the simulated triage accuracy of EMS providers. The curriculum is associated with PDT improvement for EMTs and paramedics, and yielded the greatest improvements in triage accuracy for YELLOW and RED patients.
References
- Gates JD, Arabian S, Biddinger P, et al. The initial response to the Boston marathon bombing: lessons learned to prepare for the next disaster. Ann Surg. 2014;260(6):960–6.
- Kanter R. Strategies to improve pediatric disaster surge response: potential mortality reduction and tradeoffs. Crit Care Med. 2007;35(12):2837–42.
- Hick JL, Hanfling D, Cantrill SV. Allocating scarce resources in disasters: emergency department principles. Ann Emerg Med. 2012;59(3):177–87.
- Stalcup S, Oscherwitz M, Cohen M, et al. Planning for a pediatric disaster – experience gained from caring for 1600 Vietnamese orphans. N Engl J Med. 1975;293(14):691–5.
- Lerner E, Schwartz R, Coule P, Pirrallo R. Use of SALT triage in a simulated mass-casualty incident. Prehosp Emerg Care. 2010;14(1):21–5.
- Cross KP, Cicero MX. Head-to-head comparison of disaster triage methods in pediatric, adult, and geriatric patients. Ann Emerg Med. 2013;61(6):668–76. e667.
- Frolic A, Kata A, Kraus P. Development of a critical care triage protocol for pandemic influenza: integrating ethics, evidence and effectiveness. Healthc Q. 2009;12(4):54–62.
- Claudius I, Kaji AH, Santillanes G, et al. Accuracy, efficiency, and inappropriate actions using jumpSTART triage in MCI simulations. Prehosp Disaster Med. 2015;30(5):457–60.
- Gausche-Hill M. Pediatric disaster preparedness: are we really prepared? J Trauma. 2009;67(2 Suppl):S73–6.
- Koziel JR, Meckler G, Brown L, et al. Barriers to pediatric disaster triage: a qualitative investigation. Prehosp Emerg Care. 2015;19:1–8.
- Shah MN, Cushman JT, Davis CO, Bazarian JJ, Auinger P, Friedman B. The epidemiology of emergency medical services use by children: an analysis of the National Hospital Ambulatory Medical Care Survey. Prehosp Emerg Care. 2008;12(3):269–76.
- Kannikeswaran N, Mahajan PV, Dunne RB, Compton S, Knazik SR. Epidemiology of pediatric transports and non-transports in an urban Emergency Medical Services system. Prehosp Emerg Care. 2007;11(4):403–7.
- Richards ME, Hubble MW, Zwehl-Burke S. “Inappropriate” pediatric emergency medical services utilization redefined. Pediatr Emerg Care. 2011;27(6):514–8.
- Lammers RL, Byrwa MJ, Fales WD, Hale RA. Simulation-based assessment of paramedic pediatric resuscitation skills. Prehosp Emerg Care. 2009;13(3):345–56.
- Donoghue A, Ventre K, Boulet J, et al. Design, implementation, and psychometric analysis of a scoring instrument for simulated pediatric resuscitation: a report from the EXPRESS pediatric investigators. Simul Healthc. 2011;6(2):71–7.
- Mace S, Jones J, Bern A. An analysis of Disaster Medical Assistance Team (DMAT) deployments in the United States. Prehosp Emerg Care.11(1):30–5.
- Markenson D, Foltin G, Tunik M, et al. Knowledge and attitude assessment and education of prehospital personnel in child abuse and neglect: report of a National Blue Ribbon Panel. Prehosp Emerg Care. 2002;6(3):261–72.
- Margolis GS, Romero GA, Fernandez AR, Studnek JR. Strategies of high-performing paramedic educational programs. Prehosp Emerg Care. 2009;13(4):505–11.
- Studnek JR, Fernandez AR. Organizational description and emergency preparedness of Nationally Registered First Responders. Prehosp Disaster Med. 2008;23(3):250–5.
- Schultz CH, Koenig KL, Whiteside M, Murray R, Force NSA-HDCCT. Development of national standardized all-hazard disaster core competencies for acute care physicians, nurses, and EMS professionals. Ann Emerg Med. 2012;59(3):196–208. e191.
- Cicero MX, Brown L, Overly F, et al. Creation and delphi-method refinement of pediatric disaster triage simulations. Prehosp Emerg Care. 2014;18:282–9.
- Cone DC, Serra J, Kurland L. Comparison of the SALT and Smart triage systems using a virtual reality simulator with paramedic students. Eur J Emerg Med. 2011;18(6):314–21.
- Qazi K, Kempf JA, Christopher NC, Gerson LW. Paramedic judgment of the need for trauma team activation for pediatric patients. Acad Emerg Med. 1998;5(10):1002–7.
- Romig L. Pediatric triage. A system to JumpSTART your triage of young patients at MCIs. JEMS. 2002;27(7):52–8, 60–53.
- McKenna KD, Carhart E, Bercher D, Spain A, Todaro J, Freel J. Simulation Use in Paramedic Education Research (SUPER): a descriptive study. Prehosp Emerg Care. 2015;19(3):432–40.
- Nadeau NL, Cicero MX. Pediatric disaster triage system utilization across the United States. Pediatr Emerg Care. 2016. [Epub ahead of print]
- Pediatrics EbAAo, Physicians ACoE, Trauma ACoS-Co, Society AT, Children's National Medical Center CHAI, E.ergency Medical S. Model uniform core criteria for mass casualty triage. Disaster Med Public Health Prep. 2011;5(2):125–8.
- Lerner EB, Cone DC, Weinstein ES, et al. Mass Casualty Triage: An Evaluation of the Science and Refinement of a National Guideline. Disaster Med Public Health Prep. 2011;5(2):129–37.
- Lee CW, McLeod SL, Van Aarsen K, Klingel M, Franc JM, Peddle MB. First Responder Accuracy Using SALT during Mass-casualty Incident Simulation. Prehosp Disaster Med. 2016;31(2):150–4.
- Cicero MX, Overly F, Brown L, et al. Comparing the accuracy of three pediatric disaster triage strategies: a simulation-based investigation. Disaster Med Public Health Prep. 2016;10(2):253–60.