Airway Management Proficiency Checklist for Assessing Paramedic Performance

Abstract

Objective: To develop and derive an instrument for assessing airway management proficiency for paramedics. Methods: Using a validated difficult airway model simulation, we recorded responses to a standard traumatic brain injury scenario requiring airway management in 197 certified paramedics. Discrete items (N = 131) were developed by an expert panel, and referenced to three performance standard subscales (i.e., intubation, ventilation, and backup airway). Responses were scored and subjected to an iterative process to create a more practical number of items for the final Airway Management Proficiency Checklist (AMPC). Tetrachoric correlations were used to evaluate items for relevance. Kuder-Richardson Formula 20 reliabilities were used to assess internal consistency among checklist items. Finally, a Rasch analysis on each subscale was performed to evaluate items for measurement quality. Items were retained if they were determined to fit the Rasch Model. Results: Items were deleted from the final AMPC for lack of simulation fidelity (26 items), duplicity (15 items), and poor psychometric quality (39 items). In four additional iterations, items were dropped for lack of equipment options (e.g., single mask), lack of instructional clarity (e.g., calculation of GCS score), high inference on the part of the evaluator (6 items), or inadequate measurement of behavioral performance (e.g., passes blade through lips without contacting mouth or teeth). Thirty seven items and three outcome standards (first pass success of the endotracheal tube; assisted ventilation with no interruption of 30 seconds or greater; successful placement of a backup airway device within one attempt) were retained to form three hypothesized subscales. Conclusions: The AMPC represents a psychometrically derived instrument that identified important tasks required for comprehensive airway management. The 37-item instrument will contribute to improving training and measuring the performance of paramedic's airway management skills.

Key words::

• airway management
• patient simulation
• emergency care
• prehospital
• evaluation research
• educational needs assessment
• emergency medical technicians

Introduction

Comprehensive airway management is an essential skill for paramedics, physicians, and other health care professionals working in acute care environments.^1,2 Comprehensive airway management is a series of steps and actions to ensure an open pathway for ventilation to the lungs. Sequential steps include: patient assessment, equipment selection, upper airway suctioning, administration of supplemental oxygen, insertion of oral airway adjuncts, performance of laryngoscopy, placement of an endotracheal tube (ETT) with or without sedation and neuromuscular blockade, verification of ETT placement, and the stabilization of the ETT by appropriate methods. Inadequate performance at any step during full airway management can lead to safety hazards, poor clinical outcomes, and even death.^3–5

The gold standard for advanced airway management of severely ill or injured patients has been endotracheal intubation (ETI).^1,6 The current training of prehospital providers in ETI has been found to be insufficient at improving patient outcomes, particularly in cases involving physiologic instability. Furthermore, complications and proficiency studies have shown better patient outcomes with a comprehensive airway management approach. Even the most basic tasks such as: proper equipment selection, suctioning, positioning, assisted ventilation, avoiding hypoxemia, and patient positioning, among others, have been shown to have important effects on reducing medical complications involving patients who confronted airway crises in the prehospital setting.^7,8 Consequently, developing effective training and assessment methods for the maintenance of airway management performance is critical to the improvement of patient outcomes.^9–11

While various methods of assessment of emergency airway provider performance have been proposed, none have captured the full spectrum of required competencies involved in comprehensive airway management.^12,13 In order to achieve stable and consistent performance over time, both a realistic performance assessment environment and a corresponding performance assessment instrument are needed.^14,15 Paramedic training in airway management optimally involves instruction in knowledge and skills in physiologic stabilization, and comprehensive airway management.^7,8

To meet this need, the objective of this project was to develop a performance assessment instrument, designed to isolate and measure the most important tasks involved in demonstrating proficiency in comprehensive airway management. To accomplish this, we observed and documented the performance of trained paramedics executing airway management on a previously validated traumatic brain injury simulation.¹⁶ The result was the Airway Management Proficiency Checklist, a performance assessment instrument designed to measure comprehensive airway management performance in the setting of a difficult airway.

Methods

Instrument

The Airway Management Proficiency Checklist (AMPC)

The AMPC was designed for either formative or summative assessment of a paramedic's comprehensive airway management skills. An expert panel was guided through an iterative process of deconstructing the comprehensive airway management activities, first into sequential categories of clinical performance, and then into discrete tasks within each of these categories (see Table 1). From this, a comprehensive list of performance items (N = 127) were mapped to these deconstructed airway management steps representing an entire airway management encounter. The final list of performance items was reviewed and edited by the expert panel to establish items that were considered to be a) easily observed during the simulated patient encounter and b) low inference, or easily interpreted by an airway evaluator. The expert panel included: two anesthesiologists, one emergency department (ED) nurse, one paramedic, one emergency medical service (EMS) educator, and one emergency medicine (EM) physician.

Table 1. Example of a deconstructed, step-wise, comprehensive airway management approach and categorization, with observable items of behavior and potential scoring determinants^1,2

Airway Management Category	Observable Items	Scoring Determinants
Indications for active airway management and invasive ventilation	Crash vs. emergent; team members, respiratory failure, BiPAP,^* protect airway, altered consciousness, be PREPARED (Positioning, Room, Equipment, Pharmacy, Airway, Ready, Early, Don'ts)	Recognizes and acts: crash vs emergent; Equipment and drugs correct and ordered; Position and team
Establishing airway, suctioning, optimizing supplemental oxygen delivery	Open airway, maintain patency, remove debris, suctioning, maximize pre-02,^† assisted ventilation, and 02^† supplementation	Opens and maintains airway patency; Suctions effectively and safely; Maintains O2,^† SpO2^† max, aspiration
Optimizing positioning of patient, equipment, assistance	Upright, stretcher height, airway assistant, respiratory therapy, obese pt., ETT ready	Optimal positioning, team assembly Identify high risk patient ETT optimized
Emergency sedation^‡ and neuromuscular blockade	Choices, contraindications, dosages, delivery, timing, waiting	Dosage, timing, i.v. problems
Laryngoscopy and endotracheal tube (ETT) placement	DL^§, video-laryngoscopy; placing tube, bougie, 30 seconds, first attempt, assistance, unable- 02^†monitor, repeat,	Insertion, forcing, damaging, timing; First, 2nd, or 3rd attempt, steps
Difficult airway approaches and actions	Backup airways and personnel, supra-glottic airway, alternatives, cricothyrotomy	Cardiac and chest complications, secondary airway utilization, cricothyrotomy
Confirmation of ETT secure and placement.	Breath sounds, EtCO2,^║ SpO2,^# Hold, tape, other	Rapid, proper steps, always secure
Post ETI considerations.	Critically ill patient, ventilator settings	6 mL/kg; PEEP,^** airway secured and patient still sick

^* Bilevel positive airway pressure.

^† 02 is supplemental oxygen.

^‡ Sedation and neuromuscular blockade were not included.

^§ DL is direct laryngoscopy.

^║ EtCO2 is end-tidal carbon dioxide.

^# Sp02 is saturation of O2 detected by pulse oximeter.

^** PEEP is positive end expiratory pressure.

The panel's last responsibility was to establish overarching performance standards to articulate successful performance outcomes and to function as anchors for evaluation of the performance items. The four performance standards included:

1.	success in accurately passing an endotracheal tube in the first attempt;
2.	continuous ventilation without interruptions lasting more than 30 seconds during the entire airway management encounter;
3.	success in placing a backup airway device within one attempt; and
4.	suctions the patient

The four performance standards provided a theoretical subscale structure for the instrument and were ultimately used to gauge the predictive validity of the individual checklist items thought to be associated with those subscale categories. The original 127 discrete performance tasks and their four associated performance outcomes are listed in Appendix A.

Performance Assessment

The performance assessment was designed to require the paramedic to engage in all 127 discrete performance tasks covering four key sets of skills: 1) placement of an endotracheal tube; 2) application of backup airway following a failed endotracheal intubation; 3) ventilation of the patient; and 4) achievement of airway safety and quality measures.¹⁷ Accordingly, the performance encounter included: a challenging airway case, involving a trauma scenario and a patient with the potential for a severe head injury. The patient was a realistic high-fidelity manikin (Deluxe Difficult Airway Trainer, Laerdal Medical AS, Stavanger, Norway) with a rigid cervical collar already applied and an artificially inflated tongue. The assessment took place in the back of a type three ambulance which had been converted into a mobile simulation laboratory (MSL). The MSL was equipped with four video cameras attached to a monitor that displayed the video output from all four cameras. A more detailed description of the MSL and the Difficult Airway Simulation has been documented previously.¹⁶

During the performance assessment, a trained clinical proctor read a standardized orientation to the paramedic subject. The orientation included an overview of the case, the available equipment, and a reminder to follow standard EMS protocols for airway management. The paramedic's performance was recorded and recordings were stored for scoring at a later time.

To cover all 127 airway management performance tasks, paramedics who successfully intubated the patient within three attempts were told that the endotracheal tube had become dislodged. The following instructions required the paramedic to execute the airway management tasks involved with reassessing the patient, providing proper ventilation, applying a back-up airway, and achieving airway safety and quality measures.

En route, the endotracheal tube was accidentally dislodged. You have attempted to intubate this patient twice since you're first successful endotracheal intubation. You noted that the airway was significantly edematous, full of secretions and you were unable to visualize the ET tube pass through the vocal cords. The EtCO2 is 0; Pulse oximetry is 70%. Please proceed with the scenario.¹⁶

Total participants recruited for this performance assessment was 198 paramedic subjects from the Commonwealth of Pennsylvania. Their performance on most of the 127 airway management performance tasks is described in Panchal et. al.¹⁷

Scoring, Statistical Analyses, and Data Reduction Methods

The AMPC was calibrated as part of a larger study of comprehensive airway management skills in a representative population of certified paramedics in the Commonwealth of Pennsylvania.¹⁷ All four camera angles and the audio for all interactions were digitally recorded and stored for scoring at a later time. One expert evaluator (GF) was employed to review all 198 paramedic recordings and score their performance using the 127 item checklist (see Panchal et. al. for more detail).¹⁷ The 127 items on the checklist were scored either a “1” representing successful execution of the task, or a “0” representing a failed attempt or no attempt at all. Data were entered into and analyzed with SPSS for Windows-Version 22 (IBM-SPSS for Windows, Version 22.0, Aramonk, NY) with sampling review to ensure accurate translation.

An iterative process involving four rounds of item analyses and qualitative inspection was used to reduce the original 127 items to a more practical number of items for performance measurement. Four types of statistical analyses were employed to evaluate and select items for retention and to establish the psychometric properties of the final AMPC instrument. The item analysis statistics were reported to a panel of experts for interpretation and recommendations. At the end of each round the panel would make a consensus decision as to whether an item should be retained or dropped based on the statistics and descriptions as to how the paramedics interacted with the item. The analysis was then repeated on the remaining items and reviewed again by the panel.

Ideally, an Exploratory Factor Analysis (EFA) on the tetrachoric correlation matrix of the 127 items would have been performed to reduce the data to a practical number of items and subscales. However, the EFA on this correlation matrix was found to have a determinant of zero, a common problem for this type of analysis. Consequently, other statistical methods had to be used to evaluate the AMPC items. First, tetrachoric correlations between the individual items and their associated performance standards were calculated. The tetrachoric correlation coefficient is used to estimate the Pearson Correlation between two dichotomous variables that are assumed to have underlying continua.¹⁸ While the specific performance tasks assessed in this study were scored dichotomously (performed the task or did not), we hypothesize that underlying each task is a continuum of performance. We retained items that had a positive, moderate relationship (R^tet ≥.20) with the corresponding performance standard within each subscale.

Second, the Kuder-Richardson Formula 20 (KR-20) reliabilities were calculated to assess internal consistency among the checklist items within each of the hypothesized subscales: intubation, ventilation, suction, and back-up airway skills. We retained items that contributed to the overall scale's internal consistency (KR-20). Accordingly, we retained items that had a positive, moderate point-biserial correlation with the total subscale score R_pbis ≥.20.

Finally, the EFA was replaced by Rasch Analysis of each subscale separately to evaluate items for measurement quality through fit to the Rasch Model. All Rasch Analyses were performed using Winsteps Version 3.81.0.¹⁹ We retained items that met the criteria for “reasonable item mean-square ranges for Infit and Outfit” proposed by Wright and Linacre for tests involving clinical observation (Mean Square Infit and Outfit values of 0.5–1.7).²⁰ (Table 2 shows the number of items retained at each stage and the most common reasons items were dropped.)

Table 2. Number of items retained at each stage of item analysis with number of items dropped (in parentheses) and rationale for decision to drop items

		Stage of Analysis
Subscale	Items	1	2	3	Final
Backup Airway (35)	Dropped	14	0	1	8
		-Q (3)		-Q (0)	-Q (6)
		-D (3)		-D (1)	-D (1)
		-S (8)		-S (0)	-S (1)
	Retained	21	21	20	12
Intubation (62)	Dropped	20	6	9	10
		-Q (1)	-Q (1)	-Q (5)	-Q (6)
		-D (0)	-D (2)	-D (3)	-D (4)
		-S (19)	-S (3)	-S (1)	-S (0)
	Retained	42	36	27	17
Suction (9)	Dropped	9	0	0	0
		-Q (9)
		-D (0)
		-S (0)
	Retained	0	0	0	0
Ventilation (21)	Dropped	8	2	0	3
		-Q (2)	-Q (1)		-Q (2)
		-D (0)	-D (0)		-D (1)
		-S (6)	-S (1)		-S (0)
	Retained	13	11	11	8
TOTAL (127)	Dropped	51	8	10	21
	Retained	76	68	58	37

Q = Qualitative reason; D = Duplicate item; S = Statistical reason.

Qualitative reasons for dropping items included tasks that were considered to be not fairly assessed in the clinical scenario, items that were poorly worded, and items that were considered high inference (task was difficult to observe or required to much interpretation). Duplicate items were dropped when the other items of a similar nature were found in the instrument. Statistical reasons involved items that had low correlation with the outcome items (R_tet ≤.18); had poor internal consistency as measured by the scale KR-20 with R_pbis ≤.20); and poor statistical fit with the Rasch Model (Mean Square Infit and Outfit values of less than 0.5 or greater than 1.7).

Results

We recorded the performance of 198 paramedics using the difficult airway-management simulation. The encounters between the paramedics and the simulator and their performance on the comprehensive airway management assessment were previously described in detail.¹⁷

The original 127 item instrument can be separated into four hypothetical subsets (subscales), each of which is intended to assess a separate but related skill set. There were 35 items for assessing Back-up Airway, 62 for assessing Intubation, nine for assessing Suction, and 21 for assessing Ventilation. During the first stage of review, all of the Suction items were dropped due to the lack of fidelity with the simulator to produce realistic “liquid.” Several items, which were inadvertently duplicated across subscales were identified and also dropped. However, most items dropped during Stage 1 were eliminated due to poor psychometric qualities: either low tetrachoric correlations with the outcome performance standard or poor fit statistics with the Rasch Model. At the end of Stage 1, there were 76 items remaining (see Table 2).^{Table 3}

At each of stages 2, 3, and 4, we dropped 8, 10, and 21 additional items, respectively. Items were eliminated at these stages for a variety of reasons, some qualitative and others quantitative. Qualitative reasons included limited equipment selection during the simulation, lack of clarity in instructions or expectations, lack of fidelity in the simulation, or the performance item was considered to require too much interpretation by the evaluator (high-inference item). Quantitative reasons primarily involved lack of correlation with the performance standard or lack of correlation with other items in the subscale.

The final version of the AMPC instrument contained 37 items and the three performance standards that clustered into three subscales: Back-Up Airway (KR-20 =.95), Intubation (KR-20 =.88), and Ventilation (KR-20 =.82). All 37 items were considered easily observable and easily rated by a qualified evaluator. The items were internally consistent (or highly correlated with) the other items within their associated subscale, and they were highly correlated with the intended outcome standards. Finally, all items retained in the instrument exhibited good measurement characteristics as defined by Rasch Analysis. The final AMPC instrument is shown in Table 3.

Table 3. Airway Management Proficiency Checklist Instrument (37 items with 3 performance standards) listed by subscale and original reference numbers

Scale - Item No.	Performance Task	YES	NO
V-5	Inserts oropharyngeal (adjunct) airway
V-7	Chooses correct adjunct airway size
V-9	Inserts adjunct airway to proper depth
OV-11	Ventilates patient immediately (w/in 30 sec) with BVM unattached to oxygen
V-15	Ventilates patient at rate of 10–12/min
V-16	Observes BVM technique for 30 seconds: Evaluates volumes (visible chest rise)
V-19	Orients mask correctly
V-20	Uses thenar eminence technique (E-C grip)
V-21	Maintains C-spine precautions during BVM
I-31	Uses straight-to-cuff stylette curvature technique
I-33	Checks equipment for cuff leaks
I-36	Positions head properly
I-40	Grasps laryngoscope with left hand
I-47	Elevates mandible from 45–90 degrees w/laryngoscope
I-55	Flips up epiglottis to expose larynx
I-57	Inserts laryngoscope to appropriate depth
I-58	Moves blade tip smoothly without shaking or jerking
I-61	Maintains view until ETT has stopped advancing
I-67	Passes ETT through cords with limited or no impingement
I-68	Passes tube through cords (laryngoscope in mouth to tracheal placement) in ≤20 seconds
I-70	Disconnects syringe IMMEDIATELY after inflating cuff of ET tube
I-72	Listens over each lung
I-75	Checks end-tidal CO2- After ET Tube placement
I-78	Checks pulse oximeter-After ET Tube placement
I-80	Maintains control over ET tube placement
I-81	Secures ET tube (with device)
OI-84	Successfully intubates within 1 attempts
B-106	Recognizes need for backup airway
B-108	Identifies an appropriate backup airway device
B-110	Checks equipment for cuff leaks
B-114	Immediately inflates cuff, prior to ventilation
B-117	Immediately disconnects syringe after inflating cuff
B-119	Confirms proper placement by auscultation bilaterally over each lung-Backup airway
B-121	Checks end-tidal CO2-after Backup airway placement
B-123	Checks Pulse Oximeter-after Backup airway placement
B-126	Appropriately positions equipment needed for backup airway
B-127	Maintains control over ET tube [or backup airway] after placement
B-128	Secures ET tube (or backup airway device)
B-129	Introduces ET tube or backup airway and advances to proper depth
OB-130	Successfully places backup airway within 1 attempt

O = Outcome Performance Standards; V = Ventilation; I = Intubation; B = Back-up Airway.

Discussion

A fundamental challenge of prehospital education is the confirmation of proficiency of airway procedural skills. This is particularly critical in high risk, low frequency procedures like airway management. With current overall prehospital endotracheal intubation success rates of 85% and complication rates of 48 per 1000 interventions, the need to assure high levels of skill acquisition and retention are essential.²¹

In this study, using a representative population of certified paramedics, we derived a comprehensive airway management checklist using a difficult airway simulation. The goal of this effort was to develop a performance assessment instrument for measuring airway proficiency in prehospital care providers.

The AMPC assessment instrument and the difficult airway simulation were developed because of the absence of psychometrically sound performance assessment instruments. While the National Registry for EMT/paramedics (NREMT) has an initial training checklist, which is well tested, our test cohort represented practicing paramedics, likely with typical post-initial training competencies. We sought to identify important subcategories for the conceptual deconstructed airway management profile that were identified and then used in the development of the AMPC that should serve both as a formative instrument for training or a summative instrument for certification.

The development of the AMPC instrument supplements previous work on airway management by assisting in the evolution of safety and proficiency assessment. Traditionally, paramedics are taught how to perform endotracheal intubation as an isolated skill, and initial proficiency testing remains subjective and nonstandardized. The overall goal of this method of measurement was to change the focus or airway instruction from teaching and assessing airway procedures as an isolated skill (i.e., ETI, back up airway, etc.) to teaching and assessing airway management as a comprehensive set of skills. Through this type of immersive assessment, paramedics can be instructed and assessed in a setting more closely related to that of their regular practice, which should support improved paramedic performance. Furthermore, by adopting a standardized performance assessment, involving a difficult airway, paramedic educators should be able to more accurately discriminate between basic level performers and those who should be considered masters of airway management performance.²² Mastery performance is likely a preferred airway management goal, as it is expected to provide higher levels of skill and safety through structured practice, feedback, and other methods demonstrated by educational research leaders.²³ Airways that are easily managed do not allow for the assessment of the full skill set necessary for clinical competence including back up airway utilization.

Paramedics are expected to manage difficult airways, even though their likely clinical exposure is modest, typically one-to-two cases annually, although greater in some systems.¹⁷ Promoting the use of simulation of a difficult airway and standardization of comprehensive assessment is important, including the development and promotion of mastery standards of performance.^4,21

Limitations

Unfortunately, we were unable to assess a sufficient number of paramedics in order to use more rigorous psychometric methods like exploratory factor analysis (EFA) to establish the structural validity of the AMPC. EFA requires a minimum of 10 subjects per item (i.e., for this study, we would have needed 1310 subjects). Future studies will need to be conducted to: 1) confirm the predictive validity of the derived items in the shorter, 37 item instrument; and 2) evaluate the structural or content validity of the AMPC. Additionally, we did not evaluate inter-rater reliability during the development of the AMPC. Instead we used only one rater to evaluate all 198 paramedic subjects. However, the simplicity of the data gathered (i.e., dichotomized variables) and the ability to review video recordings multiple times will facilitate future research on evaluating the inter-rater reliability of the AMPC instrument. While back up airways were generally evaluated for placement effectiveness, based upon the back-up airway commonly used by practitioners, the benefit of a specific supraglottic or other back up airways were not evaluated. Future studies should include alternative airway analyses to improve generalizability as this information becomes available.^24–26

Finally, due to the shortcomings of contemporary simulation technology, we were unable to accurately assess the performance tasks related to suctioning. As simulators improve and are able to provide viscous airway secretions, we anticipate that the AMPC will change over time to include performance tasks related to suctioning.

These results are also limited to assessing comprehensive airway management in the simulated environment. Additional research will need to be designed to see how well paramedic performance on the AMPC instrument using the difficult airway management case predicts performance in the true clinical setting.

Conclusions

The AMPC represents a psychometrically derived instrument that identified important tasks required for comprehensive airway management. The performance assessment instrument and its associated simulation will contribute to improving training and measuring the performance of paramedic's airway management skills.

Appendix A

symbols

Column 1: Sub-Scales

B =	=	Back-up Airway task
I =	=	Intubation task
O =	=	Outcome task
V =	=	Ventilation task

Column 3: Decision Codes (and Round in which

 item was eliminated)

Qualitative (Q) Items that were judged by panel not to measure well for qualitative reasons

I-	=	Irrelevant to airway management in the field or not relevant to outcome
NI-	=	Not important
F-	=	Not fairly assessed due to scenario or fidelity of simulation, incuding items that: measured selection of tools and equipment when only 1 option was available were not realistically presented because of simulation equipment or materials (suction items) measured tasks that were not performable on this simulator required equipment that was not available were not properly instructed, and not well integrated into scenario
H-	=	High inference items items that were difficulty to assess or observe (task was difficult to observe or required too much interpretation (high inference)
P-	=	Poorly worded item Duplicity (D) Item was redundant with another item Statistical (S) Item performed poorly statistically
SA-	=	Poor relationship with performance standard (outcome)
SB-	=	Poor internal consistency (KR-20)
SC-	=	Poor Rasch fit statistics

• ^*Column 4: Number and percentage of paramedics who properly executed the performance task

• ^†Column 5: KR-20 Reliability Index Value if the item was to be dropped from the related subscale

• ^‡Column 6: Tetrachoric Correlation between item and associated performance standard (outcome)

• ^§Column 7: Rasch measurement difficulty index (high positive numbers = hard; high negative numbers = easy)

• ^||Column 8: Rasch infit statistic (Good fit is considered ≥.05 and ≤ 1.7)

• ^¶Column 9: Rasch outfit statistic (Good fit is considered ≥.05 and ≤ 1.7)

• ^**Column 10: Point Biserial Correlation between item and total score on the subscale of interest