Analyzing interprofessional teamwork in the operating room: An exploratory observational study using conventional and alternative approaches

ABSTRACT

Intraoperative teamwork is vital for patient safety. Conventional tools for studying intraoperative teamwork typically rely on behaviorally anchored rating scales applied at the individual or team level, while others capture narrative information across several units of analysis. This prospective observational study characterizes teamwork using two conventional tools (Operating Theatre Team Non-Technical Skills Assessment Tool [NOTECHS]; Team Emergency Assessment Measure [TEAM]), and one alternative approach (modified-Systems Engineering Initiative for Patient Safety [SEIPS] model). We aimed to explore the advantages and disadvantages of each for providing feedback to improve teamwork practice. Fifty consecutive surgical cases at a Canadian academic hospital were recorded with the OR Black Box®, analyzed by trained raters, and summarized descriptively. Teamwork performance was consistently high within and across cases rated with NOTECHS and TEAMS. For cases analyzed with the modified-SEIPS tool, both optimal and suboptimal teamwork behaviors were identified, and team resilience was frequently observed. NOTECHS and TEAM provided summative assessments and overall pattern descriptions, while SEIPS facilitated a deeper understanding of teamwork processes. As healthcare organizations continue to prioritize teamwork improvement, SEIPS may provide valuable insights regarding teamwork behavior and the broader context influencing performance. This may ultimately enhance the development and effectiveness of multi-level teamwork interventions.

KEYWORDS:

• Operating rooms
• Performance measures
• surgery
• teamwork

Introduction

Understanding and improving operating room (OR) teamwork is essential for preventing surgical patient complications (Anderson et al., 2013; Bohnen et al., 2017; Hu et al., 2012; Jha et al., 2013; K. R. Catchpole et al., 2007; Mazzocco et al., 2009; Sacks et al., 2015; Van Beuzekom et al., 2010; Weaver et al., 2017; Weller & Boyd, 2014; Zegers et al., 2011). Currently, there are a wide range of tools available for studying OR teamwork (Boet et al., 2019; Etherington et al., 2019). Many conventional tools rely on behaviorally anchored rating scales, which assess non-technical skills such as communication, leadership, and situation awareness at the individual or team level. Two tools frequently used to assess and develop the teamwork skills of healthcare professionals are the Operating Theatre Team Non-Technical Skills Assessment Tool (NOTECHS) and Team Emergency Assessment Measure (TEAM) (Boet et al., 2019; Etherington et al., 2019). Although both are quantitative in their approach to teamwork assessment, these two tools rely on two different conceptualizations of teamwork. TEAM considers that the team is the indivisible unit of analysis while NOTECHS considers that the performance of the team is the sum of the performance of each sub-team (i.e., anesthesia, nursing, surgery). Many studies have reported the training benefits of these tools, as the numerical scores derived from their use can provide an opportunity for targeted individual feedback across various non-technical skill domains (Everett et al., 2017; Maignan et al., 2016; McCulloch et al., 2009; Mishra et al., 2008; Morgan, Hadi, et al., 2015; Lauren Morgan, Hadi, et al., 2015; Oseni et al., 2017; Wood et al., 2017).

Recently, there has been increasing recognition of the highly contextual nature of both individual and team performance, which conventional tools like NOTECHS and TEAM may have limited ability to capture (Marriage & Kinnear, 2016; McConnell et al., 2016; Schmutz et al., 2019; Tørring et al., 2019; Willemsen Dunlap et al., 2018). These tools do not account for the complexity and interdependency of individual, team and environmental factors in the OR, perhaps in part based on their intended use for in-person observation of simulated or clinical performance (Boet et al., 2019; Etherington et al., 2019). With the development and increasing use of video recording technologies in surgery (Goldenberg et al., 2017; Langerman & Grantcharov, 2017), there is an opportunity to explore alternative approaches to studying teamwork. Specifically, these technologies may facilitate capturing both quantitative and qualitative information across multiple levels (e.g., individual, team, system). The modified-Systems Engineering Initiative for Patient Safety (SEIPS) model has been used to identify system-based safety threats and resilience factors across many healthcare settings (Bergman et al., 2017; Holden et al., 2013), including the OR (Gurses et al., 2012; Kolodzey et al., 2020), but has yet to be applied specifically to intraoperative teamwork. In this study, we aimed to characterize teamwork using two conventional tools (NOTECHS, TEAM) and an alternative approach (SEIPS) using a video recording technology. We also aimed to explore their potential advantages and disadvantages to understanding teamwork in the OR, with the goal of capturing tangible and clinically relevant information to inform future interventions.

Methods

This study is reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (Von Elm et al., 2008).

Study design

This study used a prospective observational design to assess intraoperative teamwork during cases recorded by the OR Black Box® using two conventional quantitative assessment tools (NOTECHS and TEAM) as well as an alternative approach, the modified-SEIPS human factors model.

Setting

Data collection occurred from January to December 2019. Cases were recorded in one OR equipped with the OR Black Box® (Surgical Safety Technologies, Toronto, Canada) at a tertiary hospital in Ottawa, Ontario, Canada.

Participants

Research Ethics Board (REB) approval was obtained from the Ottawa Health Sciences Network (Protocol #20180152-01 H). This study focused on the three core OR sub-teams for assessment with each tool: nursing, anesthesiology, and surgery. Surgical cases were eligible to be recorded if performed in the OR room where the OR Black Box® was installed. At the time of data collection, only laparoscopic gynecological procedures were collected.

The OR Black Box® implementation at our center has been reported elsewhere (Boet et al., 2021). Briefly, we worked with key stakeholders (e.g., frontline clinicians, Ottawa Health Sciences Network Research Ethics Board) to develop a model of implied consent with several safeguards put in place. Clinicians who identified a priori that they did not want to be recorded were not scheduled to work in the OR Black Box® room. Clinicians could also decline to be recorded in the OR prior to the circulating nurse pressing “record” or inform the research team within 48 hours of the case if they would like their recording to be deleted. As patients were not the focus of the research, no identifying information was collected (e.g., faces were blurred) and the study was not interventional, the REB determined that implied consent was appropriate.

Data source

Intraoperative data were obtained with the OR Black Box® (Surgical Safety Technologies, Toronto, Canada) (Goldenberg et al., 2017) which captured video, audio, and physiological parameters from all consecutive, eligible intraoperative activities. Specific examples of data captured included: team interactions, intraoperative adverse events, rectifications, near misses, infectious risks, safety concerns, resilience supports, surgeon technical performance, the number of people in the room, changeovers, distractions (e.g. door opening, phone ringing), the primary operator (i.e. attending surgeon or trainee), and indicators of time pressure in the room (e.g. external requests for operation updates).

Conventional quantitative teamwork assessment tools: NOTECHS and TEAM

The elements of NOTECHS (Mishra et al., 2009) and TEAM (Cooper et al., 2010) are summarized in Table 1.

Table 1. Summary of tools used to analyze intraoperative teamwork with or Black Box® data.

Tool	Teamwork domains assessed	Level of rating system	Basis of rating system	Item scale
NOTECHS	− Leadership and management − Problem-solving and decision-making − Situation awareness − Teamwork and cooperation	Individual sub-teams: anesthesia, nursing, surgery (individual scores summed to generate full team score)	Based on patient safety considerations (i.e., whether behavior compromises, maintains or enhances patient safety)	1 (below standard) 2 (basic standard 3 (standard) 4 (exceed standard)
TEAM	− Leadership − Task management − Teamwork − Global rating scale	Interprofessional team as a whole for each item	Based on frequency of behaviors (individual items) and overall rating of performance (global rating)	Individual items: 0 (never/hardly ever) 1 (seldom) 2 (about as often as not) 3 (often) 4 (always/nearly always) Global rating scale: 1 (low) to 10 (high)

Evaluation system 1 (individual unit of analysis): NOTECHS

NOTECHS was developed to evaluate OR teams across four dimensions, comprised of 16 elements: (1) leadership and management (leadership, maintenance of standards, planning and preparation, workload management, and authority and assertiveness), (2) teamwork and cooperation (team building and maintaining, support of others, understanding team needs, and conflict solving), (3) problem-solving and decision-making (definition and diagnosis, option generation, risk assessment, and outcome review), and (4) situation awareness (notice, understand, and think ahead). These dimensions and corresponding elements are rated on a scale of one (Behavior compromises patient safety and effective teamwork) to four (Behavior enhances patient safety and effective teamwork) (McMullan et al., 2020). The level of assessment is individual sub-teams (i.e., anesthesia, nursing, surgery), with scores then combined for an overall team score up to 48 (score of 1–4 × 4 dimensions x 3 sub-teams = 12–48). NOTECHS has been found to have concurrent validity, predictive validity, inter-rater reliability (ICC = 0.68–0.88; Cronbach’s α = 0.88), and test-retest reliability (Etherington et al., 2019).

Evaluation system 2 (interprofessional team unit of analysis): TEAM

TEAM considers three domains (leadership, task management, teamwork) in addition to an overall global rating of team performance. Across the three domains, there are a total of 11 items, each rated from 0 (never/hardly ever) to 4 (always/nearly always). The global rating scale scores teamwork performance from 1 (low) to 10 (high). When rating each item, the focus is on the team as a whole rather than a specific individual profession. TEAM has been found to demonstrated content validity, construct validity, internal consistency (Cronbach’s α = 0.78–0.95), inter-rater reliability (ICC = 0.70–0.93), and test-retest reliability (Boet et al., 2019).

In the present study, each clinical case assessed with NOTECHS and TEAM was divided into three phases: before surgical incision (i.e. start of audio-video capture to skin incision), during procedure (i.e. skin incision to skin closure), and after surgical closure (i.e. skin closure to stop capture). For each case that was recorded, performance was rated every 20 minutes within each of these phases.

Alternative assessment tool (individual, interprofessional team, and system levels): modified-SEIPS human factors model

The Systems Engineering Initiative for Patient Safety (SEIPS) model considers the interactions of six components related to patient safety: person, tasks, tools and technology, physical environment, organization, and external environment (Carayon et al., 2006; Carayona et al., 2014; Holden et al., 2013). Within each of these components there are various sub-categories that can be classified as either resilience supports or safety threats. For example, “ambient conditions” is a sub-category of the “physical environment” component, which can be coded as “suboptimal” (i.e., safety threat) or “optimal” (i.e., resilience support). A full description of the SEIPS components, sub-categories, codes, and definitions is provided in Supplementary Material File 1. Inter-coder agreement for this model has been previously found to be moderate (Cohen’s kappa = 0.73) (Kolodzey et al., 2020). We applied the SEIPS system to intraoperative observations recorded by the OR Black Box®, which enabled us to situate teamwork behavior within the broader OR context. For example, suboptimal teamwork could be observed to occur in the context of a cluster of distractions and individual errors, representing various person- and physical environment-level safety threats. Teamwork was rated as “optimal” or “suboptimal” based on patient safety considerations.

Study size

For the purposes of this exploratory study, we determined a priori that we would record data from 50 consecutive eligible surgical cases based on feasibility following the initial implementation of the OR Black Box®. Since the primary goal of this study was to describe teamwork OR performance and explore the advantages and disadvantages of different tools we elected to use subsequent surgical cases. The first 25 were rated using NOTECHS and TEAM, while cases 26 to 50 were rated using the new modified-SEIPS human factors model. Initially, our plan was to compare NOTECHS and TEAMS only, as these were the most promising traditional tools identified in our previous systematic reviews (Boet et al., 2019; Etherington et al., 2019). After assessing 25 cases, we realized that none was as useful as anticipated, and explored other options. SEIPS was not identified by our systematic review as it had not been applied to OR teamwork at the time. We elected to test SEIPS with 25 new cases (as the previous 25 cases had already been deleted as per privacy agreements).

Rater training

All data in this study were analyzed by two expert raters with healthcare and human factors backgrounds. Different raters were involved with the NOTECHS/TEAM and modified-SEIPS assessments. Prior to this study, the raters participated in a three-month structured training program. Reviewers also undergo regular assessment of their rating quality to maintain consistency. Rater training/calibration was performed when the teamwork assessment tools (i.e., NOTECHS and TEAM) were introduced to the OR Black Box® rating process (ICC = 0.74). The cases included in this study were analyzed by one rater only, with 10% of cases analyzed in duplicate to ensure consistent reliability. We therefore report intra-class correlation coefficients (ICCs) for this data only and did not recalculate values for all 50 cases.

Data analysis

Descriptive statistics (mean/median and standard deviation/inter-quartile range) were calculated with Excel Version 16 (Microsoft, Redmond, United States) to summarize teamwork performance and related variables. Qualitative data were summarized narratively. Deductive coding based on the modified-SEIPS human factors model (Supplemental Material File 1) was then applied across each case. Coding was conducted by one research team member (CE) and verified for accuracy by a second research team member (SB, JKB).

Results

Descriptive data

The 50 cases recorded during the study period involved a range of gynecology procedures (Table 2). The most commonly observed procedures were excision of endometriosis (n = 11[22%]) and hysteroscopy for ablation/myomectomy (n = 10[20%]). The mean procedure length across both the first and second set of cases was similar (1 hour : 30 minutes :56 seconds and 1 hour : 38 minutes : 34 seconds, respectively).

Table 2. Summary of included procedures.

Procedure type	n (%)
Bilateral salpingectomy	2 (4%)
Bilateral salpingo-oophorectomy	2 (4%)
Diagnostic laparoscopy	1 (2%)
Excision of endometriosis	11 (22%)
Hysteroscopy	2 (4%)
Hysteroscopy for ablation/myomectomy	10 (20%)
Hysteroscopy for polypectomy	4 (8%)
Laparotomy for cystectomy	2 (4%)
Laparotomy for total abdominal hysterectomy	2 (4%)
Total hysterectomy	8 (16%)
Tubal ligation	2 (4%)
Vaginal hysterectomy	4 (8%)
TOTAL	50

NOTECHS and TEAM

Twenty-five cases were rated using both NOTECHS and TEAM. Teamwork performance was relatively high throughout the duration of each procedure, regardless of the assessment tool used. Median scores and interquartile ranges (IQR) across all 25 cases for each tool are provided in Table 3.

Table 3. NOTECHS and TEAM scores across all cases (n = 25).

	Median (IQR)
	Before incision	During procedure	After closure
NOTECHS
Total – full team (/48)	40 (39.5–43)	38.5 (36.5–41.5)	40 (37–43)
Anesthesia team score (/16)	13 (13–14)	12 (12–13)	14 (13–14)
Nursing team score (/16)	13 (13–15)	13 (12–14)	13 (12–15)
Surgery team score (/16)	14 (13.5–14)	13.5 (12.5–14.5)	13 (12–14)
TEAM
The team leader let the team know what was expected of them through direction and command (/4)	3.5 (3.5–4)	3 (3–3.5)	4 (3–4)
The team leader maintained a global perspective (/4)	3.5 (3–4)	3 (3–3)	4 (3.5–4)
The team communicated effectively (/4)	4 (3.5–4)	3.5 (3–4)	4 (3.5–4)
The team worked together to complete the tasks in a timely manner (/4)	3.5 (3–4)	3 (3–3.5)	4 (3.5–4)
The team acted with composure and control (/4)	3 (3–3)	3 (3–3)	3 (3–3)
The team morale was positive (/4)	3 (3–3.5)	3 (3–3)	3 (3–4)
The team adapted to changing situations (/4)	3 (3–3)	3 (3–3)	3 (3–4)
The team monitored and reassessed the situation (/4)	3 (3–3.5)	3 (3–3)	3 (3–4)
The team anticipated potential actions (/4)	3 (3–3.5)	3 (3–3)	3 (3–4)
The team prioritized tasks (/4)	3 (3–3.5)	3 (3–3)	3 (3–4)
The team followed approved standards and guidelines (/4)	3.5 (3–4)	3 (3–4)	3.75 (3–4)
Global rating of team’s non-technical performance (/10)	8.5 (8–9)	8 (8–9)	9 (8–10)

Out of a total possible score of 48, the median NOTECHS score before incision, during the procedure, and after the closure was 40.0 (IQR = 39.5–43), 38.5 (IQR = 36.5–41.5), and 40.0 (IQR = 37–43), respectively. Within the NOTECHS scale, median ratings for anesthesia, nursing, and surgical sub-teams also showed little variation (Table 3). Scores for specific non-technical skills domains are provided in Supplemental Material File 2.

The median global rating of teamwork performance when using the TEAM scale was 8.5 (IQR = 8–9) before incision, 8 during the procedure (IQR = 8–9), and 9 after closure (IQR = 8–10), out of a total possible score of 10. Little variation was observed across the scores for individual TEAM items (Table 3).

Modified-SEIPS human factors model

Across the 25 observed cases, teamwork performance was found to be relatively high (Table 4). Teamwork observations were noted as they occurred, resulting in a median of 20 observations per case (IQR = 15–21). A total of 390 optimal teamwork behaviors was observed, compared to just 16 suboptimal observations.

Table 4. Teamwork performance observations model (n = 25).

Teamwork behaviours observed*	n (%) Optimal observations	n (%) Suboptimal observations
Communication and teamwork	113 (29%)	4 (25%)
Situational awareness	208 (53.5%)	0
Decision making	10 (2.6%)	0
Leadership	30 (7.7%)	2 (12.5%)
Task management	29 (7.4%)	10 (62.5%)
TOTAL	390	16

*Based on traditional teamwork rating systems.

Optimal and suboptimal teamwork behavioral observations mapped across all 25 cases in the context of the physical environment (e.g., suboptimal ambient conditions) are summarized in Table 5. Across all surgical phases, distracting workflow sounds (e.g., door opening and closing) were the most frequently observed safety threat for the included cases (n observations = 3059; median = 81; IQR = 16.5–154.5). The number of these distractions was highest during the surgical procedure (n observations = 1046) followed by the emergence and extubation phase (n observations = 280). While these distractions occurred, teamwork performance was still observed to be optimal, with situational awareness behaviors in particular peaking during both the preoperative count (n observations = 27) and the surgical procedure itself (n observations = 143). Suboptimal teamwork was infrequently observed across the 25 cases, with six observations occurring during anesthesia setup, one observation occurring during intubation, and one observation occurring during bed transfer.

Table 5. Teamwork and distracting workflow sounds observed during the surgical procedure phase across cases (n = 25).

Observation	n
Communication and teamwork	52
Situational awareness	143
Decision-making, leadership, and task-management	33
Distracting workflow sounds	1046

Analysis of one case rated using the modified-SEIPS human factors model is provided in Table 6. Resilience supports were observed at the person (individual or team) level across all surgical phases except induction, intubation, dressings and application of other devices, bed transfer, patient exit, and room turnover. Organizational level resilience supports were noted during the timeout, surgical procedure and postop counts (effective policies/procedures; strong safety culture). Optimal teamwork was observed during most surgical phases. Of the 18 observations of optimal teamwork observed during this case, four involved the whole interprofessional team and four involved the surgical team only. Situational awareness was the most frequently observed teamwork domain.

Table 6. Case example: Intraoperative observations analyzed with the modified-SEIPS human factors model.

Procedure: Diagnostic laparoscopy CaseID: 037 Team members: surgeon, surgical resident, anesthesiologist, circulating nurse, scrub nurse (5) Primary operator: surgeon
		Resilience supports		Safety threats
Surgical phase	Start – end time	Observation	SEIPS Classification	Observation	SEIPS Classification
Induction	0:00:00 – 00:02:25	N/A	N/A	Door opened 5x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Intubation	00:02:25 – 00:06:12	N/A	N/A	Door opened 5x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Intubation until Prep and Drape	00:06:12 – 00:26:47	Situational awareness (whole team): optimizes operating conditions before starting	Person – effective teamwork – shared mental model	Door opened 11x Task management (anesthetic team): performs IV access without gloves	Physical environment – suboptimal ambient conditions – distracting workflow sounds Person – unsafe acts – protocol violation
Positioning	00:07:10 – 0:22:14	Situational awareness (surgical team): identifies anatomy/pathology clearly	Person – effective communication – verbalize/narrate action	Door opened 8x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Prep and Drape	00:26:47 – 00:37:59	Task management (nursing team): uses time to prepare other tasks	Person – anticipatory action – proactive task completion	Door opened 10x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Foley	00:37:29 – 00:38:15	N/A	N/A	N/A	N/A
Timeout	00:39:09 – 00:39:51	Communication and teamwork (whole team): provides briefing and clarifies objectives and goals before commencing operation	Person – anticipatory action – establishing next steps; Organization – effective policies/procedures – timeout	N/A	N/A
Surgical Procedure	00:45:05 – 02:09:02	Situational awareness (anesthetic team): keeps ahead of situation by giving fluids/drugs Situational awareness (surgical team): verbalizes what equipment may be required later in operation Situational awareness (surgeon to anesthesiologist): optimizes operating conditions Situational awareness (anesthesiologist to nurse): optimizes operating conditions Situational awareness (nursing team): requests equipment from appropriate person before it is required by surgeon Situational awareness (surgeon to nurse): monitors ongoing noise in the room Communication and teamwork (nurse to surgeon): using appropriate level of confidence to seek clarification Communication and teamwork (whole team): talks about the progress of the operation Decision-making (Surgical team): initiates balanced discussion of options, pros and cons with relevant team members Decision-making (anesthetic team): keeps ahead of situation by giving fluids/drugs Leadership (surgical team): provides constructive criticism to team members Surgeon OSATS score: 31/35	Person – anticipatory action – proactive task completion Person – anticipatory action – proactive team management Person – effective teamwork – shared mental model Person – effective teamwork – shared mental model Person – anticipatory action.- proactive task completion Person – effective communication – voicing concerns Person – effective communication – task verification Person – effective communication – communicating progress Person – high-performance behavior – evaluating circumstances; effective teamwork – collaborative decision-making Person – anticipatory action – proactive task completion Person – strong leadership – no criticism; Organization – strong safety culture – communicating mistakes Person – high-performance behavior – effective technique	Door opened 25x One changeover: Circulating nurse Noise occurred 3× (e.g., cell phone ringing) Two external requests for operation update Thermal injury x2 due to wrong orientation Mechanical injury due to too much force Bleeding due to inadequate visualization – corrective measure taken (hemostasis via energy) Infectious risk: instruments dropped	Physical environment – suboptimal ambient conditions – distracting workflow sounds Organization – ineffective staff management – staff change Physical environment – suboptimal ambient conditions – distracting electronic sounds Tasks – disruptions – unnecessary verbal interruption Person – unsafe acts – substandard skill/technique error Person – unsafe acts – substandard skill/technique error Tools and technology – substandard functionality/utility – malfunction Person – unsafe acts – substandard skill/technique error
Postop Counts	02:08:33 – 2:09:45	Communication and teamwork (nurse to surgeon): communicates that counts are correct	Organization – effective policies/procedures – instrument count	N/A	N/A
Emergence and Extubation	02:09:03 – 02:33:55	Communication and teamwork (whole team): cooperates with others to achieve goals Communication and teamwork (anesthetic team): providing physical, cognitive or emotional help to other members of the team	Person – effective teamwork – team harmony Person – high-performance behavior – paying attention	Door opened 22x Noise occurred 3× (e.g., cell phone ringing)	Physical environment – suboptimal ambient conditions – distracting workflow sounds Physical environment – suboptimal ambient conditions – distracting electronic sounds
Dressings and Application of Other Devices	02:10:47: 2:11:38	N/A	N/A	N/A	N/A
Bed Transfer	02:16:13 – 02:20:34	N/A	N/A	Door opened 7x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Patient Exit	02:36:53 – 02:38:27	N/A	N/A	Door opened 4x	Physical environment – suboptimal ambient conditions – distracting workflow sounds
Room Turnover	02:37:39: 02:39:16	N/A	N/A	N/A	N/A

Safety threats were primarily related to the physical environment, and in particular, suboptimal ambient conditions in the form of distracting workflow or electronic sounds. These occurred in nearly all phases of the surgery. Safety threats were not observed during the Foley insertion, timeout, postop count, dressings and application of other devices, or room turnover.

Discussion

This study characterized intraoperative teamwork using conventional rating tools (NOTECHS and TEAM) and an alternative systems-based approach (modified-SEIPS model), using the OR Black Box® across 50 gynecological procedures at a Canadian academic center. Teamwork performance was observed to be relatively high across each phase of the surgical procedure and across cases rated with NOTECHS and TEAMS, with little variation in median ratings of teamwork skills. For cases rated with the modified-SEIPS tool, both optimal and suboptimal teamwork behaviors were observed at different phases of the procedure and between cases. In addition, safety threats, such as a high level of door traffic were frequently observed but did not appear to negatively impact the quality of teamwork.

Based on our findings, there are several potential advantages and disadvantages to consider when selecting a tool for studying OR teamwork. Rating performance with NOTECHS or TEAM may be appropriate for use when summative assessment (i.e., quantify, or pass/fail) is desired or to describe overall patterns or trends, including the association between performance scores and patient outcomes. An overall rating of individual non-technical skills or interprofessional team performance may also be useful for determining whether an intervention has been effective (i.e., if scores increase) (Marriage & Kinnear, 2016). These tools may be less helpful, however, for providing substantive formative feedback to teams. This may be particularly challenging when teams exhibit little variability in their performance.

For consistently high performing teams, such as those observed in our study, NOTECHS and TEAM scores provided limited insight regarding practice implications. That is, the practice implications of a rating of 3 out of 4 may be unclear and determining whether a behavior should be rated as a 3 or a 4 may be arbitrary (Marriage & Kinnear, 2016). Although some guidance for using rating scale is provided (Cooper, 2012; Mishra et al., 2009), our practical experience of using these scales for summative assessed with research showed that extensive “rater calibration” was required to obtain inter-rater agreement on scores (Boet et al., 2013). NOTECHS and TEAM ratings are also based on different considerations: TEAM is based on behavior frequency but does not account for safety implications, while NOTECHS is based on safety implications but does not account for behavior frequency. Conversely, with the modified-SEIPS model, we were able to identify in detail what teams did well, including the content and of the interaction, the team members involved, the frequency of observed teamwork behaviors, and additional contextual information (e.g. environmental factors). This could be used to provide feedback to the observed team and others following a positive deviance approach (Lawton et al., 2014). That is, high-performing teams could be encouraged to apply the positive specific skills and behaviors they demonstrated in one intraoperative phase or during a time of numerous safety threats to other periods. In our study, few teamwork behaviors were observed during patient entry, for example. Yet, other studies suggest effective teamwork during patient entry is associated with greater patient satisfaction (Fregene et al., 2017; Goh et al., 2011; Lyu et al., 2013). Thus, just because a team is generally high-performing and achieves their intended outcomes (i.e., is a team that would score 3 out of 4 or 4 out of 4), does not mean there are not areas of practice that can be strengthened or improved moving forward. In another of our included cases where primarily positive teamwork behaviors were observed, an infection risk was still present as a result of poor task management (e.g., anesthesia not using gloves for IV access or intubation) that went undetected by the team (i.e. suboptimal situational awareness). This type of near miss and related teamwork behaviors represent opportunities to learn from and improve patient care that not necessarily captured when teams are assigned an overall positive numerical score for the surgery, like with NOTECHS or TEAM scores. Spreading real examples of effective teamwork to other lower performing teams during patient safety rounds or other team meetings may be a way to promote reflexive practice. (Schon, 1995) Reflexive practice is key to successful practice change rather than focusing solely on examples of failure or a “find and fix” reactive approach (Lawton et al., 2014; Schnittker & Marshall, 2015; Weaver et al., 2017). Focusing on “the behaviors, processes, and systems contributing to resilient, safe care” enables a proactive and longitudinal approach to patient safety that may be better suited to the complex and dynamic OR environment (Lawton et al., 2014). This may also be an opportunity to explore how organizational-level resilience supports can be better incorporated into the OR environment to support teams on a daily basis. Interestingly, we did not observe any safety threats in our case example during the surgical phases that reflected effective organizational policies and procedures (i.e. timeout and postoperative count).

While numerical ratings certainly have value, when the goal is to understand how and why teams performed in a certain way, and the path they took to get there, an alternative approach that provides a more complete picture of what takes place in the OR may be preferred. An approach like the modified-SEIPS model may also facilitate distinguishing between the different “layers” of team performance, separating individual and collective behaviors and viewing them in context (Marriage & Kinnear, 2016). As recording technologies like the OR Black Box® continue to evolve, researchers may also wish to consider additional forms of analysis that can be conducted to explore additional aspects of teamwork such as communication patterns or subtle behaviors involved in positive or negative interactions. Clinical interactions are undoubtedly complex. It is therefore also possible that combined methods of assessment may offer maximum insight when feasible, drawing on the strengths of each approach (Marriage & Kinnear, 2016).

Much of what we have learned about OR teamwork over the last decade is based on studies that have applied conventional “quantitative” tools to a limited number of in-person observations or simulated practice (K. Catchpole et al., 2008; Lingard, 2004; Mazzocco et al., 2009; Mishra et al., 2008). As a result, conclusions about teamwork are generally subject to the limitations of the available tools, which may not adequately capture the complex and multilevel nature of teamwork in the modern-day OR (Boet et al., 2019; Etherington et al., 2019; Krokos et al., 2009; Nurok et al., 2011). Specifically, the same level of performance can be reached via different routes, and assigning a single score to a complex social phenomenon such as teamwork may not always enhance our ability to make tangible recommendations for improving performance (Kell et al., 2017; McConnell et al., 2016; Willemsen Dunlap et al., 2018). As research continues to evolve, it will be important to explore how various approaches to studying teamwork can be optimized for changing practice.

Limitations

This study included only gynecology laparoscopic procedures from one academic hospital and generalizability to other specialties may thus be limited. The cases observed were relatively low risk and it is possible the teams observed would have performed differently in other circumstances. A Hawthorne effect is also a possibility; however, such an effect would be unlikely to negatively affect performance. Another potential limitation is that we did not collect personal identifiers, which prevented us from looking at the impact of specific individuals on teamwork performance, and it is possible that the same individuals were assessed multiple times. However, the goal of this study did not required collection of personal identifiers and ensuring privacy and confidentiality of data contributes to a constructive “no blame culture.” We also acknowledge that analysis using the modified-SEIPS model can require greater resources than traditional tools. This type of in-depth analysis was facilitated by the OR Black Box® and could become more accessible as the use of this or similar technologies becomes more common across various institutions. There may also be value in sharing data or lessons learned from sites equipped with an OR Black Box® with other sites who have not implemented this technology. Finally, it should be acknowledged that different sets of cases were assessed with NOTECHS/TEAMS and the modified-SEIPS human factors model; however, the goal of this work was to experience the use of assessment tools relying on several different approaches to analyze intraoperative interprofessional teamwork and explore the advantages and disadvantages of each tool, and we did not aim to directly compare psychometric properties of the assessment tools used.

Conclusions

NOTECHS and TEAM provided summative assessments and overall pattern descriptions, while SEIPS facilitated a deeper understanding of teamwork processes. As healthcare organizations continue to prioritize teamwork improvement, SEIPS may provide valuable insights regarding teamwork behavior and the broader context influencing performance. This may ultimately facilitate the development of multi-level teamwork interventions and optimize their impact on both provider and patient outcomes.

Acknowledgements

Sandy Lam, RRT, MSc for study coordination. Lisa Calder, MD for sharing advice on the implementation of the Operating Room Black Box®.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

De-identified data are available upon request.

Additional information

Funding

This study was funded by The Ottawa Hospital Academic Medical Organization (TOHAMO) Innovation Fund and by a research operating grant obtained from the University of Ottawa Department of Anesthesiology and Pain Medicine. Dr. Boet was supported by The Ottawa Hospital Anesthesia Alternate Funds Association and the Faculty of Medicine, University of Ottawa with a Tier 2 Clinical Research Chair.

Notes on contributors

Sylvain Boet

Sylvain Boet is an Associate Professor and Scientist at the University of Ottawa and Ottawa Hospital Research Institute (OHRI).

Joseph K. Burns

Joseph K. Burns is Clinical Research Coordinator at OHRI.

Jamie Brehaut

Jamie Brehaut is Senior Scientist at OHRI.

Meghan Britton

Meghan Britton is a Nurse Educator at The Ottawa Hospital.

Teodor Grantcharov

Teodor Grantcharov is Professor at the University of Toronto.

Meghan McConnell

Jeremy Grimshaw is a Senior Scientist at OHRI.

Meghan M. McConnell is Associate Professor at the University of Ottawa.

Glenn Posner

Glenn Posner is Associate Professor at the University of Ottawa.

Isabelle Raiche

Isabelle Raiche is Assistant Professor at the University of Ottawa.

Sukhbir Singh

Sukhbir Singh is Associate Professor at the University of Ottawa.

Patricia Trbovich

Patricia Trbovich is Associate Professor at the University of Toronto.

Cole Etherington

Cole Etherington is Senior Research Associate at OHRI.