The development of the helicopter non-technical skills (HeliNOTS) behavioural marker systems

Abstract

Behavioural marker systems (observational frameworks geared towards the assessment of non-technical skills by way of behavioural markers) exist across a variety of high-risk occupations, however, no identifiable system currently exists developed from rotary operative data. In this study, discussion groups (n = 9) were undertaken with subject matter experts (n = 20)—including pilots and technical crew operating across search and rescue and offshore transport environments—with the objective of identifying role-specific behavioural markers. Systems were reviewed on an iterative basis by the academic team and received final reviews by additional subject matter experts (n = 6). Two behavioural marker systems were constructed: HeliNOTS (O) for offshore transport pilots and HeliNOTS (SAR) for search and rescue crews; each with domain-specific behavioural markers. Both represent a significant step towards a nuanced approach to training and assessment of helicopter flight crews’ non-technical skills and are the first publicly available systems tailored to these distinct mission types.

Practitioner summary: There is no publicly available behavioural marker system based on data from rotary operatives. Across this study, two prototype systems were developed: HeliNOTS (SAR) for helicopter search and rescue, and HeliNOTS (O) for helicopter offshore transport. Both HeliNOTS systems represent a nuanced approach towards rotary CRM training and assessment.

Keywords:

• Crew resource management
• pilot/crew behaviour
• pilot decision making
• situation awareness
• training evaluation

Introduction

Helicopters are uniquely flexible aircraft that are able to fly lower and access more remote terrain than fixed-wing aircraft; consequently they are used for a variety of high-risk operations including offshore transport, search and rescue and emergency medical services, forestry operations, and fire-fighting (De Voogt, Uitdewilligen, and Eremenko 2009). Such operations require crews to fly in hazardous conditions, including darkness, inclement weather, and poor visibility (Aherne et al. 2019; Christensen et al. 2021; Setten and Lein 2019), as well as presenting complex landing situations, sometimes on undesignated sites (Baker et al. 2006; Hinkelbein, Schwalbe, and Genzwuerker 2010). The level of risk associated with helicopter flight is emphasised within the UK Civil Aviation Authority (CAA) aviation safety review report, with 1120 occurrences reported for offshore, onshore, and emergency service helicopter flights combined in 2019, 11 of which were rated as high severity (CAA 2019).

An important component within aviation safety management, and a key mechanism for supporting pilot effectiveness and safety, is crew resource management (CRM), defined as ‘the effective utilisation of all available resources (e.g. crewmembers, aircraft systems, supporting facilities and persons) to achieve safe and efficient operations’ (Flin 2019, 185). A fundamental aspect of CRM is the enhancement of pilot non-technical skills (NTS), the social (inter-personal) and cognitive (thinking) skills necessary, alongside technical knowledge, for safe and effective work performance (Flin, O’Connor, and Crichton 2008). The importance of these skills is emphasised within aviation safety reports, where lapses in NTS are related to adverse incidents, for example, loss of situation awareness in low visibility conditions was linked to a helicopter pilot losing their visual reference points on approach to landing (CAA 2019). In common with fixed-wing airlines, to comply with national regulations (e.g. see CAA 2016), helicopter operators must train and assess their pilots’ CRM skills on a regular basis. To be effective, the basis for NTS training within CRM should ideally be a tailored behavioural marker system (BMS); an observational tool that provides a taxonomy of the relevant skills, elements, and associated behaviours for training and assessment purposes (Flin et al. 2003).

Despite the importance of a tailored system for training purposes, no such system, to our knowledge, currently exists for civil helicopter pilots. Instead, CRM training and assessment within helicopter operators is often based upon the NOTECHS system (O’Connor et al. 2002), which was produced for the aviation context, but was primarily based on fixed-wing operations and does not distinguish between different flight, aircraft, or mission types.

Recognition of variation in skill use and emphasis across mission types appears to be particularly important for helicopter flight. Recent research has reported variations in the categories of NTS skills necessary for specific helicopter mission types (offshore transport; search and rescue), as well as differences in their emphasis and execution related to their mission aspects (Hamlet, Irwin, and McGregor 2020). This included the importance of a concept known as cognitive readiness which, while applicable to offshore transport operations, should be considered part of an everyday skillset for search and rescue personnel—where solving problems, reacting to unforeseen circumstances, and maintaining resilience are all crucial. To address the lack of specific helicopter crew NTS frameworks for particular mission types, this paper describes the development of two tailored BMSs for helicopter flight: HeliNOTS (O) (for offshore transport pilots) and HeliNOTS (SAR) (for search and rescue pilots and technical crew).

Developing a BMS

BMSs are used to support the observation, evaluation, and training of key NTS within high-risk industries and professions. Each system has unique components that are context and/or role-specific and must be developed through a rigorous research process with the workers who will be using the system (Flin and Martin 2001; Yule et al. 2006), rather than transplanting a generic system from one industry, or one job role, to another. The need for specificity is highlighted through the variation in the elements, and behaviours, required across different healthcare roles: SPLINTS (Mitchell et al. 2010), for example, breaks the core NTS skills of scrub practitioners (operating within a surgical team) into three categories (Situation awareness, Communication and teamwork, Task management) while ANTS (Fletcher et al. 2003) outlines the skills of anaesthetists in four (Task management, Teamworking, Situation awareness, Decision-making). As such, each has similar, yet role-specific, associated elements and behavioural markers. In each of the described systems, the steps towards development of the system encompassed a literature review, interviews with practitioners, and a series of focus/discussion groups alongside academic or expert review to ensure the skills, elements, and behaviours were all reflective of actual practice.

An additional consideration for the development of BMSs for helicopter flight is the number and function of flight crew. Specifically, while offshore transport (OT) helicopter crews consist of two pilots (pilot flying and pilot monitoring), search and rescue (SAR) crews encompass two pilots plus technical crew (TC) located in the rear of the aircraft. These crew members, made up of a winch-operator and winch-paramedic, aid in the planning, coordination, and preliminary medical assessment and care of a casualty. Their input is frequent; across 2380 UK-based SAR missions in 2019, 52% were classified as rescues or recoveries (Department for Transport 2020). Given that studies have noted the inputs and skills of the TC [e.g. hazard perception, communication, team coordination, mission planning (Morowsky and Funk 2016; Plant and Stanton 2014; Plant and Stanton 2016)], and their role in flight safety, it is essential to include TC behaviours and elements in the development of the SAR BMS.

Preliminary framework

The BMSs described within this paper were built upon a body of research exploring the NTS utilised by OT and SAR crews. An initial NTS framework for the pilots of both flight types was reported by Hamlet, Irwin, and McGregor (2020), derived from thematic data analysis across a critical-incident based interview study. Specifically, the framework encompassed six skill categories: situation awareness, decision-making, task management, communication, teamwork, and leadership for both flight types, plus an additional skill category of cognitive readiness for SAR crews (described previously). This interview study was followed by a focus group study designed to further explore the relevance and use of each skill, enhance the existing frameworks, ensure that industry-specific terminology was used, and identify the factors influencing NTS performance across these flight crews (Hamlet et al. 2019; Hamlet 2021). The focus group findings, in addition to a further interview critical-incident based study with SAR TC personnel (Hamlet 2021), confirmed the six-skill category framework, and the addition of the cognitive readiness skill for SAR crews.

Study aim

Building upon the Hamlet et al. studies reported above, the next step in the development of the HeliNOTS (SAR) and HeliNOTS (O) BMSs was to confirm the core NTS categories and elements identified in the research, then develop observable behavioural markers (both positive and negative) for each element. This objective was achieved through a series of subject matter expert discussion groups involving helicopter CRM trainers and pilots, followed by expert and academic reviews, to produce the prototype versions of HeliNOTS (O) and HeliNOTS (SAR). Given that the studies mentioned above identified differences between how the pilots and TC of a SAR aircraft may utilise their skills, the prototype of HeliNOTS (SAR) included both shared and role-specific behavioural markers.

Methods

Stages of development

The study was conducted over four key stages:

1. Academic review and conversion of the OT and SAR helicopter crew NTS frameworks (Hamlet, Irwin, and McGregor 2020; Hamlet 2021) into a format suitable for use within the discussion groups. This is referred to as the ‘refined version/s’ below.

2. Discussion groups with subject matter experts (helicopter pilots, CRM trainers) to confirm the key skills and elements produced via Hamlet and colleagues alongside generating observable behavioural markers.

3. Refinement of the system between discussion group sessions by way of academic review.

4. Final subject matter expert review.

Ethical approval

This research was approved by the University of Aberdeen, School of Psychology ethics committee, Scotland (PEC/4460/2020/1 and PEC/4508/2020/6). Informed consent was obtained from each participant.

Academic team

The academic team, referred to below, comprised the first, second, and third authors, two of whom have more than ten years of NTS research experience.

Participants

An essential component of BMS production is the involvement of subject matter experts (SMEs), utilising their input across various levels of development (e.g. behavioural marker production, review). Experienced OT pilots, SAR pilots, and SAR TC personnel were the SMEs for this study. They were recruited, via email and social media (LinkedIn), and originated from a SAR and OT operator based in the UK (Operator A), an OT operator based in the UK with a SAR operation in Ireland (Operator B), an Australian Air Ambulance Service (Operator C), and a Northern European military SAR operator (Operator D).

Four SAR training consultants were recruited from different consultancies in the UK (n = 2), Central Europe (n = 1), and Australia (n = 1).

Design criteria

Before devising a method for developing the HeliNOTS BMSs, several design criteria were outlined based upon those used for the development of NOTECHS (O’Connor et al. 2002):

• Categories and elements should be based upon the maximum achievable mutual exclusivity with minimal overlap

• The systems should be pragmatic, concise, and useable, relying on as few categories and elements as possible to encompass critical behaviours

• The terminology used should be recognisable to helicopter pilots and CRM trainers and assessors

• All social and personal resource behavioural markers (associated with leadership, teamwork, communication, workload management) be directly observable, and all cognitive behavioural markers (associated with decision-making, situation awareness, cognitive readiness) be either directly observable through action or the use of communication.

Procedure

Framework pre-development (stage 1)

Before liaising with SMEs, the academic team developed refined versions of each role’s NTS framework (i.e. SAR pilot, SAR tech crew, and OT pilot), derived from the research described previously (Hamlet, Irwin, and McGregor 2020; Hamlet 2021). This refinement process centred upon removing unobservable elements and amalgamating any elements interpreted as overlapping. These refined frameworks provided the basis for SME discussion groups, with the OT version containing 29 elements and the SAR version containing 27 elements. At this stage, it was decided that cognitive readiness should be included within the refined OT framework, to assess whether behaviours associated with the skill were readily volunteered by the pilots.

SME discussion groups and academic review (stages 2 and 3)

Following the method outlined by O’Connor et al. (2002), discussion groups for HeliNOTS (O) and HeliNOTS (SAR) were conducted by the first and fourth authors with SMEs (experienced pilots, TC personnel, and CRM trainers). While similar to focus groups, where participants are encouraged and led by a moderator to discuss specific topics under investigation (Acocella 2012), the term ‘discussion group’, as used here, describes the more interactive role the researchers played within the discussion. This could involve directly challenging or offering contributions as part of the discussion with the participants or playing an active role alongside the participants in developing the system collaboratively.

The first discussion group of HeliNOTS (O) and HeliNOTS (SAR) were conducted in person; at Operator B’s office in Aberdeen in the case of HeliNOTS (O) and at a Scottish SAR base in the case of HeliNOTS (SAR). However, due to Covid-19 restrictions, all other groups were conducted via Zoom meetings. In each session, members of the academic team led the group through each NTS element asking for examples of positive and negative behavioural markers associated with the element. The researchers would note suggestions and discuss research findings on that concept with the participants to identify additional behavioural markers. The researchers would also engage with the participants on potential overlaps (across categories, elements, and behavioural markers) or behavioural markers that may be problematic to observe. While no single group could dictate a categorical or elemental change in the framework they were shown, the academic team would consider this feedback against the other groups and come to a decision on pertinent amendments.

Discussion groups followed an iterative process, whereby the first SME discussion group for each system was shown the refined version of the NTS framework (mentioned previously) and asked to discuss potential behavioural markers on an element-by-element basis until the whole system was addressed or 90 min had expired. From there, the academic team would review the newly proposed behavioural markers and either approve them, edit marker wording, merge overlapping markers, or move markers to better-fitting elements. The next consecutive group/s would be given this updated version to consider in their discussion group session. This process was repeated until (a) the academic team could confirm that the design criteria were satisfied and (b) the systems were met with general approval during the final discussion group.

Discussion groups were held from February to September 2020. These ranged from an hour to 90 minutes and were recorded for reference. Each role-specific discussion group contained 2–3 participants (see Table 1).

Table 1. HeliNOTS SME discussion and review groups undertaken February–September 2020.

System	Discussion/review group	Participants
HeliNOTS (O)	Pilot discussion group 1	3 × OT captains (Operator B)
	Pilot discussion group 2	2 × OT captains (Operator B)
	Pilot review group	1 × current CRM trainer, 1 × ex-CRM trainer (Operator B)
HeliNOTS (SAR)	Pilot discussion group 1	2 × SAR captains (Operator A)
	Pilot discussion group 2	1 × SAR captain, 1 × SAR co-pilot (Operator B)
	Pilot email review	1 × SAR captain, 1 × SAR co-pilot (Operator B)
	Technical crew discussion group 1	3 × SAR dual-role technical crew members (Operator B)
	Technical crew discussion group 2	1 × SAR winch paramedic, 1 × SAR dual-role technical crew member (Operator B)
	Technical crew discussion group 3	1 × SAR paramedic, 1 × SAR winch operator (Operator A)
	Technical crew discussion group 4	1 × SAR dual-role technical crew member (Operator C), 1 × SAR training consultant (Consultant A)
	Technical crew discussion group 5	1 × SAR winch operator, 1 × SAR winch paramedic (Operator D)
	Email review	3 × SAR training consultants (Consultants B, C, and D)

Review (stage 4)

The review stage was initiated once discussion groups had ceased and preliminary behavioural markers had been established. The SMEs recruited to take part in reviewing the systems were experienced individuals within the domains who were either known to the research team or had volunteered to take part in the study. The format of review groups varied between the systems due to recently implemented Covid-19 restrictions.

The SMEs acting as reviewers were sent the current iteration of the system corresponding to their role/expertise by email and asked to review it in its entirety. They were to report any questions or concerns about (a) the system’s usability and feasibility, (b) the absence of key behaviours, (c) any overlap between elements within a skill or between skills, and (d) the wording of behavioural marker examples. For HeliNOTS (O), a review group was undertaken via a group Zoom meeting with two SMEs from the same organisation. For the pilot section of system SAR, a group of pilots (which had previously participated in a development discussion) were asked to undertake individual reviews on the entire system before the TC personnel’s input. This was due to the issues encountered with recruitment as a result of Covid-19, where in person access to participants had been restricted. However, using this group again was considered an appropriate course of action given that the participants were highly integrated into the operator’s CRM training processes and would have valuable insights into system usability and feasibility via a separate review. After TC discussion groups were concluded, the entire system (including SAR pilot and TC behavioural markers) was reviewed by three SAR training consultants.

Structure of the HeliNOTS systems

HeliNOTS (O) and HeliNOTS (SAR), are made up of five main categories, each divided into elements and associated behavioural markers (see Figure 1 for example of the category, element, behavioural marker relationship). Both systems share the same four core categories: Communication, Leadership and Teamwork, Situation Awareness, and Decision-making. System (SAR) includes the unique category of Cognitive Readiness, highlighting this skill’s fundamental importance in relation to the SAR environment. System (O) includes a fifth category of Workload Management which was considered to be essential for OT operations. The categories and elements of each system can be viewed in Table 2. Full systems, outlining all of the behavioural markers for HeliNOTS (O) and HeliNOTS (SAR), instructions for users, and the HeliNOTS rating scale, can be accessed online at: https://research.abdn.ac.uk/applied-psych-hf/helinots.

Figure 1. The structure of HeliNOTS (SAR).

A figure outlining the five non-technical skill categories of HeliNOTS (SAR). The non-technical skill of Communication is used as an example and its three elements are outlined. For the element of Exchanging Information a positive and negative behavioural marker are outlined.

Table 2. Composition of the prototype HeliNOTS (O) and HeliNOTS (SAR) behavioural marker systems.

Shared categories	Elements
Communication	Exchanging information
	Giving instructions
	Providing feedback
Leadership and teamwork	Guiding task behaviours
	Monitoring other crew members
	Sharing task activities
	Setting and maintaining crew atmosphere
Situation awareness	Gathering information
	Comprehending informational elements
	Anticipating future states
Decision-making	Identifying and selecting options
	Reviewing course of action
HeliNOTS (SAR) unique category
Cognitive readiness	Utilising preparedness
	Maintaining resilience
	Applying problem-solving
HeliNOTS (O) unique category
Workload management	Maintaining standards
	Coping with task demands
	Prioritising duties

HeliNOTS (O) and HeliNOTS (SAR), while comprising the same elements across the four shared categories possess varying behavioural markers to each other. A few examples of behavioural markers from each system can be viewed in Table 3. Included in these examples are role-specific behavioural markers from within HeliNOTS (SAR) (i.e. pilot behavioural markers, TC behavioural markers).

Table 3. Example positive and negative behavioural markers from a range of HeliNOTS (O) and HeliNOTS (SAR) categories, including HeliNOTS (SAR) role-specific markers.

Category	Elements	System	Positive behavioural marker	Negative behavioural marker
Leadership and teamwork	Sharing task activities	HeliNOTS (SAR)	Effectively synchronises task/s with other crew members (Shared marker)	Refuses to take on tasks not deemed to be part of their role (Technical crew specific marker)
		HeliNOTS (O)	Ensures both pilots are clear on role and associated activities	Acts in isolation from other pilot
Situation awareness	Gathering information	HeliNOTS (SAR)	Identifies escape points (Pilot specific marker)	Fails to recognise escape points (Pilot specific marker)
		HeliNOTS (O)	Frequently scans environment	Becomes fixated on task distractions
Cognitive readiness	Maintaining resilience	HeliNOTS (SAR)	Displays empathy and care for casualty and those involved (Technical crew specific marker)	Begins to perform outside of the standard procedures as a result of stress/distress/fatigue (Shared marker)
Workload management	Prioritising duties	HeliNOTS (O)	Shows awareness of time and manages planned tasks and activities	Spends too much time on individual tasks

The HeliNOTS rating scale

Both HeliNOTS systems include a tailored rating form and follow a generic five-point rating scale (1—poor/sub-standard, 2—marginal, 3—acceptable, 4—good, 5—very good), including a not applicable grade. Grades of 2–5 signify varying degrees of acceptable performance, whereas a grade of 1 denotes a poor/substandard performance. The rating labels and associated descriptions were based upon those outlined in NOTECHS (Flin et al. 2003) given the similarities between NTS assessment in fixed-wing and rotary environments.

The decision to utilise a five-point scale, as opposed to the alternative four-point scale (e.g. ANTS, SPLINTS, NOTSS), was based on the general familiarity of the operators with the five-point NOTECHS based rating scale that already existed. Due to this, in the review group of HeliNOTS (O), the researcher asked the participating CRM and ex-CRM trainers to discuss their previous usages of four- and five-point scales, their preferences between the two, and the justification for this preference. In this discussion, a strong inclination was noted for a five-point scale over a four-point scale; it was suggested that a five-point scale allowed more scope for degrees of passable performance while also providing more reflective feedback (Table 4).

Table 4. HeliNOTS systems rating scale.

Rating	Description
5—Very good	Behaviour optimally enhances flight/operational safety; no areas of improvements can be identified
4—Good	Behaviour is of high standard and enhances flight/operational safety
3—Acceptable	Behaviour does not endanger flight safety/operational but could be improved
2—Marginal	Behaviour indicates cause for concern and could, in other conditions, endanger flight/operational safety
1—Poor/sub-standard	Behaviour was unacceptable and directly endangered flight/operational safety; remedial action is required
N/A—Not applicable	Skill/element was not required in this case

Instructions for users

Both HeliNOTS systems were produced in the format of a handbook, which included the system, the HeliNOTS rating form, and opening guidance for users who wish to use the system. This included:

• an introduction to the handbook and contact information

• general information for users surrounding NTS and BMSs

• the rationale for the system and its associated taxonomy

• system user, and pilot (junior/trainee vs. experienced), selection and training

• suggested functions and practical recommendations.

This opening section was based upon the preceding system handbooks of ANTS, SPLINTS, and NOTSS, which all heavily emphasised the importance of such guidance for the calibration of users to ensure system reliability in use.

Discussion

The HeliNOTS (SAR) and HeliNOTS (O) systems outline the core NTS and associated elements and behavioural markers for search and rescue crews and offshore transport crews, respectively. The systems were designed utilising SME input to maximise their usability for operators while training and evaluating flight crews. The HeliNOTS systems are the first, to the authors’ knowledge, BMSs constructed specifically for the roles of OT and SAR, however given the systems are currently prototypical, further refinement should be conducted. The remainder of this section discusses the categories of the two systems.

Communication

HeliNOTS (SAR) and HeliNOTS (O) both contain a standalone category for communication. This stands in contrast to NOTECHS and the healthcare related BMSs of SPLINTS and ANTS. For example, in NOTECHS the authors suggest that communication was an inherent aspect of the other NTS categories, and that, for the purposes of NOTECHS, it was viewed as a method of observing the presence of NTS behavioural markers (O’Connor et al. 2002). While we are in agreement with this, and previously note that the content of communications be used to detect cognitive NTS behaviours, all focus groups reported a strong preference for a standalone communication category—an indication of the fundamental importance of this skill to helicopter crews. A similar finding was reported by Irwin, Tone, and Sedlar (2023) where agricultural workers emphasised the need for a standalone communication category with its own behavioural markers.

Situation awareness

The data from the preceding studies (Hamlet, Irwin, and McGregor 2020) suggested a fit for Endsley’s three-tier model of situation awareness (1995) and in the current discussion groups, helicopter crews readily proposed behavioural markers for these three levels (i.e. perception, comprehension, anticipation). This, too, appears to be the case for other BMSs such as SPLINTS, ANTS, and NOTECHS. It is acknowledged that there is an argument for a situation awareness category built instead around the concept of distributed situation awareness which may account more for the technological components of helicopter flight (Stanton 2016). Similarly, one of several team situation awareness models (She and Li 2017) could also be used to account for inherently interactive and cooperative aspects of situation awareness during flight. However, given that the coding process of the original Hamlet, Irwin, and McGregor (2020) study grouped data according to their shared underpinning meaning there was no justification to amend the systems to another model of situation awareness.

Leadership and teamwork

The NTS framework before refinement (i.e. before stage 1) encompassed four elements of leadership for each flight domain: The SAR specific element of overseeing the decision-making process; the OT specific element of training other crew members; and the shared elements of directing task behaviours, monitoring crew members, and tone-setting (Hamlet, Irwin, and McGregor 2020; Hamlet 2021). In the pre-development phase (stage 1) of the current study it was agreed by the academic team that training as an element could not be considered an everyday element of leadership in an operational, as opposed to specifically line-training, context. In addition, the SAR specific ‘overseeing the decision-making process’ element was removed during this phase due to overlap with the decision-making skill category. Further, given that both teamwork and leadership categories for both roles incorporated elements of monitoring other crew members, considerations were initially made as to which category the element of monitoring other crew members should belong.

Directing task behaviours and tone-setting were considered by the discussion groups (in stage 2) to lie ambiguously between leadership and teamwork domains. In a SAR environment, where a multidisciplinary flight crew must coordinate tasks across various stages of a rescue (e.g. transit, winching, casualty care), oversight of a particular task falls into the domain of the relevant crew-member’s expertise. Leadership within a SAR environment can be seen as distinctly transferrable in that any team member can take the lead when they see fit, referred to by Bligh, Pearce, and Kohles (2006, 305) as ‘shared leadership’: leadership as a team level phenomenon. In this sense, directing task behaviours or simply taking the lead can, for the purposes of the HeliNOTS BMSs, be considered a component of effective teamwork. Similarly, across all discussion groups, pilots and TC personnel felt that tone-setting (amended to setting and maintaining crew atmosphere) was something that each team member should undertake to avoid conflict and facilitate crew openness. It is recognised that leadership can also be a team-driven process, shared and distributed amongst a team; though clearly this is dependent on context (Day, Gronn, and Salas 2004). This precedent within the literature, combined with the data gathered from the discussion groups, led to the development of the combined teamwork and leadership NTS category.

Decision-making and the assessment and management of risk

An awareness and management of risk element was outlined in the skill lists produced by Hamlet, Irwin, and McGregor (2020) for both SAR and OT and was included in the refined versions of stage 1. However, across a substantial proportion of discussion groups from each system, it was highlighted that the assessment of the risk aspect of this element was closely related to the situation awareness element of comprehending informational elements. Indeed, Endsley (1990) describes level two situation awareness (comprehension of the current situation) as involving the understanding of indications of potential threats (e.g. warning indicators, appearance of enemy aircraft). To reduce overlap, this element was removed.

Previously pilots had indicated that decision-review was a critical aspect of decision-making (Hamlet et al. 2019; Hamlet 2021) and this was reflected in the current discussion groups. The process of reviewing decisions is suggested to be a useful method of cognitively disengaging from a task, re-establish situation awareness, and plan for future actions (Geraghty and Paterson-Brown 2020). This process of reviewing decisions is recognised in the medical BMS NOTSS which outlines an element and associated behavioural markers for implementing and reviewing decisions made by surgeons (Yule et al. 2008). In the ANTS system for anaesthetists this decision-making review was termed ‘re-evaluating’ and involved the reassessment of a situation or course of action (Flin et al. 2012).

Cognitive readiness vs. workload management

Both HeliNOTS systems share the same core structure aside from the unique categories of cognitive readiness for HeliNOTS (SAR) and workload management for HeliNOTS (O). The justification for these unique elements derives from the study of Hamlet, Irwin, and McGregor (2020).

Originally, Hamlet, Irwin, and McGregor (2020) outlined six shared NTS across SAR and OT pilots: situation awareness, decision-making, task management, communication, teamwork, and leadership. A primary finding of this study was the identification of a range of data related to dynamic emergency scenarios for SAR pilots. Cross-referenced against previous literature, the data appeared to fit the concept of cognitive readiness as outlined by Morrison and Fletcher (2002). The same data were not present for the OT sample. Cognitive readiness was, nevertheless, initially included in HeliNOTS (O) review groups to stimulate additional discussions around this skill in an OT context. By the second HeliNOTS (O) review group, it became clear that OT pilots could not readily propose behavioural markers of the skill. It was suggested by the CRM trainers in the HeliNOTS (O) review group that this may be because while cognitive readiness behaviours may appear to be important to any aviation role, it may not be a standard flight skill one could expect from an OT pilot as opposed to SAR pilots. Indeed, the basis of the concept of cognitive readiness is that it is most relevant to teams operating within dynamic and resource-limited environments (Crameri, Hettiarachchi, and Hanoun 2019).

However, it was determined that some aspects of cognitive readiness (e.g. adaptability) were relevant to the OT environment, albeit out with an emergency response context. In light of this, these behaviours were better suited towards categorisation as workload management, the elements of which covered the prioritising of duties (e.g. correct sequence of checks) and the maintaining of standards, whilst a coping with task demands element was constructed to encapsulate the ways in which OT pilots noted being flexible and rational (e.g. response to adverse weather), in relation to their operational environment.

Limitations

The HeliNOTS systems are currently pre-validated. Studies to consider the psychometric properties of BMSs are commonplace (Fletcher et al. 2003; Mitchell et al. 2012; Yule et al. 2008), and amendments may be made based on these results; for example, task management was removed as a category of NOTSS by Yule et al. (2008) based on an evaluation of the preliminary framework finding that these behaviours were too closely related to situation awareness. The two HeliNOTS systems do, however, provide the basis for further testing and validation; a future aim for the researchers.

A second limitation is the lack of a consistent data collection method across review groups—which varied from email format in the case of HeliNOTS (SAR), to a focus group for HeliNOTS (O). Additionally, for the pilot portion of system SAR, as mentioned previously, two participants who took part in a previous discussion group were also recruited to individually review the entire system handbook. While a consistent method would have improved system validity, it simply could not be achieved in the pandemic circumstances, and therefore the review groups were undertaken with available SMEs to obtain this essential industrial feedback.

Conclusion

The HeliNOTS systems were developed to outline the core NTS behavioural markers relative to the roles of OT helicopter pilots, and SAR helicopter pilots and TC. A range of SMEs took part in discussion groups and contributed reviews towards the production of behavioural markers for the systems. The two resulting systems outline NTS categories, elements, and associated behavioural markers with maximum mutual exclusivity in a simple and pragmatic format to assess observable and inferable NTS performance. These HeliNOTS systems (including the rating forms and instruction booklets) have been designed to be used for both training and assessment purposes with the aim of enhancing helicopter flight safety.

Acknowledgements

The authors would like to thank all of the pilots, technical crew personnel, CRM trainers, and aviation consultants for taking the time to participate in this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was collaboratively funded by the ESRC and Bristow Helicopters [grant number: ES/P000681/1].