First-person video recordings with eye tracking glasses and cognitive task analysis as a framework for referee decision training

ABSTRACT

Rationale

In comparison to players little is known about how sports officials integrate perception and cognition to manage in-game decisions.

Design

Using a naturalistic approach this paper uses first-person eye-tracked video footage to document the attentional demands and situation awareness (SA) of expert touch (rugby/football) referees in their real-world environment to inform decision training for amateur officials. Drawing directly from match performances, an applied cognitive task analysis (ACTA) technique investigated how three international referees manage complex attentional demands, to see what lessons could be learned for less-experienced referees.

Findings

Referees emphasised the importance of role clarity and game understanding as the foundation for effective match officiating. They used advanced cues such as player body language and movement patterns (SA1) to interpret game status (SA2) and predict likely actions and movement patterns (SA3).

Ordering abstraction, preventive communication and early positioning were used to lessen cognitive load and encourage game flow.

Practical Implications

The merits of using first-person, eye-tracked, audio-visual footage with ACTA for training less experienced sports officials through expert verbal elicitation or self-reflection are discussed.

Research Contribution

The paper proposes a decision tree for touch refereeing which emphasises a hierarchical ordering of cognitive decision points that provides the basis for training amateur referees.

KEYWORDS:

• naturalistic decision making
• visual gaze
• cognitive task analysis
• touch (rugby/football) referees

Whilst considerable attention has been given to perceptual-cognitive skills in sports performers, (see Raab et al., 2019), those whose perceptions everyone relies upon to allow the game to proceed fairly, the match officials, have received much less attention (MacMahon et al., 2015). Moreover, historically the majority of research investigating referee decisions has focused on negative attributes of the match officials (Mack et al., 2018), such as unwanted social biases in their decision-making (see Plessner & Haar, 2006). This is perhaps not surprising given that some of the earliest research on sports officials was also driven by negative expectations, such as player abuse leading to stress amongst match officials (Mack et al., 2018). A stream of early studies investigated perceptions of stress in referees (e.g. Anshel & Weinberg, 1995; Rainey, 1995). Importantly, not only did this research find generally low to moderate levels of stress (e.g. Rainey & Hardy, 1997), but more recently it seems that officials report to be much more concerned with personal errors like poor decisions, mechanics and positioning, rather than abuse from players (Anshel et al., 2013).

Reflecting on these more performance-related aspects of officiating, since the turn of the century, the rapid growth in literature on sports officials has led to more explorations into effective decision making (e.g. Kittel et al., 2019). Furthermore, a literature analysis on sports officiating research concluded by requesting more investigations into both decision making and psychology (Hancock et al., 2020), as referees are the sole judges on rule infringements in sporting competitions (Federation of International Touch, 2013). Crucially, the skills that underpin these two areas for match officials, perceptual-cognitive skills, which are arguably the most important, are fundamental to making timely and accurate judgements on passages of play (Mascarenhas et al., 2005a; Morris & O’Connor, 2017).

Such decision-making requires clear and accurate perceptual skills, often with new or changing information that needs to be integrated with existing knowledge (Farrow et al., 2007) drawn from long-term memory stores. This ability to recognise and process task-relevant information, at the right time, in order to select an appropriate response, requires perceptual-cognitive expertise (Mann et al., 2007) and is fundamental to sports officiating performance.

However, the responsibility of referees is even more nuanced than a machine-like application of the rules, as the referees’ accuracy is not only embedded in their decisions but also in their actions (Mascarenhas et al., 2002; Russell et al., 2019). Decision making is a continuous control task, not the resolution of separate conflicts (Rasmussen, 1993). Referees are responsible for ensuring safety, fairness, decision accuracy and the entertainment of a sporting context (Russell et al., 2019) by controlling and managing the flow of the game and ensuring the match result is just (Mascarenhas et al., 2002; World Rugby, 2019). With such a duty comes the responsibility of preserving the integrity of the game by making contextual judgements upon fast-paced, often ill-structured and dynamic plays, where decision accuracy is paramount (Mascarenhas et al., 2005b). In fact, upholding the spirit of the rules and the integrity of the game is often articulated in the rules of the game (e.g. FIBA, 2018) and requires the referee to apply common sense and not to interrupt the flow of the game unnecessarily in order to penalise incidental infractions, by applying preventive refereeing (see Vogt & Cotten, 2010). Thus, referees are required to find a balance between game control and game flow and should have a feeling for what the participants are trying to do and calling what is right for the game (FIBA, 2018; World Rugby, 2019). Yet, despite the clear evidence that perceptual-cognitive expertise is a core skill for the match official, the research in this area still remains limited (e.g. Helsen & Bultynck, 2004).

In addition, in the officiating environment where decisions are co-constructed between players and referees (Rix-Lièvre & Genebrier, 2011) and where officiating roles are interchangeable and decision weightings have to be compatible (Boyer et al., 2020) it is incumbent upon researchers to investigate the decisional experience in its natural context. Adopting an ecological approach, where individual constraints (e.g. declarative and procedural knowledge) task constraints, (e.g. perceptual cues and communication demands) and environmental constraints (e.g. the game context and the nuances of officiating team members) are investigated on a macro-level, provides the backdrop for systematically studying all these factors holistically (see Wattie et al., 2015), rather than systematically manipulating these constraints to study their differential effects.

Further, as decisions and actions are inextricably linked and both provide procedural tools for managing game tenor and player behaviour, a naturalistic approach is essential (Rix-Lièvrè et al., 2015). This approach is typified by exploring experts’ experience-based processes that they employ in their real-world settings (Klein, 1997). By adopting such an approach when visual displays are saturated with information, and decisions are high time-pressured, we have discovered that experts use a range of heuristics to simplify the task, such as chunking (Miller, 1956), satisficing (Simon, 1956) and hierarchical sequencing to manage complexity (Rasmussen, 1985). Unfortunately, examination of such approaches in sports officials has not been investigated (Weston et al., 2012).

In other fields, Cognitive Task Analysis (CTA) has been used as a way of “making thinking visible” so that it can be used for training purposes in performance settings that are difficult to simulate (Gordon & Gill, 1993; MacMahon, 2014). In surgical operating rooms, with complex and unfolding situations that require ongoing monitoring and management as experienced by sports officials, CTA has been shown to be significantly more effective than traditional training (Edwards et al., 2021). CTA identifies key cognitive decision points and provides learners with the procedural steps that not only develop procedural knowledge but also aid more effective technical skills (Edwards et al., 2021). In military, government, academic and medical settings CTA has also been shown to enhance declarative and procedural knowledge as well as the self-efficacy of trainees (Tofel-Grehl & Feldon, 2013). Furthermore, the latest meta-analysis of CTA-based training in surgical environments proposes that such training should occur prior to technical skills (Edwards et al., 2021). So, during a global pandemic where threats to “hands-on” experience have been high for the amateur referee and with the population more ready to embrace technological innovations to assist training (Kittel et al., 2021, 2022) the CTA approach provides a viable alternative to traditional methods that have primarily focused on the accumulation of on-field practice hours.

A recent and compelling trend has seen researchers using eye-tracking technology to explore the gaze behaviour of referees in decision making scenarios in order to develop our understanding of perceptual-cognitive expertise. To date, these studies have largely been conducted in a laboratory setting rather than in-situ environments and as such have been limited in their ecological validity.

Of note, three papers recently investigated eye-tracking with sports officials using similar lab-based methodologies, asking participants to watch videos whilst wearing eye-tracking glasses (see Hancock & Ste-Marie, 2013; Moore et al., 2019; Spitz et al., 2016). Studying soccer referees, Spitz et al. (2016) found higher decision accuracy by higher performing referees as well as more fixations in the contact zone (an area deemed to hold more task-relevant information) than their less able counterparts. Spitz and colleagues concluded that elite referees develop more specific and elaborate knowledge structures. Interestingly, Hancock and Ste-Marie (2013) found no differences in gaze behaviour between higher and lowe-level ice hockey referees, although they did find that higher-level referees made significantly more accurate decisions. Similarly, Moore et al. (2019) who examined the differences in gaze behaviour between elite and trainee rugby-union referees also found no significant differences between elite and trainee referees in fixation points when officiating the scrum, but elite referees again made more accurate decisions. These conclusions suggest that the superior decision making in higher-level referees may be more contingent upon the way information is processed, rather than the fixation point (Hancock & Ste-Marie, 2013), perhaps through peripheral processing (Hüttermann et al., 2013; Schnyder et al., 2017), or indeed some other form of advanced cognitions amongst experts. So, in short, if perceptual-cognitive processes are to be fully examined, investigating where referees are looking (perceptual information/task constraints) must be explored alongside what they are seeing (cognitive information / individual constraints) to develop understanding of officiating expertise. Unsurprisingly, focusing on not just the gaze behaviour but also the interpretation of that information has been highlighted as a key characteristic for development programmes wishing to enhance referee decision making (van Biemen et al., 2022) and has been described as the “missing link” by a recent review of gaze behaviours amongst referees (Ziv et al., 2020).

This missing link is well captured by Endsley’s (1997) situation awareness (SA) framework, whereby expertise is examined on three levels; level one, perception (what have I seen?), level two, comprehension (what is going on?) and level three, prediction (what might happen?). SA theories have been used to study perceptual-cognitive expertise in similar high-time pressured domains where the consequences of poor decision-making could be very costly, e.g. military, aviation and surgical teams. Yet, despite clear guidance on how situation awareness models could be used to investigate sports officials’ performance (Neville & Salmon, 2016), to date such approaches in this population remain unexplored.

Using first-person eye-tracked video footage, from real-life scenarios, alongside verbal reports will provide insight into how match officials manage the decisional challenges of multiple cues that may be spatially distinct. So, like a rugby player with the ball trying to see his defender as well as his own teammates, a referee may have to look at the contact between players whilst simultaneously judging players making it back to an imaginary offside line (cf., Hüttermann et al., 2013). Crucially, the peripheral vision may be as important as the foveal vision, so gaze patterns when coupled with verbalised cognitive processes would allow the trainee referee to determine the crucial information that an expert is attending to.

Such knowledge elicitation (Militello & Hutton, 1998) has been successfully investigated through applied cognitive task analysis (ACTA), incorporating a simulation interview to highlight the key events, the actions and situation assessment and the critical cues associated with each decision event, as well as gaining insight into potential decision errors that could be made (Militello & Hutton, 1998). In this context, rather than using a hypothetical scenario for the simulation interview, ecological validity was further enhanced by using point-of-view video captured with eye-tracking technology, presenting the referee’s field of vision and fixation point. This provided not just a first-person perspective representative of the task being examined, but the exact perspective of the individual performing the task of interest from which to elicit knowledge of their in-situ perceptual-cognitive expertise as put forth by Weston et al. (2012). Using a constraints-based approach with a backdrop of environmental constraints provided by the real-world context, we were particularly interested in the interaction between task and individual constraints (Livingston et al., 2020), and how experts use perception (task) to generate knowledge (individual), which informs their communication (task) of the event. Following guidance from Seamster et al. (1997) we were keen to ensure that we only selected “seasoned” experts so the criteria for selection was high to help ensure more similarity of knowledge and elicit the shared mental models that drove their decision processes (Lines et al., 2021).

Therefore, in direct response to Kittel and colleague’s (2019) call for better modalities of video in their systematic review of decision making in sports officials, this case study used in-game eye-tracking on point-of-view video footage as rich perceptual data to explore the cognitive processes of three expert touch (rugby/football) referees. Touch is a minimal contact version of rugby and is a very dynamic six-aside sport where teams try to progress the ball up the field in less than six touches, each time requiring the defence to retreat 7 m to create space for the attacking team to advance. From an officiating perspective it presents a preferable setting where all three referees rotate from sideline to central referee, providing the opportunity to also investigate team coordination strategies amongst team members as they interchange between roles. Given the hiatus in match-practice opportunities created by the global pandemic we also explored mechanisms through which this approach to CTA could form the basis for training amateur sports officials (see Zachary et al., 2000).

Method

Study design

As this is the first known study of this kind, an exploratory case study design (Fishman, 2017) was employed, using Eye-Tracking Glasses (ETG) and an ACTA simulation interview to collect, collate, analyse and synthesise the data. Consequently, only a small number of subject matter expert participants were used in accordance with ACTA (Militello & Hutton, 1998) recommendations.

Participants

Purposive sampling was used to source the most experienced and skilled referees in a European national touch federation. Three participants were used for this study, each holding the highest level of qualification available in Europe (European Federation of Touch, level 5) and all of whom have either won or been nominated for the Lucas Van Hoff Award, honouring their dedication, guidance, contribution and leadership to European touch refereeing. All three participants were male aged between 43 and 50, (M = 46.67, SD = 2.87) with between 9 and 18 years of refereeing experience (M =  12.33, SD = 4.03) and each having extensive international experience having officiated between 53 and 80 international matches (M = 63.67, SD = 11.73). Additionally, each participant was an accredited referee coach and all had experience in both playing and coaching at the international level. This study received ethical approval from the University of Edinburgh Ethics Committee and permission to use ETG during competitive regional touch games was granted from the touch league Commissioner and Health & Safety Officer. Following approval, two additional participants were used for pilot testing, to inform safety and recording protocols and to operationally test the efficacy of the equipment and its outputs.

Materials

Eye-tracking technology

The task required participants to wear the SensoMotoric Instrument (SMI Red 120) whilst officiating regularly scheduled regional touch referee appointments, officiating the matches and making decisions as they would normally. This mobile eye-tracking device collects point-of-view audio-visual recordings, together with fixation data, at a temporal resolution of 120 Hz and a spatial resolution of 0.03°. This lightweight (75 g) binocular system uses dark pupil tracking to calculate point of gaze and record the visual scene. Eye-tracked audio-visual gaze data were recorded and reviewed post recordings using a laptop (Lenovo Yoga 520) installed with IViewETG software. The gaze data were used to enhance the video data to provide information on spatial and temporal areas of interest for each of the match referees, rather than for independent eye-tracking analysis (see Figure 1). It was expected that this would enhance the recall experience for each of the referees.

Figure 1. First-person video display with visual fixation indicated by the circle.

Recording the video footage

Following a safety briefing, the ETG were fitted to the referee and calibrated to ensure accurate gaze data. The recording device was fixed to the participant either in their pocket or an arm band, whichever was preferable for the participant. Once the participant was fully prepared, each participant officiated their assigned matches as they usually would, wearing the technology for approximately 60 min of game time.

Procedure

Preparation of footage for the ACTA simulation interview

After the collection of the video and eye-tracking data the footage was replayed and analysed. Each game was segmented into passages where the participant was either the (i) control referee: when the referee was the primary on-field referee, (ii) the sideline referee: when the referee was performing assistant referee responsibilities from the sideline or (iii) in interchange: when the referee was transitioning between the control and sideline positions.¹ Two of the researchers then reviewed the footage to determine passages of play where there were consecutive touches made in one phase as sections that provided the most rich data.

ACTA interview

To further enhance the ecological validity of the ACTA approach, the video footage captured by the ETG was used as a substitute for a “hypothetical scenario” as typically used in the simulation interview. This approach to the ACTA simulation interview was novel but based on the fundamental tenets that have been found to make video a useful tool, providing detailed presentations of events in real time (Cannon-Bowers & Bell, 1997; Larkin et al., 2018) and with the addition of visual fixation points to further illuminate patterns of cognitive attention.

Each interview was arranged at the participant’s convenience within 72 h following match day. The first-person video clips with fixation points were presented on a laptop and the interviews exploring their cognitive processes were audio-recorded for analysis. Each participant was shown between 4 and 7 clips, which included at least one of each of the segments listed above and each clip lasted between 40 and 120 s (plus a 10 s pre-roll to allow the participant to become orientated to the event). They were initially asked to “recall being on the field” and informed that following the viewing they would be asked a series of questions (see below) about how they approached the situation presented in the clip. They were allowed to review the clip as many times as necessary in order to recall the events that took place. Preliminary analysis of the referees’ responses indicated that their recall of each event was detailed. Staying true to the simulation interview format each participant was questioned at length under the following five areas, with the addition of one extra element, (scanning patterns) to make best use of the eye-tracked footage:

• Events: what is taking place here?

• Actions: what are you doing and what are the players doing?

• Situation assessment: what is your understanding of the situation presented?

• Critical cues: which cues are you attending to and which are relevant to the decision-making process? Which are disrupting the decision-making process?

• Scanning patterns: how long are cues being attended to and in what order are you addressing them?

• Errors: what common errors might a novice official make in this situation?

Where necessary each video clip was sub-divided into the key decision events held within the clip, in accordance with the Federation of International Touch Rules (2013). From this data a cognitive demands table was developed for each participant that highlighted the most difficult cognitive elements of refereeing, why they are difficult, the potential errors a novice might make, and the cues and strategies utilised by experts to overcome these challenges.

Data analysis

The interview data were transcribed verbatim and thematically analysed by the first author to identify the cognitive demands of touch referee decision making (see Militello & Hutton, 1998) and the themes that arose under the sub-topics “why is this difficult?” “potential errors”, and “cues and strategies”. Where there were sections of less-clear audio, a second researcher also watched and transcribed the section to help data reliability. The latter category also used the referees’ accounts of their temporal fixation strategies that were used to overcome these difficult elements.

Following Braun and Clarke’s (2006) six-step approach to inductive thematic analysis the second author then also listened to the three interviews, read and re-read the transcripts to draw out and define recurrent themes. Finally, to add further rigour to the analysis, the verbatim transcripts, thematic analysis and the decision flow chart were returned to each referee for member checking (see Lincoln & Guba, 1985). Specifically they were asked if this presented an accurate representation of what they said, if they would like to change or add anything to the thematic analysis (see Birt et al., 2016) and if the flow chart decision tree was representative of what they do. This resulted in a change of direct wording on one referee’s quote (which the referee highlighted was inaccurate) from “tackle” to “touch”. Also, in the cognitive demands table, two of the referees both stated that game management was less about “fairness” as this can be misconstrued and more about allowing expressions of talent. This was changed accordingly. All referees unanimously agreed that the decision tree was an authentic representation of decision processes in touch.

Results

The data collected via the ACTA interviews were very rich and produced six higher order themes exposing a range of strategies for managing the challenges to officiating games of touch rugby. The foundation for officiating excellence in these three referees was driven by rich declarative knowledge, providing (i) game understanding which encompassed all three levels of (ii) situation awareness, coupled with (iii) role clarity to allow other tasks to be managed by others in the officiating team. Such understanding allowed for the development of rich, procedural knowledge, permitting (iii) advanced preventive communication, (iv) early positioning, (v) team co-ordination and backup behaviours and (vi) reactive priorities (see Table 1). These shared knowledge structures which were consistent across all the referees revealed a consistent priority system, focusing firstly on the management of the offside line, then the touch and roll-ball, with an apparent attenuation of vigilance to tertiary infringements such as dropped balls or forward passes.

Table 1. Cognitive demands table summary.

Cognitive demand (difficult cognitive element)	Why difficult? (for someone less experienced)	Potential errors (an inexperienced referee may make)	Cues & strategies (you can use/rely on)
Declarative knowledge
Game Understanding & Management: The objective of the referee is to facilitate a game where teams are able to demonstrate the best of their abilities and everyone has the opportunity to express their talent.	Requires advanced understanding of the game and a refereeing philosophy. Ability to help create a game. Allows advantage to be played when the opportunity arises (does not blow every incident and ensures the game flows).	Officiate the game strictly according to the rules, losing sight of their intention. Not applying positive referee principles such as the advantage rules.	Advanced SA, advanced cue use, positioning, team work and knowledge of the rules.
Situation Awareness: Predicting attacking patterns, player movements and active players. Recognition of defenders defensive strategies and those likely to impact play.	Requires advanced understanding of the game, patterns of play and predicting where the ball will go. Must maintain good positioning.	Reactive rather than proactive. Limited, late or poor communication.	Advanced SA, team work and knowledge of the game leads to advanced communication and positioning.
Referee Role Clarity & Responsibilities: The referees must work as a team to maintain control of the game.	Attending to important areas, identifying and interpreting cues, integrate cues with goals.	Reading the play. Limited integration of situational assessment and rule intentions. Only one approach, not adaptable.	Integrate game situation and rules to apply quality interpretations. Know which areas to look for best information, understanding of game context.
Procedural knowledge
Early & Preventive Communication: The referee must frequently and effectively communicate with the players who are likely to impact play to help prevent infringements.	Competing communication needs, focus required on play, requires anticipation.	Limited/ineffective and reactive communication.	Pre-loading players. Conspicuous language (e.g. numbers). Knowing who needs communication and when.
Early Positioning: Allows the referee the best sightlines and the best ability to both set the offside line and communicate with players.	Constant movement/anticipation, ideal position changes with game context.	Reactive positioning. Chasing the game by not anticipating the game flow.	Utilise body language cues. Understand game context and team work.
Ordering of priorities
First Tier. Offside Management: The top priority for referees.	Demanding task that requires physical and cognitive skills, requires anticipation, occurs in conjunction with other focus areas, occurs rapidly.	Poor positioning, poor communication and focusing only on involved players.	Proactive management. Strong communication recognise in advance what is going to happen and organise players accordingly. Use reference points to minimise burden and understand flow of the game.
Second Tier. The Touch and The Roll Ball: The second priority for referees	Occurs at the same time as offside line, many potential infractions, requires subjective judgement	Focus only on offside area, literal rule interpretations, limited communication	Effective pre-loading. Integrate situational awareness into judgements, use advanced cues to anticipate the action and adapt to game context.
Third Tier. Reactive Priorities: Other infringements, e.g. dropped ball. Third tier priority.	Difficult to anticipate, last priority, positioning may not be optimal.	Attention occupied by offside and roll ball, ineffective team work.	Effective teamwork to pick up ambiguous calls.
Team coordination behaviours
Communication to Aid Interchanges: Anticipate where the play is going. Referees who are less familiar with each other pre-empt interchanges by communicating their future actions with each other.	Referees must work in unison. Events and game context require referees to adapt, requires trust. Risk of losing control and ability to maintain control whilst out of position.	Resting on the side line, losing focus, will not adapt to changing contexts. Fail to prepare for the interchange or focus on getting on the field but not actually maintaining control. Limited communication and exiting without handing over control.	Recognise and anticipate interchange opportunity when teams are subbing. Proactive interchange, strong communication and positioning validate referee decisions, proactive positioning.
Backup Behaviours: Referees support one another by taking responsibility for specific players in accordance with their role. Recognising and supporting the control referee when they are unsighted to the play.	Must take responsibility for your own area of the field but also recognise when the control referee has done a long set or may need support.	Sideline referee shows a signal that the players see but the control referee does not. Can create controversy. Late interchanges and game control is lost or confused.	Awareness of the control referee's angle of view and physical state.

Declarative knowledge

Game understanding and management

This was identified frequently by all participants as the primary role of the referee. This was well articulated by referee two who explained,

There’s probably a million penalties take place during a game. But if you blew for every single one of them it wouldn’t be much fun to play in or take part in so you’ve got to think game flow or game management. For this one it was a big enough over the mark that would have given them enough of an advantage. (R2)

Situational awareness

The referees showed a clear appreciation of the critical cues that inform their decisions and actions. This included looking at both attacking and defending team’s positioning, as well as the individual player body language, as described by one referee,

I’m looking at the body language of the defender and the attacker … [in this case] it means that it’s slow ball and you can take time to set the line. (R3)

This level one situation awareness, that clearly informs level two SA as highlighted above, also allowed for accurate prediction (level three SA) to help ensure that the referees effectively positioned themselves for the next phase. This enabled them to anticipate which defender is likely to be drawn forwards to make the next touch,

I’m looking at where the players are set up to see where the gaps are. So I’ve half a mind on the defenders to see where there’s a gap in the defence and half a mind on where the player numbers are, so if you know that there is an overlap for players they’re probably going to go that route … . You can see the body language coming in to the right as we’re looking at the screen and we knew that there was gaps on that right [side]. We knew that there was an opportunity for a score. So the body language was saying “go right.” (R3)

Referee role clarity and responsibilities

The referees must work as a team to maintain control throughout the course of a game. Referees rotate between the control (on-field) and sideline positions and take on different responsibilities in these roles,

So what I’m aiming for is to find that spot. That’s 6 to 7 metres out from the 7 [metre line] where I position myself and then I’m focusing on the mids more than anything else. What I’m expecting [name of co-official] to do as she’s leaving the field is to control the link and the wing on exit. (R3)

Being able to rely on the knowledge of the officiating team around them, allows the central referee to focus primarily on the mids (middle players), whilst the support referees will manage their nearside link and then the nearside wing, thus reducing the breadth of attention and cognitive load on the control official.

Procedural knowledge

Preventive communication

Referees all showed a very clear orientation towards preventive communication to facilitate player control and importantly to allow game flow. Using their anticipation skills, referees consistently exhibited pre-emptive communications to help players remain within the rules of the game,

Ok, so that’s strong [my comms] making sure that 14 gets back to me, preloading before the touch has even gone in and you can see I am watching where the touch is going in and you have the two players up, so 14 is back with me, making sure he already has the onside line sorted so I can focus on the two who are up … (R1)

Early communication

Effective and early communication allowed these experts to reduce the cognitive load by prioritising the demands. First, referees used goal orientated, top-down processing (management of offside) in order to then allocate attention to more stimulus-driven (the touch and roll-ball) criteria. In this case, amongst many others, the referee is also heard positively reinforcing the defender for her compliance to the request,

My attention is on her [defending player], not always on the ball. The early calls are trying to get them back on side to pre-empt the offside so I can focus on the touch and the roll ball [on video “good … good”]. (R1)

Also, the timing of communication was noted by the referees,

So I was proactively calling him before the touch but you could see I was also then calling him back after the touch and that latter part is not ideal because it means that instead of focusing on other things I’m focusing on him but in this instance I felt that I had to … a lot of novice referees won’t even proactively call them back before the touch. (R3)

Early positioning

All referees acknowledged how critical positioning is to help the players understand where the offside line is, but also be able to create the best angle and proximity to the play by predicting the direction of the play,

Your comms is very much about getting the players onside first but the body language is about setting up that onside position and with your head up and so you can actually see what the players [are doing] and because you’re watching the roll ball, you are trying to use your peripheral vision to look at whether the players are trying to wrap or movement or try and look where the numbers game is. So you’re trying to scan … the idea is you want the ball to be scored at your feet. (R1)

By reading the play and early preventive communication and positioning referees were able to then focus their attention on the touch and roll ball as highlighted in the decision tree (see Figure 2).

Figure 2. Flow chart decision tree of attentional processes used to manage the competing demands when officiating the “touch and roll ball”.

Ordering of priorities

A consistent pattern from the control referee was to prioritise (i) offside management, (ii) the touch and roll ball and (iii) other infringements. For example, referee three said,

My attention is on her [defensive player], not always on the ball. The early calls are trying to get them back on side to pre-empt the offside so I can focus on the tackle and the roll ball (on video “good … good”) [to player who has retreated onside]. (R3)

Lower order reactive priorities

All referees alluded to some decisions which by their nature fell into a lower priority, where the referees are less able to be proactive and have to rely on being reactive once the incident occurs, for example, a forward pass, a dropped ball, or link and wing offside decisions,

Forward [pass] at the ruck is quite a difficult one sometimes because I’m going to be 7, 8, 9 metres behind the ball it is not always possible to see when you’re almost right in front of it whether the pass is forward or not … . Looking at the hand position for the person, the half picking the ball up and also looking over to my sideline referees if you remember to do that but it’s amazing how often you forget to do it because you’ve got so many other things to process. (R2)

Referees described that while coping with multiple decision challenges at the same time, they sometimes used advanced cues to help predict some of these types of decisions,

So the position of that player [the half] helps you determine the legality of the pass … his body is open which means it can go backwards. If it was closed then it couldn’t go backwards and also if the player was standing forward of it, it also means it cannot go backwards. (R1)

Team coordination behaviours – communication to aid interchanges

Whilst all referees acknowledged a clear pattern for interchanging with their co-officials, they all used early comms from the sideline and from on-field, as is the case here, to ensure game control was maintained at all times,

So when I say this is second [touch] I’m shouting that to the players when I’m really more shouting that at [name of co-official] for coming on yeah, just so that he’s clear. So basically, he will … I didn’t actually hear him shout it there but he probably did shout “yeah [name of referee] I’ll take it” or words to that effect and then my response to that is this is second or second coming and then as I go off you’ll notice I’m shouting at #31 which is the link so my job as I’m running behind #31 is to make sure he’s onside for the next play. (R2)

Similarly referee one explained,

Before I actually blow the whistle I’ll communicate it verbally, mainly for my sideline referee so that they’re first to get in position  … So, yeah, yeah I’ve seen it. It’s going to be an over the mark penalty. That then allows me to bail out at the side of the pitch [interchange with co-official] rather than having to blow the penalty and do the 10 m sprint to set the next 10. (R1)

Backup behaviours

Also, to take the load off the on-field referee when in the sideline position referees support the control referee with backup behaviours by communicating when a player has successfully retreated to an onside position, so that they are able to focus on their primary duties,

If I’m saying “yeah good” you’re doing something well [to the players] they are more likely to listen and do it for me again but it also acknowledges to the on-field ref that I’m happy with the players that I’m controlling, so that allows him to concentrate more on players in the middle … so he can hear it. (R2)

Referee two also indicated how his knowledge of other referees’ perspectives allows him to provide backup behaviours for his control and far sideline referee when he has the best view on the play,

I saw that as forward. I know there’s momentum but that went forward about 4, 5 metres. Now I know we’ve been coached as experienced referees that it would be the one closest to the ball and the direction the ball was going in who makes the call on a forward pass. But it was a junior referee, and I don’t even know if he’s a level one [qualified] so I was trying to help out [referee’s name] by giving him that view. (R2)

Referee errors

Referees identified poor positioning and slow interchanges as common errors, but consistently referred to likely mistakes in “pre-loading”, players which is a type of preventive communication, to ensure players are retreating onside before a touch is made. This allows for a free-flowing game and also helps to develop relationships with players,

… an experienced ref is looking for where that touch is coming in next, whilst an inexperienced is waiting for that touch and responding reactivley. (R1)

Interestingly, without prompting each referee also picked up on errors they had made during the game clips. These resulted in procedural errors by not anticipating the play (SA3), but most of which were caused by attentional errors (SA1),

 …  so the other aspect that we are picking up there was that my focus was on the wrong player. He is already up, he is making the touch. He will already know he is offside when the touch is in. It should have been the other mid and link that I was focused on. (R1)

Discussion

The objective of this study was to identify the cognitive demands experienced by elite referees in officiating games of touch , exploring the chronological critical cues through visual fixation patterns associated with managing those cues and identifying the errors that a novice might make. The results clearly show that top touch referees use both declarative and procedural knowledge to help them manage a game. Their extensive declarative knowledge was used to predict likely patterns of play. This level three situation awareness (Endsley, 1997) then allowed referees to use preventive communication and early positioning to forewarn players of potential infringements before they occur, enabling them to allocate their attention elsewhere. So in this study, officiating expertise was driven by the interpretation of advanced cues, through reading player movement patterns and body language to allow early prediction of likely actions. All referees were driven by the desire to create an attractive game with flowing passages of play, with a strong emphasis on the entertainment pillar of referee decision making (Russell et al., 2019).

Reading the game allows referees to use early and preventive communication to help maintain this balance between flow and control of the game (Russell et al., 2019) and also arguably, would serve to preserve the integrity of the game by helping to ensure that players are respectful and responsive to the referee. Crucially, by pre-empting one infringement, in this case the offside rule, with clear and timely communication, referees were able to allocate their attention to more stimulus-driven criteria. So, shifting attention from top-down preventive officiating, to bottom-up stimulus-driven processing (see attentional control theory, Eysenck et al., 2007) appears to be an effective tool for managing simultaneous demands. To create more processing resources, this study also showed that referees attenuate their attention to some infringements, where they become more reactive rather than proactive.

To further assist the referees, they have well-defined roles and an understanding with their co-officials, that is at times supported by pre-emptive communication to facilitate co-ordination and game control. Delegation of responsibility to the sideline officials is also a valuable asset to reduce the cognitive load upon the central referee. Unique to touch is the interchange of responsibilities where the sideline referee rotates on to become the central referee. Whilst some studies advise role-specific training (see Catteuw et al., 2009) this seems to be less critical in this context as officials perform a natural cross-training of both roles in match situations. In fact, team training where individuals are exposed to each others’ roles has been shown to create more team unity, a stronger push of information to the leader and enhanced anticipatory information (Marks et al., 2002; Volpe et al., 1996). So perhaps such rotational responsibilities could be used to enhance teamwork in other sports. In accordance with Boyer et al. (2013) these referees appeared to be able to support each other by reading each other’s body language and then providing validatory support through communication. These timely interactions also serve to calibrate decision judgements within the officiating team (Unkelbach & Memmert, 2008) to help develop team coherence.

Embodied in this cognitive load management strategy this investigation revealed an ordering abstraction (see Rasmussen, 1985) to referee decisions. Here, referees prioritised offside players, the touch and roll-ball and as a lower priority any other infringements that may occur. A similar hierarchical decision making approach was used successfully to train rugby-union referees (see Mascarenhas et al., 2005b) and has also been advocated to help basketball officials to adjudicate a jump shot in basketball (Richardson & Mitive, 2007). This has been termed sequencing the play and is seen as a crucial part of understanding the mechanics of basketball officiating (Wunderlich, 2020).

ETG & ACTA as a training tool

Considering the sequencing of tasks as highlighted by the decision tree (see Figure 2) this provides the operational and procedural knowledge for less experienced officials to understand the priorities for effective performance. Similar decision tree frameworks are used in other areas of sports officiating to simplify the decision process (e.g. World Rugby, 2021). The tree identifies important cognitive decision points, highlighting a structure through which learners can process knowledge of a challenging situation and has been shown to accelerate the development of competence (Edwards et al., 2021).

This study also utilised a novel approach to the ACTA simulation interview (Militello & Hutton, 1998), substituting a hypothetical scenario – “what would you do” for first-person video footage taken from the referees’ real-world performance – “why did you do”. This point of view perspective of each individual participant promotes ecological validity (Cannon-Bowers et al., 1996).

Given the dynamic nature of decision making, the high potential for sports officials to make decision errors (MacMahon & Mildenhall, 2012) and the challenges of getting sufficient on-field experience, this framework could be used for developing less expert officials’ in game performance. Historically amateur officials gain their experience through match officiating and during grading courses. Providing this sort of high-quality eye-tracked footage with expert voice-over recordings taken from the CTA process could be delivered on electronic devices as a way of overcoming environmental constraints. This would allow the amateur official to access the materials at will (Edwards, 2021) and with the opportunity to revisit in a distributed practice style, which has been shown to be crucial for both the retention and transfer of procedural knowledge (Cecilio-Fernandes et al., 2017; Moulton et al., 2006). Such materials have the potential to present more comprehensive and detailed visual materials that have been shown to be a more effective training tool than traditional hands-on training (Edwards et al., 2021). This presents a constructivist approach to learning where new knowledge is enhanced by building on pre-existing knowledge (Dennick, 2016) and provides an opportunity for abstract learning when concrete experience may not always be possible (see Kolb & Kolb, 2005).

ETG & ACTA as a reflective tool

In addition to being able to clearly articulate errors that less experienced referees might make, without prompting, all the referees were very aware and critical of their own mistakes. This is perhaps not surprising, as a feature of top referees is their ability to self-reflect and be critical of their own performance in order to learn from their errors (Slack et al., 2013). Interestingly, for these three referees the errors were largely procedural, including late communication and poor positioning having not anticipated the next play (SA3), but often this was due to poor attentional processes (SA1). The process of making errors tended to appear around goal-directed behaviours and the problems associated with attentional switching. Perhaps this could have been anticipated given that verbal and spatial reasoning have been found to place a high demand on working memory and when under pressure we become less able to flexibly switch between the two different attentional styles (Eysenck et al., 2007), becoming more hypervigilant to stimulus-driven processing. So the process of verbalisation allows us to see beyond the errors and identify the cause of those errors, which in this case showed a trend towards errors occurring as a result of task (perception or communication) constraints rather than individual (knowledge) or environmental constraints.

In fact, mitigating against the typical environmental constraints where on-the-job learning is passive, the use of first-person ETG footage, coupled with using the ACTA framework as a self-reflection tool may provide organisations with a support resource that enhances the amateur referee’s commitment to learning and accelerates their development (see, Edwards et al., 2021). As we enter a post-pandemic phase, and on-field refereeing returns for amateur referees, ETGs and ACTA may help the learners themselves to identify at which level of Endsley’s (1997) model errors occur for them, promoting autonomous learning. Also, it would be interesting to see if giving more novice referees an opportunity to articulate their embedded knowledge structures (through the ACTA events, actions, assessment, and critical cues framework) transfers to better in game communication through practice at verbalising their situation assessment of rulings. Clearly there is a financial implication to using ETGs for this purpose but with advancements in technology and the introduction of head/chest cams, lower cost alternatives are likely to emerge that could present real opportunities for sporting bodies to upskill their workforce. To understand more about the development of expertise the process could also highlight if novice referees also emphasise the entertainment pillar of Russell et al.’s (2019) model or are they more focused on safety, accuracy and fairness? So, while further research is needed to validate this approach, in this study it does appear to assist critical reflection of the participants wearing the ETGs. In an environment where the accumulation of deliberate practice hours is a challenge (MacMahon et al., 2015) it may provide a solution to fast-tracking amateur officials wishing to progress to the next level. It may also present an appropriate means through which to investigate team co-ordination, an area that currently has very limited research (Aragão e Pina et al., 2018).

Strengths, limitations and future directions

The limitations of the exploratory design used must be noted and the results taken with objective caution. The findings of this paper should be considered an introduction to the use of cognitive task analysis with sports officials, rather than conclusive evidence supporting the models proposed. The design used was susceptible to several biases. Two of the referees reported a restriction in their peripheral vision as a consequence of the ETG, which may have resulted in exaggerated head movements and stronger communication to prevent players from shooting offside early. Further enhancements in technology will hopefully improve this in future studies. Also, the sample size was small, and although representative of one-third (3 of 9) of the top qualified referees in that nation, they were all recruited from one country. Further, as with all studies using verbal recall processes that may be implicit and at times tacit, making these decision tasks more explicit by verbalising them is susceptible to confirmation bias (see Nisbett & Wilson, 1977). Also, given that some of the interviews took place nearly 72 h after the match event, recall limitations may have also contributed to this bias. Therefore, the results found here are a representation of what is happening and not necessarily what is best. However, for international sports officials it is customary for referees to justify their decisions, often to the players during the game and also in their post game reflections with the officiating team and coaches, so verbalising decision processes is actually very commonplace.

Future studies seeking to expand our understanding of referee cognition and to build useful tools for applied practice are encouraged. Investigation of expert novice differences would allow us to see the effects of poor prediction skills upon the game. For example, would a positioning error, or poor positioning lead to less accurate decisions? Which decision points in the model are most likely to break down and why?

In conclusion, the present paper demonstrates the value in examining referee cognitions in naturalistic performance environments by using technology in conjunction with expert verbalisations. Our approach has begun to illustrate the fundamental cognitive demands, both their nature and their relationships with one another, and how these mediate referee performance. This naturalistic examination emphasises the importance of game management as the overriding objective of the referee and how this acts as an omnipresent reference point for referee decision making. Once this naturalistic context is established, researchers, practitioners and referees can begin to form a more complete understanding of how referees make effective decisions. This study proposes a model of referee performance which requires further exploration and validation but functions as a suitable starting point for future research and amateur touch referee training. Finally, the study functioned as a suitable trial process for an adapted ACTA process using first-person perspective video taken from the performance environment. As established in other fields, this verbalisation of cognitions may be a pandemic-resistant approach to enhancing training experiences of amateur sports officials.

Data availability statement

The data that support the findings of this study are available on request from the corresponding author, [DRDM]. The data are not publicly available due to [restrictions e.g. their containing information that could compromise the privacy of research participants].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 A game of touch is officiated by three referees with equal responsibilities, who rotate to the central (control) position where they would take primary control of the game.