Social fidelity in cooperative virtual reality maritime training

ABSTRACT

Each year maritime accidents occur at sea causing human casualties. Training facilities serve to reduce the risk of human error by allowing maritime teams to train safety procedures in cooperative real-size immersive simulators. However, they are expensive and only few maritime professionals have access to such simulators. Virtual Reality (VR) can provide a digital all-immersive learning environment at a reduced cost allowing for increased access. However, a key ingredient of what makes all-immersive physical simulators effective is that they allow for multiple participants to engage in cooperative social interaction. Social interaction which allows trainees to develop skills and competencies in navigating situational awareness essential for safety training. Social interaction requires social fidelity. Moving from physical simulators into digital simulators based upon VR technology thus challenges us as HCI researchers to figure out how to design social fidelity into immersive training simulators. We explore social fidelity theoretically and technically by combining core conceptual work from CSCW research to the design experimentation of social fidelity for maritime safety training. We argue that designing for social fidelity in VR simulators requires designers to contextualize the VR experience in location, artifacts, and actors structured through dependencies in work allowing trainees to perform situational awareness, coordination, and communication which are all features of social fidelity. Further, we identify the risk of breaking the social fidelity immersion related to the intent and social state of the participants entering the simulation. Finally, we suggest that future designs of social fidelity should consider not only trainees in the design, but also the social relations created by the instructors’ guidance as part of the social fidelity immersion.

KEYWORDS:

• Virtual reality
• cooperative work environment
• CWE
• CSCW
• simulator
• training
• maritime
• cooperative VR
• awareness
• coordination
• communication

1. Introduction

“We chose to use [safety training simulators] with the purpose of learning how to avoid killing each other, […] We know it only takes 3 additional minutes to avoid killing each other. So, we might as well use the 3 additional minutes since the alternative is a 40-year shorter human life” (Captain & Maritime Instructor).

Training simulators for safety procedures have been around for many years exploring everything between training dangerous situations in the Air Force (Parker et al., 1974) or complex procedures in wastewater plants operating procedures (Doering et al., 1982). Situating our work in the maritime industry, we focus on maritime simulator training e.g., how to navigate large containerships in difficult waters, how to act during emergencies on oil rigs, or how to coordinate and solve engine breakdowns. Maritime training simulators are crucial for cooperative safety training, where a main learning objective is for trainees to demonstrate and perform situational awareness of not only individual activities, but also through monitoring the activities of others, taking measurements, and being proactive in resolving issues before they evolve into high-risk situations. Simulator training within the maritime industry continues to play a key role in the education and skill development of maritime practitioners (Kim, Sharma, et al., 2021; Kim, Sydnes, et al., 2021; Mallam et al., 2019), and recent research points to how technologies such as virtual reality (VR) and augmented reality (AR) can be leveraged for simulator training and potentially transform how maritime training is structured, planned, and executed (Fracaro et al., 2022).

Simulator training has received attention within research on digital technologies exploring the challenges of designing simulator training (Alharthi et al., 2018). Research includes investigations of training simulators in different application areas e.g., telemanipulation of nanotubes (Gao & Lecuyer, 2008), medical training or dental training (Blasco et al., 2013; Sallaberry et al., 2021), welding (Ipsita et al., 2022) and high-risk power line maintenance (Borba et al., 2016). Each of these training simulators is based upon insights about the practices and procedures required for the professional activity, and most of this research focuses on a single-user setting, with few exceptions such as the work on the collaborative practices in healthcare (Bjørn et al., 2021), air traffic control (Malakis et al., 2014; Pang et al., 2022), and disaster response teams (Alhartihi et al., 2018; Toups-Dugas & Kerne, 2007). VR technology offers new potentials for the cooperative simulator training (Gamito et al., 2015; Larsen et al., 2024; Maloney et al., 2020; Pederson, 2003; Rumsey & Dantec, 2020; Yin et al., 2014) – and our overall interest is to re-think the design of cooperative simulator training utilizing VR technology to increase Maritime practitioners’ skills and expertise in enacting situational awareness reducing the risk of human casualties.

When designing and implementing cooperative VR simulators, decisions on which aspects of the real-life environment is needed to be implemented, and how to design the scenarios, the artifacts, and the cooperative activities are pertinent. The accuracy of the simulation and the exactness of the digital environment in comparison to the real-life environment thus depend upon choices and design selection considering what is needed for the training outcome. Our challenge is to design a cooperative immersive training environment which allows participants to make visible their actions and monitor the actions of others while considering the contextual contingencies of the environment, and proactively acting accordingly. Concretely, our research question is: How can we design social fidelity into immersive VR training simulators?

Exploring this question, we first unpack the nature of social fidelity theoretically, then design and implement a cooperative VR training simulation prototype for mooring a containership and test the prototype involving 22 seafarers to identify challenges for designing social fidelity into the immersive simulator. We then discuss our findings and extend the conceptual work on social fidelity to be included in the design of immersive training simulators. We argue that in designing for social fidelity we need to: 1) center the interdependence in work required for the cooperative engagement and articulation work in both the training scenario (activities) and the immersive environment (artifacts and locations), and 2) enfold activities of coordination, communication, awareness into the scenario in an open-ended manner. On this basis, we suggest three design guidelines for social fidelity in an immersive training environment considering virtual bodies, gestures, and interdependencies through coordinative artifacts. Moreover, we find that when designing for the robustness in the “immersive nature” of social fidelity the design relations to interdependence, coordination, communication, and awareness might not be enough. Instead, the social fidelity immersion risks being broken (not as a technical matter), but instead, as a social matter related to the intent and social state of the participants when entering the simulation. Finding ways to reduce the risk of breaking social fidelity is thus a future design challenge for HCI researchers.

The paper is structured as follows: First, we introduce the main insights from research on training simulators while unpacking the concept of fidelity and defining the nature of social fidelity. Then we introduce our empirical case of Maritime training, introduce the design of the VR training simulator prototype and the test setup. We then demonstrate what areas of social fidelity the prototype was able to capture while identifying the situations where the social fidelity of the prototype was broken. In discussing the findings, we reflect upon the results and discuss the challenges of how to design for social fidelity in immersive training simulators. Finally, we conclude and propose new research directions for social fidelity in cooperative VR.

2. Theoretical background: training simulators and social fidelity

2.1. Training simulators

Knowing professionally what to do within concrete complex situations is a difficult task for people who are new to a specific domain, especially in situations of high risk and thus increased pressure. Professional knowledge shapes how professional actors can make sense of a situation and act appropriately (Dreyfus, 1988, 1997). For novices to become experts requires situated experiences where the novice learns to notice pertinent features of a situation, prioritize, and evaluate the situation, and then act accordingly (Schön, 1983). Novices working next to experts within a professional domain (e.g., healthcare or carpentry) can engage in situated learning experiences applying learning-by-doing principles developing required professional competencies and expertise (Lave & Wenger, 1991; Wenger, 1998). Accommodating such learning-by-doing situations and scenarios, however, requires that both experts and novices can engage in meaningful interactions in the field allowing for learning on the job. However, in professional situations of high speed, high complexity, and high risk “learning on the job” might not be an option. Thus, simulator training for developing professional skills and expertise in complex, difficult, dangerous, and hazardous situations across various domains (e.g., operators, seafarers, firefighters, medical professionals etc.) is a relevant alternative.

Immersive training simulators facilitating learning-by-doing activities have been used in the maritime domain for decades, where immersive training facilities simulate the operation of a crane, the navigation of a ship, or the controls of an engine room. The physical appearances of such simulators contain mock-up artifacts and equipment in 1–1 size, where trainees can experiment and learn the professional practice. Mock-up training simulators are used to either train procedural practices (e.g., standard operation procedures – “SOP”) or safety and emergency practices (Fracaro et al., 2022). The advantage of mock-up immersive training simulators includes the capability to practice and rehearse high-risk tasks training safety; the capability to simulate different scenarios producing flexibility in training; the capability to gain an understanding of consequences of action to learn from mistakes training skill acquisition; the opportunity to practice non-technical skills such as communication and leadership developing skill utilization; the possibility to change the speed of scenarios training efficiency; the potentials for lower operating costs compared to on-the-job being cost-effective; and finally the capability for recording training sessions supporting the pedagogical approach in training and assessment (Kim, Sharma, et al., 2021). Further what makes the strengths of mock-up simulator training is the capability to train cooperative situated social engagement between multiple actors in complex open-ended scenarios. Many immersive mock-up simulators facilitate the training experience of engaging with the mock-up artifact with digital enhancements projecting consequences of actions (e.g., turning the steering wheel on the ship bridge will change the horizon of the ship on large projector screens in the simulation), or as embedded functionality in mock-devices (e.g., the controls of engine breakdown allowing for trainees to identify problems and solve these accordantly). Yet, the main quality of the immersive mock-up simulators is the physicality of artifacts and the situated context surrounding these artifacts.

Digital immersive simulators are based upon detailed analysis of specific knowledge areas, combined with a concern for the training practice which can be developed in digitally immersive technologies such as virtual reality or augmented reality. Exploring the potentials of immersive technologies researchers have developed prototypes and explored different scenarios for professional work practices (Krauß et al., 2021; Rumsey & Dantec, 2020). Within the design of digital immersive simulators, there is an important distinction between single-user simulators (Wilhelm et al., 2002) and cooperative simulators (Paiva et al., 2015). While mock-up simulators often provide training simulator experiences for multiple participants, digital virtual reality simulations are mostly focused on the single participant, where e.g., one healthcare professional trains the use a dialysis machine (Rettinger et al., 2021) or one technical operator train the process of pump maintenance (Winther et al., 2020). Digital simulators have been used to explore the possibility of supporting firefighters in detecting objects while fighting fires allowing participants to “see” thematic heat (Abdelrahman et al., 2017), or manufacturing practitioners to experiment with robotic cells as part of a manufacturing process (Darmoul et al., 2015). Overall single-user simulators based upon virtual reality thus tend to focus on technical training of specific machines or operating procedures (Dahlstrom et al., 2009). The focus is on technical skills and the immersive opportunities are used to produce high-fidelity artifacts and interactions simulating complex technical equipment.

Cooperative simulators center interpersonal interaction into contextual situations. Prior research exploring interpersonal interaction in VR includes work on surgical teams (Paiva et al., 2015); facilitating healthcare practitioners in providing feedback on the architectural design of hospitals (Bjørn et al., 2021); supporting professional analytical practices used in marine archeology to examine the 4th-century BC Mazotos shipwreck (Katsouri et al., 2015), and finally exploring how remote medical consultation could be supported by virtual reality mediating communication between nurses and patients (Li et al., 2020).

While prior studies on digitally immersive cooperative simulators provide important insights, only a few (e.g., Larsen et al., 2024) consider the challenges of simulators which allow for participants to develop skills and competencies in situational awareness through social-professional interaction, which is key for emergency and safety training. Training such efforts based upon the “learning-by-doing” principles requires contextually situated practices, by which participants can develop their professional expertise in navigating complex, difficult, dangerous, and hazardous situations. Mock-up simulators provide the capability to train the social professional interaction, now the question is how can we transfer this capability into immersive VR simulators – or phrased differently, what entails “professional social interaction” or rather “social fidelity” as a characteristic of design for immersive VR simulators?

2.2. Social fidelity

Fidelity as a feature of immersive cooperative simulations is fundamentally about accuracy between “the real-life” contextual circumstances and the digitally created immersive environment. High-fidelity refers to the “photogenic” nature of the artificial environment and is often considered superior to low-fidelity (Dahlstrom et al., 2009), however, research has demonstrated that low- and even zero-fidelity can provide efficient training environments (Toups-Dugas et al., 2011).

Fidelity embedded within simulators is considered multi-dimensional and different classification schemes have been proposed (Hontvedt & Øvergård, 2020; Toups-Dugas et al., 2011). Physical fidelity refers to the accuracy between the physical environment and equipment – the accuracy and “photogenic” likeliness between the artifacts and surroundings and their real-life counterparts. Functional fidelity is the accuracy between the functionality and internal characteristics of artifacts and surroundings considering how metaphoric representations of action within artifacts depict the actual functionality or whether represented artifacts provide the same interactive opportunities as real-life counterparts. Psychological fidelity refers to the conceptual learning concepts and how well the “purpose” of the learning is facilitated by the simulation, and the basic assumption has been that high-fidelity physical resemblance better supports psychological fidelity. However, in recent years several studies have challenged this assumption (Dahlstrom et al., 2009; Toups-Dugas et al., 2011). Sometimes psychological fidelity is divided into behavioral fidelity (actions and activities) and perceptual fidelity (psychological experiences) to further distinguish between the different types of human experiences which the simulator environment can facilitate. Interactional fidelity (Hontvedt & Øvergård, 2020; Kim, Sharma, et al., 2021) has been introduced recently within maritime physical simulators and refers to the ability by which a simulator allows participants to interact and engage as part of the cooperative simulation, and refers to the simulation of coordination and collaboration in the immersive environment.

Most recently social fidelity has been suggested as a feature of immersive cooperative simulators. Social fidelity refers to the nature of social interactions as a part of the collaborative activities that are embedded into the simulation scenario (Hontvedt & Øvergård, 2020). The concept of social fidelity is used to explore the social factors which influence the training quality of simulator training and thus includes all the social circumstances of a training session such as the interaction between trainers and trainees as well as between trainees (Wahl, 2020). In this work, Wahl identifies and categorizes cooperative factors facilitating the quality of the training as task factors including scenario complexity, role-playing, and instructor guidance – and social factors including practical drift, storytelling, and social rank (ibid). Social fidelity is thus about relational work between all involved actors (Bjørn & Christensen, 2011; Christensen et al., 2014). The current work on social fidelity is not expanded to design, thus if we are to consider how to design for social fidelity, we need to extend the conceptual understanding to accommodate concrete ways to build and implement features of social fidelity into the immersive VR simulators.

We extend the conceptual work on social fidelity towards design by complementing prior conceptual work on the design of immersive cooperative work environments (Bjørn et al., 2021). Social fidelity is relevant to all cooperative dimensions embedded within immersive simulators, and prior work has suggested four sets of interlinked design dimensions for cooperative simulators: 1) Spatiotemporal, 2) Cooperatives, 3) Professional, and 4) Temporal orientation (see Table 1) (Ibid). The spatiotemporal dimension includes considerations for the design of the physical context (physical fidelity), in combination with the mobility of actors and artifacts interlinked with the temporal orientation of action time (the time it takes to complete activities) (Ibid). The cooperative dimension includes considerations for the design of cooperative features such as interdependence, articulation work, awareness, and grounding interlinked with the temporal orientation of sequential time (the timely order across activities) (Ibid). The professional dimension includes considerations for the design of cooperative features related to the professional domain (phenomenon/field) and includes considerations for the design of work practices (behavioral fidelity), policies, artifacts (functional fidelity), and professional language interlinked with the temporal orientation of clock time (Ibid). We suggest exploring social fidelity as it is accomplished considering the different types of design dimensions. Thus, we want to interrogate how social fidelity is shaped by the spatiotemporal dimension, the cooperative dimension, the professional dimension, and the temporal orientation reflecting on the results from our experiments with maritime VR training simulators.

Table 1. Design dimensions for immersive Cooperative Work Environment (CWE) adapted from (Bjørn et al., 2021).

Dimension	Spatiotemporal	Cooperative	Professional
	Loci	Interdependence	Work practices
	Mobility	Articulation work	Policies
		Awareness	Artefacts
		Grounding	Professional language
Temporal orientation	Action time	Sequential time	Clock time

In summary, simulator training is fundamentally about creating learning-by-doing opportunities for professional trainees allowing them training in procedural tasks or safety/emergency procedures. Mock-up cooperative simulators have been around for decades and are known to be successful for cooperative training scenarios. New digital opportunities within immersive technologies are changing simulator training, however, there is a tendency to focus only on technical procedural training, neglecting the important cooperative safety and emergency training where social fidelity is critically important. We complement existing work of social fidelity with design dimensions for immersive cooperative work environments to explore the research question: How can we design social fidelity into immersive VR training simulators?

3. Method

Investigating how to design social fidelity into immersive VR training simulators, we designed, implemented, and tested a cooperative immersive training simulator prototype.

The prototype allowed three people to use head-mounted virtual reality displays to train the safety involved when mooring a containership in the maritime domain. Below we provide more details about the empirical context, design, implementation, testing, and data analysis.

3.1. The empirical context: Maersk training

In Maersk, seven people died due to safety accidents in 2018 alone (Pico, 2019).

Maersk Training was created in 1978 to prevent accidents caused by human errors. Maersk Training’s key value is that “no loss should hit us which can be avoided with constant care” and their mission is to reduce human error and prevent human casualties in high-risk work. Their training approach is that training should be as close to real life as possible, and they have custom-made advanced mock-up simulators to provide realistic training. 3500 seafarers from all over the world travel to the island of Fyn in Denmark each year to train in social dynamics and behavioral change in high-risk situations. 90% of the training costs are related to travel and accommodation, and with 70.000 days of training each year, the CO2 cost of training is high.

To explore the potential of VR technology for safety training, Maersk Training is interested in experimenting with different types of setups to see if they can utilize the new technological opportunities. Thus, we set up a collaborative research project where we together with Maersk Training explored the potential of using VR to create immersive cooperative training environments

3.2. Design of the mooring scenario

Maritime simulators often replicate specific work domains on a ship such as the ship bridge, the cargo area, or the vessel traffic services (Kim, Sharma, et al., 2021). Together with Maersk Training, we chose to focus on the deck as the space, and the practice of mooring. We analyzed eight videos of mooring practices, both alone and together with experts using walk-through techniques (Figure 1). Due to COVID-19 we had to do the walk-through of the material remotely and were not able to observe the practices in real life.

Figure 1. Screenshot of online walk-through of mooring videos.

To moor a ship is to secure the ship on land (on the berth/shore) preventing the ship from moving. Our design focuses on mooring the ship from the front deck using heaving-lines, head-lines, and spring-lines (Figure 2). To moor the ship, it requires a minimum of three people (the officer and two sailors – also referred to as the Responsible Person and two Able Bodies in the maritime vocabulary) on the deck and at least one person on the berth who receives or releases the ropes on the bollards. The officer divides and coordinates the work between the team, makes decisions and gives orders while controlling the winches at the control panel. The two sailors fulfill the orders from the officer, prepare winches and organize the ropes. Ropes are attached to winches on the deck. Spring-lines are large heavy ropes securing the ship as close as possible to the shore. Head-lines are also heavy ropes, which restrain the ship from moving backwards or forwards (opposite positions from the spring-lines). Heaving-lines are thinner ropes which are attached to the spring- and head-lines, allowing the sailors to throw and catch before using the winches to pull or release the spring- and head-lines and secure them on the bollards.

Figure 2. Deck layout: Winches (numbers) and positions (Letters).

The activity of mooring is performed every time a ship arrives or leaves the harbor. While mooring is a repetitive activity, it differs each time depending upon the design of the berth, the weather conditions (rain, sun, wind, snow), and the people. Moving on deck between the large ropes is very dangerous, as any rope can break (or snap) causing serious injuries for the seafarers (worst case causing death). Following safety procedures, sailors should never walk over ropes, should stay away from tight ropes reducing the risk of snap-back zones, and generally ensure that winches and ropes are not causing safety hazards. The three main injuries caused by mooring incidents are injuries to the legs (23%), to the back (14%), and death (14%). Training for mooring is not done in large simulators but instead has been executed as a combination of desktop slides and side-by-side training on the ship. Most of the risk in mooring is related to the communication and cooperative engagement between the people involved, and such activities are not easily trained through desktop training. Thus, the mooring activity was an excellent scenario to explore the design of social fidelity in immersive VR training environments.

The design of the immersive environment requires us to design and implement fully-bodied avatars with functional body movements, winches with functionality, ropes (both spring-, head-, and heaving-lines), and control functions operating the winches and ropes. Finally, we needed a ship deck as well as a berth including bollards (Figure 5).

3.3. Designing for social fidelity

Our overall design choice for the prototype was to create a simulator which provided the best possible conditions for the participants to engage in social collaborative interaction supported by social fidelity. We designed social fidelity into the prototype inspired by the design dimensions for CWE (Bjørn et al., 2021) considering the spatiotemporal dimension, the cooperative dimension, the professional dimension, and temporal orientation for the specific environment and artifact design as well as the design of the cooperative task. Cooperative work is fundamentally embedded and organized through the dependencies shaped by the cooperative task (Ciolfi et al., 2023; Duckert & Bjørn, 2024; Grudin, 1994). Thus, designing for social fidelity first requires us to identify a relevant cooperative task for the simulator training, and analyze the dependencies embedded within this task, and how these dependencies are connected to artifacts in the professional work practice. We chose the mooring scenario since mooring is a task where at least three people are mutually dependent in work in order to solve a shared task. Secondly, we analyzed how the artifacts (e.g. wrenches and rope) shaped the professional context, and the required articulation work, including situational awareness and grounding activities between participants. These insights allowed us to identify the essential cooperative artifacts for the task and to understand which dependencies were crucial to embed within the prototype. Professional maritime activities are connected to specific locations within the environment (on the ship), and artifacts (e.g. ropes) change their professional meaning through the changes of different artifact states. By identifying these states, we were able to design the sequential development of the task considering all the dependencies and related situational awareness activities reducing the effort of articulation work while improving safety. Third, safety in the maritime practice is closely connected to the mobility of participants (how maritime professionals move around on the deck). Thus, we need to understand and design for social fidelity by both allowing for open-ended free mobility, while considering how the changes in artifact states impacted the environment and added additional open-ended constraints into the cooperative environment (e.g. ropes on the deck) increasing the danger of the task, allowing participants to train safety by placing their bodies appropriately to the artifacts, as well as paying attention to other cooperators bodies within the environment simultaneously. To be able to pay attention to others’ bodies, designing for social fidelity also requires full-body avatars and the ability for others to see these avatars. Our design for social fidelity thus centered around creating scenarios and environments producing the sociomaterial connections (Bjørn & Østerlund, 2014) from the real-world scenario to the simulator focusing on the design of artifacts and avatars, their mobility, and activities considering locations and state changes.

Actively designing for social fidelity, we can only design for the best possible conditions for social fidelity to emerge. Cooperative work is open-ended, thus how social fidelity is produced within the VR simulator depends on participants’ engagement with the prototype. In this way, cooperative VR simulators exist upon the same fundamental conditions for cooperative technologies as any other type of cooperative technologies, namely dependent upon collaboration readiness and collaboration technology readiness of the participants (Bjørn et al., 2014; Olson & Olson, 2000). Despite this challenge, we argue that we can design improved conditions for social fidelity to take place and emerge in practice as part of our design.

3.4. Implementation of the prototype

The digital representation of the containership is based upon a fbx-file with a real-size 3D model of the front deck of a large containership. The 3D model is detailed and includes multiple small 3D pieces without any kind of functionality (Figure 3). Because the size of the 3D file was large, we needed to reduce the size (removing all unnecessary information besides the deck) to ensure that the headset could render details without too much lag. We added colliders on the deck, but not on the objects due to the limited space and the rendering time.

Figure 3. 3D model of the deck in the VR containership from the bird’s view.

For implementing the ropes, we used Obi Rope Studio, which uses a united particles design which creates a realistic rope that swings, wraps, and folds. We also implemented six winches (Figure 4). To distinguish the ropes from each other we used different colors. For grabbing and throwing functionality we added cubes to the end of the robes.

Figure 4. Winch and rope system.

We implemented a control panel, which could control the speed of the winches. To use the winches required all three people to collaborate. The officer by the control panel controls the head- and spring-lines, while one sailor controls the button by the winch impacting the heaving-line, and the second sailor monitors the ropes and reports on the status. The task is to moor the ship with six winches at the berth by reaching the target points located near the longshoremen.

Figure 5. The berth, target points and longshoremen.

Further, we implemented three avatars using Unity’s Robot Kyle in three different colors (pink, blue and gray – Figure 6) making it easier for participants to distinguish between the different roles of officers or sailors. In the result section reporting upon our findings, we refer to the different roles as Officer, Pink, and Blue, where Pink and Blue are the two sailors. Avatars can move around the whole ship, grab, and interact with the winches, ropes, and control panels. They replicate the movements of the participants in real life following the design principle of action time (Bjørn et al., 2021). However, since the ship is large and we had limited physical space for participants, we also implemented walking in the left thumb stick on the controller. To synchronize the mooring simulation across multiple Oculus devices we used Photon Unity Networking 2 (PUN2).

Figure 6. Implementation of Avatars.

3.5. Testing setup

We tested the cooperative immersive environment in the Maritime training facility in Svendborg, Denmark through eight tests (two pretests and six main tests). The pretests were in an almost finished prototype, while the main tests were using a “stable prototype.” Each test had three maritime participants with different types of experiences (novice to expert). For each session, the participants were instructed on how to use the VR headset and controllers – and how to move around in the immersive simulation and interact with artifacts. After this initial introduction, the participants decided on roles (officers or sailors), and they logged into the environment. Each session began with a “toolbox” talk (coordination meeting), and then the task was to moor the ship with the six winches. All groups managed to moor the ship. We recorded the audio and point-of-view of all participants and one additional bird-view of each test session. After the test, we combined the four video recordings into one video, which we used as the discussion point in the post-interview with the three participants. The post-interviews took the form of reflective conversations (Bjørn & Boulus, 2011) in that we wanted to see how the participants were able to reflect upon their experiences by prompting incidents which happened during the scenarios. We recorded the audio of all the post-interviews. In total 22 maritime professionals participated in our tests in the period between May 25^th to June 11^th, 2021 (see Table 2).

Table 2. The 22 maritime participants in our prototype experiment, their roles in real life and roles they took in the test sessions (officer or sailors).

Test session	Maritime Role (real life)	Role in simulation
Pre-test 1	Operation manager	Officer
Pre-test 1	Maritime instructor	Sailor
Pre-test 1	Drilling instructor	Sailor
Pre-test 2*	Chief engineer/Dual officer	Officer
Pre-test 2	Drilling instructor	Sailor
Test 1	Ordinary ship’s assistant	Sailor
Test 1	Ordinary ship’s assistant	Sailor
Test 1	Ordinary ship’s assistant	Officer
Test 2	Master	Sailor
Test 2	Skipper	Officer
Test 2	Skipper	Sailor
Test 3	Maritime HF student	Sailor
Test 3	Maritime HF student	Sailor
Test 3	Maritime HF student	Officer
Test 4	Maritime HF student	Sailor
Test 4	Maritime HF student	Sailor
Test 4	Maritime HF student	Officer
Test 5	Student/deck cadet	Sailor
Test 5	Student/deck cadet	Officer
Test 5	Student/deck cadet	Sailor
Test 6	Maritime instructor/Deck officer	Sailor
Test 6	Sales manager	Officer
Test 6	Maritime instructor/captain	Sailor

*Pretest 2 only had two maritime professionals; the remaining role (sailor) was executed by one of the researchers.

3.6. Data analysis

All the recordings from the test sessions and post-interviews (including one audio stream) were collected and saved for analysis. In total 320 minutes of video and audio material were collected (110 minutes of mooring scenarios and 210 minutes post-interview audio). We transcribed all the material and annotated the video. We used NVivo 12 for the organization of the material. We went through the data material in several iterations. First to gain familiarity with the data as a collective form of material, and then later to identify aspects of interest which was important for the social interaction taking place. We were inspired by qualitative analysis and grounded theory (Glaser & Strauss, 1967) since we were not trying to support nor reject specific hypotheses, rather we wanted to understand the nuances of the social interaction to determine the nature of social fidelity as it was enacted by participants. We wanted to have the data material pointing us in relevant directions (Bjørn & Boulus-Rødje, 2015) unpacking social fidelity rather than evaluating pre-determined paths. In this part of the analysis we were inspired by CSCW video analysis techniques (Heath et al., 2010) in the ways that we looked for how participants moved, used artifacts, and interacted. Moreover, we had an interest in understanding how social fidelity was shaped by the cooperative practices we designed for and thus displayed within the immersive cooperative environments. Because the immersive environment was designed as open-ended with little to no pre-determined procedures (winches and ropes acted in certain ways, everything else was up to the participants), we wanted to know if participants were able to utilize the potential to develop the necessary social interaction important for safety training.

We used different tactics during the analysis such as cross-case comparison considering specific aspects such as how participants enacted situational awareness or how participants’ experience shaped the social interaction in test sessions. Experience and prior knowledge mattered for how participants were able to take advantage of the potential for social fidelity. For example, we saw how highly experienced participants – both from long-term real-life experience at sea and then as training instructors at the training facility were able to provide an extra set of meta-reflection for social fidelity in immersive cooperative environments. As a former captain and maritime instructor explained:

“This is the first time that communication and teamwork have a proper meaning in this [Virtual Reality]. You just must remember all that with heaving-lines and mooring and winches and stuff like that, it does not matter, it does not matter at all [for the important part of safety training]. This [safety training] is about communication and cooperation, and that you must perform a safe mooring operation … and we have achieved that. Therefore, this [the cooperative VR environment] differs radically from all the other high-end [virtual reality]” (Captain and Maritime Instructor).

We used all the insights from the cross-case analysis to strengthen our analytical sense, which allowed us to nuance how social interaction was accomplished providing insights for how we can design social fidelity into immersive cooperative simulators utilizing VR technologies.

Examples of empirical observations which we scrutinized in more detail were dramatic situations, frivolous interactions, breakdowns, unsafe situations (e.g., participants standing in snapback zones), or miscommunications. Through the iterative data analysis, we identified three aspects which were critical for the development of social interaction in the immersive environment, and which thus emerged as features of social fidelity: Awareness, communication, and coordination. Further, we discovered situations where the social fidelity of the immersive training environment was broken. Understanding breakdowns in social fidelity is important if we are to create guidelines for designing social fidelity into immersive cooperative training environments utilizing virtual reality.

4. Results

4.1. Awareness as a feature of social fidelity

Designing for social fidelity bringing insights from CSCW, we designed the immersive environment in such a way that participants could practice situational awareness both as a principle for maritime safety and as a feature of cooperative work. Now the question is whether we succeeded and how awareness as part of social fidelity emerged in the test sessions. Awareness as featured in cooperative work is about “making things (activities, bodies, artefacts etc.)” visibly or audibly available for others to monitor and act accordingly (Gutwin & Greenberg, 2002; Heath & Luff, 1992). Thus, analyzing our data we paid attention to the monitoring practices of participants, as well as the type of cues used by participants.

Monitoring practices included the mobility of the different participants, and how they positioned themselves to gain access to “awareness information” about other people’s activities during seamless social interaction. Further, participants were able to articulate these considerations, as the below quotation demonstrates:

“It [The location behind the controller] was the closest place I could go to without standing in the way of [The officer], to look out. Because I have agreed with [Blue] that I would keep watch, and then I found the easiest place I could stand was behind [the officer]. I could not stand in front of [the officer] because I would be too close to winch 1 which was not tightened up yet. […] If I had to find another place, then I would have to go all the way over, and over the line, and behind the one [winch 2] [Blue] was working on […] This is why I chose that place” (Test session 6, post-interview).

What is interesting to notice in the above quote is how the participant considers their own “physical” position in the ship deck environment both in gaining access to contextual information relevant to the task of mooring (being able to keep watch), and to accommodate other participants gaining access to important situational information about the context (without standing in the way). Further, the decision of physical location was also shaped by concerns for safety, namely how the location of the avatar body included “assessing” the distance to the winches and rope – as well as bearing in mind the state of the artifacts (winch 1 not yet tightened up). Situational awareness thus entailed multiple interlinked reflections, which simultaneously were relevant as single concerns as well as considerations for parallel and sequential concerns. As the participant continues to explain and reflect upon their positioning: ”[I did not want to]be under it [the rope], behind it or any places nearby it [the heaving-line].” Further, he explained that potential risks involved with a spring-line are dangerous and thus the location of bodies matters if the rope snaps during an accident. Interestingly, choosing to place the avatar body behind the officer also demonstrated the participant’s highly developed sense of safety and competencies in predicting potential risk – also for others as he explains:

“I could have stood in front of [the officer], but you do not stand in front of the person in charge because then you could risk standing and shadowing his view” (Test session 6, post-interview).

Situational awareness was also displayed in the test scenarios using audible cues. Speaking aloud participants informed others about their whereabouts, their activities, and their use of artifacts (ropes and winches). In the below example, participants continue to inform each other about the state of the situation.

Figure 7 illustrates how awareness is featured as part of the social interaction in the immersive environment connecting both visibility cues and audio cues. Interestingly in the empirical example in Figure 7, the officer never looks away from the panel, instead, he makes inquiries to others asking them to inform him about the status of the task. The two sailors (Pink and Blue) monitor the status of the rope. Blue explains later that he is color-blind and thus cannot see the changes in colors. When Pink understands Blue is unclear about the color change, he takes over. Pink informs about the color turning green (indicating that the rope has reached the bollard) and the officer and Blue continue after the task progress has been confirmed. The audible cues serve as essential markers supporting awareness features of social fidelity in the cooperative immersive environment.

Figure 7. Awareness cues and interaction.

4.2. Communication as a feature of social fidelity

Communication features in immersive environments are part of social fidelity. Communication took different forms in our test sessions both in terms of creating common ground (Bjørn et al., 2014; Olson & Olson, 2000) through grounding processes (Clark et al., 1991), as well as ways to resolve uncertainties.

Examples of communication activities which we identified in our test sessions included repeating orders and confirming information exercising the professional maritime vocabulary. One example which was repeated across test sessions was the maritime vocabulary combining bodily gestures and audible cues. In the maritime domain, a core command is “all stop” supported by an arm gesture of raising one hand above the head in a fist and keeping the hand there until people can continue. This professional command can be executed by all maritime personnel no matter rank or position and is used to ensure that anyone who notices a potential risk or predicts a risk emerging is responsible for stopping all activities until the situation is safe again. We saw how participants across all our test sessions did the “all stop” gesture several times. One example of this practice is from the situation where the spring-lines need to the tightened and thus the interrelations between the officer, Blue and Pink are important. In the below case, Blue is monitoring the berth and notices that the bollard turns from red to yellow, which means that the spring-line has reached the berth and now needs to be tightened. The below communication begins (Figure 8):

Figure 8. Confirming orders by repeating orders.

What is evident in this example is that both the officer and Pink respond together and immediately repeat Blue’s command “All stop.” Further, they also stop their activities and wait for the next step. Across all test sessions, this communication pattern emerged. Repeating orders is a key part of the professional language at sea, and to fully take advantage of the social fidelity in the immersive training environment it is critical that participants perform professionally. Confirming orders by repeating them is a vital part of how communication takes place in the maritime domain, as explained by one of the participants:

”Before you [both sailors] start anything. […] own initiative is okay but tell me what your intentions are before you perform any activities, then I will confirm what you say, so you know that I have heard it and know what your intention is.” (Test session P1)

Repeating orders served as a mechanism to create and establish common ground and acknowledge communication efforts. Below is an example:

“The team is getting ready to fasten winch 2, waiting for the officer to distribute tasks between the able bodies. The officer says “Now I would like Pink, you can go keep an eye over the side, by Blue, because I am not able to see what [he was interrupted]” Pink answers the officer: “Yes, but I can see it from here.” Blue responds: “Super” The officer focuses on Blue and says: “Okay Blue, you can throw the heaving-line now.” Blue responds: “Yes, I’m trying to grab the heaving-line […]” (Test session 5)

The above quote demonstrates how participants acknowledge each other’s utterances and thus engage in grounding activities supporting the establishment of common ground based on professional vocabulary and gestures.

4.3. Coordination as a feature of social fidelity

Coordination as a feature of social fidelity in immersive training simulators was evident in our data analysis by how participants engaged in effort of articulation work and coordination. Coordinating is the act of distributing tasks, and establishing a plan for what needs to be done (Schmidt, 1997; Schmidt & Simone, 1996) – while allowing for situated actions to unfold (Suchman, 2007). In maritime professional practice, all assignments and tasks often start with a “Toolbox” talk. Toolbox talk is the professional vocabulary for the initial coordination of a shared task before carrying out the task. Participants in our test sessions all used the Toolbox talk for coordination including task distribution, plans, and creating a structure to organize the tasks. Below we provide an example (Figure 9):

“The Officer: “Okay so we need to have moored the ship, we need to have 2 spring-lines from winch 1 and winch 2 and then we need to have 4 head-lines from the remaining 4 winches. We start by sending a spring-line from winch 1, then we send 4 head-lines and then finally we tighten up the winches, with the last spring-line on winch 2. After the task distribution, all three participants move into their assigned area, and finally, the officer arrives at the control panel, Pink positions himself at winch 1, while Blue positions himself at winch 2”. (Test session 5)

Figure 9. Coordination of task distribution during the toolbox talk.

Interestingly, Team 5 coordinated their task distribution in the same way as they would have done in real-life, and further, they even used the winches as coordinative artifacts by choosing to follow the procedure known by the officer to not tighten the first spring-line completely, until all the ropes were in place (at their target points), and then return to the first spring-line to complete the task. When asked about the task distribution activity during the post-interview, the officer explained:

“Typically, you would do at least what we did, you take your first spring-line they are the ones that go backwards towards the ship, and the front goes backwards, one at each end … you take them and tighten a bit to hold the ship in place towards the quay, and then you usually take the rest all the head-lines out and then the other spring-line at the end.’ (Test session 5, post-interview)

Coordination did not only take place during the toolbox talk; coordination also happened continuously during the whole mooring task. Moreover, it was not only the officer who delegated the task, we also saw several scenarios where the sailors would coordinate internally. Below is an example:

“Blue:

‘I’m standing right by winch 3 here. The question is should I take it and you have to go along with the wall?’

Pink:

‘Yes, I will get right behind winches 1 and 2, and then come over and find a good place to stand. Take the 3rd.’

Blue:

‘I take the 3rd so you can take over the 4th and get ready’

Pink:

‘So I’m an observer’

Officer:

‘You know what I have a good view of the 3rd and 4th. So, you can send a heaving-line each.’

Blue:

‘Okay’ “(Test session 6)

In the above example, we see how the sailors internally with the officer in the loop coordinate their tasks during the mooring activity. Besides the coordination of tasks, the coordination of sequential structures who should do what, and in which order also took place in the test sessions. In the above example, Pink is not ready for the next sequential step in the task. The sequential ordering of tasks does not just include the overall distribution of tasks as in who does what and when – but also includes the physical positioning of involved actors. Participants cannot perform some of their tasks if they are not correctly positioned, and thus the positioning as well as the status of artifacts (winches and ropes) acts as coordination mechanisms.

4.4. Social Fidelity breakdowns

While awareness, communication, and coordination each featured as part of the social fidelity in the immersive environment, there were also situations where the immersive social experience became fragile and, in some situations, was characterized by social fidelity breakdowns. One example of social fidelity breakdowns was in test session 3, where two participants fell overboard while throwing the heaving-line. The situation was as follows (Figure 10):

Figure 10. Pink (left top) and Blue (right top) in the virtual void next to the ship, Test session 3.

Pink throws the heaving-line towards the berth, however, is too close to the edge of the ship and the movement and the throw causes Pink to fall overboard (landing in the virtual void next to the ship). Blue has a different heaving-line in their hand and starts to throw the rope toward the shore. Unfortunately, Blue repeats the mistake of being too close to the edge of the ship and also ends up in the virtual void next to the ship. The officer tried to guide the sailors, by exclaiming: “Walk and press the button I am telling you! It is an order!

(Test session 3)

Falling overboard and into the virtual void created a breakdown in social fidelity. Falling overboard in real life will shift the attention toward emergency procedures of saving maritime personnel from the water. However, in the training environment, the attention was directed to the functionality of the immersive environment rather than to the cooperative work procedures of training moorings.

During the post-interview for the above test session 3, a domain expert and maritime trainer arrived and joined the conversation. When the expert instructor arrived, his presence mattered for the social interaction within the training context, as the test team clearly explained that if they had known that the expert would arrive, they would have made an effort to engage with the social fidelity features and the safety procedures of mooring in more proactive ways. The test session 3 team displayed anxiety when they learned that their performance in the training simulator was to be assessed by an expert. The domain expert carefully phrased his concerns during the post-interview by saying: “yes okay you are falling overboard it’s a bit unfortunate;” after the participants left, he continued explaining that acting professionally is critical when using simulators for training. He emphasized how experience and skills are crucial when taking the role and responsibility seriously during simulator training, and how the instructor’s articulation of the training shapes the training situation in important ways. For example, he mentioned that they would never refer to the simulator as a “game,” since it took away the seriousness of the simulation. Interestingly, what emerged in this situation was that the social fidelity was shaped by external participants to the people in the immersive environment.

Another example, where participants explained that they would have done the simulation differently had it been in real life was the behavior around the ropes. When mooring a ship is almost complete, both in real life and in our cooperative VR environment, it is very difficult to move around safely without crossing any ropes since there are ropes tight up everywhere. Still, this is a critical safety guideline, and the risk of something happening with severe consequences while walking over a tightened rope is high.

Figure 11 shows many ropes tightened across the deck. If a rope breaks and seafarers are in the snapback zone there is a high-risk situation. Across all test sessions the more inexperienced teams did not follow these safety guidelines, but instead walked over ropes and placed themselves in the snapback zones. When asking participants about their unsafe behavior walking over ropes, they explained:

Figure 11. Almost complete mooring with six winches.

I admit that I sometimes * just * went over some lines. You may not be able to do that in reality [Other participant exclaims: No!] but then . the problem is a bit that it takes so much [effort to walk around in the environment], then it is not so easy to be so mobile [because it is slow] so it takes a little shortcut, it can be when I walked around as you walk in over the mooring line, normally you would walk behind [around the mooring lines], especially if they, when they are in motion then you stay completely away from them [mooring lines]. and when they are pulled in, you stand outside the snapback zones [in a safe area].’

(Test session 5, post-interview)

In the above quote, participants explain how they did not take the safety procedures as seriously as they would have in real life. Designing for social fidelity is about creating accountability in the actions across participants related to features of awareness, communication, and coordination – thus monitoring others’ unsafe actions should cause calling out for alerts. However, we did not see examples, where people would call out the unsafe behavior of walking across ropes in the cases of less-skilled participants and no experts were present. This was mostly because they did not pay attention to the behavior of walking over ropes done by others. Without accountability in the design of social fidelity, there is a risk of breakdown, not because of technical constraints, but because of the social organization of work. The waiting, the effort, and the complexity of the scenarios are designed to produce social fidelity as closely related to real-life practices allowing for the participants to experience what it is like when they are on a ship. The design of the social interaction was not about finishing mooring fast, but about finishing mooring safely.

When the accountability required for social fidelity breaks down, the opportunities for high-quality training in high-risk tasks decrease. Social fidelity is critical to obtain in the immersive environment. Interestingly our data demonstrate that social fidelity is created, not only concerning participants within the training simulator but also shaped by the relations between people inside (trainees) and outside (instructor) of the training simulator. Social fidelity is an important part of the accountability in the immersive experience and is directly related to professional behavior and the ability to learn how to avoid safety hazards.

5. Discussion

We set out to design social fidelity into an immersive VR training simulator. Mock-up cooperative simulators have advantages for training social interaction among participants (Kim, Sharma, et al., 2021) and the question is whether our cooperative prototype would be able to complement and potentially expand these advantages into digitally designed immersive simulators using VR.

5.1. Social fidelity features

Social fidelity encompasses the nature of social interaction as they emerge in and around the cooperative simulator (Wahl, 2020), however, to develop social fidelity as a concept for design, we need to understand the complexity in more detail. By reflecting on the design dimensions for immersive cooperative work simulators concerning social fidelity in relation to the empirical observations, we found that the social skill of enacting awareness as part of mooring practices could be detected by examining how participants inform or pay attention to others’ positions and their activities. The monitoring was facilitated by the spatiotemporal context which allowed participants to connect the location of others in comparison with their own location monitoring the collected activities while predicting potential risks ahead. This was evident in the empirical observations where participants explicitly shared awareness information and engaged in seamless interaction while paying attention to and monitoring collaborators. In this way, the contextual nature of the simulator environment facilitated actors in enacting situational awareness (Heath & Luff, 1992) supported by the design of social fidelity to include the physical context (with less photogenic fidelity), the mobility of actors (each can move independently) and artifacts (moving ropes). Further, we saw how the participants used their knowledge and perception in considering the distance between artifacts and actors as well as their knowledge of action time to predict and prevent high-risk situations. We were able to detect the awareness in how participants informed each other about the positions of collaborators, informed each other about activities, and finally interacted seamlessly by implicitly paying attention to and monitoring collaborators acting appropriately. Interestingly, situational awareness was closely connected to the spatiotemporal considerations for the design of social fidelity, even if it is categorized in the cooperative dimension in the original framework (Bjørn et al., 2021). Thus, we suggest that designing for social fidelity considering the spatiotemporal design dimension produces valuable and productive insights into how situational awareness is supported. Further, it was clear that to support the spatiotemporal dimension of social fidelity accurate photogenic representation was not needed, however, the capability to “see” each other’s digital “bodies” as situated in the loci of the simulation was crucial. The spatiotemporal dimension reminds us that the “location” of bodies and the ability to assess who is close or far away from elements and entities within the simulation provides important information for social fidelity.

Social fidelity is not only related to situational awareness; our test sessions also demonstrated the importance of communication among actors as part of social fidelity. Analyzing the test sessions, participants displayed different degrees of communication skills, which in some cases followed professional guidelines while others needed improvements. The professional vocabulary and ways of communication were performed by participants in terms of certain words and phrases like “all stop,” as well as in the practice of repeating information or confirming orders. Successful communication practices supported participants in creating a common ground (Olson & Olson, 2000) decreasing uncertainties, and avoiding communication breakdowns (Clark et al., 1991) while establishing a shared meaning context (Bjørn & Ngwenyama, 2009). How participants communicated matters for how successful they were in coordinating the different interdependent sub-tasks involved in mooring activities. Thus, assessing the quality of the communication emerged as a factor for assessing the competencies, skills, and expertise demonstrated by the participants. Interestingly, the communication activities as part of social fidelity emerged across the cooperative- and professional design dimension in the original framework (Bjørn et al., 2021), since grounding, policies, and professional language were key features in the communication analyzed. The communication detected in our test session might appear simplistic as in repeating and confirming orders, however zooming out of the communication activities as part of the social fidelity in the simulation, these simple actions were fundamentally about decreasing uncertainties. Decreasing uncertainties was linked to making auditable cues to others about aspects of the work environment which needed attention with visual accessibility. Actors would have their back on each other, but by repeating or confirming orders they acknowledge others’ presence and activities aligning expectations. Aligning expectations and acknowledging each other’s activities thus emerged as important parts of the design of social fidelity.

Social fidelity also emerged as pertinent in how participants enacted their cooperative skills of coordination. In the test sessions, participants were able to provide sequential information, distribute tasks, or inform each other about tasks’ progress appropriately. Coordination takes many different forms and is highly related to the specific task and available coordinative artifacts (Dourish & Bellotti, 1992; Møller & Dourish, 2010), and our results pointed to the importance of designing coordination through mutual interdependencies as part of the use of artifacts and location of people. Through this design approach, we forced participants to engage with each other, since otherwise they could not solve the shared task. In this way, it was through the design of interdependency that participants got to exercise important articulation work as shaped by the professional work practices and use of coordinative artifacts. The connections between the three sets of coordinative artifacts (Schmidt & Simone, 1996) (winch, rope, and control panels) allowed participants a free space to decide how to act and organize their time. The open-endedness in the simulator allowed participants to go outside prescribed protocols and practices (Bjørn, 2003; Schmidt, 1997) as was evident in examples where participants changed the sequential order of the work, their mobility patterns, or even fell overboard. The design thus allows for different forms of situated action (Suchman, 1983, 1987). The flexibility of the simulator is created and co-constituted by the participants. It is through their decisions and engagement – the way they perform together – that flexibility arises. No two of the test sessions displayed the same scenario, they were all different depending on the involved people and their relation. We would argue that the strength of the cooperative simulator is to practice non-technical skills – displayed and performed by participants in terms of awareness, communication, and coordination to rehearse high-risk safety tasks. Table 3 provides an overview of the three types of social fidelity features which emerged in our testing sessions.

Table 3. Social fidelity features in immersive cooperative simulators.

Awareness	Communication	Coordination
Inform about positions of collaborators	Decrease uncertainties	Provide sequential information
Inform about activities	Repeat information	Distribute tasks
Interact seamlessly by implicit paying attention to and monitor collaborators acting appropriately	Confirm orders	Inform about task status and progress

5.2. Designing for social fidelity

All immersive training is based upon the principles of learning-by-doing (Schön, 1983) achieving professional knowledge and practice (Lave & Wenger, 1991), however, there is a distinction between technical training and then safety and emergency training (Fracaro et al., 2022). Technical training depends and focuses on details and accuracy close to photogenic likeliness; while safety and emergency training depends upon the storytelling, the scenario, and the interrelations between people. Thus, social fidelity as a feature of an immersive training environment is critically important for safety and emergency training, however less critical for technical training. Based upon our work, we find that designing social fidelity into immersive training simulators is fundamentally about identifying and emphasizing the interdependence in the common field of work, which shapes and steers the cooperative activities trainees are to navigate. Without the interdependence embedded into the scenario, artifacts, and contextual surroundings, participants will not be able to experience and engage in the articulation work required for the tasks, which is at the core of safety and emergency training. Thus, differently from ordinary CSCW designs, designing for social fidelity in immersive training simulators is not about reducing the efforts of articulation work, but instead about preserving the efforts of articulation work. In some cases, designers of immersive training environments might even strengthen the interdependence to ensure that participants notice and experience how their activities and actions impact the common field of work and thus everybody else. Thus, we confirm prior findings on cooperative immersive environments (Bjørn et al., 2021). Emphasizing interdependence is related to the scenario, activities, artifacts, and locations – and thus requires detailed knowledge about the professional practices to create relevant training experiences enfolding the social fidelity features of coordination, communication, and awareness (see Table 3).

Reflecting upon the potentials suggested in the literature for utilizing virtual reality in creating immersive training environments for safety and emergency training (Kim, Sharma, et al., 2021), our work confirms that an immersive cooperative VR simulator does indeed allow trainees to practice and rehearse high-risk tasks (e.g., safe mooring operations) and the opportunity to practice the non-technical skills such as leadership (Hontvedt & Øvergård, 2020; Kim, Sharma, et al., 2021) – thus the strength of such simulators is how they enact and provide opportunities for experiencing social interaction supported by social fidelity. On this basis, we propose three overall guidelines for designing social fidelity into immersive cooperative simulators for safety and emergency training (Table 4).

Table 4. Design Guidelines for Social Fidelity.

Design guidelines for social fidelity into immersive cooperative simulators for safety and emergency training
Designing for social fidelity includes spatiotemporal considerations for how participants can enact situational awareness (interact seamlessly by implicitly monitoring others’ activities in the contextual situation while acting appropriately) as a collective practice in terms of context, mobility, and action time in the digital simulator. This requires full-body avatars with mobility features and action time constraints.
Designing for social fidelity includes considerations for supporting communication activities allowing for grounding practices creating common ground and exercising professional vocabulary including verbal phrasing as well as gestures and bodily cues in the simulator. This requires audio links (potentially demonstrating the physical constraints in the real-life scenario) as well as access to bodily gestures and communication artefacts.
Designing for social fidelity requires us to design for mutual interdependence across actors by including coordinative artefacts to interlinked activities allowing for participants to exercise articulation work. This requires us to carefully consider task design and artefact design in terms of multiple parallel or sequential activities.

5.3. The risk of breaking social fidelity – a future design challenge

When designing for social fidelity, we also need to consider the robustness of our designs. Social fidelity is linked to the immersive nature of the social experience in the simulator, and in our test sessions, we saw examples of how this immersion broke down. Interestingly, Wahl points out that social fidelity is not only about what takes place among participants in the simulator but includes connections and relations outside of the simulator such as the instructor’s guidance and roleplay (Wahl, 2020). The social fidelity context thus expands the activities within the simulator to include the people external to the simulator, and the relations between external and internal participants. In our empirical test sessions, we took on the role of instructor in several of the sessions, and when comparing the sessions where professional instructors were present (either as participants or as instructors) it was evident that professional instructors matter for the quality of the training. We discussed with the instructors what the “quality” of the training would look like and be observable in the immersive simulator. We found that quality would entail participants’ ways to demonstrate their commitment to safety procedures. This would mean that participants acted their assigned role and responsibility by professionally avoiding safety hazards as a group and accomplishing the task safely.

We saw that in situations where the social fidelity broke down, the social relations between the participants and the “instructor” also was broken. The collaboration readiness (Bjørn et al., 2014; Olson & Olson, 2000) was lost, and the relation work (Bjørn & Christensen, 2011; Christensen et al., 2014) jeopardized. The alignment between social relations produced within the simulator and the social relations between people in the simulator and outside the simulator supports Wahl’s suggestion to include the relations with outside participants as part of social fidelity. On this basis, we suggest that when designing for social fidelity, we must include the social relations between people inside and outside the simulator. Thus, when we design a digital cooperative immersive training environment we must look beyond the digital environment for participants and include the trainers and their activities into the design space. If we neglect to do so, we might risk breaking the social fidelity of the training environment. Thus, we suggest that a future design challenge to consider for HCI researchers is to extend the design space both inside and outside the simulator reducing the risk of breaking social fidelity and improving the robustness of the immersive experience.

6. Conclusion

Initially, we asked the research question how can we design social fidelity into an immersive VR training simulator? We explored the question by theoretically expanding the conceptual work on social fidelity with CSCW insights and concepts; and practically through the design and implementation of a prototype for an immersive cooperative training simulator utilizing virtual reality. The prototype was designed for safety training of mooring practices in the maritime domain and tested with maritime professionals with different sets of expertise.

We identify a set of social fidelity features (awareness, communication, and coordination) and propose three design guidelines involving bodies, gestures, and contexts, for how to design social fidelity into digital immersive cooperative training simulators for safety and emergency training. Further, we identify the risk of breaking social fidelity as shaped by the relations between people inside the simulator and people outside the simulator. We propose that future HCI researchers should work on the robustness of social fidelity by engaging with the design challenge of extending the design space for simulator training to include external participants (e.g., instructors).

Acknowledgment

We would like to thank all the participants who joined our test sessions, as well as the University of Copenhagen, Department of Computer Science and Maersk Training, Denmark for supporting this research financially. Finally, we would like to acknowledge the efforts of Lars Skov Engdam and Thomas Bonnelund, both domain experts and Maritime trainers who supported our work, helped to recruit participants, and provided valuable feedback.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Pernille Bjørn

Pernille Bjørn is an ACM Distinguished Member specializing in Computer Supported Cooperative Work and co-author of two books: Sociomateriality & Design, 2014 and Diversity in Computer Science, 2023. This paper is her third on Cooperative VR, former research is published at JCSCW and ECSCW.

Maja Ling Han

Maja Ling Han has an MSc in Computer Science, including an exchange at Hong Kong University and interned at the Machine Agency, at the University of Washington. Her main research interests are VR, fabrication and CSCW.

Andrea Parezanovic

Andrea Parezanovic has an MSc in Computer Science. Her research interests include Human-Centered Computing (HCC) and CSCW, where she explored collaborative technologies and practices using virtual reality. She is currently developing innovative solutions to improve public services for citizens.

Per Larsen

Per Larsen is Head of Innovation in the Digital team at Maersk Training. His areas of expertise and areas in of interest include learning digitalization, learning in the flow of work, use of immersive simulated environments to build skills and strengthen safety in the maritime and energy industries.