Forming a view: a human factors case study of augmented reality collaboration in assembly

Abstract

Industry 4.0 technology is promoted as improving manufacturing flexibility, and competitiveness; though Australia has been slow to adopt. The Australian Navy shipbuilding program provides opportunities for accelerating technology adoption, revitalising manufacturing productivity and competitiveness. Adopting a sociotechnical systems lens, our research sought to identify usability, workload, and user experience of an augmented reality head-mounted display (AR-HMD) deployed to complete multiple work tasks in a workflow (electrical assembly, collaborative robot (cobot) mediated inspection, and remote troubleshooting using video call). Usability was rated ‘average’ (System Usability Scale mean = 69.8) and workload ‘acceptable’ (NASA Task Load Index mean = 25.8) for the AR-HMD alone, with usability of the integrated work system (IWS) rated ‘good’ (SUS mean = 79.2). Results suggest software interfaces, tracking, and gesturing methods for the AR-HMD require improvement. This trial shows the AR-HMD provides a versatile platform for integrating multiple digital technologies without hindering effectiveness of end-user performance, potentially benefiting productivity and quality.

Practitioner Summary:

Using an augmented reality head-mounted display (AR-HMD) to reduce and correct errors in electrical assembly identified factors influencing technology adoption in shipbuilding. Mental workload, interface design, tracking, and gesturing most hindered successful performance. AR-HMDs can facilitate the use of more complex integrated technologies (i.e. cobot), improving usability and acceptance.

Keywords:

• Augmented reality
• head-mounted display
• usability
• collaboration
• sociotechnical system

1. Introduction

1.1. Background

There is a significant opportunity for advanced technologies to contribute to the creation and expansion of capability, opportunities, and markets (Worrall et al. 2021). However, Australian manufacturing has been slow to adopt the advanced technologies and skills required to achieve production flexibility, innovation, and economic complexity essential for boosting global competitiveness (Cisco 2021; Commonwealth of Australia 2023; Dean and Spoehr 2018; World Intellectual Property Organisation 2023).

Digital technologies including augmented reality (AR) promote new production techniques, job redesign, and skills expansion, creating potential for manufacturing process innovation, employment growth, and business model transformation (Sjödin et al. 2021). Research interest in the application of AR technology in manufacturing has accelerated internationally in the last five years. Researchers have examined a variety of technology applications (e.g. assembly, training, quality control), and industry sectors using it (e.g. shipbuilding, automotive, logistics) (Howard et al. 2023). However, little of this research has addressed collaboration or integration studies examining user perspectives, including usability (e.g. Dey et al. 2018).

An advantage of AR technology is its potential to support other advanced technologies, for example, collaborative robots (cobots). Cobots are industrial robotic arms with force and range limitations, collision detection, and reduced pinch points. These functions prevent injuries and allow humans to work safely nearby (De Franco et al. 2019). Cobot utility has been enhanced by integration with AR using the Hololens 2,¹ to provide visualisation and remote-control functionalities (e.g. Demirtas, Cankurt, and Samur 2022; Lotsaris et al. 2021). Fitted with vision systems, cobots can automatically perform task-specific visual inspection (Hulgard 2021), reducing operator demands and errors. AR integration with cobot technologies was studied in a recent systematic review of 32 papers published between 2016 and 2021. Most studies investigated using AR to assist in programming and guiding cobots. Findings indicated AR head-mounted displays (HMDs) had a positive influence on user performance, task awareness, and perceived safety, although wearer comfort and ergonomics required improvement (Costa-deMoura, Petry, and Moreira 2022).

Evidence from Europe suggests HMDs, compared to other AR technology, lack durability and compatibility with existing information technology systems (Jalo et al. 2022). Understandably, business has preferentially adopted handheld AR devices for their flexibility, cost-effectiveness, and low training demands (Jalo et al. 2022). AR-HMD uptake in industry has remained limited by unresolved usability issues (Ariansyah et al. 2022), lack of integration advice for custom manufacturers (Franze et al. 2022), and low integration within the manufacturing supply chain—compared to other technologies like robots (Hopkins 2021; O’Keeffe and Howard 2023).

This case study describes multidisciplinary ergonomics and human factors (EHF) research to advance technology adoption in heavy manufacturing. Naval shipbuilding is chosen because (a) in Australia it remains an industry reliant on traditional manufacturing methods; (b) we wish to explore AR as a platform for integrating technology to increase productivity and reduce workload and (c) new technologies in naval shipbuilding and its supply chain provide process improvement opportunities, such as industrial AR for welding training, digital work instructions, maintenance, and pipe design verification (Fraga-Lamas et al. 2018).

Our study task uses AR and cobot technology to execute and inspect an electrical assembly unit. Visual inspection was initiated through the AR-HMD and completed by an automated computer-vision-enabled cobot (described further below). The AR-HMD also enabled remote troubleshooting. Central to our research is the human experience of utilising this technology in the workplace. We pose the primary research question: How does the integration of technologies (AR-HMD and cobot) affect user experience during a manufacturing assembly task? The findings extend existing knowledge of AR by providing a real-world application of integrating technologies (AR and cobotics) in a manufacturing environment—in this case in a working shipyard.

1.2. Sociotechnical systems, usability, and user experience

Addressing ergonomics and human factors (EHF) in manufacturing automation and process redesign is associated with successful technology adoption. EHF contributes to creating a competitive advantage (Co, Eddy Patuwo, and Hu 1998; Neumann et al. 2021) by prioritising understanding of process interactions through its systems- and design-driven emphasis on the relationship between performance and well-being (Dul et al. 2012). Overlooking the central role of the human in the system risks technologies being under-utilised or abandoned.

Technology adoption is a social and technical process, where increasing complexity leads to systemic changes at operational, organisational, and managerial levels, requiring new strategies to ensure technology fits the environment (Cimini et al. 2021). Recognising this, sociotechnical systems theory (STS) (Walker et al. 2008) integrates the technical (tools, equipment, and technical processes) with the social (people, relationships, and connections) producing networks of interactions (Cimini et al. 2021). The Human-Technology-Organisation (H-T-O) model (see Figure 1) emphasises interactions between subsystems, affecting overall system performance (Dregger et al. 2018). The human-technology (H-T) relationship addresses process control distribution and usability. The human-organisation (H-O) intersection represents the human’s role in meeting system demands using individual and organisational resources. The technology-organisation (T-O) intersection reflects the production process where organisational design integrates technology into work, connecting operational activities with hierarchical organisational structures, including supply chains (Dregger et al. 2018).

Figure 1. The human-organisational-technology model. Adapted from Dregger et al. (2018).

A triangle linking ‘human’ at the top with ‘technology’ on the bottom left and ‘organisation’ on the bottom right. ‘Joint optimisation’ is in the centre. A square on the top left contains usability, performance, skills and barriers. A square at the top right contains task, work design, engagement, efficiency. A square at the centre bottom contains strategy, culture, environment, use cases.

The H-T-O model represents how changes to any part of the system impact other system factors. Technology adoption must aim to achieve joint optimisation of system elements because disruption in the integration of visual, physical, temporal, and task contexts can undermine user acceptance. While a tailored approach targeting individual, job, and organisation levels (Howard et al. 2023) is important when implementing any technology, the H-T-O model provides a framework for knowledge translation. In our case, through applying the H-T-O model to the AR-HMD-mediated assembly and inspection task, lessons from our shipbuilding case can be translated to related sectors (e.g. construction, fabrication).

Technology acceptance refers to the willingness to use technology and positively impacts adoption (i.e. sustained use, dependence on, and desire to use technology) (Nadal, Sas, and Doherty 2020). Technology acceptance and adoption are direct outcomes of usability. The International Standards Organisation (ISO) defines usability as ‘the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use’ (ISO 2019, 3). Usability is mostly concerned with successful human performance (Sauer, Sonderegger, and Schmutz 2020) and includes concepts, such as ease of use, usefulness, and ease of learning (Bevan 2009). An allied concept, user experience refers to a ‘user’s perceptions and responses that result from the use and/or anticipated use of a system, product or service’ (ISO 2019, 4). Usability addresses improving performance outcomes, while user experience encompasses more diverse outcomes concerned with user satisfaction and enjoyment of technology engagement (Hassan and Galal-Edeen 2017; Jung et al. 2017).

As the ISO (2019) definition implies, the fit between task and technology is highly influenced by the context of use, especially in industry (Vidal-Balea et al. 2020) where misalignment between user needs, task structure, and environment can reduce technology usability (Spies, Grobbelaar, and Botha 2020). Yet a systematic review of AR interfaces including 291 papers published between 2005 and 2014 identifies that most studies are laboratory-based and lack pilot testing, potentially limiting application in real-world settings (Dey et al. 2018).

The user perspective on technology interaction is recognised as an important yet complex factor in successful technology adoption, as evidenced by multiple relationships within the H-T-O model. We operationalise the H-T-O model by depicting usability as the demand arising from technology-task sources requiring effort by users (including workload). This demand influences performance at the H-T (usability) and H-O (workload) levels. Workload represents the level of demand placed on the user to achieve task goals, (a function of task design) which produce the overall performance that contributes to individual and organisational success. Usability identifies and rates the relative sources of perceived interface deficiencies, while workload identifies and quantifies the human effort to achieve satisfactory performance. Together these measures target entry points for interventions to improve technology adoption. User experience, a more global concept, encompasses evaluations of overall satisfaction that influence technology acceptance, affecting the T-O level of adoption.

1.3. Augmented reality (AR) technology

AR is a relatively low-cost technology (Bottani and Vignali 2019; Miller, Hoover, and Winer 2020) with great potential for simplifying complex processes in industrial applications. AR supports task performance by overlaying real-world components with information-rich, context specific 3D interactive displays accessible in real time (Azuma 2017; Jetter, Eimecke, and Rese 2018).

Recently, AR has been increasingly applied to support manufacturing processes due to reduced hardware costs; freely available software; improved visualisation, image capture, interaction, and tracking; and development of HMD (Bottani and Vignali 2019). Assembly is the most frequently studied AR application, where decision-making support through data visualisation and reduced time latency adds value by lowering errors and improving efficiency (Danielsson, Holm, and Syberfeldt 2020; Lai et al. 2020; Miller, Hoover, and Winer 2020; Mourtzis, Zogopoulos, and Xanthi 2019). Electrical wiring assembly was chosen for our study, being one of the most common and time-consuming industrial tasks requiring trained and skilled workers (Szajna et al. 2020). In-depth AR studies of assembly performance in real-world settings are emerging, but remain sparse (Vorraber et al. 2020). Successful AR adoption in manufacturing requires the development of use cases, integration (cost-effectiveness, data security, regulatory compliance), set up time and reliability, and operational requirements (accuracy, real-time capability, ergonomics) (Quandt et al. 2018). AR adds value to assembly tasks by supporting video collaboration through access to remote supervisors and experts to understand and solve context-dependent problems (Wang, Hu, and Yang 2022). Studies are emerging examining remote human-human collaboration via AR (e.g. Vorraber et al. 2020; Wang, Li, and Freiheit 2022) showing favourable user satisfaction with visual feedback and usability, though with small sample sizes.

1.4. A-R mediated human-human collaboration

Most emerging AR-HMD human-human collaboration studies assess technical proof of capability and few report usability data. Knopp, Klimant, and Allmacher (2019) conducted a technical proof of capability trial of live-streamed guidance for the assembly/disassembly of modular moulds via AR-HMD, aiming to reduce production downtime. Participant numbers and usability data were not reported but qualitative findings indicate the remote AR was inadequate to evaluate wear on parts and intermittent tracking losses were experienced.

Wang, Hu, and Yang (2022) examined the assembly of automotive parts using ten participants in a study of AR-HMD mediated collaboration. Usability was measured with the System Usability Scale (SUS), receiving a score of 71 (acceptable). All users reported difficulty accurately determining part positioning, assembly orientation, and general visibility during collaboration.

In a shipyard example, Vidal-Balea et al. (2020) trialled AR-mediated collaborative training using the Hololens to guide the assembly of a hydraulic turbine clutch. The case study examined ten experienced male operators validating AR-HMD use with various technical measures, e.g. latency. Participants had no previous experience with AR-HMDs but found the Hololens easy and intuitive to use (90% acceptability), and useful (85% acceptability). Participants in AR-enabled remote collaboration studies reported easier communication using AR from being able to see the person providing advice and receiving visual feedback, leading to greater perceived trust in the expert’s advice.

1.5. Contribution of the current study

This case study addresses three gaps in current research on AR technology. Few applied studies exist in manufacturing using representative participants, in this case, shipbuilding workers. There are limited published usability studies, and few examining technology integration supporting remote human-human collaboration via AR.

In evaluating the research question: How does the integration of technologies (AR-HMD and cobot) affect user experience during a manufacturing assembly task, our research seeks to identify the usability, workload, and user experience impacts of integrating multiple tasks (electrical assembly, inspection, and remote troubleshooting) and technologies (AR-HMD and cobot-mediated inspection) in a simple workflow. The study aims to identify technology limitations (usability) during task execution, and perceptions of ease of use and quality of interaction (user experience). By understanding the user investment (workload) required for effective system use, design strategies can be identified to promote successful technology adoption.

2. Methods

2.1. Study design

An exploratory descriptive mixed method case study design was applied to evaluate participant experience of the use of an AR-HMD and cobot-mediated inspection in electrical assembly. The participants undertook a brief wiring assembly task on an electrical cabinet using digital work instructions. While this research is explicitly about participant experience, the research required relevant workplace experience and a level of technical skills and/or training to undertake the specific tasks, which informed participant selection and study design.

We adopted a case study approach by targeting a specific cohort of heavy industry workers—shipbuilders and production workers in their supply chain. Case studies enable the identification of essential factors, processes, and relationships within context to inform implementation (Rashid et al. 2019), in this case, technology adoption in a unique and harsh environment. Given the applied use and setting, and the burden of applying a ‘no technology’ (i.e. paper-based work instruction) condition with this field-based cohort, a controlled study was not undertaken. This decision is supported by evidence that AR integrated approaches for complex tasks outperform paper-based work instructions, based on errors and completion times (e.g. Dalle Mura and Dini 2021; Uva et al. 2018). In addition, a systematic review supports that AR assistance improves operator efficiency, usability, workload, and satisfaction during cobot interaction in assembly tasks (Costa-deMoura, Petry, and Moreira 2022).

2.2. Apparatus

2.2.1. Environment

The trial took place in a heavy manufacturing workshop partitioned from the main workspace. Participants undertook the task on a fixed height work bench where an isolated electrical cabinet was located for assembly. Risk assessments and safe work procedures were completed, and all participants wore personal protective equipment (earplugs, safety boots, long sleeves, and trousers. Hardhat and safety glasses were removed during AR-HMD use).

2.2.2. Technology and task

The task required participants wearing AR-HMD to assemble eight wires identified by colour and tag number in an electrical cabinet. Specifically, the AR-HMD (see Figure 2) provided digital work instructions integrated with coloured hologram overlays and presented wiring connections and video demonstrations to assist novices (e.g. showing tool use; given no prior electrical experience was required). On assembly completion, visual inspection of the wiring was performed by a cobot² initiated through the AR-HMD to verify accuracy (see Figures 3 and 4).

Figure 2. Worker assembling wiring in electrical cabinet with reference hologram—view from Hololens AR HMD.

Electrical cabinet containing terminals. Two hands are manipulating a tool to attach wires to the terminal. A 3D hologram appears to the right of the terminal to show correct position of wires.

Figure 3. Electrical cabinet containing completed wiring undergoing cobot inspection.

Electrical cabinet containing wiring inserted into terminals with a cobot overhead fitted with computer vision camera conducting visual inspection.

Figure 4. Display of inspection results seen through the AR-HMD.

Display of test results with pass/fail output in a billboard overlay on left side. On right side cobot camera bends over into an electrical cabinet to inspect wiring into terminals.

The research integrated two key advanced technologies. An AR-HMD (Hololens 2), delivered assembly information to participants via a digital work instruction, and enabled connection to a remote supervisor for additional support. The Hololens 2 user interface (including instruction cards and holograms) was developed using Microsoft’s Dynamics Guides 365 software, which also enables the Microsoft Teams video and audio calling function, used to complete remote troubleshooting.

The cobot (UR10e) was powered by Polyscope 5 software featuring a low code programming interface. Researchers used a touchscreen pendant for programming, meaning participant interaction was only via a dashboard (designed using Power Apps and accessed through the AR-HMD) to interact with the cobot. The cobot was fitted with a machine vision camera (IFM O2D502 2D using IFM Vision Assistant software allowing the cobot to read cable tags) (see Figure 3) and was programmed to check for errors and completeness of the task. When connecting to Wi-Fi on the HoloLens 2, participants could access the inspection dashboard through a link provided in Dynamics Guides 365 to instruct the cobot by selecting one of the inspection routines, represented as virtual buttons.

The interactions between the user and the system were:

1. The participant followed the work instructions containing written information, videos, and holograms (Figure 2) to complete the wiring task. Example images showed the correct wire alignment to ensure tags were readable.

2. The dashboard was opened via a link in the work instructions to open Hololens 2’s browser.

3. The participant activated the cobot by pressing a button to enable it to move to inspection pose, then selected the ‘inspect all’ routine on the dashboard, initiating inspection (Figure 3). The camera evaluated the result against a set threshold, determining pass or fail.

4. The cobot returned to its ‘home’ position, allowing the participant to review results via the dashboard and rectify any errors.

5. The participant initiated a remote troubleshooting call to a ‘supervisor’ (a researcher) using the Microsoft Teams function visible on the wrist menu. Errors were corrected collaboratively, and re-inspected, allowing participants to complete the task with all green passes recorded on the dashboard.

User response to technology is a product of the hardware, interface, user abilities, and the environment of use. Piloting of the user interface revealed optimal positioning for the holograms to prevent obstruction of working hands, and optimal colour contrast to distinguish between wires.

2.3. Participants

Thirty-six participants (n = 4 females) aged 18–62 years (mean 41.8 years, SD = 10.0) were recruited from a shipbuilding prime organisation (n = 27) and its manufacturing supply chain (n = 9). While no specific task or technology knowledge was required, we note most of the supply chain cohort (88.9%) held university undergraduate degrees or higher, compared with one-quarter (25.9%) of the shipbuilding cohort, who generally held vocational qualifications. All had manufacturing or construction experience, range of 2–40 years (mean 15.1 years, SD = 11.7). Participants had held their current jobs between 1 and 35 years (mean 6.8 years, SD = 6.8). English was the first language for 86% of participants. One-third (33.3%) reported needing glasses for computer work, with two reporting difficulty discriminating between colours. The research was approved by the Flinders University Human Research Ethics Committee (No. 5117) and all participants provided informed consent before commencement.

2.4. Measures and analysis

Quantitative data enable direct comparison within the sample, provide generalisability and breadth, while qualitative data capture context and depth in understanding human experiences (Friedman, Wyatt, and Ash 2022). Accordingly, mixed methods (Likert survey scales and written comments) were adopted to assess the complex phenomena of usability and user experience (Fernando et al. 2022). On task completion, participants provided verbal feedback about their experience using the AR-HMD, and its effectiveness as an integrated work system (IWS) through its initiation of cobot inspection and remote troubleshooting.

The System Usability Scale (SUS; Brooke 1996; Lewis 2018), is a ten-item scale for evaluating overall perceived system usability. Items are rated from 1 (strongly disagree) to 5 (strongly agree) as raw scores that are adjusted (including the reversal of negative statements) to scores ranging between 0 (negative rating) to 4 (positive rating). These scores are summed and multiplied by 2.5 to convert to a standard score ranging from 0 (poor usability) to 100 (excellent usability). A score of 68 represents an acceptable level of usability, while a score of 80 represents high usability (Lewis and Sauro 2018). The reliability of the SUS is described as high or strong (Taber 2018), demonstrating Cronbach alpha coefficients of 0.91 (Bangor, Kortum, and Miller 2008; 2324 cases) and 0.92 (Lewis and Sauro 2009; 324 cases).

The NASA-TLX (Hart and Staveland 1988) assesses task demands across six dimensions (physical, mental, temporal, effort, performance success, and frustration). Each dimension is ranked on a 21-point scale ranging from 0 (low demand) to 20 (high demand). An overall unweighted raw score is calculated by summing and averaging the scores for each dimension, with possible scores ranging from 0 to 100 (Hart 2006). An overall unweighted workload score of 30 or lower indicates a task has low demands, while scores exceeding this level imply high task demand (Bernard et al. 2020; Braarud 2020). The NASA-TLX is the most widely used workload measure (Hart 2006), with good internal consistency (Cronbach’s alpha ranging from 0.83 to 0.86; Braarud 2020) and very high convergent validity found with the Subjective Workload Assessment Technique (r = .98, p < .001) and the Workload Profile scale (r = .99, p < .001) (Rubio et al. 2004).

Composite scales were developed based on literature, with scales typically ranging from 1 (strongly disagree) to 5 (strongly agree). Perceived ease of use and perceived usefulness questions were based on the Technology Acceptance Model (Davis and Venkatesh 1996). Questions asked of participants regarding visual and physical experience during task completion are listed in Table 1.

Table 1. Composite scale survey questions completed for the electrical assembly and cobot-mediated inspection task.

AR-HMD composite questions	IWS composite questions
Stem: When completing the assembly task with the Hololens….	Stem: When completing the workflow using the system….
I could see the objects clearly	The steps I needed to take were clear
I was limited by the field of view	It was easy to move between assembly and inspection activities
My gestures were consistently recognised	It was easy to connect a call to communicate with a colleague
I experienced uncomfortable sensations (e.g. motion sickness)	The quality of the communication was acceptable
	I was satisfied with the image quality of the cobot camera
	I felt safe to be working near the moving cobot

Qualitative data examined perceptions of overall task experience, benefits, and barriers to uptake and general comments, typically expressed in brief statements. Thematic analysis was undertaken by an experienced qualitative researcher but was not corroborated due to low data complexity.

Table 2 summarises the self-report survey scales that were rated by participants after the completion of the entire workflow.

Table 2. Evaluation of technology—perceptions post-completion of the electrical assembly and cobot-mediated inspection task.

Assessment measures	AR-HMD	IWS
Standard quantitative scales
System Usability Scales (SUS)	✓	✓
NASA Task Load Index (TLX)	✓
Composite scales
Perceived ease of use	✓	✓
Perceived usefulness	✓	✓
Visual and physical experience	✓	✓
Qualitative/free text
Overall task experience	✓	✓
Benefits compared to current practices	✓
Concerns using in workplace	✓
Barriers to uptake	✓
General comments	✓	✓

IBM SPSS software (Version 25) was used to compute descriptive and inferential statistics. Mean, median, and standard deviation (SD) results are provided for description to allow comparison with other studies and illustrate representative values within data sets. Frequency distributions are reported for a range of Likert scale responses. Paired-sample parametric (t-test) and non-parametric (Sign Test) tests were conducted for Likert scale responses to compare SUS ratings. Statistical significance results remained the same under each assessment and only parametric results are presented for simplicity.

2.5. Procedures

2.5.1. Data collection methods

An invitation email was prepared by the researchers and distributed by selected businesses (shipyard and the supply chain) to their workers. The email included information on the trial methods, location, duration, and level of experience required (at least one year in construction or manufacturing). Interested participants were advised their involvement was voluntary and were asked to respond interest directly to the research team. Before the research activity all participants were briefed on ethical procedures and had a familiarisation and practice session (15 min duration) on each technology device. Familiarisation included functions and modes of AR-HMD gesturing in navigation, initiating cobot inspection (via the AR-HMD), and the troubleshooting videocall. Once familiar, participants assembled eight wires in the electrical cabinet guided by digital work instructions, video, and holograms, visible via the AR-HMD. On completing the wiring task (15–20 min duration), participants initiated cobot inspection and received pass/fail results. One fail result was designed into the protocol, necessitating a videocall to the ‘supervisor’ to provide feedback enabling the participant to rectify the fault. The participant then re-inspected the assembly to end the task with a pass result (overall task duration around 45 min). On completion, participants undertook separate online surveys examining use of the AR-HMD and the IWS respectively, via laptop (20 min duration), concluding by a debrief with a researcher.

Researchers observed but did not intervene during the task. However, due to the novel nature of the tasks some participants requested clarification of information in the digital work instructions or technical support when they had trouble operating the system. Minimal assistance was provided to allow the task to proceed. Participants were informed there was no time limit to completing the task.

3. Results

3.1. System usability

Overall the integrated work system (IWS) (including using the AR-HMD to deliver digital work instructions, initiating the cobot task inspection, and remote troubleshooting with a supervisor) achieved a SUS rating of 79.2 (SD = 17.9; median = 85.0), indicative of a ‘good usability’ outcome (Lewis and Sauro 2018). Usability evaluations of the AR-HMD alone were significantly muted, achieving an overall score of 69.8 [SD = 13.9; median = 70.0; t(35) = −5.23, p < .01] and aligning with an ‘acceptable usability’ outcome (Lewis and Sauro 2018). The IWS was consistently perceived as more usable than the AR-HMD alone for all SUS items (see Figure 5). In particular, participants felt significantly more confident using the IWS [t(35) = −3.42, p < .01], found it easier to use [t(35) = −3.55, p < .01], easier to learn quickly [t(35) = −3.62, p < .01] with less pre-learning before use [t(35) = −2.51, p < .05] and less need for support from a technical person [t(35) = −2.27, p < .05] than the AR-HMD alone. Additionally, participants were significantly less likely to find inconsistencies in the IWS [t(35) = −3.09, p < .01] or consider it unnecessarily complex [t(35) = −2.35, p < .05], compared to the AR-HMD alone.

Figure 5. Mean SUS adjusted scores for the HoloLens AR-HMD and IWS (higher scores reflect better usability). (Error bars represent ±1 standard deviation).

Column graph comparing average (mean) Hololens 2 and integrated work system scores on the ten-item System Usability Scale (Error bars represent ±1 standard deviation).

3.2. Workload

Workload demands using the AR-HMD were low with an overall mean score of 25.8 (SD = 10.9, median = 23.3), though subscale scores showed variability (see Figure 6). Mental demand (mean = 37.1, median = 30.0) was identified as the most taxing, requiring slightly higher levels of perceptual and processing effort to meet task outcomes. The effort also exceeded an average rating of 30, but only marginally, and ratings for this dimension showed the most variation (SD = 20.3).

Figure 6. NASA-TLX ratings using the HoloLens AR-HMD by workload dimension. (Error bars represent ±1 standard deviation).

Column graph of average (mean) workload ratings for the HoloLens 2 using the six item NASA Task Load Index (Error bars represent ±1 standard deviation).

Participants identified potential sources of mental demand as gesturing methods, positioning and locking screens, deciding which information to prioritise, and understanding progress through the task. One participant summarised this, stating:

there were three different virtual screens I had to work between plus the virtual projection in the box. I had to focus on the visual clues in the graphics to identify the correct locations. It was easy to accidentally skip a step as the images only show the current step - I thought I had skipped one and went back.

On occasions gestures were not recognised by the AR-HMD due to slow or excessive hand movement and poor orientation. One participant highlighted: ‘gestures were an obstacle, windows followed you at times when you did not want them, clicking was difficult. I struggled to click and drag, and I had windows appear when I did not want them’. Screen buttons could be activated using eye gaze and focus, and participants often used a combination of techniques. Fit and clarity of view were issues for some participants, with one stating ‘the glasses aren’t perfectly clear, the slight glare in the glasses was distracting and made it hard to see - I ended up moving them out of the way at times’. In addition, AR-HMD performance was sometimes limited by environmental issues, such as fluctuating internet connectivity, leading to unintentional screen activation, or latency and freezing.

3.3. User experience

Other important user experience measures indicated levels of satisfaction using the respective technologies. For interaction with the AR-HMD (see Figure 7), a small proportion (8%) experienced uncomfortable sensations (disorientation) though this did not interrupt task completion. Around half (48%) of participants reported challenges with consistently recognising gestures using the AR-HMD, although this was disputed by 39%. Visual performance of the AR-HMD was generally favourable, however, 25% reported the field of view was limited, and 14% indicated some dissatisfaction with object clarity.

Figure 7. Satisfaction ratings (frequency %) of specific AR-HMD functionalities. Note: For ease of reading, cells containing <5% are not labelled.

Horizontal bar graph showing satisfaction with specific AR-HMD functionalities. Ratings on scale 1–5 from strongly disagree to strongly agree.

User satisfaction with indicators of workflow in the IWS are presented in Figure 8. Participants reported feeling safe working close to the moving cobot (97%), with comments supporting the value of the cobot in visual inspection, with one participant stating: ‘I loved how the robot scanned my work and I could see the report on any errors’. Satisfaction was high with the quality of the video call to the remote ‘supervisor’ with 92% agreeing to some extent that quality of communication using the technology was acceptable. The same proportion agreed to some extent that the call was easy to initiate: ‘Having video ability to talk through and see the faults was really helpful. The supervisor could see what I see, from a distance, and give me clear instructions’.

Figure 8. Satisfaction ratings (frequency %) of specific workflow aspects of the IWS. Note: For ease of reading, cells containing <5% are not labelled.

Horizontal bar graph showing user satisfaction in percentage frequency. Ratings on scale 1–5 from strongly disagree to strongly agree.

Participants found it easy to move between functions in the IWS (86% in agreement) and reported the steps to be taken were clear (88%). Participants commented ‘once you got used to using the lens it was intuitive and easy to use. The workflow was well presented’ and ‘I think the workflow was well put together and if I had longer to use the system, I would have been a lot better and more confident using the HoloLens’.

3.3.1. User satisfaction

Comments on satisfaction with the IWS point to potential improvements. Participants were generally dissatisfied by the cobot camera precision to successfully read tags. Individual wires required careful orientation in one direction, facing upright and not obscured. This frustration was summarised: ‘the cobot was very finicky reading the tags because they had to be in exactly the right position’ and ‘the inspection failed due to criteria required to enable the inspection, not the job itself - the cable tags were not quite aligned. This was annoying and could be frustrating in production environments’.

Other comments on IWS usability emphasised the need to better signpost progress and outcome goals while working through the task, with one participant suggesting: ‘an overview of the entire task at the beginning would provide helpful context. Explicitly stating that wires of the same colour might have different number codes would reduce rework’. Another participant identified the need to simplify navigation and provide greater consistency in interface design, stating: ‘navigating the Hololens ecosystem was the most difficult part of the task. Instructions (video and 3D model) made execution of work very clear and straightforward though simpler navigation and more consistency would make a big difference to system usability. Navigation was frequently cited as a challenge: ‘working the button gestures during the task was the most difficult. If SOME gestures could be reduced or replaced with other inputs - voice, even physical buttons or touchscreen, or the reliability increased, it might gain better acceptance’.

Perceived barriers to using the AR-HMD included cost and set up (n = 4), discomfort and stress (n = 3), initial learning and support requirements (n = 5), reduced safety through lower situational awareness in the work environment (n = 5), equipment damage due to work environment (n = 4), and reliable connectivity (especially working inside a ship; n = 3). For the IWS, barriers to uptake were the long and ‘finicky’ inspection process (n = 5) and the quality of user instructions and interface (n = 3).

4. Discussion

4.1. Usability and user experience

Our study created an IWS enabled by AR-HMD technology to test user experience in human-technology interaction in the workplace. Our IWS required basic skill levels, imposed minimal demands, and provided greater functionality. Its effectiveness is reflected in consistently high usability ratings across subscales for the IWS (a ‘good’ SUS score of 79.2) compared to the AR-HMD alone (an ‘acceptable’ SUS score of 69.8) (Lewis and Sauro 2018).

AR design incorporates multiple sources (virtual and real) and types (holograms, video, work instructions) of information that can impair usability by increasing perceptual and mental processing demands (Jeffri and Rambli 2021; Sauer, Sonderegger, and Schmutz 2020). However, research on the use of AR technology has rarely tested multi-dimensional work applications or been conducted in applied or industrial environments. We note an ‘acceptable’ level of usability (SUS = 70) was found on our AR-HMD only task reflecting levels reported by others using the Hololens to deliver instructions, including Helin et al. (2018) who assessed 14 astronauts changing an engine filter (SUS = 68); Bottani et al. (2021) who examined fault diagnosis with six workers in a bottling line (SUS = 71); and Wang, Li, and Freiheit (2022) who examined team-based assembly of a motor involving ten participants (SUS = 71). However, our IWS application demonstrated that the integration of additional task-related functionality and support resulted in improved usability ratings. While this requires further examination, it suggests complex ‘real world’ applications involving the inter-relationship between context, function and usability may be more usable than those involving discrete tasks (with reduced meaning).

Our analysis and interpretation of SUS scores are supported by other quantitative and qualitative data collected as part of the trial. We note most participants reported satisfaction using the AR-HMD (89%) and felt comfortable using the cobot (97%). Perceptions about using gestures to activate the AR-HMD were mixed (48%), whereas using the IWS to activate cobot inspection and videocall was viewed favourably (86%)—reflecting the different effort required by these tasks. Similarly, participants reported higher satisfaction moving between steps for the IWS (88%) compared to the AR-HMD alone (48%).

Satisfaction with the quality of the AR-HMD visual output was similar for field of view (72%) and clarity (75%). Satisfaction with the accuracy of the IWS cobot camera was moderate (69%) and reflected frustrations with false errors from its high level of precision and specificity. Slight misalignment of wire ID tags led to false errors, as the camera could not read the data. Our findings suggest that within context the AR-HMD provides a useful and versatile tool for integration of other technologies with little extra burden on the end-user.

4.2. Workload

The IWS design aimed to achieve a seamless interface between work tasks and technologies, though limited users’ ability to isolate the workload demands of the different technologies and tasks (AR-HMD, cobot, and video call) across the IWS. Accordingly, workload was assessed on the AR-HMD only, rather than ask participants to distinguish workload demand at different points of the IWS. As such, the IWS and the AR-HMD were not independent conditions.

Our paper has provided insights into the workload demands of AR technologies in general, however, we note other AR research using Hololens 2 produced similar findings. A study of 51 students investigated different AR modalities (hand gesture vs. voice) for workload and usability effects during a 10-min maintenance assembly task (Ariansyah et al. 2022). NASA-TLX results for combined animations and hand gestures produced overall workload (mean = 26.0) and mental workload (mean = 32.0) scores similar to our study. These findings indicate hand gesturing required increased visual effort, spatial processing, and manual responses, adding to the perceived workload. As with our research, participants reported brightness levels and occlusion by animation, combined with poor contrast and detail, indicating specificity of the industrial setting is important (Ariansyah et al. 2022).

Findings from controlled studies comparing AR instructions (via Hololens 2) with either paper-based or tablet instructions produced similar results to ours. Kolla, Sanchez, and Plapper (2021) studied manual gearbox assembly with 18 participants of unspecified background. Their 15-min task compared paper-based and AR digital work instructions, finding workload demands (NASA-TLX 29.1; SD = 10.7) similar to our research (25.8) and usability (SUS 76.3; SD = 16.3) compared with 69.8. The relatively small differences between the studies may be due to chance and small sample sizes, noting workload demand for both tasks was low, and usability acceptable.

Hololens 2 and paper instructions were compared in a study of 20 student participants completing a simple pump assembly task of around 17 min duration (Brice, Rafferty, and McLoone 2020). Overall workload demand was low (NASA-TLX 29.4; SD = 11.9), although frustration and effort subscales were rated as moderate, attributed to difficulty with gesturing and connectivity to achieve desired outcomes. System usability (SUS 79.5; SD = 16. 1) was higher, but also more variable than our study, consistent with a simple task of much shorter duration. For the IWS in our trial, system usability was 79.2, indicating the overall workflow was coherent and achieved acceptable levels of demand.

Higher levels of demand, error execution, and frustration are expected during the skill acquisition gained through early use. Allocating time for training and practice, initially in short periods (e.g. 30 min) with frequent breaks, will facilitate successful technology adoption (Cometti et al. 2018). Findings highlight that use case selection must consider the social and technical system (e.g. complex relationships between hardware, software, and human characteristics), given the effect of user skill level and various functionalities (e.g. spatial mapping and eye tracking) (Seeliger, Netland, and Feuerriegel 2022).

4.3. Contribution of the study

In their review of AR technology over a decade, Bottani and Vignali (2019) highlight the trajectory of knowledge development. New knowledge in technology domains first emerges as technical papers, then progresses towards user studies and conceptual papers as capabilities and evidence increase. Our study reports an early trial of AR-HMD integrating digital work instructions, cobotic visual inspection, and video troubleshooting to complete an electrical assembly task in an industrial setting.

Integrating multiple technologies to develop smart work systems is an emerging area of research. AR-HMD research has also grown rapidly in the last five years, though few studies are applied in industrial settings, and few have examined user experience of advanced technologies in the work environment. Related studies have not used controls, report small sample sizes, and limited usability, workload, and performance data, their findings recommend improvements in interface and task design. Studies of remote collaboration also remain sparse, mainly outlining technical proof of capability with little participant or performance data. Understanding applications and user experience associated with implementing advanced technologies in workplaces is increasingly needed in countries, like Australia, with chronic skilled labour shortages and/or high labour costs. We note that all participants successfully completed our trial with little or no training, potentially reducing a barrier from skills shortages in production environments (Ariansyah et al. 2022).

Our study brings together EHF and advanced technology research in an industrial setting using Hololens 2 to integrate technologies and its remote collaboration capability to enable joint problem-solving. The findings support AR-HMD viability as vehicles for integrating technologies to increase functionality with little additional burden on end-users. Further, our trial supports the potential for tasks of higher complexity and scale (as in shipbuilding) to be automatically inspected using cobotics, replacing laborious manual inspection.

The study is limited by the lack of a control group for direct comparisons of performance outcomes, constraining the generalisability of results. However, paper-based work instructions could not adequately control for the complexity of the multi-faceted integrated design (digital work instructions, cobot activation, and remote assistance) of our research. Evidence that AR integrated approaches for complex tasks outperform paper-based work instructions based on errors and completion times (e.g. Dalle Mura and Dini 2021; Uva et al. 2018) supported our decision.

Our study could be enhanced by measuring performance time and errors to further quantify metrics for comparison across studies. Eye tracking technology could have enabled analysis of focus times and patterns, contributing to understanding of mental workload and usability findings regarding interface design, although it may have added an additional burden to the task. While our study task was longer (45 min) than most used in comparable studies, electrical assembly task durations found in industry are variable and often longer. Strengths of our study include real worker participants, real work environment, and relatively large sample size of 36 participants.

4.4. Implications and future directions

The H-T-O model depicts the system of interactions involved in technology use and acceptance. Our data supports the H-T-O model, where the H-T level represents the fit between human capacities and technology interface design. Better fit is associated with more positive usability scores, with the IWS simplified interface performing better than AR-HMD alone. At the H-O level, matching task-technology demands to reduce workload is associated with positive overall performance and organisational outcomes. At the T-O level, matching technology to organisational culture and processes is associated with positive user experience, where it was more satisfying using the IWS compared to AR-HMD alone.

Multiple factors influence effective usability, including strategic use of colour, placement of task information, and progress indicators (Neb et al. 2021). Our participants emphasised the importance of colour contrast in distinguishing cable positioning, highlighting the need for accessible guidelines to improve interface design, also supported by other research (e.g. Ariansyah et al. 2022; Evans et al. 2017; Hoover 2018). Selecting gesturing modes, presenting high-fidelity models, matching menu types, consistent mapping between colour coding, and matching to task and use case requirements are features requiring improvement. In designing for interaction continuity, Nee et al. (2012) emphasise three types of synchronisation—between media, device, and task. Our study suggests improved media presentation is likely to assist accurate tracking, reduced errors, increased responsiveness, and user acceptance.

We found the overall workload using the AR-HMD was acceptable. Mental demands were higher than desirable, but were likely due to low familiarity with the AR-HMD, given most participants had no previous exposure. Simplifying authoring conventions to promote consistency and reduce errors (Dalle Mura and Dini 2021) could improve mental demands, as could task and user skill analysis in the workplace (Radkowski, Herrema, and Oliver 2015). We found holograms are most effective for reducing mental effort, fatigue, and errors when positioned close to the related physical element and designed to minimise task switching between the virtual and physical elements. Well-designed, intuitive interfaces can minimise cognitive load and be more satisfying to use, promoting uptake, a goal of our current program of research in shipbuilding. Suggested device design improvements included greater hardware durability and ergonomics for the field of view (i.e. >100°), comfort, weight distribution, adjustability of lens position, and compatibility with industrial personal protective equipment (de Souza Cardoso, Mariano, and Zorzal 2020).

The H-T-O model is valuable for translating knowledge to heavy industries beyond shipbuilding (e.g. construction, fabrication, mining, oil and gas), seeking to improve productivity and competitiveness through technology adoption. Technology must assist in achieving functional goals, including ease of assembly, monitoring, inspection, and remote troubleshooting, which are largely functions of the H-O level and influenced by the H-T level. Success criteria specify broader outcomes where end-user satisfaction will be critical for successful technology adoption. These criteria are largely a function of the T-O level and include communication, flexibility, integration, safety and security (Wang, Li, and Freiheit 2022).

Our findings highlight the limited availability of comprehensive usability studies with consistent measures to compare cohorts, tasks, and technologies. Future research should examine occupational groups in industrial contexts to provide evidence of technology suitability in-situ (Zigart and Schlund 2020), as well as assessing technology durability and reliability (Szajna et al. 2020) to realise potential competitive advantages. Larger sample sizes with tasks of longer durations in high fidelity settings, supported by cost-benefit analyses are needed to move beyond proof of capability and inform adoption. Studies including quantitative data supported by qualitative data will contextualise user experience and identify barriers to adoption from human-technology-organisation interactions.

5. Conclusions

Advanced technology adoption promises to transform manufacturing, supporting greater flexibility, agility, and efficiencies through digitally connected processes. Research on AR technologies is rapidly expanding, along with interest in industry adoption, with far-reaching consequences for re-imagining human work. Consistent with the focus of EHF, people remain central in the work of the future and with technology used to augment their capabilities to achieve consistent quality and enhanced safety in product and service delivery. Usability remains a central focus beginning with the human-technology interface which is entrenched within the broader sociotechnical system linking organisation, human, and technology, with critical implications for sustainable adoption. Realising this potential requires testing new applications of new technologies and using systematic approaches built on usability and human-centric principles. The application of EHF principles supports the development of design guidelines for interfaces, products, processes, and jobs, improves technology performance, and enhances human skill acquisition, providing a competitive advantage for business.

Our study adds to the emerging knowledge base on AR-HMD applications, in the workplace, and specifically in a heavy industry setting. Findings highlight the potential of the AR-HMD to provide a vehicle for integrating advanced technologies to support greater functionality with minimal additional burden on end-users. Nonetheless, technical design issues (e.g. interface design, device durability and reliability, and connectivity), require resolution before uptake can be routinely applied in manufacturing operations (Park, Bokijonov, and Choi 2021). Despite these broader challenges, our findings underscore generally high levels of technology acceptance and motivation in an industrial workforce to explore technologies, with the potential to enrich work, promote task variety and learning opportunities.

Acknowledgements

This study is one part of a broader project funded by the Australian Department of Industry, Science Energy and Resources (Innovative Manufacturing CRC) in collaboration with BAE Systems Australia Maritime, which aims to de-risk and accelerate the uptake and diffusion of Industry 4.0 technology into the shipyard and its manufacturing supply chain to promote sovereign capability.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was funded by the Department of Industry, Science Energy and Resources (Innovative Manufacturing CRC) in collaboration with BAE Systems Maritime Australia.

Notes

1 A commercially available AR-HMD developed and manufactured by Microsoft powered by Windows Holographic OS with Dynamics Guides 365 software.

2 Universal robot 10e cobot fitted with 2D IFM machine vision camera.