Review of Digital Twin-based Interaction in Smart Manufacturing: Enabling Cyber-Physical Systems for Human-Machine Interaction

ABSTRACT

After its introduction around 20 years ago, the Digital Twin (DT) approach has recently attracted much interest in shaping the next generation of manufacturing. In the last years, many definitions and descriptions of the DT have been published, examining different aspects of its implementation. This paper is the first to present an analysis on the integration and interaction of human and DT in smart manufacturing systems in form of a scoping review following the PRISMA-ScR methodology. It presents the current state of the art of DT-based human-machine interaction (HMI), its implications, and future research directions. Filtering from 278 publications over the last decade, the analysis includes 23 publications, all published from 2016 to 2020. The results show the predominant scenarios and applications of DT-based HMI and identify the current division of labor between human and DT. The paper concludes with an integration of these findings into a human-centered classification of DTs as well as future research directions.

KEYWORDS:

• Human-computer interaction
• automated manufacturing systems
• collaborative manufacturing
• production planning
• robot control
• digital twin
• human-robot collaboration

1. Introduction

The idea of the Digital Twin (DT) witnessed a steep rise in popularity over the last years (Enders and Hoßbach 2019). The initial idea dates back to 2002 as a concept for Product Lifecycle Management, enhancing each physical system with a virtual representation combining all information about the original system published by Grieves and Vickers (2017). The expectations in the DT sparked research, bringing this concept into different fields of application such as manufacturing (Barricelli, Casiraghi, and Fogli 2019; Fuller et al. 2020) or aerospace (Barricelli, Casiraghi, and Fogli 2019; Glaessgen and Stargel 2012).

Due to the parallel development of DTs in a wide range of application fields, Jones et al. (2020) identified a multitude of definitions for the DT. In the context of production Negri, Fumagalli, and Macchi (2017) describe the DT as ‘[…] a virtual representation of a production system that is able to run on different simulation disciplines that is characterized by the synchronization between the virtual and real system […]’. This synchronization and integration of the digital thread into the object is an eminent aspect of the DT, similar to the usage of telemetry-data in other fields (Modoni, Sacco, and Terkaj 2016). Kritzinger et al. (2018) propose a classification and review of DT applications based on automatic or manual data flow, aggregating this to the classification of Digital Models (DM, both manual), Digital Shadows (DS, automatic update of virtual object), and Digital Twins (automatic bi-directional).

The expectations in the DT generated an immense amount of corresponding publications, complicating an overview of this field. In December 2017, Kritzinger et al. (2018) identified 34 publications (11 applications, 23 concepts) directed at the implementation of the DT idea, which rose to 87 applications and 77 concepts identified by Enders and Hoßbach (2019) in November 2018. Both reviews focus on the application of the DT concept, analyzing the general implementation. In recent time, different authors propose the DT as a means to improve human-machine interaction (HMI) and support operators (Stark, Fresemann, and Lindow 2019; Wilhelm, Beinke, and Freitag 2020; Petzoldt et al. 2020; van der Valk et al. 2020). Despite the popularity of the DT and the identified research gap (Lu et al. 2020; Kwok et al. 2020; Liu et al. 2020), no similar survey on the integration of both DT and human into smart manufacturing systems was conducted yet (see Section 2).

The objective of this paper is to gather implementations of DT-based HMI in production and manufacturing and provide an analysis of the integration of the operator into DT applications. A scoping review was conducted to gain an overview of the research done in this field and identify both existing methods and implementations as well as research gaps. Because DT applications differ widely over the use-cases, a categorization of DT-based implementations of HMI are also of interest. Therefore, this paper answers the following research questions:

RQ 1: In what scenarios in manufacturing is DT-based HMI applied?

RQ 2: How is the interaction between human and DT integrated into manufacturing systems that utilize a DT-based HMI?

The remainder of this paper is structured as follows: Section 2 presents an analysis of previously published reviews in the field of DTs, categorizing the current state of research and highlighting already identified gaps. Section 3 depicts the review method in accordance with the PRISMA-ScR statement (Tricco et al. 2018) Section 4 presents the results of the scoping review, classifying all documents based on scenario, the role of DT, and the role of human, as well as clustering specific combinations thereof. These results are critically discussed in Section 5, highlighting the potential of the DT for an improved HMI by suggesting an improved classification of the integration of the human into cyber-physical systems. Section 6 concludes the research, presenting its implications and future research directions.

2. Related works

Similar research was published in form of eleven review publications with respect to definitions, characteristics, applications, and technologies in the field of DT. All review publications found were published recently between 2017 and 2020, and are discussed more closely in the following, where the discussion point are ordered by the specifications of Table 1.

Table 1. Categorization of related review publications

Review	Definitions	Characteristics	Applications	Technologies
Barricelli, Casiraghi, and Fogli (2019)	x	x	x
Cimino, Negri, and Fumagalli (2019)			x
Enders and Hoßbach (2019)	x		x
Fuller et al. (2020)	x		x	x
Jones et al. (2020)	x	x
Kritzinger et al. (2018)			x
Liu et al. (2020)	x		x	x
Negri, Fumagalli, and Macchi (2017)	x
van der Valk et al. (2020)		x
Tao et al. (2019a)		x
Tao et al. (2019b)			x	x

In scientific literature, various definitions for DTs were proposed. The reviews by Negri, Fumagalli, and Macchi (2017), Barricelli, Casiraghi, and Fogli (2019), Enders and Hoßbach (2019), Liu et al. (2020) identify 16, 29, 10, and 21 different definitions, respectively. Negri, Fumagalli, and Macchi (2017) conclude that the DT’s role within Industry 4.0 is to predict and optimize the production system’s behavior, but without considering the interaction with humans. Several review publications study current applications for DTs. Tao et al. (2019b) consider the general industry, whereas most reviews focus on manufacturing (Barricelli, Casiraghi, and Fogli 2019; Enders and Hoßbach 2019; Cimino, Negri, and Fumagalli 2019; Fuller et al. 2020; Liu et al. 2020). Other domains are aerospace (Barricelli, Casiraghi, and Fogli 2019; Enders and Hoßbach 2019), healthcare (Barricelli, Casiraghi, and Fogli 2019; Fuller et al. 2020), and smart cities (Fuller et al. 2020). Cimino, Negri, and Fumagalli (2019) found missing implementation aspects in literature, while Tao et al. (2019b) identified a research gap in operational control by using DTs to propose new or adapted control plans based on operational conditions.

With respect to characteristics, many studies agree on the bi-directional communication between the physical and digital object as an essential characteristic of DTs (van der Valk et al. 2020; Barricelli, Casiraghi, and Fogli 2019; Kritzinger et al. 2018; Cimino, Negri, and Fumagalli 2019; Fuller et al. 2020). This need was introduced by Kritzinger et al. (2018) based on a review of literature later than 2014. Depending on either manual or automatic data flow in both directions, Kritzinger et al. (2018) suggest a distinction between Digital Model, Digital Shadow, or Digital Twin as shown in Figure 1.

Figure 1. Classification of (a) Digital model, (b) Digital shadow and (c) Digital twin by level of data flow integration (Kritzinger et al. 2018)

van der Valk et al. (2020) investigate the characteristics of DTs more comprehensively and propose a taxonomy based on a review of 233 papers. According to them, both a machine-to-machine interface for a bi-directional data link as well as a human-machine interface are important DT components. Similarly, Barricelli, Casiraghi, and Fogli (2019) also state the importance of a bi-directional, continuous data exchange based on their review of 75 papers. They extend this by the need to design such communication interfaces for three communication processes: between physical and virtual twin, between different DTs in the environment, and between the DT and interacting humans, requiring appropriate user interfaces. Further, they distinguish between two types of DTs that are linked either to an object or to a human being due to a different way of communication between the twins. While the former can exploit continuous real-time connection with its twin, the latter is connected with the human through wearable or external sensors, not necessarily guaranteeing a seamless connection and a high throughput (Barricelli, Casiraghi, and Fogli 2019). These two characteristic dimensions, i.e. firstly, bi-directional data flow and secondly, interfaces for human interaction, are also included in the 8-dimension model of the DT, proposed by Stark, Fresemann, and Lindow (2019). They consider the dimensions integration breadth, connectivity modes, update frequency, CPS intelligence, simulation capabilities, digital model richness, human interaction, and product life cycle to be essential for smart manufacturing. These dimensions were validated in related overview studies, e.g. by van der Valk et al. (2020) and Jones et al. (2020), also independently identifying eight aspects and corresponding characteristics for DTs.

Concerning future research challenges, Barricelli, Casiraghi, and Fogli (2019) highlight the need for human–work interaction design and end-user development methods and tools that allow operators to act as professionals without requiring knowledge in traditional software engineering techniques. Similarly, Liu et al. (2020) also state that human factors in the digital twin of manufacturing systems are not yet sufficiently considered.

Therefore, as introduced above, this paper differs from the related work as it focuses on user interaction with the DT. Specifically, it investigates in which scenarios the DT is utilized for human-machine interaction (RQ 1), and how the DT-based interaction is integrated and what role both humans and DTs adopt (RQ 2). As shown by the discussion of related reviews, these aspects have never been investigated by any survey yet. However, it is to highlight that the user interaction with a DT, forming an abstraction layer to the physical object for monitoring, prediction, and control of systems, is a key enabler for human-machine interaction in future Industry 4.0 systems and defines the way how operators interact with manufacturing systems.

3. Scoping review methodology

3.1. Search strategy

To answer the aforementioned questions, a scoping review based on the PRISMA-ScR statement was performed (Tricco et al. 2018). The search was conducted in the scopus database, on IEEExplore, and ScienceDirect at the end of January 2020 without limiting the scope of the review with respect to publication year or by excluding specific publication types or statuses. Based on research question RQ 1, the search strategy was designed following the CHIP framework for qualitative studies Shaw (2010). The developed items are depicted in Table 2. The search strings are given in Table 3.

Table 2. Search items developed using the CHIP framework

Area	Search items
Context	manufacturing OR production
How	prototypes OR case-studies
Issues	digital twin
Population	human operator OR user

Table 3. Search strings of the performed searches

Database	Search string
Scopus	TITLE-ABS-KEY (”digital twin” AND (manufactur* OR production) AND (user OR operator OR human))
IEEExplore	”All Metadata”:”digital Twin” AND (manufactur * OR production) AND (user OR operator OR human)
ScienceDirect	Title, abstract, keywords: ”digital twin ” AND (manufacturing OR manufacture OR production) AND (user OR operator OR human)

3.2. Inclusion criteria

After a screen for duplicates, all full-length publications with a presentation of new technological results were included (e.g. articles, conference contributions with a full report). Commentaries, abstracts, overviews, and reviews were not considered. In the first screening, all publications were evaluated based on title and abstract, and irrelevant documents removed. Publications that are not in the scope of manufacturing or production (e.g. operations research) were removed. Since many publications use the term Digital Twin in a broad sense, publications focusing on specific models for individual problems (e.g. heat flow in additive manufacturing) or training and evaluation of machine learning algorithms (e.g. imitation learning, transfer learning) were removed also. The review includes only publications in English or German. The screening on the first level was performed by one researcher (JW) and independently validated by the other two researchers (CP, TB), each assessing half the documents. Documents that were not unanimously screened out were included in the analysis.

3.3. Study selection process

On the second level of screening, data of the full-texts was extracted into three categories.

• Level of HMI based on the interaction between human and DT (bi-directional interaction (2D), uni-directional (1D), no interaction)

• Content type (case-study, prototype or tech demo, simulation study or concept)

• DT type by data transfer according to Kritzinger et al. (2018) (DM, DS, DT)

The first category was used for eligibility attribution, whereas the latter two categories are used for additional analysis. This set of data includes all documents that describe an application with a virtual representation fulfilling the definition above by Negri, Fumagalli, and Macchi (2017) and address interaction involving the DT, either one-directional with the DT integrating the human (1D) or bi-interaction with both human and DT active (2D).

The screening criteria were developed in a discussion among three researchers (JW, CP, TB). Level two screening was performed by one researcher (JW) and individually validated by two researchers (CP, TB), each assessing half the documents. Disagreements or uncertainties regarding eligibility were resolved through discussion until consensus was reached among the three reviewers. For documents that could be attributed to multiple categories, the assignment was based on the dominant category.

3.4. Data extraction

To understand the scenario in which DT and human operator are working and its implementation, data on the following categories was extracted

• The real world object of which a Digital Twin was created,

• The domain of the process (e.g. collaborative robotics),

• The specific task(s) of a human operator altered due to the usage of the Digital Twin (incl. processing), and

• The specific task(s) performed by a human operator.

The charting items were jointly developed (JW, CP, TB) to determine which features to extract. Three reviewers (JW, CP, TB) independently analyzed all eligible documents. The results were discussed continuously and disagreements resolved by discussion of the individual documents.

3.5. Synthesis

The charting data described above was aggregated in Microsoft Excel and visualized using Python. Similar data from the charting was clustered to deduce simplified categories. In a qualitative analysis each cluster was analyzed individually, as well as their relations. Within documents describing a DT-based HMI, four different categorical scenarios were identified, i.e. production planning and control (PPC), collaborative robotics, teleoperation, and maintenance. The category formation is based on the analyzed papers and the categories derived by Liu et al. (2020) with a focus on the twinned object, the DT-application, and the use-case description. On this basis, the commonalities were identified by means of cross-comparisons and the four categories were developed. These categories were chosen based on the type of activity supported by the DT and its context. Therefore, they represent the main application scenarios of the DT, not necessarily of the entire production plant (e.g. the display of live machine data is categorized as maintenance, since the data primarily supports servicing personnel). Each document is assigned to one of these categories exclusively.

To identify the means of DT-based HMI and the ways the DT supports the operator, the tasks the DT and the human operator perform were classified. The roles of the DT are structured with respect to the information processing of autonomous systems (inform, support, decide, act) according to Sheridan and Parasuraman (2005). The roles of the human are derived from classifications in expert systems, based on the way the human operator hands over control to the DT (operation, decision-making, supervision, and implementation) according toEndsley and Kaber (1999), and Cardin (2019). Allocation to these categories is not exclusive, since the roles of the operator or the DT can stretch across multiple tasks.

4. Results

The initial search lead to 278 records (after removal of duplicates), of which the authors accessed 124 full-texts and categorized 23 documents. The first documents identified were published in 2014 with an upwards trend thereafter. The first eligible document was published in 2016, with a slight increase in numbers until 2019 (Figure 2).

Figure 2. Publications in each screening stage by year

About a fifth of all accessed full-texts cover HMI (23 of 124, 19%). This excludes systems in which the human only performs data entry or is twinned for offline simulations. In 13 cases, the interaction is bi-directional (15%), whereas ten cases are only including the operator as a recipient (e.g. the human is only informed by the system).

The HMI-related publications present eleven implementations (of which five are case-studies and seven are prototypes or demonstrators) and 12 concepts of a DT-based HMI. The entire scoping procedure according to PRISMA-ScR is shown in Figure 3. The charted data is presented in Table 4.

Figure 3. PRISMA flow diagram for the conducted scoping review

Table 4. Result of the charting process

Ref.	Year	Type of work	DT Type	HMI Level	Description	Scenario	DT Object	Role of DT^a				Role of Human^b
								In	S	D	A	O	D	S	Im
Stark, Fresemann, and Lindow (2019)	2019	Concept	DT	2D	Assembly within a CPPS	Collaborative Robotics	Smart Factory Cell (CPPS) incl. Machines and robotic arms			x	x				x
Longo, Nicoletti, and Padovano (2019)	2019	Case-study	DS	1D	Creation of maintenance work plans matching production orders	PPC	Tubeproduction (CPPS) incl. Machines	x		x		x
Park et al. (2019b)	2019	Case-study	DS	1D	Optimization of a dyeing and finishing process for energy reduction	PPC	Dyeing process line (CPPS incl. Machines)			x		x			x
Braglia et al. (2019)	2019	Case-study	DT	1D	Route planning (optimization) for forklifts in a warehouse	PPC	Warehouse			x					x
Kang, Chun, and Kim (2019)	2019	Case-study	DT	1D	Production planning and control in a motor assembly line	PPC	Motor Assembly Line (CPPS incl. Transport robot)	x		x		x
Qamsane et al. (2019)	2019	Concept	DS	1D	Rerouting in a smart shop floor	PPC	Factory Floor (CPPLS) incl. Machines			x		x
Weyer et al. (2016)	2016	Concept	DS	1D	Virtual Commissioning in automotive production	PPC	CPPS incl. Machines		x				x
Graessler and Poehler (2017)	2017	Concept	DM	1D	Scheduling and allocation of human resources in a CPPS	PPC	Human Operator			x					x
Catarci et al. (2019) and Lu and Xu (2019)	2019	Concept	DT	2D	Manufacturing of card boxes with multiple companies involved	PPC	Card box Manufacturing (CPPS incl. Machines)		x	x		x	x
Lu and Xu (2019)	2019	Concept	DS	1D	On-demand manufacturing (machine as a service) including the real-time monitoring of equipment	PPC	Machine (CNC roll forming)	x	x				x
Padovano et al. (2018)	2018	Case-study	DS	1D	Reduction of setup-up times in a batch production	Maintenance	CPPS (package manufacturer)	x	x					x
Vathoopan et al. (2018)	2018	Concept	DS	2D	Component simulation for fault ascription	Maintenance	Machine component	x	x					x
Liu et al. (2019a)	2019	Demo	DS	1D	Monitoring and simulation of the milling process	Maintenance	Milling machine	x	x					x
Zhu, Liu, and Xu (2019)	2019	Demo	DS	2D	Monitoring of the milling process	Maintenance	Milling machine	x						x
Josifovska, Yigitbas, and Engels (2019)	2019	Concept	DT	2D	Assembly of cabinets	Maintenance	Environment		x		x			x
Oyekan et al. (2019)	2019	Demo	DS	2D	Safety improvements by simulation of collaborative strategies	Collaborative Robotics	Robotic Arm (virtual)	x						x
Bilberg and Malik (2019)	2019	Concept	DT	2D	Cooperative assembly (human and robot arm)	Collaborative Robotics	Assembly cell (CPPS) incl. Machines, humans, robotic arms, and environment			x	x				x
Segura et al. (2020)	2020	Concept	DT	2D	Training for a collaborative working scenario	Collaborative Robotics	Robotic Arm	x						x
Malik, Masood, and Bilberg (2020)	2020	Concept	DS	2D	Analysis and optimization of workplace layout during design phase	Collaborative Robotics	CPPS (collaborative assembly cell)	x						x
Kuts et al. (2019)	2019	Demo	DT	2D	Control of a robotic arm in education	Teleoperation	Robotic Arm	x	x	x	x		x
Tammaro et al. (2017)	2017	Demo	DT	2D	Web-based teleoperation	Teleoperation	Robotic Arm	x					x
Ardanza et al. (2019)	2019	Demo	DS	2D	Visual Scanning	Teleoperation	Robotic Arm	x				x
Horváth and Erdős (2017)	2017	Demo	DT	2D	Control of a robot in a shared workspace	Teleoperation	CPPS incl. Humans and robotic arms			x	x		x

^aRoles of DT: Inform, support, decide, act. ^bRoles of human: Operate, decide, support, implement.

4.1. RQ 1: Scenarios in manufacturing applying DT-based HMI

The authors identified four different categorical scenarios in which DT-based HMI is applied. With a total of nine, 39% of the publications are in the field of PPC, which is the predominant category both in number of case-studies as well as total references (Figure 4). Here, the DT supports along the decision-making process. Four case-studies address different aspects of PPC, especially the creation of production plans (Longo, Nicoletti, and Padovano 2019; Park et al. 2019a), and the routing of transport vehicles (Braglia et al. 2019; Kang, Chun, and Kim 2019). The remaining publications are concepts for the optimization of different aspects, e.g. production rerouting (Qamsane et al. 2019), virtual commissioning in cases of factory reconfigurations (Weyer et al. 2016), or creation of work plans for human operators (Graessler and Poehler 2017). In one scenario, such optimizations are driven by user-given goal functions (Catarci et al. 2019). Lu and Xu (2019) propose the DT for the creation of new services due to an online monitoring and planning of remote production resources.

Figure 4. Comparison of uni-directional and bi-directional interaction in the scenarios with DT-based HMI

The maintenance category aggregates case-studies and demos on the information management of manufacturing machines (5/23, 22%). Padovano et al. (2018) present the case-study of a knowledge management to support the operator during machine configuration and setup. Liu et al. (2019a) present a machine monitoring system to display information in AR, which was enriched with a user interface for modification of data (Zhu, Liu, and Xu 2019). A similar machine information system based on physical models is suggested by Vathoopan et al. (2018). Josifovska, Yigitbas, and Engels (2019) propose a concept for a user-centered interface adaptation in maintenance tasks.

Scenarios labeled as collaborative robotics include both the design and planning of collaborative tasks as well as the implementation of such control (5/23, 22%). The focus is the safety of the operator, especially by DT-driven collision detection. Oyekan et al. (2019) present a prototype for a virtual safety assessment of collaborative workplaces in which the operator is integrated into a VR environment including a twin of the robotic arm. Concepts exist for trajectory planning in collaborative assembly stations (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019), training of operators in collaborative tasks (Segura et al. 2020), and virtual design and verification of workplaces (Malik, Masood, and Bilberg 2020).

Four demo-implementations use the DT to perform teleoperation of robotic arms (4/23, 17%). The user controls the movement of the robotic arm or its joints, supported by visualizations created by the DT (Kuts et al. 2019; Tammaro et al. 2017; Ardanza et al. 2019). Horváth and Erdős (2017) propose the usage of gestures, representing individual tasks to be performed, as the commands for the teleoperation.

As stated above, one- and bi-directional interaction between human and machine were analyzed separately. In the one-directional case, the human operator benefits from additional information provided by the DT, e.g. decision-making support based on simulations, or maintenance instructions. In the bi-directional case, user and DT interact reciprocally, with both parties reacting to new input. Figure 4 depicts a comparison of the uni-directional and bi-directional interaction in each scenario. Most interaction in the PPC scenarios is purely one-directional (8/9, 89%). DT-based simulations offer new insights into the process of PPC, with the operator verifying and/or implementing these insights. Exemplary, in the concept suggested by Catarci et al. (2019), the operator determines the goal-functions, specifying the direction the DT works towards.

In the bi-directional cases in the maintenance category, the operator again has the ability to control the direction the DT works into, compared to pure implementation tasks in the uni-directional cases. The bi-directional cases include the triggering of individual simulation tasks to localize the cause of fault (Vathoopan et al. 2018), the modification of usage data of a monitored machine (Zhu, Liu, and Xu 2019), and the control of UI parameters (Josifovska, Yigitbas, and Engels 2019). In the remaining scenarios (collaborative robotics and teleoperation), the interaction in each category is always bi-directional.

4.2. RQ 2: Methods of DT-based interaction

4.2.1. Role of the DT

Categorization of the role of the DT was performed along the stages of information processing according to Sheridan and Parasuraman (2005). Since all DTs include the process of Acquisition of information, this stage was neglected, while stage two Analysis and display of information was split into Information and Support. Information collects all tasks in which the DT gathers data and represents them to the user or operator in a condensed fashion. Support includes further data analysis and optimization to support the operator during the decision-making process. Systems ascribed to Decide also conduct analysis and optimization, but furthermore determine the measures to be implemented. Act implicates systems in which the DT controls the physical system and performs actions in the physical world. This categorization is non-exclusive, so one document can be subject to multiple categories at once. The assignment was performed individually for each category, only if the DT performed this task as a necessary step in HMI. E.g. even though all systems performing decision-making tasks have to analyze data, only systems that provide this analyzed data to a human are categorized as such. Most publications use the DT to provide information to the human operator (13/23, 57%) and there are no case-studies on DT-based acting yet (Figure 5). Most case-studies are published with the DT performing decision-making, especially on production strategies in the field of PPC (Figure 6).

Figure 5. Role of the DT by publication type

Figure 6. Role of the DT in each scenario (PPC: Production Planning and Control. Mai: Maintenance. Col: Collaborative Robotics. Tel: Teleoperation.)

In more than half the documents (13/23, 57%), a main role of the DT is to provide additional information to the human. In many cases, this information is a pure visualization of the current state of the machine or specific parameters (Lu and Xu 2019; Tammaro et al. 2017; Ardanza et al. 2019; Oyekan et al. 2019; Malik, Masood, and Bilberg 2020; Zhu, Liu, and Xu 2019; Segura et al. 2020). The DT allows a view from the inside of the system, gathering all system data in a unified and pre-processed manner, e.g. the display of the pose of the robot in a 3D frame. Some authors combine this information with supporting capabilities of the DT, allowing to display the operator estimated future behavior of the system (Kang, Chun, and Kim 2019; Kuts et al. 2019; Liu et al. 2019a; Longo, Nicoletti, and Padovano 2019) or present further information for maintenance tasks (Vathoopan et al. 2018; Padovano et al. 2018).

In 35% (8/23) of the contributions, the DT supports the operator in his or her task. Examples are found in all scenarios except collaborative robotics. Most DTs perform simulations to predict and optimize the behavior of the system, offering the operator further insight into complex contexts. This is mostly used in PPC (Catarci et al. 2019; Lu and Xu 2019; Weyer et al. 2016) and maintenance (Josifovska, Yigitbas, and Engels 2019), where also historical data is combined with predictive analysis to support the operator in maintenance tasks (Vathoopan et al. 2018; Padovano et al. 2018; Liu et al. 2019a). Kuts et al. (2019) implemented the DT to predict the behavior of a robotic arm to support operators while programming the arm. Josifovska, Yigitbas, and Engels (2019) present a concept for an adaptive user interface, combining information about the operator and the environment to adjust the interface to the operator’s needs.

Almost half (11/23, 48%) of the analyzed DTs in all scenarios except maintenance offer decision-making capabilities. These systems again use simulations for optimization and prediction of future behavior, and additionally decide about the implementation of such strategies. In the context of PPC, this mainly involves the creation of work or production plans, with the plan verified Kang, Chun, and Kim (2019); Park et al. (2019a); Catarci et al. (2019); Qamsane et al. (2019); Longo, Nicoletti, and Padovano (2019) or implemented by human operators (Braglia et al. 2019; Graessler and Poehler 2017). Graessler and Poehler (2017) propose a concept in which the human operator is the object twinned by the DT, allowing the matching of specific tasks to available and capable operators. This is similar to the task of the DT in collaborative robotics, where two concepts implement a DT-based assembly plan generation implemented by a human operator (Stark, Fresemann, and Lindow 2019; Malik, Masood, and Bilberg 2020). The potential for a joint decision-making is shown by Kuts et al. (2019) and Horváth and Erdős (2017). They present two demonstrators in which the DT can disrupt the remote control of a human operator and take over control of the robot to avoid collisions.

DTs actuating and controlling the system directly are implemented in teleoperation and collaborative robotics. In all scenarios, the DT has direct control over the behavior of the robotic arm. In case of teleoperation, this is already implemented in the aforementioned demonstrators (Kuts et al. 2019; Horváth and Erdős 2017), whereas in collaborative robots it is currently evaluated in simulation studies, in which the DT controls robotic arms in collaborative assembly cells (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019).

4.2.2. Role of the human

The tasks of the human were seperated along the way the operator hands over control to the DT, namely Operation, Decision-making, Supervision, and Implementation. This order is characterized by a decreasing decision authority of the human operator; from performing all tasks independently, to the determination and monitoring of decisions, to the mere implementation of actions decided by the DT. This is similar to the task separation in the context of computer expert systems (Endsley and Kaber 1999) and general CPS (Cardin 2019). Publications on operation are most prevalent, with the remaining tasks being of approximately equal number. Most case-studies focus on systems with high autonomy (supervision and implementation, Figure 7). The studies address the different scenarios equally, while the roles of the operator are different according to the scenarios. Particularly in maintenance, the human only performs operational tasks, whereas in other scenarios he or she has at least two different roles (Figure 8).

Figure 7. Role of the human by publication type

Figure 8. Role of the human in each scenario (PPC: Production Planning and Control. Mai: Maintenance. Col: Collaborative Robotics. Tel: Teleoperation)

In the case of operation (8/23, 35%), the human operator performs his or her normal tasks at full responsibility, even though he or she might be assisted by the DT in form of additional information. Applications with this kind of cooperation are found exclusively in maintenance and collaborative robotics with five and three publications respectively. Padovano et al. (2018) present the only case-study in form of a knowledge-app offering the personnel additional information about the serviced machine. Two publications introduce a prototype for the visualization of live-data for simplified machine analysis (Liu et al. 2019b; Zhu, Liu, and Xu 2019). On a conceptual stage, the DT supports the operator by simplifying the fault analysis (Vathoopan et al. 2018) or adapting the GUI to his or hers current needs (Josifovska, Yigitbas, and Engels 2019). Most publications in collaborative robotics are proposing concepts to improve collaborating scenarios. The DT supports in training (Segura et al. 2020), layout optimization (Malik, Masood, and Bilberg 2020), and during safety assessments (Oyekan et al. 2019, demonstrated as prototype).

In decision-making (6/23, 26%), the human controls the decisions for the DT and the attached CPS, and the DT supports the operator with additional analysis, offering insight into one or multiple strategies. One publication by Catarci et al. (2019) in the field of PPC is counted as both decision-making and supervision, because the DT interacts with two different types of employees. In PPC, decision-making capabilities of the human operator are often strategic, e.g. deciding about the goal function of the scheduling optimization (Catarci et al. 2019), purchase of manufacturing services (Lu and Xu 2019), and factory reconfiguration Weyer et al. (2016). In the field of teleoperation, the human controls the behavior and the motions of the robot, either by gesture control (Horváth and Erdős 2017), via a web interface (Tammaro et al. 2017) or by remote programming (Kuts et al. 2019).

In case of supervision (6/23, 26%), the operator is not responsible for the decision but only for a verification. This can also happen in retrospect, with the operator stopping the system. A sharp distinction between decision-making and verification was not always possible, because the systems were often not described in sufficient detail. The verification of plans is the standard task of human supervisors. At this, the DT creates an optimal strategy including the predicted outcome for production or transport, which the human only verifies before it is implemented on the real system. This is shown in three case-studies (Kang, Chun, and Kim 2019; Park et al. 2019a; Longo, Nicoletti, and Padovano 2019) and two concepts (Catarci et al. 2019; Qamsane et al. 2019). The demonstrator for teleoperation by Ardanza et al. (2019) includes an operating condition in which the operator is only supervising the system.

The role of implementation (5/23, 22%) is the physical execution of tasks, which were chosen and verified by the DT, e.g. assembling a component according to given instructions. Besides concepts for manual assembly task selection (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019), and operator scheduling (Graessler and Poehler 2017), two case-studies feature the implementation of DT-based decisions, either for scheduling forklifts by Braglia et al. (2019) or reworking products by Park et al. (2019a).

4.2.3. Dependency of human role and DT role

Further insight can be generated by comparing the role of the human and the DT in the different scenarios identified. In general, a large cluster of publications in the area of human: (operate, decide) and DT: (inform, support) can be identified, suggesting that many DT applications are mainly supporting the operator during normal operation (Figure 9).

Figure 9. Dependency between role of the human and role of the DT

For some combinations of roles, human and DT are collaborating in a joint space. There are three publications, in which both the human and the DT are making decisions, and three publications in which human and robot are both performing some kind of action or operation. In the joint decision-making, human and robot are deciding on different levels of abstraction. The DT is deciding about standardized tasks (e.g. collision control), whereas the human performs ‘creative’ tasks (e.g. programming). In case of teleoperation, the human is deciding about the general behavior of the robotic arm (controlling the direct movement), with the DT as an additional controller for collision avoidance (Kuts et al. 2019; Horváth and Erdős 2017), while in the field of PPC, the human sets the high-level goal functions of optimization algorithms (Catarci et al. 2019).

Two concepts describe a joint assembly in the field of collaborative robotics, in which the human and a robotic arm are performing actions planned by the DT (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019). In the concept of Josifovska, Yigitbas, and Engels (2019), the DT performs actions mostly unrelated to the standard tasks of the human operator (e.g. adapting the brightness of the user interface). This analysis can be broken down further by evaluating the individual scenarios (Figure 10). In the field of PPC, the main task of the human is still decision-making, which is often the current main task of the human operator. Apart from ambiguities due to multiple roles of human and DT describe in a publication, a shift in responsibility is noticeable. The more sovereign the role of the DT becomes, the more the human becomes purely a tool for implementation. This is illustrated by diagonal from top left to bottom right in Figures 9 and 10.

Figure 10. Dependency between role of the human and role of the DT for different scenarios

In the field of PPC, the DT fulfils a supporting role, providing insight into the process and the effect of potential control signals. In most publications, the operator is either deciding based on simulations by the DT (Weyer et al. 2016) or supervising and verifying the decisions made by the DT. Examples for decisions made by the DT are production strategies (Kang, Chun, and Kim 2019; Park et al. 2019a; Catarci et al. 2019; Qamsane et al. 2019) and maintenance plans (Longo, Nicoletti, and Padovano 2019). In some cases the operator is also directly implementing scheduling decisions made by the DT, e.g. forklifts in a warehouse (Braglia et al. 2019), employees in a CPS (Graessler and Poehler 2017), and reworking products (Park et al. 2019a). In two cases, the DT is supplying information for decision-making tasks of a human, either in deciding about factory reconfigurations (Weyer et al. 2016) or purchasing on-demand manufacturing services (Lu and Xu 2019).

In maintenance, humans perform all operational tasks. The DT is offering support by depicting further knowledge about the process and the machine and freeing the operator from tedious tasks. The DT provides insight into the machine, either by pre-processing sensory information (Liu et al. 2019a; Zhu, Liu, and Xu 2019) or performing fault analysis simulations (Vathoopan et al. 2018), as well as additional maintenance documents (Padovano et al. 2018). The only case of an acting DT in maintenance is the user interface adaptation proposed by Josifovska, Yigitbas, and Engels (2019).

In collaborative robotics, human and DT are working alongside, both performing physical tasks. The DT manages the entire workflow, while the human operator implements tasks planned and allocated by the DT. Examples are manual assembly (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019) or the analysis of planned workplaces in an interaction with a collaborative robotic arm (Oyekan et al. 2019; Malik, Masood, and Bilberg 2020; Segura et al. 2020). The main purpose in this scenario is the enhancement of ergonomics and safety.

In teleoperation, the operator performs decision and supervision tasks. With the exception of the prototype by Tammaro et al. (2017), in which the DT is used purely for information purposes with control potentially to come, the operator is programming and controlling the system remotely. The DT supports the user by offering pre-processed information as well as performing standard tasks such as collision control.

5. Discussion

This scoping review presents the analysis of 278 unique documents published until January 2020 and identified 23 contributions addressing DT-based HMI. It focuses on English and German publications listed in SCOPUS, IEEExplore, or ScienceDirect. All eligible records were published from 2016 onwards, highlighting the increasing relevance of the DT for HMI. To answer the research questions on DT-based HMI implementations, four scenarios based on the object of the DT and its application were identified, not taking into account specific features or dimensions of the DT itself. After a discussion of the general findings, their further implications and integration into already developed systematics are presented.

5.1. General findings

Currently, there are four main scenarios with a DT-based integration of HMI. PPC is currently the predominant area, comprising 39% of all publications and 80% (4/5) of all case-studies. However, a one-directional interaction between human and DT, in which the DT is only informing the operator about the current state or made decisions, characterizes most contributions in this field.

The remaining three scenarios are distinguished by their combination of the roles of the human and the DT. Each area presents unique concepts and demonstrators. In the field of maintenance, the DT offers new ways of analyzing failures and supports the operator in his or her routine work. This is in stark contrast to teleoperation scenarios where the operator controls the DT and retains decision-making power. The DT creates additional insight into the system and oversees low-level tasks (e.g. collision detection). n scenarios with collaborative robotics, there are two dominant applications of the DT. First, the DT allows for a quick and cost-effective evaluation of the HMI during planning and design. Human operators interact with the DT instead of the physical robot, enabling easy safety assessments and early layout optimization and training. Second, during the operation stage, the DT takes full control of the CPS, controlling both the robotic arm, as well as scheduling the tasks of the human operator.

Besides the classical roles in the context of manufacturing, some contributions utilized the DT to create new roles and services. Instead of improving tasks and changing the way operators work, the DT boosts innovations by creating new ways of humans interacting with the DT. Examples are the offering of cloud-based production services directly to potential customers (Lu and Xu 2019) or, beyond the scope of manufacturing, monitoring the fit of personalized equipment (Zheng et al. 2018).

Yet, most of the identified documents exploit only a portion of the potential offered by the idea of the DT. Many contributions either limit the DT to a one-dimensional provider of information and decisions, especially in the field of PPC, or keep a static task division and fixed roles. Contrary to the idea of the DT to accompany the physical asset during all life-cycle stages and combine multiple data streams, the ways of human-DT interaction in most publications are very limited so far. Only a few contributions use the DT for cooperation or collaboration between human and CPS and enable user-centered support beyond a DT-based information processing and provision. Many case-studies and demonstrators solely display the technological aspect of a DT-based HMI, especially utilizing VR as a means of interaction. However, selected concepts already integrate this into broader applications. In particular, in the field of collaborative robotics, there are some ideas that use DT to improve human-machine collaboration (Stark, Fresemann, and Lindow 2019; Bilberg and Malik 2019). Some concepts in teleoperation and PPC allow for the joint control of tasks based on the distinction of low and high level decisions, offering new ways of HMI (Kuts et al. 2019; Catarci et al. 2019; Horváth and Erdős 2017).

A technological challenge is integrating the representation of a human into the DT. In some contributions, the human is already twinned to some extent, which will most likely increase in relevance in the near future. Thus far, almost all sensing of humans is limited to the analysis of their position and pose (Nikolakis et al. 2019). New concepts require the integration of additional information about the human, e.g. preferences and skill (Josifovska, Yigitbas, and Engels 2019; Graessler and Poehler 2017; Sun et al. 2020).

5.2. Implications for the integration of humans into DT applications

Based on this review on human interaction with DT-based systems, a distinction between unidirectional and bi-directional interaction between human operators and digital objects should be made. This distinction is similar to the data flow-based DT-classification proposed by Kritzinger et al. (2018). However, this schematic classification (see Figure 1) does not explicitly consider the human operator. Although the distinction between manual or automatic data flow inherently covers a certain extent of human interaction, this alone does not encompass all types of human interaction with DTs found in the literature review. Considering the data flow types from the perspective of human interaction more closely, manual data flow implies a connection between physical and digital object via the human operator, while automatic data flow represents a direct connection between physical and digital object. Based on this, Figure 1 can be extended to include the human operator, as shown in Figure 11 (solid lines present the original definition). In this case, the original direct manual data transfer is routed through the human operator, who explicitly performs this task. In case of automatic data flow, the operator is not involved in the original interaction.

Figure 11. Classification of a digital twin-based human-machine interaction, extending the digital twin classification proposed by .Kritzinger et al. (2018)

In various DS and DT applications studied, human operators interact with physical objects not only through the digital object, but also directly in the physical world (e.g. collaborative assembly) or observe the physical object directly (e.g. maintenance). A decoupling of the operator from the physical world through the DT has only been implemented to a very limited extent thus far. Therefore, the previous systematic was extended to include the human operator, who is connected to both the digital and the physical object (dashed lines in Figure 11). Moreover, as mentioned earlier and especially of relevance for DT applications, the immediate interaction between human operator and digital object can be either unidirectional (e.g. monitoring) or bidirectional (e.g. control), independent of the level of the DT.

6. Conclusion

This paper provides a systematic literature review on the integration and interaction of human and DT in smart manufacturing systems in form of a scoping review. The contributions were analyzed and categorized regarding their application domain, the role of the human, and the tasks of the DT. This classification helps to understand the current state of the art of DT-based HMI. Furthermore, a new classification describing the integration of humans into DT applications illustrates the potential of the DT for human-centered smart manufacturing.

Despite the vast interest in DT related applications, the integration of the operator into such systems is still at its infancy. Of the 278 analyzed documents, only 23 explicitly consider the integration of the human. The classification of the role of the human and the task of DT reveals scenario-specific characteristics. With the exception of production planning and control, each scenario has a distinct approach for DT-based HMI. In selected publications, the human and DT support each other in a collaborative manner, but most present a fixed division of labor. The DT enables human-centered design across all phases of the life cycle, which has yet to be implemented into applications. The integration of humans into the DT systematic suggested in this paper can be a basis for constructive research on this topic.

Future research activities need to address the challenges and research issues identified in this work. These include, in particular, the flexible integration of human operators and overcoming the separation between the life cycle stages. Based on this analysis and the proposed systematic, new approaches for a holistic integration of human and DT into systems are possible, redressing a static division of labor. A quantified analysis of the improvements and a human-centered validation of the identified approaches can lead way in the design of such applications. The roles identified in this paper provide a starting point for the analysis of possible work. The quality of the individual roles and the corresponding tasks, particularly with regard to the combination of the different roles for operators and DT, is an area for further investigation.

Additionally, the integration of the human into applications in smart manufacturing poses new challenges and opportunities. The latter arise from overcoming traditional roles in the context of manufacturing by engaging people directly through the DT. Rather than replacing or enhancing the operator’s tasks, the DT fosters innovation by creating a new connection between humans and objects. The human becomes a customer or recipient of the DT and its services. Examples include providing cloud-based production solutions directly to potential customers or the customization of services and products. This can be further improved by an accurate DT-representation of the human by combining external measurements with internal information such as skills, preferences, and mood. To exploit this potential, interoperable DTs of humans and objects are a next step.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the German Federal Ministry of Transport and Digital Infrastructure (BMVI) under [Grant 19H17016C].