Towards gestured-based technologies for human-centred Smart Factories

ABSTRACT

Despite the increasing degree of automation in industry, manual or semi-automated are commonly and inevitable for complex assembly tasks. The transformation to smart processes in manufacturing leads to a higher deployment of data-driven approaches to support the worker. Upcoming technologies in this context are oftentimes based on the gesture-recognition, − monitoring or – control. This contribution systematically reviews gesture or motion capturing technologies and the utilization of gesture data in the ergonomic assessment, gesture-based robot control strategies as well as the identification of COVID-19 symptoms. Subsequently, two applications are presented in detail. First, a holistic human-centric optimization method for line-balancing using a novel indicator – ErgoTakt – derived by motion capturing. ErgoTakt improves the legacy takt-time and helps to find an optimum between the ergonomic evaluation of an assembly station and the takt-time balancing. An optimization algorithm is developed to find the best-fitting solution by minimizing a function of the ergonomic RULA-score and the cycle time of each assembly workstation with respect to the workers’ ability. The second application is gesture-based robot-control. A cloud-based approach utilizing a generally accessible hand-tracking model embedded in a low-code IoT programming environment is shown.

KEYWORDS:

• Gesture-Based monitoring
• gesture-based control
• assembly
• manufacturing

1. Introduction

Within the 4^th industrial revolution, the so-called Industry 4.0, more and more smart technologies as connected intelligent devices (sensors and actuators), or augmented reality applications find their way into production systems. Nevertheless, production system still is, and will most likely remain, a sociotechnical system in the future. Hence, the employees have to interact with these smart technical systems, e.g. machinery or robots. The realization of Smart Factories needs new ways of monitoring, control and interaction between humans and technical systems to increase efficiency (Ansari et al. 2018). These ‘multimodal’ Human-Machine-Interaction mechanisms become a crucial factor in the factories of the future. The term ‘multimodal’ summarizes research that leverages speech, touch, vision and gesture (Turk 2014). Especially, gesture-based methods provide a huge potential in industry as they can be used actively to control technical devices as robots and passively to monitor and analyse the human movements of employees e.g. to assess the ergonomic situation. In this context, several approaches to track human gestures are presented, which then utilize e.g. the arm trajectory to control technical devices. This contribution reviews the different strategies of motion capturing and relates their application in industrial environment. Here, the focus is on ergonomic assessment, gesture recognition and analysis as well as gesture-based control of (industrial) robots.

Ergonomic evaluation in working scenarios requires a deep analysis of postures, loads, and frequency of movements. In the industrial field, experts commonly carry it out with observational methods that are not supported by automated, instrumental measurements and are affected by subjective bias (Roman-Liu 2014). This approach is costly and time-consuming; therefore, it is usually carried out retrospectively of an accident and by sampling only the most critical stations/processes and workers. Hence, long-term and systemic ergonomic evaluations are difficult due to current tools limitations. These are the main reasons why the ergonomic assessment is not integrated into line balancing issues as a holistic view of the process. The authors want to leverage the 4^th industrial revolution, and its integrated technologies for supporting the worker of the future, the so-called ‘Operator 4.0’ (Romero et al. 2016a). Here, bio-data monitoring (i.e. postures and workload physiological data), smart solutions (i.e. wearable trackers and sensors) and advanced Human-Machine-Interaction can contribute improving wellbeing, inclusivity, and safety of workplaces without affecting the productivity (Romero et al. 2016b).

As part of HMI (Human-Machine Interaction), Human-Robot Interaction (HRI) is a growing field of research, especially with intuitive programming and control of robots (Tsarouchi, Makris, and G. Chryssolouris 2016b).

Gesture control of industrial robots has a huge potential regarding less teaching requirements and various industrial settings, which may have restrictions in term of accessibility (Tang and Webb 2018). Furthermore, the ongoing trend of ‘mass-customization’ in industrial manufacturing leads to the requirement of flexible production equipment and processes (Schmitt et al. 2021). Pederson et al. identify robot programming as a general bottleneck for adaption and reconfiguration in production (Pedersen et al. 2016). Hence, gesture-based programming and control of robots provide a more intuitive and thus time effective way to enhance flexibility in industrial value chains. Other possibilities for eliminating bottlenecks in a human-robot production, such as optimized layout planning (Tsarouchi et al. 2017) or improved human-robot collaboration in individual activities (Dimitropoulos et al. 2021) will not be considered in this work.

Both issues, automated ergonomic assessment and gesture-based control of robots are covered in this contribution. After a literature review of gesture-based capturing technologies, ergonomic assessment, COVID-19 gesture identification and robot control, two different application scenarios are focused are presented focusing on utilizing gesture data in industry. First, the Ergo-Takt approach, which combines ergonomic assessment and assembly line balancing and second, gesture-based robot control. The contribution closes with a summary and further research activities.

2. State of research in gesture-based monitoring and control

The following paragraph gives an overview on different motion capturing technologies, which contain the basis for gesture-based monitoring and control of technical systems. In the field of gesture-based monitoring, ergonomic assessment methods utilize human gestures to evaluate, e.g. the exposure of employees to physiological stress in working environments. The captured human movements are related to different ergonomic measures-. In the research field of gesture-based monitoring, approaches of Human-Robot-Interaction are reviewed and categorized by their control strategy.

2.1. Gesture capturing technologies

Various technologies to capture human movements or gestures are present. Table 1 summarizes the most relevant ‘wearable free’ ones classified according to the ergonomic score method described in the following paragraph, captured body area and capturing technology. The term ‘wearable free’ in this context means, that capturing devices are not attached to the human body, as this can be a constraint in industrial processes. In terms of the successful application in a production environment and the necessity of a high acceptance of the capturing system is a crucial aspect. Hence, capturing technologies, which potentially has no disturbing influence on the operator in assembly are reviewed. Additionally, the score methods for ergonomic assessment are shown in Table 1. They are presented and discussed in detail in the next paragraph

Table 1. Capturing methods and ergonomic assessment measure.

Authors	Year			Depth camera	IMU	Score method
Martin et al.	2012	x		x		OWAS
Vignais et al.	2013		x	x	x	RULA
Nguyen et al.	2013	x		x		EAWS
Haggag et al.	2013		x	x		RULA
Battini et al.	2014	x			x	div.
Paliyawan et al.	2014		x	x		RULA
Peppoloni et al.	2016		x		x	RULA
Plantard et al.	2017		x	x		RULA
Manghisi et al.	2017		x	x		RULA
Bortolini et al.	2018a	x		x		REBA
Bortolini et al.	2018b	x		x		EAWS

OWAS: Ovako Working Posture Analysing System.

RULA: Rapid Upper Limb Assessment.

REBA: Rapid Entire Body Assessment.

EAWS: European Assembly Worksheet.

Table 1 indicates that two capturing techniques are common: depth camera and inertial measurement unit (IMU). Nearly all listed approaches use a Microsoft Kinect as a depth camera. The Kinect camera was originally designed for gaming, but due to its low cost and high performances it became an outstanding tool for researchers in many fields from medical (Webster and Celik 2014; Boenzi et al. 2016) to virtual and augmented reality (Cruz, Lucio, and Velho 2012; Facchini et al. 2016; Manghisi et al. 2018). Different methods utilizing other equipment, which is applied to the body (wearables), are presented in Lu et al. (2014) or Moin et al. (2021) and are not discussed in our contribution. For the first gesture-based application example, i.e. the ergonomic assessment, certain scores must be derived using the captured movements.

2.2. Motion capturing for ergonomic assessment

Ergonomics assessment methods and tools have been extensively studied and presented in scientific literature, medical knowledge, and industrial environment rules. One of the first approaches is the Ovako Working Posture Analysing System (OWAS), which divides movements into four body parts (trunk, arms, lower body and neck). It uses a series of instantaneous observations to compile a score of the harmfulness of the activity from defined tables (Karhu, Kansi, and Kuorinka 1977). The basic concept of OWAS inspired more complex methods as RULA (Rapid Upper Limb Assessment), REBA (Rapid Entire Body Assessment), and EAWS (European Assembly Worksheet), NIOSH-Eq (Revised National Institute for Occupational Safety and Health – Equation) -, Strain Index (SI) - and OCRA (McAtamney and Corlett 1993; Hignett and McAtamney 2000; Schaub et al. 2013; Waters et al. 1993; Moore and Garg 1995; Occhipinti 1998; Colombini 1998). OCRA presents results as a quotient of the theoretically best possible execution of the activity (Occhipinti 1998; Colombini 1998). Also the JSI-L (job severity index – lifting) method provides a quotient of the possible performance of an employee and the foreseen activity. This method is suitable only indirectly for the evaluation of the activities; the suitability of the employee for a certain activity is evaluated first (Liles et al. 1984). Compared to this basis, the Metabolic Energy Expenditure Rate (EnerExp) method considers the energy consumption of an activity (Garg, Chaffin, and Herrin 1978). For each type of activity an energy consumption value is assigned. Finally, the sum of the used physical energy is compared to the maximum harmless energetic performance of a worker. The presented methods share a practical limitation. To evaluate an activity, an observation must take place, through either direct annotation or video recording, followed by an assessment usually carried by specialized personnel. To overcome human error and bias, literature presents different approaches to ergonomic measurement automation. In Table 1 it can be seen, that the RULA metric is mostly used in the literature as an automated tool. RULA was developed originally for quasi-static manual evaluation. However, the ErgoSentinel tool, allows real time skeleton tracking and RULA processing (Manghisi et al. 2017).

2.3. COVID-19 gesture identification

During COVID-19 pandemic, also gesture-based issues have also been studied to support the fight against the virus spread. Here, sensor systems and data analysis strategies to identify cough and other COVID-19 related gestures have been investigated. In Chuma and Iano (2021) present a method using a K-band Doppler radar to detect coughs. The data analysis is done via Convolutional Neural Network (CNN) with an accuracy of 80%−88% under the consideration of the distance of the proband to the sensor. In Rehman et al. (2021), the authors propose a platform to identify COVID-19 symptoms by detecting hand movement, coughing, and breathing. The platform uses Channel Frequency Response (CFR) to record the minute changes in Orthogonal Frequency Division Multiplexing (OFDM) subcarriers due to any human motion over a wireless channel (Rehman et al. 2021). In Khan et al. (2020), the same research groups review non-contact sensing principles of COVID-19 symptoms by WiFi-based technologies. Manghisi et al. (2020) present a body-tracking solution to detect hand-face contact gesture as a potential pathway of contagion. The systems utilize a Microsoft Kinect camera to track especially the head area. The developed HealthShield module compares each newly acquired frame with the previous one in real-time. If the contact label changes from no-contact to contact for certain head region (e.g. mouth, nose, left/right eye) contact, a signal is sent to the interface module, indicating a contact event and the contact area. With this approach, the research group get an overall accuracy of 79.83% in seven classification-fields in the humans’ face.

Based on this literature review, it can be stated that gesture identification, monitoring and control provide a huge potential in industry for various purposes. One the one hand smart ways to utilize human movements for device control can lead to increase efficiency in production. On the other hand, recommendations can be derived to plan or (in real time) adapt workstations in a more individual and ergonomically manner. In the following we present the Ergo-Takt approach, which combines gesture-based ergonomic assessment with production planning and control issues to distribute work load with a human centred approach.

3. The Ergo-Takt approach

Although industrial processes lead toward extensive use of flexible automation, manual assembly is still inevitable due to the mounting complexity e.g. of cables, hoses or electronics, hard-to-reach places, sensitive surfaces, difficult operations, etc. (Heilala and Voho 2001). The design and levelling of a predominantly manual assembly line is since the beginning of industrial mass production, but it becomes even more critical nowadays. In fact, to compete in the global market, companies need to increase effectiveness and efficiency, especially in high-loan countries. In addition, global policies lead to an increased focus on workers’ wellness and safety. In fact, according to the Sixth European Working Conditions Survey (Eurofund 2019), exposure to repetitive arm movements and tiring positions lead to the development of work-related musculoskeletal disorders (WMSDs) with heavy costs for countries welfare. WMSDs include ‘all musculoskeletal disorders that are induced or aggravated by work and the circumstances of its performance’ (WHO 2003). WMSDs extend to almost all occupations and sectors, bearing critical physical and economic consequences for the sufferer: workers, families, businesses, and governments. These ailments are considered the most common labor medical problems among workers in the European Union. Furthermore, these issues are amplified by the aging of the workforce; in 2080, about one-third of the European population will be 65 or older (Eurostat 2019). This involves the need for preserving the operators’ wellbeing consistent with their active aging in the production environment. Therefore, the aging society, the lack of specialists or the product quality inevitably lead to high ergonomic requirements for manual assembly processes (Peruzzini and Pellicciari 2017). There is a wide range of different approaches in the literature for taking ergonomic factors into account in production and its planning. For example, factors can already be taken into account in the early design phase of processes (Sun et al. 2018) or by using simulations. For this purpose, entire work environments and processes can be virtualised (Golabchi et al. 2015) or virtual environments in which real processes are carried out (Michalos et al. 2018; Peruzzini et al. 2021). Another starting point is to achieve an ergonomic work planning mathematically. One of the possibilities is to use ergonomic factors for an adapted job rotation (Michalos et al. 2010). Another is to take ergonomics aspects into consideration when optimising work processes. The time-leveling of an assembly line is mostly named in the literature as ‘assembly line balancing’ (ALB). ALB is normally designed and evaluated by means of takt-time. As a matter of fact, takt-time levelling of an assembly line and the worker ergonomics are not directly related. This means that a well-levelled assembly line may not be ergonomic, and, conversely, a series of ergonomic workstations does not lead to a well-balanced assembly line. Ergonomic evaluation of the work shift requires a deep analysis of postures, loads, and frequency of movements. In the industrial field, it is commonly carried out by experts with observational methods that are not supported by instrumental measurements and are affected by subjective bias (Roman-Liu 2014). This approach is costly and time-consuming; therefore, it is usually carried out retrospectively of an accident and by sampling only the most critical stations/processes and workers. Hence, long term and systemic ergonomic evaluations are difficult due to current tools limitations. These are the main reasons why the ergonomic assessment is not integrated into ALB as a holistic view of the process. In this contribution, a novel approach called ‘ErgoTakt’ is proposed, which improves the standard takt-time centered balancing approach by automatic ergonomic evaluation. This is achieved by integrating the ErgoSentinel system (Manghisi et al. 2017) with an optimization algorithm to find the best solution of takt-time and ergonomic score for an entire chain of assembly stations. The approach also considers possible constraints, such as the qualification of the employees for the respective tasks, their physical impairment, etc. The ErgoTakt approach with its six layers for an entire assembly process chain is illustrated in Figure 1: data acquisition, operating resources, human resources, economic data, ergonomic data, ErgoTakt layer

Figure 1. Layered structure of the ErgoTakt approach.

The data acquisition layer provides ergonomic data captured by a Microsoft Kinect camera and processed by the ErgoSentinel software, which derives a real-time RULA-score as input data for the operator movements. For this reason, every workstation should be equipped with a camera. The operating resource layer contains the technical description of the workstation, which means the tasks to be conducted at the specific workstation, e.g. drilling, gluing, screwing and/or manual assembly. As input data the cycle time of a single task and the needed qualification for each task at the workstation are necessary. The human resource layer characterizes the operators by their qualification level, which is related in a database with the workstation description. Furthermore, the physical impairment is considered as a value [0 … 1]. This value gives, e.g. a disabled or older person more time to fulfil a specified task. Both, the qualification as well as the impairment are used as input data or the underlying calculation of advanced data. The economic data layer and ergonomic data layer import and aggregate data from the former three named layers. The balancing chart on the one hand and the ergonomic data, on the other hand, are derived with respect to a work cycle. The ErgoTakt layer computes the optimization of ErgoTakt indicator. All input data and the calculated values are aggregated and fed into an optimization algorithm. The aim is to minimize the ergonomic stress of the worker and cycle time per workstation with respect to workers’ qualification and physical impairment as well as the mandatory assembly order in the process chain. The utilized ErgoSentinel software is described by the authors in Manghisi et al. (2017, 2020) in detail.

3.1. Optimization of ergonomics and line balancing

The scheme in Figure 2 illustrates how the necessary data are provided and used to optimize the assembly process chain with respect to the operators’ workload. Starting from the ‘process engineer’, the workstation, the worker and the process data are collected, checked and brought into a feasible work sequence before these data were provided to the data-processing system. Then, the ‘assembly worker’ carries out the assembly work. The movements are recorded with the aid of a depth camera and converted into a RULA score by the ErgoSentinel tool. Under the constraints of the mandatory process order, the assembly technology, the worker qualification, and the physical impairment, the algorithm builds variants of the optimized working processes.

Figure 2. Scheme of the work preparation, data collection and optimization.

These constraints and possible resulting variants are made available to the algorithm through matrices. The following parameters are used for this purpose and for optimization:

(1) $I=\left\{\mathrm{i}|\mathrm{i}is\hspace{0.17em}task\right\}\mathrm{T}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{s}$(1)

(2) $J=\left\{j|j\hspace{0.17em}is\hspace{0.17em}workstation\right\}Workstations$(2)

(3) $K=\left\{k|k\hspace{0.17em}is\hspace{0.17em}worker\right\}\mathrm{W}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{s}$(3)

(4) $CT=\left\{ct|c{t}_i\hspace{0.17em}is\hspace{0.17em}cycletime\hspace{0.17em}of\hspace{0.17em}task\hspace{0.17em}i\in I\right\}\mathrm{C}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}$(4)

(5) $TT=\hspace{0.17em}\left\{tt|tt\hspace{0.17em}is\hspace{0.17em}takttime\right\}\mathrm{T}\mathrm{a}\mathrm{k}\mathrm{t}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{j}$(5)

(6) $T{T}_{customer}=const.\mathrm{C}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{a}\mathrm{k}\mathrm{t}-\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}$(6)

(7) $RUL{A}_i=\hspace{0.17em}\left[1,\dots ,7\right]\mathrm{R}\mathrm{U}\mathrm{L}\mathrm{A}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{i}$(7)

(8) $Re{s}_k=\left]0,\dots ,1\right]\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{o}\mathrm{f}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{w}$(8)

The assembly preparer provides most of these parameters as input data. Using the data, two matrices ‘tasks-workstations’ and ‘workers-tasks’ are formed. These represent which employee may/can do which task and which task can be executed on which station. Binary variables are used to show which combinations are feasible.

(9) $A\in I\times Jwith{a}_{ij}=1iffeasible0ifnotfeasible\text{Matrix}\text{tasks}-\text{workstations}$(9)

(10) $B\in I\times Kwith{b}_{ik}=1iffeasible0ifnotfeasible\text{Matrix}\text{workers}-\text{tasks}$(10)

The data are also used to form a three-dimensional matrix of all theoretical combinations. Each combination is a three-dimensional binary variable containing the assigned worker, task and station. This matrix is used as start point for the algorithm.

(11) $\begin{array}{rl}& C\in I\times J\times K,with{c}_{ijk}=\left\{\begin{array}{cc}1& ifassignedtosolution\\ 0& ifnotassignedtosolution\end{array}\right.\\ & Matrixworkers-tasks-workstations\end{array}$(11)

The resulting worker-workstation-task variables ($\mathrm{\forall }{c}_{ijk}=1)$ are compared under the following conditions:

1. Is the workstation feasible to fulfil the addressed task(s) and vice versa?

(12) $a_{ij}=1\mathrm{F}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{t}\mathrm{o}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$(12)

2. Is the worker able to fulfil the task with respect to the qualification level and his/her eventual physical impairment?

(13) $b_{ik}=1\mathrm{F}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}$(13)

3. Is each task exactly assigned once?

(14) ${\sum }_{y\in J}{\sum }_{z\in K}{c}_{iyz}=1\mathrm{E}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{e}\mathrm{d}\mathrm{e}\mathrm{x}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{e}$(14)

4. Is the maximum number of workers per workstation and vice versa one?

(15) $\begin{array}{rl}& \left(\sum _{x\in I}{c}_{xjk}\ge 1\right)\to \left[\sum _{x\in I}{c}_{xyk}=0\mathrm{\forall }y\in J\mathrm{\setminus }\{j\}\right]\text{^}\left[\sum _{x\in I}{c}_{xjz}=0\mathrm{\forall }z\in K\mathrm{\setminus }\left\{k\right\}\right]\\ & \mathrm{M}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{u}\mathrm{m}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{o}\mathrm{o}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{v}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{a}\end{array}$(15)

5. Is the sum of all cycle times of the workstation less than the customer takt-time?

(16) $\begin{array}{c}t{t}_{jk}:={\sum }_{x\in I}c{t}_x\cdot c_{xjk}\le T{T}_{customer}\\ Takt-time\hspace{0.17em}of\hspace{0.17em}workstation\hspace{0.17em}less\hspace{0.17em}than\hspace{0.17em}customer\hspace{0.17em}takt-time\end{array}$(16)

The optimization algorithm using the ErgoTakt as indicator for its objective function

(17) $ErgoTak{t}_{jk}={\sum }_{i\in I}\frac{RUL{A}_i\cdot c{t}_i\cdot }{Re{s}_k}\mathrm{E}\mathrm{r}\mathrm{g}\mathrm{o}\mathrm{T}\mathrm{a}\mathrm{k}\mathrm{t}\mathrm{o}\mathrm{f}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{j},$(17)

which becomes the determining factor for the objective function.

The objective function minimizes the sum of the absolute values of all differences between the individual ErgoTakt variants.

(18) $min\left({\sum }_{j,y\in J;k,z\in K;j\ne y;k\ne z}\left|ErgoTak{t}_{jk}-ErgoTak{t}_{yz}\right|\right)$(18)

with $ErgoTak{t}_{mn}={\sum }_{i\in I}\frac{RUL{A}_i\cdot c{t}_i\cdot c_{imn}}{Re{s}_n},m\in J,n\in K$ ErgoTakt of workstation m with worker n. The factor $c_{imn}$ is necessary to relate the variable to the indicator mathematically. The factor describes the parameters to be used in the calculation.

The optimization algorithm evaluates all possible task-workstation-worker combinations regarding the accomplishment of the named conditions and the predefined assembly order. Based on these results the tasks are assigned to the workstations according to the minimum ErgoTakt indicator.

3.2. Ergonomic and line balancing use-case

To illustrate the ErgoTakt approach and its results regarding an assembly scenario, a synthetic process chain is designed as a virtual simulation of nine tasks configured as shown in Figure 3.

Figure 3. Virtual process chain with 9 manual assembly steps.

The tasks can be performed by five simulated workers with respect to their qualifications and are distributed over five workstations. The assembly/task order is fixed. The cycle time (CT) for tasks and the average RULA-score are presented in Table 2 The worker data contain the worker ID from A to E and his/her physical resilience, which is represented by a percentage value (see also Table 2). In this case, it is supposed that the workers B and D have reduced physical resources, e.g. due to their age or disability. Those employees will get more time to fulfil the task according to their impairment. Furthermore, the qualification for the specified tasks (1–9) is noted as a binary variable.

Table 2. Input data for the ErgoTakt algorithm.

			1	0,8	1	0,5	1
			Worker resilience value ⇧
			Worker qualification with respect to the process ⇩
Task	CT [s]	RULA score	A	B	C	D	E
1	80	3	1	0	1	0	1
2	150	2	1	0	1	0	1
3	120	2	1	0	1	0	1
4	200	4	1	0	1	0	1
5	40	2	1	1	1	1	1
6	120	2	0	0	0	1	0
7	240	4	1	0	1	0	1
8	60	3	1	1	1	1	1
9	180	2	1	1	1	1	1

Table 3. Categorized literature review gesture-controlled robotics.

		How gestures are tracked?				What kind of robot?		How are gestures processed?
Author	Year	Wearables	IR Camera	Camera	DC	IRo	NPR	Static	Dynamic	Mirroring
Marinho et al.	2012				x	x		x		x
Moe and Schjølberg	2013	x			x	x				x
Shirwalkar et al.	2013				x	x				x
Yang et al.	2015				x	x		x
Fuad	2015				x		x			x
Chen et al.	2015		x			x			x
Tsarouchi et al.	2016a		x	x		x		x
Cueva et al.	2017				x		x	x	x
Hong et al.	2017				x		x			x
Coban and Gelen	2018	x				x		x	x
Kandaleft et al.	2018	x					x	x
Atre et al.	2018			x		x		x
Auquilla et al.	2019				x		x	x	x
Fan et al.	2019	x	x			x			x
Li	2020				x	x		x	x	x
Kaczmarek et al.	2020				x	x		x
Islam et al.	2020		x				x	x	x
Lin et al.	2020				x		x	x		x
Salamea et al.	2020				x		x	x	x

IR = Infrared; IRo = Industrial Robot; NPR =non professional robots.

The initialization of the optimization is done by various, but technically feasible, allocation of tasks to a workstation (W_i), illustrated in Figure 4. As an additional starting condition, it is defined that gluing (Task 6 in Figure 3) is only feasible at workstation 4. Worker D is the only one qualified for this task. The sum of the task cycle times per workstation defines its maximum cycle time. It can be seen, that the resulting levelling is less than ideal.

Figure 4. Initial job assignment.

Based on these input data, the optimization is carried out by the algorithm described previously. To show how the implication of ergonomic assessment influences the optimization results, the assembly scenario has been optimized in two phases:

3.2.1. Phase 1

In phase 1 no ergonomic data are considered for the line balancing optimization. Hence, the physical impairment value does not influence the algorithm even if the workers might be burdened by stress or time pressure in real assembly scenarios. The aim is here to optimize the balancing within the entire process chain. Thereby, the target is to lower the maximum cycle time of the”slowest” workstation in the process chain. The initial scenario gives the start value as 400 s. at W3 (see Figure 4).

The result of this optimization phase shows the reassignment of the tasks to the workstations as in the initial scenario and this a ‘homogenization’ of the cycle times. The maximum cycle time of the process chain is lowered to 280 s.; the total waiting time between the workstations is 130 s (see Figure 5, right).

Figure 5. Cycle time balancing chart; (left) phase 1 optimization without ergonomic, (right) phase 2 ErgoTakt optimization (right).

3.2.2. Phase 2

In phase 2, the RULA score is considered (Figure 5 right). The new target value of the optimization algorithm is to minimize the ErgoTakt indicator. As a result, the tasks are also reorganized within the process chain under the given order between W1 to W5. In addition, a technological feasible order is still present. As a result, the maximum cycle time of the process time stays at 280 s., the waiting rises slightly to 150 s (see. Figure 5, left). The mean cycle time decreases from 258 s. (phase 1) to 238 s. in phase 2. The standard deviation σ decreases also from ±57,4 s. to 31,2 s. at the expense of the waiting time (WT) between all workstations (130 s. → 150 s.).

These indicators show that the integration of the ergonomic score and resilience into the optimization leads to more WT between the workstations. The mean cycle time as well as the standard deviation σ between both optimization phases differs, which both indicate a homogenization of the assembly line especially when considering σ. Furthermore, the average RULA scores, which are known for each task, can be related to both phases even if they are not considered for the optimization in phase 1. The task-workspace- assignments then lead to a RULA scoring for W1–W5. In Figure 5 shows the occurring alignment. The physical exposure of the worker at W2 is brought down from RULA 3,5 to RULA 3, but at the expense of W4 (2,0–2,5). The RULA score is levelled in a better way by the ErgoTakt approach as in phase for this assembly scenario.

The results indicate the applicability and the advantages of the approach in a simulated assembly process chain. Based on the virtual scenario simulation, both, the line balancing and the ergonomic scores are improved. The RULA score at the most critical station 2 decreases by 30%. The increase of at WS 4 is in a rather uncritical range (2 → 2,5). The benefits of the ergonomic acquisition by the ErgoSentinel software are obvious, long-term evaluation, continuous RULA scoring or low-cost equipment are just some aspects. The combination of automated RULA evaluation and production line balancing makes it possible to optimize both dimensions, economic line balancing and ergonomic posture.

3.3. Discussion and limitations of the ErgoTakt approach

It can be seen critically, that actually and, above all, in the existing assembly process chain, many other factors influence the decision of tasks and how they are organized. For instance, the factory layout, the technical flexibility of shifting tasks between workspaces or technological incompatible combinations of workstation and tasks might be a challenge for the implementation of the ErgoTakt approach. Therefore, the first use of the ErgoTakt should already take place in the planning phase of an assembly process chain in order to evaluate the RULA-scores by ErgoSentinel, e.g. by cardboards. Then, the technical equipment of the workstations and the arrangement of the tasks can be analyzed in advance in terms of ergonomic and line balancing issues. Nevertheless, the ErgoTakt approach is applicable in an existing assembly line, when the following aspects are fulfilled predominantly:

• Workstation must be compatible with existing skeleton capturing technologies (e.g. postures, tools and safety devices, occlusions, etc.)

• Modular assembly lines with reconfigurable tasks and task technologies

• Re-configurability of the assembly line layout

• Similarity of (assembly) technologies of the workstations

Low degree of automated workstations within the process chain in comparison to manual assembly tasks. These conditions are usually verified in normal working environment, thus they are not limiting the use of this technology

The ErgoTakt use case demonstrate the utilization of gesture-based monitoring to derive ergonomic values from the workers movement (Wilhelm et al. 2021).

The next paragraph reviews gesture-based robot control technologies and show a cloud-based straight-forward approach of gesture-based robot control to demonstrate either gesture-based monitoring and control issues.

4. Gesture-based Human-Machine-Interaction

The capture of movements can additionally be used for interaction between humans and machines to provide an intuitive process by using gesture control. This allows a modification of existing production systems by new and smart interaction mechanisms. An obvious application for gesture commands is the control of robots. Industrial robots are especially used to assist humans in working environments, e.g. due to dangerous environmental conditions or high physical loads. Due to the fact that a robot is supposed to replace the movements of an employee, programming through corresponding movements is an intuitively applicable method. In the following Table 3, a review of gesture-based robot control is presented. The utilized technologies, the type of robot and the kind of control gestures are classified. The technologies are divided into four categories: wearables, camera, infrared camera (Leap Motion) and depth camera (Microsoft Kinect). The robot types are classified into professional industrial robots and non-professional/commercial robots, which functionalities are similar to industrial robots. Other areas with strong research activity in gesture control, such as humanoid robots, are left out. A distinction is made between static gestures and dynamic gestures. Static gestures are firmly assigned gestures that trigger exactly one movement in the robot. Dynamic gestures can be the indirect transmission of motion sequences, e.g. the movements of a human hand on a robot arm. Mirroring, in our context, means transmitting the (scaled) spatial coordinates of the arm movement, and thus, determining the position of the end effector.

Marinho et al. (2012) show the use of a depth camera (Microsoft Kinect) in combination with an industrial robot. The movements of the right arm are mirrored. The left arm solves predefined tasks through defined gestures, e.g. gripping objects. Only one arm can be used at a time. The transmission of the movements of the arm must be stopped by a defined gesture, if the other one should be used. Moe and Schjølberg (2013) show a similar setup with a depth camera (Microsoft Kinect) and an industrial robot with the aim of performing a pick-and-place task. An acceleration sensor (smartphone) to be able to transmit the orientation extends the depth camera. They notice an inaccuracy of the end-point and a delay, especially with fast movements. In addition, smaller movements are poorly recognized. Shirwalkar et al. (2013) use a similar setup (Microsoft Kinect + industrial robot) to perform pick-and-place and liquid-handling tasks. The focus of their work deals with different workspace sizes.The paper does provide a direct coordinate transformation of the length of the arms’ movements or a variable factor to convert the length to the robots’ workspace. In their work, the movements are continued by the robot at a constant speed until the gesture to stop is given. Other static gestures are used, especially for commands to the gripper. Yang et al. (2015) use movement control in conjunction with a collaborative work process. Using a depth camera (Microsoft Kinect v2), a defined object is recognized in one of the worker’s hands. With the other arm, commands are given in the form of static gestures. The spatial position of the object is detected and the industrial robot picks up the object from the operator or releases the grip after receiving corresponding instructions through gestures. Fuad (2015) also uses a depth camera (Kinect v2) and an industrial robot. The focus of this work is on the direct transmission of velocities at the axes by measuring the joint angle between the adjacent bones at the corresponding human joint of the arm. Chen et al. (2015) present an approach that uses an infrared camera and industrial robots. For the robot control, an IR camera detects only the hand and the coordinate transformation into movements of the robot arm is made. Tsarouchi et al.(2016b) use an IR camera (Leap Motion) in addition to the depth camera (Kinect) to send static commands to an industrial robot using ROS. The depth camera detects arm positions as commands, while the IR camera for close range detects finger positions for the same commands. Cueva, Torres, and Kern (2017) use a depth camera (Kinect v2) and an Arduino-controllable robot. Different boxes on a screen are shown to the user, whereas each of them represents an axis or the robot gripper itself. The gripper can be controlled independently of the other axes, as it is operated with the other hand. If the hand is moved from one side to the other in a box, the axis is moved. In addition, a distinction is made between an open hand and a closed hand for start or stop. In their work, Hong et al. (2017) use a depth camera (Kinect v2) and a self-built robotic arm that functions as closely as possible to a human arm. To achieve a smooth movement of the robot arm, a smoothing filter is implemented for the control commands. Erroneous values e.g. due tomeasurement errors of the depth camera are removed. Coban and Gelen (2018) also use an EMG wristband (MYO wristband) to control an industrial robot. In addition, to defined gestures and associated pre-programmed movement commands to the arms, the gyroscope, accelerometer and magnetometer are used. Roll, pitch and yaw orientation values from the sensors are utilized for spatial orientation. Kandalaft et al. (2018) apply speech control to a non-professional robotic arm in addition to an accelerometer attached to the operator’s hand. The accelerometer provides x, y, z values. Defined hand positions are calibrated beforehand and are assigned to specific commands for the robot arm. Commands can also be given by voice control, for which the spoken words are translated into text via smartphone and compared with programmed text for specific movements. Atre et al. (2018) use a USB camera and computer vision for gesture control. The camera captures images of the user. Computer vision is used to extract the hand and determine the user’s gestures. The recognised gesture is then used for control. In their work, Auquilla et al. (2019) use a similar system with boxes as described by Cueva, Torres, and Kern (2017). Boxes are displayed on the user’s screen. The control is now realized by gestures in the different boxes. Using this control, a pick-and-place task is performed. An automatic process is also tested, in which defined steps in the process are started by the user’s gestures. Fan, Yang, and Wu (2019) combine EMG (MYO wristband) and IR camera (Leap Motion) for control. The control is activated as soon as a closed hand is detected by the IR camera. When the hand is moved in the viewfield of the IR camera, the coordinates are translated and transmitted. The transmission ends as soon as the hand is opened. The wristband serves as a safety feature that detects unforeseen movements (e.g. shaking of the arm) and briefly interrupts the transmission of the data. Li (2020) shows an approach to gesture control using a depth and RGB camera (Kinect v2). Static gestures, dynamic gestures and movements are recognized by a trained Neural Network. The simulated picking and screwing is given as an application example. Kaczmarek et al. (2020) use the depth camera (Kinect v2) and voice control to control an industrial robot. The developed software is used to control different industrial robots. The robot is controlled with pre-programmed gestures. In addition, it is shown that each command is executed with a delay.

Islam et al. (2020) present a gesture-controlled non-professional robotic arm for IoT-applications. It is controlled by an IR camera (Leap Motion). Compared to a jostick controller, similar results can be achieved using gesture control. Movement sequences can be performed with similar accuracy as with a joystick. When recognizing certain commands, such as grab or rotate, there may be errors in the recognition. Lin et al. (2020) use a depth camera (Kinect v2) to recognize both hands. One hand is used to transmit real-time movements to the non-professional robot. The other hand is used for static gestures. The gestures enable the start and stop of a recording of real-time movements. Another gesture is used to run the recorded gestures through the robot arm. Salamea et al. (2020) use a depth camera (Kinect v2) to control a non-professional SCARA robot by static gestures. Here, the way of transferring the commands for teleoperations is focused. In view of the work, it is clear that depth cameras (Microsoft Kinect) are the most commonly used technology for detection. For higher precision, infrared camera technology (Leap Motion) is used. For control, the most common application is the use of static and dynamic gestures in combination.

The review provides that gesture-based control of robots seem to be feasible even for industrial application. Due to the fact, that software as a service is an increasing model to outsource computationally intensive processes, the next paragraph introduce a cloud-based approach of gesture-based robot control.

4.1. Cloud-based approach of gesture-based robot control

Two keywords of this contribution are Industry 4.0 and smart manufacturing. In this context, cloud-based services more and more find their way in production. Hence, a straight-forward approach to realize gesture-based robot control by combining existing cloud-based AI-models and graphical IoT-programming is presented.

Our approach relies on the TensorFlow model for hand-pose recoginition (MediaPipe Handpose 2022) integrated in a NodeRed flow. NodeRED is an open-source graphical development tool based on JavaScript for the integration of IoT-devices, APIs (Application Programming Interfaces) and online services by so-called flows (Gardasevic et al. 2017).

To realize the interface the robot the pythonshell-node is used to send commands to the robot control. A MECA 500 industrial 6-axis serial robot is utilized, which can be controlled via python commands. Doing this, the open-source IDE Anaconda is employed. The chosen architecture is shown in -Figure 6.

Figure 6. Architecture for cloud-based gesture control.

For the test-bench the MECA 500 robot is equipped with a measurement probe as end-effector. A camera is statically mounted over the entire setup. The aim of our use is, that the robot adopts the x-coordinate (Rx) of its program according to the x-position of the index-finger (Fx), which has the same direction. The NodeRed flow and the resulting gestured-based robot control application is structured as follows (see also Figure 7):

1. Capturing a picture: An inject node is used to start the image capturing via a USB-camera. An image preview node shows the picture taken from the operators’ hand

2. Getting the x-coordinate of the index finger (Fx): The image is passed by the handpose-node, which provides a cloud-based hand-tracking service via a Neural Network (Zhang et al. 2020). The output of this node is a JSON data object providing the x- and y-coordinate of each finger.

3. Transforming Fx to the robot coordinate (Rx): To scale the resulting index-finger coordinate according to the world reference frame of the robot a function-node linearizes Fx.

4. Transfer the transformed coordinate to the python robot script: The resulting coordinate is then transferred by a pythonshell-node to the robot program.

Figure 7. (a) top-view of the test bench with the MECA500 robot the camera; b) NodeRed flow and its direction; c) Resulting robot position after pointing a position with the index-finger.

With the presented approach, a straightforward gesture-based robot control can be realized at low-costs and low-code effort enabled by the provided visual elements of NodeRed. A crucial drawback is the high latency (>1 sec.) as the model computation is done not locally, yet.

4.2. Discussion and limitations of the cloud-based approach of gesture-based robot control

The industrial application of the described human-robot interaction approach depends on different factors, as latency, robustness and accuracy of human trajectory compared to the robot one. The control loop time for industrial PLCs (programmable logic controller) is less than 5 ms. Hence, a broadband infrastructure is essential. Here, industrial 5 G technology can provide a wireless standard, which is applicable for cloud-based and time critical scenarios. The robustness can be accociated with the repeatability and independence from the user, as industrial applications normally run 24/7 with a high availability. The accuracy depends on the real application, if the robot should be placed in sub-millimetre dimension or just roughly positioned. Here, the scaling mechanism from human to robot movements need to be adoptable.

5. Conclusion and further research

The recognition of gestures and movements in manufacturing offers a wide range of possibilities to be applied. It can be used for health prevention for employee as well as for various value-adding application scenarios in terms of line-balancing. Low-cost technologies in the form of depth cameras are available for digitising and supporting human-system interaction. With the help of these, various potential applications are already realized with different devices combined with other data-driven approaches, as artificial intelligence (neural networks) or augmented reality (AR). Further fields can be opened.

The ErgoTakt approach uses this inexpensive possibility of using a depth camera to record human movements. This is used to create a balance between the performance and health of the employee and economic interests. For this purpose, the RULA score is recorded using the ErgoSentinel software and combined with an optimization algorithm for manual assembly lines. In the developed use case two scenarios are presented, one optimization with and one without ergonomics values. The result shows an improvement of the RULA-Score of the most critical workstation of about 30% expense of increasing the waiting time by 14%.

In addition to this approach, the acquisition of gestures to control robots as a crucial automation element manufacturing systems is presented. The technique used is a low-cost, low-code solution with one USB camera with open-source IoT-environments. The camera captures movements of the hand by means of image recognition and transforms the derived finger coordinate into the robot coordinate frame for positioning tasks. The robot programm is adapted within the control loop. This cloud-based approach of teleoperative control again opens several possible applications. Scenarios can be assisted remotely even for difficult and precise tasks given by experienced users, or operations in hazardous environments can be executed at a safe distance even with the required precision.

Both approaches illustrate industrial applications for gesture-based technologies in digitized industry, but also in other fields. At this point, further research needs to adapt current sensing and processing techniques for use in real industrial environments. In addition, there is a demand for further adaptation of the optimization algorithm to be able to integrate automated processes or even teleoperative processes. In addition, the control possibilities via gesture must be improved by means of low latencies and tested for their reliability in acquisition and processing. The possibilities and effects of integrating teleoperations into existing (industrial) processes must also be investigated. In addition, human aspects must be considered. An examination whether and how employees change their behavior when their activities are recorded and processed by technologies is crucial for the acceptance of technical systems in working environments.

Acknowledgments

The authors gratefully thank the Hans-Wilhelm Renkhoff Foundation for supporting the ROKOMESS project as well as the Bayerische Forschungsallianz funding the BayIntAn- project to strengthen the international research relation of the Politechnico di Bari and the University of Applied Sciences Würzburg-Schweinfurt. Grant no. BayIntAn_FHWS_2019_159).

The Ergosentinel software is a result of the implementing agreement between the Apulia agency of the National Institute for Insurance against Accidents at work of the Apulia Region (INAIL) and the Polytechnic institute of Bari: “Sviluppo e Diffusione di un tool software low cost basato su Kinect TM v2, per il monitoraggio automatico in real time del rischio ergonomico”.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the Bayerische Forschungsallianz [BayIntAn_FHWS_2019_159]; National Institute for Insurance against Accidents at work of the Apulia Region; Hans-Wilhelm Renkhoff Foundation [ROKOMESS].