On the quantification of human-robot collaboration quality

ABSTRACT

Human-robot collaboration and interaction (HRC/I) for industrial applications is considered as a challenging topic that involves not only technological aspects but also extends to Social Sciences and Humanities (SSH) research. However, a proper and efficient human-machine cooperation can increase productivity rates and quality of the overall process. This paper proposes a novel and simple human-robot collaboration – quality index (HRC – QI), aiming to assess the flexibility, performance, cost, and quality aspects of applications involving the co-existence and collaboration of such production entities. The definition of this index considers several metrics for capturing the contributions and drawbacks introduced by the involvement of either humans or robots depending on their attributes and roles within the production operation. Four distinct case studies involving different HRC schemes in different production sectors are used to verify the effectiveness and ease of application of the suggested index.

KEYWORDS:

• Human-robot collaboration and interaction (HRC/I)
• quality index
• hybrid production systems

1. Introduction

The latest trend in manufacturing industry expresses an attempt to achieve high product customization by implementing flexibility and high reconfigurability. Traditionally, the main purposes of upgrading manual production cells to robot automation are to guarantee human workers’ safety during dangerous work activities, to improve production efficiency, and to ensure the quality of the process (Prezas et al. 2022; Charalambous, Fletcher, and Webb 2013). In this direction, the integration of new technological systems can also help to decrease product defects, product delivery times and the cost of reworking (Chryssolouris 2006). Another important parameter for production efficiency, with no interference with indicators such as quality, is the reduction of energy consumption without altering the overall cycle times (Fysikopoulos et al. 2014). Under this aspect, studies have investigated simple principles of achieving this, such as optimization of robot motion acceleration profile and in general of more efficient motion patterns (Arkouli, Aivaliotis, and Makris 2020), which strongly depend on the problem’s dynamics (Pastras, Fysikopoulos, and Chryssolouris 2019).

The smart factory represents a leap forward from traditional automation to a fully connected and flexible system. The case of Human-Robot collaborative applications is an indicative example. In modern manufacturing paradigms, industrial applications tend to involve both human and robotic resources, in an attempt to address the diversified requirements in terms of performance, flexibility (Michela and Dini 2019a), and robustness (Makris 2021). In literature commonly resources are distinguished between single, multiple, and/or team, the latter being a group acting co-ordinately and at the same time, and interacting with the environment and other resources in a specified way (X. V. Wang et al. 2017, 2019). Human-robot collaboration (HRC) has been the topic of many studies, such as in the automotive assembly industries, where the qualities of human operators related to dexterity, troubleshooting, and collaborative handling of unforeseen situations have proven to be indispensable (Tsarouchi, Makris, and Chryssolouris 2016). In such cases, the integration of collaborative robots is preferred over the one of traditional robots (Suchan and Bhatt 2017) that typically work in a separate space without any ability to collaborate with human workers. Robots on the other hand can work in the same space as humans, providing the capability of handling both small and large-scale parts (Arkouli et al. 2021; Chen et al. 2018).

However, the quality and efficiency of human-robot collaborative cells for each industrial use case depend on a large number of parameters. First of all, in each HRC environment, restrictions are induced related to human workers’ safety and their physical and mental situation during a process execution (Michalos et al., ‘Seamless human robot collaborative assembly – An automotive case study’, Michalos et al. 2018). As for the robotic resources’ integration, the level of autonomy that they can provide to the process (Huang et al. 2008), their ability of handling different types of materials (Billard and Kragic 2019), as well as their cognitive autonomy level (Krueger et al. 2019; Tai and Liu 2016), are some of the main parameters that have to be taken into consideration. SMEs often struggle to evaluate these aspects by themselves due to their lack of resources and technical knowledge. Therefore, the four aspects of economic efficiency, technical feasibility, ergonomics, and safety are taken into account when a process involves HRI systems (Weßkamp et al. 2019).

This work suggests an innovative approach for evaluating the level of quality and efficiency of human-robot collaboration, with the long-term aim of being used in the scope of planning systems that are able to consider both types of resources. It aims to capture the majority of factors contributing to an efficient collaboration experience inside any collaborative application, through a simple examination of each scenario, the collaborating entities, and the particularities of the workplace in which it takes place. Both the individual performance of humans and robots will be measured, as well as the level of their interaction that can be achieved. Several industrial case studies are examined to prove the applicability of the proposed tool for the evaluation of process steps’ assignments to flexible human-robot collaborative cells. Cumulative scores for each one of these case studies arise after their evaluation, providing an overall estimation of the HRC quality for each one of these, as well as a comparison between them.

Briefly, this manuscript is structured as follows. In section 2 the work that has been conducted, related to the evaluation of human factors for the purposes of process scheduling and assignment of tasks to both humans and robots, is presented. Section 3 describes the proposed method for quality evaluation of human-robot collaboration, based on various attributes, while the implementation of this tool in various case studies is presented in section 4. Section 5 analyses the results of the aforementioned evaluations. In the end, authors present study’s conclusions and their future actions in section 6.

2. Relevant work

In the last decades, a lot of work has been conducted focusing on planning systems for industrial cases, in which both humans and robots have an important role. Most of these decision-making methodologies, that have been investigated, use multiple criteria, focusing on the technical characteristics of the robotic manipulators (Nikolakis et al., ‘Dynamic scheduling of shared human-robot manufacturing operations’, Nikolakis et al. 2018) or human safety and ergonomic aspects (Michalos et al., ‘Workplace analysis and design using virtual reality techniques’, Michalos et al. 2018) or even on their interaction abilities (Natarajan and Gombolay 2020), in order to achieve an effective and seamless co-existence.

Human-Robot Collaborative (HRC) cells are quite complex, mainly in cases of involving multiple human operators and/or robots. In this case methods for the allocation of tasks to resources are required and these in turn need to incorporate criteria that allow the optimization of the collaborative operation (Zhu et al. 2022). One of the main parameters, that a decision-making logic evaluates during such a collaboration, is the robotic resources’ capability to deal with specific technical aspects of a process, such as the payload needed to handle the various parts or their reachability in the whole area (Tsarouchi, Makris, and Chryssolouris 2016). Decision-making algorithms have been studied, allowing both human-to-robot and robot-to-robot task allocation and cooperation in the same workspace (Tsarouchi et al. 2017). However, the approach of these papers requires a detailed cell design and a lot of information coming from simulation models in order to define efficient collaboration strategies (Mayr et al. 2021). In an effort to perform rational decisions for the organization of a HRC process, using both technical and average resource utilization parameters, Kim et al. have also proposed a CPU scheduling-based algorithm (Yongho and Matson 2016). Recently, an approach to determine task allocation by considering the actual capabilities of both humans and robots in order to improve work quality has been proposed (Ranz, Hummel, and Sihn 2017). Nikolakis et al. presented a method for scheduling shared human-robot tasks supporting online re-scheduling for unexpected situations and involving humans, robots, and/or human-robot teams (Nikolakis et al., ‘On a shared human-robot task scheduling and online re-scheduling’, Nikolakis et al. 2018). Another research has focused on performing an iterative task de-composition into sub-tasks until reaching atomic actions (Fiore, Clodic, and Alami 2014). Under this concept, Human-Aware Task Planner (HATP) module has been presented for producing plans for humans and/or robots, depending on their capabilities and their availability in an industrial environment.

Another aspect that plays an important role when humans and robots are working together is the parameters related to human safety and ergonomics. HRC workplace generation has to take into consideration such factors, while further analysis on ergonomics may be needed in 3D simulation environments, considering detailed aspects, such as human age, muscle strain, fatigue, etc. (Tsarouchi et al. 2016). In the same study, a framework has been proposed which enables designers and engineers to easily evaluate workplace layouts for human-robot collaboration in production. Reduction of the traveling distance on human-robot collaborative cells is also an important aspect that optimizes humans’ ergonomics and working quality (Michela and Dini 2019b; Conti, Varde, and Wang 2020). An attempt to address the production reconfiguration of a hybrid human-mobile robot collaborative assembly cell through a multi-criterial evaluation of the generated alternatives, in terms of weight handled or distance travelled by the operators has been presented (Wang, Dong, and Kan 2020; Kousi et al. 2020). Another research has proposed a two-staged decision-making process aiming to achieve a desired capability-based task allocation that supports job satisfaction, considering both sensory and motoric strengths and ergonomic limitations of the human (Raatz et al. 2020). Frameworks to guide design and evaluation activities that have a positive impact on work well-being have been also conducted, assessing foreseen impacts through questionnaires to different stakeholders related to work satisfaction, work engagement, safety, and user experience (H. Liu et al. 2018; Kaasinen et al. 2020). In (Evangelou et al. 2021), a human-centered approach was presented, responsible for scheduling and assignment of assembly tasks to both human and robot resources, including manufacturing throughput metrics, human RULA-based ergonomics (Michalos et al., ‘Workplace analysis and design using virtual reality techniques’, Michalos et al. 2018) and safety distance. In a similar way, a method has been tested, that enables the automatic examination of the parts and fixtures’ height, in order for the human operator to be provided with better ergonomics Michalos et al. (2018). Dimitropoulos et. al (Dimitropoulos et al. 2021), proposed a system that is able to assess the ergonomics in human-robot shared tasks and adapt the robot pose to improve ergonomics using a heuristics-based search algorithm. Most of the aforementioned studies, which focused on human-related issues, proposed mainly methods for the evaluation of human-robot collaboration performance, based on design and simulation generated data. In general, their main purpose was not to assess the quality of a potential human-robot collaboration, but by using several data, such as time for a task to be executed or the weight of parts, etc., they achieved to provide an efficient assignment of a specific task to the more suitable resource.

Last but not least, an equally important role in the evaluation of a collaborative cell has been identified for the efficient and harmonious Human-Robot Interaction (HRI). Under this concept, various studies have focused on the way that both resources can interact, proposing novel methods and technologies to enhance efficiency, but not aiming for a fast evaluation of the benefits and the quality that a proper human-robot interaction can offer to a collaborative system. In (Cirillo, Karlsson, and Saffiotti 2008), a framework for human-aware planning is presented in which three kinds of human-robot interaction approaches are examined, implicit cooperation, robot supportive role, and robot asking for feedback. The main purpose of the proposed approach is to generate a sequence of robot actions, that takes into account the status of the environment, the goals of the robot, and the forecasted plan of the human. Another study focused on the design and evaluation of Chaski, a task-level execution system that enables a robot to collaboratively execute a shared plan with a person, using insights from human-human teaming to make human-robot teaming more natural and fluid (Shah and Wiken 2011). Furthermore, research on robot decisional abilities has been conducted, taking into account explicit reasoning on the human environment and on the robot’s capacities to achieve its tasks, building a supervision system called SHARY (Supervision for Human-Robot (Y)Interaction) (Clodic et al. 2019). An interaction infrastructure called the ``Human-Robot Interaction Operating System’’ (HRI/OS) was developed by (Fong et al. 2006), aiming at building human-robot teams where they are able to work effectively together and converse about abilities, goals, and achievements. In (Trafton et al. 2013), ACT-R/E has been presented as a cognitive architecture that is used for human-robot interaction tasks, simulating how humans think, perceive, and act in the real world.

To summarize, most of the above relevant work has proposed decision-making logic, which, evaluates different criteria related to robot capabilities and/or human safety, aiming to define better HRC layouts or task assignments. Their main difference with the suggested approach of this work is that they are focusing on a limited set of HRC aspects depending on the design or planning attribute that they aim to optimize. Moreover, in order to apply/evaluate these methods/metrics, data that is generated from detailed design and/or simulations are required, making the whole evaluation quite complex and difficult to perform in real-time. The human-robot collaboration (HRC) quality index presented in the following section aims to capture the majority of aspects contributing to an efficient collaboration experience inside any collaborative application. The metric should be easily obtained by a simple examination of the collaborative scenario, the collaborating entities, and the particularities of the workplace in which it takes place. This paper does not address the planning or task assignment problem of HRC cells, however, the suggested metric can be easily incorporated into such a system and this has been the main motivation behind its development.

3. Human-robot collaboration (HRC) quality index

This paper proposes a novel yet simple approach for the evaluation of the collaboration quality that can be achieved in Human-Robot Collaborative working environments. More specifically it aims to provide a measurable index that is calculated on the basis of individual indicators which reflect the performance, cost, quality, and flexibility aspects of the HRC application.

Ultimately, the goal is to establish an easily calculated index that can be used either in the assessment/comparison of existing collaborative setups or in the decision-making processes that are employed during the design and planning of hybrid production systems.

Ultimately, the goal is to establish an easily calculated index that can be used either in the assessment/comparison of existing collaborative setups or in the decision-making processes that are employed during the design and planning of hybrid production systems.

3.1 The human co-worker paradigm

Looking at the way that robotics and HRC have been evolving over the last decades, it is evident that the ideal outcome would be the enablement of robots to act as human counterparts (Reed and Peshkin 2008). The benefits of such achievement would not only have to do with the enhancement of robotics skills, such as dexterous manipulation, autonomous decision-making, and robust operation. It would also have a great impact on the way that these machines can co-exist and collaborate with actual human operators. A short comparison between the human-human and human-robot collaboration schemes is presented hereafter.

By examining closely the example of two humans working together, it is evident that their interaction is in most cases not structured or strictly defined. Two persons are able to use different means of communication which can be direct (e.g. speech, pointing, etc.) or indirect (facial expressions, body language) and both actors are able to infer and interpret the information that needs to be conveyed (Liu et al. 2016). In the case of humans and robots, the information has to be precise (e.g. confirmation of each individual task completion) and conveyed through a predefined channel (push button, HMI, or other) with no error margin for misinterpretation.

The co-existence of these humans in the same working space presumes an understanding of the workflow execution, as well as reasoning on the other person’s actions which is dynamically adjusted as the process evolves. For instance, a human may decide to change the sequence of assembly operations and his/her co-worker can easily perceive it through observation and eventually adjust his/her future actions. Traditional robotics have very limited autonomy in implementing such dynamic behaviors even in the case of robot-to-robot collaboration.

It is therefore assumed for the scope of this paper that, if the HRC quality is to be optimized, the HRC application has to exhibit attributes that are equivalent to the case of human-to-human collaboration. For this purpose, the definition of the HRC – Quality index, as described in the following sections, relies on the comparison of such attributes between the ideal case, i.e. the robot is able to fully act as a human, and the worst case, i.e. the robot acts as a standard machine.

3.2 HRC index – general principle

Based on the analysis of the previous section, the principle under the definition of the human-robot collaboration – quality index (HRC – QI) involves the quantification of metrics belonging to three different categories as depicted in Figure 1.

Figure 1. HRC - QI overall architecture.

The three metric categories consist of:

• The Robot-related Metrics (RrM): these metrics aspire to capture the contribution of the robotic resources to the collaboration which has a direct impact on the acceptance by operators, as well as the productivity benefits that may be achieved. Robots that can exhibit autonomy, intelligence, dexterity in manipulation, and so on are able to collaborate more efficiently with humans and create a better-performing HRC application.

• The Human-related Metrics (HrM): under this category metrics that are able to quantify the perceived experience by the operators in terms of human safety, ergonomics (Arkouli, Michalos, and Makris 2022), skepticism or trust in collaborating with robots, and overall user experience. Essentially, any metric under this category will have to quantify the direct impact of the HRC application on the human operator as well as his/her perspective toward the application itself, whether it is positive or negative.

• The Human-Robot Interaction-related Metrics (HRIrM): this category of metrics targets the quantification of the interaction experience that is available through the different modalities that robots can use to interact and coexist with humans. Robots’ sensorial systems, peripheral devices, and human-machine interfaces are pieces of equipment that can enable different collaboration schemes such as workspace, time, and workpiece sharing. These schemes have a direct effect on the perceived quality of collaboration as they influence the way that tasks can be shared between humans and robots.

The next section describes the metrics that were selected to capture the individual attributes under each of these categories. In addition, the aggregation of the metric values under a single index is also presented, considering the comparison to the human-to-human collaboration paradigm that was discussed above. All the notations of the metrics of the index are listed in the Appendix at the end.

3.3 Individual metrics and HRC quality index definition

In order to cover the categories that were introduced in the previous section, a set of metrics has been selected. The selection has been based on the literature review survey, the Robotics 2020 Multi-Annual Roadmap, according to the site ‘https://old.eu-robotics.net/cms/upload/topic_groups/H2020_Robotics_Multi-Annual_Roadmap_ICT-2017B.pdf’, as well as the experience of the authors though the participation in multiple EU funded projects focusing on human-robot collaboration and their applications in an industrial environment. Indicatively projects such as x-Act, ROBO-PARTNER, THOMAS, SHERLOCK, and PENELOPE, have provided valuable lessons learned on the preferences of operators as well as the manufacturing performance-related aspects. The metrics that are considered for the scope of this paper involve:

• Robot Autonomy (RA): defines the ability of a robotic resource to work autonomously in a collaborative layout without needing the involvement of the human operator. The higher the autonomy the less effort is required on the human side in the context of collaboration.

• Robot Manipulation ability (RM): reflects the ability of a robotic system to grasp and handle an object within the context of a task. The more capable the robot, the more tasks it can undertake and contribute in more ways to the collaboration.

• Mobility (M): the ability of the robotic resource to travel or be manually repositioned inside the entire production area in order to execute tasks in different workstations. Such an ability enhances the reachability of the robot and thus the potential to work closer to humans. In combination with the ability to learn and execute autonomously new tasks, this attribute greatly enhances the collaborative applications that the robot can address.

• Cognition and Knowledge acquisition ability (CK): the ability of a robot to acquire knowledge about its environments and objects and use it to dynamically adjust its behavior over time. When coupled with the assessment of human ergonomics it can enhance the support that humans receive as the collaboration extends over time.

• Cost (C): refers to the cost of a robotic resource compared to the corresponding cost needed for a human operator to execute the same operation. Although this does not directly affect the collaboration quality, it is among the top factors to be considered when investing in any application.

• Performance for Human Safety (HS): this metric considers the number and type of safety functions that are presented in the HRC application and may allow or prohibit the robot from working at a high speed. It expresses the trade-off between productivity and speed/performance limitations that are needed to ensure the safety of humans.

• Ergonomics benefits (E): represents the physical and/or mental support that a human worker can receive while working on a specific collaborative application. The higher the level of this metric, the more support the operators receive from the integrated system and thus, the more benefits in terms of stress and strain in the long term they receive.

• User eXperience (UX): reflects the personal preference of operators and the general feeling they develop when working with robots. It also considers the level of trust or fear/skepticism that humans may develop, while working or interacting with any robotic system.

• Interaction ability (I): considers the different ways of exchanging information with a robot. The number of options available as well as the intuitive use of each option creates a different experience for the user, who may find it more or less easy to work with the system.

• Workspace sharing (WSs): denotes the ability of the human operator and the robot resource to execute operations in the same working area, regardless of the timing or the parts being processed. A traditional fenced robotic system allows workspace sharing when compared with a fully collaborative robotic cell without physical separation.

• Time Sharing (Ts): represents the ability of the human operator and the robotic resource to work on the same task in parallel.

• Workpiece sharing (WPs): evaluates if the human worker and the robot are able/allowed to work on the same part or with the same tools

In order to receive an easy assessment of each metric a set of levels has been selected considering the most representative cases (Table 1). The lowest level/value implies a non-efficient human-robot collaboration, while the highest one corresponds to the ideal scenario where the robot behaves as a second human being. Of course, in some cases it is not possible to achieve the ‘ideal’ scenario, however, it is a good reference point for comparison between similar applications. It has to be noted that the list is not exhaustive, and it is ongoing work to define more metrics and levels that can capture further aspects of the collaborative applications.

Table 1. HRC – QI metrics scale.

Category	Metric Name	Scenario description	Metric Value
RrM	Robot Autonomy (RA)	Understanding workflow and undertaking action without human involvement	5
		Supervised autonomy in terms of executing a task autonomously in most operating conditions and recognizing when input needed	4
		Traded autonomy in terms of partial workflow execution autonomously and human intervention to select subsequent tasks	3
		Execution of pre-defined actions semi-autonomously while human intervenes to control the system to carry out a task	2
		Requiring instruction/programming/confirmation in every step	1
	Robot Manipulation ability (RM)	Grasping/manipulating any part required for the process	5
		Using tool changers, allowing the use of multiple end effectors, which can handle different product types/sizes/weights	4
		Grasping/manipulating only parts within a specific size/weight/flexibility range using a single but ‘flexible’ end effector	3
		Grasping/manipulating a limited number of components using a part-dedicated end effector (e.g. suction cup)	2
		Grasping/manipulating only parts with specific geometry and size (e.g. screwdriver for a specific size of screws)	1
	Mobility (M)	A robot is able to travel or be relocated within the entire workplace to execute a task (e.g. a mobile manipulator)	3
		A robot is lightweight and it can be moved by the human	2
		Robot is stationary	1
	Cognition and Knowledge acquisition ability (CK)	Observation learning by acquiring knowledge indirectly through observing other robots or people carrying out tasks	6
		User knowledge by acquiring knowledge about the operation by observing humans in a defined area	5
		Requested knowledge by requesting human information about an object or place relevant to the task	4
		Place knowledge related to location/type of objects and environmental features by matching objects to pre-defined and known types	3
		Sensor data knowledge – accumulating knowledge about the environment based on sensors data	2
		No acquired knowledge by the robot during its operation	1
	Cost (C)	Less than an employee carrying out the same tasks	3
		Equal or a bit higher than an employee carrying out the same tasks	2
		Significantly higher than an employee carrying out the same tasks	1
HrM	Performance for Human Safety (HS)	Intrinsic safety in terms of the robot being safe by design and it cannot exert a force that is damaging the person or working at a dangerous speed for humans.	5
		Working at 100% of the designed speed at all times, also when a human is present, perceiving the environment and performing all required safety functions (e.g. collision avoidance, power and force limiting, etc.), but recognizes when this way of working may endanger the user and alters its motion/speed to ensure safe operation	4
		Adapted safety by means of robot working at a reduced speed designed for specific collaborative tasks at all times, without necessarily meaning that the person is within a close distance	3
		Basic safety in terms of the robot recognizing that a person approaches/physically inserts in its operating space and stops its operation automatically	2
		No safety – the robot is not equipped to work with a human	1
	Ergonomics benefits (E)	Full physical support (heavy/dangerous parts) and mental support, reducing human stress and fatigue in the workspace of dangerous and repetitive operations or providing indications about the workflow status	5
		Limited physical support (e.g. co-manipulation – sharing weight), but in combination with mental support during dangerous conditions	4
		Only physical support in collaborative applications, where the robot is responsible to handle heavy/dangerous parts	3
		Only mental support, where the robot provides information or indications about the workflow status (visually, audibly, or other) or reduces the stress of repetitive operations	2
		No ergonomic benefit	1
	User eXperience (UX)	Safe feeling of human-robot collaboration and full trust at all times	3
		Certified safe human-robot collaboration, but limited trust (e.g. human intimidated by a high-payload collaborative robot)	2
		Fear and skepticism of coexisting with a robot (e.g. a robot above the human’s head) - no willingness to cooperate	1
HRIrM	Interaction ability (I)	All types of interaction and functionalities are available (direct, indirect) and implemented in a very efficient – smooth, and natural experience, enabling also indirect robot control (gesture-based interaction to move the robot)	6
		Full direct interaction ability in combination with indirect interaction only for basic functions to start/stop the robot and visualize robot-related information (e.g. through AR-based tools)	5
		Direct physical interaction in terms of teaching a motion sequence to the robot through simple haptic feedback (contact)	4
		Direct control assessing the state of the robot through visual/vision feedback and providing control of the robot moment to moment (e.g. using a joystick)	3
		Limited indirect interaction ability in terms of starting/stopping the robot through a control device (e.g. smartwatch or HMI)	2
		No interaction available	1
	Workspace sharing (WSs)	Ability to work in all areas as the human counterpart without restriction	5
		Partially shared workspace	4
		Same workspace, but working on different tasks	3
		Same workspace, but the human is allowed to enter when the robot stops and vice versa	2
		Completely separated workspace	1
	Time Sharing (Ts)	Ability to work in parallel	3
		Consecutive work – each entity waits for the other to finish	2
		The human does all his tasks and then the robot starts doing its own	1
	Workpiece sharing (WPs)	Full work piece sharing in terms of humans and robots using the same parts/tools, if needed, to execute the process	4
		Workpiece sharing with minimal interference of robot/human working volumes	3
		Workpiece sharing without the interference of robot/human working volumes, being at a safe distance at all times (e.g. different sides of a lengthy part)	2
		No sharing – only a human or a robot work on the part at the same time	1

As it may be challenging to simultaneously consider all the values of the above-mentioned metrics and directly compare two HRC systems, the human-robot collaboration – quality index (HRC – QI) is introduced with the scope of providing a cumulative score and overall assessment of the collaboration quality. Since the metric levels have been defined in a ‘higher level – better performance’ notion, the HRC – QI can be calculated by the Equation 1. Two scientific theories support this equation, one of the arithmetic averages, meaning the sum of a collection of numbers divided by the count of numbers in the collection, and the theory of normalizing each metric’s value to a range between 0 and 1 (Michalos et al. 2015; Milani et al. 2005):

(1) $Q_t=\frac{{\sum }_{m=1}^M\left(\frac{v_m-v_m^{min}}{v_m^{max}-v_m^{min}}\right)}{M}$(1)

where:

• $Q_t$ is the cumulative HRC – QI score for a specific task,

• $m$ is the number of the metric that is examined each time, while M is the total number of the metrics,

• $v_m$ is the value which is set after the evaluation of the task under each metric, and

• $v_m^{min}$, $v_m^{max}\hspace{0.17em}$are the minimum and the maximum values, respectively, corresponding to each metric according to the index scale.

It follows that as there are 12 metrics used and the maximum normalized value of the index can be 1, as well representing the best case of HRC application, and the lowest value can be 0, representing a system with no collaboration potential. Nevertheless, any other similar metric can be easily integrated without affecting the quantification as long as the higher metric values signify a better performance of the system under examination.

4. Case studies

In this section, four case studies are presented aiming at the demonstration and validation of the HRC-QI applicability. These cases have been investigated in the context of EU-funded projects and represent HRC instances where different interaction/collaboration modalities had to be employed to satisfy the user requirements.

4.1 Case 1: pre-assembly of profiles on large-scale parts

The assembly of large-scale parts for the automotive sector is studied in this case study focusing on the pre-assembly of numerous aluminum profiles on top of buses’ sidewall panels.

Currently, all the processes are conducted by human workers (Figure 2), who are responsible to place glue on a series of aluminum profiles and then transfer and place them on the sidewall panel. This procedure is not considered ergonomic, since it is quite repetitive and human workers may need to travel long distances in order to apply the glue and assemble the profiles. In addition, the stability needed during the gluing process is an important issue, as the proper glue pattern application ensures the proper bonding properties.

Figure 2. Case study 1 – Manual gluing of aluminum profiles.

The envisaged solution proposed for this scenario involves Human-Robot collaborative tasks. The process starts with a mobile manipulator (mobile platform and a robot) that is assigned to place glue successively on the aluminum profiles which are located on stationary fixtures. At the same time, a vision system is used to verify that the proper amount of glue and the desired application pattern have been applied. In the case of an incorrect application, the robot dispatches an alarm for the operators to intervene. The human operators who are able to move freely in the area next to the robot are responsible to grasp these aluminum profiles and align them on the sidewall panel. The robot is able to detect human presence with the help of laser scanners located at its base and react accordingly to its presence. The process is completed with the installation of the glued profiles on the corresponding positions and the placement of weight boxes above them. The most important challenge of this case study is the proper interaction between both humans and the robot, while feedback for the completion of each process step is required. Augmented Reality (AR) tools, such as AR glasses and smartwatches, are used to facilitate this interaction, by providing robot task-related information to human workers. In Figure 3 the demonstrator of this process is presented.

Figure 3. Case study 1 layout – HRC pre-assembly of large scale-parts.

Based on the above scenario description of the envisaged solution, and by implementing the aforementioned metrics evaluation of the HRC – QI, the following results have been obtained (Table 2).

Table 2. HRC – QI results for the 1st case and justification.

Categories	Metrics	HRC – QI	Comments
RrM	RA	4/5	Through vision system and AR-based tools, human is notified when to intervene while supervising the process execution
	RM	2/5	Process dedicated end effector, able to grasp the glue dispenser
	M	3/3	The robot is placed on a mobile platform
	CK	3/6	Robot accumulates knowledge about the location/characteristics of the profiles for the gluing, only through sensors data
	C	1/3	High cost because of the two robotic components and the dispensing system
HrM	HS	4/5	Laser scanners that enable human safety, recognizing human existence and avoiding potential collisions
	E	4/5	Limited physical support (robot handles the glue dispenser), and mental support reducing repetitive tasks (glue dispensing)
	UX	3 /3	Working with a collaborative robot under full trust
HRIrM	I	6/6	Indirect: AR-based tools for robot-related information and voice commands, Direct: hand-guiding for error handling
	WSs	3/5	In the same workspace, the robot places the glue and the humans install the profiles
	Ts	3/3	The robot places the glue, while humans install the previously glued profiles
	WPs	3/4	Humans and robots work on the same (aluminum profile) with a minimal interface in case of error handling (robot hand-guiding)
Qt = 0.67 /1

4.2 Case 2: handling and lay-up of deformable materials

This case study scenario was inspired by a real manufacturing system for composite material production. According to this scenario, an innovative way of manufacturing flexible fiberglass textiles (Andronas, Kokotinis, and Makris 2020) is studied.

The main purpose and the most important challenge of this case study are to address the manipulation of such materials which are highly deformable. This composite manufacturing use case is focused on the handling and layup processes, in terms of grasping, transferring, and placing of numerous fabrics, of different dimensions, in a mould. In the current manufacturing system, these steps of the processes are performed manually (Figure 4), resulting in issues, related to production and human working conditions. Non-ergonomic tasks, such as lifting heavy parts, low quality due to humans’ difficulty to reach the entire area, or errors during fabric placement, are the most common of them.

Figure 4. Case study 2 – Manual lay-up of flexible materials.

For all these reasons, a more autonomous and reconfigurable layout has been proposed, enabling Human-Robot Interaction and Collaboration (HRI/C) by integrating an overhead dual-arm robot to assist the worker in the performance of the aforementioned processes. This robotic system will move above the mold, exploiting its full reachability during the placement in the mold tasks, with the addition of the revolute joint. In this way, issues of product quality will be addressed, avoiding humans crawling over the area. By using also an appropriate system of sensors, the robotic arms will try to follow the human moves, aiming to avoid material stretches. As for the dexterous handling system, it will offer a more accurate performance of grasping, manipulation, and placement of fiberglass plies in the mould. By using the proposed tool for this case study, the human-robot collaboration for the simultaneous lay-up process will be evaluated. The setup of this case study is shown in Figure 5.

Figure 5. Case study 2 layout – HRC fabrics handling and lay-up.

Analyzing the scenario of the envisaged solution, the following results have emerged from the metrics evaluation of the HRC – QI (Table 3).

Table 3. HRC – QI results for the 2nd case and justification.

Categories	Metrics	HRC – QI	Comments
RrM	RA	3/5	Partial workflow execution and human intervention to select the next fabric
	RM	3/5	A flexible end effector to handle parts of a specific size/weight range of fabrics
	M	3/3	Rail and the revolute joint allow the robot to move into the entire area
	CK	5/6	Robot ‘observes’ human motions, by using data from vision systems and F/T sensors, to avoid stretching for the flexible material
	C	3/3	Low-cost solution
HrM	HS	5/5	Robot work at a safe distance and full speed, recognizing by itself dangerous cases
	E	4/5	Limited physical support (textile material co-manipulation), as well as reducing crawling/walking on the mould tasks
	UX	3/3	Working with a collaborative robot under full trust
HRIrM	I	3/6	Provide ‘control’ of the robot moment to moment, based on human motions through visual and other sensors feedback
	WSs	4 /5	Partially shared workspace due to the fabrics’ lengthy size
	Ts	3/3	Humans and robots handle the fabric at the same time
	WPs	4/4	Human and robot works on the same fabric
Qt  = 0.81/1

4.3 Case 3: automotive dashboard preassembly

This examined case study aims at a passenger vehicles dashboard preassembly scenario. In its current state, a human worker is responsible for the loading and assembly of a 25 kg traverse, which is found behind the vehicle dashboard (Figure 6). Several parts are installed, such as a body computer and a cable connected to it and fixed on the traverse, by using clips in multiple points, after the loading of the traverse.

Figure 6. Case study 3 – Manual traverse assembly.

The proposed human-robot collaborative scenario is as follows. The process starts with the transport of the traverse from a loading area to an assembly bank by a Dual-Arm Robot of COMAU, where the assembly will take place. After it has been placed on the assembly base, the robot, using both arms, grasps the body computer, and places it on the traverse, while at the same time, it screws the body computer on the traverse, using a modified gripper capable of picking a conventional screwdriver. Finally, the installation of a cable is carried out in cooperation with humans. During this stage, the robot assists the human by grasping the cable and strengthen his reachability. The worker approaches the robot, gets the cable, and installs it on the traverse and the Body Computer using the plastic pins. The main challenge of this case is to achieve human safety not only during the cooperative task but also throughout the entire program’s execution by the robot. For this reason, a PILZ Safety Eye is used to overlook the overall cell and the surrounding area. The envisaged solution cell of this case study, as well as the safety zones provided by the PILZ Safety Eye, are presented in Figure 7.

Figure 7. a) Case study 3 layout – HRC dashboard preassembly, b) Safety zones.

After the evaluation of the metrics under the HRC – QI, the following scores have resulted for this case study (Table 4).

Table 4. HRC – QI results for the 3rd case and justification.

Categories	Metrics	HRC – QI	Comments
RrM	RA	3/5	Partial workflow execution and human intervention to select the next task or the next part to be assembled
	RM	4/5	Tool changers for handling the screwdriver or the cable
	M	1/3	Robot is stationary
	CK	2/6	Robot accumulates knowledge about the environment only through sensors data
	C	2/3	Almost equal cost to the human one
HrM	HS	4/5	PILZ Safety Eye provides safety zones and protects human existence
	E	3/5	Physical support by handling heavy parts
	UX	2/3	Working with a collaborative robot, but limited trust because it can be above the human
HRIrM	I	2/6	Direct interaction which provides the ability to start/stop the robot moment to moment
	WSs	3/5	Same area, but different tasks (robots support heavy parts, and humans install the lighter ones)
	Ts	3/3	Robot grasps heavy parts, while human install the cable at the same time
	WPs	2 /4	Human and robot works on the same part without interfering with the other’s tasks
Qt = 0.48/1

4.4 Case 4: rear-axle assembly

In this case, an industrial scenario is tested, which involves the assembly of a vehicle’s rear axle. The current setup of the rear axle assembly line (Figure 8) employs more than one workstation for the loading of the axle and the assembly of the rear wheel group. The majority of the operations are being performed manually. The parts to be assembled comprise a rigid and of large volume rear axle (20 kg), a smaller right wheel drum of 11 kg, the wheel drum cable of 0.5 kg, which is a flexible material, as well as a set of 4 small screws.

Figure 8. Case study 4 – Manual traverse assembly.

In the envisaged solution, a high payload robot is used in order to perform the loading of the axles. The heaviest part lifted by the operator is a screwdriver of 1.5 kg. The wheel group assembly requires cooperation between the human and the robot, where the robot is handling the weight while the human directly adjusts the position of the parts to be assembled. More specifically, the robot automatically picks up the wheel groups and brings them to the left/right of the axle, opposite to the holes that are used for screwing. In addition, manual guidance techniques are applied by a human for the correct alignment of the part against the holes. After the assembly of the first wheel group and while the human performs delicate tasks (cable assembly, clip insertion, etc.), the robot continues bringing the second wheel group, by avoiding any collision. AR-based tools are also used to provide assembly process information, visualization of the robot workspace, visual alerts, as well as production data. The main challenge of this case is that the human operator and the robot are working at a close distance, collaborating to execute the assembly process. For this reason, a lot of safety restrictions have to be taken into consideration. The setup of this case study is shown in Figure 9.

Figure 9. Case study 4 layout – HRC rear axle assembly and area monitoring using virtual zones.

For the implementation of the HRC – QI for this case study, the following scores have been used (Table 5).

Table 5. HRC – QI results for the 4th case and justification.

Categories	Metrics	HRC – QI	Comments
RrM	RA	3/5	Partial workflow execution and human intervention to select the next task or the next part to be assembled
	RM	3/5	A flexible end effector to handle within parts of a specific size/weight range of axles
	M	1/3	Robot is stationary
	CK	2/6	Robot accumulates knowledge about the environment only through sensors data
	C	2/3	Almost equal cost to the human one
HrM	HS	4 /5	Vision systems and laser scanners that enable human safety and collision avoidance
	E	3/5	Only physical support handling heavy parts (e.g. rear axle)
	UX	2/3	Working with a high-payload collaborative robot, but with limited trust
HRIrM	I	4 /6	Indirect: AR-based tools visualizing process and robot information, Direct: manual guidance techniques
	WSs	4 /5	Partially shared workspace where the human and the robot work at a close distance
	Ts	3/3	The robot grasps the heavy part, while the human is screwing
	WPs	3/4	Human and robot works on the same part with a minimal interface in case of error handling
Qt = 0.54/1

5. Results and discussion

Human-robot collaboration – quality index (HRC – QI) can drastically change the envisaged solutions for each case study, aiming at the optimization of the HRC process. It can be considered as a fast and simple way of evaluating a concept and potentially proposing different ideas in order to achieve a higher final cumulative score. As it can be observed, based each time on the corresponding scenario of the case studies, the HRC – QI offers a first sufficient evaluation of the quality of the human-robot collaborative activities.

The following charts (Figure 10) visualize the results of all four cases for the three examined aspects of metrics, the Robot-related, Human-related, and Human-Robot Interaction-related Metrics, as well as a general comparison of all the cases and metrics.

Figure 10. Overall evaluation for a) Robot-related Metrics, b) Human-related Metrics (HrM), c) Human-Robot Interaction-related Metrics (HRIrM) and d) HRC - QI Metrics.

According to the results from the evaluation of the quality of HRC for these studies, it is observed that a higher score for the Robot-related Metrics has emerged when a robot is working in an autonomous way, with a limited intervention by the human, or in case the robot is capable to handle different sizes and types of part during the whole process. In addition, in cases where the human is relieved both mentally and physically, the score for the Human-related Metrics is higher. As for the Human-Robot Interaction-related Metrics, the integration of technologies, such as AR-based tools, can offer more desirable results. An overall conclusion can be that the cumulative score of the HRC – QI depends mainly on the integration of additional technologies that provide indirect human-robot interaction, as well as ensure the human safety under collaborative tasks.

6. Conclusions and future steps

This manuscript introduced a novel approach, focusing on the assessment of Human-Robot Collaborative (HRC) activities. More specifically, by using the human-robot collaboration – quality index (HRC – QI), an easy and fast evaluation of the collaborative environment quality is conducted. The main innovation of this approach is that a metrics evaluation scale has been presented, that examines the collaboration under three main aspects, Human-related Metrics (HrM), Robot-related Metrics (RrM), and Human-Robot Interaction-related Metrics (HRIrM). The cumulative final score that emerges from this tool can characterize the quality of each HRC process and compare it with the ideal scenario of collaboration between a human and a robot that is able to act as another human being.

Subsequently, four indicative case studies are used to demonstrate the approach. For these cases an evaluation of all the metrics of the HRC – QI has been conducted and the results indicate that the metric can capture the key attributes that affect the collaboration quality. A cumulative score Q_t was obtained for each one of these scenarios, explaining how the corresponding scores arise in Tables 2–5, which aims at a useful and easy to be integrated with a future planning system, estimating their HRC quality, as well as providing an easier comparison between them. According to these results, the lowest index value is presented for the third case study concerning the vehicle dashboard preassembly. This mainly emerged because of the low level of interaction between the human and the robotic resource, as well as a close to the average value for the rest of the examined metrics. The HRC – QI final score of the second case study about the manipulation of large and flexible materials seems to be the highest one. This can be considered logical, since for case 1 the robotic resource, that is selected, is of high cost and it is responsible to execute mainly predefined operations, not acquiring too much feedback from the human. As for case study 4, the main focus of the scenario is on safety issues, not favouring the Robot-related Metrics, as well as providing limited interaction capabilities between the human and the robot.

An important limitation of the proposed approach is that it is difficult to run the evaluation in real time. This means that, firstly, a concept for each examined case study has to be taken into consideration and the available technologies for HRC must be identified. Once the role of its entity and the possible interaction modalities are defined, the user may proceed to set the metric values in order for the index to be calculated. For this reason, a user will be needed in order to implement this proposed tool each time a new collaboration concept emerges.

Future work involves the implementation of the human-robot collaboration – quality index (HRC – QI) in more case studies, aiming at the calculation of the HRC quality for a whole industrial process. Moreover, an investigation of adding more metrics will take place, which will help to cover more aspects related to human-robot collaboration. Metrics that take into consideration parts, products, and particularities, will also be added. A sensitivities analysis should also be conducted, focusing on how the potential uncertainty in the output of the provided index result can be divided and allocated to different sources of uncertainty in its inputs. These inputs will differ one each other by adding weight values to each one of the metrics, according to their importance based on the cases’ requirements. Last but not least, the integration of this index in an HRC planning module under the PENELOPE project is examined. This module will be responsible to address the task allocation in complex human-robot collaborative layouts, by using a multi-criteria decision-making logic, and to generate a list of detailed actions for the task execution by the selected alternative of resources.

Notation

HRC/I	=	Human-robot Collaboration and Interaction
SSH	=	Social Sciences and Humanities
HRC – QI	=	Human-Robot Collaboration – Quality Index
RrM	=	Robot-related Metrics
HrM	=	Human-related Metrics
HRIrM	=	Human-Robot Interaction-related Metrics
RA	=	Robot Autonomy
RM	=	Robot Manipulation ability
M	=	Mobility
CK	=	Cognition and Knowledge acquisition ability
C	=	Cost
HS	=	Performance for Human Safety
E	=	Ergonomics benefits
UX	=	User eXperience
I	=	Interaction ability
WSs	=	Workspace sharing
Ts	=	Time Sharing
WPs	=	Workpiece sharing
Qt	=	Cumulative HRC – QI score
m	=	Number of the metric that is examined each time
M	=	Total number of the metrics
vm	=	Value for each metric after the evaluation of the task
vmmin	=	The minimum value for each metric according to the index scale
vmmax	=	The maximum value for each metric according to the index scale

Acknowledgements

This research has been supported by the European project “PENELOPE Closed-loop digital pipeline for a flexible and modular manufacturing of large components” (Grant Agreement: 958303) funded by the European Commission”.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The work was supported by the PENELOPE Closed-loop digital pipeline for a flexible and modular manufacturing of large components [958303].

Appendix