Towards gestured-based technologies for human-centred Smart Factories

Figures & data

Table 1. Capturing methods and ergonomic assessment measure.

Authors	Year			Depth camera	IMU	Score method
Martin et al.	2012	x		x		OWAS
Vignais et al.	2013		x	x	x	RULA
Nguyen et al.	2013	x		x		EAWS
Haggag et al.	2013		x	x		RULA
Battini et al.	2014	x			x	div.
Paliyawan et al.	2014		x	x		RULA
Peppoloni et al.	2016		x		x	RULA
Plantard et al.	2017		x	x		RULA
Manghisi et al.	2017		x	x		RULA
Bortolini et al.	2018a	x		x		REBA
Bortolini et al.	2018b	x		x		EAWS

OWAS: Ovako Working Posture Analysing System.

RULA: Rapid Upper Limb Assessment.

REBA: Rapid Entire Body Assessment.

EAWS: European Assembly Worksheet.

Figure 1. Layered structure of the ErgoTakt approach.

Figure 2. Scheme of the work preparation, data collection and optimization.

Figure 3. Virtual process chain with 9 manual assembly steps.

Table 2. Input data for the ErgoTakt algorithm.

			1	0,8	1	0,5	1
			Worker resilience value ⇧
			Worker qualification with respect to the process ⇩
Task	CT [s]	RULA score	A	B	C	D	E
1	80	3	1	0	1	0	1
2	150	2	1	0	1	0	1
3	120	2	1	0	1	0	1
4	200	4	1	0	1	0	1
5	40	2	1	1	1	1	1
6	120	2	0	0	0	1	0
7	240	4	1	0	1	0	1
8	60	3	1	1	1	1	1
9	180	2	1	1	1	1	1

Table 3. Categorized literature review gesture-controlled robotics.

		How gestures are tracked?				What kind of robot?		How are gestures processed?
Author	Year	Wearables	IR Camera	Camera	DC	IRo	NPR	Static	Dynamic	Mirroring
Marinho et al.	2012				x	x		x		x
Moe and Schjølberg	2013	x			x	x				x
Shirwalkar et al.	2013				x	x				x
Yang et al.	2015				x	x		x
Fuad	2015				x		x			x
Chen et al.	2015		x			x			x
Tsarouchi et al.	2016a		x	x		x		x
Cueva et al.	2017				x		x	x	x
Hong et al.	2017				x		x			x
Coban and Gelen	2018	x				x		x	x
Kandaleft et al.	2018	x					x	x
Atre et al.	2018			x		x		x
Auquilla et al.	2019				x		x	x	x
Fan et al.	2019	x	x			x			x
Li	2020				x	x		x	x	x
Kaczmarek et al.	2020				x	x		x
Islam et al.	2020		x				x	x	x
Lin et al.	2020				x		x	x		x
Salamea et al.	2020				x		x	x	x

IR = Infrared; IRo = Industrial Robot; NPR =non professional robots.

Figure 4. Initial job assignment.

Figure 5. Cycle time balancing chart; (left) phase 1 optimization without ergonomic, (right) phase 2 ErgoTakt optimization (right).

Figure 6. Architecture for cloud-based gesture control.

Figure 7. (a) top-view of the test bench with the MECA500 robot the camera; b) NodeRed flow and its direction; c) Resulting robot position after pointing a position with the index-finger.

Martin, C. C., D. C. Burkert, K. R. Choi, N. B. Wieczorek, P. M. McGregor, R. A. Herrmann, and P. A. Beling. 2012. “A Real-Time Ergonomic Monitoring System Using the Microsoft Kinect.” In Proceedings of the 2012 IEEE Systems and Information Engineering Design Symposium, 50–55. IEEE. doi:10.1109/SIEDS.2012.6215130.

 Google Scholar

Vignais, N., M. Miezal, G. Bleser, K. Mura, D. Gorecky, and F. Marin. 2013. “Innovative System for Real-Time Ergonomic Feedback in Industrial Manufacturing.” Applied Ergonomics 44 (4): 566–574. doi:10.1016/j.apergo.2012.11.008.

 PubMed Web of Science ®Google Scholar

Nguyen, T. D., M. Kleinsorge, A. Postawa, K. Wolf, R. Scheumann, J. Krüger, and G. Seliger. 2013. “Human Centric Automation: Using Marker-Less Motion Captur- Ing for Ergonomics Analysis and Work Assistance in Manufacturing Processes.” In Proceedings of the 11th Global Conference on Sustainable Manufacturing. Berlin, Germany. . 586–592. https://www.gcsm.eu/Papers/132/18.3_34.pdf

 Google Scholar

Haggag, H., M. Hossny, S. Nahavandi, and D. Creighton. 2013. “Real Time Ergonomic As- Sessment for Assembly Operations Using Kinect.” In Proceedings of the 2013 UK- Sim 15th International Conference on Computer Modelling and Simulation, 495–500. IEEE. doi:10.1109/UKSim.2013.105.

 Google Scholar

Battini, D., A. Persona, and F. Sgarbossa. 2014. “Innovative Real-Time System to Integrate Ergonomic Evaluations into Warehouse Design and Management.” Computers & Industrial Engineering 77: 1–10. doi:10.1016/j.cie.2014.08.018.

 Web of Science ®Google Scholar

Paliyawan, P., C. Nukoolkit, and P. Mongkolnam. 2014. “Office Workers Syndrome Moni- Toring Using Kinect.” In Proceedings of the 20th Asia-Pacific Conference on Com- munication (APCC2014), 58–63. IEEE. doi:10.1109/APCC.2014.7091605.

 Google Scholar

Peppoloni, L., A. Filippeschi, E. Ruffaldi, and C. A. Avizzano. 2016. “A Novel Wearable System for the Online Assessment of Risk for Biome- Chanical Load in Repetitive Efforts.” International Journal of Industrial Ergonomics 52: 1–11. doi:10.1016/j.ergon.2015.07.002.

 Web of Science ®Google Scholar

Plantard, P., H. P. Shum, A. S. Le Pierres, and F. Multon. 2017. “Validation of an Ergonomic Assessment Method Using Kinect Data in Real Workplace Conditions.” Applied Ergonomics 65: 562–569. doi:10.1016/j.apergo.2016.10.015.

 PubMed Web of Science ®Google Scholar

Manghisi, V. M., A. E. Uva, M. Fiorentino, V. Bevilacqua, G. F. Trotta, and G. Monno. 2017. “Real Time RULA Assessment Using Kinect V2 Sensor.” Applied Ergonomics 65: 481–491. doi:10.1016/j.apergo.2017.02.015.

 PubMed Web of Science ®Google Scholar

Bortolini, M., M. Faccio, M. Gamberi, and F. Pilati. 2018a. “Motion Analysis System (MAS) for Production and Ergonomics Assessment in the Manufacturing Processes.” Computers & Industrial Engineering 139. doi:10.1016/j.cie.2018.10.046.

 Web of Science ®Google Scholar

Bortolini, M., M. Gamberi, F. Pilati, and A. Regattieri. 2018b. “Automatic Assessment of the Ergonomic Risk for Manual Manufacturing and Assembly Activities Through Optical Motion Capture Technology.” Procedia CIRP 72: 81–86. doi:10.1016/j.procir.2018.03.198.

 Google Scholar

Marinho, M. M., A. A. Geraldes, A. P. L. Bo, and G. A. Borges. 2012. “Manipulator Control Based on the Dual Quaternion Framework for Intuitive Teleoperation Using Kinect.” In 2012 Brazilian Robotics Symposium and Latin American Robotics Symposium, 16.10.2012 - 19.10.2012, 319–324. Fortaleza, Brazil: IEEE. doi:10.1109/SBR-LARS.2012.59.

 Google Scholar

Moe, S., and I. Schjølberg. 2013. “Real-Time Hand Guiding of Industrial Manipulator in 5 DOF Using Microsoft Kinect and Accelerometer.” In 2013 IEEE RO-MAN, 26.08.2013 - 29.08.2013, 644–649. Gyeongju: IEEE. doi:10.1109/ROMAN.2013.6628421.

 Google Scholar

Shirwalkar, S., A. Singh, K. Sharma, and N. Singh. 2013. “Telemanipulation of an Industrial Robotic Arm Using Gesture Recognition with Kinect.” In 2013 International Conference on Control, Automation, Robotics and Embedded Systems (CARE), 16.12.2013 - 18.12.2013, 1–6. Jabalpur, India: IEEE. doi:10.1109/CARE.2013.6733747.

 Google Scholar

Yang, Y., H. Yan, M. Dehghan, and M. H. Ang. 2015. “Real-Time Human-Robot Interaction in Complex Environment Using Kinect V2 Image Recognition.” In 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), 15.07.2015 - 17.07.2015, 112–117. Siem Reap, Cambodia: IEEE. doi:10.1109/ICCIS.2015.7274606.

 Google Scholar

Fuad, M. 2015. “Skeleton Based Gesture to Control Manipulator.” In 2015 International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation (ICAMIMIA), 15.10.2015 - 17.10.2015, 96–101. Surabaya, Indonesia: IEEE. doi:10.1109/ICAMIMIA.2015.7508010

 Google Scholar

Chen, S., H. Ma, C. Yang, and M. Fu. 2015. “Hand Gesture Based Robot Control System Using Leap Motion.” In Intelligent Robotics and Applications, Lecture Notes in Computer Science, edited by H. Liu, N. Kubota, X. Zhu, R. Dillmann, and D. Zhou, 581–591. Vol. 9244. Cham: Springer International Publishing. doi:10.1007/978-3-319-22879-2_53.

 Google Scholar

Tsarouchi, P., A. Athanasatos, S. Makris, X. Chatzigeorgiou, and G. Chryssolouris. 2016a. “High Level Robot Programming Using Body and Hand Gestures.” Procedia CIRP 55: 1–5. doi:10.1016/j.procir.2016.09.020.

 Google Scholar

Cueva, C. W. F., S. H. M. Torres, and M. J. Kern. 2017. “7 DOF Industrial Robot Controlled by Hand Gestures Using Microsoft Kinect V2.” In 2017 IEEE 3rd Colombian Conference on Automatic Control (CCAC), 18.10.2017 - 20.10.2017, 1–6. Cartagena: IEEE. doi:10.1109/CCAC.2017.8276455.

 Google Scholar

Hong, C., Z. Chen, J. Zhu, and X. Zhang. 2017. “Interactive Humanoid Robot Arm Imitation System Using Human Upper Limb Motion Tracking.” In 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), 05.12.2017 - 08.12.2017, 2746–2751. Macau: IEEE. doi:10.1109/ROBIO.2017.8324706.

 Google Scholar

Coban, M., and G. Gelen. 2018. “Wireless Teleoperation of an Industrial Robot by Using Myo Arm Band.” In 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), 28.09.2018 - 30.09.2018, 1–6. Malatya, Turkey: IEEE. doi:10.1109/IDAP.2018.8620789.

 Google Scholar

Kandalaft, N., P. S. Kalidindi, S. Narra, and H. N. Saha. 2018. “Robotic Arm Using Voice and Gesture Recognition.” In 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), 01.11.2018 - 03.11.2018, 1060–1064. Vancouver, BC: IEEE. doi: 10.1109/IEMCON.2018.8615055.

 Google Scholar

Atre, P., S. Bhagat, N. Pooniwala, and P. Shah. 2018. “Efficient and Feasible Gesture Controlled Robotic Arm.” In 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), 14.06.2018 - 15.06.2018, 1–6. Madurai, India: IEEE. doi:10.1109/ICCONS.2018.8662943.

 Google Scholar

Auquilla, A. R., H. T. Salamea, O. Alvarado-Cando, J. K. Molina, and P. A. S. Cedillo. 2019. “Implementation of a Telerobotic System Based on the Kinect Sensor for the Inclusion of People with Physical Disabilities in the Industrial Sector.” In 2019 IEEE 4th Colombian Conference on Automatic Control (CCAC), 15.10.2019 - 18.10.2019, 1–6. Medellín, Colombia: IEEE. doi:10.1109/CCAC.2019.8921359.

 Google Scholar

Fan, Y., C. Yang, and X. Wu. 2019. “Improved Teleoperation of an Industrial Robot Arm System Using Leap Motion and MYO Armband.” In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), 06.12.2019 - 08.12.2019, 1670–1675. Dali, China: IEEE. doi:10.1109/ROBIO49542.2019.8961758.

 Google Scholar

Li, X. 2020. “Human–robot Interaction Based on Gesture and Movement Recognition.” Signal Processing: Image Communication 81: 115686. doi:10.1016/j.image.2019.115686.

 Web of Science ®Google Scholar

Kaczmarek, W., J. Panasiuk, S. Borys, and P. Banach. 2020. “Industrial Robot Control by Means of Gestures and Voice Commands in Off-Line and On-Line Mode.” Sensors (Basel, Switzerland) 20 (21): 6358. doi:10.3390/s20216358.

 PubMed Web of Science ®Google Scholar

Islam, J., A. Ghosh, M. I. Iqbal, S. Meem, and N. Ahmad. 2020. “Integration of Home Assistance with a Gesture Controlled Robotic Arm.” In 2020 IEEE Region 10 Symposium (TENSYMP), 05.06.2020 - 07.06.2020, 266–270. Dhaka, Bangladesh: IEEE. doi:10.1109/TENSYMP50017.2020.9230893.

 Google Scholar

Lin, C.-S., P.-C. Chen, Y.-C. Pan, C.-M. Chang, and K.-L. Huang. 2020. “The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures.” Journal of Sensors 2020: 1–9. doi:10.1155/2020/9270829.

 Web of Science ®Google Scholar

Salamea, H. T., A. R. Auquilla, O. Alvarado-Cando, and C. U. Onate. 2020. “Control of a Telerobotic System Using Wi-Fi and Kinect Sensor for Disabled People with an Industrial Process.” In 2020 IEEE ANDESCON, 13.10.2020 - 16.10.2020, 1–6. Quito, Ecuador: IEEE. doi:10.1109/ANDESCON50619.2020.9272069.

 Google Scholar