Explainable human activity recognition based on probabilistic spatial partitions for symbiotic workplaces

References

• Al-Yacoub, A., A. Buerkle, M. Flanagan, P. Ferreira, E. -M. Hubbard, and N. Lohse. 2020. “Effective Human-Robot Collaboration Through Wearable Sensors.” In 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 08 - 11 September, 2020, Vienna, Austria, 651–658. doi:10.1109/ETFA46521.2020.9212100. IEEE.
   Google Scholar
• Antakli, A., T. Spieldenner, D. Rubinstein, D. Spieldenner, E. Herrmann, J. Sprenger, and I. Zinnikus. 2019. “Agent-Based Web Supported Simulation of Human-Robot Collaboration.” In Proceedings of the 15th International Conference on Web Information Systems and Technologies, Vienna, Austria, 88–99. SCITEPRESS - Science and Technology Publications.
   Google Scholar
• Bänziger, T., A. Kunz, and K. Wegener. 2020. “Optimizing Human– Robot Task Allocation Using a Simulation Tool Based on Standardized Work Descriptions.” Journal of Intelligent Manufacturing 31 (7): 1635–1648. doi:10.1007/s10845-018-1411-1.
   Web of Science ®Google Scholar
• Beschi, M., M. Faroni, C. Copot, and N. Pedrocchi. 2020. “How Motion Planning Affects Human Factors in Human-Robot Collaboration.” IFAC-PapersOnline 53 (5): 744–749. doi:10.1016/j.ifacol.2021.04.167.
   Google Scholar
• Braden, B. 1986. “The Surveyor’s Area Formula.” The College Mathematics Journal 17 (4): 326–337. doi:10.1080/07468342.1986.11972974.
   Google Scholar
• Brás, A., M. Simão, and P. Neto. 2018. “Gesture Recognition from Skeleton Data for Intuitive Human-Machine Interaction.” In Adv. Transdiscipl. Eng, edited by W. Nel, S. Josip, P. Marcello, C. Bil, and P. Margherita, Vol. 7. 271–280. The Netherlands: IOS Press BV. doi:10.3233/978-1-61499-898-3-271.
   Google Scholar
• Claudia, P.D., and J. A. Shah. 2015. “Fast Target Prediction of Human Reaching Motion for Cooperative Human-Robot Manipulation Tasks Using Time Series Classification.” In 2015 IEEE International Conference on Robotics and Automation (ICRA), 26-30 May 2015, Seattle, WA, USA, 6175–6182. IEEE. doi:10.1109/ICRA.2015.7140066.
   Google Scholar
• Dallel, M., V. Havard, D. Baudry, and X. Savatier. 2020. “InHard - Industrial Human Action Recognition Dataset in the Context of Industrial Collaborative Robotics.” In 2020 IEEE International Conference on Human-Machine Systems (ICHMS), 07-09 September 2020, Rome, Italy, Sep, 1–6. IEEE. doi:10.1109/ICHMS49158.2020.9209531.
   Google Scholar
• Díaz, I., J. Martín, X. Justo, C. Fernández, and J. Juan Gil. 2022. “Methods for User Activity Recognition in Exoskeletons.“ In Wearable Robotics: Challenges and Trends, edited by J. C. Moreno, J. Masood, U. Schneider, C. Maufroy, and J. L. Pons, Biosystems & Biorobotics 487–492. Cham: Springer International Publishing. 978-3-030-69547-7. doi:10.1007/978-3-030-69547-7_78.
   Google Scholar
• ElMaraghy, H., L. Monostori, G. Schuh, and W. ElMaraghy. 2021. “Evolution and Future of Manufacturing Systems.” CIRP Annals 70 (2): 635–658. doi:10.1016/j.cirp.2021.05.008.
   Web of Science ®Google Scholar
• Fahn, C.S., and K.Y. Chu. 2011. “Hidden-Markov-Model-Based Hand Gesture Recognition Techniques Used for a Human-Robot Interaction System.” In Human-Computer Interaction. Interaction Techniques and Environments, edited by J. A. Jacko, 248–258. Berlin, Heidelberg: Springer.
   Google Scholar
• Fera, M., A. Greco, M. Caterino, S. Gerbino, F. Caputo, R. Macchiaroli, and E. D’amato. 2019. “Towards Digital Twin Implementation for Assessing Production Line Performance and Balancing.” Sensors (Basel, Switzerland) 20 (1): 1. doi:10.3390/s20010097.
   PubMed Web of Science ®Google Scholar
• Fritzsche, L., M. Funk, P. Rosen, M. Spitzhirn, and S. Wischniewski. 2021. “Simulation and Participatory Design of HRI Work Systems in the SOPHIA Project.“ 67th Ergonomics Congress, 03-05 March, 2021, Dortmund: GfA Bochum, Germany.
   Google Scholar
• Gillies, S. 2007. “. “Shapely: Manipulation and Analysis of Geometric Objects.” https://github.com/Toblerity/Shapely.
   Google Scholar
• González, L., J. C. Álvarez, A. M. López, and D. Álvarez. 2021. “Metrological Evaluation of Human–Robot Collaborative Environments Based on Optical Motion Capture Systems.” Sensors (Basel, Switzerland) 21 (11): 3748. doi:10.3390/s21113748.
   PubMed Web of Science ®Google Scholar
• Gutzeit, L., A. Fabisch, M. Otto, J. Hendrik Metzen, J. Hansen, F. Kirchner, and E. Andrea Kirchner. 2018. “The BesMan Learning Platform for Automated Robot Skill Learning.” Frontiers in Robotics and AI 5: 5. doi:10.3389/frobt.2018.00043.
   PubMed Web of Science ®Google Scholar
• Hüsing, E., C. Weidemann, M. Lorenz, B. Corves, and M. Hüsing. 2021. “Determining Robotic Assistance for Inclusive Workplaces for People with Disabilities.” Robotics 10 (1): 1. doi:10.3390/robotics10010044.
   Web of Science ®Google Scholar
• Islam, M. M., and T. Iqbal. 2021. “Multi-GAT: A Graphical Attention-Based Hierarchical Multimodal Representation Learning Approach for Human Activity Recognition.” IEEE Robotics and Automation Letters 6 (2): 1729–1736. doi:10.1109/LRA.2021.3059624.
   Web of Science ®Google Scholar
• Jianjing, Z., H. Liu, Q. Chang, L. Wang, and R. X. Gao. 2020. “Recurrent Neural Network for Motion Trajectory Prediction in Human-Robot Collaborative Assembly.” CIRP Annals 69 (1): 9–12. doi:10.1016/j.cirp.2020.04.077.
   Web of Science ®Google Scholar
• Jonek, M., M. Manns, and T. B. Tuli. 2021. “Virtuelle Montageplanung Mit Motion Capture Systemen/Virtual Assembly Planning with Motion Capture Systems.” wt Werkstattstechnik online 111 (04): 256–259. doi:10.37544/1436-4980-2021-04-78.
   Google Scholar
• Kai, L., Q. Liu, X. Wenjun, J. Liu, Z. Zhou, and H. Feng. 2019. “Sequence Planning Considering Human Fatigue for Human-Robot Collaboration in Disassembly.” Procedia CIRP 83: 95–104. doi:10.1016/j.procir.2019.04.127.
   Google Scholar
• Kelley, R., M. Nicolescu, A. Tavakkoli, M. Nicolescu, C. King, and G. Bebis. 2008. “Understanding Human Intentions via Hidden Markov Models in Autonomous Mobile Robots.” In 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI), Mar, 12-15 March 2008, Amsterdam, Netherlands, 367–374. IEEE. doi:10.1145/1349822.1349870.
   Google Scholar
• Koch, J., L. Büsch, M. Gomse, and T. Schüppstuhl. 2022. “A Methods-Time-Measurement Based Approach to Enable Action Recognition for Multi-Variant Assembly in Human-Robot Collaboration.” Procedia CIRP 106: 233–238. doi:10.1016/j.procir.2022.02.184.
   Google Scholar
• Kousi, N., C. Gkournelos, S. Aivaliotis, C. Giannoulis, G. Michalos, and S. Makris. 2019. “Digital Twin for Adaptation of Robots’ Behavior in Flexible Robotic Assembly Lines.” Procedia Manufacturing 28: 121–126. doi:10.1016/j.promfg.2018.12.020.
   Google Scholar
• Kyrarini, M., M. Abdul Haseeb, D. Ristić-Durrant, and A. Gräser. 2019. “Robot Learning of Industrial Assembly Task via Human Demonstrations.” Autonomous Robots 43 (1): 239–257. doi:10.1007/s10514-018-9725-6.
   Web of Science ®Google Scholar
• Lee, S. U., A. Hofmann, and B. Williams. 2019. “A Model-Based Human Activity Recognition for Human–Robot Collaboration.” In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 03-08 November 2019, Macau, China, 736–743. IEEE. doi:10.1109/IROS40897.2019.8967650.
   Google Scholar
• Lin, C. J., and R. Prasetyo Lukodono. 2021. “Sustainable Human–Robot Collaboration Based on Human Intention Classification.” Sustainability 13 (11): 5990. doi:10.3390/su13115990.
   Web of Science ®Google Scholar
• Liu, Z., Q. Liu, X. Wenjun, Z. Liu, Z. Zhou, and J. Chen. 2019. “Deep Learning-Based Human Motion Prediction Considering Context Awareness for Human-Robot Collaboration in Manufacturing.” Procedia CIRP 83: 272–278. doi:10.1016/j.procir.2019.04.080.
   Google Scholar
• Liu, S., L. Wang, and X. Vincent Wang. 2020. “Symbiotic Human-Robot Collaboration: Multimodal Control Using Function Blocks.” Procedia CIRP 93: 1188–1193. doi:10.1016/j.procir.2020.03.022.
   Google Scholar
• Magrini, E., F. Ferraguti, A. Jacopo Ronga, F. Pini, A. De Luca, and F. Leali. 2020. “Human-Robot Coexistence and Interaction in Open Industrial Cells.” Robotics and Computer-Integrated Manufacturing 61: 101846. doi:10.1016/j.rcim.2019.101846.
   Web of Science ®Google Scholar
• Malik, A. A., and A. Bilberg. 2019. “Complexity-Based Task Allocation in Human-Robot Collaborative Assembly.” Industrial Robot: The International Journal of Robotics Research and Application 46 (4): 471–480. doi:10.1108/IR-11-2018-0231.
   Web of Science ®Google Scholar
• Manns, M., T. B. Tuli, and F. Schreiber. 2021. “Identifying Human Intention During Assembly Operations Using Wearable Motion Capturing Systems Including Eye Focus.” Procedia CIRP 104: 924–929. doi:10.1016/j.procir.2021.11.155.
   Google Scholar
• Martínez-Villaseñor, L., and H. Ponce. 2019. “A Concise Review on Sensor Signal Acquisition and Transformation Applied to Human Activity Recognition and Human– Robot Interaction.” International Journal of Distributed Sensor Networks 15 (6): 1550147719853987. doi:10.1177/1550147719853987.
   Web of Science ®Google Scholar
• May, G., M. Taisch, A. Bettoni, O. Maghazei, A. Matarazzo, and B. Stahl. 2015. “A New Human-Centric Factory Model.” Procedia CIRP 26: 103–108. doi:10.1016/j.procir.2014.07.112.
   Google Scholar
• McConnell, A., D. Mitchell, K. Donaldson, S. Harper, J. Blanche, T. Lim, D. Flynn, and A. Stokes. 2021. The Future Workplace: A Symbiotic System of Systems Environment, 259–329. 1st ed. Boca Raton: CRC Press.
   Google Scholar
• Menolotto, M., D.S. Komaris, S. Tedesco, B. O’flynn, and M. Walsh. 2020. “Motion Capture Technology in Industrial Applications: A Systematic Review.” Sensors 20 (19): 5687. doi:10.3390/s20195687.
   PubMed Web of Science ®Google Scholar
• Michalis, V., C. Nikou, and I. A. Kakadiaris. 2015. “A Review of Human Activity Recognition Methods.” Frontiers in Robotics and AI 2: 2. doi:10.3389/frobt.2015.00028.
   Google Scholar
• Mourtzis, D. 2020. “Simulation in the Design and Operation of Manufacturing Systems: State of the Art and New Trends.” International Journal of Production Research 58 (7): 1927–1949. doi:10.1080/00207543.2019.1636321.
   Web of Science ®Google Scholar
• Mourtzis, D. 2022. “Chapter 4 - the Mass Personalization of Global Networks.” In Design and Operation of Production Networks for Mass Personalization in the Era of Cloud Technology, edited by D. Mourtzis, 79–116. Amsterdam, Netherlands: Elsevier. doi:10.1016/B978-0-12-823657-4.00006-3. 978-0-12-823657-4.
   Google Scholar
• Mourtzis, D., J. Angelopoulos, and N. Panopoulos. 2022. “A Literature Review of the Challenges and Opportunities of the Transition from Industry 4.0 to Society 5.0.” Energies 15 (17): 6276. doi:10.3390/en15176276.
   Web of Science ®Google Scholar
• Nguyen-Duc-Thanh, N., S. Lee, and D. Kim. 2012. “Two-Stage Hidden Markov Model in Gesture Recognition for Human Robot Interaction.” International Journal of Advanced Robotic Systems 9 (2): 39. doi:10.5772/50204.
   Google Scholar
• Nikolakis, N., K. Alexopoulos, E. Xanthakis, and G. Chryssolouris. 2019. “The Digital Twin Implementation for Linking the Virtual Representation of Human-Based Production Tasks to Their Physical Counterpart in the Factory-Floor.” International Journal of Computer Integrated Manufacturing 32 (1): 1–12. doi:10.1080/0951192X.2018.1529430.
   Web of Science ®Google Scholar
• Nwakanma, C. I., F. Bushra Islam, M. Pratiwi Maharani, J.M. Lee, and D.S. Kim. 2021. “Detection and Classification of Human Activity for Emergency Response in Smart Factory Shop Floor.” Applied Sciences 11 (8): 3662. doi:10.3390/app11083662.
   Google Scholar
• Olivares-Alarcos, A., S. Foix, and G. Alenyà. 2019. “On Inferring Intentions in Shared Tasks for Industrial Collaborative Robots.” Electronics 8 (11): 1306. doi:10.3390/electronics8111306.
   Web of Science ®Google Scholar
• Papoutsis, A., G. Botilias, P. Karvelis, and C. Stylios. 2020. “A Machine Learning Approach for Human Activity Recognition.” pHealth 2020 273, 155–160. doi:10.3233/SHTI200631.
   Google Scholar
• Peng, Q., W. Chen, W. Xingming, and J. Wang. 2015. “A Novel Vision-Based Human Motion Capture System Using Dual-Kinect.” In 2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA), 15-17 June 2015, Auckland, New Zealand, 51–56. IEEE.
   Google Scholar
• Raatz, A., S. Blankemeyer, T. Recker, D. Pischke, and P. Nyhuis. 2020. “Task Scheduling Method for HRC Workplaces Based on Capabilities and Execution Time Assumptions for Robots.” CIRP Annals 69 (1): 13–16. doi:10.1016/j.cirp.2020.04.030.
   Web of Science ®Google Scholar
• Rekik, K., and J. Hoffmann. 2019. “Planning for Human-Robot Collaboration Using Markov Decision Processes. Proceedings of the Robotix Academy Conference for Industrial Robotics (RACIR'19), June 05, 2019, Liège, Belgium, Germany: Shaker Verlag. 33–37.
   Google Scholar
• Rettig, O., S. Müller, and M. Strand. 2022. “Determination of Posture Comfort Zones for Robot-Human Handover Tasks.” In Intelligent Autonomous Systems 16 Lecture Notes in Networks and Systems, edited by H. Marcelo, A. Jr, H. Asama, W. Lin, and S. Foong, 314–325. Cham: Springer International Publishing. doi:10.1007/978-3-030-95892-3_24.
   Google Scholar
• Riedelbauch, D., S. Schweizer, and D. Henrich. 2019. “Skill Interaction Categories for Communication in Flexible Human-Robot Teams.” In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 03-08 November 2019, Macau, China, 3810–3816. IEEE. doi:10.1109/IROS40897.2019.8967901.
   Google Scholar
• Roetenberg, D., H. Luinge, and P. Slycke. 2009. “Xsens MVN: Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors.” Xsens Motion Technology BV Technology Reproductive 3, 1–7.
   Google Scholar
• Sandini, G., A. Sciutti, and D. Vernon. 2021. In Encyclopedia of Robotics, edited by Ang, Marco H., Khatib, Oussama, Siciliano, Bruno, 1–7. Berlin, Heidelberg: Springer. doi:10.1007/978-3-642-41610-1_198-1.
   Google Scholar
• Scott, D. W. 2015. Multivariate Density Estimation: Theory, Practice, and Visualization. 336. Canada: John Wiley & Sons. 978047154770. doi:10.1002/9780470316849.
   Google Scholar
• Shum, H. P. H., E. S. L. Ho, Y. Jiang, and S. Takagi. 2013. “Real-Time Posture Reconstruction for Microsoft Kinect.” IEEE Transactions on Cybernetics 43 (5): 1357–1369. doi:10.1109/TCYB.2013.2275945.
   PubMed Web of Science ®Google Scholar
• Smids, J., S. Nyholm, and H. Berkers. 2020. “Robots in the Workplace: A Threat to— or Opportunity For—meaningful Work?” Philosophy & Technology 33 (3): 503–522. doi:10.1007/s13347-019-00377-4.
   Google Scholar
• Steinmetz, F., A. Wollschläger, and R. Weitschat. 2018. “Razer—a HRI for Visual Task-Level Programming and Intuitive Skill Parameterization.” IEEE Robotics and Automation Letters 3 (3): 1362–1369. doi:10.1109/LRA.2018.2798300.
   Google Scholar
• Tortora, S., S. Michieletto, F. Stival, and E. Menegatti. 2019. “Fast Human Motion Prediction for Human-Robot Collaboration with Wearable Interface.” In 2019 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), 18-20 November 2019, Bangkok, Thailand, Nov, 457–462. doi:10.1109/CIS-RAM47153.2019.9095779.
   Google Scholar
• Tuli, T. B., L. Kohl, S. Adugna Chala, M. Manns, and F. Ansari. 2021. “Knowledge-Based Digital Twin for Predicting Interactions in Human-Robot Collaboration.” In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 07-10 September 2021, Vasteras, Sweden, Sep, 1–8. IEEE. doi:10.1109/ETFA45728.2021.9613342.
   Google Scholar
• Tuli, T. B., M. Manns, and S. Zeller. 2022. “Human Motion Quality and Accuracy Measuring Method for Human–Robot Physical Interactions.” Intelligent Service Robotics 15 (4): 503–512. doi:10.1007/s11370-022-00432-8.
   Web of Science ®Google Scholar
• Tuli, T. B., V. Mukesh Patel, and M. Manns. 2022. “Industrial Human Activity Prediction and Detection Using Sequential Memory Networks.” Proceedings of the Conference on Production Systems and Logistics: CPSL 2022,17-20 May, 2022, Vancouver, Canada: 62–72. Publish-Ing. doi:10.15488/12314.
   Google Scholar
• Uzunovic, T., E. Golubovic, Z. Tucakovic, Y. Acikmese, and A. Sabanovic. 2018. “Task-Based Control and Human Activity Recognition for Human-Robot Collaboration.” In IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society, 21-23 October 2018, Washington, DC, USA, Oct, 5110–5115. IEEE. doi:10.1109/IECON.2018.8591206.
   Google Scholar
• Wang, L., S. Liu, H. Liu, and X. Vincent Wang. 2020. “Overview of Human-Robot Collaboration in Manufacturing.” In Proceedings of 5th International Conference on the Industry 4.0 Model for Advanced Manufacturing, edited by L. Wang, V. D. Majstorovic, D. Mourtzis, E. Carpanzano, G. Moroni, and L. M. Galantucci, Lecture Notes in Mechanical Engineering, Cham, 15–58. Springer International Publishing.
   Google Scholar
• Wang, Z., A. Peer, and M. Buss. 2009. “An HMM Approach to Realistic Haptic Human-Robot Interaction.” In World Haptics 2009 - Third Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 18-20 March 2009, Salt Lake City, UT, USA, 374–379. IEEE. doi:10.1109/WHC.2009.4810835.
   Google Scholar
• Wang, L., and X. Vincent Wang. 2018. “Context-Aware Human-Robot Collaborative Assembly.” In Cloud-Based Cyber-Physical Systems in Manufacturing, edited by L. Wang and X. V. Wang, 261–294. Cham: Springer International Publishing.
   Google Scholar
• Xin, M., F. Tao, M. Zhang, T. Wang, and Y. Zuo. 2019. “Digital Twin Enhanced Human-Machine Interaction in Product Lifecycle.” Procedia CIRP 83: 789–793. doi:10.1016/j.procir.2019.04.330.
   Google Scholar
• Zolotová, I., P. Papcun, E. Kajáti, M. Miškuf, and J. Mocnej. 2020. “Smart and Cognitive Solutions for Operator 4.0: Laboratory H-CPPS Case Studies.” Computers & Industrial Engineering 139: 105471. doi:10.1016/j.cie.2018.10.032.
   Web of Science ®Google Scholar