Hand Gesture Recognition of Methods-Time Measurement-1 Motions in Manual Assembly Tasks Using Graph Convolutional Networks

References

• Agethen, P., M. Otto, F. Gaisbauer, and E. Rukzio. 2016. Presenting a novel motion capture-based approach for walk path segmentation and drift analysis in manual assembly. Procedia CIRP 52:286–1303. https://doi.org/10.1016/j.procir.2016.07.048. doi:10.1016/j.procir.2016.07.048.
   Google Scholar
• Almeida, D., and J. Ferreira. 2009. Analysis of the Methods Time Measurement (MTM) methodology through its application in manufacturing companies. 19th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM 2009), Middlesbrough, United Kingdom. Middlesbrough, 10.13140/RG.2.1.2826.1927.
   Google Scholar
• Baines, T., L. Hadfield, S. Mason, and J. Ladbrook. 2003. Using empirical evidence of variations in worker performance to extend the capabilities of discrete event simulations in manufacturing. Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No. 03EX693), Xi'an, China, 1210–16. New Orleans, IEEE. 10.1109/WSC.2003.1261552.
   Google Scholar
• Bokranz, R., and K. Landau. 2012. Handbuch Industrial Engineering: Produktivitätsmanagement Mit MTM. 2nd ed. Stuttgart: Schäffer-Poeschel.
   Google Scholar
• Bures, M., and P. Pivodova. 2015. Comparison of time standardization methods on the basis of real experiment. Procedia Engineering 100: 466–74. doi:10.1016/j.proeng.2015.01.392.
   Google Scholar
• Cao, Z., G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. 2021. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (1):172–86. doi:10.1109/TPAMI.2019.2929257.
   PubMed Web of Science ®Google Scholar
• Chryssolouris, G., D. Mavrikios, D. Fragos, and V. Karabatsou. 2000. A virtual reality-based experimentation environment for the verification of human-related factors in assembly processes. Robotics and Computer-Integrated Manufacturing 16 (4):267–76. doi:10.1016/S0736-5845(00)00013-2.
   Web of Science ®Google Scholar
• Fan, J., and J. Dong. 2003. Intelligent virtual assembly planning with integrated assembly model. SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No. 03CH37483), Washington, DC, USA, 5: 4803–8vol.5. doi:10.1109/ICSMC.2003.1245743.
   Google Scholar
• Fang, W., and L. Zheng. 2020. Shop floor data-driven spatial–temporal verification for manual assembly planning. Journal of Intelligent Manufacturing 31 (4):1003–18. doi:10.1007/s10845-019-01491-y.
   Web of Science ®Google Scholar
• Fantoni, G., S. Q. Al-Zubaidi, E. Coli, and D. Mazzei. 2021. Automating the process of method-time-measurement. International Journal of Productivity and Performance Management 70 (4):958–82. doi:10.1108/IJPPM-08-2019-0404.
   Web of Science ®Google Scholar
• Foret, P., A. Kleiner, H. Mobahi, and B. Neyshabur. 2020. Sharpness-Aware minimization for efficiently improving generalization, October. http://arxiv.org/abs/2010.01412.
   Google Scholar
• Genaidy, A. M., A. Mital, and M. Obeidat. 1989. The validity of predetermined motion time systems in setting production standards for industrial tasks. International Journal of Industrial Ergonomics 3 (3):249–63. doi:10.1016/0169-8141(89)90025-5.
   Google Scholar
• Gomes de Sá, A., and G. Zachmann. 1999. Virtual reality as a tool for verification of assembly and maintenance processes. Computers & Graphics 23 (3):389–403. doi:10.1016/S0097-8493(99)00047-3.
   Web of Science ®Google Scholar
• Hochreiter, S., and J. Schmidhuber. 1997. Long short-term memory. Neural Computation 9 (8):1735–80. doi:10.1162/neco.1997.9.8.1735.
   PubMed Web of Science ®Google Scholar
• Keskar, N. S., D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. Peter Tang. 2016. On large-batch training for deep learning: generalization gap and sharp minima, September. http://arxiv.org/abs/1609.04836.
   Google Scholar
• Kreiss, S., L. Bertoni, and A. Alahi. 2019. PifPaf: Composite fields for human pose estimation. March. http://arxiv.org/abs/1903.06593.
   Google Scholar
• Li, S., T. Peng, X. Chi, F. Yan, and Y. Liu. 2009. A mixed reality-based assembly verification and training platform BT - virtual and mixed reality . In ed. R. Shumaker, Virtual and Mixed Reality, 576–85. Berlin, Heidelberg: Springer Berlin Heidelberg.
   Google Scholar
• Li, Y., H. Zihang, Y. Xiang, H. Zuguo, and K. Han. 2019. Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition. EURASIP Journal on Image and Video Processing 2019 (1):78. doi:10.1186/s13640-019-0476-x.
   Web of Science ®Google Scholar
• Lin, T.-Y., P. Goyal, R. Girshick, H. Kaiming, and D. Piotr. 2017. Focal loss for dense object detection. August. http://arxiv.org/abs/1708.02002.
   Google Scholar
• Liu, J., G. Wang, H. Ping, L.-Y. Duan, and A. C. Kot. 2017. Global context-aware attention LSTM networks for 3D action recognition. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 3671–80. IEEE. 10.1109/CVPR.2017.391.
   Google Scholar
• Liu, L., Y.-C. Lin, and J. Reid. 2019. Improving the performance of the LSTM and HMM model via hybridization. July. http://arxiv.org/abs/1907.04670.
   Google Scholar
• Manns, M., M. Otto, and M. Mauer. 2016. Measuring motion capture data quality for data driven human motion synthesis. Procedia CIRP 41:945–50. doi:10.1016/j.procir.2015.12.068.
   Google Scholar
• Masood, T., and J. Egger. 2019. Augmented reality in support of industry 4.0—implementation challenges and success factors. Robotics and Computer-Integrated Manufacturing 58:181–95. doi:10.1016/j.rcim.2019.02.003.
   Web of Science ®Google Scholar
• Maynard, H. B., G. J. Stegemerten, and J. L. Schwab. 1948. Methods-Time Measurement. Methods-Time Measurement. New York, NY, US: McGraw-Hill.
   Google Scholar
• Menolotto, M., D.-S. Komaris, S. Tedesco, B. O’Flynn, and M. Walsh. 2020. Motion capture technology in industrial applications: A systematic review. Sensors 20 (19):5687. doi:10.3390/s20195687.
   PubMed Web of Science ®Google Scholar
• Min, Y., Y. Zhang, X. Chai, and X. Chen. 2020. An efficient pointLSTM for point clouds based gesture recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 5760–69. IEEE. doi:10.1109/CVPR42600.2020.00580.
   Google Scholar
• Ovur, S. E., S. Hang, Q. Wen, E. De Momi, and G. Ferrigno. 2021. Novel adaptive sensor fusion methodology for hand pose estimation with multileap motion. IEEE Transactions on Instrumentation and Measurement 70:1–8. doi:10.1109/TIM.2021.3063752.
   PubMed Web of Science ®Google Scholar
• Paszke, A., S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S. 2019. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , edited by H. Wallach, H. Larochelle, and A. Beygelzimer, F d\textquotesingle Alchéd\textquotesingle AlchéBuc, E Fox, and R Garnett. Vol. 32. Curran Associates, Inc., Vancouver, Canada. https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
   Google Scholar
• Pham, V. T., Q. Qiu, A. A. P. Wai, and J. Biswas. 2007. Application of ultrasonic sensors in a smart environment. Pervasive and Mobile Computing 3 (2):180–207. doi:10.1016/j.pmcj.2006.07.002.
   Web of Science ®Google Scholar
• Polotski, V., Y. Beauregard, and A. Franzoni. 2019. Combining predetermined and measured assembly time techniques: Parameter estimation, regression and case study of fenestration industry. International Journal of Production Research 57 (17):5499–519. doi:10.1080/00207543.2018.1530469.
   Web of Science ®Google Scholar
• Prabhu, V. A., B. Song, J. Thrower, A. Tiwari, and P. Webb. 2016. Digitisation of a moving assembly operation using multiple depth imaging sensors. The International Journal of Advanced Manufacturing Technology 85 (1–4):163–84. doi:10.1007/s00170-015-7883-7.
   Web of Science ®Google Scholar
• Puthenveetil, S. C., C. P. Daphalapurkar, W. Zhu, M. C. Leu, X. F. Liu, J. K. Gilpin-Mcminn, and S. D. Snodgrass. 2015. Computer-automated ergonomic analysis based on motion capture and assembly simulation. Virtual Reality 19 (2):119–28. doi:10.1007/s10055-015-0261-9.
   Web of Science ®Google Scholar
• Qi, W., S. Hang, and A. Aliverti. 2020. A smartphone-based adaptive recognition and real-time monitoring system for human activities. IEEE Transactions on Human-Machine Systems 50 (5):414–23. doi:10.1109/THMS.2020.2984181.
   Web of Science ®Google Scholar
• Rabiner, L. R. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77 (2):257–86. doi:10.1109/5.18626.
   Web of Science ®Google Scholar
• Roetenberg, D., H. Luinge, and P. J. Slycke. 2009. Xsens MVN: Full 6DOF human motion tracking using miniature inertial sensors. In.
   Google Scholar
• Rozemberczki, B., P. Scherer, H. Yixuan, G. Panagopoulos, A. Riedel, M. Astefanoaei, O. Kiss, Ferenc, B., Lopez, G., Collignon, N., Sarkar, R. 2021. PyTorch geometric temporal: Spatiotemporal signal processing with neural machine learning models CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge ManagementGold Coast, Queensland, Australia. April. doi:10.1145/3459637.3482014.
   Google Scholar
• Rude, D. J., S. Adams, and P. A. Beling. 2018. Task recognition from joint tracking data in an operational manufacturing cell. Journal of Intelligent Manufacturing 29 (6):1203–17. doi:10.1007/s10845-015-1168-8.
   Web of Science ®Google Scholar
• Salmi, A., P. David, E. Blanco, and J. D. Summers. 2016. A review of cost estimation models for determining assembly automation level. Computers & Industrial Engineering 98:246–59. doi:10.1016/j.cie.2016.06.007.
   Web of Science ®Google Scholar
• Shafaei, A., and J. J. Little. 2016. “Real-time human motion capture with multiple depth cameras.2016 13th conference on computer and Robot Vision (CRV), Victoria, BC, Canada, 24–31. IEEE. doi:10.1109/CRV.2016.25.
   Google Scholar
• Shi, L., Y. Zhang, J. Cheng, and L. Hanqing 2018. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. May. http://arxiv.org/abs/1805.07694.
   Google Scholar
• Stump, B., and F. Badurdeen. 2012. Integrating lean and other strategies for mass customization manufacturing: A case study. Journal of Intelligent Manufacturing 23 (1):109–24. doi:10.1007/s10845-009-0289-3.
   Web of Science ®Google Scholar
• Su, H., S. E. Ovur, X. Zhou, Q. Wen, G. Ferrigno, and E. De Momi. 2020. Depth vision guided hand gesture recognition using electromyographic signals. Advanced Robotics 34 (15):985–97. doi:10.1080/01691864.2020.1713886.
   Web of Science ®Google Scholar
• Tadayon, M., and G. Pottie. 2020. Comparative analysis of the hidden Markov model and LSTM: A simulative approach. August. http://arxiv.org/abs/2008.03825.
   Google Scholar
• Tran, D., H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. 2018. A closer look at spatiotemporal convolutions for action recognition.2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 6450–59. IEEE. doi:10.1109/CVPR.2018.00675.
   Google Scholar
• Wang, X., S. K. Ong, and A. Y. C. Nee. 2016. Real-virtual components interaction for assembly simulation and planning. Robotics and Computer-Integrated Manufacturing 41 (October):102–14. doi:10.1016/j.rcim.2016.03.005.
   Google Scholar
• Yan, S., Y. Xiong, and D. Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition, January. http://arxiv.org/abs/1801.07455.
   Google Scholar
• Yin, X.-Q., W. Tao, Y. Feng, H.-W. Yang, H. Qiao-Zhi, and H. Zhao. 2019. A robust and accurate breakpoint detection method for line-structured laser scanner. Optics & Laser Technology 118:52–61. doi:10.1016/j.optlastec.2019.03.037.
   Web of Science ®Google Scholar
• Zandin, K. B. 2020. MOST® work measurement systems. Edited by T. M. Schmidt, Fourth edition Boca Raton, FL: CRC Press. 2021. | On title page the registered trademark follows “MOST” in the title.: CRC Press. doi:10.1201/9780429326424.
   Google Scholar
• Zhang, F., V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. 2020. MediaPipe Hands: On-device real-time hand tracking, June. http://arxiv.org/abs/2006.10214.
   Google Scholar
• Zhang, P., C. Lan, J. Xing, W. Zeng, J. Xue, and N. Zheng. 2017. View adaptive recurrent neural networks for high performance human action recognition from skeleton data.2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2136–45. IEEE. doi:10.1109/ICCV.2017.233.
   Google Scholar
• Zhong, R. Y., Q. Y. Dai, T. Qu, G. J. Hu, and G. Q. Huang. 2013. RFID-enabled real-time manufacturing execution system for mass-customization production. Robotics and Computer-Integrated Manufacturing 29 (2):283–92. doi:10.1016/j.rcim.2012.08.001.
   Web of Science ®Google Scholar