2,643
Views
12
CrossRef citations to date
0
Altmetric
Research Article

Digital twin for human-machine interaction with convolutional neural network

, , , , &
Pages 888-897 | Received 30 Mar 2019, Accepted 29 Mar 2021, Published online: 31 May 2021
 

ABSTRACT

Digital twin (DT) technology aims to create a virtual model of a physical entity and efficiently analyze the intelligent manufacturing system. Based on the DT, human-machine interaction (HMI) is a typical application. Deep learning technology is employed in the digital twin to realize and strengthen HMI while analyzing the physical and virtual data. The convolutional neural network (CNN) is used for analyzing visual information. For dealing with the HMI task in DT, two CNN models, Visual Geometry Group Network (VGG) and Residual Network (ResNet), are adopted. Modified 3D-VGG and 3D-ResNet models are proposed in this paper, which is an improvement over existing VGG and ResNet models. The models focus on humans’ information in videos that are captured with the HMI system’s sensors. The information, which can be regarded as the digital twin data, includes the action and the position of the human skeleton. Additionally, the proposed models are end-to-end. The experiments show that both models perform well on the human motion recognition task. The model can effectively generate skeletal data from video data. With the generated information, the human and the machine can interact well with the aid of the digital twin data analysis.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [61972016, 62032016]; Beijing Natural Science Foundation [L191007, JQ19011]; Beijing Science and Technology Major Project [Z191100002719004]; Fundamental Research Funds for the Central Universities [YWF-20-BJ-J-612].

Log in via your institution

Log in to Taylor & Francis Online

PDF download + Online access

  • 48 hours access to article PDF & online version
  • Article PDF can be downloaded
  • Article PDF can be printed
USD 61.00 Add to cart

Issue Purchase

  • 30 days online access to complete issue
  • Article PDFs can be downloaded
  • Article PDFs can be printed
USD 528.00 Add to cart

* Local tax will be added as applicable

Related Research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.