2,643
Views
12
CrossRef citations to date
0
Altmetric
Research Article

Digital twin for human-machine interaction with convolutional neural network

, , , , &
Pages 888-897 | Received 30 Mar 2019, Accepted 29 Mar 2021, Published online: 31 May 2021
 

ABSTRACT

Digital twin (DT) technology aims to create a virtual model of a physical entity and efficiently analyze the intelligent manufacturing system. Based on the DT, human-machine interaction (HMI) is a typical application. Deep learning technology is employed in the digital twin to realize and strengthen HMI while analyzing the physical and virtual data. The convolutional neural network (CNN) is used for analyzing visual information. For dealing with the HMI task in DT, two CNN models, Visual Geometry Group Network (VGG) and Residual Network (ResNet), are adopted. Modified 3D-VGG and 3D-ResNet models are proposed in this paper, which is an improvement over existing VGG and ResNet models. The models focus on humans’ information in videos that are captured with the HMI system’s sensors. The information, which can be regarded as the digital twin data, includes the action and the position of the human skeleton. Additionally, the proposed models are end-to-end. The experiments show that both models perform well on the human motion recognition task. The model can effectively generate skeletal data from video data. With the generated information, the human and the machine can interact well with the aid of the digital twin data analysis.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [61972016, 62032016]; Beijing Natural Science Foundation [L191007, JQ19011]; Beijing Science and Technology Major Project [Z191100002719004]; Fundamental Research Funds for the Central Universities [YWF-20-BJ-J-612].

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.