An Intelligent Shadow Play System With Multi-dimensional Interactive Perception

Abstract

We developed an intelligent shadow play system (ShadowTouch) based on multi-dimensional interactive perception technologies to let user interact with the shadow play figure naturally. ShadowTouch included a shadow play hardware subsystem and a multi-dimensional interactive software subsystem. In shadow play hardware subsystem, each joint of the shadow play figure was actuated by the connecting rod which was rotated through one steering gear. In multi-dimensional interactive software subsystem, a somatosensory interaction module synchronized motions between shadow play figure and human body based on skeleton point detection, and an automatic choreography module generated the motions automatically for the shadow play figure based on musical emotion recognition. The proposed system improved the immersive interaction experience and let users feel involved during the performance of shadow play, and the motions of shadow play figure (steering gear motion angles) could be saved digitally to further be applied for the reactivation and inheritance of traditional culture.

Disclosure statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Funding

This work was supported by the Philosophy and Social Science Planning Fund Project of Zhejiang Province [22NDJC007Z, 20NDQN260YB], the Natural Science Foundation of Zhejiang Province [LY20F020028], the Key Research and Development Program of Zhejiang Province [2022C03148], and the Fundamental Research Funds for the Provincial Universities of Zhejiang [GB201901006].

Notes on contributors

Zhichuan Tang

Zhichuan Tang received a PhD degree in Computer Science at Zhejiang University, China. He is an associate professor at Zhejiang University of Technology, China. His research interests include human-computer interface, deep learning and artificial intelligence.

Yidan Hu

Yidan Hu is currently a master student majoring in industrial design at Zhejiang University of Technology, China. Her research interests include human-computer interface and intelligent design.

Weining Weng

Weining Weng is currently a Ph.D. candidate in Computer Science at Chinese Academy of Sciences, China. His research interests include deep learning, affective computing and pattern recognition.

Lekai Zhang

Lekai Zhang received a PhD degree in Computer Science at Zhejiang University, China. He is a lecturer at Zhejiang University of Technology, China. His research interests include human-computer interface and affective computing.

Lingtao Zhang

Lingtao Zhang is currently a master student majoring in industrial design at Zhejiang University of Technology, China. His research interests include human-computer interface and deep learning.

Jichen Ying

Jichen Ying is currently a master student majoring in industrial design at Zhejiang University of Technology, China. His research interests include human-computer interface and intelligent system.