Perception and Decision-Making for Multi‑Modal Interaction Based on Fuzzy Theory in the Dynamic Environment

Abstract

The projection-augmented reality assembly assistance system, which supports natural multi-modal human–computer interaction, has the potential to improve interaction efficiency. However, the effectiveness of the interaction method is compromised by the interference caused by the dynamically changing industrial environment. To address this issue, this study proposed a multi-model interaction model based on fuzzy theory and developed an environment perception decision-making interaction system. This system continuously monitors real-time environment data, evaluates interaction performance, and visually recommends the optimal method to assist users in completing their tasks. A comparative experiment was conducted between the proposed system and a non-environment perception interaction system. The collected data, including interaction success rate, task time, system usability, and user experience, were analyzed. The results demonstrated that the proposed system effectively enhances the adaptability and comfort of computer systems in complex environments and improves communication efficiency between humans and machines.

Keywords:

• Projection augmented reality
• human–computer interaction
• multi-modal interaction
• fuzzy theory
• environment perception

Acknowledgments

This work was supported by National Key R&D Program of China (Grant Nos. 2019YFB1703800, 2021YFB1716200, 2021YFB1714900), the National Natural Science Foundation of China (Grant Nos. 52375515, 52275513), the Programme of Introducing Talents of Discipline to Universities (111 Project), China (Grant No. B13044), the Fundamental Research Funds for the Central Universities, NPU (Grant No. 3102020gxb003).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Jie Zhang

Jie Zhang is a Ph.D. student in the School of Mechanical Engineering at Northwestern Polytechnical University. His research interests include human-computer interaction, projection augmented reality, assembly guidance, and part code identification.

Shuxia Wang

Shuxia Wang is a professor in the School of Mechanical Engineering at Northwestern Polytechnical University. She is interested in human-computer interaction, cognitive computing, assembly guidance, virtual training, EEG, and AR/VR/MR.

Weiping He

Weiping He is a professor in the School of Mechanical Engineering at Northwestern Polytechnical University. He is interested in geometrical modeling, 2D bar-code recognition, direct part marking and automatic identification, AR, and MR.

Jianghong Li

Jianghong Li is a Ph.D. student in the School of Mechanical Engineering at Northwestern Polytechnical University. Her research interests include eye movement interaction, cognitive computing, and assembly guidance.

Shixin Wu

Shixin Wu is a Ph.D. student in the School of Mechanical Engineering at Northwestern Polytechnical University. His research interests include deep learning, assembly guidance, and part code identification.

Zhiwei Cao

Zhiwei Cao was a Ph.D. student in the School of Mechanical Engineering at Northwestern Polytechnical University. His research interests include projection augmented reality, gesture interaction, and natural interaction.

Manxian Wang

Manxian Wang is an expert in AECC Xi’an Aero-Engine Ltd. His research interests include aero-engine assembly, projection augmented reality, and field application.