The Effect of Audiovisual Spatial Design on User Experience of Bare-Hand Interaction in VR

Abstract

This study investigated whether the audio-visual spatial design could improve and how it improved the user experience of bare-hand interaction (BHI) in virtual reality (VR), by adjusting spatial audio accordingly (no sound, audiovisual spatial congruency, audio-visual spatial in congruency). We asked 32 participants to complete the grasping task in two interaction scenarios with three audio-visual spatial conditions. We evaluated the user experience in terms of both intrinsic cognitive load and usability through four metrics: subjective cognitive load scores, HbO₂ data in the corresponding brain regions, task time, and user satisfaction. The results showed that in near range interaction, although there was no significant difference between audio-visual spatial in congruency and congruency in terms of cognitive load and usability, they both significantly improved the user experience compared to the no sound group. In distant range interaction, audio-visual spatial in congruency significantly reduced the cognitive load and increased usability, thus improving the user experience.

Acknowledgements

We are grateful to Shandong University for the support of the experimental site and equipment.

Ethics statement

This study was approved by the Human Ethics Committee of Shandong University and conducted in the Industrial Design Science Laboratory, School of Mechanical Engineering, Shandong University. The experiments were performed according to the Declaration of Helsinki. All participants voluntarily participated in this study and gave informed consent before starting the experiment.

Authors’ contribution

SZ was involved in experimental design, software development and draft writing. YL was involved in the conception experiment and reviewed the original draft. FS contributed to project administration and original draft review. DY was involved in the draft writing and data sorting. ZB contributed to the draft writing and experimental design. ZZ contributed to the draft writing and experimental design.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was funded by the key project of Shandong University Education Teaching Reform Research [2022Z46]. This information is available upon request.

Notes on contributors

Shiyi Zhang

Shiyi Zhang received the bachelor’s degree in industrial design from the School of Art, Shandong University, Jinan, Shandong, China, in 2020. She is currently pursuing the Master of Mechanical Engineering, Shandong University, Jinan, Shandong, China. Her research interests are VR interaction design, usability and user experience.

Yan Liu

Yan Liu received the Doctor of Arts degree in folk art from the School of Arts, Shandong University, Jinan, Shandong, China. She is currently a Professor and a PhD Supervisor with the School of Mechanical Engineering, Shandong University, and the Director of the Department of Industrial Design, School of Mechanical Engineering.

Fanghao Song

Fanghao Song received the Doctor of Arts degree in aesthetic cultures from the Aesthetics of Literature and Art Research Centre, Shandong University, Jinan, Shandong. He is currently a Professor and a PhD Supervisor with the School of Mechanical Engineering, Shandong University, and the Director of the Institute of Modern Industrial Design.

Dong Yu

Dong Yu received the bachelor’s degree in industrial design from the University of Jinan, Jinan, Shandong, China, in 2018. He is currently pursuing the master’s degree with the School of Mechanical Engineering, Shandong University, Jinan. His research interests include industrial design and interaction design research.

Zhenming Bo

Zhenming Bo received the bachelor’s degree in industry design from the Mechanical Engineering College, Guizhou University. He is currently pursuing the Master of Mechanical Engineering, Shandong University, Jinan, Shandong, China. His research interests include user experience research, human–computer interaction, and VR interaction design research.

Zihan Zhang

Zihan Zhang received the bachelor’s degree in art of visual communication design from the School of Art, Shandong University, Jinan, Shandong, China. She is currently pursuing the PhD degree with the School of Mechanical Engineering, Shandong University, Jinan, Shandong, China. Her research interests include human–computer interaction and virtual presence.