Multimodal interaction in virtual reality supported education: a systematic review

ABSTRACT

This study reviews empirical studies on applying Multimodal Interactions in Virtual Reality (MIVR) environments in education. We selected 25 papers and reviewed them based on (1) educational research domains, (2) MIVR modalities, (3) MIVR devices/tools, and (4) the methodology and major findings. The results suggest that MIVR focused on language and mathematical science. The common modalities were Visual + Haptic and Visual + Auditory. The mature haptic feedback devices and Virtual Reality (VR) app have advantages in devices. In methodology and key findings, the participants were primarily from middle school and university; the duration of the studies commonly less than one day, and the number of participants were mostly less than 50; the research design more used quasi-experimental; the research methods were quantitative and mixed; the common data collection approaches were tests and questionnaires; scholars evaluated learners’ knowledge, skills development, learning experience, learning attitude, and opinion about the MIVR system. The advantages of MIVR in education include natural and real learning experiences, flexibility and efficiency in obtaining information etc., disadvantages were availability issues, cognitive load and personalized insufficiency.

KEYWORDS:

• A systematic review
• multimodal interaction
• VR
• education
• learning effect

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural science foundation of China (NSFC) project “Shared Cluster Structure Mining and application Research for Multi-view Educational Data” [grant number: 62277043]; the National Natural Science Foundation of China (NSFC) Project “Research on Key Technologies of Deep Cognitive Tracking and Learning Intervention Model in VR Environment” [grant number: 62277044]; and the Chongqing Graduate Research Innovation Project “Design and Application Research of Multimodal Learning Activities in Virtual Reality Environment” [grant number: CYB22093].

Notes on contributors

Hanlin Hu

Hanlin Hu, is a doctoral student at Southwest University. He is currently involved in research on VR technology-supported learning and Smart learning environment.

Geping Liu

Geping Liu, is a professor of educational technology at Southwest University. His research interests cover VR technology-supported learning, smart learning environment and online education system.

Tao Xie

Tao Xie, is an associate professor of educational technology at Southwest University. His research interests cover on the key technology and application of intelligent education.