Foreign language acquisition via artificial intelligence and extended reality: design and evaluation

Abstract

Artificial Intelligence (AI) and Extended Reality (XR) have been employed in several foreign language education applications to increase the availability of experiential learning methods akin to international immersion programs. However, research in multi-modal spoken dialogue in L2 combined with immersive technologies and collaborative learning is thin, limiting students’ experiences to solo interactions focused mostly on vocabulary and grammar in such settings. We intend to fill this gap as we present the Cognitive Immersive Language Learning Environment (CILLE). The AI in CILLE can hear, see, and understand its users and can engage with them in non-dyadic multimodal conversations. The XR offers students a feeling of being somewhere else without the use of intrusive devices and supports multi-party, multi-modal interactions. Together, AI and XR create naturalistic conversational interactions targeted towards comprehensive foreign language acquisition. We evaluate CILLE as a Chinese-as-a-foreign-language (CFL) education tool through a seven-week, mixed-methods study with university students (N = 10). Results display statistical significance and retained improvement in CFL vocabulary, comprehension, and conversation skills. Coupled with an analysis of student feedback and researcher observations, we show how CILLE is designed and experienced by students to learn CFL.

Keywords:

• Artificial Intelligence
• augmented
• virtual and extended reality
• chatbots
• conversational agents
• foreign language acquisition
• human-computer interaction
• multi-modal interaction
• multi-party interaction
• spoken dialogue systems

Acknowledgements

The authors would like to thank Dr. Michael Wei, professor at the University of Missouri-Kansas City, for sharing his expertise in statistical analysis and for helping with user speaking test scoring. The authors would also like to thank RPI and IBM for funding this project as a part of their collaboration through the AI Horizons Network (AIHN)

Ethics statement

The user study and the experiment were conducted at Rensselaer Polytechnic Institute, Troy NY, USA. The user study and the experiment were approved by the Rensselaer Polytechnic Institute’s Institutional Review Board (IRB) under IRB # 1867. Rahul R. Divekar, Jaimie Drozdal and Prof. Yalun Zhou were the PI and Prof. Hui Su was the listed faculty advisor.

Notes

1 Hanzi are the logograms and most popular way of writing and reading Simplified Chinese.

2 Pinyin is the Romanized spelling of Chinese words that conveys pronunciation and tone.

3 https://www.ibm.com/cloud/watson-visual-recognition.

4 Demo Video: https://youtu.be/KBV9z9fLAD0.

5 The user study was approved by the university Institutional Review Board (IRB). Participants were compensated for their time with a gift card.

6 National Council of State Supervisors for Languages (NCSSFL) and the American Council on the Teaching of Foreign Languages (ACTFL) can do statements: https://www.actfl.org/publications/guidelines-and-manuals/ncssfl-actfl-can-do-statements.

Additional information

Notes on contributors

Rahul R. Divekar*

Rahul R. Divekar graduated with a Ph.D. in Computer Science from Rensselaer Polytechnic Institute where he did this work as a part of his dissertation. He has since joined Educational Testing Service (ETS) as an Associate Research Scientist and continues to work on the interdisciplinary field of computer-assisted language learning, AI, and user experiences. He can be reached at [email protected].

Jaimie Drozdal is a Research Specialist at the Cognitive Immersive Systems Lab at Rensselaer Polytechnic Institute. She holds a B.S. in Cognitive Science from RPI.

Samuel Chabot is a PhD student of Architectural Acoustics in the Cognitive and Immersive Systems Lab. His area of research focuses on the development and use of immersive virtual environments for human-scale, collaborative learning. He has worked with professors and classes of architecture, lighting, and language acquisition to best realize the potential of these spaces.

Yalun Zhou, Ph.D., is an Associate Professor of Communication and Media at Rensselaer Polytechnic Institute. Her research interests lie in the intersection of SLA, L2 pedagogy, and L2 users. Her recent research activities involve the instructional and user study design of Chinese learning with games and artificial intelligence in cognitive immersive learning environments.

Prof. Hui Su is the director of the Cognitive Immersive Systems Lab. Previously, he has worked at IBM for several years as the director of IBM Research Lab in Cambridge, MA.

Yue Chen is an undergraduate student at Rensselaer Polytechnic Institute.

Houming Zhu is an undergraduate student at Rensselaer Polytechnic Institute.

Prof. James Hendler is the Director of the Institute for Data Exploration and Applications and the Tetherless World Professor of Computer, Web and Cognitive Sciences at RPI. He is a Fellow of the AAAI, AAAS, ACM, IEEE and the National Academy of Public Administration.

Prof. Jonas Braasch is the Associate Director at the Cognitive Immersive Systems Lab (CISL). He is an Associate Professor at the School of Architecture at Rensselaer Polytechnic Institute, where he teaches in the Graduate Program in Architectural Acoustics. His research interests span binaural hearing, auditory modeling, multimodal integration, sensory substitution devices, aural architecture and creative processes in music improvisation.