Cognitive Shifts in Bilingual Speakers Affect Speech Interactions with Artificial Agents

Abstract

A major research question in psycholinguistics relates to the phenomenon of linguistic relativity, which contends that the language one speaks influences how one thinks. Of particular interest is whether bilingual speakers shift cognitive paradigms when speaking different languages. Here, we conducted a human-agent interaction (HAI) study using a bilingual virtual avatar capable of autonomous speech during cooperative gameplay in two languages (Korean and English). We ran 40 participants, including 20 monolingual speakers (10 Korean, 10 English) and 20 Korean/English bilingual speakers, engaging the avatar during 30-minute game sessions. Comparison of speech patterns showed that bilingual speakers exhibited notable “cognitive shifts” in both languages while interacting with the avatar, which were markedly different from their monolingual counterparts. Interestingly, the virtual avatar’s own speech behavior also significantly changed during interaction with bilingual speakers, despite identical programming. As evidenced here, such cognitive shifts appear to impact the way humans interact with artificial agents.

Keywords:

• Human-robot interaction
• social cognition
• bilingualism
• virtual avatar
• linguistic relativity
• cross-cultural robotics

Acknowledgments

We would also like to thank our various collaborators, including Jaeyoung Suh, Jihong Jeong, and Seeun Lee (Hanyang University), for their assistance in this work.

Data availability statement

The data used in this study included audiovisual recordings of participants during the experiment. However, it may be made available in de-identifiable form upon reasonable request.

Disclosure statement

The authors report there are no competing interests to declare.

Additional information

Funding

This work was supported by a grant from the National Research Foundation of Korea (NRF) under grant number: 2021R1G1A1003801.

Notes on contributors

Casey C. Bennett

Casey C. Bennett is an Associate Professor in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, as well as DePaul University in Chicago IL (USA). He specializes in artificial intelligence and robotics in healthcare, including use of data science and machine learning to create better human-robot interaction.

Say Young Kim

Say Young Kim is an Associate Professor in the Department English Language & Literature at Hanyang University in Seoul, Korea. He specializes in the study of language and bilingualism. He completed his PhD at Maryland University in the US.

Benjamin Weiss

Benjamin Weiss is a researcher at Quality and Usability Lab at Technische Universität Berlin in Germany and has also worked at Hanyang University in Seoul Korea previously. He specializes in natural language processing and speech systems. He received his PhD from Humboldt University in Berlin Germany.

Young-Ho Bae

Young-Ho Bae is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.

Jun Hyung Yoon

Jun Hyung Yoon is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.

Yejin Chae

Yejin Chae is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).

Eunseo Yoon

Eunseo Yoon is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).

Uijae Ryu

Uijae Ryu is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).

Hansae Cho

Hansae Cho is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.

Yesung Shin

Yesung Shin is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.