Multiparty conversation via multirobot system: incorporation of nonverbal user responses for continued conversation

Abstract

Recent years have seen the advent of conversational humanoid robots. Implementing multiple robots is a promising approach for continuing human–robot conversations when a robot encounters speech-recognition errors. This strategy is effective for the user's verbal responses; however, people generally use nonverbal responses, such as nods or smiles, to respond to the interlocutor. In this study, we proposed a conversational strategy for twin robots in which the second robot recognized a human's nonverbal responses and interrupted the conversation by mentioning these responses. Moreover, we developed an interrogative dialogue system using simple nonverbal recognition modules. To verify the effectiveness of this conversational strategy, a subject-based experiment was conducted in which humans conversed with the two robots, and their impressions were evaluated through a questionnaire. We compared the outcomes under two conditions: whether the second robot's interruption was inserted or otherwise. Our results indicate that language-only ineffectiveness in responding to ambiguous responses regarding satisfaction and comprehension was mitigated. Thus, such a dialogue system is advantageous because it can facilitate robust conversation without relying solely on speech recognition.

GRAPHICAL ABSTRACT

Keywords:

• Human-robot interaction
• social robots
• multiple robots
• dialogue strategy

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API

Additional information

Funding

This work was supported by JSPS KAKENHI Grant Number JP19H05691, and JST, Moonshot R&D Grant Number JPMJPS2011.

Notes on contributors

Kazuki Sakai

Kazuki Sakai received a Ph.D. degree in engineering science from Osaka University, Japan, in 2020. He is currently a specially appointed assistant professor of the Department of Systems Innovation in the Graduate School of Engineering Science at Osaka University. His research interests include human-robot interaction and dialogue systems.

TingHao Hsieh

TingHao Hsieh received his master's degree in engineering from Osaka University, Japan, in 2020. From 2018 to 2020, he was a graduate student at Intelligent Robotics and Communication Laboratories. His research interests include human–robot interaction and human nonverbal behavior detection systems.

Yuichiro Yoshikawa

Yuichiro Yoshikawa received a Ph.D. degree in engineering from Osaka University, Japan, in 2005. From 2005, he was a Researcher at Intelligent Robotics and Communication Laboratories, Advanced Telecommunications Research Institute International. Since 2006, he has been a Researcher at Asada Synergistic Intelligence Project, ERATO, Japan Science and Technology Agency. He is a member of the Japanese Society of Robotics, the Japanese Society of Cognitive Science, the Virtual Reality Society of Japan, the Japanese Society for Child and Adolescent Psychiatry, and the Japanese Society of Pediatric Psychiatry and Neurology.

Hiroshi Ishiguro

Hiroshi Ishiguro received a D. Eng. in systems engineering from Osaka University, Japan in 1991. He is currently Professor of Department of Systems Innovation in the Graduate School of Engineering Science at Osaka University (2009-) and a Distinguished Professor of Osaka University (2017-). He is also visiting Director (2014-) (group leader: 2002–2013) of Hiroshi Ishiguro Laboratories at the Advanced Telecommunications Research Institute and an ATR fellow. His research interests include sensor networks, interactive robotics, and android science.