An emotion-driven and topic-aware dialogue framework for human–robot interaction

ABSTRACT

Developing dialogue services for robots has been promoted nowadays for providing natural human–robot interactions to enhance user experiences as conversation is a key instrument for creating and maintaining their mutual relationships. In this work, we present a trainable framework for modeling context-aware human–robot dialogues, and apply it to a live streaming application to demonstrate the need for social robots. Our social chatting robot framework takes into account both emotional context and topic-focused content information to generate appropriate responses. This framework has some unique features. It adopts a multimodal deep learning model to recognize the participants’ emotions. Moreover, our work includes a topic-aware neural model that enables the social robot to follow certain conversation topics to fulfill the specific dialoguing goals and to enhance the chatting coherence. Most importantly, we design a strategy able to take emotions as the drive to control over the neural method of generating utterances. To evaluate the proposed approach, we have conducted several sets of quantitative and quantitative experiments. The results highlight the importance of multimodal emotion recognition and topic-awareness in dialoguing and confirm the feasibility and effectiveness of our framework for human–robot interaction.

GRAPHICAL ABSTRACT

KEYWORDS:

• Human–robot dialogue
• Live streaming
• Multimodal emotion recognition
• Topic modeling
• Deep learning

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research work was supported in part by the National Science and Technology Council of Taiwan: under Contract NSTC 112-2221-E-110-054.

Notes on contributors

I.-Chun Lu

I-Chun Lu received the M.A. degree from the Department of Information Management, National Sun Yat-sen University, Kaohsiung, Taiwan. Her research interests include machine learning, dialoguing system, and information retrieval.

Jhih-Yuan Huang

Jhih-Yuan Huang received the M.A. degree from the Department of Information Management, National Sun Yat-sen University, Kaohsiung, Taiwan, where he is currently pursuing the Ph.D. degree. His research interests include deep learning, natural language processing, machine vision, and parallel computation.

Wei-Po Lee

Wei-Po LEE received the Ph.D. degree in artificial intelligence from The University of Edinburgh, U.K. He is currently a Professor at the Department of Information Management, National Sun Yat-sen University, Kaohsiung, Taiwan. His research interests include artificial intelligence, autonomous robot, human–machine interaction and machine learning.