An emotion-driven and topic-aware dialogue framework for human–robot interaction

References

• Fortunati L, Cavallo F, Sarrica M. Multiple communication roles in human–robot interactions in public space. Int J Soc Robot. 2020;12:931–944. doi:10.1007/s12369-018-0509-0
   Web of Science ®Google Scholar
• Huang J-Y, Lee W-P, Lin T-A. Developing context-aware dialoguing services for a cloud-based robotic system. IEEE Access. 2019;7:44293–44306. doi:10.1109/ACCESS.2019.2905616
   Web of Science ®Google Scholar
• de Wit J, van der Kraan A, Theeuwes J. Live streams on twitch help viewers cope with difficult periods in life. Front Psychol. 2020;11:586975, doi:10.3389/fpsyg.2020.586975
   PubMed Web of Science ®Google Scholar
• Hepp A. Artificial companions, social bots and work bots: communicative robots as research objects of media and communication studies. Media, Culture Soc. 2020;42(7-8):1410–1426. doi:10.1177/0163443720916412
   Web of Science ®Google Scholar
• Khaund T, Kirdemir B, Agarwal N, et al. Social bots and their coordination during online campaigns: A survey. IEEE Trans. Comput Soc Sys 2022;9(2):530–545. doi:10.1109/TCSS.2021.3103515
   Web of Science ®Google Scholar
• Cheng X, Zhang X, Cohen J, et al. Human vs. human vs. AI: understanding the impact of anthropomorphism on consumer response to chatbots from the perspective of trust and relationship norms. Inform Proc Manag. 2022;59(3):102940, doi:10.1016/j.ipm.2022.102940
   Web of Science ®Google Scholar
• Huang M, Zhu X, Gao J. Challenges in building intelligent open-domain dialog systems. ACM Trans. Inform Sys. 2020;38(3) Article 21. doi:10.1145/3383123
   Web of Science ®Google Scholar
• Yan R, Li J, Yu Z. Deep learning for dialogue systems: chit-chat and beyond. Found Trends® in Inform Retriev. 2022;15(5):417–589. doi:10.1561/1500000083
   Web of Science ®Google Scholar
• Chen C-H, Lee W-P, Huang J-Y. Tracking and recognizing emotions in short text messages from online chatting services. Informat Process Manag. 2018;54(6):1325–1344. doi:10.1016/j.ipm.2018.05.008
   Web of Science ®Google Scholar
• Ma Y, Nguyen KL, Xing FZ, et al. A survey on empathetic dialogue systems. Inf Fusion. 2020;64:50–70. doi:10.1016/j.inffus.2020.06.011
   Web of Science ®Google Scholar
• Huang J-Y, Lee W-P. Exploring the effect of emotions in human–machine dialog: An approach toward integration of emotional and rational information. Knowl Based Syst. 2022;243:108425, doi:10.1016/j.knosys.2022.108425
   Web of Science ®Google Scholar
• Mai S, Hu H, Xing S. Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019: 481–492. doi:10.18653/v1/P19-1046
   Google Scholar
• Majumder N, Hazarika D, Gelbukh A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowl Based Syst. 2018;161:124–133. doi:10.1016/j.knosys.2018.07.041
   Web of Science ®Google Scholar
• Poria S, Majumder N, Mihalcea R, et al. Emotion recognition in conversation: research challenges, datasets, and recent advances. IEEE Access. 2019;7:100943–100953. doi:10.1109/ACCESS.2019.2929050
   Web of Science ®Google Scholar
• Huddar MG, Sannakki SS, Rajpurohit VS. Multi-level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification. Comput Intell. 2020;36:861–881. doi:10.1111/coin.12274
   Web of Science ®Google Scholar
• Gao J, Li P, Chen Z, et al. A survey on deep learning for multimodal data fusion. Neural Comput. 2020;32(5):829–864. doi:10.1162/neco_a_01273
   PubMed Web of Science ®Google Scholar
• Hong A, Lunscher N, Hu T, et al. A multimodal emotional human–robot interaction architecture for social robots engaged in bidirectional communication. IEEE Trans. on Cybernetics. 2021;51(12):5954–5968. doi:10.1109/TCYB.2020.2974688
   PubMed Web of Science ®Google Scholar
• Wang H-Y, Huang J-Y. Integrating scene image and conversational text to develop human–machine dialogue. Int J Semant Comput. 2022;16(3):425–447. doi:10.1142/S1793351X22430012
   Google Scholar
• Hossain MS, Muhammad G. Emotion recognition using deep learning approach from audio-visual emotional big data. Inf Fusion. 2019;49:69–78. doi:10.1016/j.inffus.2018.09.008
   Web of Science ®Google Scholar
• Schoneveld L, Othmani A, Abdelkawy H. Leveraging recent advances in deep learning for audio-visual emotion recognition. Pattern Recognit Lett. 2021;146:1–7. doi:10.1016/j.patrec.2021.03.007
   Web of Science ®Google Scholar
• Praveen RG, Granger E, Cardinal P. Cross attentional audio-visual fusion for dimensional emotion recognition. Proceedings of the 16th IEEE international conference on automatic face and gesture recognition; 2021. pp. 1–8.
   Google Scholar
• Tzirakis P, Trigeorgis G, Nicolaou AM, et al. End-to-end multimodal emotion recognition using deep neural networks. IEEE J Sel Top Signal Process. 2017;11:1301–1309. doi:10.1109/JSTSP.2017.2764438
   Web of Science ®Google Scholar
• Zhang S, Zhang S, Huang T, et al. Learning affective features With a hybrid deep model for audio–visual emotion recognition. Circ Syst Video Techn. 2018;28(10):3030–3043. doi:10.1109/TCSVT.2017.2719043
   Web of Science ®Google Scholar
• Yadav A, Vishwakarma DK. Sentiment analysis using deep learning architectures: A review. Artif Intell Rev. 2020;53:4335–4385. doi:10.1007/s10462-019-09794-5
   Web of Science ®Google Scholar
• Abdu SA, Yousef AH, Salem A. Multimodal video sentiment analysis using deep learning approaches, a survey. Inf Fusion. 2021;76:204–226. doi:10.1016/j.inffus.2021.06.003
   Web of Science ®Google Scholar
• Devlin J, Chang M-W, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of annual conference of the north American chapter of the association for computational linguistics: human language technologies; 2019: 4171–4186.
   Google Scholar
• Xing C, Wu W, Wu Y, et al. Topic aware neural response generation. Proceedings of the 31th AAAI conference on artificial intelligence; 2017: 3351–3357.
   Google Scholar
• Wu Y, Li Z, Wu W, et al. Response selection with topic clues for retrieval-based chatbots. Neurocomputing. 2018;316:251–261. doi:10.1016/j.neucom.2018.07.073
   Web of Science ®Google Scholar
• Serban IV, Sordoni A, Bengio Y, et al. Building end-to-end dialogue systems using generative hierarchical neural network models. Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016: 3776–3783.
   Google Scholar
• Zhang Y, Sun S, Galley M, et al. Dialogpt: large-scale generative Pre-training for conversational response generation. Proceedings of the 58th annual meeting of the association for computational linguistics; 2020: 270–278. doi:10.18653/v1/2020.acl-demos.30
   Google Scholar
• Firdaus M, Chauhan H, Ekbal A, et al. Emosen: generating sentiment and emotion controlled responses in a multimodal dialogue system. Affect Comp. 2022;13(3):1555–1566. doi:10.1109/TAFFC.2020.3015491
   Web of Science ®Google Scholar
• Kingma DP, Welling M. Auto-encoding variational Bayes. Proceedings of international conference on learning representation; 2014.
   Google Scholar
• Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. Proceedings of Neural Information Processing Systems. 2014: 2672–2680.
   Google Scholar
• Tolstikhin I, Bousquet O, Gelly S, et al. Wasserstein auto-encoders. Proceedings of international conference on learning representation; 2018.
   Google Scholar
• Gu X, Cho K, Ha J-W, et al. DialogWAE: multimodal response generation with conditional wasserstein auto-encoder. Proceedings of international conference on learning representation; 2019.
   Google Scholar
• Peng D, Zhou M, Liu C, et al. Human–machine dialogue modelling with the fusion of word- and sentence-level emotions. Knowl Based Syst. 2020;192:105319, doi:10.1016/j.knosys.2019.105319
   Web of Science ®Google Scholar
• Luo B, Lau RYK, Li C, et al. A critical review of state-of-the-art chatbot designs and applications. Wiley Interdiscipl Rev Data Min Knowled Discovery. 2022;12:e1434, doi:10.1002/widm.1434
   Web of Science ®Google Scholar
• Wang Y, Song W, Tao W, et al. A systematic review on affective computing: emotion models, databases, and recent advances. Inf Fusion. 2022;83-84-84:19–52. doi:10.1016/j.inffus.2022.03.009
   Web of Science ®Google Scholar
• Abdollahi H, Mahoor M, Zandie R, et al. Artificial emotional intelligence in socially assistive robots for older adults: a pilot study. IEEE Trans Affect Comp. 2022. (early access, doi: 10.1109/TAFFC.2022.3143803
   PubMed Web of Science ®Google Scholar
• Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the association for computational linguistics; 2002: 311–318.
   Google Scholar
• Busso C, Bulut M, Lee C-C, et al. IEMOCAP: interactive emotional dyadic motion capture database. Language Res Eval. 2008;42(4):335–359. doi:10.1007/s10579-008-9076-6
   Web of Science ®Google Scholar
• Li Y, Su H, Shen X, et al. Dailydialog: A manually labelled multi-turn dialogue dataset. Proceedings of the eighth international joint conference on natural language processing; 2017. 986–995.
   Google Scholar
• Godfrey J, Holliman E. Switchboard-1 release 2: Linguistic data consortium. Switchboard: a user’s manual, 1997.
   Google Scholar
• Poria S, Cambria E, Hazarika D, et al. Context-Dependent sentiment analysis in user-generated videos. Proceedings of the 55th annual meeting of the association for computational linguistics; 2017: 873–883. doi:10.18653/v1/P17-1081
   Google Scholar
• Hazarika D, Poria S, Zadeh A, et al. Conversational memory network for emotion recognition in dyadic dialogue videos. Proceedings of annual conference of the north American chapter of the association for computational linguistics: human language technologies; 2018: 2122–2132.
   Google Scholar
• Park Y, Ho JC, Kim G. A hierarchical latent structure for variational conversation modeling. Proceedings of annual conference of the north American chapter of the association for computational linguistics: human language technologies; 2018. 1792–1801.
   Google Scholar
• Plutchik R. The nature of emotions. Am Sci. 2001;89(4):344–350. doi:10.1511/2001.28.344
   Web of Science ®Google Scholar