What multimodal components, tools, dataset and focus of emotion are used in the current research of multimodal emotion: a systematic literature review

Figures & data

Figure 1. PRISMA multimodal emotion stages of data collaction.

Figure 2. Bibliographical information.

Table 1. Multimodal research in understanding emotion.

Authors	Participants	Multimodal components	Tools and technology
Arthanarisamy and Palaniswamy (2022), India	32	Audio + Visual + Text	EEG
Bhattacharya et al. (2023), Singapore	Not reported	Audio + Visual	Audio-Video Emotion Recognition (AVER)
Cai et al. (2020), China	10	Audio + Visual	CNN-LSTM model
Caihua (2019), China	10	Audio + Visual	SVM
Chudasama et al. (2022), India	13	Audio + Visual + Text	Multi-modal Fusion Network (M2FNet)
Dahmane et al. (2022), Canada	Not reported	Audio + Visual	CNN and LSTM
Garg et al. (2021), India	Not reported	Audio + Visual	EEG
Ghaleb et al. (2019a), Netherland	7442 clips from 91 actors (43 females and 48 males)	Audio + Visual	Radial basis function (RBF) based support vector machine (SVM)
Ghaleb et al. (2019b), Netherland	Not reported	Audio + Visual	Audio-Video Emotion Recognition (AVER)
Gordon et al. (2018), Australia	Four narrative videos	Audio + Visual	EEG
Guanghui and Xiaoping (2021), China	Not reported	Audio + Visual	Two-dimensional convolutional neural network (2D-CNN) and three-dimensional convolutional neural network (3D-CNN)
Hu et al. (2022), China	256-dimension facial feature vector and 512-dimension emotion feature speech emotion	Audio + Visual	MTCNN (Multi-task Cascaded Convolutional Networks)
Jo et al. (2020), USA	30	Audio + Visual	ROSbag-based
Lee et al. (2020), South Korea	64 Dimensions	Audio + Visual + Text	FFMPEG
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	9.13 m tweets	Audio + Visual + Text	Twitter’s streaming API
Liu and Tan (2020), China	777 Songs	Audio + Text	Late fusion method (LFSM)
Priyasad et al. (2020), Australia	Five sessions of utterances for 10 unique speakers	Audio + Visual	Deep Convolutional Neural Networks (DCNN) and Recurrent Neural Networks (RNN)
Song et al. (2019), China	28 Videos from 23 participants	Audio + Visual	A-LSTM, LSTM, SVM and KNN
Sun et al. (2020), China	291 Videos and 70 host speakers collected from YouTube.	Audio + Visual + Text	LSTM
Tung et al. (2019), Taiwan	528 (33 subjects × 16 videos)	Audio + Visual	EEG, ECG and GSR
Zhang et al. (2020), China	12 h of audiovisual data from 10 actors, including video, voice, facial motion capture and text transcription.	Audio + Visual + Text	LSTM, DenseNet and GloVe.

Table

Authors	Dataset	Focus of emotions
Arthanarisamy and Palaniswamy (2022), India	DEAP	Not reported
Bhattacharya et al. (2023), Singapore	CREMA-D and RAVDESS	Positive + negative
Cai et al. (2020), China	IEMOCAP	Excite, angry, aad and neutral.
Caihua (2019), China	Not reported	Anger, boredom, disgust, fear (Anxiety/Fear), happiness (sadness) and sadness, neutral.
Chudasama et al. (2022), India	MELD and IEMOCAP	(Anger, Disgust, Sadness, Joy, Surprise, Fear and Neutral).
Dahmane et al. (2022), Canada	One-Minute Gradual (OMG) 2018	Negative emotions
Garg et al. (2021), India	AMIGOS	Not reported
Ghaleb et al. (2019a), Netherland	eNTERFACE and CREMA-D	Anger, disgust, fear, happy, neutral and sad
Ghaleb et al. (2019b), Netherland	CREMA-D and RAVDESS.	Not reported
Gordon et al. (2018), Australia	Not reported	Not reported
Guanghui and Xiaoping (2021), China	RML, eNTERFACE05, BAUM-1	Not reported
Hu et al. (2022), China	Not reported	Angry, disgust, fear, happy, sad, surprise and neutral
Jo et al. (2020), USA	International Affective Pictures System (IAPS) and the International Affective Digitized Sounds (IADS)	Happiness, sadness, anger, fear, surprise, disgust and neutral
Lee et al. (2020), South Korea	TV series	Angry, disgust, fear, happy, neutral, sad and surprise
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	Twitter	Positive + negative
Liu and Tan (2020), China	Music Mood Classification Data Sets	Exuberance, anxious, contentment and depression
Priyasad et al. (2020), Australia	IEMOCAP	Anger, happiness, neutral and sadness
Song et al. (2019), China	Not reported	Positive emotions and negative emotions
Sun et al. (2020), China	MuSe-CaR	Arousal and valence
Tung et al. (2019), Taiwan	AMIGOS	Positive emotions and negative emotions
Zhang et al. (2020), China	IEMOCAP	Anger, excitement, happiness, neutrality and sadness

Table

Authors	Scientific field	Model of inquiry process
Arthanarisamy and Palaniswamy (2022), India	Psychology	Investigation
Bhattacharya et al. (2023), Singapore	Psychological	Explore
Cai et al. (2020), China	Computer	Development
Caihua (2019), China	Language	Identifying
Chudasama et al. (2022), India	Language	Development
Dahmane et al. (2022), Canada	Psychology	Investigation
Garg et al. (2021), India	Psychology	Investigation
Ghaleb et al. (2019a), Netherland	Education	Investigation
Ghaleb et al. (2019b), Netherland	Psychology	Investigation
Gordon et al. (2018), Australia	Psychology	Investigation
Guanghui and Xiaoping (2021), China	Psychology	Investigation
Hu et al. (2022), China	Computer	Investigation
Jo et al. (2020), USA	Human-computer interaction (HCI)	Investigation
Lee et al. (2020), South Korea	Psychology	Investigation
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	Psychology	Investigation
Liu and Tan (2020), China	Music	Investigation
Priyasad et al. (2020), Australia	Music	Investigation
Song et al. (2019), China	Psychology	Investigation
Sun et al. (2020), China	Psychology	Investigation
Tung et al. (2019), Taiwan	Psychology	Investigation
Zhang et al. (2020), China	Psychology	Identifying

Figure 3. Scientific information.

Figure 4. Frequency of multimodal emotional research by research locations.

Figure 5. Frequency of multimodal emotions research by multimodal components.

Figure 6. Frequency of multimodal emotions research by tools and technologies.

Figure 7. Frequency of multimodal emotions research by dataset.

Table 2. Frequency of multimodal emotions research by focus of emotions.

Focus of emotions	Frequency	Percentage (%)
No mentioned	5	23.81
Positive and negative emotions	4	19.05
Negative emotions	1	4.76
Excited, angry, sad and neutral.	1	4.76
Angry, bored, disgusted, scared, happy, sad and neutral	1	4.76
Angry, disgusted, sad, happy, shocked, scared and neutral	1	4.76
Angry, disgusted, scared, happy, neutral and sad	1	4.76
Angry, disgusted, scared, happy, sad, shocked and neutral	1	4.76
Happy, sad, angry, scared, shocked, disgusted and neutral	1	4.76
Angry, disgusted, scared, happy, neutral, sad and shocked	1	4.76
Happy, worried, satisfied and depressed	1	4.76
Angry, happy, neutral and sad	1	4.76
Arousal and valant	1	4.76
Angry, glad, happy, neutral and sad	1	4.76

See Table 3 for illustration and identification of emotions focus.

Table 3. Frequency of multimodal emotions research by emotions focus cluster.

Emotion focus cluster	Frequency	Percentage (%)
Arousal and valant	1	4.76
Negative emotions	1	4.76
Positive and negative emotions	5	23.81
Positive, negative and neutral emotions	9	42.86
No reported	5	23.81

Arthanarisamy, R. M. P., & Palaniswamy, S. (2022). Subject independent emotion recognition using EEG and physiological signals–a comparative study. Applied Computing and Informatics. http://doi.org/10.1108/aci-03-2022-0080

 Google Scholar

Bhattacharya, P., Gupta, R. K., & Yang, Y. (2023). Exploring the contextual factors affecting multimodal emotion recognition in videos. IEEE Transactions on Affective Computing, 14(2), 1547–1557. https://doi.org/10.1109/TAFFC.2021.3071503

 Web of Science ®Google Scholar

Cai, L., Dong, J., & Wei, M. (2020). Multi-modal emotion recognition from speech and facial expression based on deep learning [Paper presentation]. 2020 Chinese Automation Congress (CAC) (pp. 5726–5729). https://doi.org/10.1109/CAC51589.2020.9327178

 Google Scholar

Caihua, C. (2019). Research on multi-modal mandarin speech emotion recognition based on SVM [Paper presentation]. 2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS) (pp. 173–176). https://doi.org/10.1109/ICPICS47731.2019.8942545

 Google Scholar

Chudasama, V., Kar, P., Gudmalwar, A., Shah, N., Wasnik, P., & Onoe, N. (2022). M2FNet: Multi-modal fusion network for emotion recognition in conversation. Proceedings of theCVF Conference on Computer Vision and Pattern Recognition (pp. 4652–4661). https://doi.org/10.48550/arXiv.2206.02187

 Google Scholar

Dahmane, M., Alam, J., St-Charles, P. L., Lalonde, M., Heffner, K., & Foucher, S. (2022). A multimodal non-intrusive stress monitoring from the pleasure-arousal emotional dimensions. IEEE Transactions on Affective Computing, 13(2), 1044–1056. https://doi.org/10.1109/TAFFC.2020.2988455

 Web of Science ®Google Scholar

Garg, S., Patro, R. K., Behera, S., Tigga, N. P., & Pandey, R. (2021). An overlapping sliding window and combined features based emotion recognition system for EEG signals. Applied Computing and Informatics, https://doi.org/10.1108/ACI-05-2021-0130

 Google Scholar

Ghaleb, E., Popa, M., & Asteriadis, S. (2019a). Metric learning-based multimodal audio-visual emotion recognition. IEEE Multimedia, 27(1), 1–1. https://doi.org/10.1109/MMUL.2019.2960219

 Web of Science ®Google Scholar

Ghaleb, E., Popa, M., & Asteriadis, S. (2019b). Multimodal and temporal perception of audio-visual cues for emotion recognition [Paper presentation]. 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 552–558). https://doi.org/10.1109/ACII.2019.8925444

 Google Scholar

Gordon, R., Ciorciari, J., & van Laer, T. (2018). Using EEG to examine the role of attention, working memory, emotion, and imagination in narrative transportation. European Journal of Marketing, 52(1–2), 92–117. https://doi.org/10.1108/EJM-12-2016-0881

 Web of Science ®Google Scholar

Guanghui, C., & Xiaoping, Z. (2021). Multi-modal emotion recognition by fusing correlation features of speech-visual. IEEE Signal Processing Letters, 28, 533–537. https://doi.org/10.1109/LSP.2021.3055755

 Web of Science ®Google Scholar

Hu, L., Li, W., Yang, J., Fortino, G., & Chen, M. (2022). A sustainable multi-modal multi-layer emotion-aware service at the edge. IEEE Transactions on Sustainable Computing, 7(2), 324–333. https://doi.org/10.1109/TSUSC.2019.2928316

 Google Scholar

Jo, W., Kannan, S. S., Cha, G. E., Lee, A., & Min, B. C. (2020). Rosbag-based multimodal affective dataset for emotional and cognitive states [Paper presentation].2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 226–233). https://doi.org/10.1109/SMC42975.2020.9283320

 Google Scholar

Lee, J. H., Kim, H. J., & Cheong, Y. G. (2020). A multi-modal approach for emotion recognition of TV drama characters using image and text [Paper presentation]. 2020 IEEE International Conference on Big Data and Smart Computing (BigComp) (pp. 420–424). https://doi.org/10.1109/BigComp48618.2020.00-37

 Google Scholar

Li, M., Ch’ng, E., Chong, A. Y. L., & See, S. (2018). Multi-class Twitter sentiment classification with emojis. Industrial Management & Data Systems, 118(9), 1804–1820. https://doi.org/10.1108/IMDS-12-2017-0582

 Web of Science ®Google Scholar

Li, X., Song, D., Zhang, P., Zhang, Y., Hou, Y., & Hu, B. (2018). Exploring EEG features in cross-subject emotion recognition. Frontiers in Neuroscience, 12, 162. https://doi.org/10.3389/fnins.2018.00162

 PubMed Web of Science ®Google Scholar

Liu, G., & Tan, Z. (2020). Research on multi-modal music emotion classification based on audio and lyirc [Paper presentation]. 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC) (Vol. 1, pp. 2331–2335). https://doi.org/10.1109/ITNEC48623.2020.9084846

 Google Scholar

Priyasad, D., Fernando, T., Denman, S., Sridharan, S., & Fookes, C. (2020). Attention driven fusion for multi-modal emotion recognition [Paper presentation]. ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3227–3231). https://doi.org/10.1109/ICASSP40776.2020.9054441

 Google Scholar

Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., & Cui, Z. (2019). MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access, 7, 12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579

 Web of Science ®Google Scholar

Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020). Multi-modal continuous dimensional emotion recognition using recurrent neural network and self-attention mechanism [Paper presentation]. Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-Life Media Challenge and Workshop (pp. 27–34). https://doi.org/10.1145/3423327.3423672

 Google Scholar

Tung, K., Liu, P. K., Chuang, Y. C., Wang, S. H., & Wu, A. Y. A. (2019). Entropy-assisted multi-modal emotion recognition framework based on physiological signals [Paper presentation]. 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), IEEE (pp. 22–26). https://doi.org/10.1109/IECBES.2018.8626634

 Google Scholar

Zhang, X., Wang, M. J., & Guo, X. D. (2020). Multi-modal emotion recognition based on deep learning in speech, video and text [Paper presentation]. 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP) (pp. 328–333). https://doi.org/10.1109/ICSIP49896.2020.9339464

 Google Scholar