Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture

Figures & data

Figure 1. Comparison between bimodal dual-task emotion recognition methods and traditional methods.

Figure 2. Network architecture diagram.

Table 1. Algorithm comparison on the IEMOCAP dataset.

Algorithm	Mode	WA	UA
Sajjad and Kwon (2020)	Voice	72.3%	72.25%
Lu et al. (2020)	Voice	71.7%	72.6%
Wang et al. (2020)	Voice	72.7%	73.3%
Chen et al. (2024)	Voice	73.7%	74.3%
Padi et al. (2022)*	Voice + Text	75.9%	75.0%
Kumar and Mahajan (2019)	Voice + Text	71.7%	75.0%
Feng, Ueno, and Kawahara (2020)*	Voice + Text	68.6%	69.7%
Morais et al. (2022)*	Voice + Text	77.3%	77.7%
Kakuba, Poulose, and Han (2023)	Voice + Text	74.8%	74.2%
Proposed	Voice + Text	82.3% ± 0.2%	83.0% ± 0.3%

Table 2. Algorithm comparison on the RAVDESS dataset.

Algorithm	Mode	WA	UA
Zeng et al. (2019)	Voice	64.5%	64.6%
Xu, Zhang, and Zhang (2021)*	Voice	77.8%	77.4%
Li et al. (2020)	Voice	72.0%	70.8%
Singh et al. (2021)	Voice	81.2%	79.7%
Kakuba, Poulose, and Han (2023)	Voice	94.0%	93.65%
Proposed	Voice	83.5% ± 0.5%	82.4% ± 0.3%

Figure 3. Confusion matrix of the model in the IEMOCAP dataset.

Figure 4. Confusion matrix of the model in the REVDESS dataset.

Figure 5. Feature distribution learned under the supervision of cross-entropy loss function on the IEMOCAP dataset.

Figure 6. Feature distributions learned under the supervision of both cross-entropy and N-pair loss function on the IEMOCAP dataset.

Figure 7. Ablation experiments based on the IEMOCAP dataset.

Sajjad, M., and S. Kwon. 2020. Clustering-based speech emotion recognition by incorporating learned features and deep BiLSTM. Institute of Electrical and Electronics Engineers Access 8:79861–75. doi:10.1109/ACCESS.2020.2990405

 Google Scholar

Lu, Z., L. Cao, Y. Zhang, C. Chiu, and J. Fan 2020. Speech sentiment analysis via pre-trained features from end-to-end asr models. In IEEE ICASSP, Barcelona, Spain, 7149–53. doi:10.1109/ICASSP40776.2020.9052937

 Google Scholar

Wang, J., M. Xue, R. Culhane, E. Diao, J. Ding, and V. Tarokh 2020. Speech emotion recognition with dual-sequence LSTM architecture. In IEEE ICASSP, Barcelona, Spain 6474–78. doi:10.1109/ICASSP40776.2020.9054629

 Google Scholar

Chen, W., X. Xing, P. Chen, and X. Xu (2024). Vesper: A compact and effective pretrained model for speech emotion recognition. IEEE Transactions on Affective Computing. doi:10.1109/TAFFC.2024.3369726

 Google Scholar

Padi, S., S. O. Sadjadi, D. Manocha, and R. D. Sriram 2022. Multimodal emotion recognition using transfer learning from speaker recognition and BERT-based models. arXiv preprint arXiv:2202.08974.

 Google Scholar

Kumar, Y., and M. Mahajan. 2019. Machine learning based speech emotions recognition system. International Journal of Scientific and Technology Research 8 (7):722–29.

 Google Scholar

Feng, H., S. Ueno, and T. Kawahara. 2020. End-to-end speech emotion recognition combined with acoustic-to-word ASR model. In Interspeech, 501–05. doi:10.21437/Interspeech.2020-1180

 Google Scholar

Morais, E. D., R. Hoory, W. Zhu, I. Gat, M. Damasceno, and H. Aronowitz 2022. Speech emotion recognition using self-supervised features. arXiv preprint arXiv:2202.03896.

 Google Scholar

Kakuba, S., A. Poulose, and D. S. Han. 2023. Deep learning approaches for bimodal speech emotion recognition: Advancements, challenges, and a multi-learning model. Institute of Electrical and Electronics Engineers Access 11:113769–89. doi:10.1109/ACCESS.2023.3325037

 Google Scholar

Zeng, Y., H. Mao, D. Peng, and Z. Yi. 2019. Spectrogram based multi-task audio classification. Multimedia Tools and Applications 78 (3):3705–22. doi:10.1007/s11042-017-5539-3

 Web of Science ®Google Scholar

Xu, M., F. Zhang, and W. Zhang. 2021. Head fusion: Improving the accuracy and robustness of speech emotion recognition on the IEMOCAP and RAVDESS dataset. IEEE Access 9:74539–49. doi:10.1109/ACCESS.2021.3067460

 Web of Science ®Google Scholar

Li, H., W. Ding, Z. Wu, and Z. Liu 2020. Learning fine-grained cross modality excitement for speech emotion recognition. arxiv preprint arxiv:2010.12733.

 Google Scholar

Singh, P., R. Srivastava, K. P. Rana, and V. Kumar. 2021. A multimodal hierarchical approach to speech emotion recognition from audio and text. Knowledge-Based Systems 229:107316. doi:10.1016/j.knosys.2021.107316

 Web of Science ®Google Scholar

Data Availability Statement

The data used to support the findings of this study are included within the article.