Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture

ABSTRACT

In the field of emotion recognition, analyzing emotions from speech alone (single-modal speech emotion recognition) has several limitations, including limited data volume and low accuracy. Additionally, single-task models lack generalization and fail to fully utilize relevant information. To address these issues, this paper proposes a new bi-modal bi-task emotion recognition model. The proposed model introduces multi-task learning on the Transformer architecture. On one hand, unsupervised contrastive predictive coding is used to extract denser features from the data while preserving self-information and context-related information. On the other hand, model robustness against interfering information is enhanced by employing self-supervised contrastive learning. Furthermore, the proposed model utilizes a modality fusion module to incorporate textual and audio information to implicitly align features from both modalities. The proposed model achieved accuracy rates of 82.3% and 83.5% on the IEMOCAP and RAVDESS datasets, respectively, when considering weighted accuracy (WA). When weight is not considered (unweighted accuracy (UA)), the model achieved 83.0% and 82.4% accuracy. Compared to the existing methods, the performance is further improved.

Introduction

In the early stage of emotion recognition, only a single mode is used for speech emotion recognition. The acoustic characteristics, including formant frequency, speech speed, energy characteristics, and time domain characteristics, are used as the identification criteria. Text is also commonly used for speech emotion recognition. Different from acoustic information, text information focuses more on the information contained in the text in the speech. The extracted features include vocabulary, grammar, syntax, and other information related to language structure. For speech data that has been converted to text, such as speech translation and chat conversations, natural language processing (NLP) techniques can be used to analyze the emotional information in the text (Min et al. 2023; Samant et al. 2022).

With the rapid advancement of deep learning technology, emotion recognition technology is becoming increasingly sophisticated. This progress has also highlighted the limitations of relying on a single modality (like speech) for emotion recognition. While single-mode data is limited, training accurate emotion recognition models requires large amounts of labeled speech data covering a wide range of emotions and speech variations. Moreover, the emotion in speech is usually subjective, and the same emotion shows significant differences when different people use different ways of speech expression. The single-modality emotion recognition model faces difficulty in generalization when dealing with various speech variants. This is where multi-modality becomes a valuable approach. By integrating other modal information, such as facial expression and body language, the emotional state of the speaker can be more holistically captured, improving the accuracy of emotion detection. (Cai, Li, and Li 2023; Huang et al. 2023)

In the field of mood analysis, multi-task learning has gained significant attention because of its powerful performance. Multi-task learning is a branch of computational learning that utilizes simultaneous learning of multiple related tasks to achieve the goal of enhancing the overall efficiency of the model through the interrelationship between tasks (Yang et al. 2023). Figure 1(a,b) show the traditional method and the proposed bi-modal bi-task approach, respectively. Specifically, the traditional method solely extracts emotional information from a single modality through a single backbone network. This dependence on a single modality limits the model’s generalization capability. In contrast, the proposed bi-modal bi-task approach further expands the modality range of input information and facilitates the complementary fusion of information between different modalities. This makes the model more robust and adaptable. Furthermore, the joint training of multiple tasks assists in emotion recognition while addressing other tasks further improves the performance of the model.

Figure 1. Comparison between bimodal dual-task emotion recognition methods and traditional methods.

The literature shows that multimodal and multitasking learning techniques in emotion recognition tasks have better results and higher recognition accuracy. This work proposes using self-supervised models for emotion recognition. Sel-supervised learning can utilize the inherent properties or structures of the data for learning, significantly reducing the cost and workload of data annotation. Hence, this approach facilitates the practical deployment of the model. Additionally, self-supervised learning can assist the model in learning richer feature representations from different modalities of data, enhancing the model’s generalization ability and performance. Therefore, this paper proposes a bi-modal bi-task model to analyze emotion more comprehensively and accurately from the aspects of both mode and learning tasks. The main contributions include:

1. A novel multi-modal emotion analysis model. Inspired by the Transformer architecture, the proposed model utilizes both text and acoustic modes for interaction to obtain more comprehensive emotion information in speech. This approach effectively reduces the bias in sentiment analysis that can occur when relying on a single mode.

2. The accuracy of emotion recognition can be improved by introducing two auxiliary tasks. The features of relevant data are extracted through Contrastive Predictive Coding (CPC) (Meng and Yolwas 2023), enabling the sharing of basic information among different parts of the high-dimensional signal. This approach employs Enhanced Contrastive Learning (ECL) (Wang et al. 2023) to improve feature learning. In addition, conditional BERT contextual augmentation (CBCA) and Linear Spectral Adjustment (LSA) are utilized to enrich the input data. By comparing and contrasting data samples, ECL helps the model extract more robust and distinctive features.

Related Work

The development of emotion recognition technology is inextricably linked to the development of various machine learning algorithms. Early machine learning algorithms mainly include Linear Regression (Ribeiro and Schön 2023), Logistic Regression, k-nearest neighbor algorithm (Goshvarpour and Goshvarpour 2023), decision tree, Naive Bayes and support vector machine. Early neural networks were relatively small due to computational resource constraints. With the evolution of deep learning in recent years, there has been a surge in interest and development in neural networks, leading to advancements in various technologies. In the realm of emotion recognition, algorithms such as convolutional neural networks (Datta and Chakrabarti 2022; Feng et al. 2023; Sarvakar et al. 2023), recurrent neural networks, and long short-term memory networks, have also shown promising performance.

Emotion Recognition Model

In the further deepening of emotion recognition research, multi-modal speech emotion recognition has come into public view. Different from the single feature vector of a single mode, multi-modal speech emotion recognition has a wider range of application scenarios and a stronger generalization ability. Kakuba et al. proposed an attention-based multi-learning approach, utilizing residual dilated causal convolution blocks, dilated convolution layers, and multi-head attention, to learn emotional cues from extracted features (Kakuba, Poulose, and Han 2022a). However, further research is still needed for the robustness of multimodal emotion recognition. Subsequently, they found that attention mechanisms still play a significant role in bi-modal speech emotion recognition. Therefore, they proposed the DBMER model to improve the performance of bimodal speech emotion recognition by combining traditional methods (Kakuba, Poulose, and Han 2023). Sharafi et al. introduced a novel multimodal neural architecture that integrates both audio and visual data as inputs into a composite network composed of a bidirectional Long Short-Term Memory (BiLSTM) network and a pair of Convolutional Neural Networks (CNNs). Experimental investigations demonstrate that this approach outperforms the existing algorithms in adapting emotion recognition (Sharafi et al. 2022). Wen et al. presented a cross-modal dynamic convolution method and experimentally validated its competitive performance and higher efficiency (Wen, You, and Fu 2021). Chang et al. defined a multimodal residual perceptron network that learns end-to-end through multimodal network branches. The concept behind this multimodal residual perceptron network has promise for applications beyond just optical and acoustic signal sources (Chang and Skarbek 2021). Several studies have also integrated information from different modalities. Lian, Liu, and Tao (2021) proposed the use of a transformer-based structure to model intra-modal and cross-modal interactions among multimodal features (Lian, Liu, and Tao 2021). Additionally, they introduced intra-modal interaction modules and cross-modal interaction modules to capture contextual information between different modalities (Lian, Liu, and Tao 2022).

Transformer Architecture

The Transformer was first proposed in 2017 by the Google Machine translation team (Vaswani et al. 2017). This new architecture revolutionized the field by moving away from traditional RNN and CNN toward a stacked structure. Transformer has achieved a surprising effect in the field of NLP. Various emotion recognition algorithms based on Transformer architecture have been launched. Le et al. (2023) established a Transformer-based fusion and representation learning method to fuse and enrich multi-modal features for emotion recognition in multi-label videos. Experimental results demonstrate that this approach outperforms other robust baseline methods and current techniques in the field of multi-label video emotion recognition (Le et al. 2023). MA et al. proposed a Transformer-based self-distillation model that can transfer knowledge of both hard and soft labels to each mode. Experimental results from the dataset indicate that the Single Decision Tree (SDT) surpasses the performance of the previous best baseline model (Ma et al. 2023). Xie et al. utilized a Transformer-based cross-modal fusion technique, called the EmbraceNet network architecture, for emotion prediction. The developed model achieved 65% precision, surpassing the single-mode model used for comparison (Xie, Sidulova, and Park 2021). Similarly, Kakuba et al. also investigated the importance of multi-level fusion for emotion recognition. They utilized multi-level fusion with the Transformer’s multi-head attention mechanism to concurrently learn spatial, temporal, and semantic feature representations, further enhancing the effectiveness and robustness of the model (Kakuba, Poulose, and Han 2022b).

Multi-Task Learning

Emotion recognition usually uses a large pool of labeled data when training models. In multi-task learning (MTL), emotion recognition tasks can be combined with other related tasks to share the same dataset, efficiently utilizing the limited annotated data. Latif et al. proposed an MTL framework for learning generalized representations from augmented data (McL-aug). The semi-supervised nature of the McL-aug makes it possible to leverage large amounts of unlabeled data to further improve the efficiency of identification (Latif et al. 2022). Atmaja et al. used two audiovisual data-based affective attribute prediction methods based on MTL and fusion strategies. The use of MTL in single-modal and early fusion methods has shown better results than single-tasking learning, achieving an average Concordance Correlation Coefficient (CCC) score of 0.431 (Atmaja and Akagi 2020). Bendjoudi and colleagues proposed a novel context-dependent multi-label multi-task deep learning structure for identifying emotions. The findings from the EMOTIC dataset indicate that the label with a lower frequency outperforms the existing technological standard (Bendjoudi et al. 2021).

The above-mentioned studies exhibit the superior results of the Transformer architecture and the positive role of MTL in emotion recognition. Therefore, this paper improves Transformer architecture and incorporates MTL. This allows the model to learn multiple emotional aspects simultaneously, resulting in a more comprehensive understanding of emotions.

Methodology

The overall network framework is divided into two tasks: CPC and ECL. These tasks share parameters within the core parts of the network. In the CPC task, the audio sequences are encoded using wav2vec-2.0, while the text sequences are encoded using DistilBERT. Subsequently, a Cross-Modal coding Block (CCB) is utilized to fuse the representation of different modalities. Then, BiGRU is employed to effectively merge information from different modalities, and finally, the extracted features are used to compute cross-entropy. The process of the ECL task is similar to that of the CPC task, with the difference lying in the input stage, where text and audio are subjected to text enhancement techniques CBCA and spectral enhancement techniques LSA, respectively. In the output stage, unlike the CPC task, the features obtained are used for contrastive loss computation. This article assumes a dataset $\mathit{C}$containing $\mathit{N}$sentences $\left\{c_1,c_2,\right.c_3,\dots ,c_N\}$and their corresponding labels $\left\{x_1,x_2,\right.x_3,\dots ,x_N\}$. Here, it is assumed that each sentence$c_i$ has a voice cue $a_i$and a text cue$t_i$, where $c_i\in (a_i,t_i)$. The task of emotion recognition is converted to assigning each sentence $c_i$an emotion label $x_i$, which $x_i$ represents the emotion probability distribution of the first $\mathit{i}$ sentence studied in the dataset.

Feature Extraction Module

Unlabeled speech data was pre-trained on a 960-hour Librispeech using wav2vec-2.0 (Schneider et al. 2019) as the original waveform encoder. During pre-training, the wav2vec-2.0 model learned the speech feature representation by reordering random subsequences of inputs into the correct order with predictions. The speech features $a_i$ of each discourse $c_i$were fed to the wav2vec-2.0, and feature extraction was performed at the CNN layer of wav2vec-2.0. Frames with a step size of 20 ms and a jump size of 25 ms were extracted.

Text feature coding with DistilBERT. DistilBERT is a compressed and streamlined BERT model proposed by Hugging Face (Sanh et al. 2019). Think of distilling a large model into a smaller model like this: the large model acts as a teacher and the smaller model learns everything it can from the teacher’s outputs. The model uses a distillation loss, which is calculated using soft labels. When T is set to 1, the distillation loss becomes similar to the regular loss function used for training. It retains most of the language understanding capabilities of the original BERT model. By learning from the original BERT model, it achieves a significantly smaller size and faster processing speed, making it suitable for running in resource-constrained environments.

Mode Fusion Module

The Modal fusion module (MFM) consists of two CCBs and a BiGRU module. It incorporates a multi-head attention mechanism module (Tsai et al. 2019), a residual connection and a feedforward layer. Each CCB is constructed into a common Transformer layer (Vaswani et al. 2017), as shown in yellow and green boxes in Figure 2. Each Transformer layer is made up of a multi-head attention mechanism module (Tsai et al. 2019), residual connections, and feedforward layers.

Figure 2. Network architecture diagram.

Speech-Aware Text Representation

To enhance token representations in text, associated audio is utilized as guidance for learning. Specifically, during the computation of Cross-Modal Attention, audio features extracted by wav2vec-2.0 serve as queries $\mathit{A}\in R^{\mathit{d}\times \mathit{J}}$ for embedding, while features extracted by DistilBERT serve as keys and values $\mathit{T}\in R^{\mathit{d}\times \mathit{M}}$ for embedding. Here, $\mathit{J}$, $\mathit{M}$and $\mathit{d}$ represent the dimensionalities of the audio, text and embedding features, respectively. The output of Cross-Modal Attention in CBB1 is as follows:

(1) $\mathit{CMA}=\mathit{softmax}(\frac{{[{\mathit{W}}_{\mathit{qi}}\mathit{A}]}^{\mathit{T}}[W_{\mathit{ki}}\mathit{T}]}{\sqrt{\mathit{d}/\mathit{m}}})[W_{\mathit{Vi}}\mathit{T}{]}^T$(1)

Herein, $\{W_{\mathit{qi}},W_{\mathit{ki}},W_{\mathit{vi}}\}\in R^{\mathit{d}/m\times h}$ represents the weight matrices of the query, key, and value for the ith attention head, respectively. The final output of CCB1 is denoted as $\mathit{H}=(H_0,H_1,\dots ,H_m)$.

Text-Aware Speech Representation

To enhance the token representations in audio, related text is utilized for guidance in learning. Specifically, during the computation of Cross-Modal Attention, features extracted by DistilBERT are used as queries $\mathit{T}\in R^{\mathit{d}\times \mathit{M}}$ for text embeddings, while features extracted by wav2vec-2.0 are used as keys and values $\mathit{A}\in R^{\mathit{d}\times \mathit{J}}$ for audio embeddings. Here, $\mathit{J}$, $\mathit{M}$ and $\mathit{d}$ represent the dimensionalities of the audio, text and embedding features, respectively. The final output of CCB2 is $\mathit{G}=(g_0,g_1,\dots ,g_j)$. Phoneme alignment holds a significant research position in speech science and acoustics, allowing each word to allocate weights to the speech embeddings according to their importance.

BiGRU Module

BiGRU is an improved RNN unit that receives output from a CCB as input and models the data through a bidirectional loop structure. It consists of two directional GRUs (Gated Recurrent Units), one responsible for forward sequential processing and the other responsible for reverse sequential processing. Despite the preliminary fusion of information from different modalities by the CCB module, BiGRU still contributes to modeling contextual information in the input data, allowing it to capture contextual relationships between different modalities even when speech and text information have been integrated. Additionally, it effectively learns long-term dependencies in sequential data. Its bidirectional structure allows it to simultaneously analyze information both before and after the current time step, facilitating more detailed modeling of the temporal relationships between speech and text information. Furthermore, BiGRU can extract abstract representations of the input data through hidden layer states, capturing key features in speech and text information.

Multi-Task Learning

The model introduces two emotion recognition learning tasks. This section will introduce the principle of two loss functions and two tasks and the main tasks in emotion recognition.

Cross Entropy Loss

Cross-entropy loss is a commonly used function for training speech emotion recognition (SER) models. When calculating cross-entropy loss with emotional labels, the maximum pooled mp(.) is first used independently on the wav2vec-2.0 speech A and MFM modules on the time step axis. The embeddings are then concatenated to obtain the final embed. The final embedding is passed by the linear transformation and softmax activation function, as shown in formula (2):

(2) $\stackrel{\wedge }{\mathit{y}}=\mathit{soft}max(W_p^T[\mathit{mp}(\mathit{A});\mathit{mp}(\mathit{M})]+B_P)$(2)

where $\stackrel{\wedge }{\mathit{y}}\in R^4$is a single vector representation of each discourse, $W_P\in R^{2\mathit{d}\times 4}$is a weight matrix, $\mathit{soft}max(\cdot )$ represents softmax activation function and $\mathit{mp}(\cdot )$ indicates an attention-pooling operation. After this step, the cross-entropy is calculated by $\mathit{LCE}=\mathit{CrossEntropy}(\stackrel{\wedge }{\mathit{y}},y_i)$.

N-Pair Loss Function

The N-Pair loss function is used for contrast learning. The input samples are mapped to the feature space by constructing a network of shared parameters, and the similarity between positive and negative samples is maximized by the N-pair loss function. The loss function is defined as:

(3) $L_{\mathit{N}-\mathit{Pair}}=\frac{1}{N}\sum _{\mathit{i}=1}^{N}log(1+\sum _{\mathit{j}\ne \mathit{i}}exp({a_i}^{\mathit{T}}{a}_j^{-}-{a_i}^{\mathit{T}}{a}_i^{+}))$(3)

where $a_i^{+}$and $a_j^{-}$represent $a_i$positive and negative cases, respectively, where $a_i^{+}$and $a_i$belong to the same class, and $a_j^{-}$ belong to $a_i$ different classes.

Contrastive Predictive Coding

CPC is a self-supervised learning method based on prediction tasks. It works by making the model predict what comes next in a sequence of information and then comparing the predicted results with the actual inputs to learn the input representation. The CPC method predicts contextual information within a modality, which enables the model to learn representations that capture shared semantic information between two modalities. Furthermore, through contrastive loss functions, CPC encourages the model to differentiate representations of semantic information in different modalities. This helps the model learn more universal and generalized feature representations.

Enhanced Contrast Learning

ECL helps the model become more efficient by learning the similarities and differences between samples. This allows the model to develop more robust and discriminative feature representations. Traditionally, emotion recognition focused on classifying emotions based on static features. However, this paper considers multiple modes of input and enhances the original data to enable the model to learn through comparisons across a richer dataset. In the aspect of speech, LSA spectrum enhancement technology is introduced to reduce noise by noise estimation and spectrum modification of input signal. The short-time Fourier transform (STFT) is used to convert the input signal into the time-frequency domain. Noise estimation and gain calculations are then used to determine the region of the spectrum that needs to be modified. Finally, the enhanced signal can be obtained by modifying the frequency spectrum of these regions and performing inverse STFT. In terms of text, text enhancement technology based on context information (Wu et al. 2019) (CBCA) is introduced. This approach begins with a pre-trained language model that randomly removes a word from the initial text that requires improvement. The residual text is then fed into the language model. The model’s first k predicted words are utilized to substitute the erased words in the original text, resulting in k new texts. Finally, to optimize the model, the minimum is reduced to: $\mathit{L}=\mathit{L}+\mathit{\alpha }{L}_{\mathit{SCL}}+\mathit{\beta }{L}_{\mathit{ACL}}$, where$\mathit{\alpha }$and $\mathit{\beta }$ are the adjusted hyperparameters.

Experiments

Dataset and Parameter Settings

The proposed model underwent training and assessment using the IEMOCAP and RAVDESS datasets. The IEMOCAP dataset, developed by the University of Southern California’s Speech Analysis and Interpretation Laboratory (SAIL), comprises recordings of 10 actors engaged in two-person dialogs. These recordings are marked with comprehensive details about the actors’ facial expressions and gestures during both scripted and spontaneous conversational scenes. The actors performed scripts specific to certain emotions and improvised hypothetical situations intended to elicit various emotional responses such as joy, anger, sorrow, frustration, and neutral states. The entire collection consists of approximately 12 hours of data. The RAVDESS is an emotion dataset for the study and analysis of emotional expression in speech and song. The dataset was created by the Department of Psychology at Ryerson University in Canada to provide a diverse range of emotional voice and song samples. It contains approximately 1,500 voice and song samples from 24 actors. Each actor performed eight emotional states in the dataset, including surprise happiness, anger, fear, disgust, sadness, calm and neutral. These emotional states are expressed through speech and song.

To accurately evaluate the model’s performance, a 5-fold cross-validation experiment was employed. The steps are as follows: The data was divided into five equally-sized folds, keeping each actor’s session intact. For each experiment, the sessions of one actor were selected as the verification set and the remaining sessions of all other actors were combined to form the training set. In each experiment, the hyperparameters were kept consistent. The training set was utilized for model learning, followed by the validation set for model validation. The above steps were repeated until each actor’s session was tested once as a verification set. The accuracy of each experiment was a weighted average and used as the evaluation index of hyperparameters. The results of evaluation indicators were then used to select the appropriate hyperparameters. Finally, the model was trained with the selected hyperparameters using all the data as the training set and the final model was obtained. This approach allows for a more precise evaluation of hyperparameters, ultimately leading to the selection of the best hyperparameters and their corresponding models.

Both DistilBERT and wav2vec-2.0 use basic architectures. The model was implemented on PyTorch (Paszke et al. 2019) and optimized by Adam. The cumulative gradient of training and evaluation with a batch size of 4 and 100 rounds was raised to 4. For an MTL setup, a training model was used where the search was performed$\mathit{\alpha },\mathit{\beta }\in \{1,0.1,0.01,0.001\}$ between and grid search. The proposed model has approximately 180 M parameters.

Baseline Model

In the early stage, emotion recognition relied on a single mode of speech, and in the later stage, utilized multi-mode for recognizing emotions. This study compares the emotion recognition models with different modes and algorithms in recent years.

Mustaqeem et al. used similarity measures based on radial function network (RBFN) to select key sequence segments and extract discriminant features and significant features from speech spectrograms (Sajjad and Kwon 2020). Lu et al. proposed a method to solve speech sentiment analysis, which uses the pre-learned characteristics of the end-to-end automatic speech recognition (ASR) model as inputs for downstream tasks (Lu et al. 2020). Wang et al. proposed a new two-layer model to predict emotions based on Mel spectra generated by MFCC features and raw audio signals (Wang et al. 2020). Padi et al. developed a neural network-based emotion recognition framework, utilizing transfer learning and fine-tuning models to combine speech and text modal information (Padi et al. 2022). Kumar et al. obtained audio and text attributes using attention-based gated loop units (GRUs) and pre-trained BERT, respectively, and then combined these attributes to forecast the ultimate emotional category (Kumar and Mahajan 2019). Feng et al. suggested an end-to-end fusion of the voice-word automatic speech recognition (ASR) model and the speech emotion recognition (SER) model (Feng, Ueno, and Kawahara 2020). Morais et al. presented a modular end-to-end (E2E) SER system following the upstream + downstream architectural paradigm (Morais et al. 2022).

Zeng et al. designed a time-aligned mean-maximum pooling mechanism (Zeng et al. 2019). Xu et al. proposed a multi-task model based on deep neural networks (DNN) and used it to simultaneously process multiple audio classification tasks (Xu, Zhang, and Zhang 2021). Li et al. proposed a head fusion approach based on a multi-head attention mechanism and established an attention-based convolutional neural network (ACNN) model (Li et al. 2020). Singh et al. developed a deep learning-based layered approach (Singh et al. 2021). Kakuba et al. proposed a Transformer-based model that further improves the performance by learning multi-level fusion of spatial, temporal, and semantic features (Kakuba, Poulose, and Han 2023). Chen et al. proposed an improved pre-trained encoder Vesper, which enhances the performance of speech emotion recognition through emotion-guided masking strategy and hierarchical cross-layer methods (Chen et al. 2024).

Comparison with Baseline Model

The evaluation indexes used in the experiments were weighted average accuracy (WA) and unweighted accuracy (UA). The proposed model was evaluated on IEMOCAP and RAVDESS datasets, with the results shown in Tables 1 and 2, respectively. The asterisk * denotes a multi-task model. Table 1 shows that in the IEMOCAP dataset, multi-modal input improves the overall performance of the model compared with single-modal input (Lu et al. 2020; Sajjad and Kwon 2020; Wang et al. 2020). It is worth noting that there are also models with multi-modal input, but the enhancement effect is not significant, such as in the literature (Feng, Ueno, and Kawahara 2020). This may be because multi-modal sentiment recognition requires designing models that work with multiple types of data. The previous multi-modal fusion methods adopted simple linear or splicing methods, ignoring the relevance and importance of different modes. Compared with the model that also chooses speech and text as modal inputs (Kumar and Mahajan 2019; Padi et al. 2022; Feng, Ueno, and Kawahara 2020; Morais et al. 2022), the proposed model has outstanding advantages in terms of performance. Padi et al. (2022) used additional tasks, categorical markers, and independent fragment markers for transfer learning. However, the algorithm proposed in this paper still improves WA by 6.4% and UA by 8%. This is because the supervised contrast learning utilized by the proposed model does not require large-scale labeling of datasets and can better handle data scarcity and noise conditions in emotion recognition tasks. Feng, Ueno, and Kawahara (2020) and Morais et al. (2022) use end-to-end models but require a large number of labeled data to train the entire system. When the training data is not enough or there is noise, the accuracy is worse than the proposed model. Sebastian et al. (2019) used the GRU fusion attention mechanism to extract audio features, which shows the applicability of the GRU module in emotion classification. However, the proposed model uses a bidirectional BiGRU module and achieves superior efficiency. Overall, the proposed bi-modal bi-task model demonstrates the best performance, with WA and UA reaching 82.3% and 83.0%, respectively. This achievement is attributed to the synergistic interaction of two key factors. Firstly, a bi-task learning approach is adopted, which allows the model to simultaneously consider different tasks and extract more informative features from the data. Secondly, the CCB module enables the audio and text modalities to perceive each other, facilitating the adaptive alignment of key components between different modalities. Additionally, BiGRU further captures the temporal relationships and long-term dependencies of the data features. The mutual coordination between CCB and BiGRU enables the model to comprehensively understand the input data and enhances the performance and robustness of the emotion recognition task.

Table 1. Algorithm comparison on the IEMOCAP dataset.

Algorithm	Mode	WA	UA
Sajjad and Kwon (2020)	Voice	72.3%	72.25%
Lu et al. (2020)	Voice	71.7%	72.6%
Wang et al. (2020)	Voice	72.7%	73.3%
Chen et al. (2024)	Voice	73.7%	74.3%
Padi et al. (2022)*	Voice + Text	75.9%	75.0%
Kumar and Mahajan (2019)	Voice + Text	71.7%	75.0%
Feng, Ueno, and Kawahara (2020)*	Voice + Text	68.6%	69.7%
Morais et al. (2022)*	Voice + Text	77.3%	77.7%
Kakuba, Poulose, and Han (2023)	Voice + Text	74.8%	74.2%
Proposed	Voice + Text	82.3% ± 0.2%	83.0% ± 0.3%

Table 2. Algorithm comparison on the RAVDESS dataset.

Algorithm	Mode	WA	UA
Zeng et al. (2019)	Voice	64.5%	64.6%
Xu, Zhang, and Zhang (2021)*	Voice	77.8%	77.4%
Li et al. (2020)	Voice	72.0%	70.8%
Singh et al. (2021)	Voice	81.2%	79.7%
Kakuba, Poulose, and Han (2023)	Voice	94.0%	93.65%
Proposed	Voice	83.5% ± 0.5%	82.4% ± 0.3%

To test the generalization performance of the proposed model, experiments were conducted on the RAVDESS dataset. The results are presented in Table 2. During the testing phase, only the audio information from the dataset was utilized. The results were then compared with five other emotion recognition models that have been tested on the RAVDESS dataset in recent years (Kakuba, Poulose, and Han 2023; Li et al. 2020; Singh et al. 2021; Xu, Zhang, and Zhang 2021; Zeng et al. 2019). All models were for audio-based single-modal emotion recognition. The results revealed that the proposed model achieved WA and UA of 83.5% and 82.4%, respectively. Despite the absence of interaction between different modalities, CCB still played a critical role. The CCB module incorporated an attention mechanism, enabling the model to effectively capture long-range dependencies between features. Furthermore, BiGRU was still capable of modeling sequential data, which is crucial for handling temporal data like audio. The combination of these two aspects allowed the model to perform remarkably well even with single-modal data. It is worth noting that although the DBMER model proposed by Kakuba et al. achieved better results on single-modal datasets, the proposed model surpasses its performance when dealing with multi-modal datasets. This is because the DBMER model employs multiple deep learning techniques, including CNNs, RNNs, and multi-head attention mechanisms. These techniques enhance the model’s feature extraction capability and exhibit significant advantages on single-modal data. However, on multi-modal data, despite extracting high-quality features, the DBMER model overlooks the alignment of features between different modalities, resulting in semantic misalignment. In contrast, the proposed model maintains feature quality and effectively aligns the semantic information of different modalities through the CBB module and BiGRU when modeling multi-modal.

Visual Analysis

To visually illustrate the effectiveness of the proposed model within and across various emotional categories, the confusion matrices of the model on the IEMOCAP and RAVDESS datasets are displayed in Figures 3 and 4, respectively. It can be observed from the figures that the proposed model performs best in identifying happy and sad emotions with accuracy of 87.5% and 84.8% (IEMOCAP dataset) and 89.5% and 90% (RAVDESS dataset), respectively.

Figure 3. Confusion matrix of the model in the IEMOCAP dataset.

Figure 4. Confusion matrix of the model in the REVDESS dataset.

This is because happy and sad emotions often have distinct phonetic and textual features. Happiness is often expressed by high pitch, fast speech speed, bright sound quality, and a greater tendency to use positive sentence structures, such as affirmations and exclamations. Sad feelings are often expressed by low pitch, slow speech speed, heavy voice quality, and a greater tendency to use negative sentence structures, such as negative sentences and questions. The lowest recognition rate of surprise can be seen on the RAVDESS dataset, at only 76%, because surprise can be accompanied by short pauses, sudden changes in pitch, and changes in speech speed. These features can be confused with other emotions, making recognition difficult. Furthermore, it can be noticed from Figures 3 and 4 that calmness, an emotion in the middle of the emotional dimension, is also misclassified as other emotions.

By preserving the final layer results of the model and utilizing the t-SNE visualization method, features can be plotted on a plane for visual analysis. For the IEMOCAP dataset, Figure 5 shows the feature distribution learned using the cross-entropy loss function, while Figure 6 illustrates the positions of deep features learned after incorporating enhanced contrastive learning. Different shapes of points represent features of different categories. It can be observed from Figure 5 that the cross-entropy loss function is capable of separating different emotion features, yet the differences within features remain substantial. However, Figure 6 indicates that after incorporating enhanced contrastive learning and sufficient training, the CCB module seems to better combine information across different modalities, leading to a clearer clustering structure in the t-SNE plot. Additionally, the BiGRU module can extract abstract representations of input data, making the feature points in the t-SNE plot more representative and discriminative. Overall, the introduction of the bi-task model enables the model to learn more about emotion representation. This allows the model to better capture the variations and complexities within the data, subsequently improving feature discriminability and clustering effectiveness. Moreover, parameter sharing and feature learning further enhance the model’s generalization capability and robustness. This is reflected in the t-SNE plot, where the feature points appear more consistent and stable. This visualization analysis provides a comprehensive understanding of the feature distribution learned by the model under different conditions and the influence of different tasks on features.

Figure 5. Feature distribution learned under the supervision of cross-entropy loss function on the IEMOCAP dataset.

Figure 6. Feature distributions learned under the supervision of both cross-entropy and N-pair loss function on the IEMOCAP dataset.

Ablation Experiment

Ablation tests were conducted on the IEMOCAP dataset to evaluate its impact on overall performance by progressively reducing parts of the model. In experiments, other parameters were fixed and only specific parts of the model were modified one by one. The performance of the model was evaluated after each change. This helps validate whether each part is truly necessary for the model to function effectively.

First, the two task modules CPC and ECL were deleted. Figure 7 shows that the model accuracy rate decreases the most in the absence of tasks, WA decreases by 11.7% and UA decreases by 12.9%. The increase in tasks has no significant impact on the number of model parameters, but it impacts the recognition efficiency. This is because, in MTL, two tasks share the model’s parameters. Hence, the parameter count does not increase significantly. The introduction of another task can provide more training data and training signals, speeding up the rate of convergence training efficiency of the model. Secondly, only the original speech and text were used for extraction, and voice and text enhancements were not carried out. At this point, WA was 70.6% and UA was 70.1%. WA and UA were 11.7% and 12.9% lower than the original model, respectively. This is because the enhancement of speech and text data can make the model more robust to changes in diversity factors such as noise, accent, and tone. This ultimately leads to a more accurate and consistent model. Finally, in the model fusion module, only CCB 2 and CCB 1 were used for modal fusion. It can be seen from the figure that the recognition efficiency is reduced. In the fusion phase, CCB 1 uses text-assisted speech, while CCB 2 uses voice-assisted text. The partial accuracy of missing CCB 1 decreases even more significantly, with WA decreasing by 5.3% and UA decreasing by 5.6%. This shows that the effect of audio features is more obvious in the two modes of speech and text, which is also an important reason why speech features are often used to identify emotions in previous single-modal studies. Ablation experiments demonstrate the functions of the model components and evaluate the contribution of each module to the performance.

Figure 7. Ablation experiments based on the IEMOCAP dataset.

Conclusions

This paper presents a novel and innovative bi-modal bi-task model for emotion recognition. The Transformer architecture is improved and a new bi-modal bi-task emotion recognition model is developed. The proposed model first uses wav2vec-2.0 to encode speech features and DistilBERT to encode text. These speech and linguistic features are then fed into the cross-modal coding block respectively for modal fusion. After fusion, the combined features are input into the BiGRU module, whose features and gating unit further extract the information in the features. In addition, contrastive predictive coding and enhanced contrastive learning tasks are added to the model to improve the recognition accuracy. The proposed model is compared with the existing advanced methods. The experimental comparison demonstrates that the proposed model performs the existing methods. To confirm the model’s effectiveness and importance, a range of ablation tests were carried out. These ablation experiments provide an understanding of the role of each part of the model, highlight the key points of performance improvement, and can help in optimizing the model design. It can enhance the model’s effectiveness and offer superior benchmarks and guidance for future research.

However, the proposed method still has some limitations. Firstly, constrained by the modalities of input data, the model can only handle information from two modalities: audio and text. The model is unable to process other types of inputs such as images or videos, which limits the model’s generalization capability. Secondly, there is a high demand for language proficiency, requiring further research on the model’s robustness to different languages, accents, and other interference factors. Lastly, it is crucial to design lightweight models to improve computational efficiency and facilitate the practical deployment of motion recognition technology in real-life scenarios. This means reducing the model parameters as much as possible without compromising accuracy.

Acknowledgements

The authors would like to thank the editor and anonymous reviewers for their contributions toward improving the quality of this paper.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

The data used to support the findings of this study are included within the article.

Additional information

Funding

This work was supported by the Guangdong Provincial Major Research Platform Ordinary University Characteristic Innovation Project Fund [No.2022KTSCX204].