Multi-branch feature learning based speech emotion recognition using SCAR-NET

Abstract

Speech emotion recognition (SER) is an active research area in affective computing. Recognizing emotions from speech signals helps to assess human behaviour, which has promising applications in the area of human-computer interaction. The performance of deep learning-based SER methods relies heavily on feature learning. In this paper, we propose SCAR-NET, an improved convolutional neural network, to extract emotional features from speech signals and implement classification. This work includes two main parts: First, we extract spectral, temporal, and spectral-temporal correlation features through three parallel paths; and then split-convolve-aggregate residual blocks are designed for multi-branch deep feature learning. The features are refined by global average pooling (GAP) and pass through a softmax classifier to generate predictions for different emotions. We also conduct a series of experiments to evaluate the robustness and effectiveness of SCAR-NET which can achieve 96.45%, 83.13%, and 89.93% accuracy on the speech emotion datasets EMO-DB, SAVEE, and RAVDESS. These results show the outperformance of SCAR-NET.

Keywords:

• Affective computing
• speech emotion recognition
• convolutional neural network
• parallel paths
• feature learning

1. Introduction

Emotions are vehicles for personal feelings and feedback. The understanding of emotions plays a crucial role in human-human interaction and is likewise an effective means to improve human-computer interaction. Therefore, the area of affective computing has gained great attention and made significant developments in the last decade (Zeng et al., 2009). The ultimate goal of affective computing is the automatic understanding and recognition of human emotions, and there are still many difficulties in achieving this. In fact, people often express their emotions through various mediums, such as body movements, facial expressions, speech, and physiological changes. In human-computer interaction, computers could capture these key mediums to recognise the emotional state of the users. Noticing this, researchers applied techniques from psychology, signal processing, and deep learning to affective computing, allowing computers to learn mediums such as body movements (Noroozi et al., 2021; Pławiak et al., 2016), facial expressions (Kulkarni et al., 2018; Shojaeilangari et al., 2016), speech signals (Kamińska & Pelikant, 2012; Kamińska et al., 2017; Noroozi et al., 2017), and physiological signals (Greco et al., 2016; Jenke et al., 2014), ultimately enabling emotion recognition. In earlier studies, facial expressions have received a lot of attention because of their rich emotional expressions. About 95% of the literature is based on facial expressions for affective computing (De Gelder, 2009). However, due to the advantages of low acquisition difficulty and little private information, affective computing based on speech signals is becoming the mainstream of current research.

Affective computing based on speech signals is also known as speech emotion recognition (SER). Computers with SER systems can grasp the emotional state of the users in real time and make targeted responses. This can facilitate the development of many fields such as education, medical care, and services, which is of great social importance. SER applies speech signal processing techniques to extract low-level descriptors (LLDs) from the collected speech signals such as frequency, amplitude, and pitch. And find the mapping relationship between these features and human emotions. There are already various SER methods, including traditional machine learning methods and deep learning methods. However, the acoustic features are largely influenced by the accent and intonation of different speakers, which makes identify accurate emotional features a challenging task for these methods. As a result, the existing SER methods have less accuracy in predicting emotions and tend to confound similar emotions.

We hope SER methods can be free from the influence of individual factors and learn emotional features with generalizability. To this end, we apply asymmetric convolution and multi-branch convolution to propose a novel method for SER called SCAR-NET. The main contributions of our method can be summarised as follows:

• We provide a block for multi-dimensional feature extraction called parallel path, which is an improvement of the asymmetric convolution block (Szegedy et al., 2016). The parallel paths extract spectral features, temporal features, and spectral-temporal correlation features through three independent convolutional neural networks (CNNs) with different-sized convolutional kernels.

• We design a multi-branch convolution structure to replace the single-channel one in the residual block (He et al., 2016), and propose a split-convolve-aggregate residual (SCAR) block. The SCAR blocks implement deep feature learning and help to obtain more general features.

• We select only 40 Mel frequency cepstral coefficients (MFCCs) as feature input to lessen the computational load of the network, and substitute the fully connected layers of the CNN with GAP to reduce the overall parameter quantity.

We evaluate the proposed method on the speech emotion datasets EMO-DB (Burkhardt et al., 2005), SAVEE (W. Wang, 2010), and RAVDESS (Livingstone & Russo, 2018), and we obtain 96.45%, 83.13%, and 89.93% classification accuracy. Our method provides better recognition performances than state-of-the-art methods, which indicates its application value.

2. Literature review

Emotions have the specificity of crossing language, culture, and ethnicity, giving SER great social significance. The earliest SER research emerged in the mid-1980s. Since then, a large number of researchers have devoted themselves to this field, and many effective solutions have been proposed. The original SER methods mainly relied on acoustic statistical features. The researchers manually extracted low-level features from the speech signal, such as zero-crossing rate, energy, formants, and MFCCs to implement emotion classification. Later researchers brought traditional machine learning algorithms to this field, among which the most widely used are hidden Markov models (HMMs) (Nwe et al., 2003), Gaussian mixture models (GMMs) (Daniel et al., 2006), support vector machines (SVMs) (Chavhan et al., 2010), and so on.

With the development of SER, traditional machine learning methods cannot meet the demands of continuously growing amount of data and computation (X. Wang, Yang, et al., 2022). Deep learning-based methods gradually replace traditional machine learning methods (Ren, Meng, et al., 2020). Deep neural networks (DNNs) can extract complex features from speech signals through the process of learning, thereby increasing the accuracy of recognition. With the unique advantage of ignoring task-irrelevant information (Gu et al., 2018) and processing complex unstructured signals, CNNs have made great achievements in the area of SER (Kwon, 2019). Khamparia et al. (2019) utilised CNNs to learn high-level hidden features from spectrograms to recognise emotions. The spectrograms are two-dimensional plots of the speech signals frequency with respect to time. Therefore, it is effective to obtain the features in the spectrograms in a similar way to how 2D-CNNs process images. Yenigalla et al. (2018) also took the spectrograms as input and added multiple parallel paths into CNNs, which helped to achieve better recognition rate.

The application of CNNs increased the research interest of the SER. Some researchers have further noticed that speech signals are time-series signals, which means that speech signals have a strong context dependency. Learning this dependency is of great help in identifying emotions. So recurrent neural networks (RNNs) began to be applied to learn context information in speech signals. Many researchers combined CNNs and RNNs into SER networks. Chen et al. (2018) proposed a CNN-RNNs based on a 3D attention mechanism, which can better preserve contextual emotion-related information. Ma et al. (2018) applied a similar structure to make the network capable of handling longer speech signals, where the CNN was used to extract spatial features while the RNNs was used to solve timing problems. Long short-term memory (LSTM), as an improved RNN that solves the long-term dependencies problem, was also widely used (X. Wang, Ren, et al., 2022). Trigeorgis et al. (2016) fused CNNs with LSTM and proposed a end-to-end SER method that can automatically learn the best feature description for speech signals directly from raw time series rather than the handcrafted features. Kwon (2020) utilised hierarchical blocks of the convolutional LSTM to extract spatio-temporal features in the hierarchical correlated form of speech signals.

With the help of the literature, we realise the limitations in the various existing SER methods. Traditional machine learning algorithms require manual feature extraction, which is difficult to adapt to tasks with large amounts of data. CNN-based deep learning methods are weak in processing time-series signals such as speech signals, and need to be combined with RNNs to make up for this defect (Ren, Dong, et al., 2020). However, the ability of CNN-RNNs to capture emotional features is still less than ideal. In addition, advanced SER methods often need to obtain complex acoustic feature sets from speech signals as input, which makes the computational load of the network and the time cost of training high. For example, The INTERSPEECH 2009 Emotion Challenge (IS09) feature set, a commonly used feature set in SER (Mirsamadi et al., 2017), contains 384 features obtained by applying statistical functions to LLDs.

In response to the above limitations, we propose an improved method for SER named SCAR-NET. As an enhanced CNN, SCAR-NET excels at capturing spatial features. The design of the parallel paths improves its ability to extract temporal features. The SCAR blocks enable the network to learn more sparse and diverse features. The use of GAP reduces the parameter scale of the network. Moreover, the input of SCAR-NET is only 40 MFCCs, which lowers the computational load and training cost. The experimental evaluations corroborate that our method achieves better results than state-of-the-art SER methods.

3. Proposed method

In this section, we explain the proposed method specifically. We propose a SER network named SCAR-NET which consists of four main parts: feature input, parallel paths, SCAR module, and classifier. Figure 1 and Algorithm ?? respectively illustrate the overall structure and algorithm description of SCAR-NET, and further details of each part are described below.

Figure 1. The overall structure of SCAR-NET.

3.1. Feature input

The initial input of the network is waveform audio files. However, the task-irrelevant information contained in speech signals will greatly increase the computational complexity of the network. To address this issue, feature selection is required. We perform a series of signal processing operations as follows to extract the desired features from speech signals.

1. Segmentation and Normalization: Divide the speech signals into equal-length segments of 3 s. For segments with insufficient length, zero padding is used to make up the difference. Then normalise them between −1 and 1.

2. Frame Blocking and Windowing: Split the segments into 64ms frames, with 16 ms overlaps to ensure their continuity. Then use a Hamming window to avoid the risk of spectral leakage.

3. Fast Fourier Transform: Perform a 1024-point fast Fourier transform (FFT) on each frame. Convert it from a time-domain signal to a frequency-domain signal for the subsequent frequency analysis.

4. Mel Scale Filter Bank: Use a Mel scale filter bank in the range of 40 Hz to 7600 Hz to remove the redundant part of the frequency-domain signals.

5. Inverse Discrete Cosine Transform: Perform an inverse discrete cosine transform (DCT) on the filtered signals, and then the MFCCs of each frame are calculated.

The first 40 MFCCs are selected as features, which are input to the subsequent parts of the network for training.

3.2. Parallel paths

The second part of the network implements the extraction of advanced feature information from MFCCs. Figure 2 shows the MFCC spectrum obtained from the first part. The spectrum contains information in three dimensions: spectral, temporal, and spectral-temporal correlation. For this feature input that contains multiple dimensions, we adopt the method of processing each dimension separately and then integrating them. Therefore, three parallel paths are designed to achieve multi-dimensional feature extraction. The second box in Figure 1 illustrates the structure of the parallel paths.

Figure 2. The MFCC spectrum and the receptive fields of the parallel paths.

The three paths are very similar in structure. They all achieve feature extraction through convolution, batch normalisation, activation function, and average pooling. The only difference is the size of convolution kernels. Considering that the spectral dimension and the temporal dimension are equally important for SER, we hope to apply a square convolution kernel. The asymmetric convolution blocks use $n\times 1$ and $1\times n$ sized kernels as a replacement for $n\times n$ sized one to conserve computational effort. Therefore, we draw on the asymmetric convolution blocks to use convolution kernels of sizes $9\times 1$ and $1\times 9$ to extract spectral features and temporal features. However, the spectral-temporal correlation dimension is just as critical for SER. We add a $3\times 3$ sized convolution kernel on the basis of the asymmetric convolution block to handle these features. This design brings two benefits to the network:

• Through the three independent parallel paths, multi-dimensional features can be effectively extracted. Moreover, the design of asymmetric convolution reduces the size of parameters on each path, thereby curtailing the risk of overfitting and improving the final classification accuracy of the network.

• The parallel paths effectively decrease parameter quantity. In Figure 2, the receptive fields of $9\times 1$, $1\times 9$, and $3\times 3$ size convolution kernels are represented by orange, blue, and yellow rectangles. After the combination of the three, the actual receptive field is supposed to be $9\times 9$ size, which is represented by a red dotted rectangle. The receptive field is of equal length and width to ensure that the network is not biased towards a single dimension. There is a positive relationship between classification accuracy and the size of receptive fields (Araujo et al., 2019). Therefore, compared with the single path with a $9\times 9$ size convolution kernel, the parallel paths can cut down the number of parameters in this part of the network while ensuring classification accuracy. Theoretically, the parameter quantity can be reduced to $\frac{9\times 1+1\times 9+3\times 3}{9\times 9}$ of the original size.

After the feature extraction with the parallel paths, the obtained high-level feature information of three dimensions is concatenated and then fed into the third part of the network.

3.3. SCAR module

The SCAR module is the main part of the network. The third box in Figure 1 shows the structure of this part. The beginning of this part is a transition layer that includes convolution, batch normalisation, activation function, and average pooling. The transition layer processes the high-level feature information from the parallel paths as a legitimate input. Then this information will undergo two consecutive SCAR blocks for feature learning.

In the process of feature learning, there is a problem called degradation. After reaching a certain depth, although the network continues to deepen, the training loss does not decrease but increases. The left side of Figure 3 is a residual block, which is designed to solve the degradation problem. For the same input x, a common CNN needs to fit the output $H(x)$, while a residual block is supposed to fit $F(x)=H(x)-x$. When the network reaches a certain depth, there is always a certain layer whose output is close to the optimal solution. As the next layer expects to be closer to the solution, it needs to update its weights. Since the input x is already close to the desired output $H(x)$, a residual block can only update a small part of the weights to achieve fitting. Consequently, the degeneration problem of deep networks is avoided.

Figure 3. The residual block and the SCAR block.

The SCAR block is an improvement from the residual block. The right side of Figure 3 illustrates the specifications of the SCAR block. We replace the single-path convolution of the residual block with 8 parallel identical topological structures. The entire block afterward forms an architecture of split-convolve-aggregate (SCA). The high-level feature information is split into 8 feature subspaces after being input to the SCAR block, and then the feature learning will be realised through a series of convolutional layers that gradually increase in depth. After that, the output will be aggregated and delivered to the classifier. The SCAR block inherits the advantage that the residual block can avoid network degradation, and also benefits from the SCA structure. When splitting, the original feature space is divided into 8 subspaces. Each path learns the features in its own subspace, making the learned features relatively sparse and alleviating the problem of overfitting. While aggregating, the features of each subspace are merged to enhance the width of the network. This allows the network to obtain the feature information from different subspaces, and comprehend features from multiple perspectives. As a result, the final classification accuracy is improved.

At the tail of the SCAR module, we use GAP as a replacement for several fully connected layers of CNNs. Figure 4 explains how GAP works (Lin et al., 2013). It calculates the mean and outputs data nodes for each feature map eventually output by the network. In the proposed network, the last layer outputs 64 feature maps, so GAP will output 64 data nodes as a feature vector and send it into the classifier for classification calculation. Compared to the fully connected layer, GAP itself has no parameters, which reduces the number of parameters and avoids overfitting effectively.

Figure 4. The working principle of GAP.

The last part of the network is the classifier. It maps nonlinear input space into each linearly separable subspace, which corresponds to a specific emotion classification. As shown in the last box in Figure 1, the classifier includes a Dropout layer and a softmax layer. The former ignores a part of neurons with a certain probability to reduce the occurrence of overfitting, and the latter converts the network output into the probabilistic form.

4. Experimental evaluations and analysis

In this section, we train and evaluate the proposed network on public speech emotion datasets, and the experimental results will be compared with recent papers.

4.1. Datasets

To demonstrate the robustness and effectiveness of the proposed network, we select three public speech emotion datasets to evaluate them, namely EMO-DB, SAVEE, and RAVDESS.

EMO-DB. The Berlin Emotional Speech Database is a commonly used dataset in the SER domain. The recording of the dataset was completed by 10 professional actors and actresses (5 males and 5 females) in a professional recording studio. The corpus consists of 5 long and 5 short sentences selected under the principles of neutral semantics and daily spoken language. Actors and actresses are supposed to express 7 emotions in these sentences: anger, fear, happiness, neutral, sadness, disgust, and boredom. The dataset finally retains 535 audio files with a 3–5 seconds length using a 16 kHz sampling rate after the auditory discrimination experiments.

SAVEE. Surrey Audio-Visual Expressed Emotion is a dataset consisting of recordings of 4 male actors from the University of Surrey. The dataset contains 480 audio files expressing 7 emotions (surprise, anger, happiness, neutral, sadness, disgust, and fear). The audio sampling rate is 44.1kHz, and the corpus used is British English sentences selected from the standard TIMIT corpus.

RAVDESS. The Ryerson Audio-Visual Database of Emotional Speech and Song is a newly introduced emotion corpus that is widely used in the SER methods. It contains 24 professional actors and actresses (12 males and 12 females) expressing 8 emotions in a neutral North American accent, including angry, calm, disgust, fearful, happy, neutral, sad, and surprise. Each emotion, except neutral, is produced at both normal and strong intensities. The dataset includes 1440 audio files with a sample rate of 48 kHz and an average length of 3.5 s.

Table 1 illustrates the emotion classifications of the three datasets and the proportion of each emotion.

Table 1. The detailed descriptions of EMO-DB, SAVEE, and RAVDESS datasets.

	EMO-DB		SAVEE		RAVDESS
Emotions	Total files	Proportion(%)	Total files	Proportion(%)	Total files	Proportion(%)
Anger	127	23.7	60	12.5	192	13.3
Fear	69	12.9	60	12.5	192	13.3
Happiness	71	13.3	60	12.5	192	13.3
Neutral	79	14.8	120	25.0	96	6.7
Sadness	62	11.6	60	12.5	192	13.3
Disgust	46	8.6	60	12.5	192	13.3
Boredom	81	15.1	–	–	–	–
Surprise	–	–	60	12.5	192	13.3
Calm	–	–	–	–	192	13.3

4.2. Experimental setup

Experimental Environment. The proposed network is implemented in a Python 3.7 virtual environment, using version 2.3.0 of the TensorFlow deep learning framework. Both training and evaluation are run on a single Nvidia GeForce RTX 2080 Ti GPU.

Hyperparameters. We train the network model for 300 epochs and 32 batch sizes. The initial learning rate is ${10}^{-4}$. According to the decay strategy, starting from 50 rounds, the learning rate decays at a rate of ${\mathrm{e}}^{-0.3}$ every 20 rounds. The ratio of Dropout in the classifier is set to 0.3.

Evaluation Strategy. Our experimental evaluations apply a leave-one-speaker-out (LOSO) (Schuller et al., 2009) cross-validation strategy. LOSO is a special K-fold cross-validation strategy, where K is the number of speakers in the dataset. LOSO divides the dataset into K groups so that each group corresponds to a speaker. Each time we select the audio data of one speaker as the test set, while the audio data of the remaining K-1 people are used as the training set for experiments. After repeating the experiment for K times, we take the mean value as the final experimental results. LOSO can make full use of the data so that the experimental results better represent the classification accuracy of the network. Among the three datasets we selected, EMO-DB has 10 speakers and SAVEE has 4 speakers. Therefore, the experimental evaluations are based on 10-fold cross-validation and 4-fold cross-validation, respectively. RAVDESS, with 24 speakers, is not suitable for LOSO. We divide the speakers into 8 groups and perform 8-fold cross-validation.

Evaluation Metrics. As the number of audio files for each emotion classification in the dataset is not balanced, using a single evaluation metric can only reflect the performance of the network one-sidedly. We choose a variety of evaluation metrics to evaluate the network, including precision, recall, F1_score, as well as their macro average and weighted average. Accuracy and confusion matrices are also used as evaluation metrics.

4.3. Experimental evaluations

Experimental evaluation is an important link in the deep learning process and a key step to verify the effectiveness of the network. We evaluate the practical performance of the proposed network on EMO-DB, SAVEE and RAVDESS using multiple evaluation metrics, demonstrating that our method can provide better classification accuracy. The evaluation results will be presented in the form of a classification report and three confusion matrices.

Table 2 is a classification report generated after evaluating SCAR-NET on the datasets. It lists the precision, recall, and F1_score of the network model for recognising each emotion classification. There are also the macro average, weighted average, and accuracy for the overall evaluation of the datasets. Through this report, a comprehensive understanding of the recognition strength of each emotion classification, as well as the overall performance of the model can be gained.

Table 2. The classification report of SCAR-NET evaluated on datasets.

	Emo-DB			SAVEE			RAVDESS
Emotions	Precision	Recall	F1_score	Precision	Recall	F1_score	Precision	Recall	F1_score
Anger	0.9542	0.9843	0.9690	0.8793	0.8500	0.8644	0.9278	0.9375	0.9326
Fear	0.9444	0.9855	0.9645	0.7895	0.7500	0.7692	0.8744	0.9062	0.8900
Happiness	0.9545	0.8873	0.9197	0.7727	0.8500	0.8095	0.9231	0.8750	0.8984
Neutral	0.9747	0.9747	0.9747	0.8819	0.9333	0.9069	0.8495	0.8229	0.8360
Sadness	0.9836	0.9677	0.9756	0.8154	0.8833	0.8480	0.8703	0.8385	0.8541
Disgust	1.0000	0.9565	0.9778	0.9070	0.6500	0.7573	0.9399	0.8958	0.9173
Boredom	0.9634	0.9753	0.9693	–	–	–	–	–	–
Surprise	–	–	–	0.7500	0.8000	0.7742	0.9211	0.9115	0.9162
Calm	–	–	–	–	–	–	0.8692	0.9688	0.9163
Macro	0.9678	0.9616	0.9644	0.8280	0.8167	0.8185	0.8969	0.8945	0.8951
Weighted	0.9648	0.9645	0.9643	0.8347	0.8313	0.8296	0.9001	0.8993	0.8991
Accuracy	96.45%			83.13%			89.93%

In addition to the classification report, we also compare the actual emotion classification labels provided by the datasets with the predicted labels from the network model and draw the confusion matrix. The confusion matrix shows the proportion of audio data that is correctly or incorrectly classified by the model. The horizontal axis represents the predicted labels, and the vertical axis represents the actual labels. The intersection of each emotion classification with others represents the confusion ratio between them. The diagonal values are the recall for each emotion classification.

Figures 5–7 show the confusion matrices of SCAR-NET for EMO-DB, SAVEE, and RAVDESS. It can be seen that our method achieves excellent evaluation results on EMO-DB. Except for happiness, the classification recall is above 95% for all emotions. The evaluation results on SAVEE and RAVDESS are relatively lower than those on EMO-DB. However, they are still better than the SER method proposed in recent years, which will be explained detailedly in the experimental results.

Figure 5. The confusion matrix of SCAR-NET evaluated on EMO-DB.

Figure 6. The confusion matrix of SCAR-NET evaluated on SAVEE.

Figure 7. The confusion matrix of SCAR-NET evaluated on RAVDESS.

4.4. Experimental results and analysis

For the purpose of determining the optimal structure of the network, we conduct a series of ablation experiments and comparative experiments on EMO-DB. After discussing and analysing the experimental results, we also compare our method with the state-of-the-art SER methods to demonstrate its effective improvement in evaluation results.

Effect of Parallel Paths. We evaluate the improvement effect of parallel paths on the network through an ablation experiment. Table 3 shows the experimental results. After replacing the single path ($9\times 9$ kernel) of the same size receptive field with the parallel paths ($9\times 1$, $1\times 9$, and $3\times 3$ kernels), the average precision, recall, and F1_score of the experimental evaluations are improved. Furthermore, the number of parameters in this part is reduced from 2624 to 960, and the final classification accuracy is improved by 1.31%, which proves that the multi-dimensional feature extraction using the parallel paths can effectively improve the classification accuracy while reducing the parameter quantity. This is achieved because the parallel paths decrease the number of parameters on each path, thereby suppressing the occurrence of overfitting.

Table 3. The effect of parallel paths on experimental evaluation.

		Methods
Metrics		Single Path	Parallel Paths
Macro	Precision	0.9581	0.9678
	Recall	0.9437	0.9616
	F1_score	0.9494	0.9644
Weighted	Precision	0.9529	0.9648
	Recall	0.9514	0.9645
	F1_score	0.9508	0.9643
Parameters		2624	960
Accuracy(%)		95.14	96.45

Impact of SCA Structures. To determine the impact of SCA structures on the network, controlled experiments are required. Firstly, we set a network with 1 branch as a control group, which represents a network without the SCA structure. Then we set networks with 2, 4, 6, 8, and 10 branches as comparative experiments. The experimental results are shown in Table 4. Compared with the control group, the networks with the SCA structure obtains higher evaluation metrics, which proves that the application of the SCA structure can play a positive role in classification accuracy. For the networks with different numbers of branches, as the number increases to 8, the evaluation metrics gradually improve to the highest. When the number continues to increase, not only does it take more training time, but the evaluation metrics drop. This may be caused by the phenomenon of overfitting, which is triggered by a huge parameter quantity from excessive branches. Consequently, we apply the SCA structure with 8 branches in the network to ensure optimal classification accuracy.

Table 4. The impact of SCA structure with diverse branch numbers on experimental evaluation.

		Number of SCA branches
Metrics		1	2	4	6	8	10
Macro	Precision	0.9546	0.9595	0.9626	0.9656	0.9678	0.9554
	Recall	0.9475	0.9525	0.9532	0.9576	0.9616	0.9504
	F1_score	0.9502	0.9546	0.9568	0.9598	0.9644	0.9524
Weighted	Precision	0.9523	0.9566	0.9600	0.9626	0.9648	0.9536
	Recall	0.9514	0.9551	0.9589	0.9607	0.9645	0.9533
	F1_score	0.9510	0.9545	0.9585	0.9600	0.9643	0.9531
Accuracy(%)		95.14	95.51	95.89	96.07	96.45	95.33

Impact of SCAR Blocks. There is a direct relation between the number of SCAR blocks and the depth of networks. Properly deepening networks is beneficial to feature learning. However, excessive network depth will cause a decrease of classification accuracy. Accordingly, we design controlled experiments to evaluate networks containing 1 to 4 SCAR blocks. Table 5 shows the experimental results. It can be seen that the network containing 2 SCAR blocks has the highest evaluation metrics. Afterward, the classification accuracy goes down with the number of blocks increases. This indicates that 2 SCAR blocks allow the network to reach an appropriate depth, and further increasing the depth will result in overfitting problems leading to network performance degradation. Eventually, we determine to utilise 2 SCAR blocks in the network for feature learning.

Table 5. The impact of diverse SCAR blocks numbers on experimental evaluation.

		Number of SCAR blocks
Metrics		1	2	3	4
Macro	Precision	0.9603	0.9678	0.9554	0.9453
	Recall	0.9560	0.9616	0.9520	0.9383
	F1_score	0.9576	0.9644	0.9531	0.9410
Weighted	Precision	0.9592	0.9648	0.9557	0.9447
	Recall	0.9589	0.9645	0.9551	0.9439
	F1_score	0.9586	0.9643	0.9549	0.9437
Accuracy(%)		95.89	96.45	95.51	94.39

Comparison with Recent Methods. After determining the optimal structure of the network, we choose 6 recent papers on each of the three datasets, summary their methods, and compare them with ours. The comparative results can be found in Table 6. Compared with the state-of-the-art methods, our method improves the classification accuracy by 1.45%, 1.13%, and 2.09% respectively on EMO-DB, SAVEE, and RAVDESS. Compared to EMO-DB (535) and SAVEE (480), RAVDESS (1440) has a larger data size and obtains a greater improvement. This may indicate that the large dataset is more suitable for our deep network. The evaluations above prove the robustness and effectiveness of the proposed network. Furthermore, the optimal structure determined on EMO-DB also achieves ideal results on SAVEE and RAVDESS, proving its excellent generalizability.

Table 6. Comparative analysis of SCAR-NET with recent SER methods.

Datasets	Reference	Methods	Accuracy(%)
EMO-DB	Sajjad and Kwon (2020)	CNN-BiLSTM with RBFN (Redial Based Function Network)	85.57
	Anvarjon and Kwon (2020)	CNN with the plain rectangular filter and modified pooling strategy	89.02
	Kwon and Mustaqeem (2021)	Dilated CNN with RBSC (residual blocks with a skip connection)	90.01
	Mustaqeem and Kwon (2021)	CNN with HFLBs (hierarchical features learning blocks) and BiGRU (bi-directional gated recurrent unit)	91.14
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	95.00
	Aftab et al. (2022)	Fully CNN with LFLBs (local feature learning blocks)	94.21
	Our Method	SCAR-NET	96.45
SAVEE	Thakare et al. (2019)	CNN with MFCCs input	72.92
	Avots et al. (2019)	SVM with MFCCs input	77.40
	Nguyen et al. (2020)	VGG-16 (visual geometry group) with LReLU (leaky rectified linear units)	69.00
	Assunção et al. (2020)	VGGVox with spectrograms input	70.40
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	82.00
	Li et al. (2022)	SVM with ReliefF feature selection algorithm	81.25
	Our Method	SCAR-NET	83.13
RAVDESS	Mustaqeem and Kwon (2021)	CNN with HFLBs (hierarchical features learning blocks) and BiGRU (bi-directional gated recurrent unit)	78.01
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	85.00
	Luna-Jiménez et al. (2021)	Pre-trained xlsr-Wav2Vec2.0 transformer using embedding extraction and fine-tuning	86.70
	Shinde et al. (2022)	SVM with PCA (principle component analysis) feature selection algorithm	80.21
	Sowmya et al. (2022)	MLP (multilayer perceptron) with the converged communication of frequency	85.00
	Alaparthi et al. (2022)	Pre-trained Wav2Vec2.0 with the supervised contrastive pipeline	87.84
	Our Method	SCAR-NET	89.93

5. Conclusion and future direction

In this paper, we propose a more effective SER network named SCAR-NET in view of the defects existing in the current methods. SCAR-NET computes MFCCs from the input audios and uses parallel paths with three convolution kernels of different sizes to extract multi-dimensional features. Feature learning is achieved by the residual block with the split-convolve-aggregate structure. In addition, we discuss the impact of the number of SCA branches and SCAR blocks on the network performance through experiments and finally determine the optimal structure of the network. Based on the above work, our method achieves 96.45%, 83.13%, and 89.93% classification accuracy on the speech emotion datasets EMO-DB, SAVEE, and RAVDESS, proving its robustness and effectiveness. The limitation of our method is: the SCAR module improves network performance by using a number of repetitive structures, which raises the overall parameter quantity of the network.

In the future, we will further tune and improve SCAR-NET in pursuit of better performance and lower parameter quantity, as well as evaluate its capability in applications over a more realistic corpus. We will also try to apply SCAR-NET in the process of depression recognition. Depression, as an affective disorder, changes the emotional response of patients. Extracting emotional features in speech with SER methods can be great helpful in the recognition of depression.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Basic Public Welfare Research Project of Zhejiang Province [grant number LGG22F020014] and the National Natural Science Foundation of China [grant number 62072410].