Multi-branch feature learning based speech emotion recognition using SCAR-NET

Figures & data

Figure 1. The overall structure of SCAR-NET.

Figure 2. The MFCC spectrum and the receptive fields of the parallel paths.

Figure 3. The residual block and the SCAR block.

Figure 4. The working principle of GAP.

Table 1. The detailed descriptions of EMO-DB, SAVEE, and RAVDESS datasets.

	EMO-DB		SAVEE		RAVDESS
Emotions	Total files	Proportion(%)	Total files	Proportion(%)	Total files	Proportion(%)
Anger	127	23.7	60	12.5	192	13.3
Fear	69	12.9	60	12.5	192	13.3
Happiness	71	13.3	60	12.5	192	13.3
Neutral	79	14.8	120	25.0	96	6.7
Sadness	62	11.6	60	12.5	192	13.3
Disgust	46	8.6	60	12.5	192	13.3
Boredom	81	15.1	–	–	–	–
Surprise	–	–	60	12.5	192	13.3
Calm	–	–	–	–	192	13.3

Table 2. The classification report of SCAR-NET evaluated on datasets.

	Emo-DB			SAVEE			RAVDESS
Emotions	Precision	Recall	F1_score	Precision	Recall	F1_score	Precision	Recall	F1_score
Anger	0.9542	0.9843	0.9690	0.8793	0.8500	0.8644	0.9278	0.9375	0.9326
Fear	0.9444	0.9855	0.9645	0.7895	0.7500	0.7692	0.8744	0.9062	0.8900
Happiness	0.9545	0.8873	0.9197	0.7727	0.8500	0.8095	0.9231	0.8750	0.8984
Neutral	0.9747	0.9747	0.9747	0.8819	0.9333	0.9069	0.8495	0.8229	0.8360
Sadness	0.9836	0.9677	0.9756	0.8154	0.8833	0.8480	0.8703	0.8385	0.8541
Disgust	1.0000	0.9565	0.9778	0.9070	0.6500	0.7573	0.9399	0.8958	0.9173
Boredom	0.9634	0.9753	0.9693	–	–	–	–	–	–
Surprise	–	–	–	0.7500	0.8000	0.7742	0.9211	0.9115	0.9162
Calm	–	–	–	–	–	–	0.8692	0.9688	0.9163
Macro	0.9678	0.9616	0.9644	0.8280	0.8167	0.8185	0.8969	0.8945	0.8951
Weighted	0.9648	0.9645	0.9643	0.8347	0.8313	0.8296	0.9001	0.8993	0.8991
Accuracy	96.45%			83.13%			89.93%

Figure 5. The confusion matrix of SCAR-NET evaluated on EMO-DB.

Figure 6. The confusion matrix of SCAR-NET evaluated on SAVEE.

Figure 7. The confusion matrix of SCAR-NET evaluated on RAVDESS.

Table 3. The effect of parallel paths on experimental evaluation.

		Methods
Metrics		Single Path	Parallel Paths
Macro	Precision	0.9581	0.9678
	Recall	0.9437	0.9616
	F1_score	0.9494	0.9644
Weighted	Precision	0.9529	0.9648
	Recall	0.9514	0.9645
	F1_score	0.9508	0.9643
Parameters		2624	960
Accuracy(%)		95.14	96.45

Table 4. The impact of SCA structure with diverse branch numbers on experimental evaluation.

		Number of SCA branches
Metrics		1	2	4	6	8	10
Macro	Precision	0.9546	0.9595	0.9626	0.9656	0.9678	0.9554
	Recall	0.9475	0.9525	0.9532	0.9576	0.9616	0.9504
	F1_score	0.9502	0.9546	0.9568	0.9598	0.9644	0.9524
Weighted	Precision	0.9523	0.9566	0.9600	0.9626	0.9648	0.9536
	Recall	0.9514	0.9551	0.9589	0.9607	0.9645	0.9533
	F1_score	0.9510	0.9545	0.9585	0.9600	0.9643	0.9531
Accuracy(%)		95.14	95.51	95.89	96.07	96.45	95.33

Table 5. The impact of diverse SCAR blocks numbers on experimental evaluation.

		Number of SCAR blocks
Metrics		1	2	3	4
Macro	Precision	0.9603	0.9678	0.9554	0.9453
	Recall	0.9560	0.9616	0.9520	0.9383
	F1_score	0.9576	0.9644	0.9531	0.9410
Weighted	Precision	0.9592	0.9648	0.9557	0.9447
	Recall	0.9589	0.9645	0.9551	0.9439
	F1_score	0.9586	0.9643	0.9549	0.9437
Accuracy(%)		95.89	96.45	95.51	94.39

Table 6. Comparative analysis of SCAR-NET with recent SER methods.

Datasets	Reference	Methods	Accuracy(%)
EMO-DB	Sajjad and Kwon (2020)	CNN-BiLSTM with RBFN (Redial Based Function Network)	85.57
	Anvarjon and Kwon (2020)	CNN with the plain rectangular filter and modified pooling strategy	89.02
	Kwon and Mustaqeem (2021)	Dilated CNN with RBSC (residual blocks with a skip connection)	90.01
	Mustaqeem and Kwon (2021)	CNN with HFLBs (hierarchical features learning blocks) and BiGRU (bi-directional gated recurrent unit)	91.14
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	95.00
	Aftab et al. (2022)	Fully CNN with LFLBs (local feature learning blocks)	94.21
	Our Method	SCAR-NET	96.45
SAVEE	Thakare et al. (2019)	CNN with MFCCs input	72.92
	Avots et al. (2019)	SVM with MFCCs input	77.40
	Nguyen et al. (2020)	VGG-16 (visual geometry group) with LReLU (leaky rectified linear units)	69.00
	Assunção et al. (2020)	VGGVox with spectrograms input	70.40
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	82.00
	Li et al. (2022)	SVM with ReliefF feature selection algorithm	81.25
	Our Method	SCAR-NET	83.13
RAVDESS	Mustaqeem and Kwon (2021)	CNN with HFLBs (hierarchical features learning blocks) and BiGRU (bi-directional gated recurrent unit)	78.01
	Kwon (2021)	Two-stream DNN with an INCA (interative neighbourhood component analysis)	85.00
	Luna-Jiménez et al. (2021)	Pre-trained xlsr-Wav2Vec2.0 transformer using embedding extraction and fine-tuning	86.70
	Shinde et al. (2022)	SVM with PCA (principle component analysis) feature selection algorithm	80.21
	Sowmya et al. (2022)	MLP (multilayer perceptron) with the converged communication of frequency	85.00
	Alaparthi et al. (2022)	Pre-trained Wav2Vec2.0 with the supervised contrastive pipeline	87.84
	Our Method	SCAR-NET	89.93

Sajjad, M., & Kwon, S. (2020). Clustering-based speech emotion recognition by incorporating learned features and deep BiLSTM. IEEE Access, 8(2020), 79861–79875. https://doi.org/10.1109/Access.6287639

 Google Scholar

Anvarjon, T., & Kwon, S. (2020). Deep-net: A lightweight CNN-based speech emotion recognition system using deep frequency features. Sensors, 20(18), 5212. https://doi.org/10.3390/s20185212

 PubMed Web of Science ®Google Scholar

Kwon, S., Mustaqeem (2021). MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach. Expert Systems with Applications, 167(2021), Article 114177. https://doi.org/10.1016/j.eswa.2020.114177

 Google Scholar

Mustaqeem,  & Kwon, S. (2021). 1D-CNN: Speech emotion recognition system using a stacked network with dilated CNN features. CMC-Computers Materials & Continua, 67(3), 4039–4059. https://doi.org/10.32604/cmc.2021.015070

 Web of Science ®Google Scholar

Kwon, S. (2021). Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network. International Journal of Intelligent Systems, 36(9), 5116–5135. https://doi.org/10.1002/int.v36.9

 Web of Science ®Google Scholar

Aftab, A., Morsali, A., Ghaemmaghami, S., & Champagne, B. (2022). Light-SERNet: A lightweight fully convolutional neural network for speech emotion recognition. In ICASSP 2022–2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6912–6916). IEEE.

 Google Scholar

Thakare, C., Chaurasia, N. K., Rathod, D., Joshi, G., & Gudadhe, S. (2019). Comparative analysis of emotion recognition system. International Research Journal of Engineering and Technology, 6(12), 380–384.

 Google Scholar

Avots, E., Sapiński, T., Bachmann, M., & Kamińska, D. (2019). Audiovisual emotion recognition in wild. Machine Vision and Applications, 30(5), 975–985. https://doi.org/10.1007/s00138-018-0960-9

 Web of Science ®Google Scholar

Nguyen, D., Sridharan, S., Nguyen, D. T., Denman, S., Tran, S. N., Zeng, R., & Fookes, C. (2020). Joint deep cross-domain transfer learning for emotion recognition. arXiv preprint arXiv:2003.11136.

 Google Scholar

Assunção, G., Menezes, P., & Perdigão, F. (2020). Speaker awareness for speech emotion recognition. International Journal of Online and Biomedical Engineering, 16(4), 15–22. https://doi.org/10.3991/ijoe.v16i04.11870

 Web of Science ®Google Scholar

Li, G. M., Liu, N., & Zhang, J. A. (2022). Speech emotion recognition based on modified reliefF. Sensors, 22(21), 8152. https://doi.org/10.3390/s22218152

 PubMed Web of Science ®Google Scholar

Luna-Jiménez, C., Kleinlein, R., Griol, D., Callejas, Z., Montero, J. M., & Fernández-Martínez, F. (2021). A proposal for multimodal emotion recognition using aural transformers and action units on RAVDESS dataset. Applied Sciences, 12(1), 327. https://doi.org/10.3390/app12010327

 Google Scholar

Shinde, A. S., Patil, V. V., Khadse, K. R., Jadhav, N., Joglekar, S., & Hatwalne, M. (2022). ML based speech emotion recognition framework for music therapy suggestion system. In 2022 6th international conference on computing, communication, control and automation (ICCUBEA) (pp. 1–5). IEEE.

 Google Scholar

Sowmya, G., Naresh, K., Sri, J. D., Sai, K. P., & Indira, D. V. (2022). Speech2Emotion: Intensifying emotion detection using MLP through RAVDESS dataset. In 2022 international conference on electronics and renewable systems (ICEARS) (pp. 1–3). IEEE.

 Google Scholar

Alaparthi, V. S., Pasam, T. R., Inagandla, D. A., Prakash, J., & Singh, P. K. (2022). ScSer: Supervised contrastive learning for speech emotion recognition using transformers. In 2022 15th international conference on human system interaction (HSI) (pp. 1–7). IEEE.

 Google Scholar