Knowledge enhancement for speech emotion recognition via multi-level acoustic feature

Figures & data

Figure 1. The framework of multi-level feature cross-fusion for SER.

Figure 2. The structure of multi-head cross-level attention for the cross-fusion module.

Table 1. The shape settings of the input and output of each module of cross-fusion.

Layers	Input shape	Output shape
Alexnet	[3, 256, 348]	[256, 36]
Bilstm	[300, 40]	[300, 512]
Wav2vec2.0	[32, 48000]	[149, 768]
Conv1d(s)	[256, 36]	[256, 60]
Conv1d(m)	[300, 512]	[300, 60]
Conv1d(w)	[149, 768]	[149, 60]
Cross-fusion module	[256, 60], [300, 60], [149, 60]	[120]
Linear classifier	[120]	[4]

Table 2. Performance of different methods for emotion recognition on the dataset IEMOCAP using 5-fold and 10-fold cross-validation. The scores of WA and UA are reported.

		5-fold		10-fold
Prefrence	Models	WA	UA	WA	UA
Li et al. [30]	CNN_TF_Att.pooling	/	/	71.75	68.06
Wu et al. [31]	CNN GRU-SeqCap	/	/	72.73	59.71
Peng et al. [32]	ASRNN	/	62.60	/	/
Wu et al. [33]	Audio25	60.64±1.96	61.32±2.26	/	/
Guo et al. [34]	CNN-ELM+STC attention	61.32	60.43	/	/
Cao et al. [35]	HNSD	/	/	70.50	72.50
Zou et al. [6]	Co-attention	68.06	69.14	69.70	70.67
Ours	Cross-fusion	70.98	71.76	72.04	73.26

Figure 3. Final normalised confusion matrix: comparison of baseline's reproduced results and experimental results of the proposed fusion model under 10-fold cross-validation.

Table 3. The classification report of Cross-Fusion evaluated on IEMOCAP.

Emotions	Precision	Recall	F1_score
Anger	0.7477	0.8200	0.7845
Sad	0.7163	0.7500	0.7328
Happy	0.7175	0.6877	0.7022
Neutral	0.6927	0.6534	0.6725

Table 4. The table shows the results of the ablation study of cross-fusion on our model. The best results are highlighted in bold font.

Model	Description	WA	UA
S	Only spectrogram features	60.59	60.90
M	Only MFCC features	56.83	58.36
W	Only Wav2vec2 features	64.39	64.83
M+S+W	Connect the three features	67.38	68.84
S${}^{\prime }$(W&M-¿S)	Separately fuse W and M with S	71.68	72.61
M${}^{\prime }$(W&S-¿M)	Separately fuse W and S with M	70.09	71.59
W${}^{\prime }$(M&S-¿W)	Separately fuse M and S with W	71.26	71.61
S${}^{\prime }$+M${}^{\prime }$+W${}^{\prime }$	Connect the three fusion features	70.41	71.01
(M+S+W) ⊕ G	Connection and multi-task learning	68.00	69.48
(W&M-¿S) ⊕ G	Cross-fusion and multi-task learning	72.04	73.26

Table 5. The table displays the emotion recognition accuracy attained by our model under varying initial values of the parameter α.

	5-fold			10-fold
α	WA	UA	UA_gen	WA	UA	UA_gen
0	70.18	71.13	45.52	71.68	72.61	47.04
0.01	70.23	71.27	53.54	71.76	72.76	54.84
0.1	71.48	72.26	83.74	72.34	73.76	85.92
1	64.18	65.94	96.05	65.47	66.93	98.14