Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Figures & data

Figure 1. The process of multi-modal feature extraction with cross-modal modeling.

Figure 2. The overview of multi-tensor fusion network with three-peak modalities.

Table 1. Hyperparameters of MTFN-CMM

Hyperparameters	CMU-MOSI	CMU-MOSEI
Batch Size	24	8
Optimizer	AdamW	AdamW
Bi-LSTM Hidden Unit Size	128	128
Attention Dense Layer Unit Size	64	128
Dense Dropout	0.2	0.1
FC-Layer Unit Size	64	128
Output Dropout	0.3	0.1
Gradient Clip	1.0	1.0
Epochs	50	30

Table 2. Regression experimental results compared with other baselines

Metric	CMU-MOSI					CMU-MOSEI
	MAE	r	F1	A2	A7	MAE	r	F1	A2	A5	A7
DFG	–	–	–	-	–	0.710	0.540	77.0	76.9	45.1	45.0
EF-LSTM	1.023	0.622	75.2	75.4	32.4	0.642	0.616	77.9	76.2	44.7	43.9
MARN	0.968	0.625	77.0	77.1	34.7	–	–	–	-	-	–
MFM	0.951	0.662	78.1	78.1	36.2	–	–	–	-	-	–
MCTN	0.909	0.676	79.1	79.3	–	0.609	0.670	80.6	79.8	-	–
CIA	0.914	0.689	79.5	79.9	38.92	0.680	0.590	78.2	80.4	49.2	50.1
TFN	0.946	0.640	77.1	77.2	34.9	0.617	0.646	76.6	76.4	44.9	44.0
RTN	0.942	0.661	79.2	79.2	35.2	0.609	0.650	78.2	77.9	45.6	45.1
MTFN-CMM(ours)	0.891	0.691	81.0	80.9	38.92	0.589	0.674	81.3	80.8	49.3	50.1
T-test	0.0019	0.0002	0.005	0.0018	0.0003	0.0002	0.00001	0.036	0.037	0.00003	0.000001

Table 3. Coupling experimental results on multi-tensor fusion network with cross-modal modeling with different attention mechanisms

Metric	CMU-MOSI			CMU-MOSEI
	MAE	r	F1	MAE	r	F1
MTFN	0.942	0.661	79.2	0.609	0.650	78.2
MTFN-CMM(SM)	0.918	0.612	78.6	0.605	0.670	77.2
MTFN-CMM(MM)	0.908	0.684	80.1	0.598	0.671	80.4
MTFN-CMM(ours)	0.891	0.691	81.0	0.589	0.674	81.3

Figure 3. An example of the cross attention weight matrix heat map of ten video segments.