An Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning

Figures & data

Figure 1. The main framework of the proposed MMJL network.

Figure 2. Network structure of MSTF.

Figure 3. Network structure of MCIF.

Figure 4. Cross-modal feature fusion module EMT, where only the first and Lth layers are shown.

Table 1. Statistical table of dataset division.

Dataset	Positive	Neutral	Negative	Total
MSVA-Single	2683	470	1358	4511
MSVA-Multiple	11318	4408	1298	17024

Table 2. Performance comparison of MVSA dataset.

Model	MVSA-Single		MVSA-Multiple		Complexity
	Accuracy (%)	F1 (%)	Accuracy (%)	F1 (%)	FLOPs (M)
MultiSentiNet (Xu and Mao 2017)	69.84	69.63	68.86	68.11	25.3
CNN-Multi (Cai and Xia 2015)	61.20	58.37	66.39	64.19	29.7
DNN-LR (Yu et al. 2016)	61.42	61.03	67.86	66.33	31.1
VAuLT (Chochlakis et al. 2022)	72.80	71.80	70.30	67.00	34.7
MVAN (Yang et al. 2020)	72.98	72.98	72.36	72.30	32.9
ITIN (Zhu et al. 2023)	75.19	74.97	73.52	73.49	35.1
MLFC (Wang et al. 2023)	75.33	73.46	70.53	67.97	38.6
MultiPoint (Yang et al. 2023)	70.13	68.26	68.55	66.33	27.7
LXMERT-MMSA (Yin et al. 2024)	-	-	69.40	67.90	37.2
MDSE (Li et al. 2024)	76.22	75.71	72.31	70.12	35.1
CIGNN (Wang, Ren, and Yu 2024)	75.11	73.33	73.41	70.96	29.6
MMJL(Ours)	76.98	76.23	75.32	75.29	33.6

Table 3. Results of ablation studies on the MVSA dataset.

Method	MVSA-Single		MVSA-Multiple
	Accuracy (%)	F1 (%)	Accuracy (%)	F1 (%)
MMJL w/o MCIF only GGCN	74.32	73.95	73.17	72.97
MMJL w/o MCIF only MHSA	73.98	73.55	73.10	72.96
MMJL w/o MSTF only ViT	73.96	73.46	73.06	72.86
MMJL w/o EMT	72.36	72.10	71.68	71.54
MMJL w/o INIT	75.64	75.12	74.39	74.20
MMJL w/o AMTF	74.65	73.23	74.39	74.20
MMJL(Ours)	76.98	76.23	75.32	75.29

Figure 5. Hyperparameter analysis of different $\mathit{\lambda }$ in Equation (16)(16) $\mathit{Loss}=\mathit{Los}{s}_t+\mathit{Los}{s}_v+\mathit{Los}{s}_f+\mathit{\lambda gLos}{s}_{\mathit{all}}$(16) for proposed MMJL.

Figure 6. The impact of initial learning rate on sentiment classification results.

Table 4. Case analysis results of the proposed MMJL model on the MVSA-Multiple dataset.

Sample	Text	Image	True value	MMJL w/o EMT	MMJL w/o INIT	MMJL w/o AMTF	MMJL(Ours)
(a)	Happy new year~		Positive	P (√)	NT (×)	P (√)	P (√)
(b)	This rain won’t stop team		Positive	P (√)	P (√)	P (√)	P (√)
(c)	So my family just tried kicking me off the tv no bitch		Negative	P (×)	NG (√)	NG (√)	NG (√)
(d)	We just can’t get enough of these summer sunsets		Negative	NG (√)	NG (√)	NT (×)	NG (√)
(e)	Luke when he see you aren’t voting		Neutral	NG (×)	NT (√)	NT (√)	NT (√)
(f)	Most Alberta trout streams threatened: survey		Negative	NT (×)	NT (×)	NT (×)	NT (×)

*P represents Positive, NT represents Neutral, and NG represents Negative.

Xu, N., and W. Mao. 2017. MultiSentiNet: A deep semantic network for multimodal sentiment analysis. In Proceedings of the 2017 ACM on Conference on Information & Knowledge Management, 2399–402. doi:10.1145/3132847.3133142.

 Google Scholar

Cai, G., and B. Xia. 2015. Convolutional neural networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Computing: 4th CCF Conference, NLPCC 2015, Proceedings 4, Nanchang, China, October 9–13, 2015, 159–167. Springer International Publishing.

 Google Scholar

Yu, Y., H. Lin, J. Meng, and Z. Zhao. 2016. Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9(2):41. doi:10.3390/a9020041.

 Web of Science ®Google Scholar

Chochlakis, G., T. Srinivasan, J. Thomason, and S. S. Narayanan. 2022. VAuLT: Augmenting the vision-and-language transformer for sentiment classification on social media. arXiv preprint arXiv:2208.09021.

 Google Scholar

Yang, X., S. Feng, D. Wang, and Y. Zhang. 2020. Image-text multimodal emotion classification via multi-view attentional network. IEEE Transactions on Multimedia 23:4014–26. 10.1109/TMM.2020.3035277.

 Web of Science ®Google Scholar

Zhu, T., L. Li, J. Yang, S. Zhao, H. Liu, and J. Qian. 2023. Multimodal sentiment analysis with image-text interaction network. IEEE Transactions on Multimedia 25:3375–85. doi:10.1109/tmm.2022.3160060.

 Web of Science ®Google Scholar

Wang, H., X. Li, Z. Ren, M. Wang, and C. Ma. 2023. Multimodal sentiment analysis representations learning via contrastive learning with condense attention fusion. Sensors 23(5):2679. doi:10.3390/s23052679.

 PubMed Web of Science ®Google Scholar

Yang, X., S. Feng, D. Wang, P. Hong, and S. Poria. 2023. Few-shot multimodal sentiment analysis based on multimodal probabilistic fusion prompts. In Proceedings of the 31st ACM International Conference on Multimedia, 6045–6053.

 Google Scholar

Yin, Z., Y. Du, Y. Liu, and Y. Wang. 2024. Multi-layer cross-modality attention fusion network for multimodal sentiment analysis. Multimedia Tools & Applications 83(21):60171–87. doi:10.1007/s11042-023-17685-9.

 Google Scholar

Li, J., C. Wang, Z. Luo, Y. Wu, and X. Jiang. 2024. Modality-dependent sentiments exploring for multi-modal sentiment classification. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7930–7934, IEEE.

 Google Scholar

Wang, H., C. Ren, and Z. Yu. 2024. Multimodal sentiment analysis based on cross-instance graph neural networks. Applied Intelligence 54(4):3403–16. doi:10.1007/s10489-024-05309-0.

 Web of Science ®Google Scholar