An Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint Learning

ABSTRACT

Multimodal sentiment analysis is a technical approach that integrates various modalities to analyze sentiment tendencies or emotional states. Existing challenges encountered by this approach include redundancy in independent modal features and a lack of correlation analysis between different modalities, causing insufficient fusion and degradation of result accuracy. To address these issues, this study proposes an innovative multi-channel multimodal joint learning method for image-text sentiment analysis. First, a multi-channel feature extraction module is introduced to comprehensively capture image or text features. Second, effective interaction of multimodal features is achieved by designing modality-wise interaction modules that eliminate redundant features through cross-modal cross-attention. Last, to consider the complementary role of contextual information in sentiment analysis, an adaptive multi-task fusion method is used to merge single-modal context features with multimodal features for enhancing the reliability of sentiment predictions. Experimental results demonstrate that the proposed method achieves an accuracy of 76.98% and 75.32% on the MVSA-Single and MVSA-Multiple datasets, with F1 scores of 76.23% and 75.29%, respectively, outperforming other state-of-the-art methods. This research provides new insights and methods for advancing multimodal feature fusion, enhancing the accuracy and practicality of sentiment analysis.

Introduction

Through processing and analysis (Riyadh and Shafiq 2022; Zhang, Yu, and Zhu 2022). Sentiment analysis is a technological way to determine the sentiment tendency or emotional state conveyed in text (Khan et al. 2023), photos (Jia et al. 2023), voice (Zhao et al. 2023), and other information. Multimodal sentiment analysis (Yan et al. 2022) is an extended application of sentiment analysis that combines information from multiple modalities. Multimodal sentiment analysis (MSA) methods have potential applications, such as in personalized advertising (Xie et al. 2023), where they can provide advertisers with insights into user emotions and enable targeted advertisement placements. Sentiment cross-modal retrieval (Manek and Shenoy 2022) allows users to obtain more accurate emotional information through queries in different modalities. In terms of opinion mining (Liu et al. 2023), multimodal sentiment analysis helps in the analysis of user emotions expressed across different media, providing businesses or researchers with a comprehensive market feedback. Furthermore, multimodal sentiment analysis provides decision-makers with a comprehensive information foundation in the field of intelligent decision-making (Angamuthu and Trojovský 2023), aiding in the formulation of intelligent and personalized decision strategies.

Analysis and processing of information derived from multiple modalities exhibit both advantages and disadvantages over single-modal analysis. Numerous prompt multimodal approaches rely heavily on manually crafted components. However, such features are usually based on restricted human knowledge and lack the ability to completely characterize highly abstract sentiments, which can cause unsatisfactory outcomes. Convolutional neural networks (CNN) were applied to MSA since the advent of deep learning, and exhibited promising results. Nevertheless, the majority of the techniques based on CNN either connect features gathered from various modalities (Guo et al. 2021) or enable basic-level comprehension of the relationship across images and text (Gandhi et al. 2022). Human sentiments are primarily prompted by certain regions (She et al. 2020) in images, and are expressed by specific words in the accompanying commented text. Existing methods for image sentiment analysis still face some outstanding problems that should be solved urgently, e.g., the image regions that the feature extraction networks focus on are not accurate enough, the simple process of feature extraction for independent modalities causes information redundancy, and the semantic correlation between multimodalities is not analyzed and fused sufficiently. This research proposes a multi-channel multimodal joint learning (MMJL) image-text interaction network to overcome the aforementioned problems and improve the performance of MSA. Using cross-modal cross-attention, it extracts multichannel picture and text information and merges them. Three modules make up the suggested network: an adaptive sentiment classification module, a cross-modal fusion module, and a single-modal feature extraction module. The following are the primary contributions of our work:

1. Vision Transformer (ViT) (Liu, Wu, and Guo 2022) and Faster R-CNN (Zhang 2022) are combined to propose a new Multi-channel Comprehensive Image Features (MCIF) method, which is employed to comprehensively focus on emotion regions within images. Simultaneously, a novel Multi-channel Syntactic Text Features (MSTF) method based on syntactic features and self-attention mechanisms is utilized to extract global textual emotion features. And apply the proposed MCIF and MSTF methods to extract information from different modalities for MMJL.

2. Intra-modal interaction modules are designed to remove redundant features. A new Efficient Multimodal Transformer (EMT) fusion module is introduced, which leverages cross-modal attention mechanisms to acquire fused image-text features, and thoroughly explores inherent correlations between two input feature sequences.

3. A novel Adaptive Multitasking Fusion (AMTF) method is applied to integrate single-modal contextual emotion features with multimodal features, resulting in a more comprehensive multimodal sentiment prediction.

4. Numerous experiments are carried out on two public datasets, and the outcomes confirm the effectiveness of the proposed MSA method. Furthermore, the proposed method outperforms a number of other advanced MSA techniques.

The rest of the paper is organized as follows: Section 2 carries out a literature review of ViT, BERT, multitask learning and multimodal sentiment classification. The proposed MSA approach is described in Section 3. Section 4 provides the results of MSA experiments on the MVSA-Single and MVSA-Multiple datasets. Section 5 outlines the achievements and limitations of the proposed MSA approach, and future research plans for this topic.

Related Work

Multimodal Sentiment Classification

The integration of verbal and nonverbal data, e.g., visual and acoustic data, has made MSA a significant research topic. Traditionally, researchers concentrated on representation learning and multimodal fusion. Wang et al. (2019) constructed a change embedding network with recurrent change engagement to generate multimodal shifts for the representation learning method. Hazarika, Zimmermann, and Poria (2020) suggested modality-invariant and modality-specific representations for multimodal representation learning. Based on the stage at which fusion is carried out, existing work on multimodal fusion can be separated into two groups: early fusion and late fusion. Early fusion approaches typically employed subtle attention mechanisms for cross-modal fusion. For instance, Tsai et al. (2019) suggested a cross-modal converter to learn cross-modal attention for improving the target modality. After learning the intra-modal representation, post-fusion methods were used to achieve inter-modal fusion using a tensor fusion network. However, this method lacked the ability to interact between multimodal. In order to address this issue, Yan et al. (2021) introduced a multi-tensor fusion method based on cross modal feature extraction. However, this method fused image and text modalities only roughly during the feature fusion stage, and ignored any information exchange within the modalities. Although Yan et al. (2022) introduced a multi-tensor fusion network to model interactions between multiple bimodal pairs, they did not consider the global context of modalities. Additionally, Meena et al. (2023) proposed a hybrid model combining CNN and LSTM, incorporating a knowledge graph to enhance model performance. This approach leverages the local feature extraction capabilities of CNNs and the temporal data memory functions of LSTMs. However, the method relies on the construction of the knowledge graph, and the model may still exhibit instability when dealing with complex linguistic phenomena such as sarcasm and puns.

Our work enhances the ability to extract private and shared information across different modalities and fuse multimodal features at a later stage by using self-supervised and multi-task learning strategies to efficiently learn the information. Moreover, a framework to adaptively combine decision-level and feature-level fusion is designed that significantly improves the sentiment classification accuracy. The proposed method is extensively evaluated on benchmark datasets, demonstrating its effectiveness in extracting and fusing multimodal features.

ViT and BERT

A Transformer is a non-looping sequence-to-sequence architecture (Xu et al. 2021), which is employed to model sequential data and can offer superior results, efficiency, and depth compared to cyclic structures. Two models based on the Transformer are ViT and BERT. ViT (Paul and Chen 2022; Xie and Liao 2023) is a vision model based on the Transformer architecture that achieves impressive performance in computer vision tasks with promising applications. It is based on the principle of segmenting an image into small plots and employing a self-attention mechanism for image feature extraction and modeling. BERT (Bao, Dong, and Wei 2021) is a successful application of Transformer for language models, achieving excellent accuracy in various text processing tasks. The open-source pre-trained BERT and ViT models are utilized for obtaining features from text and image data, respectively.

Multi-Task Learning for Multimodal Sentiment Classification

Multi-task learning (MTL) (Gao et al. 2022; Tan et al. 2023) is a machine learning method that designs a model to simultaneously learn multiple related tasks. It can boost the model’s generalization capacity and efficiency to some extent. Information sharing and interaction between various tasks can be achieved by considering correlation and differences between them via multitask learning for designing an appropriate model architecture and loss function. Multi-task learning uses shared parameters in the backpropagation process, which allows features to be used by other tasks. This provides learning features that can be applied to several different tasks for improving generalization performance across them. The parameters can be shared in two ways: soft sharing and hard sharing.

Multi-task learning (Yang et al. 2022; Yu et al. 2020; Zhang et al. 2023) has been extensively used for MSA. For example, Yu et al. (2020) proposed a multi-task learning framework based on late fusion, which showed state-of-the-art performance on the collected multimodal dataset referred to as CH-SIMS. This framework was used by many researchers as a baseline for research in the multimodal sentiment classification field. Yang et al. (2022) combined pretrained models with multi-task learning strategies to propose a two-stage multi-task sentiment analysis (TPMSA) framework, which improved multimodal sentiment classification performance and achieved good results on multimodal datasets CMU-MOSI and CMU-MOEI. Zhang et al. (2023) proposed a multi-task learning model that integrated cascading and specific scoring, and simultaneously performed inter modal and intra modal interactions to enhance multimodal sentiment representation capabilities. Inspired by these MSA methods based on multi-task learning, we use a hard-sharing approach to share parameters between sub-tasks, and utilize a weight modification strategy for balancing the learning process of each task.

Proposed MSA Approach

Figure 1 depicts the primary framework of the proposed MMJL network. The MMJL jointly learns a multimodal task, two single-modal subtasks and a joint task, with the aim to develop a rich modal representation and a powerful sentiment classification capability. Its main modules include MSTF, INIT, MCIF, EMT and AMTF. Textual features are primarily extracted through the MSTF module, which uses a BERT-based pre-trained language model for text sequence embedding and Bi-LSTM for capturing hidden states, while considering word dependencies and context. The MSTF introduces a dual-channel mechanism. The first channel extracts syntactic features using a syntactic dependency tree, and the second one uses position encoding and multi-head self-attention for global features, including context and key emotional words. For image features, the MCIF module integrates ViT and Faster R-CNN, where the former extracts global features and the latter detects local target information. The final image feature representation includes concatenated features from both channels.

Figure 1. The main framework of the proposed MMJL network.

MSTF for Multi-Channel Syntactic Text Feature Extraction

Figure 2 depicts the MSTF’s overall network structure whose main objective is to comprehensively extract text features by utilizing multi-channel information, including a BERT-based pre-trained language model, syntactic dependency tree, and global self-attention mechanism. Although its structure is similar to that of the MGGCN (Xiao et al. 2022), the MSTF directly utilizes the textual content for GGCN computation that further amplifies the differences between words and sentences, and captures information at different levels. This is unlike MGGCN that shares input data between the GGCN and Bi-LSTM, which introduces additional interference and also extracts redundant features.

Figure 2. Network structure of MSTF.

For simplicity, a text sequence with n number of words is defined as $\mathit{Input}\_\mathit{t}$. A pre-trained language model based on BERT is employed to extract the text features. First, the sentence is transformed into a word sequence, and the parameters of the BERT model are utilized for word vector feature representation as follows:

(1) $e_i^c=\mathit{BERT}(\mathit{Input}\_t_i;{\theta }_t^{BERT})\in R^{\mathit{d}\times 1}$(1)

In (1), $e_i^c$ represents the embedding representation of the $\mathit{i}$-th word for $\mathit{i}\in [1,\mathit{n}]$, $\mathit{Input}\_t_i$ denotes the $\mathit{i}$-th word, and ${\theta }_t^{BERT}$ represents the BERT model parameters. Second, $e_i^c$ is fed into the Bi-LSTM to obtain the hidden state representation of the text, which includes dependencies between adjacent words and contextual information. This process can be expressed as follows:

(2) $h_i^c=[\overrightarrow{LSTM(e_i^c)};\overleftarrow{LSTM(e_i^c)}]$(2)

where $\overrightarrow{LSTM(e_i^c)}$ and $\overleftarrow{LSTM(e_i^c)}$ denote feeding of the word vector representation into the forward and backward LSTM units, respectively.

A total of two channels are used to extract syntactic and global features. The first channel involves construction of a syntactic dependency tree for the text sequence and utilizes the Gated Graph Convolutional Network (GGCN) to extract syntactic features. The output of this channel can be expressed by (3) as follows:

(3) $\mathit{ou}{t}_{\mathit{sy}{\mathit{n}}_{\mathit{\tau }+\mathit{j}}}=\mathit{GGCN}(\mathit{Input}\_t_{\mathit{\tau }+\mathit{j}},e_{\tau +j}^c;{\theta }_t^{GGCN})$(3)

where $\mathit{\tau }+\mathit{j}(\mathit{j}\in [1,\mathit{m}])$ denotes the aspect words, e.g., food, service, etc., in the text sequence.

The second channel combines positional encoding and a multi-head self-attention mechanism to obtain global features that contain contextual information and key sentiment word information. The output of this channel is shown in (4) as

(4) $\mathit{ou}{t}_{at{t}_i}=\mathit{MHSA}(e_i^c;{\theta }_t^{MHSA})$(4)

where MHSA stands for Multiple Self Attention Layer. Last, the obtained syntactic and global features are reduced separately via aspect-word-based local average pooling and global max pooling operations, as shown in (5) and (6), respectively. Subsequently, the final text features $F_t$ are concatenated using (7).

(5) $h_{avg}^s=\mathit{f}(\mathit{ou}{t}_{\mathit{sy}{\mathit{n}}_{\mathit{\tau }+1}},\mathit{ou}{t}_{\mathit{sy}{\mathit{n}}_{\mathit{\tau }+2}},\dots ,\mathit{ou}{t}_{\mathit{sy}{\mathit{n}}_{\mathit{\tau }+\mathit{m}}})$(5)

(6) $h_{max}^a=\mathit{g}(\mathit{ou}{t}_{at{t}_1},\mathit{ou}{t}_{at{t}_2},\dots ,\mathit{ou}{t}_{at{t}_n})$(6)

(7) $F_t=[h_{avg}^s;h_{max}^a]$(7)

MCIF for Multi-Channel Image Feature Extraction

We utilize ViT and Faster R-CNN to capture both local and global image features. While these components are used for feature extraction by many approaches (Zhang et al. 2023), we simplify the network structure by focusing on these two main components. This simplification reduces the complexity and optimizes the features to improve semantic understanding, as shown in Figure 3. ViT extracts global features by dividing the image into patches and aggregating patch information into a vector. The resulting global feature vector captures the overall semantic information and provides robust support for the subsequent sentiment analysis. Local features are captured by using Faster R-CNN for object detection and region extraction. This framework identifies objects and extracts the corresponding regions, effectively capturing local details and key features. The resulting region features offer a rich representation of local contents in the image.

Figure 3. Network structure of MCIF.

Global and local information is integrated by concatenating the global and local features extracted by ViT and Faster R-CNN, respectively. The resulting fused feature vector is denoted by $F_v$ and combines global and local image information, offering a comprehensive and detailed representation for sentiment classification tasks. This feature vector is used as the input for the final sentiment classification, and is computed according to (8).

(8) $F_v=[\mathit{ViT}(\mathit{Input}\_\mathit{v};{\theta }_v^{ViT});\mathit{Faster}(\mathit{Input}\_\mathit{v};{\theta }_v^{Fa\mathrm{ster}})]$(8)

Cross-Modal Attention-Based Feature Fusion

We introduce EMT for feature fusion, which enhances multimodal sentiment analysis using a symmetric cross-modal attention MPU to explore correlations between image and text sequences, as shown in Figure 4 4. This facilitates beneficial information interaction and achieves efficient multimodal fusion. The proposed design reduces redundancy in single-modal features, especially for images. However, we prevent overfitting by balancing the interaction between the global multimodal context and local single-modal features, shown as follows in (9):

(9) $\begin{array}{rl}& H_t^{[l+1]},G_{t\to g}^{[l]}=\mathit{MP}{U}_{t\leftrightarrow g}^{[l]}(H_t^{[l]},G^{[\mathit{l}]})\\ & H_v^{[l+1]},G_{v\to g}^{[l]}=\mathit{MP}{U}_{v\leftrightarrow g}^{[l]}(H_v^{[l]},G^{[\mathit{l}]})\end{array}$(9)

Figure 4. Cross-modal feature fusion module EMT, where only the first and Lth layers are shown.

where $G^{[0]}=\mathit{Concat}(H_t^{[0]},H_v^{[0]})\in {\mathbb{R}}^{2\times \mathit{d}}$, $\mathit{l}\in [0,\mathit{L}]$ denotes the l-th layer. In this way, a one-to-many global cross-modal interaction within a single MPU can be captured. The global multimodal context and local unimodal features are refined progressively by stacking multiple MPU modules. After multiple stacking layers, a pooling layer is applied to aggregate sentiment information that facilitates subsequent fusion. The specifics of this operation are illustrated in (10) as follows:

(10) $G^{[\mathit{l}+1]}=\mathrm{softmax}(V^{T}tanh(W^T{G}_g^T+\mathit{b}))G_g$(10)

where $G_g=\mathrm{Concat}(G_{t\to g}^{[l]},G_{v\to g}^{[l]})$, and $\mathit{V},\mathit{W}$, and $\mathit{b}$ denote the parameters that can be learned. The entire fusion process can be succinctly described by (11). This process fuses global multimodal context and local unimodal features, ultimately providing the final output of sentiment information.

(11) $F_f=\mathit{EMT}(F_t,F_v)$(11)

Sentiment Classification

After image-text feature fusion, the next step is the sentiment classification prediction, as illustrated in Figure 1. In the single-modal framework, INIT uses the multi-head self-attention mechanism to select sentiment-relevant features, which prevents interference from irrelevant information. A comprehensive integration of single-modal and multi-modal predictions is achieved by feature-level and decision-level fusion using adaptive attention mechanisms, which results in the classification performance optimization. The final sentiment classification combines predictions from textual, image, and multimodal modalities, as shown in (12).

(12) $\left\{\begin{array}{c}{p}_t=\mathit{softmax}(\mathit{TextClassifier}(F_t))\\ p_v=\mathit{softmax}(\mathit{ImageClassifier}(F_v))\\ p_f=\mathit{softmax}(\mathit{FusionClassifier}(F_f))\end{array}\right.$(12)

In addition, learnable parameters are designed as shown in (13)-(15) for tuning the final classification results.

(13) $\left\{\begin{array}{c}{w}_t=\mathit{Fc}(F_t)\\ w_v=\mathit{Fc}(F_v)\\ w_f=\mathit{Fc}(F_f)\end{array}\right.$(13)

(14) $w_t,w_v,w_f=\mathit{softmax}([w_t,w_v,w_f])$(14)

(15) $\mathit{p}=p_t\cdot w_t+p_v\cdot w_v+p_f\cdot w_f$(15)

The final loss function is defined as follows:

(16) $\mathit{Loss}=\mathit{Los}{s}_t+\mathit{Los}{s}_v+\mathit{Los}{s}_f+\mathit{\lambda gLos}{s}_{\mathit{all}}$(16)

where $\mathit{Los}{s}_t$, $\mathit{Los}{s}_v$ and $\mathit{Los}{s}_f$ represent the text, image and multimodal classification losses, respectively, and $\mathit{Los}{s}_{\mathit{all}}$ is the predicted value loss obtained using (15). Furthermore, $\mathit{\lambda }$ is a hyperparameter that modifies the impact of multimodal classification loss on the overall model.

Experiments

Datasets

Two publicly accessible MSA databases, MVSA-Single and MVSA-Multiple, are used to evaluate the model under discussion (Niu et al. 2016). 5129 image-text pairings tagged with good, neutral, and negative feelings are included in the former, which is sourced from Twitter and represents actual social media emotions. The latter database contains 19,600 image-text pairs, annotated by three independent annotators for diversity. These datasets pose challenges to sentiment analysis models in real scenarios due to a significant amount of emotional information. Both datasets represent user emotions and offer quality data for advancing multimodal sentiment analysis research and practical applications.

The two datasets are preprocessed following the steps described in (Xu and Mao 2017) to remove the image-text pairs whose images are completely different from the text labels. Furthermore, when one modality is labeled as positive (or negative) while the equivalent label of the other modality is neutral, the polarization of the image-text combination sentiment is identified as positive (or negative). Table 1 displays the statistical information of the preprocessed MSVA-Single and MSVA-Multiple datasets.

Table 1. Statistical table of dataset division.

Dataset	Positive	Neutral	Negative	Total
MSVA-Single	2683	470	1358	4511
MSVA-Multiple	11318	4408	1298	17024

Implementation Details

All experiments were conducted using the PyTorch framework on an NVIDIA GeForce RTX 4090 GPU with CUDA version 11.8 and torch version 2.0. For data preprocessing, we manually padded and clipped text content to a maximum length of 50 words and resized the images to 224 × 224 pixels. Following data transformation, the dataset was randomly split into training, validation, and test sets in an 8:1:1 ratio, ensuring the original label distribution was maintained. Using a stratified sampling strategy, we further constructed a meta set from the training set, with each dataset containing 256 instances. The Adam optimizer was used, with an initial learning rate of 0.001, which was reduced by a factor of 10 every 10 epochs, and a weight decay set to 1e-5. Based on previous research, the evaluation metrics for this study are accuracy and F1 scores, as described below:

(17) $\mathit{Accuracy}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{FN}+\mathit{TN}+\mathit{FP}}$(17)

(18) $\mathit{Precision}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(18)

(19) $\mathit{Recall}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$(19)

(20) $F_1=\frac{2\cdot \mathit{Precision}\cdot \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}$(20)

where TP, TN, FP and FN refer to True Positive, True Negative, False Positive and False Negative, respectively.

Baseline Techniques

The proposed image-text sentiment analysis method is compared to a number of conventional and cutting-edge techniques, which are described next. MultiSentiNet (Xu and Mao 2017) extracts objects and scenes from images in the form of visual semantic data. After identifying important words through attention using a visual feature-guided LSTM technique, it integrates all of these characteristics to produce the final label. CNN-Multi (Cai and Xia 2015) learns visual and textual properties independently using two different CNNs, and then applies the input to a third CNN for sentiment prediction. DNN-LR (Yu et al. 2016) uses logistic regression to classify sentiment and makes use of different pre-trained CNNs for text and pictures. The primary contribution and originality of the VAuLT (Chochlakis et al. 2022) approach are the utilization of the output representation generated by a prominent language model, such as BERT, to inform the linguistic input of ViLT. This enables the exploitation of the contextual representation capabilities offered by a large language model.

MVAN (Yang et al. 2020) is an MSA method that uses memory network modules and multi-view attention networks to repeatedly collect semantic image text characteristics. Zhu et al. (2023) describe ITIN as having a cross-modal alignment module that logs word correspondences between different regions. This approach achieves the present state-of-the-art accuracy by utilizing an adaptive cross-modal gating tool to integrate multimodal data. MultiPoint (Yang et al. 2023) designs unified multimodal prompts and introduces probabilistic fusion to reduce discrepancies between modalities and enhance the model robustness, respectively. MLFC (Wang et al. 2023) addresses the issue of redundant modal features using a CNN and a Transformer. It leverages supervised contrastive learning to improve the model’s ability for learning standard sentiment features.

Comparison with the State-of-the-Art

Table 2 provides a comparison of MMJL with several other classical and state-of-the-art methods in terms of accuracy and F1 scores with the MSVA datasets. The following conclusions can be derived:

Table 2. Performance comparison of MVSA dataset.

Model	MVSA-Single		MVSA-Multiple		Complexity
	Accuracy (%)	F1 (%)	Accuracy (%)	F1 (%)	FLOPs (M)
MultiSentiNet (Xu and Mao 2017)	69.84	69.63	68.86	68.11	25.3
CNN-Multi (Cai and Xia 2015)	61.20	58.37	66.39	64.19	29.7
DNN-LR (Yu et al. 2016)	61.42	61.03	67.86	66.33	31.1
VAuLT (Chochlakis et al. 2022)	72.80	71.80	70.30	67.00	34.7
MVAN (Yang et al. 2020)	72.98	72.98	72.36	72.30	32.9
ITIN (Zhu et al. 2023)	75.19	74.97	73.52	73.49	35.1
MLFC (Wang et al. 2023)	75.33	73.46	70.53	67.97	38.6
MultiPoint (Yang et al. 2023)	70.13	68.26	68.55	66.33	27.7
LXMERT-MMSA (Yin et al. 2024)	-	-	69.40	67.90	37.2
MDSE (Li et al. 2024)	76.22	75.71	72.31	70.12	35.1
CIGNN (Wang, Ren, and Yu 2024)	75.11	73.33	73.41	70.96	29.6
MMJL(Ours)	76.98	76.23	75.32	75.29	33.6

The MMJL model exhibits optimal performance on both datasets. It provides detailed descriptions of multi-channel text through image feature extraction modules like MCIF, MSTF, and the cross-modal fusion module EMT. This is unlike VAuLT, which mainly focuses on textual input in images and does not fully explore the interaction information. The MMJL model thoroughly explores potential information in image regions and textual words. This exploration achieves cross-modal interaction at both global and local levels that enhances the overall sentiment classification performance.

While ITIN also uses a cross-modal alignment module, it emphasizes local relationships, and faces problem in finding effective connections between images and text. The MMJL model, with its fine-grained interaction, densifies relationships between features that improves its performance. Comparing computational complexity using FLOPs, the MMJL’s multi-channel modules extract information more efficiently, reducing the number of redundant calculations and network complexity. On the MVSA-Single dataset, MMJL achieves 1.79% and 1.26% increase in accuracy and F1 score, respectively, compared to TIIN. Similarly, on the MVSA-Multiple dataset, the proposed model demonstrates performance improvements of 1.80% in both accuracy and F1 score. Compared to the recent MDSE, although its performance is very close to our MMJL, MDSE introduces complex feature learning modules and contrastive loss, increasing the model’s complexity and the number of parameters. Additionally, MDSE overly emphasizes the handling of private features, neglecting the interaction between features across modalities. In contrast, MMJL achieves a balance between intra-modal feature extraction and inter-modal feature interaction, enhancing the model’s understanding of multimodal information and improving its generalization performance.

Ablation Experiments

Ablation experiments are carried out on the MVSA datasets to validate each module’s efficacy in the proposed model. The MHSA and GGCN in text features, Faster R-CNN in image modal features, feature fusion module, intra-modal interaction module, and AMTF module are removed from the MMJL model, respectively. The resulting models without different modules are denoted as “MMJL w/o MCIF only GGCN,” “MMJL w/o MCIF only MHSA,” “MMJL w/o MSTF only ViT,” and “MMJL w/o EMT,” “MMJL w/o INIT” and “MMJL w/o AMTF” in Table 3, respectively. We explain this terminology through two examples: 1) “MMJL w/o AMTF” indicates that only simple fusion classifiers are used for prediction while additional loss functions have been removed. 2) “MMJL w/o EMT” denotes simple splicing and dimensionality reduction of multimodal features. Table 3 provides the outcomes of these studies, which reveal the following key findings: 1) First, the multi-channel feature extraction module improves the model’s comprehension of diverse modalities, which enhances the sentiment classification performance. 2) Second, EMT effectively explores semantic correlations between modalities. This improves the model’s understanding of complex relationships and results in enhanced multimodal sentiment analysis performance: On the MVSA-Multiple dataset, accuracy and F1 scores increase by 3.64% and 3.75%, respectively. 3) Additionally, INIT uses a multi-head self-attention mechanism to select sentiment-relevant features, increasing both accuracy and F1 scores by 0.93% on the MVSA-Multiple dataset. 4)Finally, AMTF improves the model’s ability to integrate information from various modalities, increasing the sentiment classification accuracy: On the MVSA-Multiple dataset, both accuracy and F1 scores increase by 0.93%.

Table 3. Results of ablation studies on the MVSA dataset.

Method	MVSA-Single		MVSA-Multiple
	Accuracy (%)	F1 (%)	Accuracy (%)	F1 (%)
MMJL w/o MCIF only GGCN	74.32	73.95	73.17	72.97
MMJL w/o MCIF only MHSA	73.98	73.55	73.10	72.96
MMJL w/o MSTF only ViT	73.96	73.46	73.06	72.86
MMJL w/o EMT	72.36	72.10	71.68	71.54
MMJL w/o INIT	75.64	75.12	74.39	74.20
MMJL w/o AMTF	74.65	73.23	74.39	74.20
MMJL(Ours)	76.98	76.23	75.32	75.29

In summary, each module provides distinct information processing features. The integration of these features allows the model to comprehensively capture complex relationships between images and text that improves the multimodal sentiment analysis.

Hyperparameter Study

Adjustment Parameters of Loss Function

In the experiments, $\mathit{\lambda }$ in (16) is considered as a key parameter that affects the prediction performance. Therefore, the values of $\mathit{\lambda }$ are varied to analyze the performance of the proposed method on two MVSA datasets. Figure 5 displays the outcomes of this experiment. When $\mathit{\lambda }=0.5$, the proposed MMJL achieves the best accuracy and F1 scores on the MVSA-Single and MVSA-Multiple datasets. When $\mathit{\lambda }$ is too large, the model excessively emphasizes the fusion of multimodal information and neglects the unimodal features, causing a performance decline. Conversely, if $\mathit{\lambda }$ is too small, the model focuses more on a single modality and ignores the correlation between multimodalities, which also degrades the performance. Therefore, both unimodal and multimodal aspects should be considered for comprehensive sentiment classification. Based on the above analysis, $\mathit{\lambda }=0.5$ is used in the proposed MMJL network.

Figure 5. Hyperparameter analysis of different $\mathit{\lambda }$ in Equation (16)(16) $\mathit{Loss}=\mathit{Los}{s}_t+\mathit{Los}{s}_v+\mathit{Los}{s}_f+\mathit{\lambda gLos}{s}_{\mathit{all}}$(16) for proposed MMJL.

Initial Learning Rate

The setting of the initial learning rate significantly affects the sentiment classification performance (An, Zainon, and Hao 2023). Therefore, initial learning rates are varied for validation experiments, and the corresponding sentiment classification results are shown in Figure 6. The figure shows that the model is prone to underfitting for a high initial learning rate. On the other hand, when the initial learning rate is too small, the model is prone to overfitting, which will also increase training costs and reduce the convergence efficiency. Therefore, based on the comprehensive performance analysis of the two datasets, the initial learning rate of the proposed MMJL model is set as 0.001.

Figure 6. The impact of initial learning rate on sentiment classification results.

Cases Analysis

The performance of MMJL is evaluated on six random samples taken from the MVSA-Multiple dataset and used as input to MMJL w/o EMT, MMJL w/o INIT, MMJL w/o AMTF, and MMJL. Table 4 summarizes the classification results. Out of the six samples, MMJL w/o INIT and MMJL w/o AMTF correctly predict four samples, while MMJL w/o EMT predict only three samples, emphasizing the crucial role of the cross-modal attention fusion module. The MMJL model achieves five correct predictions, showing a 83.33% accuracy rate and indicating the effectiveness of EMT, INIT and AMTF modules in improving the sentiment features. Particularly, their fusion effectively enhances multimodal sentiment classification. However, all models make incorrect predictions for the example (f), possibly due to ambiguous meaning in the text and images that suggests the need to further improve the MMJL.

Table 4. Case analysis results of the proposed MMJL model on the MVSA-Multiple dataset.

Sample	Text	Image	True value	MMJL w/o EMT	MMJL w/o INIT	MMJL w/o AMTF	MMJL(Ours)
(a)	Happy new year~		Positive	P (√)	NT (×)	P (√)	P (√)
(b)	This rain won’t stop team		Positive	P (√)	P (√)	P (√)	P (√)
(c)	So my family just tried kicking me off the tv no bitch		Negative	P (×)	NG (√)	NG (√)	NG (√)
(d)	We just can’t get enough of these summer sunsets		Negative	NG (√)	NG (√)	NT (×)	NG (√)
(e)	Luke when he see you aren’t voting		Neutral	NG (×)	NT (√)	NT (√)	NT (√)
(f)	Most Alberta trout streams threatened: survey		Negative	NT (×)	NT (×)	NT (×)	NT (×)

*P represents Positive, NT represents Neutral, and NG represents Negative.

Discussion

Different comparison experiments show that the MMJL model demonstrates superior performance in multimodal sentiment analysis compared with other methods. Key findings of the experiments show the effectiveness of different modules included in the model: MSTF and MCIF extract text and image features, EMT introduces global context, INIT facilitates adaptive interactions, and AMTF integrates multimodal information. These modules function collaboratively to enhance the model’s understanding of sentiments in text and images, improving its overall performance. The MMJL model excels in handling global and local image-text correlations, achieving outstanding results in multimodal sentiment analysis.

The MMJL model encounters problems in generating a complete understanding of the emotional content when subtle emotional features in images are not adequately captured by the text. The fine-grained interaction in the MMJL model is limited in such scenarios, as the cross-modal interaction mechanism cannot emphasize the correlation between image regions and text. In the absence of any textual information, the interaction mechanism cannot effectively guide the model, which leads to a heavier reliance on the single-modality information and reduced robustness in multimodal sentiment analysis tasks. To address these issues, it is recommended that future research should focus on the following four aspects to improve the MMJL model’s performance in real-world multimodal sentiment analysis challenges:1) A detailed study of mutual influence between images and text at both global and local levels should be carried out. 2) Existing research should be extended to multimodal continuous sentiment intensity. 3) Consistency between different modalities should be ensured. 4) Methods for enhancing model robustness in challenging scenarios should be explored.

Conclusions

This study introduced the Image-Text Interaction Network (MMJL) with innovative text and image feature extraction modules (MCIF, MSTF) to improve multimodal sentiment analysis. These modules helped in overcoming traditional limitations by capturing latent information from image regions and text words. The Cross-Modal Fusion module (EMT) introduced a global multimodal context for effective cross-modal interactions, while the Adaptive Multitasking Fusion module (AMTF) achieved adaptive fusion of multimodal and single-modal results. The MMJL model consistently outperformed existing methods on public databases, demonstrating significant industrial application value by extracting crucial features from concurrent image and text data in complex environments. A limitation of the proposed model was acknowledged. Future work aims to enhance model robustness by studying mutual influence between images and text to achieve a more comprehensive cross-modal interaction, and exploring multimodal continuous sentiment intensity beyond classification tasks.

Disclosure S tatement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This article is the research result of the 2023 Zhejiang Province Vocational Education “the 14th Five-Year Plan“ Teaching Reform Project (Exploration and Practice of Collaborative Education Model Based on ”Anheng Digital Security Industry College”).