An enhanced text classification model by the inverted attention orthogonal projection module

ABSTRACT

The orthogonal projection method has made significant progress in text classification, especially in generating discriminative features. This method obtains more pure and suitable for classification features by projecting text features onto the orthogonal direction of common features (which are not helpful for classification and actually confuse performance). However, this approach requires an additional branch network to generate these common features, which reduces the flexibility of this method compared to representation optimisation methods such as self-attention mechanisms, as it requires significant modification of the base network structure to use. To address this issue, this paper proposes the Inversed Attention Orthogonal Projection Module (IAOPM). IAOPM uses inversed attention (IA) to iteratively reverse the attention map on text features, encouraging the network to remove discriminating features from the text features and obtain potential common features. Unlike the original orthogonal projection method, IAOPM can extract common features within a single network without any branch networks, increasing the flexibility of the orthogonal projection method. We also use an orthogonal loss to ensure the quality of the common features during training, so IAOPM also has better purity performance than the original method. Experiments show that text classification models based on IAOPM outperform the baseline models, self-attention mechanisms, and the original orthogonal projection method on multiple text classification datasets with an average accuracy increase of 1.02%, 0.44%, and 0.52%, respectively.

KEYWORDS:

• Text classification
• orthogonal projection
• inverted attention

1. Introduction

Text classification is an important task in natural language processing, such as the sentiment classification, question classification, etc. For text classification, deep learning models have been shown to outperform the traditional classification methods. Many neural networks and embedding techniques have been devised and applied, such as the Convolution Neural Network (CNN), Recurrent Neural Network (RNN), Long short-term memory (LSTM), Gated Recurrent Unit (GRU), etc. However, these neural network methods cannot take full advantage of discriminative features, and also be disturbed by other common features (or invariant features (Ganin & Lempitsky, 2015; Zhang et al., 2019), which do not help, rather confuse the classification) without class tendencies. Even the attention mechanism is able to give higher weights to the discriminative features and lower weights to the common features to alleviate this problem to some extent. However, due to the idiosyncrasy of the data and the inaccuracy of the attention mechanism (Qin et al., 2020), these low-weight common features still interfere with the representation learning and ultimately affect the text classification accuracy.

To address this issue, Qin et al. (2020) proposed the use of an orthogonal projection to purify text features. By removing common features from the text, the system focuses more on discriminative features in the text, thereby improving the accuracy of text classification. Specifically, this method projects the text features onto the orthogonal direction of common features, and the resulting features are more discriminative for classification due to being perpendicular to the common features. As shown in Figure 1, assuming the text content is “This movie is very good”, common features such as “This”, “movie”, “is”, and “very” can be eliminated through orthogonal projection, leaving the discriminating feature “Good” remaining. This allows the network to be unaffected by the influence of common features. However, the purity of this method depends on the quality of the common features, thus requiring an additional branch network to separately extract these features to ensure their quality. Clearly, the additional network increases the cost of using this method and is less convenient compared to other representation optimisation methods such as attention mechanisms, which act as plug-in modules. Its advantages are also not obvious. If high-quality common features can be extracted within a single network, the flexibility of this method can be increased, making it more suitable for wider adoption and development. Therefore, this method still has room for improvement.

Figure 1. Orthogonal projection purifies text features.

In this study, we designed a simple module, called the Inversed Attention Orthogonal Projection Module (IAOPM), for extracting high-quality common features within a single network, thereby increasing the flexibility of the orthogonal projection method. Specifically, common features without classification bias are typically assigned lower weights in the attention distribution map of text features. IAOPM iteratively reverses the attention distribution map of text features through inversed attention (IA) to remove discriminative features, inducing the system to generate complementary latent common features, and then refines the text features through orthogonal projection. Finally, we propose a self-supervised loss function called Orthogonal Projection Loss (OPL). It ensures the shared nature of extracted common features within one or several batches, thereby ensuring high-quality common features extracted by IA. Additionally, OPL stabilises the training process and aids in generalisation. Compared to the original orthogonal projection method (Qin et al., 2020), IAOPM is relatively more flexible because all operations are performed within a single network, and therefore it does not add any branch networks. IAOPM can simply be added after a feature extractor (such as RNN, LSTM, CNN, etc.) to purify text features. In addition, through experiments and visualisation, we have demonstrated that IAOPM extracts higher quality common features, resulting in better purity performance compared to the original method.

We summarise the main contributions of this work as follows:

1. We propose using IA to generate common features, which runs only on the feature extraction network (also called the backbone network) and does not require any additional branch networks.

2. We propose the use of Orthogonal Projection Loss (OPL) to ensure the quality of the common features extracted by IA. OPL is used only as a form of regularisation without adding any additional network parameter.

3. Experimental results show that on benchmark text classification datasets (CR, MR, SST2, and Subj), IAOPM improves accuracy by 0.44% and 0.52% compared to self-attention mechanisms (Vaswani et al., 2017) and FP-Net (Qin et al., 2020).

2. Related work

Text classification is broadly divided into two main categories, supervised and unsupervised methods. Our work focuses on improving the orthogonal projection method under supervised classification. Therefore, we primarily discuss supervised methods while briefly introducing common features.

2.1. Supervised methods

Supervised methods obtain training directions from labelled data that improve the utilisation efficiency of the data and extract discriminative features. The most commonly used of these methods are deep neural networks, e.g. CNN use windows of different sizes to capture the key local features that are important for classification (Jing, 2019; Kim, 2014; Zhou et al., 2022). Unlike CNNs, the sequence models process text in a sequential manner by sharing recurrent units to learn a feature representation of texts and are then used for text classification, such as RNN, LSTM, GRU, etc. A large number of text classification studies based on RNN and its variants for feature extraction and downstream tasks (Huan et al., 2022; Jing, 2019; Sivakumar & Rajalakshmi, 2021). These neural network methods generally rely on the final hidden state of a network (such as the final hidden state of RNN or CNN) to create the final instance-level representation. However, they are unable to sift through the plethora of information for discriminative features that are helpful for classification. Attention mechanism (Mnih et al., 2014) is used for text classification to resolve this problem. By quickly filtering out the high-valued information from a large amount of information, the attention mechanism can guide the system to focus on these discriminative features. For example, Hu and Zhao (2021) proposed a Bi-direction Gated Recurrent Unit (Bi-GRU) model based on pooling and attention mechanism, which utilises self-attention mechanisms to obtain information about the influence of words and sentences for text classification. Ruan et al. (2022) proposed an ATT-CN-BILSTM Chinese news classification model based on the attention mechanism. The model uses the attention mechanism to improve the feature extraction process of CNN and BiLSTM. In addition, the Transformer (Vaswani et al., 2017) is a method that relies solely on self-attention mechanisms to compute text feature representations. This model has achieved excellent results in multiple natural language processing domains, including text classification. Bert (Devlin et al., 2018), Transformer-XL (Dai et al., 2019) and XLNet (Yang et al., 2019) combined a transformer with large corpora to further improve the accuracy of text classification (Lv et al., 2022; Shaheen et al., 2020; Wang & Zhang, 2022). However, these attention-based text classification methods cannot completely eliminate the influence of common features on classification as these features are given low weights, instead of being removed.

To eliminate the influence of common features, Qin et al. (2020) first applied orthogonal projection to filter common features in text, which improved the text classification based on CNN, RNN, Transformer, and Bert and achieved good classification results. Inspired by this, more and more scholars have started to apply orthogonal projection to text classification, such as Liu et al. (2022), who applied orthogonal projection to sentiment analysis tasks and proposed an Aspect Feature Distillation and Enhancement Network (AFDEN). Wei et al. (2022) proposed GP-GCN to accomplish aspect-level sentiment analysis. Through orthogonal projection, GCN not only weakens the dependency of the graph node in updating process but also reduces the dependency between node and corpus. However, these orthogonal projection methods all use extra branch networks, thus still have room for improvement in terms of flexibility.

2.2. Common features

The concept of common features comes from domain adaptation, which refers to some shared features between the source and target domains. Among them, Blitzer et al. (2006) proposed a method, called as the Structural Correspondence Learning (SCL), which mainly exploits multiple shared features to predict tasks.

Afterwards, Pan et al. (2010) proposed Spectral Feature Alignment (SFA) algorithms to solve the feature mismatch problem by aligning domain-specific words with the help of domain-independent common features. But all the above methods require manual selection of common features. In recent years, many researchers have studied methods to automatically identify common features based on neural networks so as to improve the efficiency of common feature acquisition. Yu and Jiang (2016) proposed two auxiliary tasks to learn sentence embeddings for CNN. An Adversarial Memory Network (AMN) was proposed by Li et al. (2017), which automatically identifies common features by applying attention mechanisms and adversarial training. Zhang et al. (2019) proposed an Interactive Attention Transfer Network (ATN) for cross-domain sentiment classification, which extracts common features through an independent branch network.

Inspired by these studies, Qin et al. (2020) used common features and orthogonal projection to improve the accuracy of text classification, and proposed the Feature Purification Network (FP-Net). Subsequently, researchers began using common features to purify text features in order to improve text classification performance (Liu et al., 2022; Wei et al., 2022).

Finally, our work is related to several other works. In object detection, Huang et al. (2020) used an efficient and fine-grain mechanism, called as the Inverted Attention (IA), to improve object detectors. By iteratively inverting the attention weights of features, the detector network with IA can discover new discriminative clues and put more attention on complementary objects, feature channels and even context. This ability to expand the system's vision allows IA to discover the hidden common features in texts. No branch network is added here as IA works only on text features.

Ranasinghe et al. (2021) developed a novel loss function, termed as the Orthogonal Projection Loss (OPL). It can impose orthogonality in the feature space and directly enforce the inter-class separation alongside the intra-class clustering in the feature space through orthogonality constraints on the mini-batch level. Inspired by Ranasinghe et al. (2021), we design an OPL in IAOPM to resolve the problem that the common features generated by IA are too similar to text features. Unlike Ranasinghe et al. (2021), we do not pay attention to know whether the features of different classes are orthogonal. Thus, our work is significantly different.

3. Inverted attention orthogonal projection module

The IAOPM (Figure 2(right)) proposed in this work can extract common features from a single network, thus increasing the flexibility of the original orthogonal projection method (FP-Net, Figure 2(left)) to some extent. The IAOPM consists of three parts: common feature extraction based on IA, orthogonal projection, and orthogonal projection loss. Firstly, the attention map of text features is iteratively reversed using the IA module, which removes the discriminative features and thereby induces the system to generate complementary latent common features. Then, the text features are projected onto the orthogonal direction of these common features using the orthogonal projection module, resulting in purer discriminative features. Finally, the orthogonal projection loss is used to ensure the quality of the common features, improving the feature purification performance. Next, we will introduce each component of the proposed IAOPM in detail.

Figure 2. Structure of FP-Net and structure of IAOPM.

3.1. Common feature extraction based on IA

The goal of IAOPM is to extract common features from a single network. Its core technology is the use of IA to extract common features, as shown in Figure 3. This method consists of only two simple steps: (1) Attention Generation and (2) Inverted Attention to Generate Common Features. Firstly, IA generates the corresponding attention distribution map based on the attention scores of the text features. Then, unlike traditional attention mechanisms, IA removes the discriminative features with high weights by iteratively reversing the attention map, thereby guiding the system to obtain complementary low-weight common features. Through the IA method, we can extract common features within a single network.

Figure 3. IA generated common features, including Attention Generation and Inverted Attention to Generate Common Features.

3.1.1. Attention generation

Given a dataset $D=\left\{\right(x_i,y_i){\}}_{i=1}^N$, where $x_i$ is a text of length $L$ and $y_i$ is the corresponding label for sample $x_i$. The dataset $D$ is transformed into text features $f_t$ through the feature extractor $F$. In order to obtain the common features corresponding to the text features $f_t$, it is necessary to first calculate the attention map $M$ corresponding to the text features $f_t$. For simplicity, we use a parameter matrix $W_Q$ to generate the attention map, as shown in Equation (1). (1) $M=softmax\left(W_Q{f}_t\right)$(1)

3.1.2. Inverted attention to generate common features

Traditional attention mechanisms assign higher weights to discriminative features in text features and lower weights to features that are not conducive to classification. These low-weight features often belong to common features because they do not point to any specific category. Therefore, we use the inverted attention (IA) method to iteratively reverse the original attention map $M$ into an inverted attention map, forcing the network to focus on low-weight common features. Specifically, we identify discriminative features and common features through the inverted attention map $A=\left\{a_i\right\}$, and remove the high-weight discriminative features, retaining the low-weight common features. The specific calculation method is: (2) $\begin{array}{rl}{a}_i& =\{\begin{array}{ll}0& if{m}_i>T\\ m_i& else\end{array}\end{array}$(2) (3) $\begin{array}{rl}{f}_g& =(A\cdot f_t)W_V\end{array}$(3) where $a_i$ and $m_i$ are the inverted attention score and attention score of the $i-th$ position, respectively. $T$ is a hyperparameter representing the Inverted attention threshold. Finally, the text features $f_t$ are multiplied by the inverted attention map $A$ and the weight $W_V$ to obtain the final common features $f_g$.

As can be seen from Equation (2), in the attention distribution graph, the part with weight greater than $T$ will be removed, and the part with weight less than $T$ will be retained as shown in Figure 4 (assuming $T=7$). Then, the generated inverted attention score matrix $A=\left\{a_i\right\}$ is multiplied by the text feature $f_t$ and a mapping is performed to generate common features. These operations contain only 2 parameter matrices, the rest are math operations. Therefore, the method does not rely on any external branch network and it has a simple structure.

Figure 4. IA obtained low-weight information by resetting weight of high-weight features to 0.

3.2. Orthogonal projection

The Orthogonal Projection method removes irrelevant features from the text features by projecting the text features onto the orthogonal direction space of common features (which should contain purely discriminative and useful for classification features). The specific calculation method is: (4) $\begin{array}{rl}{f}_{t,proj}& =\frac{f_t\cdot f_g}{|f_g|}\frac{f_g}{|f_g|}\end{array}$(4) (5) $\begin{array}{rl}{f}_{t,orth}& =f_t-f_{t,proj}\end{array}$(5) where $f_t\cdot f_g$ is dot product operation and $|f_g{|}^2$ is the $L_2$norm of $f_g$.

To be more intuitive, the process is represented by a 2D image as shown in Figure 5. Specifically, the text feature vector $f_t$ consists of $f_{t,proj}$ and $f_{t,orth}$. Since $f_{t,proj}$ is a component of common feature $f_g$, $f_{t,proj}$ still consists of common features. Then, we can be found that $f_g{f}_{t,orth}^T=f_{t,proj}{f}_{t,orth}^T=0$. So, $f_{t,orth}$ is independent of $f_{t,proj}$ and $f_g$. Therefore, $f_{t,orth}$ contains only discriminative features, which can be called as the pure features. Finally, use $f_{t,orth}$ for classification as expressed by Equation (6). (6) $Y_{IAOM}=softmax(f_{t,orth}\cdot W_{IAOM}+b_{IAOM})$(6) where $W_{IAOM}$ and $b_{IAOM}$ are, respectively, the weights and biases of the classifier, and $Y_{IAOM}$ is the predicted result.

Figure 5. Orthogronal projection in 2D space.

3.3. Orthogonal projection loss

In general, common features shared by all categories should be concentrated in the feature space, as shown in Figure 6(a), where blue and yellow represent positive and negative classes, respectively, and the corresponding common features are represented in green and red. However, the Inverted Attention (IA) method uses text features $f_t$ to generate common features $f_g$, making the text features $f_t$ and common features $f_g$ more similar. This results in the common features $f_g$ losing shared features to some extent and not being able to represent common features of the sample, as shown in Figure 6(b). Therefore, these low-quality common features may not be pure enough and still contain a lot of discriminative information, which can affect the effectiveness of subsequent orthogonal projection.

Figure 6. (a): The distribution of common features in 2D space in normal circumstances (b): The distribution of common features generated by IA in 2D space

To address the issue of the common features $f_g$ losing shared characteristics due to the IA method, we use the Orthogonal Projection Loss (OPL). OPL references the Orthogonal Loss proposed by Ranasinghe et al. (2021). and is used to separate $f_g$ and $f_t$ and guide the network to aggregate $f_g$. In other words, OPL is a constraint on the common features $f_g$ to ensure that $f_g$ does not lose shared characteristics. The calculation formula for OPL is as follows: (7) $\begin{array}{rl}s& =\frac{{\sum }_{i\in B}{\sum }_{(j\in B/i\ne j)}<f_g^i,f_g^j>}{{\sum }_{i\in B}{\sum }_{(j\in B/i\ne j)}1}\end{array}$(7) (8) $\begin{array}{rl}d& =\frac{{\sum }_{i\in B}{\sum }_{(j\in B/i\ne j)}<f_g^i,f_t^j>}{{\sum }_{i\in B}{\sum }_{(j\in B/i\ne j)}1}\end{array}$(8) where $<\cdot >$ is the cosine similarity operator applied on two vectors, $|\cdot |$ is the absolute value operator, and $B$denotes the batch size. Note that the cosine similarity operator (Ranasinghe et al., 2021) used in Equations (7) and (8) is as follows: (9) $<x,y>=\frac{x\cdot y}{||x|{|}_2\cdot ||y|{|}_2}$(9) where $||\cdot |{|}_2$is the $L_2$norm. In Equation (7), $s\in (-1,1)$ represents the average similarity of all $f_g$ in batch $B$ after calculating cosine similarity with each other (except itself). In Equation (8), $d\in (-1,1)$ represents the average similarity after calculating the cosine similarity of all $f_g$ and $f_t$ in batch $B$ (except for $f_g$and $f_t$ with the same serial number).

To ensure the shared characteristics of the common features, the average similarity between common features should be maximised, in which case $s=1$; similarly, to separate common features from text features, the average similarity between common features and text features should be minimised, in which case $d=0$. The above calculation method is shown in Equation (10). (10) $L_{OPL}=(1-s)+|d|$(10) The final loss function is a weighted combination of CE and OPL: (11) $L=L_{CE}(Y_{truth},Y_{IAOM})+\lambda L_{OPL}(f_g,f_t)$(11) where $L_{CE}$ is the cross-entropy loss and $\lambda$ is the OPL weight hyperparameter.

Figure 7(a) shows the 2D distribution of text features and common features for CNN (only common features are generated, but text features are not purified), IAO (only IA and orthogonal projection operations are included), and IAOPM. As can be seen from IAOPM, OPL forces the network to separate $f_g$(red)and $f_t$(blue). Figure 7(b) shows the 2D spatial distributions are generated by IAO, IAOPM and FP-Net, with red for positive classes and blue for negative classes. Compared with IAO and FP-Net, the common features of IAOPM are significantly more aggregated. It is shown that the optimisation of common features by OPL is effective.

Figure 7. (a): Visualisation of the common features and text features of CNN, IAO, and IAOPM in 2D space. (b): Visualisation of the common features (different classes) of IAO, IAOPM, and FP-Net in 2D space.

The complete algorithm of the proposed IAOPM is given in Algorithm 1, which is self-explanatory.

Table

Algorithm 1 Inverted Attention Orthogonal Projection
Input:
Dataset $D=\left\{\right(x_i,y_i){\}}_{i=1}^N$,$x_i$’s embedding matrix $X_i\in {\mathbb{R}}^{M\times K}$
Output: text classification model.
1:	for each iteration do
2:	Feature map $f_t$ from feature extractor$F$.
3:	Attention generation part:
4:	Generate attention map $M$ (Equation (1))
5:	Attention inversion part:
6:	Inversed attention map$M$, generate inverted attention map $A=\left\{a_i\right\}$ (Equation (2))
7:	Generate common feature part:
8:	Generate common features. (Equation (3))
9:	Orthogonal projection part:
10:	Get the pure features. (Equations (4) and (5))
11:	Perform classification.
12:	Orthogonal projection loss part:
13:	Compute CE loss and OPL (orthogonal projection loss). (Equation (11)), where OPL is computed using Equations (7)–(10)
14:	Back-propagation.
15:	end for

4. Experiments

4.1. Experiment details

4.1.1. Dataset

We carried out experiments with the following four diverse benchmark datasets:

• MR (Pang & Lee, 2005): This is a sentiment classification dataset, which is a collection of people's evaluations of movies, where each sample is divided into positive and negative.

• SST2 (Liu et al., 2016): This is the Stanford Sentiment Treebank dataset, where each sample is divided into positive and negative.

• CR (Hu & Liu, 2004): This is a sentiment classification dataset, which is a collection of customers’ reviews, where each sample is divided into positive and negative.

• Subj (Pang & Lee, 2004): This is a question classification dataset, where each sample is divided into objective and subjective aspects.

4.1.2. Baselines

We use feature extractors common in NLP as the baseline models:

• RNN (Jordan, 1997): Recurrent Neural Network (RNN), where the current output of a sequence is related to the previous output.

• LSTM (Hochreiter & Schmidhuber, 1997): Long short-term memory network (LSTM) for solving the gradient disappearing problem of the traditional RNN.

• GRU (Cho et al., 2014): Gated Recurrent Unit (GRU), which can be considered as a variant of LSTM, but it is easier to implement and compute.

• CNN (Kim, 2014): Multiple parallel convolution kernels used to extract text local features.

• BERT (Devlin et al., 2018): We use the bert-base-cased and fine-tuned on the trained Bert.

• XLNet (Yang et al., 2019): We use xlnet-base-cased and fine-tuned on the trained XLNet.

We compare IAOPM with the self-attention mechanism and FP-Net based on the above baseline models.

4.1.3. Experimental parameters

In our experiments, all the word embeddings use 300-dimensional vectors trained by Word2vec (except BERT and XLNet). The specific settings for each feature extractor are shown in Table 1, where the CNN has a window size of [2, 3, 4]. We maintain these parameters consistent across all experiments. Additionally, all other adjustable parameters in the experiments were the optimal parameters determined through multiple trials.

Table 1. Parameter settings of feature extractors.

Model	lr	Dropout	Embedding	Batch size
RNN	1e-4	0.5	300	256
LSTM	1e-4	0.5	300	256
GRU	1e-4	0.5	300	256
CNN	1e-4	0.5	300	256
Bert	2e-5	0.1	768	32
XLNet	2e-5	0.1	768	32

4.2. Experiment results

Firstly, we evaluate the proposed method on four widely used benchmark datasets and compare it with a self-attention mechanism and FP-Net. Our goal is to verify whether the proposed method is general and effective against different baseline models on different datasets. Note that all networks have the same basic structure (from input to FC layer), except for the components of IAOPM, self-attention mechanism and FP-Net. The experimental results are shown in Table 2, where X + A means the adding of self-attention mechanism on X feature extractor, X + FP means that the FP-Net uses X as the feature extractor, and X + IAOPM means the adding of an IAOPM in the X feature extractor. We measure the performance of text classification models through the highest accuracy. From Table 2, we can draw the following conclusions.

Table 2. Results of IAOPM, Self-Attention and FP-Net for four benchmark datasets.

Model	CR	MR	SST2	Subj
RNN	76.92	76.31	79.97	90.40
RNN + A	80.63	78.65	81.66	90.80
RNN + FP	80.11	78.28	80.51	90.80
RNN + IAOPM	80.64	79.40	81.88	91.30
LSTM	79.84	78.65	81.44	90.80
LSTM + A	81.17	78.84	81.49	91.10
LSTM + FP	80.78	78.84	81.82	91.20
LSTM + IAOPM	81.17	80.24	83.58	91.40
GRU	80.10	78.28	82.27	91.50
GRU + A	80.90	80.15	82.37	91.60
GRU + FP	80.64	79.50	82.52	91.55
GRU + IAOPM	81.17	81.18	82.92	91.70
CNN	84.35	80.24	85.28	92.20
CNN + A	85.94	81.27	85.06	92.20
CNN + FP	85.15	81.18	85.35	92.70
CNN + IAOPM	85.68	81.46	85.45	92.70
Bert	90.60	83.63	90.44	95.80
Bert + A	91.00	83.91	90.82	96.25
Bert + FP	91.26	84.80	90.95	95.85
Bert + IAOPM	91.13	84.47	91.16	95.85
XLNet	76.82	73.26	78.53	90.20
XLNet + A	76.91	73.73	78.65	89.55
XLNet + FP	78.28	73.51	78.81	90.15
XLNet + IAOPM	79.07	73.92	79.24	89.90

In our experiments, both IAOPM and FP-Net consistently enhanced the performance of various baseline feature extractors, including RNN, LSTM, GRU, Bert, and XLNet. For instance, using FP-Net on RNN-MR resulted in a 1.97% improvement, while IAOPM yielded a 3.09% improvement. The only exception was when using IAOPM with XLNet-Subj, where it performed slightly worse than the self-attention mechanism. This demonstrates the effectiveness of the orthogonal projection in purifying text features.

In most experiments, IAOPM outperformed the self-attention method in improving the baseline. The only exceptions were the LSTM-CR, CNN-CR, and Bert-Subj experiments, where IAOPM performed slightly worse or equally to the self-attention mechanism. This demonstrates that IAOPM can effectively eliminate the influence of common features on representation learning by removing these features from the text, allowing the system to discover more discriminative features for object classes. Simply assigning lower weights to common features does not fully mitigate their impact on the final classification, which is why IAOPM offers more improvements compared to the self-attention mechanism.

Table 3 shows the time overhead of self-attention mechanism, FP-Net, and IAOPM on the CNN model. As shown in Table 3, the time overhead of self-attention mechanism, FP-Net, and IAOPM increased by an average of 12.16%, 77.395%, and 69.93% respectively compared to the baseline CNN on the four datasets. Although FP-Net and IAOPM had a larger increase in time overhead than the self-attention mechanism, they also had a higher relative accuracy. In addition, IAOPM had a lower time overhead increase than FP-Net, while also having a higher accuracy, indicating better performance of IAOPM.

Table 3. Time overhead table.

Model	Batch size	epoch	CR (seconds)	MR (seconds)	SST2 (seconds)	Subj (seconds)
CNN	256	20	92.48s	247.06s	120.82s	244.94s
CNN + A	256	20	93.88s	205.36s	160.18s	262.40s
CNN + FP	256	20	159.24s	346.08s	254.58s	457.06s
CNN + IAOPM	256	20	147.8s	343.38s	249.10s	428.02s

In summary, IAOPM has higher accuracy and lower time overhead compared to FP-Net. This demonstrates that IAOPM is able to generate more pure features, thereby improving text classification accuracy. For practical applications, the modular structure of IAOPM applied within a single network is more flexible than the dual stream network structure of FP-Net, as it does not require the construction of an additional independent branch structure and can be rapidly applied as a plug-in module to a base network without significant changes to the overall structure.

4.3. Visual analysis

In the orthogonal projection method, the quality of common features is crucial as it can affect the effectiveness of data purification. If the quality of common features is high, it is possible to more effectively extract discriminating features from the original data, thereby improving the performance of purification. To more intuitively compare IAOPM and FP-Net, we visualise the common features extracted by both methods and analyse whether they cluster in space. This allows us to evaluate the quality of the common features and determine which method extracts common features of higher quality.

First, we visualise the common features generated by IAOPM and FP-Net in two-dimensional space (Figure 8(left half), where red indicates positive, and blue represent negative.). On the two datasets in two-dimensional space, the common features generated by IAOPM are more clustered due to the continual constraint on the common features by OPL during training. However, IAOPM generates two clusters on MR, as OPL typically only affects one or a few batches, and is unable to control the overall clustering of common features (Algorithm 1, shown in line 13). Therefore, if the system divides features into multiple parts during training, then the common features generated by IAOPM will also produce multiple clusters correspondingly.

Figure 8. The visualisation of common features in 2D and polar coordinate space for IAOPM and FP-Net.

Then, we also visualise the common features generated by IAOPM and FP-Net in polar coordinate space (Figure 8(right half), where red indicates positive, and blue represent negative.) and three-dimensional space (Figure 9, where red represents positive and blue represents negative.) to ensure the generalizability of the results. In polar coordinate space, the degree of clustering of the common features is reflected in the distance between them and the points. It can be observed that in polar coordinates, the common features generated by IAOPM are clearly more clustered, almost converging to one point, while the common features generated by FP-Net are more dispersed, with greater distances between them. In three-dimensional coordinate space, there is hardly any difference between the common features generated by IAOPM and FP-Net.

Figure 9. The visualisation of common features in 3D space for IAOPM and FP-Net.

According to the above experiments, IAOPM's method of generating common features through IA (shown in Algorithm 1, lines 3–8) and optimising them using OPL (shown in Algorithm 1, line 13) is superior to FP-Net.

4.4. Ablation experiments and analysis

In order to analyse the effectiveness of each component of IAOPM, we performed the ablation experiments as reported in Table 4. Note that in CNN + O + OPL, we use Equation (3) to replace $(A\cdot f_t)$ with $f_t$ for generating common features and ensuring that the experiments can be carried out. From the results reported in Table 4, where CNN + X means the adding of X components to CNN, IA means inverted attention module, O means orthogonal projection module, and OPL means orthogonal projection loss. we can observe the following:

Table 4. Ablation experiments.

Model	CR	MR	SST2	Subj
CNN	84.35	80.24	85.28	92.2
CNN + IA	66.31	76.87	73.91	90.6
CNN + IA + O	84.08	80.99	84.29	92.0
CNN + IAOPM	85.68	81.46	85.45	92.7
CNN + IA + OPL	64.19	72.66	76.39	90.3
CNN + O + OPL	84.08	80.52	83.64	92.3

In the experiments of CNN + IA + O, we remove OPL, which means that it no longer constrains the common features. The results show that the accuracy rate is decreased by 1.6% on the CR dataset, 0.47% on the MR dataset, 1.16% on the SST2 dataset, and 0.7% on the Subj dataset. In conjunction with Figures 7(a) and (b), it is shown that OPL (i.e. Equation (10)) constraining the common features is valid.

In the CNN + IA and CNN + IA + OPL experiments, we removed the orthogonal projection module. Obviously, this will lose a large number of high-weight features. Compared with IAOPM, the accuracy is decreased by 19.37% (CNN + IA + O) and 21.49% (CNN + IA + OPL) in the CR dataset. In the CNN + O + OPL experiment, we removed IA and only get common features through $W_V$. Compared to IAOPM, the accuracy drops by 1.6%. The above experiments show that it is effective and feasible to generate common features through IA and purify combined text features with orthogonal projection.

These results indicate that each component in IAOPM is important and the absence of any one component will result in a loss of accuracy.

Finally, with all other parameters held constant, we conducted relevant experiments on the threshold parameter $T$ in formula 2, as shown in Table 5, where hyperparameters $T$ (Equation (2)) is expressed as a percentage. For example, T = 10 means removing the top 90% of the text features, T = 90 means removing the top 10% of the text features. As shown in Table 5, it is generally recommended to set the $T$for sequence models such as RNN and LSTM between 20% and 30%, while the $T$for CNN is recommended to be set between 60% and 70%. However, for the best purification performance, the threshold should be adjusted according to the specific situation. This characteristic of being able to flexibly adjust parameters according to the actual situation to control the purification effect also indicates the potential for IAOPM to have certain industrial applications.

Table 5. Hyperparametric $T$ experiment.

Model	$T$ (%)	CR	MR	SST2	Subj
LSTM + IAOPM	10	79.58	79.18	82.15	91.2
	20	81.17	79.03	82.48	91.3
	30	80.64	80.24	82.59	91.5
	40	80.37	79.53	82.45	91.1
	50	80.50	79.75	82.49	91.3
	60	80.64	79.09	82.42	91.1
	70	79.58	79.21	83.29	91.2
	80	80.11	79.06	83.58	91.1
	90	80.30	79.01	83.31	91.2
CNN + IAOPM	10	83.55	80.43	85.45	92.1
	20	83.91	80.56	85.21	92.0
	30	83.82	80.80	85.17	92.4
	40	84.08	80.81	85.09	92.2
	50	84.08	81.37	85.06	92.4
	60	85.15	81.46	84.56	92.5
	70	85.15	81.27	85.06	92.7
	80	84.08	81.09	84.46	92.6
	90	84.35	81.02	84.62	92.2

5. Conclusion

In this study, we proposed the Inversed Attention Orthogonal Projection Module (IAOPM). This method extracts common features by iteratively reversing the attention distribution map of text features using inverse attention, and ensures the quality of these common features through OPL. Finally, the text features are purified using orthogonal projection. Compared to the original orthogonal projection, IAOPM does not add any branches and is more flexible without sacrificing performance. Through a large number of comparison experiments, it was proved that the proposed method performs better than the baseline model, self-attention mechanism, and original orthogonal projection method on multiple text classification datasets, with an average accuracy increase of 1.02%, 0.44%, and 0.52%, respectively. Currently, our method is only suitable for traditional text classification methods such as sequence models (RNN, LSTM, GRU, etc.), Bert, etc. and can only ensure the sharing of common features under one or a few small batches. In the next step, we will try to guarantee the sharing of common features as a whole and further reduce the number of parameters.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work is supported by the National Science Foundation of China [grant number62166025], and the Science and technology project of Gansu Province [grant number 21YF5GA073].