An enhanced text classification model by the inverted attention orthogonal projection module

Figures & data

Figure 1. Orthogonal projection purifies text features.

Figure 2. Structure of FP-Net and structure of IAOPM.

Figure 3. IA generated common features, including Attention Generation and Inverted Attention to Generate Common Features.

Figure 4. IA obtained low-weight information by resetting weight of high-weight features to 0.

Figure 5. Orthogronal projection in 2D space.

Figure 6. (a): The distribution of common features in 2D space in normal circumstances (b): The distribution of common features generated by IA in 2D space

Figure 7. (a): Visualisation of the common features and text features of CNN, IAO, and IAOPM in 2D space. (b): Visualisation of the common features (different classes) of IAO, IAOPM, and FP-Net in 2D space.

Table

Algorithm 1 Inverted Attention Orthogonal Projection
Input:
Dataset $D=\left\{\right(x_i,y_i){\}}_{i=1}^N$,$x_i$’s embedding matrix $X_i\in {\mathbb{R}}^{M\times K}$
Output: text classification model.
1:	for each iteration do
2:	Feature map $f_t$ from feature extractor$F$.
3:	Attention generation part:
4:	Generate attention map $M$ (Equation (1))
5:	Attention inversion part:
6:	Inversed attention map$M$, generate inverted attention map $A=\left\{a_i\right\}$ (Equation (2))
7:	Generate common feature part:
8:	Generate common features. (Equation (3))
9:	Orthogonal projection part:
10:	Get the pure features. (Equations (4) and (5))
11:	Perform classification.
12:	Orthogonal projection loss part:
13:	Compute CE loss and OPL (orthogonal projection loss). (Equation (11)), where OPL is computed using Equations (7)–(10)
14:	Back-propagation.
15:	end for

Table 1. Parameter settings of feature extractors.

Model	lr	Dropout	Embedding	Batch size
RNN	1e-4	0.5	300	256
LSTM	1e-4	0.5	300	256
GRU	1e-4	0.5	300	256
CNN	1e-4	0.5	300	256
Bert	2e-5	0.1	768	32
XLNet	2e-5	0.1	768	32

Table 2. Results of IAOPM, Self-Attention and FP-Net for four benchmark datasets.

Model	CR	MR	SST2	Subj
RNN	76.92	76.31	79.97	90.40
RNN + A	80.63	78.65	81.66	90.80
RNN + FP	80.11	78.28	80.51	90.80
RNN + IAOPM	80.64	79.40	81.88	91.30
LSTM	79.84	78.65	81.44	90.80
LSTM + A	81.17	78.84	81.49	91.10
LSTM + FP	80.78	78.84	81.82	91.20
LSTM + IAOPM	81.17	80.24	83.58	91.40
GRU	80.10	78.28	82.27	91.50
GRU + A	80.90	80.15	82.37	91.60
GRU + FP	80.64	79.50	82.52	91.55
GRU + IAOPM	81.17	81.18	82.92	91.70
CNN	84.35	80.24	85.28	92.20
CNN + A	85.94	81.27	85.06	92.20
CNN + FP	85.15	81.18	85.35	92.70
CNN + IAOPM	85.68	81.46	85.45	92.70
Bert	90.60	83.63	90.44	95.80
Bert + A	91.00	83.91	90.82	96.25
Bert + FP	91.26	84.80	90.95	95.85
Bert + IAOPM	91.13	84.47	91.16	95.85
XLNet	76.82	73.26	78.53	90.20
XLNet + A	76.91	73.73	78.65	89.55
XLNet + FP	78.28	73.51	78.81	90.15
XLNet + IAOPM	79.07	73.92	79.24	89.90

Table 3. Time overhead table.

Model	Batch size	epoch	CR (seconds)	MR (seconds)	SST2 (seconds)	Subj (seconds)
CNN	256	20	92.48s	247.06s	120.82s	244.94s
CNN + A	256	20	93.88s	205.36s	160.18s	262.40s
CNN + FP	256	20	159.24s	346.08s	254.58s	457.06s
CNN + IAOPM	256	20	147.8s	343.38s	249.10s	428.02s

Figure 8. The visualisation of common features in 2D and polar coordinate space for IAOPM and FP-Net.

Figure 9. The visualisation of common features in 3D space for IAOPM and FP-Net.

Table 4. Ablation experiments.

Model	CR	MR	SST2	Subj
CNN	84.35	80.24	85.28	92.2
CNN + IA	66.31	76.87	73.91	90.6
CNN + IA + O	84.08	80.99	84.29	92.0
CNN + IAOPM	85.68	81.46	85.45	92.7
CNN + IA + OPL	64.19	72.66	76.39	90.3
CNN + O + OPL	84.08	80.52	83.64	92.3

Table 5. Hyperparametric $T$ experiment.

Model	$T$ (%)	CR	MR	SST2	Subj
LSTM + IAOPM	10	79.58	79.18	82.15	91.2
	20	81.17	79.03	82.48	91.3
	30	80.64	80.24	82.59	91.5
	40	80.37	79.53	82.45	91.1
	50	80.50	79.75	82.49	91.3
	60	80.64	79.09	82.42	91.1
	70	79.58	79.21	83.29	91.2
	80	80.11	79.06	83.58	91.1
	90	80.30	79.01	83.31	91.2
CNN + IAOPM	10	83.55	80.43	85.45	92.1
	20	83.91	80.56	85.21	92.0
	30	83.82	80.80	85.17	92.4
	40	84.08	80.81	85.09	92.2
	50	84.08	81.37	85.06	92.4
	60	85.15	81.46	84.56	92.5
	70	85.15	81.27	85.06	92.7
	80	84.08	81.09	84.46	92.6
	90	84.35	81.02	84.62	92.2