GP-GCN: Global features of orthogonal projection and local dependency fused graph convolutional networks for aspect-level sentiment classification

Abstract

Aspect-level sentiment classification, a significant task of fine-grained sentiment analysis, aims to identify the sentimental information expressed in each aspect of a given sentence The existing methods combine global features and local structures to  obtain good classification results. However, the introduction of global features will bring noise and reduce the classification accuracy. To solve this problem, a new method is proposed, named GP-GCN. In our proposed method, the global feature is further simplified to reduce the noise . The local structures and global features obtained by orthogonal feature projection are introduced into aspect-level sentiment classification. First, the simplified global feature structures of text are built. Through orthogonal projection, GCN not only weakens the dependency of the graph node in updating process but also reduces the dependency between node and corpus. Next, syntactic dependency structure and sentence sequence information are utilised to mine the local dependency structure of sentences. A percentage-based multi-headed attention mechanism is proposed to measure the critical output of GCN, which can better represent sentences for given aspects. Finally, location coding is input to simulate aspect-specific representations between each aspect and its context such that the text becomes more discriminative in sentiment classification. The experimental results show that the proposed method effectively improves the accuracy of text sentiment classification.

KEYWORDS:

• Aspect-based sentiment classification
• BERT
• GCN
• attention

1. Introduction

At present, sentiment analysis research for the whole text has been relatively perfect (Zhang et al., 2021; Zhang et al., 2022; Zhou et al., 2020) and has obtained good classification results. However, the sentiment analysis of the whole text will cover up its details, so it cannot reflect people's fine-grained sentimental expressions. Therefore, it is necessary to find all aspects of text reviews and determine the sentimental polarity expressed by the text. These can provide strong technical support for text sentiment classification, recommendation systems (Geler et al., 2021; Wu et al., 2022), question-answering systems, etc. As shown in Figure 1, in the sentiment sentence ‘computer screen is very good, but the price is not friendly’, ‘computer screen’ and ‘price’ has two aspects. Their views are ‘very good’ and ‘not friendly’ respectively, and their sentimental polarity is positive and negative.

Figure 1. An example of aspect-based sentiment classification.

In recent years, local information has been used to capture rich semantic information (Cai et al., 2021; Mansouri-Benssassi & Ye, 2021; Zhang et al., 2022). Although this work has achieved good results, its limitations are mainly concentrated on the use of local features such as order information and syntactic dependence in a given sentence. The global features of a sentence are not fully considered, so it is difficult to conduct a more complete sentiment analysis of the text. Specifically, global features refer to the global dependency information between words, which is often hidden in the whole corpus to reveal the global relationship between words. Recent studies have shown that using global structural dependence can usually significantly improve the performance of text classification. More and more experts use GCN to learn global features (Zargari et al., 2021; Zhou et al., 2019) and capture high-order neighbourhood information. However, when considering global dependency information, it is often overdependent on the corpus. Specifically, the existing methods usually model the corpus, resulting in too many interference factors. It lacks effective modelling of global dependence information, so it is difficult to truly mine valuable global dependence information. Different from previous work, the sentiment analysis in this paper is fine-grained aspect-level sentiment analysis. A GCN model integrating simplified global features and local features is constructed based on feature projection.

Given the above problems, effective fine-grained sentiment expression research should fully consider the following two keystones. (1) How to obtain global structural features from the text. The global structural features are hidden in the whole database. When modelling the database, the interference words are eliminated, and the global features in the database are fully excavated, which can reveal the global relationship between words. (2) How to fuse global features and local features effectively. The global features and local dependency information are fused to obtain a complete sentiment features vector for text sentiment classification.

Based on the above considerations, we introduce the local structure and global features obtained by orthogonal feature projection into the aspect-level sentiment classification. Furthermore, a new aspect-level sentiment classification method is proposed.

The motivation of this paper is to further simplify the global features from the view of reducing the interference of global features. Furthermore, the accuracy of text sentiment classification can be improved by fusing the global features and local dependency. The framework is shown in Figure 2. Two GCN networks are used to learn the global features and local dependency information of the text, respectively, and the two features are fused. Thus, the interdependent information can be accurately extracted from the dependency tree. The model framework is divided into two aspects: (1) Modelling the text corpus by GCN to obtain valuable global features (G-GCN). The output of the orthogonal projection is the input of the GCN. (2) Exploring each sentence by fusing GCN to obtain local features dependency information (P-GCN). The input of GCN is the output of BERT and Bi-LSTM. The advantage of G-GCN is to simplify global features by reducing the noise of the global features. The advantage of P-GCN is to make the local features more representative and learn more discriminative features.

Figure 2. The framework of GP-GCN.

The main contributions of this paper are as follows:

• The acquisition of global and local features for the text. Global features obtained by orthogonal feature projection and local dependence are better for solving the long-term problem of multi-word dependence. It not only retains the overall attributes of global features but also reduces noise interference for further refining global features. At the same time, the influence of local features is considered on text sentiment classification, and local features are refined. The effective integration of the two improves the accuracy of text sentiment classification.

• The fusion of global features and local dependency information. Percentage-based multi-head attention mechanism to accurately measure the key output of GCN and obtain a sentence representation for a given aspect. It can not only capture long-distance dependencies but also prevent over-fitting by setting super parameters. The accuracy of text sentiment classification is improved.

The division of this paper is as follows. In Section 2, we summarise the related work. Then, we detail our GP-GCN model in Section 3. In Section 4, we set up experiments based on the dataset and analyse the results. Finally, we summarise our work and put forward some ideas for future work in Section 5.

2. Related works

This section reviews related research work of traditional learning method and graph neural network method in aspect-level sentiment classification.

2.1. Aspect-based sentiment analysis (ABSA)

The purpose of ABSA is to analyze the sentimental polarity of specific aspect items in a sentence (Cambria, 2016; Cambria et al., 2022; De Meo et al., 2020). Its subtasks mainly include aspect item extraction, opinion item extraction, Explainable Artificial Intelligence (XAI), and aspect item sentiment classification. Zhang et al. (2020) jointly extracted aspect items, opinion items, and sentiment polarity and named it a new subtask in aspect-level sentiment analysis task as opinion triplet extraction (OTE). As shown in Figure 1, the sentence ‘the screen of computer is very good, but price is unfriendly’, ‘computer screen’ and ‘price’ have two aspects, their views are ‘very good’ and ‘unfriendly’, sentiment polarity is positive and negative. Usually, the task is described as a triad consisting of aspects, opinions, and sentiment polarity. Given the low efficiency of artificial annotation features, early aspect-level sentiment analysis is mainly based on neural networks. Especially, since Tang et al. (2016) proposed the challenge of establishing a semantic correlation between context and aspect words, attention-based and recurrent neural network methods are usually used in ABSA. Chen et al. (2017) introduced an attention mechanism into regular RNN to further solve the long-term dependence problem. At the same time, LSTM based on the attention network has also been proved to identify important sentimental information related to targets. However, due to the lack of a mechanism to explain relevant syntactic constraints and long-distance word dependence in such models, Xiao et al. (2020) composed multiple attention and improved GCN established on the sentence dependence tree for dealing with complex text structures. Zuo et al. (2020) used token semantics and proposed a GCN framework with heterogeneous context, which can better reflect the structural context information.

Although the above work has achieved good results, they mainly focus on the use of local features such as sequence information and syntactic dependence in a given sentence, which is their limitation. So, we need to consider the global features of the sentence to obtain better accuracy.

2.2. Traditional learning method

The traditional deep learning methods mainly include convolutional neural networks, neural memory networks, and circulatory neural networks. Ruder et al. (2016) embed words into sentence-level Bi-LSTM to learn intra-sentence and inter-sentence relationships. Tay et al. (2017) used a binary memory network to optimise memory selection operation and improve binary interaction between aspects and word embedding. However, In the research process, some given aspects are easily ignored. To strengthen attention to given aspects, Wang et al. (2016) set an attention vector for each target based on LSTM to respond to specific targets. Huang et al. (2018) introduced aspect word information into CNN through parameterised filters to effectively obtain the features vector of a given aspect. Wu et al. (2018) combined the target as a feature and its context features to conduct aspect-level sentiment classification. When dealing with the interaction between context and target, LSTM cannot explain the importance of each word in the full text. To address this issue. Yang et al. (2017) introduce an attention mechanism into Bi-LSTM to improve classification results. Li et al. (2017) used a target recognition task to simulate aspectual sentiment interaction. Fan et al. (2018) used the attention mechanism to capture the words and multi-word expressions in the sentence simultaneously, which can more accurately locate the sentiment polarity. To extend the attention model, an attention mechanism with a memory network (Li et al., 2021) can capture the importance of each context word for a given target. Inspired by this, Basiri et al. (2021) proposed an attention-based bidirectional CNN-RNN learning model (ABCDM), which can extract past and future contexts in the time dimension and measure the importance of different features.

With the deepening of the research, the sentimental contextual knowledge of specific entities needs to be further explored. (Xue & Li, 2018; Zhang et al., 2016) used the gating neural network structure to control the propagation deviation and model the interaction between aspect words and context words. (Li et al., 2018; Thesia et al., 2022) adjusted the input of the convolution layer by using the position correlation between words and aspect words for better input representation. Liu and Zhang (2017) used two attention to sentence-level semantics and context. At the same time, the order and correlation of words are considered. Huang et al. (2018) modelled aspects and sentences jointly and captured the interaction between aspects and context sentences. To solve the problem that implicit sentiments are difficult to be mined, the capsule network is used to extract additional semantic information to obtain high-level feature information (Hussain et al., 2021; Yan et al., 2021). But Gupta et al. (2022) use an improved binary particle swarm algorithm to improve sentiment classification accuracy by global and local optimisation. Inspired by this, our model is to model the text corpus to obtain valuable global sentiment features and enhance sentiment mining for a given aspect.

2.3. Graph convolutional networks (GCNs)

A graph is a data structure representing the multi-to-multi relationship between objects, covering the representation of a set of one-to-one and one-to-many relationship structures, highlighting the strong representation ability of the graph structure. GCNs can obtain the information of adjacent nodes. It can fully excavate interdependent information from complex relational data.

With the update of sentiment analysis requirements, data in many practical application scenarios are generated from non-European spaces. Traditional deep learning methods have successfully extracted features of Euclidean spatial data, but their performance in dealing with non-European spatial data is still unsatisfactory. GCNs are more competent in this respect, and they capture graph dependency through message passing between graph nodes (Zhang et al., 2019; Zhang et al., 2020). It can be used to explain the mechanism of related syntactic constraints and long-distance word dependence and better identify context words unrelated to grammar. Many studies applied GCNs to aspect-level sentiment classification. GCNs is a simple and effective deep learning network. It can obtain not only the information of adjacent nodes but also the information of other nodes of adjacent nodes. It can fully excavate the information of interdependence from rich relational data. Hou et al. (2021) selected the GCNs model based on the dependency tree, which used multiple attention to select attention for each word in the sentence. Zhao et al. (2020) used GCNs to capture the sentiment dependence between different aspects of sentences on the bidirectional attention mechanism of location coding. Liang et al. (2022) propose a SenticGCN model, which combines the sentimental dependence of sentences to mine the specific aspects of sentences.

The existing methods often rely too much on the whole corpus when considering the global dependency information, resulting in too many interference factors and a lack of effective modelling of global dependency signals, which makes it difficult to truly mine valuable global dependency information. For example, Zhou et al. (2020) used a syntactic dependency tree and common-sense map to enhance the representation of each given aspect in the sentences, which can effectively incorporate external knowledge into the sentiment classification model to improve the accuracy. Zhu et al. (2021) constructed a document map to model the whole corpus to mine the global dependency between words. Different from the above, our model can effectively obtain global dependence and local features. Specifically, the global features obtained by orthogonal feature projection and local dependence can better solve the long-term problem of multi-word dependence. It not only retains the overall attributes of global features but also reduces noise interference, further refines the global features, and improves the accuracy of text sentiment classification.

3. GP-GCN of our proposed method

To improve the accuracy of text sentiment classification, we further consider the noise interference of global features and refine the global features. Combining the advantages of the BERT model, GCN model, and attention mechanism, we introduce the global features by orthogonal features projection and local structure obtained into the aspect-level sentiment classification (GP-GCN). We use two fusion graph convolution neural networks to learn the global features (G-GCN) and local dependence (P-GCN) information of the text, respectively. The model framework is shown in Figure 2. It is composed of an embedded layer, convolution layer, pooling layer, attention layer, and full connection layer.

3.1. Embedded layer

The main role of the input embedding layer is to represent the text statement as a word vector matrix and map each word into the vector space of high latitude. In traditional processing tasks, words usually use One-Hot Encoding to represent text with discrete values. However, this method ignores the correlation between words and context information, which is easy to cause data disaster. To solve these problems, researchers proposed to represent words as continuous dense low latitude vectors (Giatsoglou et al., 2017). At present, the commonly used methods are Glove (Pennington et al., 2014), Word2Vec, and BERT (Devlin et al., 2018).

In our model, we use the Glove and BERT pretraining models to obtain word embedding representations. In Glove, $E_t\in R^{d_{emb}\times v_i}$ is a word embedding matrix, where $d_{emb}$ is the dimension of a word vector. $v_i$ represents the size of a vocabulary. Word embedding $w_t$ is $x_t\in R^{d_{emb}\times 1}$, And it can be calculated by the word embedding matrix $E_t$. In BERT, we use parameters BERT_base as the experimental basis. As the given sentence S =  {w₁, w₂, w₃, … , w_n}, the input is out = {[CLS], w₁, w₂, w₃, … , w_n, [SET]}. $Z=BERT(out)\in R^{d_{bert}\times N}$, where $d_{bert}$ is the size of the hidden dimension. Word embedding $w_t$ is $z_t\in R^{d_{bert}}$, they can map words into high latitude vector spaces. Then sentiment lexicon $senti(w_i)$ (Xu et al., 2022) is used to weight the word vector, and the formula is as follows: (1) $z_i=w_i\ast senti(w_i)$(1) where $w_i$ is the weighted word vector, the matrix is used as the output of the embedding layer.

3.2. Bi-LSTM

Since Bi-LSTM can process variable-length sequence information (Tang et al., 2015; Wei et al., 2020), we summarise two directions of information through three gated units that control information transmission to obtain context information more accurately. The calculation formula is shown in Formula (2). (2) $h_i=[\overrightarrow{lstm}(x_i,{\mathrm{\wp }}_l),\overleftarrow{lstm}(x_i,{\mathrm{\wp }}_l),i\in [1,n]],j\in [n,1]$(2) where ${\mathrm{\wp }}_l$ is the parameter of the LSTM model. We connect the hidden state to obtain $h_i$. Input word embedding $Z=[z_1,z_2,z_3,\dots ,z_n]$, we use $h_i$ to represent the given the word $w_i$ and get to $H=[h_1,h_2,h_3,\dots ,h_n]\in R^{d_h\times N}$. Where $d_h$ represents the dimension of the hidden state. The word input into GCN is the features matrix of this node.

3.3. Graph convolutional network

GCNs is a multi-layer neural network that is suitable for the convolution operation of nodes in the graph. Formally, we consider a graph G =  (V, E, A), |V| and |E| of G denote the order of graph G and the number of edges. $A\in R^{N\times N}$ is the corresponding adjacency matrix of G. In the L^th layer of GCN, $H_i^{(l+1)}$ is Layer L output as the i node. The convolution operation formula is shown in Formula (3). (3) $H_i^{(l+1)}=\sigma (\sum _{j=1}^k{A}_{ij}{W}^{(l)}{H}_i^{(l)}+B^{(l)})$(3) Where l∈ {1, … , L}, $\sigma$ is the sigmoid function. In the function, $W^{(l)}$and $B^{(l)}$ is weight and biased, respectively. Through the above operation of GCN, the aggregated information of each node is not only itself but also its direct neighbour. Then the information of adjacent internal nodes was collected by the L+1 layer GCN.

3.4. Neutral word vectors and CPD-NET

In this paper, some words with special meaning are given appropriate weight, that is, by calculating the relevance of the context of neutral words. Then, the selected neutral word vector is projected into the sentimental polarity word vector space, so the neutral word vector is given appropriate weight. When tagging text, there are generally more explicit sentimental features, such as ‘perfect’ representing positive sentiments, ‘bad’ and ‘disgusting’ representing negative sentiments. However, there are also words in the text called neutral words (e.g. ‘general’, ‘normal’, ‘?’). Their sentiment characteristics are not obvious, and there is no ambiguity in the classification. They all belong to the neutral word vector, but the neutral word vector will affect the results of text sentiment classification to some extent. In this paper, we propose a new feature projection method. We project the neutral feature vector to the orthogonal direction of the emotional polarity feature vector to generate the final and more comprehensive features for classification. Obviously, this orthogonal projection is to get rid of common features, so that the model not only pays attention to those distinctive features but also pays attention to neutral features with special significance. The specific projection method is explained later.

Definition 1. The Degree of Word Correlation (DWC). It is used to measure the correlation between sentimental polarity words and neutral words in the context. That is, to calculate the sentimental polarity intensity of neutral words from the context. Words or characters with special significance can be more accurately identified through the semantic correlation between words. The integrity of feature extraction can be improved. The calculation formula is shown in Formula (4). (4) $DWCi=|Ci\times Wi\times Li|$(4) where $Ci$ represents the correlation between neutral words and polar words, and the calculation formula is shown in Formula (5). (5) $Ci=\frac{\sum (Ru,i-Ru)(Ru,j-Ru)}{{\sqrt{\sum (Ru,i-Ru)}}^2\sqrt{\sum {(Ru,j-Ru)}^2}}\times \left(\frac{R\text{i}\times Rj}{\text{R}{\text{u}}^{\text{2}}}\right)$(5) where $Ru$ represents the average value in the whole review corpus. $Ru,i$ represents the correlation between words i and the average value. $Wi$ represents the initial weight of neutral words and $Li$ represents the position of words.

Because only a small part of the neutral words in the text has special significance, we use the LTP tool of the Harbin Institute of Technology to segment the text according to the part of speech. It calculates the semantic association between words and selects the neutral words or symbols with special significance. As Figure 3 shows, ‘This computer is really unusual!’, where ‘computer’, ‘general’ and ‘!’ are all neutral words. By calculating the semantic association between neutral words and polar words, the neutral words ‘general’ and ‘!’ with special significance are screened out. The words greater than the threshold are selected as the screening result to provide data support for subsequent feature projection operations.

Figure 3. Sample description of word relevance.

CPD-NET framework is shown in Figure 4. The framework comprises two parts: the feature projection network (CP-NET) and the sentimental polarity features learning network (D-NET). The model projects the complete information vector of the learned text into a more recognisable semantic space. It can identify special neutral word vectors affecting text sentiment classification, so the global features are simplified. The purpose of CP-NET is to extract the neutral word feature vector with special significance and give appropriate weights. D-NET is to obtain sentiment polarity word vector. CP-NET and D-NET are composed of three parts. The first two layers are the input layer and the feature extraction layer. The input layer has the same input $x_i$, where $x_i$ is an input document with the length L. The feature extraction layer uses VD-CNN (Conneau et al., 2017). The third layer is the orthogonal projection layer (OPL) and the reverse gradient layer (GRL). The core idea of the model is to project the feature vector $f_{cp}$ to the orthogonal direction of $f_d$, $f_{cp}$ is projected to a more discriminative semantic space. It can identify the noise interference of the global features and simplify the global features. And the special neutral word vector, which has an impact on text sentiment classification, is given appropriate weight. The CP-NET feature extraction layer is extracted, and the D-NET feature extraction layer is extracted. Next, each proposed orthogonal projection model component is described in detail.

Figure 4. Orthogonal projection (CPD-NET) model.

When the feature extraction layer of CP-NET receives $X_i$ from the Bert model, $X_i\in R^{L\times k}$ is the embedding matrix of $x_i$. The advanced features $f_{cp}$ are extracted in the form of n-grams from $X_i$ by $F_{cp}$. The specific extraction method $f_{cp}$ is: (6) $f_{cp}=[c_1,c_2,\dots ,c_{(l-n+1)}]=[c_j{]}_{j=1}^{l-n+1}$(6) Where $c_j$ represents the filtering output of $X_i$, A convolution operation for filtering is composed of a filter $W\in R^{n\times k}$ and a bias $\text{b}\in R$. $c_j$ can be expressed as: (7) $c_j=g(W\cdot {\mathrm{X}}_i[j:j+n-1:]+b)$(7) $\text{g}$ is ReLU, and Maxpooling is applied to feature mapping. $f_{cp}=max\{f_{cp}\}$ is used as the corresponding feature of the filter $\text{v}$. The same D-NET feature extraction layer also receives advanced features $f_d$ from the input layer when $X_i$ is received. (8) $\begin{array}{rl}{f}_{cp}& =VD-CN{N}_{cp}(X)\end{array}$(8) (9) $\begin{array}{rl}{f}_d& =VD-CN{N}_d(X)\end{array}$(9) Orthogonal Projection Layer (OPL), As shown in Figure 5. The two-dimensional space $f_d$ represents the feature vector of sentiment polarity words. $f_{cp}$ represents the feature vector of neutral words and $f_{c{p}^{\ast }}$ represents the final weighted neutral word feature vector.

Figure 5. Orthogonal projection principle.

The feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ is set $\widetilde{{\text{f}}_{\text{cp}}}=GRL(f_d)$ by GRL, and the formula is shown as: (10) $\begin{array}{rl}{Y}_{GRL}& =\mathrm{softmax}(\widetilde{{\text{f}}_{\text{cp}}}\cdot W_d+b_d)\end{array}$(10) (11) $\begin{array}{rl}Los{s}_d& =CrossEntorpy(Y_{truth},Y_{GRL})\end{array}$(11) The purpose of D-NET is to extract sentiment polarity feature vectors $f_d$. In the text, sentiment classification tasks can obtain discriminative semantic information. Then, the sentiment classifier uses sentiment polarity words to distinguish different sentiment categories. The purpose of CP-NET is to extract the feature vector $f_{cp}$ of neutral words, which means to obtain semantic information that is not discriminative in text sentiment classification tasks. And then project it to the semantic  space of sentiment polar words for reinforcement learning. This can give the neutral word vector appropriate weight to identify valuable sentiment features. To accurately obtain the feature vector, GRL is added after the feature extractor $F_{cp}$ and $F_d$ to reverse the direction of the gradient (Ganin et al., 2017). Through this training module, we can obtain the neutral word feature vector with low discrimination and the sentiment polarity feature vector with high discrimination in the text. To identify the valuable sentimental features of neutral words, the extracted feature vector $f_{cp}$ is projected onto the orthogonal direction of $f_d$ the sentimental polarity word vector. The feature space orthogonal to the feature vector of sentimental polarity words should contain pure and efficient features for text sentiment classification. $f_{cp}$ is projected into the orthogonal feature space. It calculates not only the similarity between the neutral word feature vector and the polar word feature vector but also refines according to the orthogonal distance between the two. The special neutral word feature vector is given the appropriate weight, so that valuable sentiment features can be well identified.

In mathematics, the neutral word feature vector $f_{cp}$ is projected onto the sentiment polarity word feature vector $f_d$, and it is shown in formula (12). (12) $f_{cp\ast }=Orj(f_{cp},f_d)$(12) where $Orj$ is a projection function, and it is shown in Formula (13). (13) $Orj(u,v)=\frac{u\cdot v}{|u|}\frac{v}{|v|}$(13) where $u,v$ are vectors. Then, the orthogonal direction of the projection feature $f_{cp}$ is projected to obtain the classification feature vector with high discrimination. The calculation formula is shown in Formula (14). (14) $\widetilde{{\text{f}}_{\text{cp}}}=Orj(f_{cp},(f_{cp}-f_{cp\ast }))$(14) It can be seen from the above that the feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ is equivalent to $f_{c{p}^{\ast }}-f_{cp}$. The feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ can be obtained by using the neutral word feature vector $f_{c{p}^{\ast }}$ projection, and a plane (in 3D) can be constructed. The intersection of this plane and the orthogonal plane of the projection feature vector $f_{c{p}^{\ast }}$ is a high-resolution sentimental feature vector.

In addition, the projection in Formula (13) is a constraint on the feature vector of sentimental polarity words. That is, the modulus of sentimental polarity word feature vector is limited when $f_{cp}$ is projected to $f_d$. So the semantic information of sentimental polarity word feature vector obtained by projection is the projection feature $f_{c{p}^{\ast }}$. It only contains sentimental polarity semantic information related to the neutral word feature vector. This makes the final high-resolution sentiment feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ come from the neutral word feature vector $f_{cp}$, rather than the vector on any plane orthogonal to the sentimental polarity feature vector $f_d$. Finally, the feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ obtained by projection is used for text sentiment classification. (15) $\begin{array}{rl}{Y}_{OPL}& =\mathrm{softmax}(\widetilde{{\text{f}}_{\text{cp}}}\cdot W_{cp}+b_{cp})\end{array}$(15) (16) $\begin{array}{rl}Los{s}_{cp}& =C\text{ross}E\text{ntorpy}(Y_{truth},Y_{OPL})\end{array}$(16) where $Los{s}_{cp}$ function and $Los{s}_d$ function are trained simultaneously, but ADAM and SGD optimiser are used, respectively. The gradient is passed back through the characteristic when the loss function is optimised. Although these two loss functions are opposite in the optimisation objective of the feature extractor $F_d$, the influence $F_d$ is in the orthogonal direction $f_d$. This can find a balance point so that the extracted features are closer to the real sentiment polarity features. Algorithm 1 gives the proposed complete training algorithms for CP-NET and D-Net.

Table

Algorithm 1. CPD-NET model obtains neutral word feature vector with weight.

Input: Dataset $D=\{(x_i,y_j){\}}_{i=1}^N$, xi's embedding ${\text{X}}_{\text{i}}ϵ{\text{R}}^{\text{L}\times \text{k}}$

Output: The feature vectors of the neutral word with weight

1. Randomly CP-NET, D-Net's parameters $\theta$

2. for each iteration $b=1,2,3,\dots ,n$ do

3. Sample one batch $X_b$ from D

4. D-NET part:

5. Generate sentiment polarity word features (EPFs) (Eq. 7)

6. EPFs go through GRL (Eq. 8)

7. Obtain precise sentiment polarity word feature vector $f_d$ (Eq. 10)

8. CP-net part:

9. Generate neutral word features (NFs) (Eq. 6)

10. NFs EPFs go through GRL (Eq. 8)

11. Get neutral word feature vectors $f_{cp}$

12. Get $f_{c{p}^{\ast }}$ by Projecting (Eq. 12)

13. Obtain a weighted neutral feature vector $\widetilde{{\text{f}}_{\text{cp}}}$ (Eq. 14)

14. Update parameters:

15. CPD-NET's parameters $\theta$ are updated together (Eq. 11& Eq. 16)

16. End for

Algorithm 1 shows that the first row is the parameter initialisation of the feature network. Lines 4–7 obtain sentiment polarity word feature vectors from the text. Lines 8–11 obtain feature vectors of neutral words from a text. Line 12–13 projects the neutral word feature vector into the orthogonal space of the sentimental polarity feature vector. It obtains the neutral word feature vector with weight, the sentiment feature with high discrimination. Line 14–15 is based on the loss function results to optimise the parameters of the feature network.

3.5. Global structure dependency (G-GCN)

The motivation of this section is to learn global dependency attention weights. So, we need to extract global information of words. We prune the corpus by orthogonal projection to reduce interference factors and give some words with special meanings appropriate weights. When we are committed to extracting the global features of words, a graph of the word co-occurrence and text graph is prevalent. Compared with the construction of the word co-occurrence graph (Peng et al., 2018), the text graph (Yao et al., 2019) is represented as nodes, which consists of words and sentences. Taking sentence nodes as a bridge, we can capture the long-term word relations in the whole corpus, thus further enhancing word representation learning. Therefore, we choose a text graph to input information into the GCN model.

The original Text-GCN began to learn the sentence representation of test data, which will significantly limit its practical application. Therefore, we train the corpus's matrix of word embedding from the perspective of global dependence, which will be more conducive to the training effect of the model. Formally, the weights i and j of edges between nodes are defined as: (17) $A_{ij}=\{\begin{array}{ll}PMI(i,j)& i,jarewords,PMI(i,j)>0\\ TF-IDF(i,j)& iisdocument,jisword\\ 1& 0\\ 0& otherwise\end{array}$(17) The PMI value i, j is calculated as: (18) $PMI(i,j)=\log \frac{\mathrm{\#}N(i,j)\mathrm{\#}N}{\mathrm{\#}N(i)\mathrm{\#}N(j)}$(18) where the value of N(i) is the sliding window of i, so is N(j). #N is the sliding windows’ number. It will be considered when the PMI between the word nodes is positive. Two-layer GCN is applied to the text graph. The formula is as follows: (19) $Y=softmax({\tilde{D}}^{-\frac{1}{2}}\tilde{A}\tilde{D}{}^{-\frac{1}{2}}s({\tilde{D}}^{-\frac{1}{2}}\tilde{A}{\tilde{D}}^{-\frac{1}{2}}X{W}_0)W_1)$(19) where output is Y, W₀ and W₁ are training weight matrices.

In the whole corpus, TextGCN is pre-trained, which can get the embedding matrix of words. Word semantic relationships are reflected by learned word representations from the view of a global feature. Eg is obtained by embedding the original input sentence into the learned word embedding matrix. Then, we combine the output H gate mechanism of Eg using Bi-LSTM.

3.6. Local dependency attention

To obtain aspect-oriented attention, local dependence structures are considered. A GCN is applied in the dependency tree of sentences and used with Bi-LSTM. This process can capture long-term relations. Specifically, according to the words of a given sentence, a dependency tree is constructed at first, and its adjacency matrix $A\in R^{N\times N}$ is obtained. We apply dependency tree modelling by the graph convolution. A self-loop is used for each node, and the activation is normalised before the nonlinear transformation. The formalisation is as follows: (20) $h_i^{(l)}=\sigma \left(\sum _{j=1}^n{A}_{ij}{W}^{(l)}{h}_j^{(l)-1}/d_i+b^{(l)}\right)$(20) where A = A + I, I as an n × n unit matrix, $d_i=\sum _{j=1}^n{A}_{ij}$ represents the scope of the i^th node. W^(l) and b^(l) is the training parameters of a specific layer. The GCN layer on the final output dependency tree is $h^L=(h_1^L,h_2^L,h_3^L,\dots ,h_n^L)$, and the aspect-specific masking layer at the top is further applied, replacing the non-aspect state with zero. In the masking layer, $H_{mask}^L=(0,\dots ,h_{\xi +1}^L,\dots ,h_{\xi +m}^L,\dots ,0)$ as an output. So, we can determine the local dependency attention weight α as the following formula. (21) $\begin{array}{rl}{a}_t& =\frac{\exp (e_t)}{\sum _{i=1}^n\exp (e_t)\mathrm{\Delta }x}\end{array}$(21) (22) $\begin{array}{rl}{e}_t& =\sum _{i=\xi +1}^{r+m}{h}_t^T{h}_t^L\end{array}$(22)

3.7. Percentage multiple attention mechanism

To improve the suitability of the weight assignment of the multi-attention mechanism, a pooling is proposed. In most cases, the weighted average method is selected because the average pool performance is better than the maximum pool performance. However, the average pool sometimes misses the optimal choice because it takes the average as the final standard. So, the average sometimes is not robust. To solve these problems, we propose a new method of Percentage Pooling. The motivation of pth percentage is to minimise p% value of the set-in sorting elements. In general, the 50th percentile is more robust than the average and has more advantages than the maximum pool. For example, with large outliers in a set, the maximum pool as output is not the best choice. And the 80h percentage is better to exclude the abnormal value. In this paper, the pth percentage of vector R as function $f_m(r)$, m can be adjusted as a super parameter. The range of m is between 0 and 100.

4. Experiments

Comparative experiments are carried out on real datasets as shown in Table 1 to verify that the proposed method is effective. The effectiveness of the proposed GP-GCN is verified by the experimental results..

Table 1. Official link of datasets.

Dataset	Official link
SemEval-2014	https://alt.qcri.org/semeval2014/task4/index.php?id = data-and-tools
SemEval-2015	https://alt.qcri.org/semeval2015/task4/index.php?id = data-and-tools
SemEval-2016	https://alt.qcri.org/semeval2016/task4/index.php?id = data-and-tools

4.1. Experimental data

SemEval dataset is the most commonly used dataset in aspect-level sentiment analysis. Its primary task is to analyse the sentiment of Twitter texts. SemEval datasets links are shown in Table 1. The SemEval dataset is used in this experiment, includes 14Res and 14Lap from SemEval-2014, 15 Res from SemEval-2015, and 16 Res from SemEval-2016. In Table 2, the statistical results of the dataset are shown.

Table 2. Statistics of the datasets.

	Positive		Negative		Neutral
Datasets	Train	Test	Train	Test	Train	Test
Twitter	1553	158	3139	395	1490	273
14lap	980	338	912	114	398	172
14res	2210	713	812	187	642	185
15res	958	653	392	487	30	28
16res	1520	572	542	187	69	39

4.2. Experimental setup and evaluation measurement

Because the proportion of N labels in edge labels is much larger than in other tag categories, the data category is not balanced. To solve the imbalance of data, we set the weighted gate of the relationship distance weight. It is not involved in graph calculation when the relationship distance weight is less than the threshold. In the model training process, different optimisers are selected for optimisation so that it makes the model training to bemore effective and convenient for experimental comparison. The selection of each model parameter and optimiser is shown in Table 3.

Table 3. Parameter setting.

	Parameters	Values/Descriptions
Word embedding	Embedding size	300
Bi-LSTM	Hidden layers	2
	Hidden neurons	180
	Max time step	140
GCN	Layers	4
	Network size	4[360,7]
Pooling	Pooling method	50th percentile pooling
Regularisation	Loss function	Cross entropy
	L2 regularisation	1×10^−8
	Orthogonal regularisation	1×10^−8
Training	Optimiser	Adam/RMSProp
	Weight gate	0.28/0.24
	batch size	32
	epochs	60
	Learning rate	0.001
	Weight decay rate	1×10^−8

In this paper, the feature extraction of text sentiment classification combines multiple extraction methods based on orthogonal projection. To verify the classification performance of the proposed GD-GCN model, we conduct experiments on real datasets. The specific experimental steps are as follows.

Step1: Experimental data acquisition. The part of the original AFOE data is used as experimental data. We add sentiment polarity annotation based on the SemEval dataset to obtain sentiment polarity review data. After denoising, 10000 high-quality text reviews are screened.

Step2: Calculate the embedding matrix of text. This paper selects the Skip-gram structure to train the input text and obtain the word vector and embedding matrix, which can better represent the information between characters or words and context.

Step3: Simplifying global features. Eliminating Interference in Global Features by Orthogonal Projection.

Step4: The aspect-level sentiment classification of comment texts. Features with high discrimination are learned by combining simplified global and local features with a multi-headed attention mechanism. So that the text is more discriminative in aspect-level sentiment classification. Acc calculates the accuracy of text sentiment classification, Pre, Rec and F1 are used as evaluation criteria to compare the text sentiment classification results of multiple models.

4.3. Experimental results

Experimental results on Restaurants14, Laptop14, Restaurants15, Restaurants16 and Twitter are shown in Table 4. The glove model obtained in each set and the best results based on the BERT model are marked in bold and underlined, respectively.

To verify the effect of different percentage pooling on (PD-GCN) model classification, full connection layer, 30^th,50th, 60^th, 70^th, 80^th, 90^th, 100^th, and average pooling participation graphs were set to calculate and compare.

Table 4. Result of experimenting.

		Res14		Lap14		Res15		Res16		Twitter
		Accurcy	F1	Accurcy	F1	Accurcy	F1	Accurcy	F1	Accurcy	F1
Baselines	AdaRNN	60.01	45.10	–	–	–	–	78.43	56.43	66.79	65.78
	PhraseRNN	66.78	59.90	–	–	–	–	–	–	–	–
	LSTM + SynATT + TA + SA	79.89	69.77	70.90	66.57	78.09	58.98	83.89	64.79	70.78	67.89
	Sentic LSTM + TA + SA	79.10	70.78	70.77	67.23	79.89	60.78	83.89	68.23	70.90	67.56
	ContextAvg	71.19	58.98	61.33	53.47	73.67	47.90	79.89	55.56	68.47	65.34
	BERT-pair-QA-M	81.89	73.29	78.89	73.23	81.45	65.22	84.9	66.89	74.89	72.02
	LSTM	74.23	59.89	66.99	59.23	75.56	53.89	80.18	54.78	69.90	66.56
	TD-LSTM	78.82	68.78	71.44	68.90	76.23	58.67	83.10	54.78	70.67	68.56
	ATAE-LSTM	78.89	67.90	68.34	63.78	78.23	62.09	83.09	61.56	68.89	66.78
	MemNet	78.89	65.67	70.89	64.23	77.89	60.01	83.89	57.67	69.78	68.38
	LAN	77.90	66.67	71.89	65.78	77.90	54.99	82.78	57.89	71.78	69.11
	RAM	78.10	68.89	72.90	68.56	79.10	59.67	83.56	62.78	70.90	67.43
	PRET + MULT	79.01	69.67	71.78	67.34	81.89	65.90	85.89	69.67	–	–
Ablated Models	G-GCN(no CPD + GloVe)	75.12	59.91	67.89	59.90	74.12	51.56	78.78	59.78	70.45	68.27
	G-GCN(GloVe)	80.98	70.04	72.34	67.89	78.98	61.67	84.67	68.59	71.78	68.45
	P-GCN(GloVe)	79.10	68.67	71.90	67.45	78.90	60.23	83.15	64.67	71.93	69.90
	G-GCN(no CPD + BERT)	81.89	71.23	77.34	73.10	81.89	61.89	84.56	66.76	73.45	71.67
	G-GCN(BERT)	83.21	75.98	78.23	74.78	82.78	62.89	85.17	66.79	74.89	72.87
	P-GCN(BERT)	82.01	72.67	78.28	75.23	82.01	65.89	86.19	67.98	75.44	73.78
Full Models	GP-GCN(GloVe) + 50th	80.89	70.90	73.90	68.67	79.89	61.78	83.90	64.67	71.67	69.45
	GP-GCN(BERT) + 50th	83.89	75.09	79.90	75.89	83.90	66.89	87.78	72.89	75.90	73.90

Experimental results in Table 5 show that the maximum pooling operation has achieved good classification results, which is also the reason why it can be widely used. However, compared with the maximum pooling method, the percentile pooling method shows better performance. The experimental results are the best when the percentile pooling is set to 50th. Compared with maximum pooling and average pooling increased by 1.25% and 2.31%, respectively, which indicates that percentile pooling is suitable for the grammar-based GCN model. The experiment shows that the percentile combination can improve the measurement value of microflow in our model by 3.55%.

Table 5. Result of different percentile pooling.

Pooling Method	Macro-measure	Micro-measure
Fully Connected Layer	73.93	75.25
Average Pooling	76.26	79.02
30^th percentile	72.15	74.26
50th percentile	80.12	81.37
60th percentile	75.32	78.24
70th percentile	76.92	78.26
80th percentile	75.72	70.25
90th percentile	79.43	80.25
100th percentile	78.28	77.31

4.4. Experimental discussion

To further demonstrate the superiority of the GP-GCN model, this paper explores the accuracy of the training process of each model on the validation set, as shown in Figures 6 and 7.

1. The classification accuracy of this model is higher than other models. AdaRNN and PhraseRNN have similar classification results, but the classification results are relatively low. ContextAvg and TD-LSTM classify text by SoftMax function and bidirectional LSTM, respectively, which is better than RNN, LSTM, and its variants in classification effect. However, the words with significant influence on a given target are not given enough attention, so the classification effect needs to be improved. Compared with LAN and TD-LSTM models, this model not only considers the optimisation of local information extraction but also considers the global dependency between words. Different from PRET + multi model, some interference factors are eliminated by feature projection when dealing with the database to extract the emotional features with higher discrimination. It can be seen from the above experimental results that the GP-GCN model combined with G-GCN and P-GCN can effectively improve the text classification effect.

2. G-GCN is a more effective method in ABSA. GP-GCN has three modules: G-GCN, P-GCN, and CPD-NET. The ablation experiments on the dataset show that compared with P-GCN and G-GCN (no CPD-NET), G-GCN respectively improves the classification performance by 1.19% and 1.32%. It proves that simplified global dependency information between words plays a vital role in ABSA. The training result of the GP-GCN (Bert) model is the best, which is 0.79 percentage points higher than that of P-GCN, 1.88% higher than that of D-GCN, 2.25% higher than that of the P-GCN + D-GCN hybrid (Glove), and 1.93% higher than that of G-GCN (no CPD-NET).

3. The parameter optimisation route of this model is more efficient than other models. As shown in Figures 6 and 7, in the early training (the first ten rounds), the accuracy of each model is low, and the cross-entropy loss is high, but GP-GCN is in a leading position. In the middle of training (10–20 rounds), the accuracy of each model gradually increased. But the accuracy of GP-GCN has tended to be stable. After training (20–50 rounds), the accuracy of other models tends to be stable.

Figure 6. Accuracy curve of calibration set (Baselines).

Figure 7. Accuracy curve of calibration set (Ablated Models and Full Models).

The traditional deep learning model has a significant fluctuation in the early stage of training, and it began to converge in the fifth period. The fluctuation of a single GCN learning model is relatively stable, but the convergence speed is normal. The GP-GCN model proposed in this paper has a fast convergence rate and relatively flat training curve. The training process is stable, the accuracy is high, and it has been in a leading position in the subsequent model training process, reflecting the advantages of the GP-GCN model.

5. Conclusions

To improve the accuracy of text sentiment classification, global features are further simplified from the view of reducing the noise in this paper. An aspect-level sentiment classification method is proposed, named GP-GCN. The local structure and global features obtained by orthogonal feature projection are introduced into the aspect-level sentiment classification. The contribution of this paper mainly includes the following two aspects:

• An aspect-level sentiment classification method is proposed. Text sentimental features are effectively extracted. Different from the traditional methods, it not only weakens the noise interference but also refines the global features. In addition, global and local features are introduced into the aspect-level sentiment classification to improve the accuracy of the results.

• The percentage multi-headed attention mechanism is proved effective. Compared with the traditional attention mechanism, this method is more robust and insensitive to outliers. It can accurately measure the critical output of GCN and obtain better sentence representation of given aspects. The classification accuracy and generalisation performance of the model are improved, which can provide technical support for text sentiment classification.

The experimental results show that the proposed model can accurately and effectively represent all aspects of sentimental tendencies. The calculation speed can be improved by optimising the global features. This model can help users better understand the fine-grained sentimental tendencies in all aspects. In future work, we will fully consider combining local dependence information and knowledge map to learn more discriminative text sentiment features. At the same time, it is considered to be extended to sentiment analysis and sentiment value calculation to obtain better results.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by National Natural Science Foundation of China: [grant number 62076006]; University Synergy Innovation Program of Anhui Province: [grant number GXXT-2021-008]; 2019 Anhui Provincial Natural Science Foundation Project: [grant number 1908085MF189]; Graduate Innovation Foundation Project of Anhui University of Science & Technology: [grant number 2021CX2110]; Anhui Provincial Key R&D Program: [grant number 202004b11020029].