Global heterogeneous graph convolutional network: from coarse to refined land cover and land use segmentation

ABSTRACT

The abundant details embedded in very-high-resolution remote sensing images establish a solid foundation for comprehending the land surface. Simultaneously, as spatial resolution advances, there is a corresponding escalation in the required granularity of land cover and land use (LCLU) categories. The coarse classes identified necessitate further refinement into more detailed categories. For instance, the ‘built-up’ class can be subdivided into specific categories such as squares, stadiums, and airports. These refined LCLU classifications are better equipped to support diverse domains. Nonetheless, most studies simply adopt methods initially designed for coarse LCLU when addressing the challenging refined LCLU segmentation. Few studies have considered the inherent relationships between coarse and refined LCLU, overlooking the potential exploitation of the numerous recently released LCLU products. To better leverage this prior knowledge, we propose the Global Heterogeneous Graph Convolutional Network (GHGCN). The GHGCN introduces a heterogeneous graph and excels in establishing relationships between coarse and refined LCLU, which can extract long-distance dependencies more effectively than convolutional neural networks. Furthermore, this model is performed end-to-end, eliminating the necessity for presegmentation and facilitating training acceleration. GHGCN exhibits competitive performance compared to state-of-the-art models, indicating its effective design in exploiting coarse LCLU data, especially for categories with limited samples. The source code is released at: https://github.com/Liuzhizhiooo/GHGCN.

KEYWORDS:

• Land cover and land use (LCLU)
• semantic segmentation
• heterogeneous graph
• graph convolutional network (GCN)

1. Introduction

1.1. Motivation

Land cover and land use (LCLU) information are essential for comprehending the land surface since they can reflect both physical characteristics and the socioeconomic activities on the ground surface. Therefore, recognizing LCLU supports various research domains, including environment science (Foley et al. 2005; Sterling, Ducharne, and Polcher 2013), ecosystem deterioration (Song et al. 2020; Vitousek et al. 1997), and climate monitoring (Feddema et al. 2005; Pielke 2005). With the increase in the spatial resolution of remote sensing imagery, tremendous demands exist for refined LCLU segmentation. They can better depict the ground surface and benefit land resource management, urban planning, and operation (Zhu et al. 2019; Ge et al. 2019; Zhang, Du, and Zhang 2019).

Most investigations have been conducted to identify coarse LCLU types due to limitations in temporal and spatial resolution (Esch et al. 2017; Gong et al. 2019; Zhu et al. 2022). It can be defined as a coarse category system containing a few classes that have a relatively straightforward correlation with the physical characteristics of the Earth’s surface (Comber, Brunsdon, and Farmer 2012). As a result, coarse LCLU can often be distinguished solely based on spectral features when employing low – or medium-resolution images. In contrast, this refined classification provides a detailed and precise depiction of land surface characteristics and human activities. To illustrate, the coarse category ‘built-up’ can be further subdivided into squares, stadiums, and airports. In essence, refined LCLU entails a detailed and specific classification within the coarse LCLU system, wherein coarse LCLU is systematically dissected into more granular subcategories.

However, a persistent semantic gap (Durand et al. 2007; Zhao, Zhong, and Zhang 2016) between images and a refined LCLU poses a formidable challenge for accurate identification. This gap is defined as the gap between the low-level information extracted from images and the high-level semantic interpretation. Moreover, refined LCLU classification can be divided into two steps: first, basic surface recognition, followed by a spatial reclassification that is performed based on the local spatial pattern of coarse LCLU results (Barnsley and Barr 1996; Wharton 1982). Compared to classifications based solely on images, this two-step approach significantly reduces complexity and aligns more seamlessly with the human cognitive process. Therefore, incorporating coarse LCLU in addition to images is feasible and attractive for bridging this gap (Albert, Rottensteiner, and Heipke 2015). The coarse LCLU information involves essential aspects of the land surface and human activities, facilitating the recognition of intricate patterns in refined LCLU. Additionally, a variety of medium – to high-resolution coarse LCLU classification products, such as FROM-GLC10 (Gong et al. 2019), Dynamic World (Brown et al. 2022), World Cover (Zanaga et al. 2022), Land Use/Land Cover Time Series (Karra et al. 2021), and Sinolc-1 (Li et al. 2023), have been released by the research community. These products provide a crucial data foundation for refined LCLU classification. Consequently, this investigation is significant for enhancing refined LCLU recognition and exploring the utilization of coarse LCLU results.

1.2. Related work

Harnessing the potential of coarse LCLU results to assist in obtaining precise LCLU categories remains a difficult undertaking. The key to this challenge lies in capturing the implicit and intricate relationships between them. Most existing approaches rely on artificially engineered features to express this relationship, and they can be broadly classified into two categories: (1) sliding window-based methods and (2) region-based methods. The former encompasses frequency-based (Eyton 1993; Wharton 1982) and co-occurrence matrix-based (Barnsley and Barr 1996; Van der Kwast et al. 2011; Zhang, Du, and Zhang 2018) techniques, and they necessitate manual specification of the window size. However, these methods impose limitations by assuming rectangular shapes for ground objects, thereby restricting the field of view for recognizing spatial patterns and scene structures. To overcome this, the use of an adaptively generated region has been proposed to improve spectral homogeneity (Lv et al. 2023). In contrast, region-based methods predominantly rely on adjacency graphs (Barnsley and Barr 1997; Barnsley and Barr 2000; Barr and Barnsley 1997; Walde et al. 2014), which enable a more comprehensive description of the spatial and structural relationships among ground objects. Nevertheless, implementing graph-based methods requires a preliminary image segmentation process, which segments an image into numerous parcels. This step is time-consuming and requires human involvement.

Recently, deep learning has achieved remarkable success in remote sensing due to its powerful feature extraction capabilities (Zhu et al. 2017; Zhang, Zhang, and Du 2016), including object detection (Tang et al. 2021; Zhao et al. 2022), image fusion (Sun et al. 2022; Liu et al. 2022; Sun et al. 2023), and semantic segmentation (Jiao et al. 2021; Zhang, Tang, and Zhao 2019). Equipped with a powerful framework for local pattern modeling by end-to-end and hierarchical learning, Convolutional Neural Networks (CNNs) have emerged as the most successful architectures for image segmentation. Prominent examples include the FCN (Long, Shelhamer, and Darrell 2015), DeepLabV3+ (Chen et al. 2018), BiSeNetV2 (Yu et al. 2021), and ConvNeXt (Liu et al. 2022). Moreover, Graph Convolution Networks (GCNs) (Kipf and Welling 2016) are a generalization of CNNs from raster data to graph data, demonstrating impressive performance in capturing intricate relationships. This is attributed to their ability to perform effective long-distance feature extraction. Consequently, CNNs and GCNs can be regarded as prime candidates for improving window-based and region-based methods, respectively. Furthermore, there is a growing trend to integrate both CNNs and GCNs for semantic segmentation tasks, and fusing these two architectures has proven beneficial (Hu et al. 2020; Li and Gupta 2018; Niu et al. 2022). Specifically, CNNs are treated as backbones for local visual feature extraction, upon which subsequent GCNs extract the global semantic features. Therefore, the core of the overall model resides in the GCN module, given its strong association with high-level semantic feature extraction and its pivotal role in comprehending the intricate relationships embedded in images.

Current works often fall short in effectively exploiting the abundant semantic information contained in coarse LCLU maps since they are built on homogeneous graphs (Niu et al. 2022; Su et al. 2022). As shown in Figure 1(a), nodes from different categories are treated equally. Moreover, all the nodes are globally connected, which hinders the aggregation of distinguishable features for image segmentation. To overcome this, the CDGC (Hu et al. 2020) employs a coarse prediction map to construct class-wise graphs, where edges between nodes of different classes are removed. While this approach facilitates the clustering of nodes belonging to the same class, it ignores inter-class relationships. Similarly, a KNN graph is employed, and only nodes that are close to each other in the feature space are connected (Su et al. 2022). Moreover, inter-class and global dependency relations have been further explored (Liu, Schonfeld, and Tang 2021). However, dependency reasoning is performed by group weighted convolutions, which cannot fully express the complicated relationships between visual objects.

Figure 1. Graph comparison of different graph designs. (a) fully connected homogeneous graph; (b) spatially connected homogeneous graph; (3) class-wise connected homogeneous graph; (d) spatially connected heterogeneous graph; (e) fully connected heterogeneous graph.

Compared to globally connected graphs, spatially connected graphs (Hong et al. 2020; Liu et al. 2020; Wan et al. 2019) focus more on capturing the topological relationships among objects, as shown in Figure 1(b). By assuming that spatially adjacent objects tend to share similar properties, the extracted features are less likely to be contaminated by nodes from different categories. A co-occurrence matrix of node types is employed to introduce prior category distributions to enhance the modeling capability of homogeneous graphs (Cui et al. 2021). However, prior knowledge is globally shared and not sufficiently precise to instruct node-level feature extraction. Furthermore, CHeGCN introduces heterogeneous graphs and incorporates node class information in the feature aggregation (Liu et al. 2022), providing a good solution for exploiting coarse LCLU classification, as depicted in Figure 1(d). Nevertheless, spatially connected graphs are applied only after the images are segmented into parcels, which is time-consuming and lacks adaptability. Moreover, because these methods are composed of local connections, they are limited in global feature extraction and cannot be efficiently accelerated by GPUs.

Furthermore, the coarse LCLU result is employed solely in the graph convolution operation, which is implicit and ineffective. Motivated by the incorporation of depth data (Hazirbas et al. 2017), coarse LCLU data are believed to offer valuable geometric cues that can mitigate uncertainty even when identifying refined LCLU categories in homogeneous areas. In summary, existing models exhibit the following limitations: (1) ineffective utilization of coarse LCLU results, (2) insufficient integration of the coarse node category into node feature aggregation, and (3) incomplete training in a fully end-to-end manner due to the need for presegmentation.

1.3. Our contribution

To address the above issues, we propose the Global Heterogeneous Graph Convolutional Neural Network (GHGCN), as shown in Figure 2. The competitive performance of GHGCN can be attributed to the following factors: (1) introducing a heterogeneous graph that effectively incorporates heterogeneous data (images and coarse LCLU data) for semantic segmentation, (2) effective exploiting of coarse LCLU data through three different approaches, and (3) end-to-end training of the GHGCN. Inspired by GloRe (Chen et al. 2019), the graph convolution operations in heterogeneous graphs are realized using CNNs. Moreover, this approach eliminates the need for image presegmentation required for graph construction. These properties make the training process end-to-end and well supported by GPUs. Moreover, the issue of indistinguishable features aggregated from nodes of different categories can be addressed by incorporating heterogeneous graphs, where the edge weights are influenced by the classes of the corresponding nodes.

Figure 2. The architecture of the proposed Global Heterogeneous Graph Convolutional Neural Network (GHGCN). Conv, feat, cls, G-prj, and G-reprj are abbreviations for convolutional neural network, feature map, classifier, graph projection, and graph reprojection, respectively. $X_0$, $X$, $V^l$, $Z$, and $\hat{Y}$ correspond to the symbols used in the formulas presented in the methodology section.

The main contributions of this paper are as follows:

1. The proposed framework can efficiently extract high-level semantic features crucial for the LCLU segmentation task from heterogeneous data. Instead of concatenating the input image and coarse LCLU mask, GHGCN innovatively employs heterogeneous graphs to extract information from both images and coarse LCLU data in a more integrated manner. Moreover, this framework is adaptable for addressing other inference tasks that involve the transitioning from coarse to refined results.

2. We explore various methods for deep mining prior knowledge, including both direct and implicit approaches. Specifically, GHGCN adopts three strategies to incorporate coarse LCLU data: (1) leveraging spatial cues extracted by an additional CNN branch with coarse LCLU as input, enhancing the identification of spatial patterns and guiding the spatial constraint of the coarse LCLU data, (2) improving the construction of long-distance relationships by globally creating nodes and aggregating features at the node level, and (3) applying category constraints from the class-wise classifier, significantly simplifying reclassification as coarse LCLU categories are ensured in advance.

3. The proposed GHGCN efficiently performs graph convolution in heterogeneous graphs and extracts long-distance relationships effectively. The graph-based GHGCN model is achieved in an end-to-end fashion and can be accelerated well by GPUs. Specifically, the graph nodes are dynamically generated in a data-driven manner without the requirement for presegmentation, enabling flexible adjustments of node creation during the training process. Moreover, the feature aggregation among heterogeneous nodes is implemented by a CNN, which can be supported by GPUs well.

In the dataset with various-grained labels, our model outperforms the current state-of-the-art segmentation algorithms. Utilizing local patterns extracted from coarse LCLU data, coupled with the relationships established through heterogeneous graphs, significantly improves the performance of fine-grained LCLU classification. Remarkably, the benefits derived from the coarse LCLU results become more evident as the classification system becomes more precise.

2. Methodology

In this section, the architecture of the proposed GHGCN is described in detail. It follows an encoder-decoder structure, as shown in Figure 2. The encoder can be divided into two components: (1) the CNN module and (2) the GCN module. The CNN module contains two CNN branches. The upper branch extracts geometric cues from coarse LCLU, while the lower branch focuses more on capturing local features from images. The GCN module applies graph convolution to heterogeneous graphs to establish node-level long-distance relationships. Additionally, the decoder is a class-wise classifier composed of multiple classifiers, where pixels of the same coarse LCLU class share a classifier.

2.1. CNN module

The selection of CNNs as the backbone network for semantic segmentation is prevalent owing to their ability to proficiently capture local patterns. Since the pretrained model is recommended given limited samples, ResNet (He et al. 2016) pretrained with ImageNet (Krizhevsky, Sutskever, and Hinton 2012) is adopted as the CNN backbone. The CNN module consists of two CNN branches, both of which remove the last four convolutional layers of ResNet-50, specifically the conv5_x layer and the classifier. This adjustment effectively controls the number of model parameters within a desirable range. This reduction significantly decreases the number of parameters in our model and enhances its resistance to overfitting.

Since the pretrained backbone has already provided abundant local features prepared for high-level semantic feature extraction, the backbone network in the CNN module is frozen except for the first convolutional layer in the upper CNN branch because the input coarse LCLU data are converted using one-hot encoding and the input channel of the first convolutional layer is adjusted to 5. Freezing the CNN backbone can reduce the time consumed in training and dramatically improve the resistance to overfitting. Moreover, the primary focus of our research lies in capturing, rather than extracting, abstract semantic information from local features. Subsequently, the outputs from the two branches are concatenated along the feature dimension. In practice, before being fed into the GCN module and the feature fusion module, the dimensions of the stacked features are reduced from 2048 to 512 by a $1\times 1$ convolutional layer.

2.2. GCN module

The introduction to the GCN module contains three components: (1) the basic concepts of the GCN, (2) the implementation of the GCN using a CNN, and (3) the global heterogeneous GCN. A graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is determined by vertices $\mathcal{V}$ and edges $\mathcal{E}$. In practice, the vertex set $\mathcal{V}$ is represented by a vertex feature matrix $V\in {\mathbb{R}}^{N\times C}$, and the edge set $\mathcal{E}$ is denoted by an adjacency matrix $A\in {\mathbb{R}}^{N\times N}$, where $N$ and $C$ represent the number of nodes and the dimensions of the node features, respectively. The $i$-th row of $V$ corresponds to the feature vector ${\mathbf{v}}_i\in {\mathbb{R}}^C$ of node $i$. Node features are obtained through a process called graph projection, where pixels belonging to the same superpixel are assigned to their corresponding node. In graph projection, the grid-structured feature maps $X_0\in {\mathbb{R}}^{H\times W\times C}$ are transformed to a graph-structured data representation $V$, where node features are calculated as the average signatures of the pixels involved. The edge weight $A_{\mathrm{ij}}$ represents the strength of the connection between node $i$ and node $j$, with higher values indicating a closer relationship between these two nodes.

Each pixel as a node often leads to a large graph with intractable computations. To address this issue, many approaches adopt the Simple Linear Iterative Clustering (SLIC) (Achanta et al. 2012) algorithm to generate superpixels that serve as nodes. In GHGCN, the nodes are generated automatically and can be adjusted adaptively during the training process. Specifically, a learnable projection matrix $B\in {\mathbb{R}}^{N\times L}$ is generated through a 1D convolutional layer, with the reshaped output of the CNN module $X\in {\mathbb{R}}^{L\times C}$ as its input. Here, $L=H\times W$, where $H$ and $W$ represent the height and width of the feature map, respectively. As a result, the projection function $V=f(X)$, as depicted in Equation 1(1) $V=f(X)=\mathrm{BX}=\mathrm{Conv}1\mathrm{D}(X)X$(1) , is formulated to project the feature maps from the coordinate space to the interaction space, enabling more effective reasoning for long-distance relationships. (1) $V=f(X)=\mathrm{BX}=\mathrm{Conv}1\mathrm{D}(X)X$(1) After acquiring the node feature matrix $V$, the forward propagation process of a GCN layer can be expressed as: (2) $V^{l+1}=\sigma (\tilde{A}{V}^l{W}^l)$(2) where $V^l\in {\mathbb{R}}^{N\times C^l}$ and $V^{l+1}\in {\mathbb{R}}^{N\times C^{l+1}}$ represent the input and output node feature matrices of the $l$-th layer, respectively. The feature dimension of $V^{l+1}$ is determined by the learnable linear transformation matrix $W^l\in {\mathbb{R}}^{C^l\times C^{l+1}}$. The matrix $\tilde{A}$ denotes a symmetric normalized adjacency matrix computed based on $A$. The activation function $\sigma (\cdot )$ is chosen to be Rectified Linear Units (ReLU) (Nair and Hinton 2010). The feature aggregation process in a GCN layer can be divided into two steps: (1) information diffusion and (2) state update (Chen et al. 2019). The former involves aggregating features across nodes, which corresponds to $\tilde{A}V$. The latter is used for transforming the feature dimensions, which corresponds to $(\tilde{A}V)W$. Based on the above analysis, graph convolution can be achieved by employing two 1D convolution layers in different directions, namely channel-wise and node-wise layers. Therefore, for fully connected graphs, Equation 2(2) $V^{l+1}=\sigma (\tilde{A}{V}^l{W}^l)$(2) can be rewritten as follows: (3) $V^{l+1}=\mathrm{Conv}1\mathrm{D}(\mathrm{Conv}1\mathrm{D}{(V^l)}^T{)}^T$(3) In heterogeneous graphs, considering the node type becomes crucial for feature aggregation. Assigning node categories occurs during graph projection, which transforms feature maps from the coordinate space to the interaction space. The determination of node labels is based on the most frequently occurring land cover category among the associated pixels. In contrast to SLIC, the nodes in our model are generated using a data-driven convolutional layer. However, during the early training phase, there is no guarantee that the pixels within a node will be homogeneous. To mitigate this issue, pixels within a node that do not belong to the coarse LCLU class of the node are excluded. This can greatly decrease the variance in pixels located in a node. In the information diffusion step, the consideration of meta-paths, denoted as $P$, is introduced. A meta-path $P$ describes the relationship between node $i$ and node $j$, where the relationship is determined based on the coarse LCLU labels of the node pair $(i,j)$. In fact, the meta-path can be interpreted as a specific type of edge in heterogeneous graphs. As depicted in Figure 2, the edge $e_{\mathrm{ij}}$ is visualized with a gradient color, which is determined by the categories associated with nodes $i$ and $j$. It is worth noting that the meta-path of node pair $(i,j)$ is the same as that of node pair $(j,i)$ since the graphs are undirected. Consequently, the total number of meta-path types is ${\displaystyle \frac{m(m+1)}{2}}$, where $m$ denotes the number of coarse LCLU classes.

After the information diffusion step, the node features ${\hat{V}}^l$ in heterogeneous graphs can be formulated as: (4) ${\hat{\mathbf{v}}}_i^l=\sum _j(a_P\mathrm{Conv}1{\mathrm{D}}_P({\mathbf{v}}_j^l)+b_P)$(4) where ${\mathbf{v}}_j^l\in {\mathbb{R}}^N$ represents the $j$-th column of $V^l$, $\mathrm{Conv}1{\mathrm{D}}_P(\cdot )$ denotes the 1-D convolutional kernel associated with meta-path $P$, $a_P$ and $b_P$ are scalar parameters used to adjust the importance of meta-path $P$, and ${\hat{\mathbf{v}}}_i^l\in {\mathbb{R}}^N$ represents the $i$-th column of ${\hat{V}}^l$. The category of a node pair (i.e. meta-path $P$) influences the feature aggregation of nodes in heterogeneous graphs via two approaches: the meta-path based transformation function $\mathrm{Conv}1{\mathrm{D}}_P(\cdot )$ and meta-path based aggregation coefficients $a_P$ and $b_P$. Finally, the computation of the GCN in heterogeneous graphs can be obtained using the following equation: (5) $V^{l+1}=\mathrm{Conv}1\mathrm{D}({\hat{V}}^l)$(5)

2.3. Feature fusion and classifier

The representations of VHR images extracted from different network architectures exhibit significant variations. CNNs primarily emphasize spectral-spatial features, while GCNs focus on capturing long-distance relationships among nodes. However, relying solely on features provided by a single architecture often limits the ability to achieve optimal results. Consequently, the GCN module is enhanced by incorporating CNN features to achieve better performance. Additionally, the residual nature of the GCN module facilitates gradient propagation and enables easier integration with various CNN backbones. Prior to feature fusion, it is necessary to map the node-level outputs of the GCN module back to the pixel-level, which involves an inverse process of graph projection known as graph reprojection. To accomplish this, we reuse the projection matrix $B$, enhancing computational efficiency while ensuring minimal performance loss. The underlying assumption is that the adjacency matrix $B$ tends to be a symmetric orthogonal matrix that satisfies $B^{T}B=E$. This assumption enables us to achieve no loss reversion after the graph projection and graph reprojection of feature maps, i.e. $B^{T}V=B^T\mathrm{BX}=X$. Consequently, the graph reprojection can be formulated as follows: (6) $Z=B^{T}V$(6) where $Z\in {\mathbb{R}}^{C\times L}$ indicates the reverse projected feature maps. During graph reprojection, the node features are assigned to the pixels located within that node. Subsequently, element-wise addition is performed between the reprojected GCN feature maps and the CNN feature maps.

The output $Y\in {\mathbb{R}}^L$ is calculated after feeding the fused feature maps into the class-wise classifier, which contains $m$ individual classifiers. Each classifier comprises a $1\times 1$ convolutional layer, a softmax function, and an $\arg \max$ operation applied to the feature dimension, as shown in Equation 7(7) $Y=\underset{C}{\arg \max }(\mathrm{softmax}(\mathrm{Conv}1\mathrm{D}(X+Z)))$(7) . This process involves assigning pixels from different categories to their respective classifiers, essentially performing a reclassification of the coarse LCLU data. Finally, the pixel-level segmentation results $\hat{Y}\in {\mathbb{R}}^{H\times W}$ are obtained by reshaping the output $Y\in {\mathbb{R}}^L$. (7) $Y=\underset{C}{\arg \max }(\mathrm{softmax}(\mathrm{Conv}1\mathrm{D}(X+Z)))$(7)

3. Experiments

In the experiments, our proposed GHGCN model is evaluated on the generated dataset. Eleven state-of-the-art deep learning methods are compared against our method: FCN (Long, Shelhamer, and Darrell 2015), DeepLabV3+ (Chen et al. 2018), UNet++ (Zhou et al. 2018), DANet (Fu et al. 2019), OCNet (Yuan et al. 2018), GloRe (Chen et al. 2019), DGCNet (Zhang et al. 2019), BiSeNetV2 (Yu et al. 2021), SegFormer (Xie et al. 2021), ConvNeXt (Liu et al. 2022), and SegNeXt (Guo et al. 2022). These models encompass a diverse range of architectures, including CNNs, GCNs, and attention networks. Three metrics, namely, the overall accuracy (OA), kappa coefficient, and mean intersection over union (mIoU), are employed to evaluate the performance of these models. Finally, ablation experiments and visualization results are conducted to demonstrate the benefits of coarse LCLU data and our model design.

3.1. Dataset

We generated datasets based on the Gaofen Image Dataset (GID) (Tong et al. 2020), the GID-15 dataset (Yang et al. 2022), and the Five-Billion-Pixels dataset (Tong, Xia, and Zhu 2023). The GID contains 150 Gaofen-2 (GF-2) images and the associated manual annotations with 5/15 categories. The Five-Billion-Pixel dataset is further completely labeled with 24 classes. For clarity, we refer to the category systems with 5, 15, and 24 types as GID5, GID15, and GID24, respectively. These three category systems are determined with reference to the Chinese Land Use Classification Criteria (GB/T 21010-2017). Our task is to predict the refined LCLU (GID15/GID24) with images and coarse LCLU (GID5) data.

GID5 comprises five primary classes: built-up, forest, farmland, meadow, and water. Unlabeled areas in GID5 are miscellaneous or unclear places that are highly challenging to annotate. This is the same for both GID15 and GID24. The GID15 expands upon the GID5 and includes 15 categories: industrial land, urban residential, rural residential, traffic land, garden land, arbor forest, shrub land, paddy field, irrigated land, dry cropland, natural meadow, artificial meadow, river, lake, and pond. Building upon GID15, GID24 possesses a more complete category system. It retains most of the classes from GID15 while further refining some categories. For example, traffic land is further divided into roads, overpasses, rails, and airports. The hierarchical classification system is described in Figure 3. Notably, snow and bare land in GID24 are excluded due to the absence of annotations for GID5 and GID15.

Figure 3. The hierarchical category systems of GID5, GID15, and GID24 and the boxes are rendered with the corresponding colors. The abbreviations for GID5 are: Farm – farmland, Buil – build-up, Mead – meadow, Wate – water, Fore – forest, Unla – unlabeled area. The abbreviations for GID15 are: Urba – urban residential, Rura – rural residential, Indu – industrial area, Traf – traffic land, Gard – garden land, Arbo – arbor forest, Shru – shrub forest, Padd – paddy field, Irri – irrigated field, Dryc – dry cropland, Natu – natural meadow, Arti – artificial meadow, Rive – river. The abbreviations not mentioned for GID24 are as follows: Stad – stadium, Squa – square, Over – overpass, Rail – railway station, Airp – airport, Fish – fish pond, and Bare – bare land.

In our task, GID5 represents the coarse LCLU types, while GID15 and GID24 represent refined LCLU classes. As depicted in Figure 4, nine GF-2 images are selected. The selection follows specific criteria to ensure dataset diversity: (1) inclusion of all the categories in GID24 (except snow and bare land), (2) considerable distinction between coarse land cover labels and refined land use annotations, and (3) sufficient spatial and temporal variations to ensure generalizability across different geographical contexts and seasons. Table 1 lists the basic information about these images. The GF-2 MSI images provide a spatial size of $6800\times 7200$ pixels, a spatial resolution of 4 m, and four bands: blue, green, red, and near-infrared. The first three visible bands are selected. These images, along with the corresponding annotation masks, are simultaneously cropped into patches of $256\times 256$ pixels. Patches with too many unlabeled areas are discarded, yielding a total of 5218 patches. The class distributions of these patches are listed in Table 2. The statistics indicate a category imbalance in this dataset, while each of the eight categories, including airports, parks, and stadiums, constitute less than 1%. This category imbalance greatly challenges the ability of models to learn from classes with small sample sizes. These patches are then randomly divided into training, validation, and test sets at a nearly 1:1:2 ratio. Moreover, the category distributions across these three sets remain similar.

Figure 4. Nine Gaofen-2 (GF-2) images and their annotations on GID5, GID15, and GID24. The color systems of GID5, GID15, and GID24 are displayed at the bottom.

Table 1. The basic information about the selected GF-2 images, including the product name, date, and longitude and latitude of the left top point of the images.

Product Name	Date	Lon	Lat
GF2_PMS2__L1A0001471436	16/03/16	120.201	30.374
GF2_PMS2__L1A0001448894	15/02/17	116.300	39.968
GF2_PMS1__L1A0001395956	16/02/07	114.000	30.246
GF2_PMS2__L1A0001015594	15/09/02	113.989	30.908
GF2_PMS1__L1A0001293546	15/12/19	113.363	23.096
GF2_PMS2__L1A0001396040	16/02/07	114.186	30.021
GF2_PMS1__L1A0001660727	16/06/23	116.527	28.455
GF2_PMS1__L1A0001680858	16/07/04	120.895	49.236
GF2_PMS2__L1A0001787564	16/08/27	116.535	39.979

Table 2. The category distribution of the generated dataset.

Category	Indu	Urba	Rura	Road	Padd	Irri
Percent (%)	7.06	13.41	5.32	6.85	1.84	28.07
Category	Dryc	Gard	Arbo	Shru	Natu	Arti
Percent (%)	0.36	2.07	13.43	0.27	2.66	1.05
Category	Rive	Lake	Pond	Stad	Squa	Over
Percent (%)	7.61	4.20	1.34	0.05	0.05	0.62
Category	Rail	Airp	Park	Fish	Snow	Bare
Percent (%)	0.18	0.19	0.13	3.26	0	0

3.2. Hyperparameter configuration

The output channel of each branch in the CNN module is 1024 after removing the last four convolutional layers, specifically the conv5_x layer and the classifier. Before feeding to the GCN module, the dimension of the concatenated feature is reduced to 512. Moreover, there is 1 layer with 64 nodes in the GCN module, followed by a batch normalization (Ioffe and Szegedy 2015) layer and a ReLU activation layer. We use the cross-entropy loss and Adam optimizer (Kingma and Ba 2014) to train the GHGCN algorithm for 50 epochs. We employ a ‘poly’ learning rate policy: $\mathrm{lr}=l{r}_{\mathrm{init}}\ast (1-{(\mathrm{iter}/\mathrm{epochs})}^{\mathrm{power}})$, where the power equals 0.9 and the initial learning rate is 0.001 with a batch size of 8. All hyperparameters are determined by the performance in the validation dataset, and the same hyperparameter configuration is used for both qualitative and quantitative results. Our code is implemented in Python-3.8 and PyTorch-1.10.0.

All compared models are equipped with their corresponding pretrained model parameters, except for UNet++. To ensure a fair comparison, the majority of the selected state-of-the-art models employ the pretrained ResNet-50, with no convolutional layers removed. This ensures that the channel dimension of the feature maps in the encoders of these compared models is 2048, as specified by the CNN module in GHGCN. Moreover, the classifiers of all the compared models are replaced by class-wise classifiers to ensure fairness in the comparison. Furthermore, the compared models and our GHGCN model share the same hyperparameter setup. It is worth noting that all the quantitative metrics are obtained by the model with the maximum OA index on the validation dataset.

3.3. Comparison of classification performance

Table 3 displays the segmentation scores (%) of GHGCN and seven state-of-the-art models in terms of OA, Kappa, and mIoU. Due to the imbalanced GID category distribution, the mIoU metric is considered a more comprehensive evaluation metric for model performance comparisons. Consequently, the subsequent analysis will primarily focus on the mIoU.

Table 3. Quantitative comparison of different methods.

Model	Params(M)	FLOPs(G)	GID15			GID24
			OA	Kappa	mIoU	OA	Kappa	mIoU
FCN	33.00	34.77	87.81	85.72	68.46	86.71	84.40	51.22
DeepLabV3+	40.47	46.29	88.41	86.39	70.71	86.48	84.19	51.64
UNet++	9.17	35.13	87.74	85.61	66.31	86.44	84.10	47.18
DANet	47.49	49.61	86.25	83.86	66.05	86.56	84.27	50.46
OCNet	33.92	35.85	87.88	85.75	69.66	86.49	84.21	51.16
GloRe	11.62	12.84	88.18	86.11	71.17	86.28	84.00	51.29
DGCNet	48.71	50.18	87.77	85.64	68.21	87.05	84.81	52.14
BiSeNetV2	3.45	3.28	89.50	87.65	72.60	87.74	85.65	53.13
SegFormer	13.71	3.42	87.24	85.02	69.52	86.35	83.99	52.96
ConvNeXt	37.06	19.25	89.35	87.49	72.07	88.75	86.81	54.90
SegNeXt	3.82	1.25	87.59	85.44	70.63	85.48	82.99	50.28
GHGCN	29.13	31.82	90.15	88.44	75.45	89.28	87.45	60.69

GHGCN achieves the highest OA, Kappa, and mIoU on both the GID15 and GID24 datasets, surpassing the mIoU of the second-ranked model by approximately 2.8% and 5.8%, respectively. This highlights its effectiveness in capturing intricate semantic features and providing accurate segmentation results. Moreover, GHGCN demonstrates competitive performance with a moderate number of parameters (29.13 M) and computational cost (31.82 G FLOPs), indicating efficient resource utilization compared to some other models with higher complexity. Given the challenge of achieving both lightweight parameters and high accuracy, the GHGCN is a promising choice due to its balanced trade-off between model complexity and performance. Furthermore, the competitive performance of GHGCN on the test dataset demonstrates its ability to generalize across diverse geographical contexts and seasons.

Overall, the performance of the compared models exhibits the following trends: (1) state-of-the-art CNNs, exemplified by BiSeNetV2, outperform state-of-the-art attention models such as SegFormer. In succession, attention models surpass graph convolutional models, represented by GloRe, and these models, in descending hierarchies, outperform conventional convolutional models (e.g. DeepLabV3+) and attention models (e.g. DANet). This trend highlights the advantages of GCN models over traditional CNN models in long-distance relationship capture and emphasizes the potential for improvement in graph models compared to state-of-the-art convolutional models. (2) Models with fewer parameters achieve higher accuracy on the GID15 dataset, characterized by fewer categories, while models with a larger parameter count demonstrate superior performance on the more challenging GID24 dataset. For instance, BiSeNetV2 excels on GID15, while the ConvNeXt model exhibits superior performance on GID24. This observation suggests that lightweight models are better suited for GID15, where they are less prone to overfitting under relatively limited samples.

Specifically, we find that DeepLabV3 + outperforms the FCN and UNet++, which can be attributed to the effectiveness of the Atrous Spatial Pyramid Pooling (ASPP) design. Moreover, UNet++ performs worst in CNNs since it is trained from scratch, and the disparity between UNet++ and the FCN is further amplified in GID24 dataset. This suggests that employing pre-trained models in scenarios with relatively limited samples can provide considerable benefits. Furthermore, we notice that DANet performs even worse than UNet++, which reveals that the attention networks may not be as competitive as CNNs for this task.

Among the GCN models, GloRe surpassed DGCNet on the GID15 dataset but demonstrated slightly inferior performance on GID24 compared to DGCNet, which can be attributed to its smaller parameter count. This highlights the significance of proper graph construction in achieving image segmentation. DGCNet ineffectively generates nodes through pooling, where a node represents a rectangular area. In contrast, GloRe can dynamically adjust the projection matrix, allowing nodes to have arbitrary shapes. This adaptability proves advantageous in effectively capturing long-distance relationships. The GHGCN model further improves the edge weight calculation by incorporating coarse LCLU results, leading to the highest mIoU. Notably, the advantage of GHGCN over GloRe is amplified in GID24, proving the effectiveness of our proposed model.

Furthermore, Table 4 presents the IoU values for all the categories in GID24. GHGCN achieved the optimal or near-optimal mIoU in most categories, particularly those characterized by limited samples (comprising less than 1% of the total). For instance, GHGCN achieves significant improvements in the airport category (30% increase), the park category (11% increase), the overpass category (10% increase), and the shrub forest category (7.5% increase). Moreover, in homogeneous built-up areas, the improvements in GloRe in industrial areas (2.56% increase), urban residential areas (1.3% increase), rural residential areas (1.85% increase), and roads (2.06% increase) are also not marginal.

Table 4. Quantitative comparison of different methods in terms of mIoU in GID24.

Model	Indu	Urba	Rura	Road	Padd	Irri	Dryc	Gard	Arbo	Shru	Natu
FCN	61.40	72.18	64.03	63.91	53.78	93.97	59.54	38.07	89.85	35.89	89.83
DeepLabV3+	59.66	72.14	65.25	61.52	55.30	95.01	70.23	52.00	90.28	32.80	92.55
UNet++	62.21	71.33	65.81	65.61	52.50	96.40	36.34	40.70	89.19	29.81	91.34
DANet	61.84	71.62	63.49	63.34	52.25	93.01	44.01	43.25	90.00	35.93	88.04
OCNet	62.51	73.04	64.88	62.10	52.90	92.64	73.25	54.11	91.76	20.61	89.68
GloRe	61.97	72.00	63.99	63.25	49.74	92.39	69.89	56.53	89.54	17.72	90.92
DGCNet	62.44	72.74	64.67	64.11	54.54	94.14	52.68	36.68	89.21	33.07	90.70
BiSeNetV2	63.67	73.02	66.19	67.27	62.84	96.76	45.34	51.33	92.36	33.48	91.81
SegFormer	61.50	73.48	66.45	65.74	59.86	96.10	81.98	38.50	88.56	41.66	91.60
ConvNeXt	62.28	73.26	65.27	66.08	68.57	96.56	75.06	49.45	91.73	36.55	90.32
SegNeXt	61.55	71.63	63.10	56.29	45.70	93.12	79.92	53.47	90.99	33.68	89.12
GHGCN	64.53	73.30	65.84	65.31	77.14	97.22	72.33	63.83	94.39	49.17	93.60
	Arti	Rive	Lake	Pond	Stad	Squa	Over	Rail	Airp	Park	Fish
FCN	75.94	80.54	57.66	22.60	3.51	0.99	38.56	12.59	29.51	5.51	76.91
DeepLabV3+	81.79	76.01	49.43	17.04	3.79	0.00	24.36	13.15	23.71	16.64	83.36
UNet++	79.44	75.08	48.71	27.14	0.00	0.00	13.11	0.67	17.05	0.29	75.16
DANet	72.51	81.74	62.78	27.00	2.81	1.11	40.28	11.89	23.23	3.32	76.65
OCNet	76.52	75.46	51.34	28.08	0.07	0.00	33.01	9.14	15.55	16.39	82.58
GloRe	79.07	77.28	54.92	26.61	0.48	0.39	37.38	8.34	17.42	13.36	85.18
DGCNet	77.51	79.90	60.36	36.43	7.03	2.57	40.97	15.12	20.39	15.52	76.38
BiSeNetV2	79.94	75.50	49.19	13.49	26.37	0.64	48.68	1.01	21.99	21.99	85.94
SegFormer	80.44	67.53	45.74	19.90	3.46	0.48	42.60	9.64	24.92	23.67	81.28
ConvNeXt	77.97	84.51	67.05	37.94	4.08	0.00	42.78	9.69	14.56	9.80	84.24
SegNeXt	77.40	68.27	45.44	31.59	0.07	0.00	25.90	9.18	14.50	13.69	81.62
GHGCN	84.25	82.68	57.30	28.61	22.41	0.08	58.65	12.31	49.54	34.60	88.18

We further display the qualitative segmentation results of the nine test samples in Figure 5. The comparison includes representative CNNs (ConvNeXt), attention models (SegFormer), and GCNs (DGCNet). Overall, the boundaries between different categories are more accurate and smoother in GHGCN and ConvNeXt. For instance, roads and water bodies are more complete in samples 2, 4, and 5. With its shallower convolutional structure, the CNN module retains more spatial details, which facilitates the recognition of narrow objects. Moreover, the use of coarse LCLU data as a spatial constraint among different coarse LCLU categories is beneficial. Even in homogeneous areas, GHGCN shows superiority. The classification results tend to be consistent in homogeneous areas, which is due to long-distance node-level feature aggregation. Moreover, the control of each node’s corresponding area to belong to the same category plays a crucial role in preventing feature confusion among different classes, especially when nodes are dynamically generated.

Figure 5. Visualization comparison of the prediction results obtained by DeepLabV3 (Dplb3), OCNet, GloRe, and GHGCN. The first three rows indicate the images of the RGB bands, the GID5 labels, and the GID24 labels. The last four rows represent the predictions of these four models. Areas with large differences are highlighted with white boxes.

As mentioned above, GHGCN shows significant superiority, especially in categories with limited samples. Specifically, these nine columns in Figure 5 represent the distinctions of these models in segmenting airports, parks, overpasses, shrub forests, paddy fields, stadiums, garden lands, fish ponds, and industrial areas. Additionally, it is observed that both DGCNet and SegFormer achieve similar results and underperform ConvNeXt, which is consistent with the quantitative evaluation.

3.4. Ablation experiments

The improvement in the GHGCN performance can be attributed to three key designs: (1) the integration of the CNN branch with coarse LCLU data as input (+LCLUCNN), (2) the employment of the GCN module (+GCN), and (3) the incorporation of the meta-path in feature aggregation (+Meta-path). To demonstrate the effectiveness of these factors, we conducted additional experiments as presented in Table 5. We developed five comparable models and tested their performance on both the GID15 and GID24 datasets. These five models share the same decoder and training configuration but vary in encoder design. The first model’s encoder comprises a CNN branch with images as input. Building upon this, two additional models were created, each incorporating an extra CNN branch with a coarse LCLU mask as input (+LCLUCNN) and the GCN module. These two models were designed to analyze the individual impacts of the dual-branch CNN and the GCN module. The fourth model combines both. Finally, the meta-path is further incorporated into the GCN module, which is actually the proposed GHGCN.

Table 5. Quantitative comparison of ablation models.

	+LCLUCNN	+GCN	+Meta-path	mIoU	$\mathrm{\Delta }$mIoU
GID15				66.66	–
	✓			68.37	+1.71
		✓		71.99	+2.62
	✓	✓		73.53	+1.54
	✓	✓	✓	75.45	+1.92
GID24				51.86	–
	✓			54.84	+2.98
		✓		55.82	+0.98
	✓	✓		56.98	+1.16
	✓	✓	✓	60.69	+3.71

Through the ablation study, we observed that all the three design factors contribute significantly to the improved performance of our GHGCN model. When none of these factors are applied, our model is equivalent to the FCN model with the last four convolutional layers of ResNet-50 removed. Compared to the performance of FCN in Table 3, the removal of these layers in our model is advantageous because it preserves more spatial details. When only the LCLUCNN factor is added, our model becomes a double-branch FCN model. This results in improvements of 1.71% and 2.98% in the mIoU on GID15 and GID24, respectively. Moreover, when only the GCN factor is employed, our model degrades to GLoRe. The increase in the mIoU over the FCN demonstrates the effectiveness of the GCN module. Notably, we find that the GCN factor has a more significant impact than the LCLUCNN factor does, which additionally improves GID15 and GID24 by 2.62% and 0.98%, respectively. Furthermore, when combining both, the resulting model further promotes the mIoU metric. Finally, introducing a meta-path leads to additional performance enhancements based on the former. These findings demonstrate the positive effects of all these designs on the use of coarse LCLU labels. Furthermore, we observed that the impact of the additional LCLUCNN branch is more pronounced in GID24. This may be attributed to the increased significance of the spatial constraints introduced by the additional branch when dealing with more complex categories. Moreover, the benefits of the meta-path factor have also been enhanced in GID24, as it optimizes the aggregation of node features.

3.5. Visualization analysis

In this section, we visualized the following components to gain a better understanding of the mechanism of our GHGCN model: (1) the CNN features generated from the coarse LCLU data, (2) the automatically created nodes in the heterogeneous graph, and (3) the node feature distributions before and after the GCN layer. These visualizations can assist in comprehending how GHGCN efficiently utilizes coarse LCLU data.

We visualize the feature maps of the CNN branch with coarse LCLU data as input to illustrate how these features work in Figure 6. We select 6 representative features of three layers from samples 1 and 3 as examples. The brighter the pixel is, the stronger the feature response. Overall, shallower layer features tend to focus more on edge information, which is beneficial for narrow object segmentation and can provide more precise boundaries. Deeper layers tend to generate spatial cues that concentrate more on topological positions. Specifically, the features of the conv4_x layer in sample 1 are concentrated on different topological positions, including the two sides (the 2nd column), the top (the 3rd column), and the center (the 4th column) of the built-up mask. Correspondingly, the features of the conv4_x layer in sample 3 are concentrated on the left side (the 2nd column), the top (the 1st column), and the center (the 6th column) of the built-up mask. The spatial cues facilitate segmentation since the topological position can benefit the recognition of refined LCLU. For example, pixels located in the center of the built-up mask are more likely to belong to the urban residential area than those located on the edges are.

Figure 6. Visualization of CNN feature maps with coarse LCLU data as input. Each row includes 6 typical features of the same layer. Each image is normalized by its minimum and maximum values, and brighter pixels indicate higher values. The spatial cues manifest from the edge information to the topological locations as the layer deepens.

To present the adaptively generated nodes, the learned projection weights of the nodes are shown in Figure 7. After the weights are produced, the node features are determined by the weighted sum of the pixel features. The projection weights can be viewed as a special kind of presegmentation, where they are dynamically created without any spatial restriction and can be any value rather than the average sum. Specifically, we selected 4 nodes from samples 1, 2, and 5 in Figure 5 for illustrative examples. The automatically generated nodes can achieve similar and even more flexible results than presegmentation. (1) Each node corresponds to a specific area. For example, node 1 in the first row represents the airport area, node 3 in the second row indicates the park area, and node 4 in the third row corresponds to the water body. (2) The generated nodes can represent areas on a coarser scale. For instance, node 1 in the second row represents the built-up areas, including the industrial, road, overpass, and urban residential areas. (3) The generated nodes exceed the limitation of representing single local areas, which is more flexible than presegmentation-based node production (e.g. SLIC). In the last row, nodes 1 and 2 include two isolated areas that belong to the same categories. These properties offer greater adaptability in capturing global relationships between nodes. Moreover, we find that the projection weights of the same area can be either positive or negative, such as for node 1 and node 2 in the last row. In other words, the features of different nodes pointing to the same region may have contrasting values.

Figure 7. Visualization of the adaptively generated nodes and the nodes segmented by Simple Linear Iterative Clustering (SLIC). Each row contains an image, the corresponding GID24 label, nodes of SLIC, and the learned projection weights of 4 nodes. The yellow line in the third column indicates the boundaries of different nodes. In columns four to seven, the color red represents positive values, and the color blue denotes negative values. The darker the color is, the greater the value. The adaptively generated nodes can yield comparable and even more flexible results than the presegmentation output.

Furthermore, we generated the nodes via SLIC for comparison, as shown in the third column in Figure 7. The parameters for SLIC, including segment number, compactness, and sigma, are set to 128, 10, and 2, respectively. Since SLIC segments images based on spatial-spectral features, the generated nodes possess three characteristics: (1) they are locally homogeneous, with spatially nonadjacent pixels not assigned to the same node, (2) there is no intersection between different node, and (3) the pixels within a node are treated equally. In contrast, the dynamically generated nodes of the GHGCN break the constraint of spatial adjacency. Moreover, node features are calculated by the sum of pixel features with adaptively determined weights. Therefore, compared to the pre-processing SLIC method, which requires manual determination of segmentation parameters, the end-to-end and adaptive approach of GHGCN is more flexible and powerful.

The feature distributions before and after the GCN layer are visualized to demonstrate how the GHGCN model improves the separability of different categories. Principal component analysis (PCA) was used to reduce the feature dimension to two in Samples 5 and 4 for illustration. Typically, the ideal feature distribution should ensure that the clusters of different categories are separable: (1) the points of different classes (clusters) should be far apart, and (2) the points of the same class should be tightly distributed. As shown in Figure 8, after the GCN layer, we observe a noticeable improvement in the separability of the feature distributions. The points belonging to the same category now form a tighter cluster, whereas points from different classes are distributed farther apart. The tighter intra-class distribution and increased inter-class separation make it easier to distinguish among different categories and contribute significantly to the improved performance of our model. For example, in the first row, the mixed clusters are split apart after the application of the GCN layer, allowing for improved discrimination of nodes with similar CNN features. In the second row, the clusters become tighter and more distinguishable, especially for the water and forest regions. This finding demonstrated the positive effects of the GCN module.

Figure 8. Visualization of node feature distributions before and after the GCN layer. Each row indicates an individual sample, and the feature dimension is reduced to two by principal component analysis for better visualization. After the GCN layer, the clusters of different categories are divided further apart while the clusters of the same class are gathered more tightly, which greatly benefits the classification.

4. Discussion

4.1. Going deeper with GCN

Despite showing promise, GCNs typically have shallower architectures than CNNs. This is mainly due to the vanishing gradient problem, which limits the number of layers in most current GCNs to fewer than 5. The depth of a model can significantly impact its performance, especially when the number of layers is limited due to the vanishing gradient problem. To investigate this impact, we conduct experiments in GID24 to evaluate the performance of GHGCN with 1–3 layers, where each layer has 64 hidden units. All other hyperparameters remain unchanged from the previous experiments.

As shown in Table 6, the best model depth is 1, which is consistent with the results of GloRe. This result aligns with our earlier observation that the generated nodes in CHeGCN cover global regions and are fully connected, enabling the effective construction of long-distance relationships. In contrast, presegmentation-based graphs can only represent local areas with spatially connected nodes, requiring more layers for long-distance relationship extraction.

Table 6. Quantitative comparison of models with different GCN layer numbers in GID24.

Layer	OA	Kappa	mIoU
1	89.28	87.45	60.69
2	88.43	86.46	57.69
3	88.41	86.44	56.86

4.2. The influence of node number

The graph size, determined by the number of nodes, plays a crucial role in GCNs. The number of nodes in the graph affects the size of superpixels and has a significant impact on the GCN performance. If the number of nodes $N$ is too small, it may result the oversmoothing of large objects, while too many nodes can lead to noisy representations of tiny objects. Hence, finding an optimal value for the number of nodes is a challenging task, and it often depends on the dataset and task requirements. In our study, we conduct further experiments on GID24 to investigate the influence of the number of nodes on the performance of GHGCN. Specifically, we evaluated the performance of GHGCN with different node numbers ranging from 16 to 256 in Table 7, using a single GCN layer. All the hyperparameters were kept consistent with those used in the previous experiments.

Table 7. Quantitative comparison of models with different node numbers in GID24.

Node	OA	Kappa	mIoU
16	88.01	85.99	55.67
32	89.25	87.41	59.89
64	89.28	87.45	60.69
256	89.14	87.28	59.39

The performance variations of the GHGCN models with 32–256 nodes are marginal, indicating that GHGCN is robust to changes in the number of nodes. The performance variations in GHGCNs with 32–256 nodes are marginal. This robustness can be attributed to the advantages of the node generation method in GHGCNs. The adaptive creation of the node projection matrix over the entire image allows for flexibility in defining the corresponding area of a node. The correlated area can be of any shape and can even cover multiple isolated regions, as shown in Figure 7. This adaptability enables the model to effectively capture spatial relationships and maintain consistent performance across different node numbers. However, when the node number is decreased to 16, there is a considerable decrease in performance. This is because, with only one GCN layer and too few nodes, the model capability is significantly limited. Therefore, selecting a relatively large value for the number of nodes is recommended.

4.3. The quality of coarse LCLU data

The employed coarse LCLU mask shares the same spatial resolution as the refined LCLU labels to explore the theoretical positive effects of the coarse LCLU data. Moreover, their category systems are consistent. However, in practice, the available coarse LCLU data often fail to meet these criteria. Commonly used coarse LCLU products typically have spatial resolutions of 10, 30 m, or even lower and may exhibit category system mismatches. Therefore, the proposed model may be limited by these two mismatches in practical applications, and further research is needed on these two types of mismatches. Furthermore, we have observed that the benefits of the coarse LCLU mask for identifying refined vegetation categories are marginal, as their identification relies more on temporal features than visual patterns.

4.4. Limitations and future work

The GHGCN model is built upon the features of a pretrained CNN, while multi-scale information is not fully exploited. Multi-scale information has been shown to be beneficial for image segmentation tasks. Techniques such as pyramid pooling and ASPP have demonstrated their effectiveness in capturing contextual information at different scales, leading to more accurate and robust segmentation results. Incorporating multi-scale CNN features into the GHGCN model can enhance the model’s ability to capture spatial context at different scales, which is particularly valuable when dealing with objects or structures of various sizes. Moreover, the idea of creating multi-scale heterogeneous graphs on multi-scale CNN features aligns well with the idea of aggregating information from various levels of spatial detail. In the future, we will investigate the potential to build multi-scale heterogeneous graphs on multi-scale CNN features, which is indeed an interesting and promising direction for further improving the segmentation performance.

5. Conclusion

This work highlights the importance of incorporating prior knowledge (i.e. coarse LCLU information) for refined LCLU classification in addition to imagery. The proposed framework, which combines CNNs and GCNs and effectively utilizes coarse LCLU data, shows promising results and contributes to the advancement of image segmentation techniques for remote sensing applications. Specifically, we explore three different approaches to utilizing this information. Each of them has positive effects on improving the segmentation results. Notably, incorporating coarse LCLU in the feature aggregation process within heterogeneous graphs has shown significant success. In our proposed model, graph convolutions in heterogeneous graphs are implemented in an end-to-end manner. This streamlines the training process and facilitates better optimization. Our model not only improves segmentation performance but also demonstrates the potential of harnessing coarse LCLU results effectively. The design of this model can lead to further advancements in LCLU classification and offer valuable contributions to the remote sensing community.

Acknowledgments

The authors gratefully acknowledge the free access to the GID dataset (https://x-ytong.github.io/project/GID.html), the GID-15 dataset (https://captain-whu.github.io/HPS-Net/), and the Five-Billion-Pixels dataset (https://x-ytong.github.io/project/Five-Billion-Pixels.html).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data used in this study are available by contacting the authors.

Additional information

Funding

This work was supported by the National Key R&D Program of China under grant number 2021YFB3900503.