Incorporating convolutional and transformer architectures to enhance semantic segmentation of fine-resolution urban images

ABSTRACT

Though convolutional neural networks (CNN) exhibit promise in image semantic segmentation, they have limitations in capturing global context information, resulting in inaccuracies in segmenting small object features and object boundaries. This study introduces a hybrid network, ICTANet, which incorporate convolutional and Transformer architectures to improve the segmentation performance of fine-resolution remote sensing urban imagery. The ICTANet model is essentially a Transformer-based encoder-decoder structure. The dual-encoder architecture, which combines CNN and Swin Transformer modules, is designed to extract both global and local detail information. The feature information at various stages is collected by the Feature Extraction and Fusion modules (FEF), enabling multi-scale contextual information fusion. In addition, an Auxiliary Boundary Detection (ABD) module is introduced at the end of the decoder to enhance the model’s ability to capture object boundary information. Numerous ablation experiments have been conducted to demonstrate the efficacy of various components within the network. The testing results have proven that the proposed model can achieve satisfactory performance on the ISPRS Vaihingen and Potsdam dataset, with overall accuracies of 91.9% and 92.0%, respectively. Simultaneously, the proposed model is also compared to the current state-of-the-art methods, exhibiting competitive performance, particularly in the segmentation of diminutive objects like cars and trees.

KEYWORDS:

• Image semantic segmentation
• transformers
• convolutional neural networks
• feature extraction
• remote sensing

Introduction

With the advancement of science and technology, satellite remote sensing has been providing comprehensive earth observations at various spatial resolutions. Notably, fine-resolution optical imagery captures a wealth of spatial details that find extensive applications in change detection (M. Wang et al., 2019; Xing et al., 2018), road and building extraction (Griffiths & Boehm, 2019; Shamsolmoali et al., 2021), environmental protection (Samie et al., 2020), and various other practical domains (Ding et al., 2024; Shen et al., 2019; Tang & Werner, 2023). In particular, semantic segmentation of urban scenes serves as an essential descriptor for the development of urbanization (T. Li et al., 2020). Traditionally, the extraction of these terrestrial features from large remote sensing images can be performed manually by human experts. However, this approach is time-consuming and inefficient, primarily due to the vast quantity of satellite data and the subjectivity and variability among different observers in identifying urban features (Vobecky et al., 2022; J. Wang et al., 2015). As a consequence, there is an urgent community need to automatically, effectively, and accurately segment complex urban scenes that consist of numerous objects.

In the early days, numerous efforts were made to automatically extract urban features using various methods, including spectral analysis (F. Chen et al., 2017; Clark & Kilham, 2016), morphological index techniques (Huang & Zhang, 2012; Huang et al., 2017), and multi-information fusion methods (Mezaal et al., 2017; L. Yang et al., 2003). These methods primarily differentiate the urban objects from the background by analyzing the differences in reflected spectral characteristics among various ground objects and incorporating geometrical and contextual information (Ok, 2013). However, they rely heavily on prior knowledge of specific data and can be influenced by intra-class spectral discrepancies and inter-class similarities, making them inadequate for images with complex features (Abdollahi & Pradhan, 2021; Yuan & Mohd Shafri, 2022). In recent years, machine learning-based methods have garnered significant attention due to their potential in semantic segmentation, which have been widely used in remote sensing fields. For example, Ünsalan and Boyer (2005) simultaneously detected houses and streets using IKONOS-2 data by introducing a k-means clustering process that considers spatial information. F. Chen et al. (2018) employed a support vector machine (SVM) method to derive land cover maps of urban environments from high-resolution hyperspectral data. These simple machine learning methods do not fully leverage the rich spatial-spectral information in satellite imagery, resulting in relatively poor overall accuracy and necessitating adjustments for different scenarios (Coseo & Larsen, 2019; X. Ma et al., 2023; Ok, 2013).

Convolutional neural network (CNN) methods have emerged as a pivotal framework in semantic segmentation tasks using high-resolution satellite imagery (Kemker et al., 2018; Kotaridis & Lazaridou, 2021; L. Ma et al., 2019; Tong et al., 2020). This type of method has the capability to automatically and adaptively capture hierarchical context information, resulting in significant capabilities for feature extraction and representation (Griffiths & Boehm, 2019). The fully convolutional network (FCN), as proposed by Long et al. (2015), has been widely acknowledge as an effective architecture for semantic segmentation (Caye Daudt et al., 2018; Mou & Zhu, 2018; B. Zhang et al., 2019). While multiple convolutions and pooling operations can extract information and expand the receptive fields, they may also lead to the loss of global context information (R. Liu et al., 2022). Addressing this limitation, L.-C. Chen et al. (2014) introduced atrous convolutions that enlarge receptive fields without reducing spatial resolution. To maximize the utilization of contextual information in the encoder-decoder FCN architecture, Ronneberger et al. (2015) introduced skip connections, integrating high-level and low-level features to facilitate more precise predictions. Furthermore, several studies have explored the use of the spatial pyramid pooling (SPP) module to capture multi-level contextual features and enhance performance in complex environments (He et al., 2015; Yu et al., 2018; Zhao et al., 2017). Meanwhile, Fu et al. (2019) integrated spatial path attention and channel path attention modules into a dilated FCN, allowing dynamic integration of local features with their global dependencies. Objects in urban scenes often display diverse shapes and scale variability. Despite significant progress, challenges remain in processing images of varying sizes or objects at different scales. This is particularly evident in the inadequate segmentation of small-scale objects and imprecise delineation of image boundaries (Audebert et al., 2018; Kampffmeyer et al., 2016; Yuan & Mohd Shafri, 2022; Yuan et al., 2021).

Compared to CNN-based methods, Transformer-based methods typically use self-attention mechanisms to enhance global information extraction and sequence-to-sequence modeling capabilities, yielding superior segmentation results (Vaswani et al., 2017; Z. Zhang et al., 2023). Moreover, this method integrates broader contextual information to capture extended dependencies across larger image areas, enhancing semantic segmentation capabilities (Ding et al., 2022). Most existing Transformers for semantic segmentation still follow the encoder-decoder structure, which can be roughly categorized into three groups. The first category is the encoder-decoder-based Transformers, known as pure Transformer, including models such as Segmenter (Strudel et al., 2021) and SwinUNet (Cao et al., 2023). These models capitalize on the transformer’s ability to enhance global contextual understanding, thereby improving segmentation precision (Carion et al., 2020). However, their deployment is hindered by substantial computational demands and a dependency on extensive annotated datasets, which pose significant challenges in resource-limited settings (Vaswani et al., 2017). Moreover, they struggle with segmenting small-objects and delineating image boundaries (Hou et al., 2019). The second category typically involves a combination of a Transformer-based encoder and a CNN-based decoder (J. Chen et al., 2021). For instance, DC-Swin is an architecture that combines a Swin Transformer-based encoder (Z. Liu et al., 2021) with a decoder utilizing a densely connected convolutional architecture for segmenting fine-resolution remote sensing images (L. Wang, Li, Duan, et al., 2022). These architectures enhance segmentation accuracy through contextual awareness and adaptability across different scales. However, they face challenges due to high computational demands and reliance on extensive training datasets (Z. Wang et al., 2021). In contrast, the third category consists of a CNN-based encoder and a Transformer-based decoder, as seen in Unetformer (L. Wang, Li, Zhang, et al., 2022). While numerous achievements, Transformer-based models have limitations in remote sensing tasks that involve multi-objective segmentation (Z. Zhang et al., 2023). These models not only lack the inductive biases found in CNN, potentially leading to decreased performance on small datasets, but also require improvements in extracting complex and small target features, as well as capturing boundary feature information (Dosovitskiy et al., 2021; Lin et al., 2021). Additionally, the computational complexity of these models is higher compared to CNN-based models, which can impact their suitability for urban-related applications (L. Wang, Li, Zhang, et al., 2022; Xie et al., 2021).

In this paper, the ICTANet model is proposed, which integrates convolution and transformer architectures. This design enhances multi-scale feature extraction and fusion, improving the segmentation of small and medium-sized targets in urban scenes and addressing issues of inaccurate boundary segmentation. Hybrid CNN-transformer models combine the strengths of CNNs and transformers to effectively leverage both local and global information. However, the wide variety of natural and man-made features at various scales presents challenges in segmenting fine-resolution remote sensing images, especially when features overlap or when the boundaries between different features are not well-defined. In this study, the ICTANet model focuses on two key components: the Feature Extraction and Fusion (FEF) model and the Auxiliary Boundary Detection (ABD) module. These components are integrated within the main structure of the hybrid CNN-transformer to effectively process multi-scale information and enhance boundary detection in urban remote sensing images. Our study makes the following key contributions: (1) ICTANet adopts a hybrid architecture with a dual encoder and a symmetric Transformer-based decoder, enabling competitive results in semantic segmentation of fine-resolution remote sensing urban images. (2) the FEF module, incorporated into the encoder, employs a global-local attention mechanism to effectively integrate local features with global representations and fuse multi-scale information. (3) To enhance the extraction of target boundary feature information, the ABD module is introduced at the end of the decoder, effectively improving the model’s performance. (4) The performance of ICTANet is evaluated using numerical indices and visualization inspections. The results demonstrate that ICTANet achieves competitive performance on the ISPRS Vaihingen and Potsdam dataset, with overall accuracy of 91.9%, and 92.0%, respectively.

The remainder of this paper is organized as follows: the data source and preprocessing is presented in Section 2. Materials and the details of the proposed ICTANet model are introduced in Section 3. Section 4 describes the experiential results and discussions. The conclusions of this study are drawn in Section 5.

Data source and preprocessing

To analyze the performance of the proposed model for urban scene classification quantitatively and qualitatively, we utilized the Vaihingen and Potsdam datasets from the International Society for Photogrammetry and Remote Sensing (ISPRS) Test Project in our experiments. Both of these public datasets are freely accessible at https://www.isprs.org/education/benchmarks/UrbanSemLab/default.aspx. The details of datasets are presented in SI (Table S1).

The Vaihingen dataset consists of 33 patches with varying sizes, covering the Vaihingen region of Germany (as shown in Figure 1a). For each patch, a true orthophoto and a digital surface model (DSM) are provided, both with a ground sampling distance of 9 cm. In our experiments, the model input exclusively consists of true orthophotos, with their RGB bands corresponding to the near infrared (NIR), red, and green channels of the camera. The high-quality labeled ground truths for these patches, which serve as the true labels in model building, can also be obtained (as shown in Figure 1b). In this dataset, it contains five foreground classes, namely impervious surface, building, low vegetation, tree, and car, along with a background class referred to as “clutter”. Therefore, its complex environments, multiscale objects, and diverse distributions provide an opportunity to evaluate the capability for automatic segmentation and generalization of the proposed model.

Figure 1. Overview of the ISPRS Vaihingen 2D dataset. There are consists of 33 tiles of true orthophotos (TOP) tiles and their corresponding digital surface models (DSMs). The numbers in each tile represents the ID number of the tile. The figure source from: (https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-vaihingen/). (a) represents the original image (IRRG), (b) shows the ground truth of image.

Similarly, the Potsdam dataset is named after the region it covers and consists of 38 patches, all with the same size of 6000 × 6000. As depicted in Figure 2, the true orthophotos with four channels (red, green, blue and NIR) and their corresponding labeled ground truths are picked to further verify the performance or accuracy of the developed model. In this region, it contains the same six urban categories as the Vaihingen dataset. However, compared to Vaihingen, Potsdam exhibits larger building blocks, narrow streets, and a denser settlement structure. Additionally, the Potsdam dataset offers a greater number of training samples.

Figure 2. Overview of the ISPRS Potsdam 2D dataset. There are 38 tiles of true orthophotos. The numbers in each tile represents the ID number of the tile. The figure source from: (https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-potsdam/). (a) represents the original image (IRRG), (b) shows the ground truth of image.

Prior to training, it is necessary to preprocess these datasets due to the large size of the high-resolution images and the constraints imposed by GPU memory. In this study, patches with IDs 1, 3, 5, 7, 11, 13, 15, 17, 21, 23, 26, 28, 32, 34, and 37 (2_11, 2_12, 3_10, 3_11, 3_12, 4_10, 4_11, 4_12, 5_10, 5_11, 5_12, 6_07, 6_08, 6_09, 6_10, 6_11, 6_12, 7_07, 7_08, 7_09, 7_10, 7_11, 7_12) from the Vaihingen (Potsdam) dataset are used to generate the training dataset. Patch ID 30 (2_10) from the Vaihingen (Potsdam) dataset is utilized to construct the validation dataset, while the remaining 17 (14) patches from the Vaihingen (Potsdam) dataset are used to build the testing dataset. Notably, only red, green and blue channels in Potsdam images are used in our experiments. The training and validation patches are randomly cropped into 512 × 512 batches using a sliding window approach. As noted by Y. Liu et al. (2017), the segmentation accuracy tends to be highest when the overlap size of the sliding window reaches 75%, while there are less gains for further increases. Therefore, the overlap rate of the crop is set at 75% to preserve the spatial structure of the objects and enhance segmentation accuracy. In addition, a strategy incorporating random horizontal and vertical flipping, rotation, and scaling is employed to augment the dataset and enhance the generalization ability of the proposed model. The details of the training, validation, and testing datasets in this study are listed in Table S1.

All models in the experiments were implemented with the Pytorch framework on a single NVDIA GeForce RTX 2080 Ti GPU, 11GB graphics memory, running on the Ubuntu 18.04.6 LTS operating system and Python 3.8 environment. The AdamW optimizer is used to train and optimize the model, accelerating its convergence speed. The initial value of the learning rate is set to 0.001, the momentum coefficient and weight attenuation are set to 0.9 and 0.01, respectively. The proposed model is retrained on the Vaihingen and Potsdam datasets for semantic segmentation, maintaining the same hyperparameters and training process.

Methodology and materials

Overall architecture

The overall structure of the ICTANet model, mainly consisting of a dual encoder and a Transformer-based decoder, is depicted in Figure 3a. It is evident that this backbone is primarily based on the Swin Transformer, which employs spatial-wise self-attention mechanisms to capture spatial dependencies and facilitate long-range interactions across the entire image. As illustrated in Figure 3c, each Swin Transformer block consists of several components, including multi-head self-attention, Multi-Layer Perceptron (MLP) layers, residual connections, and layer normalization.

Figure 3. The flowchart of the ICTANet.

Since the CNN has the capability to capture spatially local features and extract hierarchical representations of image data, we combine a CNN module and a Swin Transformer module as a dual encoder to effectively capture fine-grained local features and achieve a comprehensive global contextual understanding. In this study, the input images with dimensions of H×W×C (where H, W, and C represent the height, width, and number of channels for the image, respectively) are simultaneously processed by both the convolutional and Swin Transformer modules. Three convolutional blocks are stacked in the CNN module to progressively capture informative features from the input images. As shown in Figure 3b, each convolutional block contains a 3 × 3 convolutional layer for extracting local features, followed by batch normalization, a ReLU activation function for introducing non-linearity, a pooling layer for reducing spatial dimensionality, and a 1 × 1convolution layer for adjusting channel dimensions and fusing multi-scale information. The spatial size of the output feature maps from each convolutional block is halved, while the depth is doubled.

Similarly, the Transformer module also consists of three blocks. Initially, the input images are transformed into smaller one-dimensional sequences by a Patch Partition operation. These sequences are then projected into a lower-dimensional space using Linear Embedding, facilitating effective processing by subsequent Transformer operations. Furthermore, two stacked Swin Transformer blocks are employed to enable efficient attention-based modeling of spatial dependencies and to efficiently process high-resolution image data. The output of the Swin Transformer block and the integrated feature maps from the FEF module are subsequently aggregated in the Patch Merging process. Lastly, the most important features of the input data are captured by the bottleneck, which encompasses two successive Swin Transformer modules.

In the decoder section, the encoding feature maps are reconstructed and transformed into the final outputs using a symmetric Transformer-based module. As illustrated in the right part of Figure 3a, the feature maps from the decoder are concatenated with the corresponding feature maps from the encoder at the same depth using skip connection. This process aims to preserve spatial details and integrate multi-scale information. Prior to this integration, the feature maps from the decoder are upsampled to ensure seamless integration. Subsequently, these feature maps are then passed to the two stacked Swin Transformer blocks, which play a vital role in generating meaningful and contextually relevant output sequences. After three iterations, the reconstructed features are directed to a Linear Projection layer to generate pixel-level segmentation predictions. At the final stage of this model, the Auxiliary boundary detection module is incorporated to enhance boundary detection or segmentation performance by producing sharper boundaries.

Feature extraction and fusion module (FEF)

Both global contextual information and local details are crucial for the semantic segmentation of complex urban scenes. In the encoder section, the FEF is introduced to efficiently integrate local and global information to enhance the representation learning capabilities of the model. As shown in Figure 4, the FEF consists of local global integration branches and fusion stages. Within the local extraction branch, 1 × 1 convolutions and 3 × 3 convolutions are utilized to integrate local information in a multi-scale manner. In addition, a batch normalization layer is added before the cascaded summation operation to expedite the convergence of the model. In the global extraction branch, the multi-head self-attention mechanism plays a significant role in integrating global information. The one-dimensional sequence from the Swin Transformer block is split into query (Q), key (K), and value (V) matrices through linear projection. These matrices are then divided into multiple heads to attend to different positions and aspects of the input. For each head, the Q and K metrics are subjected to dot product operation and normalized by the softmax function to quantify the importance of each position in the input. This process is defined as follows:

Figure 4. Illustration of the feature extraction and fusion module (FEF).

(1) $F_{\mathit{ji}}=\frac{exp\left(Q_i\times K_j^T\right)}{{\sum }_{j=1}^N\mathit{exp}\left(Q_i\times K_j^T\right)}$(1)

Where F_ji represents the attention scores between the ith and jth elements of Q and K, respectively. N represents the number of elements in the K. Then, they are multiplied with V to determine the contribution of each value, with greater attention placed on positions that are more informative for capturing global dependencies. The reshaped results are then scaled by α and subjected to element-wise addition with the input features E_j, resulting in the attended representation G_j (Equation 2).

(2) $G_j=\mathit{\alpha }\sum _N^{\mathit{i}=1}(F_{ji}^T\times V_i)+E_j$(2)

Finally, the feature maps obtained from the global and local branches are concatenated along the channel dimension, which may result in an increase in the output channels. To address this, a convolutional module consisting of a 3 × 3 deep convolutional layer, batch normalization, and a standard 1 × 1 convolutional layer is employed to combine the global and local contextual features and fused the features.

Auxiliary boundary detection module (ABD)

The diversity and complexity of objects in remote sensing images present challenges in accurately identifying object boundaries, frequently resulting in the common problem of pixel confusion around the boundaries and making it challenging to differentiate between similar categories. To address this issue, an auxiliary boundary detection module is devised to improve the generation of thin and accurate boundaries in image segmentation tasks. Its function is to provide a loss that guides the model to generate sharper and more precise boundaries. As shown in Figure 5, ABD consists of a segmentation prediction and a boundary extraction section. As shown in Equation (3)(3) $\mathit{los}{s}_{\mathit{ce}}=H_x\left(x^{\prime }\right)=-\sum _i{x}_i\mathit{log}\left(\mathit{x}{ }_i^{\prime }\right)$(3) , the pixel-based cross entropy loss (i.e. loss_ce) is used to process each pixel individually in the semantic segmentation result, without considering the semantic relationship between adjacent pixels and different objects. Consequently, this approach fails to capture the overall geometric boundaries of the objects.

Figure 5. Illustration of the auxiliary boundary detection (ABD) module.

(3) $\mathit{los}{s}_{\mathit{ce}}=H_x\left(x^{\prime }\right)=-\sum _i{x}_i\mathit{log}\left(\mathit{x}{ }_i^{\prime }\right)$(3)

where x_i is the ith value of the label, and x’_i represents the corresponding prediction.

In the boundary detection process, a boundary detector is employed to accurately locate the object boundary from the ground truth, resulting in the acquisition of the corresponding boundary truth values. Subsequently, the boundary prediction errors are calculated using the boundary perception loss, effectively enhancing the features surrounding the boundaries and suppressing noise features in the background. This process proves beneficial for improving the accuracy of boundary pixel classification. The boundary detection involves a binary classification task, with the loss function described by Equation (4)(4) $\mathit{H}\left(x_i\right)=-\mathit{p}\left(x_i\right)\mathit{log}\left(\mathit{p}\left(x_i\right)\right)+\left(1-\mathit{p}\left(x_i\right)\right)\mathit{log}\left(1-\mathit{p}\left(x_i\right)\right)$(4) . Then, points in the boundary regions can obtain larger loss weight coefficients, indicating that the model will focus more on learning the boundary regions (Equation 5(5) $\mathit{los}{s}_{\mathit{ee}}=\frac{\mathit{nu}{\mathit{m}}_{\mathit{NP}}}{\mathit{nu}{m}_{\mathit{EP}}+\mathit{nu}{m}_{\mathit{NP}}}\sum _{\mathit{i}=1}^{nu{m}_P}\mathit{H}\left(x_i\right)+\frac{\mathit{nu}{\mathit{m}}_{\mathit{EP}}}{\mathit{nu}{m}_{\mathit{EP}}+\mathit{nu}{m}_{\mathit{NP}}}\sum _{\mathit{j}=1}^{nu{m}_N}\mathit{H}\left(y_j\right)$(5) ).

(4) $\mathit{H}\left(x_i\right)=-\mathit{p}\left(x_i\right)\mathit{log}\left(\mathit{p}\left(x_i\right)\right)+\left(1-\mathit{p}\left(x_i\right)\right)\mathit{log}\left(1-\mathit{p}\left(x_i\right)\right)$(4)

(5) $\mathit{los}{s}_{\mathit{ee}}=\frac{\mathit{nu}{\mathit{m}}_{\mathit{NP}}}{\mathit{nu}{m}_{\mathit{EP}}+\mathit{nu}{m}_{\mathit{NP}}}\sum _{\mathit{i}=1}^{nu{m}_P}\mathit{H}\left(x_i\right)+\frac{\mathit{nu}{\mathit{m}}_{\mathit{EP}}}{\mathit{nu}{m}_{\mathit{EP}}+\mathit{nu}{m}_{\mathit{NP}}}\sum _{\mathit{j}=1}^{nu{m}_N}\mathit{H}\left(y_j\right)$(5)

where p(x_i) represents the probability of pixel x_i being classified as a boundary pixel. H(x_i) and H(y_j) denote the cross-entropy loss for boundary and non-boundary pixels, respectively. num_EP and num_NP represent the number of boundary and non-boundary pixels, with num_NP typically being much larger than num_EP. Consequently, points in the boundary regions can obtain larger loss weight coefficients, indicating that the model will focus more on learning the boundary regions. Finally, the overall loss function is calculated as the sum of the segmentation prediction loss and the boundary detection loss:

(6) $\mathit{Loss}=\mathit{los}{s}_{\mathit{ce}}+\mathit{los}{s}_{\mathit{ee}}$(6)

Evaluation metrics

In this study, the overall accuracy (OA), F1-Score, and mean intersection over union (mIoU) are selected as evaluation metrics to assess the accuracy of the proposed model. These metrics are computed based on the accumulated confusion matrix as follows:

(7) $\mathit{OA}=\frac{{\sum }_{k=1}^N\mathit{T}{P}_k}{{\sum }_{k=1}^N\mathit{T}{P}_k+\mathit{F}{P}_k+\mathit{T}{N}_k+\mathit{F}{N}_k}$(7)

(8) $\mathit{F}1-\mathit{Score}=2\times \frac{\mathit{precision}\times \mathit{recall}}{\mathit{precision}+\mathit{recall}}$(8)

(9) $\mathit{mIoU}=\frac{1}{N}{\sum }_{k=1}^N\frac{T{P}_k}{\mathit{T}{P}_k+\mathit{F}{P}_k+\mathit{F}{N}_k}$(9)

Where TP_k, FP_k, TN_k, and FN_k represent the true positive, false positive, true negative, and false negatives of class k, respectively. OA is computed for all categories including the background. In addition, the frames per second (FPS) was used to evaluate how many frames (images) the segmentation algorithm processes in one second. Meanwhile, the memory footprint (MB) and the number of model parameters (M) were used to assess the memory requirements.

Experimental results and discussions

In this study, the proposed model is retrained six times on both the Vaihingen and Potsdam datasets, respectively. From the Table 1, it can be found the proposed model performs relatively stable, as evidenced by the low variability in the evaluation outcomes. Consequently, the experiments with the best performance have been selected for showcasing, specifically Experiment 4 for the Vaihingen dataset and Experiment 6 for the Potsdam dataset.

Table 1. Quantitative evaluation results of six experiments on the Vaihingen and Potsdam datasets.

ICTANet (number)	Imp. Surf.	Building	Lowveg.	Tree	Car	F1-Score	OA	mIoU
	Vaihingen
1	92.8	95.7	84.7	91.2	89.6	90.8	90.2	83.3
2	93.2	95.9	85.1	91.6	89.7	91.1	90.6	83.6
3	93.7	96.2	85.4	92.3	90.3	91.6	91.5	84.1
4	93.7	96.3	85.6	92.5	90.4	91.7	91.9	84.2
5	93.5	96.1	85.4	91.9	90.1	91.4	91.3	84.0
6	93.3	95.9	85.2	91.8	89.9	91.2	91.1	83.9
Std*	0.356	0.276	0.330	0.430	0.340	0.306	0.563	0.310
	Potsdam
1	92.5	96.5	87.4	89.0	96.1	92.3	90.7	87.0
2	92.8	96.5	87.6	89.0	96.3	92.4	90.8	87.2
3	93.6	97.1	88.5	89.6	97.0	93.2	91.7	88.1
4	93.3	96.9	88.1	89.3	96.8	92.9	91.4	87.8
5	93.0	96.6	87.8	89.1	96.5	92.6	91.0	87.5
6	93.8	97.4	88.7	89.9	97.2	93.4	92.0	88.4
Std*	0.479	0.335	0.467	0.334	0.386	0.412	0.475	0.489

Std means the calculated standard deviation for the experiments.

Ablation study

Each component of ICTANet

To assess the individual performance of the modules within the ICTANet model, a series of ablation experiments are conducted using testing datasets. Specifically, the relative importance of the FEF module and the ABD module is evaluated by removing certain components and analyzing the impact on the overall performance of the system. In the experiments, four variations of the ICTANet model incorporating different modules are tested: (a) without the FEF and ABD modules, (b) with only the FEF module, (c) with only the ABD module, and (d) with both the FEF and ABD modules. During these ablation experiments, it is important to note that the model, when lacking a specific component, is retrained using the same hyperparameters as the original ICTANet model, with all other aspects of the model and training process remaining unchanged. For a fair comparison, the segmentation accuracy of each class, F1-Score, OA, and mIoU are calculated. Tables 2 and 3 report the experiment results for the Vaihingen and Potsdam datasets, respectively. Significantly, an FCN-32s model (Long et al., 2015), which set the backbone as the ResNet18 model, is employed as the baseline to evaluate the improvements of the proposed model.

Table 2. Ablation study on the effects of various components in ICTANet architecture, evaluated on the Vaihingen dataset. (“–” indicates without FEF and ABD modules).

Backbone	Model	Surface (%)	Building (%)	Lowveg (%)	Tree (%)	Car (%)	F1-Score (%)	OA (%)	mIoU (%)
ResNet18	FCN-32s	91.57	94.35	82.88	89.34	84.27	88.48	89.15	77.6
	–*(a)	92.13	93.04	82.90	90.26	87.22	89.11	89.31	78.6
	+FEF (b)	93.60	95.62	84.65	88.70	89.58	90.43	90.52	80.8
	+ABD (c)	93.66	94.32	84.04	91.11	89.12	90.45	90.39	80.7
	+FEF +ABD (d)	93.72	96.31	85.59	92.53	90.40	91.71	91.90	84.2

– indicates ICTA model without FEF and ABD modules.

Table 3. Same as table 2, but on the Potsdam dataset.

Backbone	Model	Surface (%)	Building (%)	Lowveg (%)	Tree (%)	Car (%)	F1-Score (%)	OA (%)	mIoU (%)
ResNet18	FCN-32s	91.49	94.52	86.53	88.24	94.19	90.99	90.08	81.5
	–*(a)	91.95	94.84	86.69	88.76	94.11	91.27	90.12	81.7
	+FEF (b)	93.39	96.53	86.37	89.49	96.77	92.51	90.90	83.5
	+ABD (c)	92.55	95.66	87.34	89.63	95.52	92.14	90.78	83.1
	+FEF +ABD (d)	93.78	97.37	88.71	89.94	97.22	93.40	92.00	88.4

It is evident that the model without the FEF and ABD modules performs similarly on the Vaihingen dataset to the baseline model (i.e. FCN-32s). The performance gradually improves when equipped with the FEF and ABD modules, achieving the best results with both modules incorporated into the ICTANet model. The model reaches a peak F1-Score of 91.71%, an OA of 91.9%, and a mIoU of 84.2%. Specifically, when only the FEF module is incorporated, the proposed model achieves improvements of 1.95% in F1-Score, 1.37% in OA, and 3.2% in mIoU compared to the baseline model. It also reveals an increase of 5.31% in the segmentation accuracy for small targets (i.e. car). This demonstrates that the FEF module, integrating features with varying semantic information, can effectively enhance the model performance through context and local feature extraction, and improve the segmentation accuracy of small objects. In addition, incorporating the ABD module, which reduces the impact of fuzzy boundaries, results in an F1-Score of 90.45% and an OA of 90.39%. These figures mark improvements of 1.97% and 1.24%, respectively, over the baseline model. In addition, the model that incorporates both the FEF and ABD modules outperforms configurations that include only one of these modules. As shown in Table 3, the results on the Potsdam dataset demonstrate similar performance. The results of the ablation experiments highlight the effective enhancement in fine-resolution remote sensing image semantic segmentation by incorporating the FEF and ABD modules.

Encoder-decoder combination

To demonstrate the effectiveness of the hybrid structure, the ablation experiments are conducted on the Vaihingen and Potsdam datasets, which are also compared with the ERFNet, SwinUNet, and TransUNet model results (shown in Table 4). We report the training time for each epoch, inference time. In terms of mIoU on the Vaihingen and Potsdam datasets, the ICTANet model maintains competitive accuracy. Notably, when compared to the pure CNN structure (i.e. ERFNet), the ICTANet model shows improvements with an increase of 15.1% and 12.2% on the Vaihingen and Potsdam datasets, respectively. Compared to pure Transformer networks like SwinUNet, the ICTANet model achieves a 50% reduction in computational complexity. Although it has more parameters than Transformer-based encoder-decoder architectures, the ICTANet model can still achieve higher mIoU on both datasets. Additionally, the ICTANet model has fewer parameters than the models consisting of a Transformer-based encoder and a CNN-based decoder (TransUNet) (J. Chen et al., 2021). Therefore, the hybrid structure of the ICTANet model, consisting of a CNN & Transformer-based encoder and a Transformer-based decoder, offers both high execution speed and superior accuracy in remote sensing segmentation.

Table 4. Ablation study on the effects of various encoder-decoder combinations, evaluated on the Vaihingen and Potsdam datasets. The complexity and speed are measured by a 512 × 512 input image and give the inference speed measured in frames per second (FPS) on a single NVIDIA RTX 2080 Ti GPU. The last column mIoU is divided two types, where the first number is the mIoU on the Vaihingen dataset and the second one is the Potsdam dataset.

Method	Backbone	Encoder	Decoder	Complexity (G)	Parameters (M)	Speed (FPS)	Train Time (s)/epoch	Inference Times (s)	mIoU (%)
ERFNet (Romera et al., 2018)	FCN	CNN	CNN	14.75	2.06	48.62	125	6.85	69.10/76.20
SwinUNet (Cao et al., 2023)	Swin-Tiny	Transformer	Transformer	33.91	10.91	33.20	108	5.43	77.20/81.30
TransUNet (J. Chen et al., 2021)	UiT-R50	Transformer	CNN	35.42	20.10	21.70	101	4.62	79.80/83.20
ICTANet	ResNet18	CNN&Transformer	Transformer	16.28	14.02	60.21	138	7.24	84.20/88.40

Assessment of network efficiency

Due to the importance of complexity and speed in fine-resolution remote sensing image segmentation, evaluating the efficiency of the neural network becomes essential. To evaluate the efficiency of the encoder-decoder architecture with different components, metrics such as complexity, parameters, FPS, training time per epoch, inference time, and mIoU are reported in Table 4. While the ICTANet model does not offer the lowest computational cost (FPS = 60.21), it still achieves a competitive computational cost compared to other networks. In addition, the mIoU values on the ISPRS Vaihingen and Potsdam datasets demonstrate that the proposed model outperforms other models in semantic segmentation. These comparisons underscore the suitability of ICTANet for fine-resolution urban image segmentation.

In addition, the ICTANet model is compared with other networks on the Vaihingen and Potsdam testing datasets in terms of complexity, parameter counts, FPS, and mIoU. As demonstrated in Table 5, the ICTANet model has comparable complexity and size to other ResNet18-based networks. Although the ICTANet model has greater complexity and a higher parameter count compared to lightweight networks such as UNetFormer and DANet, it maintains relatively high operational efficiency. Specifically, the ICTANet model achieves a competitive inference speed of 60.21 FPS. Its complexity and parameter counts are comparable to models with similar architectures, yet it achieves notably higher segmentation accuracy. This is evidenced by its performance, surpassing other networks by over 15.6% and 12.5% in mIoU on the Vaihingen and Potsdam datasets, respectively. The relatively high accuracy and speed of the ICTANet model demonstrate the efficiency of its hybrid structure and the effectiveness of the newly introduced FEF and ABD modules. This not only highlights the competitive performance of the ICTANet model but also shows that it can achieve satisfactory segmentation outcomes within limited computational resources.

Table 5. The evaluation results of the ICTANet model compared with other networks. “G” means Gillion (i.e. units for the number of floating point operation), and “M” indicates Million (i.e. units for the number of parameters). The speed is measured by a 512 × 512 input image and give the inference speed measured in frames per second (FPS) on a single NVIDIA RTX 2080 Ti GPU. The mIoU is divided into two types: the first number represents the mIoU on the Vaihingen dataset, and the second one represents the Potsdam dataset.

Method	Backbone	Complexity (G)	Parameters (M)	Speed (FPS)	mIoU
PSPNet (Zhao et al., 2017)	ResNet18	12.55	24.03	98.52	68.60/75.90
DANet (Fu et al., 2019)	ResNet18	9.90	12.68	112.05	69.40/80.30
FANet (Hu et al., 2021)	ResNet18	21.66	13.81	58.64	75.60/84.20
SwiftNet (Oršić & Šegvić, 2021)	ResNet18	13.08	11.80	86.35	78.30/83.80
ResU-Net (R. Li et al., 2022)	ResNet18	10.15	26.27	84.90	83.30/88.10
MAResU-Net (R. Li et al., 2022)	ResNet18	25.43	16.17	30.62	78.60/83.90
UNetFormer (L. Wang, Li, Zhang, et al., 2022)	ResNet18	10.30	10.5	65.48	82.70/86.80
ICTANet	ResNet18	16.28	14.02	60.21	84.20/88.40

Assessment of performance on the Vaihingen dataset

The performance of the ICTANet model is further evaluated using the Vaihingen dataset and compared with some leading networks. From the evaluation results presented in Table 6, it can be found that the ICTANet model exhibits strong performance relative to other networks. Based on evaluation results, the ICTANet model achieves the highest segmentation accuracy in terms of F1-Score (91.7%), OA (91.9%), and mIoU (84.2%). Specifically, the ICTANet model effectively identifies buildings with an impressive 96.3% segmentation accuracy. This is due to the larger size and greater data volume of the buildings in the dataset, which enable the accurate capture of boundaries and enhance the stability of the proposed model. Additionally, the ICTANet model has achieved optimal segmentation accuracies for tree and car objects, underscoring its effectiveness in extracting irregular and small-scale ground objects. While the segmentation accuracy of impervious surfaces and low vegetation is relatively low compared with SwinB-CNN-BD (X. Zhang et al., 2022) and MFNet (Su et al., 2022), attributed to the challenges of distinguishing them from building or trees, the proposed model still achieved competitive performance compared to other networks. Figure 6 illustrates two cases (i.e. ID 11 and 2 in Vaihingen Dataset) to demonstrate that the ICTANet model can effectively segment fine-resolution remote sensing images with relatively high accuracy. In summary, ICTANet shows great potential for urban scene segmentation.

Figure 6. Segmentation results for ID 11 and 2 in the Vaihingen dataset. The first column represents the input RGB images. The second and third columns display the ground truth and the segmentation outcomes generated by the ICTANet model, respectively.

Table 6. Quantitative evaluation results on the Vaihingen testing dataset.

Method	Backbone	Imp. Surf.	Building	Lowveg.	Tree	Car	F1-Score	OA	mIoU
HSN (Pan et al., 2018)	VGG16	92.4	95.1	78.9	88.7	82.4	87.5	89.6
V-FuseNet (Audebert et al., 2018)	FuseNet	91.0	94.4	84.5	89.9	86.3	89.2	90.0	–
UFMG-4 (Nogueira et al., 2019)	–	91.1	94.5	82.9	88.8	81.3	87.7	89.4	–
HUSTW5 (Sun et al., 2019)	SegNet	93.3	96.1	86.4	90.8	74.6	88.2	91.6	–
DDCM-Net (Q. Liu et al., 2020)	ResNet50	92.7	95.3	83.3	89.4	88.3	89.8	90.4	–
HCANet (X. Li et al., 2021)	–	94.2	96.2	83.3	92.3	88.8	91.0	88.8	–
AFNet (X. Yang et al., 2021)	ResNet50 + 18	93.1	96.5	85.8	90.6	88.8	91.0	91.7	–
STransFuse (Gao et al., 2021)	–	88.3	91.5	79.0	85.5	77.1	78.7	86.1	66.7
BANet (Z. Wang et al., 2021)	–	92.2	95.2	83.7	89.9	86.7	89.5	90.4	–
SERNet (X. Zhang et al., 2022)	–	91.7	91.6	81.6	81.6	75.6	84.4	88.1	–
SwinB-CNN+BD (X. Zhang et al., 2022)	Swin-Base	95.3	86.9	83.6	92.2	89.6	89.5	90.4	–
MFNet (Su et al., 2022)	ResNet50	93.4	96.4	85.9	90.5	88.3	90.9	91.7	83.5
UPerNet (Y. Li et al., 2023)	MiT-b3	93.6	96.3	82.4	92.0	87.9	90.1	90.9	73.7
ICTANet	ResNet18	93.7	96.3	85.6	92.5	90.4	91.7	91.9	84.2

In Figure 7, the segmentation details of the ICTANet model and other networks on the Vaihingen dataset are compared. The enlarged partial regions of the original remote sensing images and their corresponding ground truth labels are shown in the first and second columns, respectively. The third to fifth columns display the segmentation results from the Swin-B+BD model, the AFNet model (X. Yang et al., 2021), and the ICTANet model, respectively. As shown in the original images in Figure 7a–f, the tops of buildings and impervious surfaces exhibit similar texture characteristics due to their similar materials and the absence of height information in remote sensing images. Despite their similar features, the ICTANet model shows a relatively accurate ability to distinguish cement roads from low buildings with cement roofs. This can be attributed to the incorporation of the FEF module, which enables the learning and extracting of more comprehensive features from global and local contexts. As illustrated in rows (a) and (e) of Figure 7, tree segmentation presents a significant challenge due to limited contextual and spatial information available to the network. The introduced ABD module in this study enhances the capability to capture relevant information, partially addressing challenges related to shadow occlusion in tree segmentation. The ICTANet model also achieves improved segmentation performance for small objects like cars partially occluded by tree and low vegetation shadows, as shown in Figure 7b,c. Even in scenes with dense car arrangements, the ICTANet model can effectively capture car boundaries under various conditions, thanks to the enhanced ability provide by the FEF and ABD modules. Additionally, the complex and compact ground features can be accurately distinguished by the ICTANet model to a certain degree (Figure 7d–f), resulting in closer alignment with the ground truths.

Figure 7. Comparison of segmentation results on the Vaihingen test dataset. The first column displays the enlarged original image. The second column represents the corresponding ground truth, with impervious surfaces, buildings, low vegetation, trees, cars, and the background denoted as white, blue, cyan, green, orange and red, respectively. The third, fourth, and fifth columns showcase instance segmentation results obtained through various methods.

Assessment of performance on the Potsdam dataset

The comparison of the segmentation accuracy between the ICTANet and other methods on the Potsdam dataset are listed in Table 7. The Potsdam dataset is characterized by its large size and diversity, enabling the proposed model to leverage the advantages of large-scale data and mitigate the risk of overfitting. It’s evident that the ICTANet model outperforms other methods, achieving optimal segmentation results. Due to their similar color characteristics, low vegetation and trees pose challenges in segmentation. Moreover, sparse low vegetation plants can be easily confused with the impervious surface category, potentially resulting in incorrect segmentation. Significantly, the segmentation accuracy of the ICTANet model for impervious surfaces is slightly inferior to that of AFNet, UPerNet (Y. Li et al., 2023) and MFNet, which excel in capturing and processing fine-scale textural in complex urban environments. Nevertheless, the ICTANet model excels in low vegetation and tree segmentations, demonstrating its capability to distinguish complex features. Compared to other objects, cars constitute a relatively small proportion in the image and are often obscured by shadows cast, buildings, and trees, posing a challenge to effective feature extraction. The segmentation accuracy of cars of the ICTANet model reaches 97.2%, demonstrating its capability to effectively discern subtle differences between classes. Despite a slight reduction in building segmentation accuracy (97.4%) compared to the UperNet model, the ICTANet model still achieves comparable performance with other methods. The slight decline in building segmentation may be caused by excessive recognition of uniform textures, resulting in over-segmentation. In contrast, the sensitivity of the UPerNet model to subtle texture variations enhances its building segmentation accuracy. In addition, the ICTANet model achieves a high accuracy of F1-Score (93.4%) and mIoU (88.4%), demonstrating the effectiveness of the dual encoder structure and FEF module proposed in this study. The ICTANet model accurately extracts and fuses available feature information from various objects, thereby enhancing the overall model segmentation performance to 92.0%. Figure 8 illustrates the prediction results for ID 3_12 and 4_11, further affirming the effectiveness of the ICTANet model in capturing intricate object details.

Figure 8. Visualization results of ID 3_12 and 4_11 from the Potsdam test set. The first column represents the input RGB images. The second column shows the ground truth, and the third column denotes the segmentation results of the ICTANet.

Table 7. Quantitative evaluation results on the Potsdam testing dataset.

Method	Backbone	Imp. Surf.	Building	Lowveg.	Tree	Car	F1-Score	OA	mIoU
HSN (Pan et al., 2018)	VGG16	92.2	95.2	88.2	82.9	89.8	89.7	89.7	–
V-FuseNet (Audebert et al., 2018)	FuseNet	92.7	96.3	87.3	88.5	95.4	92.0	90.6	–
UFMG-4 (Nogueira et al., 2019)	–	90.8	95.6	84.4	84.3	92.4	89.5	87.9	–
HUSTW5 (Sun et al., 2019)	ResegNets	93.6	97.6	88.5	88.8	94.6	92.6	91.6	–
DDCM-Net (Q. Liu et al., 2020)	ResNet50	92.9	96.9	87.7	89.4	94.9	92.3	90.8	–
HCANet (X. Li et al., 2021)	–	92.8	96.9	87.2	88.1	93.8	91.8	90.6	–
AFNet (X. Yang et al., 2021)	ResNet50 + 18	94.1	97.6	88.7	89.7	97.1	93.4	92.1	–
STransFuse (Gao et al., 2021)	–	89.8	93.9	82.9	83.6	88.5	82.1	86.7	71.5
BANet (Z. Wang et al., 2021)	–	93.3	96.6	87.3	89.1	95.9	92.5	91.0	–
SERNet (X. Zhang et al., 2022)	–	91.7	95.3	73.7	82.2	92.1	87.0	90.2	–
SwinB-CNN+BD (X. Zhang et al., 2022)	Swin-Base	92.2	95.3	83.6	89.2	86.9	89.4	90.4	–
MFNet (Su et al., 2022)	ResNet50	93.9	97.4	87.8	89.0	96.7	93.0	91.6	87.1
UPerNet (Y. Li et al., 2023)	InternImage-B	94.1	97.9	88.1	88.7	96.3	92.9	91.4	79.7
ICTANet	ResNet18	93.8	97.4	88.7	89.9	97.2	93.4	92.0	88.4

Figure 9 illustrates the comparison of image segmentation details between the proposed model and other methods on the Potsdam dataset. The ICTANet model has demonstrated overall good accuracy in segmenting fine-resolution remote sensing urban images. As shown in rows (a) to (c) of Figure 9, the ICTANet model performs relatively high accuracy in segmenting large-scale ground features, such as buildings and low vegetation. In addition, the ICTANet model enhances boundary detail recognition and reduces false segmentation issues caused by the background. Though distinguishing low vegetation from low buildings with rooftop lawns and impervious surfaces with lawns remains challenging, the ICTANet model achieves commendable segmentation results in these cases (rows (a) and (b)). This can be attributed to the self-attention mechanism’s capacity to extract global context information and the ability of Transformer module to capture long-term dependencies. Furthermore, in rows Figure 9d,e, the network demonstrates improved car segmentation. It successfully distinguishes various car types and demonstrates improved boundary accuracy for densely arranged small ground objects. The dual encoder structure of the ICTANet model facilitates the simultaneous learning and integration of both overall and detailed information. Figure 9f shows that the ICTANet model accurately identifies thin and long ground objects. The high inter-class similarity between low vegetation and trees often poses challenges in accurately identifying and distinguishing their boundaries. Nonetheless, the ICTANet model overcomes this challenge by employing the ABD module, which enhances boundary information and improves internal pixel object classification.

Figure 9. Comparison of segmentation results on the Potsdam test dataset. The first column displays the enlarged original image. The second column represents the corresponding ground truth, with impervious surfaces, buildings, low vegetation, trees, cars, and the background denoted as white, blue, cyan, green, orange, and red, respectively. The third, fourth, and fifth columns showcase instance segmentation results obtained through various methods.

Conclusions

In this study, Transformer-based network that combines convolutional operations and a self-attention mechanism is proposed for fine-resolution remote sensing urban image segmentation. A dual encoder structure is designed to extract both global context and local detail information from remote sensing images. Compared to typical hybrid CNN-transformer models, our approach introduces a Feature Extraction and Fusion module (FEF) to enhance network feature representation by optimizing and fusing local features and global representations at different stages. Additionally, an auxiliary boundary detection (ABD) module is added at the end of the model to enhance the extraction of object boundary information. As a result, the ICTANet model is not only effective in segmenting fine-resolution remote sensing urban images but also improve the performance for small objects.

Based on two publicly available fine-resolution remote sensing image datasets, namely, the ISPRS Vaihingen and Potsdam datasets, the ICTANet model effectively segments fine-resolution remote sensing urban images with relatively high accuracy. It achieves 91.9% (92.0%), 91.7% (93.4%), and 84.2% (88.4%) in terms of overall accuracy, F1-Score, and mean intersection over union on the Vaihingen (Potsdam) dataset. Compared to various other methods, the proposed model demonstrates competitive performance for large objects like buildings and low vegetation. The ICTANet model excels in accurately capturing the distribution and boundary details of small objects like cars and sparse trees, outperforming other methods. Comprehensive benchmark and ablation experiments further demonstrate the effectiveness and efficiency of the proposed method for fine-resolution remote sensing urban images.

However, balancing computational efficiency with performance remains crucial. There is potential to reduce the model size and improve inference speed in future studies. Future efforts should focus on reducing consumption cost while maintaining the efficiency and accuracy of the proposed model.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets supporting this study’s findings are available on the website (https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-vaihingen/; https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-potsdam/).

Additional information

Funding

This work was supported by the Collaborative Innovation Center for Natural Resources Planning and Marine Technology of Guangzhou [No.2023B04J0301, No.2023B04J0326], Key-Area Research and Development Program of Guangdong Province [NO.2020B0101130009], Guangdong Enterprise Key Laboratory for Urban Sensing, Monitoring and Early Warning [No.2020B121202019] and The Science and Technology Foundation of Guangzhou Urban Planning & Design Survey Research Institute [RDI2230201063].