A method for building extraction in remote sensing images based on swintransformer

ABSTRACT

Remote sensing image building segmentation, which is essential in land use and urban planning, is evolving with advancements in deep learning. Conventional methods using convolutional neural networks face limitations in integrating local and global information and establishing long-range dependencies, resulting in suboptimal segmentation in complex scenarios. This paper proposes LMSwin_PNet, a novel segmentation network that addresses the SwinTransformer encoder's deficiency in local information processing through a local feature extraction module. Additionally, it features a multiscale nonparametric merging attention module to enhance feature-channel correlations. The network also incorporates the pyramid large-kernel convolution module, replacing the traditional 3 × 3 convolution in the decoder with multibranch large-kernel convolution, thereby achieving a large receptive field and detailed information capture. Comparative analyses on three public building datasets demonstrated the model's superior segmentation performance and robustness. The results show that LMSwin_PNet produced outputs closely matching labels, showing its potential for broader application in remote sensing image segmentation tasks. It achieved achieving an IoU of 72.35% on the Massachusetts Building Dataset, 91.30% on the WHU Building Dataset, and 78.99% on the Inria aerial-image building dataset. The source code will be freely available at https://github.com/ziyanpeng/pzy.

KEYWORDS:

• Remote sensing image
• building segmentation
• deep learning
• improved SwinTransformer
• large-kernel convolution

1. Introduction

The study of building extraction from satellite remote sensing images has particular significance as it contributes to the comprehension of building information within these images. It also provides essential data support for population forecasting (Wang, Wang, et al. 2022), urban planning (Wang and Qu 2022), and agricultural production (Huang et al. 2018).

In recent years, convolutional neural networks (CNNs) have gained widespread utilization in remote sensing image analysis, including detection (Wang, Lv, et al. 2022), classification (Duan, Duan, and Ding 2021), and semantic segmentation (Sun et al. 2021), yielding notable results. Unlike conventional machine learning techniques such as support vector machines (Razaque et al. 2021), random forests (Linhui, Weipeng, and Huihui 2021), and conditional random fields (Wang, Mu, et al. 2021), CNN-based approaches excel at capturing complex local contextual details. This, in turn, enhances their ability in feature representation and pattern recognition. In 2014, the fully convolutional neural network (FCN) (Long, Shelhamer, and Darrell 2015) replaced fully connected layers with fully convolutional networks to accomplish image segmentation. However, the simplification of the FCN decoder has led to coarse resolution segmentation (Wu et al. 2018). Afterward, in 2015, U-Net (Ronneberger, Fischer, and Brox 2015) achieved significant success in image segmentation owing to its distinctive skip connections and encoder–decoder architecture. While its encoder extracts hierarchical features by gradually downsampling the spatial resolution of feature maps, the decoder learns more contextual information by progressively restoring the spatial resolution. Therefore, the encoder–decoder framework has become the standard for remote sensing image segmentation networks. While CNN segmentation networks using the encoder–decoder structure have achieved commendable segmentation results, they primarily focus on local semantic feature extraction, thus lacking the ability to capture global semantic features. Previous research has explored the application of dilated convolutions (Deng, Shi, and Li 2021a; Wang et al. 2018) to expand the receptive field. However, this approach is prone to information loss (Dai et al. 2017; Yu, Koltun, and Funkhouser 2017) and may fail to capture contextual information from a global perspective.

Obtaining long-range dependencies and a large receptive field proves crucial in distinguishing features, particularly in remote sensing images featuring complex patterns and low contrast, manmade objects. Identifying the objects requiring segmentation solely through global information is challenging. Recently, the vision transformer (Vit) (Vaswani et al. 2017) has presented a substantial challenge to CNNs. It has demonstrated superior performance and is gaining prominence in modern vision tasks. Originally proposed for natural language processing, the Vit structure excels in handling global information dependencies, possessing a global receptive field for extracting more comprehensive semantic information from features. Chen et al. (2021) utilized CNNs to extract features and then employed a transformer for long-term dependency modeling, aiming to address neural network deficiencies in global modeling. Wang, Xie, et al. (2021) introduced the pyramid Vit model for extracting multiscale feature maps, albeit with the complexity scaling to some extent with image size. Liu et al. (2021) presented the SwinTransformer model, utilizing a sliding-window strategy to significantly reduce computational complexity and achieve superior performance in cross-window information exchange. Cao et al. (2022) replaced the convolutional encoding and decoding operations in U-Net with SwinTransformer modules, thereby establishing the Swin_UNet model.

Transformer models exhibit nearly flawless performance in visual tasks. However, these models also have certain flaws. For instance, they typically rely on pretrained networks to achieve peak performance, which requires a large amount of training data for the former (Kornblith, Shlens, and Le 2019; Raffel et al. 2020). Consequently, applying transformer models to acquire global information may prove unsuitable for smaller remote sensing datasets (Devlin et al. 2018; Yang et al. 2019). Notably, the use of large convolutional kernels can yield a substantial receptive field. The principal factor contributing to the performance disparity between CNNs and Vits is in the disparity in their receptive field sizes. Large convolutional kernels can expand the receptive field, compensating for the limitations of transformers on smaller datasets. In addition, transformer models lacks the capacity to extract local information. In recent research, more transformer architectures have focused on extracting local information. Zhu et al. (2023) proposed a simple and effective dynamic sparse attention for two-layer routing to achieve more content-aware, flexible-computing allocation. Furthermore, Ren et al. (2022) proposed a new method for addressing the efficiency of Vit in capturing small objects; it allows Vit to model attention at a mixed scale of each attention layer while preserving fine-grained features. Consequently, extracting local information while extracting global information based on the transformer model is crucial for semantic segmentation tasks.

This paper proposes a novel segmentation network designed for building segmentation in remote sensing imagery. It leverages a fusion of transformer and CNN elements, distinct from purely transformer-based networks. In particular, the improved SwinTransformer is used as an encoder for global feature extraction, the LFEM + MNPA is used for extracting local information, and the PLKC module is employed to obtain multiscale large receptive fields, solving the limitations of pure transformers on smaller datasets and the loss of detail information by SwinTransformer.

Our main contributions are summarized as follows:

1. Introduction of a new segmentation network called LMSwin_PNet, representing a hybrid of SwinTransformer and U-Net architectures. This network was successfully applied to three publicly available building datasets;

2. Proposal of the local feature extraction module (LFEM), combining the potent global information extraction capability of the SwinTransformer encoder to complement the local information that SwinTransformer may overlook;

3. Proposal of the nonparametric merging attention (MNPA) module. This module enhances features at each layer, with the number of channels in MNPA adapting to the feature maps. The greater the number of channels, the more feature map groups that are employed, with the maximum number of feature map groups increasing alongside the channel count. The designed MNPA module collaborates with the LFEM module to concentrate on pertinent building areas;

4. Proposal of the PLKC module, which is characterized by varying convolution kernel sizes. The PLKC block size adjusts to the feature map's dimensions. For larger feature maps, smaller convolution kernel sizes are employed within the PLKC block. This pyramid structure, coupled with a large receptive field, allows for the capture of more information, thereby significantly enhancing network-segmentation representation and performance.

2. Related work

2.1. Semantic segmentation of buildings

Traditional semantic segmentation methods are often used for semantic segmentation of buildings. Li et al. (2015) obtained the initial classification of image pixels based on unsupervised segmentation, and then developed a new conditional random field formula to achieve accurate building segmentation. Du, Zhang, and Zhang (2015) learned a random forest classifier from a large number of imbalanced samples with high-dimensional features to perform more precise semantic segmentation of buildings. Although traditional semantic segmentation algorithms are suitable for extracting buildings, they often rely on manual feature extraction and rule design, making it difficult to handle complex scenes and diverse architectural styles.

The development and application of the CNN are also applicable to the field of semantic segmentation of remote sensing images. The CNN is currently the main technology for constructing semantic segmentation. Liu et al. (2019) proposed a lightweight deep learning model that combines spatial pyramids with the encoder–decoder structure, enhancing segmentation accuracy while maintaining compact model parameters. Deng, Shi, and Li (2021b) uses the stable encoder–decoder architecture, combined with a grid-based attention gate and atrous spatial-pyramid-pooling module, to capture and restore features progressively and effectively. Tian et al. (2021) elevated building segmentation accuracy by incorporating attention mechanisms and a dense feature pyramid into the segmentation network. Yuan, Wang, and Xu (2022) proposed the shift pooling PSPNet, addressing the limitations of pyramid pooling and effectively capturing complete local information for building segmentation. This continuous development of CNN based feature learning structures continually enhances the efficacy of building extraction from remote sensing images.

In remote sensing images, nonbuilding objects often exhibit complex structures, making it difficult to achieve accurate segmentation without global semantic information. In the past 3 years, transformer technology has made significant progress in semantic segmentation, with some transformer-based models achieving better performance than the CNN. Renhe, Qian, and Guixu (2023) introduces a novel shunted transformer, which enables the model to capture multiscale information internally while fully establishing global dependencies, which enables the construction of a pure ViT-based U-shaped model for building extraction. However, there are fewer building extraction methods for pure transformer models.

Although semantic-segmentation models based on a CNN and transformer have achieved good segmentation results, the former approach falls short in capturing multiscale global features, while the latter can lead to the loss of local information, as depicted in Figure 1. For semantic segmentation, if only local information is modeled, the classification of each pixel will often be fuzzy. The combination of a CNN and transformer architectures may be an effective method to improve semantic-segmentation performance. Liegang et al. (2023) uses convolution to extract the local features of buildings while extracting global features in the encoder. Xiao et al. (2022) integrates a U-shaped network in a novel manner to achieve the feature-level fusion of local and large-scale semantics. The above methods provide the possibility for the fusion architecture of a CNN and transformer to achieve global and local feature fusion of buildings.

Figure 1. Explanation of local information and global information. The image on the left is from the WHU Building Dataset, and the image on the right is from Massachusetts Building Dataset, where the local contextual information is modeled using convolution (in yellow). The global contextual information is modeled through the dependency relationship between long-range windows (in red).

2.2. Local feature extraction

In Vit, images are tagged with fragments, and then long-range contextual information is obtained by examining the relationship between representation and self-attention structures. The standard Vit includes multi-head self-attention (MSA), multilayer perceptron (MLP), and layer normalization (LN), as shown in Figure 2(a). Compared with conventional Vit, SwinTransformer is a hierarchical transformer that uses efficient self-attention strategies. As shown in Figure 2(b), the conventional MSA method is replaced by the rule-based window MSA method and the moving SW-MSA method. In the SwinTransformer model, although the alternating execution of W-MSA and SW-MSA increases the interaction between information, this approach restricts the focus on local information within the window, thus weakening the global modeling capability of the encoder (Xia et al. 2023; Xu et al. 2023). Hence, we use the LFEM to address the additional detailed information that SwinTransformer may inadvertently neglect, as shown in Figure 2(c).

Figure 2. (a) Architecture of a standard transformer block; (b) Two consecutive window-based transformer blocks (W-TB) and shift window-based transformer group (SW-TB); (c) Novel SwinTransformer block with a multiscale nonparametric merging attention mechanism module (MNPA) and local feature extraction module (LFEM).

2.3. Nonparametric merging attention module

The concept of grouping feature maps has been a recurring topic of research. In the context of deep separable convolutions (Chollet 2017; Howard et al. 2017), the grouping operation of feature maps has reached an extreme, with the number of groups matching the number of channels in the feature maps. Each group undergoes independent convolution operations, enabling the capture of spatial information within the feature maps. Furthermore, in EPSANet (Zhang et al. 2022), the grouping operation is used to capture dependencies between different positions of the feature maps, thus enhancing the interconnection among the channels within the feature maps. Building upon these ideas, we developed an MNPA module that focuses on the pertinent channel features through the utilization of the SimAM attention mechanism (Yang et al. 2021). The grouping methodology and the number of groups are tailored to accommodate the varying channel numbers within the encoder network at distinct stages. In cases where feature maps contain a large number of channels, the number of groups correspondingly increases, further complicating the grouping approach. This architectural design facilitates more efficient interaction among feature channels with differing grouping strategies and numbers, ultimately making the fused features resulting from multiscale integration more interpretable. This, in turn, contributes to achieving superior segmentation performance.

2.4. Large-kernel convolution

The convolutional operation is a core component of CNNs, and the choice of convolutional kernels is crucial for network performance. Previous studies have widely used 3 × 3 convolutional kernels owing to advantages such as fewer parameters, more nonlinear transformations, and better parameter-sharing effects (Simonyan and Zisserman 2014). However, these kernels have a small receptive field, making it difficult to capture global features. Convolutional kernels larger than 5 × 5 are usually referred to as large kernels. In the early days, large kernels were frequently used, such as in AlexNet (Krizhevsky, Sutskever, and Hinton 2012). Although large kernels can achieve a larger receptive field, they have drawbacks, such as low efficiency and the overlooking of details, so they have been replaced by 3 × 3 convolutional kernels for a long time. Recently, the resurgence of large kernels has introduced a fresh perspective. RepLKNet (Ding et al. 2022) proposed the idea that small kernels can compensate for information loss. The fundamental concept involves the simultaneous use of small kernels alongside large kernels to capture complex details that large kernels may disregard. Empirical results substantiate the superiority of this approach. In RepLKNet, a dimension-reduction operation precedes large-kernel convolution, effectively curtailing the computational burden of the network. Based on these principles, this study proposes the concept of PLKC. Prior to each large-kernel convolution, dimension reduction is applied, and each PLKC module integrates 3 × 3 convolutions in parallel to enhance the extraction of fine-grained features. The pyramid structure aids in amalgamating the receptive fields of convolutional kernels at various scales, thereby enhancing segmentation accuracy. In particular, the configuration of the PLKC module adapts to the feature map's size. Larger feature maps entail the application of smaller convolutional kernels within the PLKC module, thus effectively mitigating the computational demands associated with large-kernel convolution.

3. Method

The novel architectural segmentation network proposed in this research is illustrated in Figure 3. This network, denoted as LMSwin_PNnet, comprises an enhanced SwinTransformer encoder and a decoder founded on large-kernel convolution. In its entirety, this network accepts remote sensing building images as input and seamlessly produces building segmentation results. The methodology presented in this paper can achieve accurate building segmentation while maintaining network efficiency. The subsequent sections offer comprehensive explanations for each constituent element.

Figure 3. LMSwin_PNet for building segmentation in remote sensing images.

3.1. LFEM

Although global contextual information plays a pivotal role in the semantic segmentation of complex architectural scenes, retaining rich details through local information is equally important. To address this requirement, we devised two parallel branches dedicated to separately extracting global and local contexts. global contextual information is combined with local contextual information insights obtained from LFEM to yield comprehensive global–local contextual information.

In datasets 1 and 3, most individual images contain multiple small and irregularly spaced buildings, and the spacing between buildings is tight. In contrast, Dataset 2 has relatively high image accuracy, with most individual images containing few or even no buildings, and a relatively large proportion of individual target buildings, with sparse intervals between buildings. Recognizing the differences between Dataset 1, Dataset 2, and Dataset 3, we tailored two different methods for extracting local features. Based on the data characteristics of dataset 1 and dataset 3, we use a size of 1 × 1. 3 × 3 and 5 × Three convolution kernels of 5 are used to extract local contextual information. Conversely, for dataset 2, we use a size of 1 × 1. 3 × 3. 5 × 5 and 7 × The four convolution kernels of 7 are used to extract local contextual information. Both methods for extracting local contextual information combine batch normalization before addition or superposition operations. Finally, three different pooling methods are applied to balance the features in the feature map. The structural details are shown in Figure 4(a and b).

Figure 4. (a) The LFEM used in Dataset 1 and Dataset 3; (b) The LFEM used in Dataset 2

3.2. MNPA

For the LFEM-derived features, we employ MNPA for supervision. This supervised grouping of channel features effectively highlights the objects that require detection, enabling the network to adaptively learn the critical aspects of the input data. Our MNPA approach is composed of different cascaded grouping methods. When the feature map has a channel count of 128, no grouping is applied, as illustrated in Figure 5(a). When the channel count reaches 256, the feature map is divided into groups of 1 and 2, as depicted in Figure 5(b). For a feature map with 512 channels, grouping involves 1, 2, and 4 groups, as shown in Figure 5(c). Finally, in the case of a feature map containing 1,024 channels, grouping includes 1, 2, 4, and 8 groups, as illustrated in Figure 5(d).

Figure 5. Cascading of groups corresponding to different feature channel numbers at different stages of MNPA, where G represents the number of groups in the feature map.

The grouped feature map is subsequently input into the SimAm attention mechanism. SimAm operates without generating extra parameters and assigns weight values to individual neurons through an energy function, eliminating the need for complex structures: (1) ${\mathrm{e}}_t(w_t,b_t,y,x_i)={\displaystyle \frac{1}{M-1}\sum _{i=1}^{M-1}{(-1-(w_t{x}_i+b_t\left)\right)}^2+(1-(w_{t}t+b_t){)}^2+\lambda w_t^2,}$(1) where $i$ is the index over the spatial dimension; $t$ and $x_i$ represent a neuron and other neurons in the feature map, respectively; $M$ represents the number of neurons in a channel; $w_t$ and $b_t$ represent the weights and biases of the neuron, respectively; and $\lambda$ represents a hyper-parameter in the function, usually set to 10⁻⁴. Furthermore, by calculation and inference, the average value and variance of all neurons, along with the weights $w_t$ and biases $b_t$, can be obtained. By substituting these analytical solutions into it, we can obtain the minimum energy function: (2) $e_t^{\ast }={\displaystyle \frac{4(\stackrel{\wedge }{{\sigma }^2}+\lambda )}{{(t-\stackrel{\wedge }{\mu })}^2+2\stackrel{\wedge }{{\sigma }^2}+2\lambda },}$(2)

which is an energy function that quantifies the disparity between the target neuron and other neurons. where the definitions of $\stackrel{\wedge }{{\sigma }^2}$ and $\stackrel{\wedge }{\mu }$ are given in Equations (3(3) $\stackrel{\wedge }{{\sigma }^2}={\displaystyle \frac{1}{M}{\sum }_{i=1}^M{(x_i-\stackrel{\wedge }{\mu })}^2,}$(3) ) and (4(4) $\stackrel{\wedge }{\mu }={\displaystyle \frac{1}{M}{\sum }_{i=1}^M{x}_i.}$(4) ), respectively: (3) $\stackrel{\wedge }{{\sigma }^2}={\displaystyle \frac{1}{M}{\sum }_{i=1}^M{(x_i-\stackrel{\wedge }{\mu })}^2,}$(3) (4) $\stackrel{\wedge }{\mu }={\displaystyle \frac{1}{M}{\sum }_{i=1}^M{x}_i.}$(4) In general, a smaller value signifies greater importance assigned to the target neuron. To address the need for processing all channels within the feature map efficiently, we use SimAM to enhance and optimize the energy function, as depicted in Equation (5(5) $\stackrel{\wedge }{X}=\mathrm{sigmoid}({\displaystyle \frac{1}{E})\odot X.}$(5) ). (5) $\stackrel{\wedge }{X}=\mathrm{sigmoid}({\displaystyle \frac{1}{E})\odot X.}$(5) Among them, $E$ groups all $e_t^{\ast }$ across channel and spatial dimensions. sigmoid limits the larger value in $E$, and $X$ represents the input neurons. As shown in Figure 6, the specific processing flow of the SimAM when the number of feature map groups is 4 is as follows:

Figure 6. Cascading unit when the number of groups is 4(G = 4), where G represents the number of groups in the feature map, X represents the feature map, C represents the number of feature channels, and SimAMAttention represents the SimAM attention mechanism.

3.3. PLKC

In the decoder, we replace the original 3 × 3 convolution module with the PLKC module, which primarily comprises five distinct structures, ranging from PLKC0 to PLKC4. Each PLKC module consists of four parallel branches arranged in a sequential manner. Each branch includes convolution operations, batch normalization, and the ReLU activation function.

Moreover, this paper uses a 3 × 3 convolution on convolutions with kernel sizes greater than or equal to 5 × 5 to supplement fine-grained information. Concurrently, each convolution with a large kernel employs 1 × 1 convolutions for dimension reduction and dimension expansion to mitigate computational complexity. We use convolution kernels of different sizes based on the size of the feature map. When the size of the feature map is small, we use a larger convolution kernel in the PLKC model. When the size of the feature map is large, we use a smaller convolution kernel in the PLKC module. The design concept of PLKC, in one sense, benefits from large-kernel convolution, effectively improving the prediction accuracy of the model and, in another sense, reducing the enormous computational burden brought by large-kernel convolution.

The convolution kernels used in PLKC0 include 1 × 1, 3 × 3, and 5 × 5, as illustrated in Figure 7(a). For PLKC1, the convolution kernels include 1 × 1, 3 × 3, 5 × 5, and 7 × 7, as shown in Figure 7(b). In the case of PLKC2, the convolution kernels comprise 3 × 3, 5 × 5, and two 7 × 7 kernels, as depicted in Figure 7(c). PLKC3 employs 3 × 3, 5 × 5, 7 × 7, and 9 × 9 convolution kernels, as presented in Figure 7(d). Finally, PLKC4 utilizes 5 × 5, 7 × 7, 9 × 9, and 11 × 11 convolution kernels, as outlined in Figure 7(e).

Figure 7. PLKC block (a) PLKC0. (b) PLKC1. (c) PLKC2. (d) PLKC3. (e) PLKC4.

4. Experiment and results

4.1. Datasets

To ensure a fair comparison with existing research, researchers typically perform the evaluation of building segmentation accuracy using publicly available building datasets. In our study, we employed three distinct datasets to assess the model's performance.

Dataset 1 (Mnih 2013) is the Massachusetts Building Dataset, which comprises aerial images with R, G, and B bands and corresponding labels. The size of all images in this dataset is 1500 × 1500, with a spatial resolution of 1 m. The Massachusetts dataset has a lower spatial resolution and contains more buildings in each image. Dataset 2 (Ji, Wei, and Lu 2018) is the WHU Building Dataset, provided by Wuhan University, which has original data from the New Zealand Land Information Service website. This dataset includes a total of 187,000 buildings and is a visible-light dataset with a ground resolution of 0.3 m. The dataset contains 4,736 training images (130,500 building areas), 1036 validation images (14,500 building areas), and 2,416 testing images (42,000 building areas), with an image size of 512 × 512. Dataset 3 (Maggiori et al. 2017) is the Inria aerial-image building dataset, which does not split adjacent parts of the same image into training and testing subsets, but includes different cities in each subset. It covers different urban settlements, including densely populated areas and high mountain towns, covering an area of 810 square km with a spatial resolution of 0.3 m. Since only the masks of the images in the training set are available publicly, these data were used in this study.

4.2. Evaluation metrics

To examine the building extraction performance of various networks, we employed well-established metrics, including intersection over union (IoU), recall, and F1 score. These metrics were computed based on the counts of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). TP indicates that the model correctly identifies building pixels as building pixels, FP indicates that the model incorrectly identifies non building pixels as building pixels, FN indicates that the model incorrectly identifies building pixels as non building pixels, and TN indicates that the model correctly identifies non building pixels as non building pixels.

IoU was used to measure the degree of overlap between the segmentation results predicted by the model and the actual annotated segmentation results. The value range of IOU is between 0 and 1, with a larger value indicating a higher degree of overlap between the predicted segmentation results and the actual segmentation results, resulting in better model performance. The formula for IoU is (6) $I\mathrm{oU}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}.}$(6) Precision was used to measure how much of the model's predicted positive class samples were truly positive class samples. The formula for precision is (7) $\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}.}$(7) Recall indicates that the model successfully predicted the proportion of the number of pixels in a certain category to the actual number of pixels in that category. The formula for recall is (8) $\mathrm{Re}\mathrm{call}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}.}$(8) F1 score is a comprehensive evaluation index for model performance, which combines precision and recall indicators to measure the predictive performance of the model on positive and negative categories. The formula for F1 score is (9) $F1\mathrm{score}={\displaystyle \frac{2\times \mathrm{Recall}\times \mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}.}$(9)

4.3. Implementation details

All experiments were conducted based on experimental data with the same training parameters used three times, and the column with the highest accuracy in the three training sessions was selected for inclusion in the table. The images in datasets 1 and 2 were cropped to 256 × 256, while the images in dataset 3 were cropped to 512 × 512. In addition, for dataset 1, images with blank areas larger than 5% were excluded. The model was trained for 50 epochs with a batch size of 4. To ensure fairness, we conducted all experiments in the same environment, using the cross-entropy loss function as the model constraint for training, with unified hyper-parameter settings. The Adam optimizer was used with a learning rate set to 0.0001. All experiments were implemented on a NVIDIA GeForce RTX 4090 24-GB GPU.

4.4. Ablation study

4.4.1. The effectiveness of PLKC, LFEM, and MNPA

In this section, we present our validation of the effectiveness of the LFEM, MNPA, and PLKC modules within the network, as presented in Table 1. Ablation experiments were conducted on dataset 1, dataset 2, and dataset 3. The baseline network was established by removing the LFEM, MNPA, and PLKC modules from our network architecture. Table 1 exhibits the results obtained by comparing our method with models containing different components. The baseline network exhibited robust segmentation performance, achieving an IoU of 69.89% and an F1 score of 81.11% on dataset 1, an IoU of 90.70% and an F1 score of 93.61% on dataset 2, and an IoU of 78.55% and an F1 score of 85.56% on dataset 3.

Table 1. Results of ablation experiments with our proposed method.

	Methods	IoU(%)	F1 score(%)	Recall(%)
Dataset 1	Baseline	69.89	81.11	79.08
	Baseline + LFEM + MNPA	71.00	81.97	80.13
	Baseline + PLKC	71.86	82.36	81.12
	Baseline + LFEM + PLKC	71.59	82.17	80.05
	Baseline + LFEM + MNPA + PLKC	72.35	82.93	81.10
Dataset 2	Baseline	90.70	93.61	94.55
	Baseline + LFEM + MNPA	91.01	93.83	94.62
	Baseline + PLKC	91.09	93.92	94.69
	Baseline + LFEM + PLKC	91.12	93.90	94.63
	Baseline + LFEM + MNPA + PLKC	91.30	94.07	94.74
Dataset 3	Baseline	78.55	85.56	86.02
	Baseline + LFEM + MNPA	78.91	85.89	86.72
	Baseline + PLKC	78.77	85.75	85.85
	Baseline + LFEM + PLKC	78.96	85.93	86.14
	Baseline + LFEM + MNPA + PLKC	78.99	85.92	86.46

First, we introduced the LFEM + MNPA module to the baseline network, denoting this model as BaseLine + LFEM + MNPA. The results show that on dataset 1, the IoU and F1 score metrics increased by 1.11% and 0.86%, respectively. On dataset 2, there was an improvement of 0.33% in IoU and 0.27% in the F1 score. On dataset 3, there was an improvement of 0.36% in IoU and 0.33% in the F1 score. The LFEM module extracts local features, while MNPA suppresses irrelevant information within the feature channels, thereby mitigating interference from nonprominent objects. The LFEM + MNPA module enhances the global–local information balance by increasing local information features.

Second, we investigated the effectiveness of the PLKC module. We added the PLKC module to the base network and called this model BaseLine + PLKC. Compared with the basic network, for dataset 1, the IoU and F1 score index increased by 1.97% and 1.25%, respectively; on dataset 2, it increased by 0.39% and 0.31%; and on dataset 3, it increased by 0.22% and 0.19%, respectively. The PLKC module uses cascaded large-kernel convolution and small-kernel convolution, enabling the decoder to capture local information while capturing a large amount of receptive field information, compensating for the traditional 3 × 3 defect of the small receptive field of convolution.

Afterward, we investigated the effectiveness of the MNPA module. By adding the LFEM + PLKC module to the baseline network, denoted as BaseLine + LFEM + PLKC, we assessed its effect. The integration of the PLKC and LFEM + MNPA modules into the baseline network led to the creation of the proposed model, BaseLine + LFEM + MNPA + PLKC. The results show that on dataset 1, the proposed model improved the IoU and F1 score metrics by 0.76% and 0.76%, respectively. On dataset 2, there was an enhancement of 0.18% in IoU and 0.17% in the F1 score. On dataset 3, there was an enhancement of 0.03% in IoU but an decrease 0.01% in the F1 score.

Ultimately, the proposed model (BaseLine + LFEM + MNPA + PLKC) delivered significant improvements, achieving an increase in IoU of 2.46% and an increase in the F1 score of 1.82% on dataset 1. On dataset 2, the model demonstrated an improvement of 0.6% in IoU and 0.46% in the F1 score. On dataset 3, the model demonstrated an improvement of 0.44% in IoU and 0.36% in the F1 score. The integration of the PLKC, LFEM, and MNPA modules facilitates more effective interactions between features, contributing to enhanced segmentation performance.

4.4.2. Comparison of different combinations of convolutions in LFEM

For datasets 1, 2, and 3, owing to the differences in their data, the convolution combinations used for extracting local information were different, resulting in different segmentation effects. In our proposed model, we adopted three distinct convolution combinations tailored to the specific datasets, guided by the evaluation metrics. The convolution combinations corresponding to different evaluation metrics are outlined in Table 2. For datasets 1 and 3, the 1 × 1, 3 × 3, and 5 × 5 convolution combinations had higher IoU and F1 score metrics than for other combinations. However, for dataset 2, the 1 × 1, 3 × 3, and 5 × 5 convolution combinations that performed well on dataset 1 and dataset 3 did not achieve the best segmentation results on dataset 2. Instead, the 1 × 1, 3 × 3, 5 × 5, and 7 × 7 convolution combinations had higher IoU and F1 score metrics on dataset 2, but they were not suitable for dataset 1 and dataset 3. Given the distinctive characteristics of the datasets, the utilization of different convolution combinations to extract local information for each dataset proved beneficial in achieving enhanced segmentation results.

Table 2. The effect of different combinations of convolutional kernels on segmentation effectiveness.

	Methods	IoU(%)	F1 score(%)	Recall(%)
Dataset 1	(1,3)	72.24	82.69	80.57
	(1,3,5)	72.35	82.93	81.10
	(1,3,5,7)	71.15	82.66	80.84
Dataset 2	(1,3,5)	91.10	93.87	94.49
	(1,3,5,7)	91.30	94.07	94.74
	(3,3,5,7)	91.25	94.03	94.66
Dataset 3	(1,3)	78.79	85.79	85.94
	(1,3,5)	78.99	85.92	86.46
	(1,3,5,7)	78.85	85.85	86.29

4.4.3. The impact of the dimensionality-reduction operation in 1 × 1 convolution

Within the PLKC module, by using 1 × 1 convolution to reduce dimensionality, we effectively reduced the computational complexity associated with large-kernel convolution and accelerated the training speed of the model. However, the accuracy of the model could be impacted when using 1 × 1 convolutions to reduce dimensionality. We referred to the model without dimensionality reduction using 1 × 1 convolutions as Ours + NODR (None Dimension Reduction).

To investigate the influence of 1 × 1 convolution dimensionality reduction on segmentation results, we compared all three datasets, namely datasets 1, 2, and 3. As illustrated in Table 3, the segmentation effect of using 1 × 1 convolution without dimensionality reduction varied across datasets. While increasing model complexity, it was likely to increase segmentation accuracy. On dataset 1, it resulted in an increase of 0.27% in the IoU and 0.09% in F1 score. On dataset 2, there was an enhancement of 0.19% in the IoU and 0.17% in F1 score. On dataset 3, it resulted in a decrease of 0.23% in IoU and 0.14% in the F1 score.

Table 3. Influence of 1 × 1 convolution on dimension reduction for segmentation results.

	Methods	IoU(%)	F1 score(%)	Recall(%)
Dataset 1	Ours	72.35	82.93	81.10
	Ours + NODR	72.62	83.02	80.93
Dataset 2	Ours	91.30	94.07	94.74
	Ours + NODR	91.49	94.24	94.94
Dataset 3	Ours	78.99	85.92	86.46
	Ours + NODR	78.70	85.71	85.34

4.4.3. Comparing with the state-of-the-art methods

To validate the effectiveness of our proposed method, we conducted a comparative analysis between LMSwin_PNet and seven advanced segmentation methods, including U-Net, PSPNet (Zhao et al. 2017), Deeplabv3plus (Chen et al. 2018), TransUNet, MANet (Li et al. 2021), Swin_UNet, and BANet (Graham et al. 2021). U-Net, PSPNet, Deeplabv3plus, TransUNet, Swin_UNet, and BANet are representative deep learning models in semantic segmentation. They are widely used in various scenarios and have achieved good segmentation results. MANet is a well-known model in the field of remote sensing segmentation, with good performance. U-Net, PSPNet, Deeplabv3plus, and MANet are all based on CNN architecture, while Swin_UNet and BANet are all based on Transformer architecture. The presence of both Transformer and CNN elements in LMSwin_Pnet and TransUNet has gradually become a research hotspot in recent years, alleviating the problem of insufficient segmentation accuracy caused by insufficient sample data. We carried out both quantitative and qualitative evaluations for all these methods. Tables 4–6 present the quantitative evaluation results on datasets 1, 2, and 3, respectively, and we provide the parameter quantities and Flops for each model in Table 6. Notably, our method consistently outperformed the other segmentation methods in terms of IoU and F1 score, showcasing its superior segmentation capabilities.

Table 4. Comparing our method with other advanced segmentation methods on dataset 1.

Methods	IoU(%)	F1 score(%)	Recall(%)
U-Net	68.52	79.83	77.79
PSPNet	52.50	66.31	60.44
Deeplabv3plus	59.50	72.37	69.05
TransUNet	66.25	77.62	75.04
MANet	66.86	78.48	76.41
Swin_UNet	64.52	76.68	74.45
BANet	65.94	73.14	78.22
LMSwin_PNet(our)	72.35	82.94	81.07

Table 5. Comparing our method with other advanced segmentation methods on dataset 2.

Methods	IoU(%)	F1 score(%)	Recall(%)
U-Net	89.00	92.09	94.21
PSPNet	85.68	90.00	90.29
Deeplabv3plus	85.89	89.64	91.73
TransUNet	87.63	90.78	93.17
MANet	89.74	92.76	93.51
Swin_UNet	84.86	88.88	92.30
BANet	87.63	91.08	93.66
LMSwin_PNet(our)	91.30	94.07	95.99

Table 6. Comparing our method with other advanced segmentation methods on dataset 3.

Methods	IoU(%)	F1 score(%)	Recall(%)	Param(byte)	Flops(G)
U-Net	71.45	79.81	81.12	7766018	219.966
PSPNet	67.61	76.99	76.38	2375874	39.586
Deeplabv3plus	69.07	78.08	77.56	5813266	159.849
TransUNet	78.82	80.91	80.98	93230690	517.17
MANet	75.65	83.28	83.55	35859458	309.820
Swin_UNet	76.71	84.14	84.79	41341632	181.846
BANet	70.20	78.66	78.89	12663460	51.159
LMSwin_PNet(our)	78.99	85.92	86.46	126732802	541.1

Further, we compared the segmentation results of various methods, as illustrated in Figure 8, showcasing visual comparisons on dataset 1, dataset 2, and dataset 3. Notably, our method consistently delivered superior segmentation results. When confronted with relatively blurry images in dataset 1 and dataset 3, our method excelled in distinguishing buildings and handling boundary information. Similarly, when dealing with remote sensing images in dataset 2 characterized by similar contrast, our proposed method accurately identified building areas, demonstrating its robustness and effectiveness.

Figure 8. Visual comparison of segmentation results from different methods on datasets 1, 2, and 3. The first and second columns are from dataset 1, the third and fourth columns are from dataset 2, and the fifth columns are from dataset 3.

5. Discussion

In the encoder, we introduce a three-branch structure for global–local attention, thereby enabling the extraction of a significant amount of global contextual information while preserving complex local details. Global contextual information is obtained through cascaded SwinTransformer branches, whereas local information is captured via cascaded LFEM modules. The MNPA modules further enhance model interpretability by suppressing irrelevant information output by LFEM. Although the combination of global–local information and local enhancement has been achieved, there remains room for research into more effective global–local information extraction techniques.

In the decoder, we propose a fusion of large-kernel convolutions and multiscale concepts. This approach leverages large-kernel convolutions of various sizes for feature extraction, making full use of the large receptive field to strengthen long-range connections between different features. Different convolution sizes allow for the extraction of multiscale information, compensating for information loss associated with the small receptive field and single scale, as seen in traditional neural network structures. The unique PLKC structure with varying kernel sizes provides a substantial receptive field while adding only a modest computational load. Experimental results validate its effectiveness in remote sensing image segmentation.

Traditional CNN structures are currently being influenced by transformer and other architectures owing to their limited receptive fields. This study not only suggested novel directions for future CNN applications in remote sensing image segmentation but also proposed a method to integrate the advantages of a large receptive field into CNN network structures. Experimental findings indicate that combining large-kernel convolutions and small kernel convolutions can enhance model performance. Although large-kernel convolutions introduce additional parameters, the parameter size can be mitigated through dimensionality reduction using 1 × 1 convolutions, albeit with a marginal loss in accuracy. Thus, our future research will focus on more effective ways to combine large- and small-kernel convolutions within CNNs.

6. Conclusions

In this article, we proposed a segmentation network named LMSwin_Pnet, which combines the strengths of both the CNN and transformer architectures to effectively capture global and local information for building remote sensing image segmentation. We conducted extensive experiments on the WHU building dataset, the Massachusetts building remote sensing dataset and the Inria aerial-image building dataset, demonstrating the effectiveness of our model. We achieved an IoU of 91.49% on the WHU dataset, an IoU of 72.62% on the Massachusetts dataset, and an IoU of 78.93% on the Inria aerial-image building dataset.

The core structure of LMSwin_PNet was built upon U-Net, SwinTransformer, LFEM, MNPA, and PLKC. The LFEM plays a crucial role in extracting local information and effectively integrating it with the global information obtained from the original encoder. MNPA enhances the correlation between feature channels without increasing the model's parameters. When used alongside LFEM, it helps the model focus on building details more effectively. The PLKC introduces a novel approach by combining large- and small-kernel convolutions. We designed a PLKC structure with different sizes of convolution kernels, which expands the receptive field while adding minimal computational complexity. We believe that our proposed network can serve as a dependable computer-aided system for building remote sensing image segmentation, thereby facilitating accurate identification of building features in remote sensing images. In the future, we may address computational efficiency concerns associated with models such as Vit by exploring lightweight architectures such as knowledge distillation, tensor decomposition, and depthwise separable convolutions, aiming to achieve minimal accuracy loss in lightweight segmentation networks.

Author contributions

W.Z. and X.Z. designed and completed the experiments and wrote the paper. N.H. revised the paper and analyzed the data. Y.X., T.C. supervised the study. Y.L. and Y.H. guided the process and helped with the writing of the paper. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

We would like to thank the anonymous reviewers for their constructive and valuable suggestions on earlier drafts of this manuscript.

Data availability statement

The data used in this study are from open-source datasets. The datasets can be downloaded from Road and Building Detection Datasets (toronto.edu), https://gpcv.whu.edu.cn/data/building_dataset.html and Download – Inria Aerial Image Labeling Dataset (accessed on 17 January 2024).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by the National Natural Science Foundation of China (grant number 42371441) and the Scientific Innovation Program Project by the Shanghai Committee of Science and Technology (grant number 20dz1206501).