A cross-stage features fusion network for building extraction from remote sensing images

ABSTRACT

The deep learning-based building extraction methods produce different feature maps at different stages of the network, which contain different information features. The detailed information of the feature maps decreases along the depth of the network, and insufficiently detailed information results in limited accuracy. However, existing methods are incapable of making full use of low-level feature maps with rich details. To overcome these shortcomings, we proposed a Cross-stage Features Fusion Network (CFF-Net) for building extraction from remote sensing images. In the CFF-Net, we innovatively proposed a Cross-stage Features Fusion (CFF) module that fuses different features generated at different stages. And we used the attention mechanism to make the network more focused on important information at different scales. To further improve the accuracy of building extraction, we designed the Prediction Enhancement (PE) module, where the last convolutional layer and the feature map generated in the intermediate stage are used for prediction at the same time to enhance the final result. To evaluate the effectiveness of the proposed network, we conduct quantitative and qualitative experiments on the two publicly available datasets, i.e. the Inria dataset and the WHU datasets. CFF-Net outperformed other state-of-the-art algorithms on the two datasets in IoU and F1 metrics. The efficiency analysis reveals that the proposed CFF-Net achieves a great balance between building extraction performance and complexity/efficiency, with faster convergence and higher robustness.

KEYWORDS:

• Remote sensing
• cross-stage features fusion
• building extraction
• attention mechanism

1. Introduction

Building extraction from remote sensing images is an important application, which has an important role in urban planning, smart city construction, and disaster management (Cai et al. 2023; Chen et al. 2020; Ding et al. 2022; Dong and Shan 2013; Shao et al. 2021; Trinder and Liu 2020). In recent decades, the development of Very High Resolution (VHR) remote sensing satellites and unmanned aerial vehicles, which can generate meter- and submeter-level remote sensing images easily, has further promoted the development of building extraction techniques (Li, Huang, et al. 2022). VHR remote sensing images include more detailed feature information, from which the finer results of building extraction can be achieved. And the obtained products have wider applications (Huang et al. 2019). However, the VHR remote sensing images also make the homogeneous and heterogeneous effects more obvious and introduce more noise, for example, houses and roads are very similar, making it difficult to extract the fine building results (Guo et al. 2022).

The current methods of extracting buildings based on optical remote sensing images mainly include the traditional hand-craft feature – methods and deep-learning-based methods. The traditional methods extract hand-craft features from remote sensing images, including color, spectrum, and texture, and apply traditional machine learning methods to recognize buildings (Dornaika et al. 2016; Zhang 1999). For example, Huang et al. (2016) took advantage of the difference in morphological profiles for building extraction from remote sensing images. Hossain and Chen (2022) used spectral features of VHR remote sensing images for the image segmentation task to finally achieve building extraction. However, traditional hand-craft features are often designed relying on experience, and these features vary in different weather conditions and sensors, which have little robustness (Zhu et al. 2021). In recent years, deep learning has been a mainstream approach to learning features autonomously. Semantic segmentation, where the goal is to classify each pixel into a fixed set of categories without differentiating object instances (He et al. 2017), is a main task of computer vision and is widely used for building extraction from remote sensing images (Heipke and Rottensteiner 2020). Convolutional Neural Network (CNN) is currently the main deep learning approach for building extraction, which can automatically learn various types of information through convolutional layers, including textures, spectrums, spatial context, and the interactions among geo-objects (Bai et al. 2023; Li, Zheng, et al. 2022; Wang et al. 2023).

A Fully Convolutional Network (FCN) derived from CNNs can achieve pixel-level category prediction on images (Long, Shelhamer, and Darrell 2015), which is widely used in the satellite building extraction task. Ronneberger, Fischer, and Brox (2015) proposed U-Net, which downsamples the images to extract the feature first, and then upsamples the image to the original size while obtaining the extraction results. SegNet adds the pooling index for upsampling based on U-Net (Badrinarayanan, Kendall, and Cipolla 2017). Deeplab-v3 uses atrous separable convolution for feature extraction to obtain the information of different receptive fields (Chen et al. 2017). HRNet extracts the feature through parallel branches simultaneously and integrates to achieve the segmentation task (Sun, Xiao, et al. 2019). In recent years, there have been many researches focused on building extraction in the remote sensing field, Z. Zhang et al. (2018) proposed a Siamese U-Net with shared weights in two branches, SiU-Net, for building extraction. Chen, Zhang, et al. (2022) designed CLNet, which improves the final accuracy by enhancing the features extracted in the network. The transformer was first used in the field of Natural Language Processing (NLP) (Vaswani et al. 2017) and has now been applied to the field of computer vision, Vision Transformer (ViT) (Dosovitskiy et al. 2020) and DEtection TRansformer (DETR) (Carion et al. 2020). BuildFormer uses ViT and CNN dual path for information extraction to finally achieve building extraction for remote sensing images (Wang et al. 2022).

The key to the deep learning-based methods to extract building consists of acquiring the semantic information of the remote sensing images and applying the extracted features appropriately for the final prediction results. The deep learning network normally includes several stages, which produce feature maps containing different feature information, respectively (Xu et al. 2022). Low-level feature maps have more detailed spatial information but are less capable of global perception while high-level feature maps are able to have a clearer understanding of semantic information (Zhu et al. 2020). The proper fusion of low-level and high-level is the key to extracting buildings accurately. The texture details contained in the feature maps weaken as the network deepens. In contemporary deep learning methodologies, efforts have been made to integrate intricate low-level details into high-level feature maps. For instance, U-Net employs skip connections to fuse feature maps from the encoder, which contain finer spatial details, into the decoder. Furthermore, feature fusion techniques have found application in the domain of building extraction (Ronneberger, Fischer, and Brox 2015). BMFR-Net adopts a multi-stage approach within the decoder for result prediction, where results from different stages are fused (Ran et al. 2021). Sun et al. (2019) proposed a fusion strategy based on parallel Support Vector Mechanism (SVM) to utilize the deep features extracted from multi-scale CNN structures at different scales. Additionally, Chen et al. (2022) employed transformer architectures for multi-scale feature extraction, fusing these features to facilitate building extraction. However, the ways are too simple to make full use of the information in high-level feature maps, which exhibit major limitations in difficult identification of buildings at image boundaries, confusion between buildings and non-buildings, and blurred boundaries of complex buildings.

In order to solve the above problems, we proposed a Cross-stage Feature Fusion network (CFF-Net) for building extraction from VHR remote-sensing images. The proposed CFF-Net is able to fuse features from multiple feature maps generated in different stages and make full use of the global semantic and spatially local information through the attention mechanism. Moreover, the CFF-Net is capable of utilizing more information during the prediction by prediction enhancement. We took the feature maps generated in the intermediate stage for the prediction to enhance the prediction results of the final convolutional layer, which is capable of obtaining more accurate extraction results. The main contributions of this study are as follows:

1. We proposed a CFF-Net, a novel cross-stage feature fusion network designed for precise building extraction from VHR remote sensing images. This network adeptly amalgamates cross-stage semantic features derived from multiple processing stages, demonstrating exceptional proficiency across various scales, ensuring accurate predictions for buildings of diverse sizes. Specifically, the network leverages intermediate-stage feature maps to refine predictions generated by the final convolutional layer, significantly enhancing prediction accuracy. And the CFF-Net obtained the best building extraction results on two public data sets compared with the traditional methods.

2. The cross-stage feature fusion module is proposed to fuse the feature information generated in multiple stages. In contrast to conventional methods, which integrate feature via skip connections simply, the CFF module employs the attention mechanism to amalgamate features extracted from different stages within the encoder and pass them into the decoder. This mechanism facilitates the integration of low-level spatial details and high-level semantic information, enriching the ability of network to comprehend multiple scales. The embedded attention mechanism makes the network pay more attention to building areas, improving its capacity to discriminate between building and non-building regions. Notably, the network exhibits superior performance in extracting buildings located at the image peripheries, which are often ignored.

3. To effectively leverage the details information extracted from the intermediate process of network for refining final predictions, we designed a prediction enhancement module in CFF-Net. Diverging from conventional deep learning methodologies that solely rely on the last convolutional layer of the FCN for final predictions, our prediction enhancement module leverages predictions from the intermediate layer of network. This strategic integration enhances the predictions generated by the final layer, leading to more accurate building extraction. Compared to the last convolutional layer, the feature maps in the intermediate stage bring more detailed information. Consequently, this finer level of information empowers the final predictions, ensuring superior accuracy in delineating complex-shaped buildings and small-scale structures.

We organize the rest of this article as follows. We describe the related work involving attention mechanisms and feature fusion in Section 2. The structure and components of CFF-Net are presented in Section 3. Then we give the experiments and result analysis in Section 4, and the conclusions are presented in Section 5.

2. Related work

2.1. Attention mechanisms

Attention mechanisms introduced into computer vision is aiming at imitating people, who can naturally and effectively find salient regions in complex scenes (Guo et al. 2022). Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. The process of attention mechanisms working can be formulated as:

(1) $\begin{array}{c}\mathit{Attention}=\mathit{f}(\mathit{g}(\mathit{x}),\mathit{x})\end{array}$(1)

where $\mathit{g}(\mathit{x})$ represents to generate attention on the whole image; $\mathit{f}(\mathit{g}(\mathit{x}),\mathit{x})$ means processing input $\mathit{x}$ based on the attention $\mathit{g}(\mathit{x})$.

In recent years, many studies have been put into attention mechanisms and have made some progress in computer vision tasks. SENet is a novel channel-attention network that implicitly and adaptively predicts the potential key features (Jie, Shen, and Sun 2018). The network learns the relationship between channels and the importance of each channel in a set of feature maps, and the features of the channels are weighted and fused to improve the representing capacity of the network. Similar to the channel attention mechanism, which aims to capture the important channels, the spatial attention mechanism aims to enable the model to adaptively learn the attention weights of different regions by introducing an attention module (Zhu et al. 2019). The model pays more attention to the important image regions and ignore the unimportant ones. The self-attention mechanism is one type of spatial attention mechanism that captures long-range dependencies between elements adaptively by computing the relative importance between elements, with some popular models being Non-local (Wang et al. 2018), GCNet (Cao et al. 2019), and ViT (Dosovitskiy et al. 2020). Convolutional Block Attention Module (CBAM) (Woo et al. 2018) is an attention mechanism that combines the channel attention mechanism and spatial attention mechanism. The coordinate attention mechanism considers both inter-channel relationships and long-distance location information with only a small increase in computational effort, which can effectively improve the accuracy of the model (Hou, Zhou, and Feng 2021). In the field of remote sensing, numerous studies have introduced attention mechanisms tailored to remote sensing tasks. Zhao et al. (2023) proposed a multiple attention network, including a plug-and-play attention module, for spartina alterniflora segmentation using multitemporal remote sensing images. Y. Gao et al. (2023) introduced a cross-scale mixing attention mechanism to facilitate the fusion and classification of multisource remote sensing data. Li et al. (2022) used the self-attention mechanism for Hyperspectral and SAR Image Classification in AsyFFNet. In SOT-Net (M. Zhang et al. 2022), a cross-attention module was employed to enhance the spectral information of HyperSpectral Imagery (HSI). Additionally, the DFINet featured a depth-wise cross-attention module meticulously crafted to extract both self-correlation and cross-correlation patterns from multisource feature pairs (Y. Gao et al. 2021).

2.2. Feature fusion

Efficient fusion of features during the network, absorbing what is good and rejecting what is bad, is the key to improving the extraction accuracy of the model. Many studies have been conducted on how to better fuse the features of multiple channel feature maps at a single stage, where the feature maps have the same size, to fully exploit the extracted to information (Pan et al. 2023; Yuan et al. 2023). SENet recalibrates channel-wise feature responses by explicitly modeling interdependencies between channels (Jie, Shen, and Sun 2018). GCNet captures global contextual information through the global average pooling layer, and fuses global contextual information with local features when fusing multiple feature maps (Cao et al. 2019). DANet uses dual-attention paths of spatial attention mechanism and channel attention mechanism for the acquisition of important information and integrates local features with their global dependencies in the feature fusion process adaptively (Fu et al. 2019).

The improvement of the feature fusion method in single-stage channels can obtain useful information. The size of the feature maps differ in multiple stages, and the scales of the contained features vary more. The fusion of the feature maps of multiple stages can provide better knowledge of the details and global information. Feature Pyramid Network (FPN) enables the upper-level features to receive high-resolution information from the lower level and maintain high-quality semantic information during feature fusion by adding lateral connections in the feature pyramid at different levels (Lin et al. 2017). Atrous Spatial Pyramid Pooling (ASPP) is a pyramid fusion method using atrous convolution, where atrous convolution operations are performed at different atrous rates to obtain features with different scales, which are then fused together (Chen et al. 2017). ExFuse introduces semantic information into low-level features and high-resolution details into high-level features to achieve the integration of multi-stage features (Zhang et al. 2018).

The above attention mechanisms and feature fusion approaches have good results in semantic segmentation tasks. Unlike natural images, remote sensing images cover a wider range and contain more complex feature information. These methods may not necessarily achieve good results when applied directly to building extraction from remote sensing images. Therefore, for building extraction of remote sensing images, the features fusion methods deserve further exploration.

3. The proposed method

3.1. Overview

To make full use of the spatial details and global semantic information of the feature maps to extract buildings from remote sensing images, we proposed a Cross-stage Feature Fusion Network (CFF-Net). The proposed network fuses cross-stage features in an optimal manner (Figure 1). The image feature extraction component extracts multi-level features from the input image; the feature fusion component fuses these cross-stage features obtained from the previous stage.

Figure 1. The structure of the proposed CFF-Net.

3.2. CFF network

CFF-Net was designed based on the U-net (Ronneberger, Fischer, and Brox 2015), which has been demonstrated to perform well in building extraction. The U-Net contains an encoder, decoder, and skip connection components. In CFF-Net, Resnet-34 was chosen as the feature extraction network (He et al. 2016). Deep-residual networks have been demonstrated to reduce gradient degradation problems in model training and have the ability to locate the building features accurately (Audebert, Saux, and Lefèvre 2018). As shown in Figure 1, the feature extraction stage contains one down-sampling convolutional layer and four residual layers, where five feature maps are obtained. The width and height of each level are 1/2 of the upper level. The size of the convolution kernel of the first convolutional layer is 7 × 7, and the strider is 2. The residual layer consists of residual blocks (Figure 2), and the number of residual blocks in the four residual layers is 3, 4, 6, 3, respectively. A maxpool layer is included between the convolutional layer and the first residual layer to down-sample the feature map. The first residual block of the remaining residual layers is the down-sampling residual module, and the rest of the residual blocks are normal residual blocks. Decoder, whose function is to fuse the current feature maps and the feature maps in the skip connection and upsample, includes four up-sampling modules (Figure 3). We obtain the feature map with the same size as the input image through the decoder. Note that each up-sampling operation changes the width and height of the feature map to twice of the previous one by linear interpolation.

Figure 2. The structure of the residual module. (a) Down-sampling residual module. (b) Normal residual module.

Figure 3. The structure of the up-sampling module.

3.3. Cross-stage feature fusion module

To make full use of features extracted during the feature extraction stage, we proposed a Cross-stage Feature Fusion (CFF) module, which fuses cross-stage lateral encoder features (Figure 4). Inspired by Zhang et al. (2018), CFF – Net fuses the two adjacent features, which can take advantage of the spatial detail and global semantic information. The fused feature map is passed into a decoder with the skip connection to transmit feature information.

Figure 4. The structure of the cross-level feature fusion module.

Cross-stage feature fusion is the process of fusing the feature maps generated by two stages to obtain their respective useful information. The feature maps generated in each stage own different sizes. The features are first interpolated to the same size and fused by concat operation. To capture the exact precise positional information and dependency, we draw on the operation proposed by Hou, Zhou, and Feng (2021). As shown in Figure 4, two adjacent feature maps are linearly interpolated to the same size and then concatenated. In order to obtain the relative positional relationships of the features, the features concatenated are averaged pooled in the both horizontal direction and vertical direction, respectively. For the feature map $\mathit{X}\in {\mathbb{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$, we use the pooling kernel $(1,\mathit{W})$ for horizontal pooling operations. And the output of the c-th channel at height h can be formulated as:

(2) $z_c^h(\mathit{h})=\frac{1}{W}{\sum }_{0\le \mathit{i}<\mathit{W}}{X}_c(\mathit{h},\mathit{i})$(2)

Similarly, the pooling kernel $(\mathit{H},1)$ is used to perform vertical pooling operations. And the output of the c-th channel at width w can be written as:

(3) $z_c^w(\mathit{w})=\frac{1}{H}{\sum }_{0\le \mathit{j}<\mathit{H}}{X}_c(\mathit{j},\mathit{w})$(3)

Two attention maps obtained by pooling horizontally and pooling vertically are concatenated and switched through the non-linear operation. The concatenation of two attention maps enables the network to learn the interconnections between the horizontal and vertical directions of the feature maps. After that, the maps are split and two attention weighed maps are obtained by two 1 $\times$ 1 convolution, each element in which reflects whether the object of interest exists in the corresponding row and column. The features concatenated are weighted with the two attention maps and two 3 $\times$ 3 convolutional transformations are utilized to obtain the output.

There are five stages in the decoder and encoder of the network respectively. Let $F^i$ denotes the feature map input into the $\mathit{ith}$ stage of decoder, then we have:

(4) $F^i=\left\{\begin{array}{cc}{l}^{5-\mathit{i}}\oplus l^{6-\mathit{i}}& \mathit{i}=1\\ \mathit{CFF}(l^{5-\mathit{i}},l^{6-\mathit{i}})\oplus F^{\mathit{i}-1}& \mathit{i}=2,3,4\\ \mathit{Upsample}(F^{\mathit{i}-1})& \mathit{i}=5\end{array}\right.$(4)

where $\oplus$ indicates the skip connection operation, $\mathit{Upsample}\left(.\right)$ represents the processed feature by Up-sampling module, $\mathit{CFF}\left(.\right)$ represents the processed feature by the CFF module, and $l^i$ denotes the feature map generated from $\mathit{ith}$ layer during the decoder process.

3.4. Predication enhancement module

To make full use of the feature detail obtained from the hidden layers for the final result, inspired by the Deeply-Supervised (Li, Huang, et al. 2022), we proposed the prediction enhancement module (Figure 5). Rather than obtaining the result from the only last convolutional layer, we also predict the extraction result using the feature map with the 1/4 size of the original image generated in third stage during the encoder process. The prediction results of the two stages are fed into the PE module to obtain the final building extraction results. Furthermore, the prediction enhancement module can directly supervise the hidden layer to ensure the network can converge more quickly. The feature map generated in third stage of a decoder is chosen to balance the global semantic and spatial detail information. Figure 5 depicts the prediction enhancement module and Residual Block. The prediction results from the two stages are fed into the PE module, and linear interpolation is operated to make the feature maps have the same size. Then the feature maps concatenated are normalized by BN operation, further fused by two sets of Conv-BN-ReLU operation and residual block operation. The final building extraction result is obtained by a 1 × 1 convolution.

Figure 5. The structure of the prediction enhancement (PE) module. (a) Prediction enhancement module. (b) Residual block.

3.5. Model implementation and loss function

All experiments run on NVIDIA GeoForce RTX 3090 GPUs. The operating system is Ubuntu 20.04. And all programs were implemented using Pytorch library. We used Adam as the optimizer. The initial learning rate is 0.001 and is decreased to 0.7 of the current learning rate if the accuracy does not increase on the validation set. The training process stops when the epoch reaches 100 or the learning rate is lower than 1e-7. During the training, random horizontal and vertical flipping were selected as data augmentation strategies. The sigmoid binary cross-entropy loss was selected as the loss function of CFF-Net, considering the task of the pixel-wise binary classification. For the position $(\mathit{i},\mathit{j})$, the loss can be computed as:

(5) $\begin{array}{c}\mathit{Los}{s}_{\mathit{ij}}=[{\mathrm{y}}_{\mathit{ij}}\ast \ln P_{\mathit{ij}}+(1-{\mathrm{y}}_{\mathit{ij}})\ast \ln (1-P_{\mathit{ij}})]\end{array}$(5)

(6) $\begin{array}{c}{P}_{\mathit{ij}}=\mathrm{Sigmoid}(\mathit{logis}{t}_{\mathit{ij}})=\frac{1}{1+{\mathrm{e}}^{-\mathit{logis}{\mathit{t}}_{\mathit{ij}}}}\end{array}$(6)

where $\mathit{logis}{t}_{\mathit{ij}}$ represents the predicted result, and ${\mathrm{y}}_{\mathit{ij}}\in \{0,1\}$ indicates the ground truth. The sigmoid function was applied to logits to ensure that $\mathit{P}\in [0,1]$. The loss value is the average of $\mathit{logis}{t}_{\mathit{ij}}$ at all positions for an input image.

4. Experimental and analysis

4.1 datasets

To evaluate the performance of the proposed CFF-Net, we conducted a comprehensive experiment on two publicly available building datasets, including the WHU building dataset (Shunping, Wei, and Lu 2018) and the Inria aerial image dataset (Maggiori et al. 2017).

The WHU building dataset includes aerial and satellite subsets. In our experiment, we selected the aerial subset, which overs over a 450 km2 area with about 22,000 buildings. The buildings with various appearances and scales can evaluate the robustness of the proposed algorithm. Each image has three bands, corresponding to Red (R), Green (G), and Blue (B) wavelengths, with each image size of 512 × 512 pixels. And the spatial resolution of the R-G-B composite aerial images is 0.3 m. There are a total of 8188 tiles of images, including 4736, 2416, and 1036 tiles as training, test, and validation data sets, respectively. We conducted our experiment at its original provided data set partitioning.

The Inria aerial image dataset includes 360 aerial images with 5000 × 5000 pixels from 5 cities (Austin, Chicago, Kitsap, Tyrol, and Vienna). The images cover dissimilar urban buildings, ranging from dense urban villages to alpine towns. Only 180 training images have labels, and we cropped them into the images with 512 × 512 pixels. We got a total of 12,095 tiles, and these images were randomly assigned to the training set and the test machine in an 8:2 ratio. Finally, there are 9676 and 2419 tiles as training and validation data sets.

4.2. Evaluation metrics

The building extraction is the classification result of each pixel, so we apply the pixel-level metrics to assess the performance of the methods, including precision, recall, the F1-score, and the Intersection-Over-Union (IoU). Precision represents the percentage of the correct areas that are predicted as buildings, while recall indicates the proportion of the correct predictions in the building ground truths. The F1-score is the harmonic mean of precision and recall; the IoU represents the ratio of the intersection areas over the union areas between the building predictions and the ground truths. There are four classifying conditions: True prediction on a Positive sample (TP), False prediction on a Positive sample (FP), True prediction on a Negative sample (TN), and False prediction on a Negative sample (FN). Equations are given as follows:

(7) $\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$(7)

(8) $\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$(8)

(9) $\mathrm{F}1=\frac{2\ast \mathrm{Recall}\ast \mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}$(9)

(10) $\mathrm{IoU}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}}$(10)

4.3. Performance evaluation

4.3.1. Comparison methods

We selected five state-of-the-art methods, including U-net, BiSeNet v2, Deeplab v3, HRNet, and SegNet, and four recent methods, including SiU-Net, BuildFormer, CLNet, and BalNet, for comparison. We will introduce them briefly.

U-net was proposed at 2015 for Biomedical Image Segmentation (Ronneberger, Fischer, and Brox 2015), which includes an encoder, decoder, and skip connection. It won the ISBI cell tracking challenge 2015 and has good performance in semantic segmentation and building extraction tasks. BiseNet v2 (Yu et al. 2021) is a lightweight semantic segmentation network that improved on BiseNet. BiSeNet v2 processes these spatial details and classification semantics separately to achieve highly accurate and efficient semantic segmentation in real-time. Deeplab v3 (Chen et al. 2017) used the Multi-Grid strategy based on the previous model and adds batch normalization to the ASPP module. The ASPP module with different atrous rates can effectively capture multi-scale information. The method obtained the SOTA performance at VOC2012 at that time. HRNet (Sun et al. 2019) is a high-resolution network that maintains high-resolution representations throughout the processing of the network. SegNet (Badrinarayanan, Kendall, and Cipolla 2017) explicitly proposes the encoder-decoder architecture and proposed the maxpool index to decode, which can save memory.

SiU-Net (Shunping, Wei, and Lu 2018) is a Siamese U-Net with shared weights in two branches and original images and their down-sampled counterparts as inputs, which significantly improves the segmentation accuracy, especially for large buildings. BuildFormer (Wang et al. 2022) is a novel ViT with a dual-path structure, including a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. CLNet (Chen et al. 2022) is fully convolutional neural network consisting of the spatial fusion module, the focus enhancement module, and the feature decoder module. BalNet (He and Jiang 2021) is a fully convolutional network which embeds a boundary learning task and the spatial variation fusion module, adopts separable convolution with a larger kernel, and utilizes the convolutional block attention module.

4.3.2. Quantitative comparison

The quantitative evaluations of our proposed CFF-Net and the comparison methods on the Inria aerial dataset are shown in Table 1, and the evaluations of the WHU building dataset are shown in Table 2. Our proposed CFF-Net outperformed the comparative methods on the two datasets, which demonstrates the effectiveness of our proposed method. CFF-Net reached the highest IoU and F1 for both two datasets. Compared with BuildFormer proposed recently, CFF-Net increased the IoU by 0.12% and 0.43%, respectively. CFF-Net reached the highest and second highest Recall on the Inria Dataset and WHU building dataset, respectively. The higher Recall means that the more buildings could be extracted. We noticed that CLNet had a poor IoU, but reached the highest Precision on both datasets, indicating that it had more omissions in the process of building extraction. In terms of Precision, for the WHU dataset, CFF-Net achieved the second highest Precision after CLNet, which comes at the cost of missing many buildings. For the Inria dataset, there is the same phenomenon that Bal-Net and CLNet have the highest Precision but the low IOU. CFF-Net has a high Precision while having the highest IOU and F1. On the other hand, the proposed CFF-Net algorithm is able to maintain a high level of identifying the buildings and non-buildings in the process of building extraction. BuildFormer, obtaining multi-scale semantic information from the dual-path structure of ViT and CNN, reached the second highest IoU and F1 on the both datasets, which demonstrates it is crucial for building extraction to acquire and fuse multi-scale feature.

Table 1. Quantitative comparison with SOTA methods on the Inria dataset.

Method	IoU(%)	Recall(%)	Precision(%)	F1%)
Bal-Net	70.57	74.67	92.78	82.74
CLNet	72.60	76.78	93.03	84.13
U-Net	77.12	87.37	87.37	87.08
HRNet	78.14	87.36	88.09	87.72
SiU-Net	78.37	87.49	88.26	87.87
SegNet	78.67	87.35	88.78	88.06
BiSeNet v2	79.17	87.76	89.00	88.37
Deeplab v3	79.91	88.21	89.46	88.83
BuildFormer	80.29	88.56	89.59	89.07
Ours	80.41	89.01	89.27	89.14

Bolded font represents the highest value and underlined font represents the second highest value.

Table 2. Quantitative comparison with SOTA methods on the WHU dataset.

Method	IoU(%)	Recall(%)	Precision(%)	F1%)
CLNet	79.08	81.24	96.74	88.32
Bal-Net	88.17	92.65	94.80	93.71
SegNet	89.51	93.67	95.27	94.46
BiSeNet v2	89.88	94.15	95.19	94.67
HRNet	89.98	93.69	95.78	94.72
U-Net	90.37	95.27	94.61	94.94
SiU-Net	90.77	95.65	94.68	95.16
Deeplab v3	90.95	95.25	95.26	95.26
BuildFormer	91.31	95.15	95.77	95.46
Ours	91.74	95.32	96.06	95.69

Bolded font represents the highest value and underlined font represents the second highest value.

4.3.3. Qualitative comparison

To further analyze the limitations and advantages of the compared methods, we visualized some prediction examples of the top five IoU methods on the two publicly available datasets. As shown in Figures 6 and 7, consistent with the quantitative metrics, our proposed CFF-Net can obtain more precise building extraction results than the other competing methods.

Figure 6. Examples of building extraction results of the proposed CFF-Net and other comparison methods on the inria dataset.

Figure 7. Examples of building extraction results of the proposed CFF-Net and other comparison methods on the WHU dataset.

In Figure 6 row 3 and Figure 7 row 2, there are surface features with similar characteristics to buildings in the remote sensing images of the two datasets, and the comparison methods all incorrectly recognize them as buildings. Our proposed CFF-Net in this paper is able to distinguish the heterogeneity of different features well through the fusion of multi-stage information and identify the real buildings. In Figure 7 (row 1 and row 3), there are small holes in the building extraction results of Deeplab-v3 and U-Net, but the CFF-Net proposed is able to extract the whole building more successfully. For buildings with complex shapes, CFF-Net proposed can present their complex boundaries completely, as shown in Figure 6 (row 1, row 4) and Figure 7 (row 4), and the results obtained by other comparative algorithms are quite different from the ground truth. Moreover, in Figure 6 (row 2), CFF-Net is capable to detect small buildings accurately and obtain the complete boundaries, while other comparative methods occur omission or get broken buildings.

4.4. Ablation studies

To examine the functionality of the CFF module and PE module proposed in our CFF-Net, we set up a series of ablation experiments. We tested baseline, baseline plus CFF module, baseline plus CFF module, and PE module, which is CFF-Net, on the Inria and WHU datasets. The results of the ablation experiments on the two datasets are shown in Tables 3 and 4, respectively.

Table 3. Ablation experimental results on the Inria dataset.

Method	IoU(%)	Recall(%)	Precision(%)	F1%)
baseline	79.37	88.30	88.69	88.49
baseline + PE	79.79	87.43	90.13	88.767
baseline + CFF	79.98	87.98	89.80	88.87
baseline + CFF + PE	80.41	89.01	89.27	89.14

Bolded font represents the highest value.

Table 4. Ablation experimental results on the WHU dataset.

Method	IoU(%)	Recall(%)	Precision(%)	F1%)
baseline	90.89	95.38	95.07	95.23
baseline + PE	91.36	95.12	95.85	95.48
baseline + CFF	91.32	95.40	95.53	95.46
baseline + CFF + PE	91.74	95.32	96.06	95.69

Bolded font represents the highest value.

As shown in Table 3, the baseline obtained 79.37% IoU on the Inria dataset. Adding the CFF module achieved a 0.61% improvement in IoU. And the Precision increased from 88.69% to 89.80%, indicating that the fusion of multi-scale feature information can reduce the false identification of buildings. Furthermore, the addition of PE module for baseline and baseline plus CFF module increases the IoU by 0.42%, 0.43% respectively, which indicates that the combined effect of multi-stage prediction results can extract buildings more accurately. As shown in Table 4, for the WHU dataset, the addition of CFF module and PE module also obtain an increase of IoU by 0.43%, 0.39% and the combination of the two modules gets an increase of IoU by 0.85%, which illustrates the effectiveness of our proposed method.

To further illustrate the contribution of the proposed CFF module, which fuses the cross-scale information for the final building extraction, as shown in Figure 8, we visualized the feature maps of baseline and baseline with the CFF module in the process of building extraction on the WHU dataset. The fusion of multi-scale features enables the network to have a more accurate perception at any scale. As shown in Figure 8 (row 2, row 3, and row 4), the CFF module is able to suppress non-building regions while having better attention to complex buildings. In addition, the fusion of multi-scale information enables the network to have more accurate judgments on regions that are easily ignored. As illustrated in Figure 8 (row 1 and row 5), the network is able to focus more on the buildings at the image boundaries, making the final results more accurate.

Figure 8. Feature visualization from baseline and baseline with CFF module on the WHU dataset. The first column presents feature visualization and the second column presents the prediction result of baseline and baseline with CFF module in (c) and (d).

4.5. Complexity of CFF-Net

To validate the efficiency and accuracy of the proposed CFF-Net algorithm, as shown in Figure 9, we visualized the IoU, parameters, and model size of the CFF-Net with the compared SOTA methods. As shown in Figure 9, CFF-Net with 27.08 M parameters achieved the highest IoU, 91.74%. Deeplab-v3 obtains 90.95% IoU with 40.29 M parameters, which has much more parameters than CFF-Net. In summary, our proposed CFF-Net achieves higher accuracy and owns lower complexity compared to the competing SOTA methods.

Figure 9. Comparison of complexity and accuracy of CFF-Net and the comparison methods.

5. Conclusions

The difficult identification of buildings at the boundary of images and confusion between buildings and non-buildings are the main challenges in building extraction from remote sensing. To address the above challenges, we proposed a Cross-stage Feature Fusion Network (CFF-Net) for building extraction. The Cross-stage Feature Fusion (CFF) module was proposed to fuse the feature maps generated in different stages, making the network a better grasp of both detailed features and global information, and accurately identifying buildings in complex areas. Furthermore, the results are predicted using the feature map generated in the intermediate stages and the last convolutional layer. We fused the two prediction results to improve the final extraction results through the Prediction Enhancement (PE) module.

Experiments on the Inria and WHU datasets demonstrate the effectiveness and robustness of the proposed method for building extraction from remote sensing images. CFF-Net achieves the highest IoU, 80.41% and 91.74%, on the two datasets, respectively, and has the best extraction results. The ablation experiments confirm the effectiveness of the proposed CFF module and PE module, where the CFF module can accurately guide the network to focus on important regions. In addition, CFF-Net achieves a better balance between building extraction performance and network complexity/efficiency. The Precision of our method does not achieve the highest value, which indicates that our algorithm can misjudge when recognizing buildings, which deserves further research. Moreover, further work will improve the network’s performance while simplifying the structure, and apply CFF-Net to other remote sensing tasks.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Additional information

Funding

This article was supported by the following projects: National Natural Science Foundation of China [42090012], Guangxi Science and Technology Plan Project [Guike 2021AB30019], Hubei Province Key R&D Project [2022BAA048], Sichuan Province Key R&D Project [2022YFN0031, 2023YFN0022, 2023YFS0381], Zhuhai Industry-University-Research Cooperation Project [ZH22017001210098PWC], Shanxi Provincial Science and Technology Major Special Project [202201150401020], Guangxi Key Laboratory of Spatial Information and Surveying and Mapping Fund Project [21-238-21-01].

Notes on contributors

Xiaolong Zuo

Xiaolong Zuo received the B.S. degree in spatial information and digital technology from Wuhan University, Wuhan, China, in 2022. He is currently working toward the Ph.D. degree in the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University. His research field includes image processing and the applications of deep learning in remote sensing.

Zhenfeng Shao

Zhenfeng Shao received the Ph.D. degree in photogrammetry and remote sensing from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan, China, in 2004. Since 2009, he has been a Full Professor with LIESMARS, Wuhan University. He has authored or coauthored over 50 peer-reviewed articles in international journals. His research interests include high-resolution image processing, pattern recognition, and urban remote sensing applications.

Jiaming Wang

Jiaming Wang received his B.S. degree from Wuhan University, Wuhan, China, in 2015, M.S. degree from the Geographic Information Science and Technology, Georgia Institution of Technology, Atlanta, GA, USA, in 2016, and Ph.D. degree in Geography from the University of South Carolina, Columbia, SC, USA, in 2020. He is currently an Assistant Professor with the Department of Environmental Sciences at Emory University. His research interests cover GeoAI, deep learning, and human-environmental interactions.

Xiao Huang

Xiao Huang received the Ph.D. degree from the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, in 2022. He is currently working at the School of Computer Science and Engineering, Wuhan Institute of Technology. His research interests include image/video processing, computer vision, and artificial intelligence.

Yu Wang

Yu Wang received a master’s degree in engineering from Wuhan Institute of Technology, Wuhan, China, in 2021. He is currently pursuing the Ph.D. degree under the supervision of Prof. Zhenfeng Shao with the State Key Laboratory for Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan. His research field includes image/video processing and computer vision.