MDSNet: a multiscale decoupled supervision network for semantic segmentation of remote sensing images

ABSTRACT

Recent deep-learning successes have led to a new wave of semantic segmentation in remote sensing (RS) applications. However, most approaches rarely distinguish the role of the body and edge of RS ground objects; thus, our understanding of these semantic parts has been frustrated by the lack of detailed geometry and appearance. Here we present a multiscale decoupled supervision network for RS semantic segmentation. Our proposed framework extends a densely supervised encoder-decoder network with a feature decoupling module that can decouple semantic features with different scales into distinct body and edge components. We further conduct multiscale supervision of the original and decoupled body and edge features to enhance inner consistency and spatial boundaries in remote sensing image (RSI) ground objects, enabling new segmentation designs and semantic components that can learn to perform multiscale geometry and appearance. Our results outperform the previous algorithm and are robust to different datasets. These results demonstrate that decoupled supervision is an effective solution to semantic segmentation tasks of RS images.

KEYWORDS:

• Semantic segmentation
• remote sensing images
• edge supervision
• multiscale

1. Introduction

Semantic segmentation plays an important role in RSI applications, such as resource exploration (Shirmard et al. 2022), environmental monitoring (Li et al. 2020b), natural disaster prediction (Munawar, Hammad, and Waller 2022), etc. Prior to the advent of semantic segmentation, manual visual interpretation was the primary technique employed for segmenting RSI (Xu et al. 2019). However, this method has limitations in processing large volumes of RS data, and it is unable to achieve efficient and fast extraction of semantic information. With the increasing volume of RS data being collected, manual segmentation methods must be reconsidered to meet the demands of the era of big data. As a result, the need for automated segmentation techniques, such as semantic segmentation, is crucial for overcoming these limitations and maximizing the potential of RS data.

Recent advancements in deep learning have revolutionized the field of computer vision and enabled us to accurately allocate every pixel within an image to a specific class. Semantic segmentation has emerged as a prominent research focus in this field, particularly in the context of RSI. RS semantic segmentation involves segmenting satellite images at the pixel level based on various factors, including color, morphology, and texture features. These segmented image features, such as buildings, vegetation, rivers, and roads, are labeled appropriately. The inception of fully convolutional networks (FCNs) (Long, Shelhamer, and Darrell 2015) marked a significant milestone in integrating deep learning-based approaches for semantic segmentation, leading to many promising advancements since then.

Numerous studies have explored the potential of FCN-based approaches in semantic segmentation and have shown promising results (Yuan, Shi, and Gu 2021). However, these approaches still face some limitations. One key limitation is that the receptive field (RF) of FCNs increases linearly with depth, which can hinder their ability to capture long-range semantic correlations among image pixels (Luo et al. 2016). This, in turn, makes pixel classification difficult, particularly when there is ambiguity and noise in object boundaries. Another shortcoming of FCNs is that they use downsampling operations, which can lead to blurry predictions due to the loss of edge details at low resolution. This can result in segmented objects that look blobby with inadequate boundary details, leading to a decrease in performance, particularly for smaller objects. These issues are particularly significant when working with remotely sensed images, which often involve complex scenes with a high level of heterogeneity and variability. To overcome the limitations of RF in FCNs, researchers have proposed various methods, including dilation convolution (Yu and Koltun 2015) and pyramidal pooling modules (Zhao et al. 2017). Some approaches incorporate low-level features that contain edge information with high-level features (Zhang et al. 2018), while others refine the output directly (Bertasius, Shi, and Torresani 2016). However, these methods may not fully consider the intricate interplay between the contours of objects and their interior regions.

Recently, Li et al. (2020) introduced an innovative semantic segmentation approach, DecoupleSegNet. This method divides the feature map into body and edge features. The goal is to obtain smooth representations of the interior areas of objects with low frequencies using body features while capturing high-frequency details along object boundaries with edge features. DecoupleSegNet employs separate supervised signals for each subtask, helping tackle body consistency and edge preservation issues in semantic segmentation. However, implementing DecoupleSegNet in high-resolution RSIs poses challenges. First, it necessitates improving the intricate texture features of high-resolution RSIs, which contain a wealth of texture information. Second, pixel stacking is common in RSIs, leading to pixels with the same label displaying varied characteristics. Since DecoupleSegNet solely focuses on segmenting low-resolution features, its effectiveness in this task needs enhancement. Consequently, there is a requirement to integrate detailed information from multiple scales to identify better subtle structural differences between classes in high-resolution RSIs.

In this study, we present an RSI semantic segmentation model utilizing a multiscale decoupled supervision framework. Inspired by the DecoupleSegNet architecture, our framework separates semantic features into body and edge components across different scales, which are then supervised independently. Unlike the DecoupleSegNet approach, our framework effectively captures both local and global background information by considering body and edge features at various scales. To further improve the performance of our model, we designed a novel hybrid loss function for edge supervision, which combines Binary Cross Entropy (BCE) and Intersection over Union (IOU) at both the pixel and map levels. This hybrid loss function enhances the network’s sensitivity to edge pixels, preserves edge details, and ultimately boosts segmentation accuracy. In summary, our contributions are as follows:

1. We propose MDSNet, an RSI semantic segmentation framework with multiscale decoupled supervision, which can effectively capture local and global features in complex backgrounds. Specifically, MDSNet utilizes the characteristics of RS objects to extract more differences between different classes based on supervising the body and edge features of RSIs at different scales.

2. To enhance the performance of edge segmentation, we introduce a novel joint loss function for edge supervision that merges BCE and IOU at both pixel and map levels. Considering the significant imbalance between classes (with the edge features being sparse), it calculates the posterior probability ratio between the edge class and the body classes, thereby further magnifying the probability distinction between the edge and body classes. Moreover, this innovative approach can be implemented in all semantic segmentation techniques incorporating nested edge detection.

3. Extensive experimental results on the Beijing Land Use (BLU) and the Potsdam datasets demonstrate the effectiveness of our model. Our findings are in remarkable alignment with theoretical predictions, further solidifying the validity and reliability of our approach.

2. Related work

Semantic segmentation has gained significant attention due to its wide range of applications. With the introduction of Fully Convolutional Networks (FCNs) for semantic segmentation, deep learning-based methods have gradually taken the lead in this field. Subsequently, a series of encoder-decoder based network architectures have been proposed, including U-Net (Ronneberger, Fischer, and Brox 2015), SegNet (Badrinarayanan, Kendall, and Cipolla 2017), PSPNet (Zhao et al. 2017), and the Deeplab series (Chen et al. 2018). These methods extract features progressively through the encoder part and perform upsampling and feature fusion through the decoder part, effectively capturing the contextual information in images. However, they still face challenges in achieving high segmentation accuracy for small or complex textured objects. In recent years, there has been a growing trend of using Transformer-based methods for semantic segmentation, such as BANet (Wang et al. 2021a), SegFormer (Xie et al. 2021), TransUNet (Chen et al. 2021), DC-Swin (Wang et al. 2022) and so on. These methods fully leverage the advantages of Transformers in capturing global context and modeling long-range dependencies, enabling the networks to better handle image semantic segmentation tasks. In addition, WiCoNet (Ding et al. 2022) addresses the context modeling issue in high-resolution RSI semantic segmentation by employing a wide-context Transformer. However, due to the locality of the self-attention mechanism in Transformers, it struggles to effectively capture edge information far from the center pixel, presenting challenges in accurately identifying edge pixels and preserving fine details.

Several recent approaches have recognized the importance of edge information in semantic segmentation, as demonstrated by GSCNN (Takikawa et al. 2019) and Zhen et al. (2020). These methods consider the ability to perceive object shapes as a priority and aim to improve the accuracy of semantic segmentation and the capture of fine-grained edges through edge supervision. However, these networks may tend to overlook the sparse nature of edge pixels, which can limit the effectiveness of edge supervision in improving segmentation performance. Further research is needed to address these limitations and develop more sophisticated approaches that can better leverage edge information to enhance segmentations while also overcoming the challenges of sparsity in edge pixels.

Meanwhile, other innovative methods for semantic segmentation are also emerging continuously. PWC-Net (Sun et al. 2018) employs a pyramid optical flow estimation approach, leveraging multiple-scale feature maps and a pyramid-based flow estimation process for multiscale flow estimation. Zhou et al.(2021) utilize flow learning between two features to align the positions of high-level and low-level feature mappings. Different from these methods, we utilize a flow field to warp the features toward the interior of the object. Moreover, RSS-PV (Zhou et al. 2022) enhances the ability to capture unique features and changes within each semantic class by defining prototype samples for each class, resulting in more accurate and accurate semantic segmentation results. Wang et al. (2021) enhance segmentation performance by leveraging the cross-image pixel contrast in a collection of images. Zhou et al. (2021a) improve weakly supervised semantic segmentation through group-wise learning. These methods offer new insights and directions for semantic segmentation but still face challenges in handling complex boundary details in RSIs. Recently, Segmenting Anything (SAM) (Kirillov et al. 2023) has achieved promising results in general segmentation tasks. However, SAM only performs segmentation on visual objects and lacks detailed semantic information.

Although these studies have improved the accuracy of semantic segmentation to some extent, none of them has fully considered the relationship between the object's interior and its surrounding boundary. In light of this, DecoupleSegNet proposes an innovative decoupling method to separate the feature maps of semantic segmentation into body and edge features and designs a specialized supervisory framework. However, in the DecoupleSegNet method, the feature map may lose some detailed information after multiple downsampling operations. This limitation poses significant challenges in segmenting RSIs, which often contain abundant texture information, complex land boundaries, and objects of various scales. Furthermore, DecoupleSegNet also fails to fully address the sparsity problem of edge pixels, which may be disregarded due to insufficient saliency. Consequently, developing of a network suitable for RSI semantic segmentation remains a challenging issue within the field of semantic segmentation.

3. Methods

Compared to prior semantic segmentation methods for RSIs, our approach employs a unique approach by separating the problem into two fundamental components: the low-frequency spatial component, which accounts for more gradual changes, and the high-frequency details component, which accounts for more abrupt changes. By supervising both components, we have achieved superior performance in semantic segmentation for RSIs. Additionally, we have devised a multiscale output structure that captures object details across differing scales within RSI. A detailed overview of our network architecture and supervision framework is presented below.

3.1. Decoupling module

As shown in Figure 1, given a feature map $F\in R^{C\times H\times W}$, where $C$ is the channel dimensions, and $H\times W$ represents the spatial resolution, the decoupling module outputs body features and boundary features of the same size. To achieve this, we use a body feature generation module that aggregates detailed characterization of the ground objects to create a distinct body part for each object. We first generate the body feature using this module and then obtain the edge features by subtracting the body features from the original feature map. In other words, $F_{body}=\alpha (F)$, $F_{edge}=F-F_{body}$, where $\alpha$is the body generation module.

Figure 1. Illustration of our decoupling module. The decoupling module consists of two steps: first, the body feature generation module generates body features, and then edge features are obtained through explicit subtraction.

Our body feature generation module is inspired by DecoupleSegNet (Li et al. 2020). We consider the low-resolution feature $F_{low}$ corresponding to the feature $F$ as representing the central part of the object's interior, which we view as a collection of seed points. Specifically, We utilize two consecutive convolutional layers with a kernel size of 3 × 3 and a stride of 2 to compress $F$ into a low-frequency $F_{low}$, and then upsampled it to the original size by bilinear interpolation. Then, we concatenate $F$ and $F_{low}$, and utilize a convolutional layer with a kernel size of 3 × 3 to predict the flow field $\delta \in R^{H\times W\times 2}$. The expression for flow field generation can be represented as follows: (1) $\delta =f_{conv}(F,F_{low}^{\Uparrow })\in R^{H\times W\times 2}$(1) where $\Uparrow$ represents upsampling with bilinear interpolation, $f_{conv}$ represents the convolutional layer.

In the feature warping part, the point $p$ at each position can be mapped to a new point $\hat{p}$ on a standard spatial grid $\mathrm{\Omega }$ through $\hat{p}=p+\delta$(p) in which $\delta (p)$ represents the point at the same position in the flow field. This results in a warped spatial grid ${\mathrm{\Omega }}_w$, which is then used to approximate each point $p_x$ in $F_{body}$ by using differentiable bilinear sampling. The sampling mechanism was introduced in the spatial transformer networks (Jaderberg et al. 2015). This mechanism allows for linear interpolation of the values of the four pixels closest to the target pixels. The interpolation formula is as follows: (2) $F_{body}(p_{body})=\sum _{p\in N(p_i)}{\omega }_{p}F(p)$(2) where ${\omega }_p$ denotes the weights assigned to the bilinear kernel on the deformed spatial grid ${\mathrm{\Omega }}_w$, $N(p_i)$ refers to the neighboring pixels that are associated with the point $p_i$, $F(p)$ indicates the value of the point $p$ in the feature map $F$.

Our edge feature generation module obtains edge features by subtracting the body features from the original feature map $F$. This approach differs from DecoupleSegNet, which fuses edge features with low-level features. We believe that merging edge features with low-level features may introduce non-edge information, which can impact edge supervision. In contrast, direct display subtraction offers a straightforward and effective approach to obtaining edge features.

3.2. Network architecture

As shown in Figure 2, our proposed model is composed of a densely supervised encoder-decoder network with U-Net-like architecture, as well as six feature decoupling modules. By utilizing skip connections across different layers, this network is able to capture both broad contextual information and detailed low-level features, which addresses the problem of losing detailed information that was associated with decoupled edge features in DecoupleSegNet.

Figure 2. An illustration of the complete structure of our proposed network, which incorporates a decoupling module targeting specific features.

In our model, the encoder is comprised of an input convolutional layer and six basic res-blocks. The input convolutional layer has 64 filters with a 3 × 3 kernel size and a stride of 1. The following four levels of the input convolutional layer use ResNet-34 to extract features. However, we don't utilize pooling operations after the input layer, unlike ResNet-34, which enables our network to acquire higher-resolution feature maps in the early layers while reducing the overall perceptual domain. To match the same perceptual field as ResNet-34, we added two extra stages following the fourth stage of ResNet-34, consisting of three basic res-blocks each and employing 512 filters after a non-overlapping maximum pooling layer of size 2.

To augment the global contextual information, we introduced an intermediate stage between the encoder and decoder. This stage includes three convolutional layers with 512 dilated 3 × 3 filters. After every convolutional layer, we applied batch normalization and ReLU activation functions.

The decoder's architecture is symmetrical to that of the encoder comprising three convolutional layers in each stage. Each layer is followed by a batch normalization layer and ReLU activation function. At the input of each stage, a concatenated feature map is formed by upsampling the output of the previous stage and the corresponding stage in the encoder. In our model, the last layer of each decoder is partitioned into two separate components, namely, $F_{body}$ and $F_{edge}$, by the decoupling module.

Taking inspiration from the HED approach proposed by Xie and Tu (2015), we adopt a multiscale supervision framework to enhance our network’s ability to effectively learn multilevel and multiscale features. As discussed by Lee et al. (2015), this approach can boost the learning efficiency and generalization performance of neural networks by providing extra and more immediate training signals to the preceding layers through intermediate network outputs during training. By incorporating supervision at multiple scales, our model can effectively capture larger structures and utilize the RF by encoding significant features in the deeper layers.

Specifically, we provide supervision for each decoder-level feature as well as the decoupled body and edge features. To generate the side output, each feature at the decoder level, along with separate body and edge features, undergoes a 3 × 3 convolutional operation and is then upsampled to the input image size using bilinear interpolation. Furthermore, we process the edge features through the sigmoid function. This process results in our model generating corresponding probability maps for the features and their decoupled body and edge features at six scales. The specific supervisory framework for the probability maps is described in detail in Section 3.3 of the paper.

3.3. Supervision module

Our proposed supervisory framework consists of three parallel branches, $L_{seg}$, $L_{body}$ and $L_{edge}$. The $L_{seg}$ branch is dedicated to supervising the segmentation probability map at each decoder level, the $L_{body}$ branch supervises the body features at each decoder level and the $L_{edge}$ branch supervises the edge features at each decoder level. The total loss is calculated as: (3) $L={\lambda }_1\sum _{i=1}^6{L}_{seg}(s_i,\hat{s})+{\lambda }_2\sum _{i=1}^6{L}_{body}(s_{bod{y}_i},\hat{s})+{\lambda }_3\sum _{i=1}^6{L}_{edge}(b_i,s_i,\hat{b},\hat{s})$(3) where $\hat{s}$ represents ground truth (GT), $\hat{b}$ is the GT binary mask which is generated by $\hat{s}$. $b_i$_, $s_{bod{y}_i}$ and $s_i$ represent segmentation i-th map prediction from $F_{edge}$_, $F_{body}$ and $F_{seg}$ respectively. ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ represent the three hyperparameters associated with these tasks. Since all three tasks involve CE loss, we set them to the default value of 1. $L_{seg}$ is a cross-entropy loss in the segmentation task. Then, we discuss the losses of $L_{body}$ and $L_{edge}$ in detail as follows.

For supervision of the $F_{body}$ part, we relaxed the object boundaries during the training process and utilized a boundary relaxation loss that only samples pixels within the object (Zhu et al. 2017). Specifically, we defined boundary pixels as any pixel with neighbors by different labels. For simplicity, we consider the scenario where a pixel is being classified on the boundary between two classes, $A$ and $B$. Rather than optimizing for the likelihood of the target label given by the annotations, we suggest optimizing for the likelihood of the combination of $A$ and $B$. As they are mutually exclusive, our objective is to maximize the union of both classes: (4) $P(A\cup B)=P(A)+P(B)$(4) where $P()$ refers to the probability distribution obtained through the softmax function for each category. Then, we define the loss of $L_{body}$ as follows: (5) $L_{body}(s_{body},\hat{s})=-\log \sum _{C\in N(p_i)}P(C)$(5)

In which, $N(p_i)$ represents the collection of classes inside a 3 × 3 neighborhood surrounding pixel $p_i$. $C$ represents the complete set of classes inside a 3 × 3 region centered at pixel $p_i$.

In the edge supervision portion, we observed that many semantic segmentation methods utilizing nested edge supervision employ Binary Cross Entropy (BCE) loss for guiding the edge prediction maps. Unfortunately, this method neglects the inherent sparsity of edge pixels, which often results in insignificant edge losses. Our experiments showed that merely increasing the weight of BCE loss was insufficient for enhancing the network's accuracy. To tackle this challenge, we propose a hybrid loss function that fuses BCE and Intersection over Union (IOU) loss for effectively supervising edge pixels. The IOU loss, when supervising data with a limited number of positive samples (such as those found in edge pixels), causes the network to prioritize segmentation boundary accuracy more. Consequently, the IOU loss allows the network to better capture the shape and boundary information of objects. Our combined loss function facilitates pixel-level and map-level supervision of boundaries in remote sensing images (RSI), ultimately improving the network’s sensitivity to edge pixels. Furthermore, we adopt a comprehensive loss function encompassing boundary priors deduced from edge features. Since the boundary pixels of objects in RSI tend to be particularly challenging to classify, our approach is centered on these pixels and implements online hard example mining using a predetermined edge threshold during the training process. Therefore, $L_{edge}$ includes $L_{bce}$ and $L_{iou}$, which respectively refer to the categorization of edge pixels and the depiction of object contours. $L_{ohem}$ mines the edge parts of objects in RSI through online hard example mining. The expression of $L_{edge}$ is shown as follows: (6) $L_{edge}(b,s,\hat{b},\hat{s})={\lambda }_4{L}_{bce}(b,\hat{b})+{\lambda }_5{L}_{iou}(b,\hat{b})+{\lambda }_6{L}_{ohem}(s,\hat{s},b)$(6) where ${\lambda }_4$, ${\lambda }_5$, and ${\lambda }_6$ are three hyper-parameters, and we set them to default values of 1. The calculation of $L_{bce}$ and $L_{iou}$ is as follows in the equation: (7) $L_{bce}(b,\hat{b})=-\hat{b}log(b)+(1-\hat{b})log(1-b)$(7) (8) $L_{iou}(b,\hat{b})=1-{\displaystyle \frac{\hat{b}\cdot b}{\hat{b}+b-\hat{b}\cdot b}}$(8) To $L_{ohem}$, it merges the edge prior from $\hat{b}$ with the weighted bootstrap cross-entropy. The calculation of $L_{ohem}$ is as follows: (9) $L_{ohem}(s,\hat{s},b)=-{\displaystyle \frac{1}{K}\sum _{i=1}^N{w}_i\cdot I\left[s_{i,{\hat{s}}_i}⟨t\cap \sigma (b_i)⟩t_b\right]\cdot log{s}_{i,{\hat{s}}_i}}$(9) where $K=0.10\times N$, in which $N$ represents the number of pixels in the RSI, ${\hat{s}}_i$ represents the target class of pixel $i$, $s_{i,j}$ is the predicted posterior probability that pixel $i$ belongs to class $j$, $w_i$ is the weight value of the corresponding class of pixel $i$. $I[x]=1$ if $x$ is true or $I[x]=0$ if x is false. $\sigma (x)$ is the sigmoid function to determine if pixel $i$ falls on the object boundary. We use a threshold $t$ to select the pixel-matching probability and a threshold $t_b$ to determine whether pixel $i$ is on the boundary.

We achieve separate supervision of the object body and boundary by measuring sampled pixels from different parts of the RSI separately. By doing those, we can improve the performance of the prediction results.

4. Experiments

4.1. Datasets

To assess the effectiveness of the proposed approach, we conducted tests on two datasets, including the Beijing Land-Use dataset (Ding et al. 2022) and the Potsdam dataset. The Beijing Land-Use dataset was obtained in June 2018 using the Beijing-2 satellite, which twenty-first Century Space Technology Co provided. The dataset consists of RGB optical images with a ground sampling distance (GSD) of 0.8 m, acquired over the region of Beijing. The Potsdam dataset comprises 38 UAV images of 6000 × 6000 pixels collected from urban scenes with six categories. The dataset was split into training and test sets, with 24 and 14 images, respectively. The images were then cropped into non-overlapping patches of size 512 × 512. The training set comprises 2904 patches, while the test set contains 1694 patches.

4.2. Experimental setup and evaluation metrics

This experiment is implemented based on PyTorch. Our hardware environment for the experiments is a workstation equipped with an I9 10920 K CPU, 64 GB of RAM, and an RTX3090 GPU. For the BLU and Potsdam datasets, we set the hyperparameters of the training process to fixed values, including 80 epochs, a batch size of 4, an initial learning rate of 0.002, a momentum of 0.9, and a weight decay of 1e-4. The data enhancement contains techniques such as random level flipping.

In this study, we used evaluation metrics widely adopted in semantic segmentation, including recall, precision, overall accuracy (OA), F1 score, and overall and individual MIOU for each class. The equations for these evaluation measures can be formulated based on the counts of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) pixels, which are calculated as: (10) $\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$(10) (11) $\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$(11) (12) ${\mathrm{F}}_1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$(12) (13) $\mathrm{OA}={\displaystyle \frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{T}\mathrm{P}+\mathrm{TN}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$(13)

4.3. Experimental results

4.3.1. Comparison with existing deep learning-based methods

To assess the effectiveness of our proposed model, we will contrast it with other approaches for deep learning based on the BLU dataset and Potsdam dataset, including FCN (Long, Shelhamer, and Darrell 2015), DeepLabv3+ (Chen et al. 2018), DecoupleSegNet (Li et al. 2020), BANet (Wang et al. 2021a), SegFormer (Xie et al. 2021), TransUNet (Chen et al. 2021), DC-Swin (Wang et al. 2022), WiCoNet (Ding et al. 2022) and SAM (Kirillov et al. 2023). As the segmentation produced by SAM lacks semantic information, we solely visualize the segmentation results of SAM.

Using the BLU dataset, we conducted five repeated experiments on various semantic segmentation models. Table 1 presents the performance evaluation and corresponding standard deviations on multiple metrics, including recall, precision, overall accuracy (OA), and F1 score. The experimental results demonstrate the superiority of our MDSNet model, outperforming other models in nearly all metrics. It achieves the highest recall rate, OA, and F1 score, with slightly lower precision than DC-Swin.

Table 1. Performance comparison of existing methods on the BLU dataset using Recall (%), Precision (%), OA (%), and F1 (%) metrics.

Method	Recall	Precision	OA	F1
FCN (2015)	82.59 ± 0.16	79.42 ± 0.21	83.39 ± 0.17	80.84 ± 0.26
DeepLabv3+ (2018)	81.85 ± 0.14	81.32 ± 0.18	83.78 ± 0.23	81.57 ± 0.14
DecoupleSegNet (2020)	82.65 ± 0.12	82.59 ± 0.10	84.80 ± 0.15	82.61 ± 0.17
BANet (2021)	84.42 ± 0.06	81.67 ± 0.09	83.97 ± 0.08	81.43 ± 0.12
SegFormer (2021)	84.26 ± 0.20	81.93 ± 0.23	84.76 ± 0.19	82.02 ± 0.16
TransUNet (2021)	83.64 ± 0.32	81.96 ± 0.28	83.42 ± 0.17	82.14 ± 0.26
DC-Swin (2022)	84.39 ± 0.13	82.64 ± 0.12	84.68 ± 0.08	83.06 ± 0.14
WiCoNet (2022)	84.33 ± 0.10	80.03 ± 0.07	84.43 ± 0.12	81.76 ± 0.20
MDSNet (Ours)	84.61 ± 0.18	82.14 ± 0.11	85.28 ± 0.09	83.33 ± 0.12

Table 2 shows that our proposed MDSNet achieved a higher mIoU score than other models. Furthermore, MDSNet outperformed other models in terms of IoU for most classes, notably for water bodies and roads. This superior performance can be attributed to MDSNet’s ability to model spatial continuity using contextual information from neighboring pixels at multiple scales. Instead of relying solely on the characteristics of individual pixels, MDSNet captures contextual information from different scales by using multiscale supervision, leading to better recognition of continuous features. However, further studies are required to improve the inference speed.

Table 2. Performance comparison of existing methods on the BLU dataset using IoU (%), mIoU (%), and FPS (Hz) scores for each category.

Method	Background	Built-up	Vegetation	Water	Agricultural	Road	mIoU	FPS
FCN (2015)	57.17	74.84	78.02	63.77	72.08	63.93	68.26 ± 0.36	37.8
DeepLabv3+ (2018)	58.25	75.92	77.81	65.14	72.45	65.43	69.03 ± 0.22	24.5
DecoupleSegNet (2020)	60.72	78.30	78.63	66.86	72.98	66.28	70.63 ± 0.27	12.6
BANet (2021)	59.87	78.42	78.59	67.31	72.17	64.83	70.24 ± 0.14	8.7
SegFormer (2021)	60.08	78.47	78.46	67.08	71.89	63.98	69.95 ± 0.33	15.5
TransUNet (2021)	60.15	78.13	78.02	66.92	71.58	64.42	69.88 ± 0.22	7.3
DC-Swin (2022)	61.21	79.12	79.14	67.98	72.47	66.52	71.18 ± 0.13	10.9
WiCoNet (2022)	61.50	78.29	78.32	62.07	73.07	63.90	69.58 ± 0.16	15.8
MDSNet (Ours)	60.06	78.57	79.50	70.75	73.14	67.33	71.55 ± 0.13	12.9

The visualized results in Figure 3 showcase the segmentation performance of all compared methods. Our proposed MDSNet accurately segmented different objects with less internal noise and more accurate spatial boundaries compared to other models. Compared with DecoupleSegNet, our model performs better in segmentation performance. Although DecoupleSegNet also supervises the body and edge regions of the object, it only supervises low-resolution features, leading to the loss of some fine-grained information and a decrease in performance. Compared to other Transformer-based semantic segmentation methods, MDSNet achieves more accurate edge segmentation. This is because Transformer-based methods typically focus more on modeling global context and semantic information, while edge information may not be explicitly modeled sufficiently. This can lead to certain difficulties in pixel-level edge recognition. Additionally, while SAM achieves precise segmentation of objects, it falls short in recognizing the semantic information of ground objects in RSI and may misclassify objects of the same category as different ones. Based on the visualization results, it is apparent that MDSNet demonstrates enhanced continuity and superior edge segmentation when recognizing the water class. This finding aligns with our experimental results and analysis.

Figure 3. Comparison of visualization results on BLU dataset with different semantic segmentation methods.

We also conducted five repeated experiments using these semantic segmentation models on the Potsdam dataset. The specific evaluation metrics and standard deviations are shown in Table 3. The experimental results indicate that our MDSNet model performs exceptionally well on most metrics.

Table 3. Performance comparison of existing methods on the Potsdam dataset using Recall (%), Precision (%), OA (%), and F1 (%) metrics.

Method	Recall	Precision	OA	F1
FCN (2015)	84.93 ± 0.38	81.12 ± 0.32	87.61 ± 0.14	82.97 ± 0.21
DeepLabv3+ (2018)	83.92 ± 0.37	82.54 ± 0.29	87.53 ± 0.17	83.15 ± 0.13
DecoupleSegNet (2020)	84.52 ± 0.24	82.16 ± 0.22	87.30 ± 0.05	83.25 ± 0.14
BANet (2021)	84.92 ± 0.23	82.94 ± 0.34	87.69 ± 0.07	84.07 ± 0.12
SegFormer (2021)	83.60 ± 0.17	81.96 ± 0.14	86.43 ± 0.10	82.64 ± 0.20
TransUNet (2021)	83.41 ± 0.21	82.34 ± 0.25	86.52 ± 0.04	82.54 ± 0.21
DC-Swin (2022)	85.48 ± 0.12	83.79 ± 0.14	88.46 ± 0.18	84.27 ± 0.13
WiCoNet (2022)	83.21 ± 0.06	83.58 ± 0.16	87.47 ± 0.12	83.47 ± 0.17
MDSNet (Ours)	85.63 ± 0.11	83.61 ± 0.24	88.55 ± 0.09	84.56 ± 0.16

Table 4 shows that our proposed MDSNet achieves higher mIoU scores compared to other state-of-the-art semantic segmentation models on the Potsdam dataset. Additionally, in the Potsdam dataset, the Cluster category refers to clustered buildings in urban areas, which are often more densely packed, complex, and prone to overlapping than other building classes such as Residential and Industrial. This makes it challenging to distinguish the Cluster category from other categories. Compared to other models, our proposed MDSNet shows significantly more accurate classification results for the Cluster class. This is because our network learns through multiscale side outputs and can better adapt to the shape and size of Cluster buildings. This demonstrates that our network is better suited for identifying complex RS objects. However, in terms of inference speed, MDSNet only achieved a frame rate of 12.8 FPS, indicating the need for further improvements in the future.

Table 4. Performance comparison of existing methods on the Potsdam dataset using IoU (%), mIoU (%), and FPS (Hz) scores for each category.

Method	Imp.surf	Building	Low Veg.	Tree	Car	Cluster	mIoU	FPS
FCN (2015)	81.32	90.51	72.51	72.31	79.88	39.15	72.55 ± 0.23	36.4
DeepLabv3+ (2018)	81.03	90.16	70.56	73.53	78.44	41.52	72.81 ± 0.20	24.3
DecoupleSegNet (2020)	80.81	89.63	70.25	72.92	80.25	42.50	72.73 ± 0.14	12.6
BANet (2021)	81.83	90.36	71.29	71.76	79.60	43.29	73.07 ± 0.12	8.5
SegFormer (2021)	81.26	89.83	71.92	72.32	79.89	42.57	72.94 ± 0.06	15.4
TransUNet (2021)	81.94	89.42	71.37	72.45	80.12	42.68	73.03 ± 0.15	7.2
DC-Swin (2022)	82.62	90.61	72.02	73.65	81.47	43.75	74.08 ± 0.13	10.9
WiCoNet (2022)	81.58	90.01	70.59	72.36	80.85	42.95	73.13 ± 0.15	15.8
MDSNet (Ours)	82.90	91.64	72.56	73.82	80.99	45.53	74.58 ± 0.07	12.8

Figure 4 presents the visualized results of various semantic segmentation models on the Potsdam dataset. The results demonstrate that our proposed approach can accurately segment diverse objects with high precision and outperform other models regarding spatial boundaries and internal consistency. This highlights the efficacy of our decoupled supervision framework. Notably, MDSNet performs significantly better than several other models in edge pixel segmentation. An important reason, MDSNet can capture edge information at various scales, and its hybrid loss function is better at achieving the supervised effect on edge pixels. While SAM also performs well in edge segmentation, its lack of semantic information makes it unsuitable for the semantic segmentation of remote sensing objects. Clearly, our MDSNet network significantly outperforms other networks in segmenting edge pixels, incredibly complex edge pixels. This further highlights the efficacy of our proposed hybrid loss function in enhancing the supervised effect on edge pixels and establishes the superiority of the MDSNet network in RSI segmentation.

Figure 4. Comparison of visualization results on Potsdam dataset with different semantic segmentation methods.

To gain a more intuitive understanding of the differences between our method and other approaches, we present in Figure 5 the variations of mIoU during the training process for different methods on the BLU dataset. The results in Figure 5 demonstrate that in the early stages of training, our network does not outperform some Transformer-based segmentation methods. This is because Transformer models have strong modeling capabilities and a better understanding of global context, enabling the network to quickly learn key features and semantic information from the data. However, as the number of epochs increases, MDSNet exhibits better performance. We believe that this is due to the training strategy of multiscale decoupled supervision, which allows better focus on the features at each level of the decoder and independently optimize them. This training approach enables the network to adjust the parameters of each layer more quickly in the later stages, thereby improving the accuracy of segmentation.

Figure 5. The variation of mIoU during the training process on the BLU dataset.

Our model exhibits higher segmentation accuracy and better segmentation results in conducting experiments on BLU and Potsdam datasets. In particular, the problems of inconsistent object interiors and unsmooth segmentation edges have been important problems plaguing the field in semantic segmentation tasks. Our model makes substantial improvements in this area, resulting in more accurate and smoother segmentation results. These experimental results further demonstrate the effectiveness and practicality of our proposed method and provide new ideas and methods for further research and applications in the semantic segmentation of RSI.

4.3.2. Ablation studies

Table 5 showcases the ablation experiments conducted on the baseline model. In these experiments, we maintain consistency with other conditions while utilizing ResNet-34 as the encoder structure for our UNet baseline. We compare this baseline model with an extended version that incorporates two extra stages, corresponding to the last two steps of our encoder structure. Lastly, we introduce an intermediate stage between the encoder and decoder to achieve a comprehensive network architecture. The results of the experiments demonstrate that the inclusion of the ‘two extra stages’ significantly expands the receptive field, allowing for the capture of a broader range of contextual information and resulting in improved performance. Furthermore, the introduction of the intermediate stage further enhances the overall network performance.

Table 5. Ablation experiment on the baseline model.

Architecture	mIoU (%)
	BLU dataset	Potsdam dataset
Baseline	70.43	73.31
+ two extra stages	71.32	74.18
+ two extra stages + intermediate stage	71.56	74.57

To show the efficacy of our decoupled supervision framework, we systematically investigated its performance through a series of ablation experiments presented in Table 6. In these experiments, we progressively added loss functions for supervising the body and edge parts of the network, denoted as $L_{body}$ and $L_{edge}$ respectively, to assess their impact on segmentation performance. The results demonstrate that adding $L_{body}$ alone improved our mean Intersection over Union (mIoU) by 0.66% and 0.95% on BLU and Potsdam datasets, respectively, and the supervision from $L_{body}$ effectively reduced uncertain noise along boundaries. Similarly, adding $L_{edge}$ alone improved our mIoU by 0.97% and 1.43% on BLU and Potsdam datasets, respectively. Finally, combining $L_{body}$ and $L_{edge}$ yielded significant improvements of 1.85% and 2.39% on the respective datasets, thus providing robust evidence for the efficacy of our decoupled supervision approach for the body and edge components.

Table 6. Ablation experiments on decoupling module.

Dataset	$L_{seg}$	$L_{body}$	$L_{edge}$	mIoU
BLU	√			69.71
BLU	√	√		70.37
BLU	√		√	70.68
BLU	√	√	√	71.56
Potsdam	√			72.19
Potsdam	√	√		73.14
Potsdam	√		√	73.62
Potsdam	√	√	√	74.57

For a more intuitive presentation, we display the visualization results based on the BLU dataset (see Figure 6) and the Potsdam dataset (see Figure 7). From the visualization results in Figures 6 and 7, we observe inconsistencies in the interior of objects and inaccurate edge pixel partitioning in both Figures 6 and 7(c). By adding $L_{body}$, we found that the noise inside the objects decreased, as shown in Figures 6 and 7(d). This indicates that we achieved better internal consistency by supervising the body features. Similarly, with the addition of $L_{edge}$ we achieved more accurate segmentation boundaries, as shown in Figures 6 and 7(e). This demonstrates the effectiveness of our edge supervision module. Finally, by combining $L_{body}$ and $L_{edge}$, we achieved better internal consistency and spatial boundaries.

Figure 6. Visualization results on BLU dataset. (a) Images, (b) GT, (c) MDSNet without $L_{body}$ and $L_{edge}$, (d) MDSNet without $L_{edge}$, (e) MDSNet without $L_{body}$, and (f) MDSNet.

Figure 7. Visualization results on the Potsdam dataset. (a) Images, (b) GT, (c) MDSNet without $L_{body}$ and $L_{edge}$, (d) MDSNet without $L_{edge}$, (e) MDSNet without $L_{body}$, and (f) MDSNet.

To show the effectiveness of our hybrid loss for the edge, we conducted a series of experiments by utilizing our loss combination. The experimental results presented in Table 7 demonstrate that the performance is significantly improved by our proposed hybrid loss, which combines IOU and BCE. To further illustrate the qualitative impact of the loss, we trained the model with different losses, as shown in Table 6. When we used only BCE loss for edge supervision, there was slight performance improvement, likely due to the sparsity of edge pixels. We attempted to increase the coefficient of the BCE loss but found that it did not lead to significant performance improvement. However, when we used the hybrid IOU and BCE loss, the performance improved significantly, demonstrating the effectiveness of our hybrid loss. In addition, by incorporating hard pixel mining on Ledge, we observed a performance improvement of around 0.47% and 0.65% on the BLU and Potsdam datasets, respectively, indicating the effectiveness of our adopted boundary comprehensive loss.

Table 7. Ablation experiment of edge loss.

Dataset	$L_{bce}$	$L_{iou}$	$L_{ohem}$	mIoU (%)
BLU				70.37
BLU	√			70.37
BLU	√	√		71.09
BLU	√	√	√	71.56
Potsdam				73.14
Potsdam	√			73.18
Potsdam	√	√		73.92
Potsdam	√	√	√	74.57

To evaluate the effectiveness of our multiscale supervision network, we conducted a series of experiments and summarized the results in Table 8. In these experiments, we examined the impact of different supervision scenarios, including the absence of supervision at a particular scale and supervision of only the prediction with the highest accuracy. It's important to note that our loss function encompasses features at different scales, including body and edge features. The results of our experiments reveal that the absence of supervision at any scale leads to a slight decrease in the mIoU index. Additionally, we observed that supervising only the prediction with the highest accuracy caused a reduction in the mIoU index of BLU and Potsdam datasets by 1.27% and 1.50%, respectively. This can be attributed to the lack of supervision in the middle layer, which limits the network's ability to receive direct feedback during training. These findings indirectly demonstrate the efficacy of our multiscale supervision. They suggest that each supervision scale plays a significant role in the final semantic segmentation result, thereby strengthening the viability and effectiveness of our proposed MDSNet.

Table 8. Ablation experiment of multiscale supervision.

Method	mIoU (%)
	BLU dataset	Potsdam dataset
MDSNet (without Loss 1)	71.42	74.48
MDSNet (without Loss 2)	71.37	74.33
MDSNet (without Loss 3)	71.27	74.43
MDSNet (without Loss 4)	71.40	74.30
MDSNet (without Loss 5)	71.22	74.32
MDSNet (only with Loss 6)	70.29	73.07
MDSNet	71.56	74.57

5. Conclusion

In this study, we propose an RSI segmentation model called MDSNet, which utilizes a multiscale decoupled supervised framework to improve segmentation performance. By supervising decoupled body and edge features, our approach can significantly enhance the internal consistency of RS objects and learn spatial boundaries. Moreover, the multiscale output enables our model to learn objects at different scales, which makes our model more adaptable to RS semantic segmentation. Notably, we propose a novel edge hybrid loss, which effectively improves the network's ability to recognize edge pixels and further enhances the segmentation performance of our model. We validate the effectiveness of the proposed multiscale decoupled supervision framework and edge hybrid loss through ablation experiments on BLU and Potsdam datasets. Our proposed model's superior performance in RSI segmentation is demonstrated by the experimental results on the BLU and Potsdam datasets, highlighting its robustness across various datasets and scenarios in comparison to other existing methods.

Despite our proposed model's strengths, it still has limitations. The multiscale output strategy we employed has augmented the network's intricacy, resulting in an expanded model size. As a result, we plan to investigate ways to decrease network resource consumption in future research while enhancing or retaining the model's efficiency.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The BLU dataset and Potsdam dataset in this study are downloaded at https://rslab.disi.unitn.it/dataset/BLU/ and https://www.isprs.org/

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant number 41971365], the Major Science and Technology Project of the Ministry of Water Resources [grant number SKR-2022037], and the Chongqing Graduate Research Innovation Project [grant number CYS22448].