Hybrid transformer-CNN networks using superpixel segmentation for remote sensing building change detection

ABSTRACT

Convolution in convolutional neural network(CNN) essentially uses a filter (kernel) with shared parameters to achieve feature extraction by computing the weighted sum of the centre pixel and adjacent pixels. The transformer divides the input image into patches and adds position encodings, then learns global semantic information and performs remote modelling through a self-attentive mechanism. However, CNNs are good at extracting local features but have difficulty in capturing global cues; the Transformer uses the self-attention mechanism for remote modelling. However, relative to CNN, local feature details are ignored to a certain extent. We believe that CNN and Transformer are complementary and will show better results if they are fused. Therefore, in this work, we propose a Hybrid Transformer-CNN Networks based on the fusion of CNN and Transformer branches for remote sensing change detection. In the CNN branch, we use the classical U-Net architecture to learn local semantic features. In the Transformer branch, we use Transformer-based progressive sampling to focus the model’s attention on objects of interest and prevent corrupting object structure. Subsequently, we propose an adaptive feature merging module to fully fuse the features of CNN and Transformer to enhance feature representation. At the same time, we introduce a differentiable superpixel branch to take advantage of the superpixel segmentation algorithm to accurately identify object boundaries, preserve boundary information and reduce noise in pixel-level features. We supplement the fused enhanced features into the superpixel branch features using a feature refinement module. After our experiments, we demonstrate the superiority of our model over other State of the art methods.

KEYWORDS:

• High-resolution optical remote sensing image
• superpixel segmentation
• change detection(CD)
• transformer
• convolutional neural network (CNN)

Acknowledgements

The authors acknowledge the National Natural Science Foundation of China (Grant nos. 61772319, 62002200, 62202268 and 61972235), the Shandong Natural Science Foundation of China (Grant no. ZR2021MF107, ZR2022MA076) Youth Innovation Technology Project of Higher School in Shandong Province under (Grant No. 2021KJ069, 2019KJN042) and Yantai science and technology innovation development plan(2022JCYJ031).

Disclosure statement

The authors declare that no potential competing interests exist. There is no an undisclosed relationship they may pose a competing interest. There is no an undisclosed funding source that may pose a competing interest.

Data availability statement

Data available on request from the authors https://study.rsgis.whu.edu.cn/pages/download/building_dataset.html; https://ieee-dataport.org/open-access/oscd-onera-satellite-change-detection.