Spatiotemporal Fusion of Remote Sensing Images using a Convolutional Neural Network with Attention and Multiscale Mechanisms

ABSTRACT

In this paper, we propose a new spatiotemporal fusion method based on a convolutional neural network to which we added attention and multiscale mechanisms (AMNet). Different from the previous spatiotemporal fusion methods, the residual image obtained by subtracting moderate resolution imaging spectroradiometer (MODIS) images at two times is directly used to train the network, and two special structures of multiscale mechanism and attention mechanism are used to increase the accuracy of fusion. Our proposed method uses one pair of images to achieve spatiotemporal fusion. The work is mainly divided into three steps. The first step is to extract feature maps of two types of images at different scales and fuse them separately. The second step is to use the attention mechanism to focus on the important information in the feature maps. And the third step is to reconstruct the image. We used two classical datasets for the experiment, and compared our experimental results with the other three state-of-the-art spatiotemporal fusion methods. The results of our proposed method have richer spatial details and more accurate prediction of temporal changes.

Acknowledgements

We sincerely thank Mr. Zhenyu Tan for providing the source code for DCSTFN and Irina V. Emelyanova and colleagues for providing the dataset, which were very helpful in our research.

Disclosure statement

The authors declare no conflict of interest.

Additional information

Funding

This work was supported in part by the Chongqing Graduate Student Scientific Research Innovation Project under Grant CYB19174; the Natural Science Foundation of China under Grant 61972060; and Grant U1713213; the Natural Science Foundation of Chongqing under Grant cstc2019cxcyljrc-td0270; and Grant cstc2019jcyj-cxttX0002