Hyperspectral and multispectral image fusion via residual selective kernel attention-based U-net

ABSTRACT

The fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) is a crucial technology for producing high-resolution hyperspectral images. Most existing image fusion algorithms based on deep learning do not fully utilize the ability of neural network to extract and process multi-scale features, which leads to the problem of difficulty in fully learning features and ambiguity of features. In order to overcome these issues, a residual selective kernel attention-based U-net named RSKAU-net is designed for LR-HSI and HR-MSI fusion. RSKAU-net is constructed by a residual selective kernel module with an attention mechanism and a channel attention block. The residual selective kernel attention-based (RSKA) module is designed to process images of different resolutions, which adaptively extracts multi-scale features and efficiently emphasizes significant features through the attention mechanism. The channel attention (CA) module retains important spectral information by assigning different weights to each channel of LR-HSI. The proposed network can enhance the spatial information of LR-HSI while preserving its spectral information. Meanwhile, it effectively fuses the features from the source image to obtain the HR-HSI with rich details. The experimental results demonstrate that the proposed network has advantages in terms of both visual effect and objective quantitative indices when compared to existing HSI-MSI fusion approaches.

KEYWORDS:

• Hyperspectral image (HSI)
• multispectral image (MSI)
• residual selective kernel attention-based U-net
• image fusion

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was in part supported by the National Natural Science Foundation of China under Grant 61871210.