Multi-scale spatial-spectral Transformer for spectral reconstruction from RGB images

ABSTRACT

Spectrometers can obtain fine spectral reflectance of substances through high-resolution spectral sampling, which has important advantages in identifying the categories of ground substances. As a complement to high-cost spectrometers, reconstruction from RGB images provides an economical and convenient way to acquire hyperspectral images. Although the current mainstream reconstruction methods based on deep convolution network can directly learn the inverse reconstruction mapping in data-driven way, they still have shortcomings in the representation of multi-scale and long-range spatial-spectral correlations within hyperspectral images. To conquer these issues, we propose a novel multi-scale spatial-spectral Transformer network for spectral reconstruction. The proposed network consists of a cascade of multiple-scale spatial contextual module and spatial-spectral fusion Transformer module. Specifically, the multiple-scale spatial contextual module performs three-level feature extraction to learn the spatial contextual structures of each spectral band from the RGB image. The spatial-spectral fusion Transformer module employs three parallel spatial-spectral united Transformers and one fusion Transformer to enhance the spatial and spectral consistency of the reconstructed spectral image. Parallel deployment of spatial-spectral united Transformers is beneficial to reduce the over-smooth effect caused by stacking too many Transformers. Therefore, the proposed network can better recover hyperspectral images with both fine spatial structures and accurate spectral signatures. Comprehensive experiments show that our method can achieve better reconstruction performance than the state-of-the-art methods.

KEYWORDS:

• Spectral reconstruction
• Transformer
• multi-scale context
• spatial-spectral correlation

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was enabled in part by support provided by the National Natural Science Foundation of China, funding reference number [Grants 62276139, U2001211 and U21B2044].