Multi-scale spatial-spectral Transformer for spectral reconstruction from RGB images

References

• Aeschbacher, J., W. Jiqing, and R. Timofte. 2017. “In Defense of Shallow Learned Spectral Reconstruction from RGB Images.” In Proceedings of the IEEE International Conference on Computer Vision Workshops, 471–479. https://doi.org/10.1109/iccvw.2017.63.
   Google Scholar
• Alvarez-Gila, A., J. Van De Weijer, and E. Garrote. 2017. “Adversarial Networks for Spatial Context-Aware Spectral Image Reconstruction from Rgb.” In Proceedings of the IEEE International Conference on Computer Vision workshops, 480–490. https://doi.org/10.1109/iccvw.2017.64.
   Google Scholar
• Arad, B., and O. Ben-Shahar. 2016. “Sparse Recovery of Hyperspectral Signal from Natural RGB Images.” In European Conference on Computer Vision, 19–34. https://doi.org/10.1007/978-3-319-46478-7_2.
   Google Scholar
• Arad, B., R. Timofte, O. Ben-Shahar, Y.-T. Lin, and G. D. Finlayson. 2020. “Ntire 2020 Challenge on Spectral Reconstruction from an Rgb Image.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 446–447. https://openaccess.thecvf.com/content_CVPRW_2020/html/w31/Arad_NTIRE_2020_Challenge_on_Spectral_Reconstruction_From_an_RGB_Image_CVPRW_2020_paper.html.
   Google Scholar
• Arad, B., R. Timofte, R. Yahel, N. Morag, A. Bernat, Y. Cai, J. Lin, et al. 2022. “NTIRE 2022 Spectral Recovery Challenge and Data Set.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 863–881. https://doi.org/10.1109/cvprw56347.2022.00102.
   Google Scholar
• Arnab, A., M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. 2021. “Vivit: A Video Vision Transformer.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6836–6846. https://doi.org/10.1109/iccv48922.2021.00676.
   Google Scholar
• Banerjee, A., and A. Palrecha. 2020. “MXR-U-nets for Real Time Hyperspectral Reconstruction.” arXiv preprint. https://doi.org/10.48550/arXiv.2004.07003.
   PubMedGoogle Scholar
• Becerir, B. 2017. “Color Concept in Textiles: A Review.” Journal of Textile Engineering & Fashion Technology 1 (6). https://doi.org/10.15406/jteft.2017.01.00039.
   Google Scholar
• Bichen, W., X. Chenfeng, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda. 2020. “Sustained Release of Tulobuterol from Graphene Oxide Laden Hydrogel to Manage Asthma.” Journal of Biomaterials Science Polymer Edition 32 (4): 524–535. https://doi.org/10.1080/09205063.2020.1849921.
   PubMed Web of Science ®Google Scholar
• Cai, Y., L. Jing, L. Zudi, H. Wang, Y. Zhang, H. Pfister, R. Timofte, and L. Van Gool. 2022 “Mst++: Multi-Stage Spectral-wise Transformer For Efficient Spectral Reconstruction.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 745–755. https://doi.org/10.1109/cvprw56347.2022.00090.
   Google Scholar
• Cai, Y., J. Lin, H. Xiaowan, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool. 2022. “Mask-Guided Spectral-Wise Transformer for Efficient Hyperspectral Image Reconstruction.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June, 17502–17511. https://doi.org/10.1109/cvpr52688.2022.01698.
   Google Scholar
• Cao, X., D. Hao, X. Tong, Q. Dai, and S. Lin. 2011. “A Prism-Mask System For Multispectral Video Acquisition.” IEEE Transactions on Pattern Analysis and Machine Intelligence 33 (12): 2423–2435. https://doi.org/10.1109/TPAMI.2011.80.
   PubMed Web of Science ®Google Scholar
• Cao, H., Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, and M. Wang. 2023. “Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation“ Computer Vision – ECCV 2022 Workshops, 205–218. https://doi.org/10.1007/978-3-031-25066-8_9.
   Google Scholar
• Carion, N., F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. 2020. “End-To-End Object Detection with Transformers.” In European conference on computer vision, 213–229. https://doi.org/10.1007/978-3-030-58452-8_13.
   Google Scholar
• Chen, C.-F. R., Q. Fan, and R. Panda. 2021. “Crossvit: Cross-Attention Multi-Scale Vision Transformer for Image Classification.” In Proceedings of the IEEE/CVF international conference on computer vision, 357–366. https://doi.org/10.1109/iccv48922.2021.00041.
   Google Scholar
• Chen, L., L. Xin, J. Zhang, X. Chu, and C. Chen. 2021. “HINet: Half Instance Normalization Network for Image Restoration.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 182–192. https://doi.org/10.1109/cvprw53098.2021.00027.
   Google Scholar
• Dai, Z., B. Cai, Y. Lin, and J. Chen. 2021. “Up-Detr: Unsupervised Pre-Training for Object Detection with Transformers.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1601–1610. https://doi.org/10.1109/cvpr46437.2021.00165.
   Google Scholar
• Descour, M., and E. Dereniak. 1995. “Computed-Tomography Imaging Spectrometer: Experimental Calibration and Reconstruction Results.” Applied Optics 34 (22): 4817–4826. https://doi.org/10.1364/AO.34.004817.
   PubMed Web of Science ®Google Scholar
• Ding, Y., Z. Zhang, X. Zhao, D. Hong, W. Cai, Y. Chengguo, N. Yang, and W. Cai. 2022. “Multi-Feature Fusion: Graph Neural Network and CNN Combining for Hyperspectral Image Classification.” Neurocomputing 501:246–257. https://doi.org/10.1016/j.neucom.2022.06.031.
   Web of Science ®Google Scholar
• Ding, Y., Z. Zhang, X. Zhao, D. Hong, L. Wei, W. Cai, and Y. Zhan. 2022. “AF2GNN: Graph Convolution with Adaptive Filters and Aggregator Fusion for Hyperspectral Image Classification.” Information Sciences 602:201–219. https://doi.org/10.1016/j.ins.2022.04.006.
   Web of Science ®Google Scholar
• Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani. 2020. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale“ arXiv preprint. https://doi.org/10.48550/arXiv.2010.11929.
   Google Scholar
• Hendrycks, D., and K. Gimpel. 2023. “Gaussian Error Linear Units (gelus).” arXiv preprint. https://doi.org/10.48550/arXiv.1606.08415.
   Google Scholar
• Hong, D., L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot. 2021. “Graph Convolutional Networks for Hyperspectral Image Classification.” IEEE Transactions on Geoscience & Remote Sensing 59 (7): 5966–5978. https://doi.org/10.1109/TGRS.2020.3015157.
   Web of Science ®Google Scholar
• Hong, D., L. Gao, N. Yokoya, J. Yao, J. Chanussot, D. Qian, and B. Zhang. 2021. “More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification.” IEEE Transactions on Geoscience & Remote Sensing 59 (5): 4340–4354. https://doi.org/10.1109/TGRS.2020.3016820.
   Web of Science ®Google Scholar
• Hong, D., N. Yokoya, J. Chanussot, and X. Xiang Zhu. 2019. “An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing.” IEEE Transactions on Image Processing 28 (4): 1923–1938. https://doi.org/10.1109/TIP.2018.2878958.
   Web of Science ®Google Scholar
• Huang, G., Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. 2017. “Densely Connected Convolutional Networks.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 4700–4708. https://doi.org/10.48550/arXiv.1608.06993.
   Google Scholar
• Jiaojiao, L., W. Chaoxiong, R. Song, L. Yunsong, and F. Liu. 2020. “Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 462–463. https://doi.org/10.1109/cvprw50498.2020.00239.
   Google Scholar
• Jia, Y., Y. Zheng, G. Lin, A. Subpa-Asa, A. Lam, Y. Sato, and I. Sato. 2017. “From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping.” In Proceedings of the IEEE international conference on computer vision, 4705–4713. https://doi.org/10.1109/iccv.2017.504.
   Google Scholar
• Kaiming, H., X. Zhang, S. Ren, and J. Sun. 2016. “Deep Residual Learning for Image Recognition.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778. https://doi.org/10.1109/cvpr.2016.90.
   Google Scholar
• Koundinya, S., H. Sharma, M. Sharma, A. Upadhyay, R. Manekar, R. Mukhopadhyay, A. Karmakar, and S. Chaudhury. 2018. “2d-3d Cnn Based Architectures for Spectral Reconstruction from Rgb Images.” In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 844–851. https://doi.org/10.1109/cvprw.2018.00129.
   Google Scholar
• Kwon, H., and Y.-W. Tai. 2015. “RGB-Guided Hyperspectral Image Upsampling.” In Proceedings of the IEEE International Conference on Computer Vision, 307–315. https://doi.org/10.1109/iccv.2015.43.
   Google Scholar
• Lee, S.-H., M.-J. Park, and J.-O. Kim. 2020. “Progressive Deep Network with Channel Back-Projection for Hyperspectral Recovery from RGB.” In 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 1257–1261. https://ieeexplore.ieee.org/abstract/document/9306454.
   Google Scholar
• Lim, B., S. Son, H. Kim, S. Nah, and K. Mu Lee. 2017. “Enhanced Deep Residual Networks for Single Image Super-Resolution.” In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 136–144. https://doi.org/10.1109/cvprw.2017.151.
   Google Scholar
• Liu, Z., Y. Lin, Y. Cao, H. Han, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 2021. “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10012–10022. https://doi.org/10.1109/iccv48922.2021.00986.
   Google Scholar
• Li, K., S. Wang, X. Zhang, X. Yifan, X. Weijian, and T. Zhuowen 2021. “Pose Recognition with Cascade Transformers.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1944–1953. https://doi.org/10.1109/cvpr46437.2021.00198.
   Google Scholar
• Meng, Z., M. Jiawei, and X. Yuan. 2020. “End-to-end low cost compressive spectral imaging with spatial-spectral self-attention.” In European Conference on Computer Vision, 187–204. https://doi.org/10.1007/978-3-030-58592-1_12.
   Google Scholar
• Nguyen, R. M., D. K. Prasad, and M. S. Brown. 2014. “Training-Based Spectral Reconstruction from a Single RGB Image.” In European Conference on Computer Vision , 186–201. https://doi.org/10.1007/978-3-319-10584-0_13.
   Google Scholar
• Parmar, M., S. Lansel, and B. A. Wandell. 2008. “Spatio-Spectral Reconstruction of the Multispectral Datacube Using Sparse Recovery.” In 2008 15th IEEE International Conference on Image Processing, 473–476. https://doi.org/10.1109/icip.2008.4711794.
   Google Scholar
• Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” In International Conference on Medical image computing and computer-assisted intervention, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28.
   Google Scholar
• Ruikang, X., M. Yao, C. Chen, L. Wang, and Z. Xiong. 2023. “Continuous Spectral Reconstruction from RGB Images via Implicit Neural Representation.” Computer Vision – ECCV 2022 Workshops, 78–94. http://dx.doi.org/10.1007/978-3-031-25072-9_6.
   Google Scholar
• Shi, W., J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. 2016. “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 1874–1883. https://doi.org/10.1109/cvpr.2016.207.
   Google Scholar
• Shi, Z., C. Chen, Z. Xiong, D. Liu, and W. Feng 2018. “Hscnn+: Advanced Cnn-Based Hyperspectral Recovery from Rgb Images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 939–947. https://doi.org/10.1109/cvprw.2018.00139.
   Google Scholar
• Shi, H., W. Xiong, Y. Hanhan, W. Shichao, F. Zhu, L. Zhiwei, H. Luo, L. Chao, and X. Wang. 2023. “High Resolution Fourier Transform Spectrometer for Ground-Based Verification of Greenhouse Gases Satellites.” Remote Sensing 15 (6): 6. https://www.mdpi.com/2072-4292/15/6/1671.
   Web of Science ®Google Scholar
• Stiebel, T., S. Koppers, P. Seltsam, and D. Merhof. 2018. “Reconstructing Spectral Images from Rgb-Images Using a Convolutional Neural Network.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 948–953. https://doi.org/10.1109/cvprw.2018.00140.
   Google Scholar
• Strudel, R., R. Garcia, I. Laptev, and C. Schmid. 2021. “Segmenter: Transformer for Semantic Segmentation.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 7262–7272. https://doi.org/10.1109/iccv48922.2021.00717.
   Google Scholar
• Timofte, R., V. De Smet, and L. Van Gool. 2014. “A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution.” In Asian conference on computer vision, 111–126. https://doi.org/10.1007/978-3-319-16817-3_8.
   Google Scholar
• Valero, E. M., J. L. Nieves, S. M. C. Nascimento, K. Amano, and D. H. Foster. 2007. “Recovering Spectral Data from Natural Scenes with an RGB Digital Camera and Colored Filters.” Color Research & Application 32 (5): 352–360. https://doi.org/10.1002/col.20339.
   Web of Science ®Google Scholar
• Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. “Attention is All You Need.” Advances in Neural Information Processing Systems.” 30. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
   Google Scholar
• Y. Verdie, L. Juwei, and K. N. Plataniotis. 2022. “Efficient Hyperspectral Reconstruction from RGB Images with Line-Pixel Deconvolution.” In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 1–5. https://doi.org/10.1109/ivmsp54334.2022.9816227.
   Google Scholar
• Wagadarikar, A., R. John, R. Willett, and D. Brady. 2008. “Single Disperser Design for Coded Aperture Snapshot Spectral Imaging.” Applied Optics 47 (10): B44–B51. https://doi.org/10.1364/AO.47.000B44.
   PubMed Web of Science ®Google Scholar
• Wang, Z., X. Cun, J. Bao, W. Zhou, J. Liu, and L. Houqiang 2022. “Uformer: A General U-Shaped Transformer for Image Restoration.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17683–17693. https://doi.org/10.1109/cvpr52688.2022.01716.
   Google Scholar
• Wang, N., M. Shaohui, Y. Zhang, B. Zhang, M. Mingyang, and Z. Xiangqing. 2022. “Reconstructing Hyperspectral Images from RGB Inputs Based on Intrinsic Image Decomposition.” In IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium, 2374–2377. https://doi.org/10.1109/igarss46834.2022.9883751.
   Google Scholar
• Xiaowan, H., Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool. 2022. “Hdnet: High-Resolution Dual-Domain Learning for Spectral Compressive Imaging.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17542–17551. https://doi.org/10.1109/cvpr52688.2022.01702.
   Google Scholar
• Xin, W., D. Hong, and J. Chanussot. 2023. “UIU-Net: U-Net in U-Net for Infrared Small Object Detection.” IEEE Transactions on Image Processing 32:364–376. https://doi.org/10.1109/TIP.2022.3228497.
   Web of Science ®Google Scholar
• Xin, W., D. Hong, J. Chanussot, M. Zheng, P. Zhang, D. Xu, and J. Wang. 2022. “Convolutional Neural Networks for Multimodal Remote Sensing Data Classification.” IEEE Transactions on Geoscience and Remote Sensing 60:1–16. https://doi.org/10.1109/TGRS.2022.3210948.
   Web of Science ®Google Scholar
• Xiong, Z., Z. Shi, L. Huiqun, L. Wang, D. Liu, and W. Feng 2017. “Hscnn: Cnn-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections.” In Proceedings of the IEEE International Conference on Computer Vision Workshops, 518–525. https://doi.org/10.1109/iccvw.2017.68.
   Google Scholar
• Yang, S., Z. Quan, M. Nie, and W. Yang. 2021. “Transpose: Keypoint localization via transformer.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11802–11812. https://doi.org/10.1109/iccv48922.2021.01159.
   Google Scholar
• Yao, D., Z. Zhi-Li, Z. Xiao-Feng, C. Wei, H. Fang, C. Yao-Ming, and W.-W. Cai. 2023. “Deep Hybrid: Multi-Graph Neural Network Collaboration for Hyperspectral Image Classification.” Defence Technology 23:164–176. https://doi.org/10.1016/j.dt.2022.02.007.
   Web of Science ®Google Scholar
• Yuan, X., D. J. Brady, and A. K. Katsaggelos. 2021. “Snapshot Compressive Imaging: Theory, Algorithms, and Applications.” IEEE Signal Processing Magazine 38 (2): 65–88. https://doi.org/10.1109/MSP.2020.3023869.
   Web of Science ®Google Scholar
• Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang. 2022. “Restormer: Efficient Transformer for High-Resolution Image Restoration.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5728–5739. https://doi.org/10.1109/cvpr52688.2022.00564.
   Google Scholar
• Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. Shahbaz Khan, M.-H. Yang, and L. Shao. 2020. “Learning Enriched Features for Real Image Restoration and Enhancement.” In European Conference on Computer Vision, 492–511. https://doi.org/10.1007/978-3-030-58595-2_30.
   Google Scholar
• Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. Shahbaz Khan, M.-H. Yang, and L. Shao. 2021. “Multi-Stage Progressive Image Restoration.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14821–14831. https://doi.org/10.1109/cvpr46437.2021.01458.
   Google Scholar
• Zhang, L., Z. Lang, P. Wang, W. Wei, S. Liao, L. Shao, and Y. Zhang. 2020. “Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution.” In Proceedings of the AAAI Conference on Artificial Intelligence, 12821–12828. https://doi.org/10.1609/aaai.v34i07.6978.
   Google Scholar
• Zhao, Y., P. Lai-Man, Q. Yan, W. Liu, and T. Lin. 2020. “Hierarchical Regression Network for Spectral Reconstruction from RGB Images.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 422–423. https://doi.org/10.1109/cvprw50498.2020.00219.
   Google Scholar
• Zheng, C., S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding. 2021. “3d Human Pose Estimation with Spatial and Temporal Transformers.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11656–11665. https://doi.org/10.1109/iccv48922.2021.01145.
   Google Scholar