An enhanced spatiotemporal fusion model with degraded fine-resolution images via relativistic generative adversarial networks

References

• Arjovsky M, Chintala S, Bottou L. 2017. Wasserstein generative adversarial networks. In: Proc. 34th Int. Conf. Mach. Learn.; vol. 70; Aug; Sydney, Australia. p. 214–223.
   Google Scholar
• Baby D, Verhulst S. 2019. Sergan: speech enhancement using relativistic generative adversarial networks with gradient penalty. In: Proc. 2019 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP); May; Brighton, UK. p. 106–110.
   Google Scholar
• Benzenati T, Kessentini Y, Kallel A. 2022. Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks. Expert Syst Appl. 188:115996.
   Web of Science ®Google Scholar
• Chen Y, Shi K, Ge Y, Zhou Y. 2022. Spatiotemporal remote sensing image fusion using multiscale two-stream convolutional neural networks. IEEE Trans Geosci Remote Sens. 60:1–12.
   Web of Science ®Google Scholar
• Emelyanova IV, McVicar TR, Van Niel TG, Li LT, Van Dijk AI. 2013. Assessing the accuracy of blending Landsat–MODIS surface reflectances in two landscapes with contrasting spatial and temporal dynamics: a framework for algorithm selection. Remote Sens Environ. 133:193–209.
   Web of Science ®Google Scholar
• Gao F, Masek J, Schwaller M, Hall F. 2006. On the blending of the Landsat and MODIS surface reflectance: predicting daily Landsat surface reflectance. IEEE Trans Geosci Remote Sens. 44(8):2207–2218.
   Web of Science ®Google Scholar
• Gastineau A, Aujol JF, Berthoumieu Y, Germain C. 2022. Generative adversarial network for pansharpening with spectral and spatial discriminators. IEEE Trans Geosci Remote Sens. 60:1–11.
   Web of Science ®Google Scholar
• Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. 2014. Generative adversarial nets. Adv Neural Inf Process Syst. 27:2672–2680.
   Google Scholar
• Guo D, Shi W, Hao M, Zhu X. 2020. FSDAF 2.0: improving the performance of retrieving land cover changes and preserving spatial details. Remote Sens Environ. 248:111973.
   Web of Science ®Google Scholar
• Haut JM, Paoletti ME, Plaza J, Plaza A, Li J. 2019. Visual attention-driven hyperspectral image classification. IEEE Trans Geosci Remote Sens. 57(10):8065–8080.
   Web of Science ®Google Scholar
• He KM, Zhang XY, Ren SQ, Sun J. 2016. Identity mappings in deep residual networks. In: 14th Eur. Conf. Comput. Vis. (ECCV). Lect. Notes Comput. Sci.; vol. 9908; Oct; Amsterdam, Netherlands. p. 630–645.
   Google Scholar
• Hou S, Sun W, Guo B, Li X, Zhang J, Xu C, Li X, Shao Y, Li C. 2022. RFSDAF: a new spatiotemporal fusion method robust to registration errors. IEEE Trans Geosci Remote Sens. 60:1–18.
   Web of Science ®Google Scholar
• Hu J, Shen L, Albanie S, Sun G, Wu EH. 2020. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell. 42(8):2011–2023.
   PubMed Web of Science ®Google Scholar
• Huang B, Song H. 2012. Spatiotemporal reflectance fusion via sparse representation. IEEE Trans Geosci Remote Sens. 50(10):3707–3716.
   Web of Science ®Google Scholar
• Jang DW, Park RH. 2019. Densenet with deep residual channel-attention blocks for single image super resolution. In Proc. 32nd IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshops; Jun; Long Beach, CA, USA. p. 1795–1803.
   Google Scholar
• Jolicoeur-Martineau A. 2019. The relativistic discriminator: a key element missing from standard GAN. In: 7th Int. Conf. Learn. Represent. (ICLR); May; New Orleans, LA, USA. p. 1–26.
   Google Scholar
• Kingma DP, Ba JL. 2015. Adam: a method for stochastic optimization. In: 3rd Int. Conf. Learn. Represent. (ICLR); May; San Diego, CA. p. 1–15.
   Google Scholar
• Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In: Proc. 30th IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR); Jul; Honolulu, HI, USA. p. 105–114.
   Google Scholar
• Li J, Huo H, Li C, Wang R, Feng Q. 2021a. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks. IEEE Trans Multimed. 23:1383–1396.
   Web of Science ®Google Scholar
• Li J, Li Y, Cai R, He L, Chen J, Plaza A. 2022a. Enhanced spatiotemporal fusion via MODIS-like images. IEEE Trans Geosci Remote Sens. 60:1–17.
   Web of Science ®Google Scholar
• Li J, Li Y, He L, Chen J, Plaza A. 2020a. Spatio-temporal fusion for remote sensing data: an overview and new benchmark. Sci China Life Sci. 63(4):140301.
   Google Scholar
• Li Q, Lu L, Li Z, Wu W, Liu Z, Jeon G, Yang X. 2021b. Coupled gan with relativistic discriminators for infrared and visible images fusion. IEEE Sens J. 21(6):7458–7467.
   Web of Science ®Google Scholar
• Li W, Yang C, Peng Y, Du J. 2022b. A pseudo-siamese deep convolutional neural network for spatiotemporal satellite image fusion. IEEE J Sel Top Appl Earth Obs Remote Sens. 15:1205–1220.
   Web of Science ®Google Scholar
• Li W, Zhang X, Peng Y, Dong M. 2021c. Spatiotemporal fusion of remote sensing images using a convolutional neural network with attention and multiscale mechanisms. Int J Remote Sens. 42(6):1973–1993.
   Web of Science ®Google Scholar
• Li X, Foody GM, Boyd DS, Ge Y, Zhang Y, Du Y, Ling F. 2020b. SFSDAF: an enhanced FSDAF that incorporates sub-pixel class fraction change information for spatio-temporal image fusion. Remote Sens Environ. 237:111537.
   Web of Science ®Google Scholar
• Li Y, Li J, He L, Chen J, Plaza A. 2020c. A new sensor bias-driven spatio-temporal fusion model based on convolutional neural networks. Sci China Life Sci. 63(4):140302.
   Google Scholar
• Liu M, Yang W, Zhu X, Chen J, Chen X, Yang L, Helmer EH. 2019a. An improved flexible spatiotemporal data fusion (IFSDAF) method for producing high spatiotemporal resolution normalized difference vegetation index time series. Remote Sens Environ. 227:74–89.
   Web of Science ®Google Scholar
• Liu Q, Zhou H, Xu Q, Liu X, Wang Y. 2021. PSGAN: a generative adversarial network for remote sensing image pan-sharpening. IEEE Trans Geosci Remote Sens. 59(12):10227–10242.
   Web of Science ®Google Scholar
• Liu S, Zhou J, Qiu Y, Chen J, Zhu X, Chen H. 2022. The FIRST model: spatiotemporal fusion incorrporting spectral autocorrelation. Remote Sens Environ. 279:113111.
   Web of Science ®Google Scholar
• Liu X, Deng C, Chanussot J, Hong D, Zhao B. 2019b. StfNet: a two-stream convolutional neural network for spatiotemporal image fusion. IEEE Trans Geosci Remote Sens. 57(9):6552–6564.
   Web of Science ®Google Scholar
• Luo P, Ren J, Peng Z, Zhang R, Li J. 2019. Differentiable learning-to-normalize via switchable normalization. In: 7th Int. Conf. Learn. Represent. (ICLR); May; New Orleans, LA, USA.
   Google Scholar
• Ma Y, Wei J, Tang W, Tang R. 2021. Explicit and stepwise models for spatiotemporal fusion of remote sensing images with deep neural networks. Int J Appl Earth Obs. 105:102611.
   Web of Science ®Google Scholar
• Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S. 2017. Least squares generative adversarial networks. In Proc. 16th IEEE Int. Conf. Comput. Vision (ICCV); Oct; Venice, Italy. p. 2813–2821.
   Google Scholar
• Miyato T, Kataoka T, Koyama M, Yoshida Y. 2018. Spectral normalization for generative adversarial networks. In: Int. Conf. Learn. Represent. (ICLR); May; Vancouver, BC, Canada.
   Google Scholar
• Mou L, Zhu XX. 2020. Learning to pay attention on spectral domain: a spectral attention module-based convolutional network for hyperspectral image classification. IEEE Trans Geosci Remote Sens. 58(1):110–122.
   Web of Science ®Google Scholar
• Nwankpa CE, Ijomah W, Gachagan A, Marshall S. 2021. Activation functions: comparison of trends in practice and research for deep learning. In: 2nd Int. Conf. on Comput. Sci. Tec.; Dec; Jamshoro, Pakistan. p. 124–133.
   Google Scholar
• Peng Y, Li W, Luo X, Du J, Gan Y, Gao X. 2021. Integrated fusion framework based on semicoupled sparse tensor factorization for spatio-temporal–spectral fusion of remote sensing images. Inf Fusion. 65:21–36.
   Web of Science ®Google Scholar
• Shang C, Li X, Yin Z, Li X, Wang L, Zhang Y, Du Y, Ling F. 2022. Spatiotemporal reflectance fusion using a generative adversarial network. IEEE Trans Geosci Remote Sens. 60:1–15.
   Web of Science ®Google Scholar
• Simonyan K, Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition. In: 3rd Int. Conf. Learn. Represent. (ICLR); May; San Diego, CA, USA.
   Google Scholar
• Song H, Huang B. 2013. Spatiotemporal satellite image fusion through one-pair image learning. IEEE Trans Geosci Remote Sens. 51(4):1883–1896.
   Web of Science ®Google Scholar
• Song H, Liu Q, Wang G, Hang R, Huang B. 2018. Spatiotemporal satellite image fusion using deep convolutional neural networks. IEEE J Sel Top Appl Earth Obs Remote Sens. 11(3):821–829.
   Web of Science ®Google Scholar
• Tan Z, Di L, Zhang M, Guo L, Gao M. 2019. An enhanced deep convolutional model for spatiotemporal image fusion. Remote Sens. 11(24):2898.
   Google Scholar
• Tan Z, Gao M, Li X, Jiang L. 2022. A flexible reference-insensitive spatiotemporal fusion model for remote sensing images using conditional generative adversarial network. IEEE Trans Geosci Remote Sens. 60:1–13.
   Web of Science ®Google Scholar
• Tan Z, Yue P, Di L, Tang J. 2018. Deriving high spatiotemporal remote sensing images using deep convolutional network. Remote Sens. 10(7):1066.
   Google Scholar
• Vivone G, Dalla Mura M, Garzelli A, Pacifici F. 2021. A benchmarking protocol for pansharpening: dataset, preprocessing, and quality assessment. IEEE J Sel Top Appl Earth Obs Remote Sens. 14:6102–6118.
   Web of Science ®Google Scholar
• Wang X, Xie L, Dong C, Shan Y. 2021. Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: Proc. 18th IEEE Int. Conf. Comput. Vis. Workshops; Oct; Virtual, Online, Canada. p. 1905–1914.
   Google Scholar
• Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C. 2019. Esrgan: enhanced super-resolution generative adversarial networks. In: 15th Eur. Conf. Comput. Vis. (ECCV). Lect. Notes Comput. Sci. (LNCS); vol. 11133; Sep; Munich, Germany. p. 63–79.
   Google Scholar
• Wang Z, Simoncelli EP, Bovik AC. 2003. Multiscale structural similarity for image quality assessment. In: Proc. 37th Asilomar Conf. Signals, Syst. Comput; vol. 2. p. 1398–1402.
   Google Scholar
• Woo S, Park J, Lee JY, Kweon IS. 2018. Cbam: convolutional block attention module. In: 15th Eur. Conf. Comput. Vis. (ECCV); Sep. p. 3–19.
   Google Scholar
• Wu M, Niu Z, Wang C, Wu C, Wang L. 2012. Use of modis and landsat time series data to generate high-resolution temporal synthetic landsat data using a spatial and temporal reflectance fusion model. J Appl Remote Sens. 6(1):063507.
   Google Scholar
• Wu Y, Feng S, Lin C, Zhou H, Huang M. 2022. A three stages detail injection network for remote sensing images pansharpening. Remote Sens. 14(5):1077.
   Google Scholar
• Yuhas RH, Goetz AF, Boardman JW. 1992. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In: Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop; Jun; Pasadena, CA, USA. p. 147–149.
   Google Scholar
• Zhang H, Song Y, Han C, Zhang L. 2021. Remote sensing image spatiotemporal fusion using a generative adversarial network. IEEE Trans Geosci Remote Sens. 59(5):4273–4286.
   Web of Science ®Google Scholar
• Zhao H, Gallo O, Frosio I, Kautz J. 2017. Loss functions for image restoration with neural networks. IEEE Trans Comput Imaging. 3(1):47–57.
   Web of Science ®Google Scholar
• Zhou J, Chen J, Chen X, Zhu X, Qiu Y, Song H, Rao Y, Zhang C, Cao X, Cui X. 2021. Sensitivity of six typical spatiotemporal fusion methods to different influential factors: a comparative study for a normalized difference vegetation index time series reconstruction. Remote Sens Environ. 252:112130.
   Web of Science ®Google Scholar
• Zhu M, Jiao L, Liu F, Yang S, Wang J. 2021. Residual spectral–spatial attention network for hyperspectral image classification. IEEE Trans Geosci Remote Sens. 59(1):449–462.
   Web of Science ®Google Scholar
• Zhu X, Chen J, Gao F, Chen X, Masek JG. 2010. An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions. Remote Sens Environ. 114(11):2610–2623.
   Web of Science ®Google Scholar
• Zhu X, Cheng D, Zhang Z, Lin S, Dai J. 2019. An empirical study of spatial attention mechanisms in deep networks. In Proc. 17th IEEE Int. Conf. Comput. Vision (ICCV); Oct; Seoul, Korea. p. 6688–6697.
   Google Scholar
• Zhu X, Helmer EH, Gao F, Liu D, Chen J, Lefsky MA. 2016. A flexible spatiotemporal method for fusing satellite images with different resolutions. Remote Sens Environ. 172:165–177.
   Web of Science ®Google Scholar
• Zhu X, Zhan W, Zhou J, Chen X, Liang Z, Xu S, Chen J. 2022. A novel framework to assess all-round performances of spatiotemporal fusion models. Remote Sens Environ. 274:113002.
   Web of Science ®Google Scholar
• Zhukov B, Oertel D, Lanzl F, Reinhackel G. 1999. Unmixing-based multisensor multiresolution image fusion. IEEE Trans Geosci Remote Sens. 37(3):1212–1226.
   Web of Science ®Google Scholar