Spatiotemporal Fusion of Remote Sensing Images using a Convolutional Neural Network with Attention and Multiscale Mechanisms

References

• Ancuti, C. O., and C. Ancuti. 2013. “Single Image Dehazing by Multi-scale Fusion.” IEEE Transactions on Image Processing 22 (8): 3271–3282. doi:10.1109/TIP.2013.2262284.
   PubMed Web of Science ®Google Scholar
• Belgiu, M., and A. Stein. 2019. “Spatiotemporal Image Fusion in Remote Sensing.” Remote Sensing 11 (7): 818. doi:10.3390/rs11070818.
   Web of Science ®Google Scholar
• Bhattarai, N., L. J. Quackenbush, M. Dougherty, and L. J. Marzen. 2015. “A Simple Landsat–MODIS Fusion Approach for Monitoring Seasonal Evapotranspiration at 30 M Spatial Resolution.” International Journal of Remote Sensing 36 (1): 115–143. doi:10.1080/01431161.2014.990645.
   Web of Science ®Google Scholar
• Chen, B., B. Huang, and X. Bing. 2015. “Comparison of Spatiotemporal Fusion Models: A Review.” Remote Sensing 7 (2): 1798–1835. doi:10.3390/rs70201798.
   Web of Science ®Google Scholar
• Chen, B., B. Huang, and X. Bing. 2017. “A Hierarchical Spatiotemporal Adaptive Fusion Model Using One Image Pair.” International Journal of Digital Earth 10 (6): 639–655. doi:10.1080/17538947.2016.1235621.
   Web of Science ®Google Scholar
• Dai, P., H. Zhang, L. Zhang, and H. Shen. 2018. “A Remote Sensing Spatiotemporal Fusion Model of Landsat and Modis Data via Deep Learning.” Paper presented at the IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.
   Google Scholar
• Emelyanova, I. V., T. R. McVicar, T. G. Van Niel, L. T. Li, and A. I. Van Dijk. 2013. “Assessing the Accuracy of Blending Landsat–MODIS Surface Reflectances in Two Landscapes with Contrasting Spatial and Temporal Dynamics: A Framework for Algorithm Selection.” Remote Sensing of Environment 133: 193–209. doi:10.1016/j.rse.2013.02.007.
   Web of Science ®Google Scholar
• Fung, C. H., M. S. Wong, and P. W. Chan. 2019. “Spatio-Temporal Data Fusion for Satellite Images Using Hopfield Neural Network.” Remote Sensing 11 (18): 2077. doi:10.3390/rs11182077.
   Web of Science ®Google Scholar
• Gao, F., J. Masek, M. Schwaller, and F. Hall. 2006. “On the Blending of the Landsat and MODIS Surface Reflectance: Predicting Daily Landsat Surface Reflectance.” IEEE Transactions on Geoscience Remote Sensing 44 (8): 2207–2218. doi:10.1109/TGRS.2006.872081.
   Web of Science ®Google Scholar
• Gupta, H. V., H. Kling, K. K. Yilmaz, and G. F. Martinez. 2009. “Decomposition of the Mean Squared Error and NSE Performance Criteria: Implications for Improving Hydrological Modelling.” Journal of Hydrology 377 (1–2): 80–91. doi:10.1016/j.jhydrol.2009.08.003.
   Web of Science ®Google Scholar
• Hu, J., L. Shen, and G. Sun. 2018. “Squeeze-and-excitation Networks.” Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT.
   Google Scholar
• Huang, B., and H. Song. 2012. “Spatiotemporal Reflectance Fusion via Sparse Representation.” IEEE Transactions on Geoscience Remote Sensing 50 (10): 3707–3716. doi:10.1109/TGRS.2012.2186638.
   Web of Science ®Google Scholar
• Jaderberg, M., K. Simonyan, and A. Zisserman. 2015. “Spatial Transformer Networks”. Paper presented at the Advances in neural information processing systems, Montreal, Canada.
   Google Scholar
• Khan, M. M., L. Alparone, and J. Chanussot. 2009. “Pansharpening Quality Assessment Using the Modulation Transfer Functions of Instruments.” IEEE Transactions on Geoscience Remote Sensing 47 (11): 3880–3891. doi:10.1109/TGRS.2009.2029094.
   Web of Science ®Google Scholar
• Korhonen, J., and J. You. 2012. “Peak Signal-to-noise Ratio Revisited: Is Simple Beautiful?” Paper presented at the 2012 Fourth International Workshop on Quality of Multimedia Experience, Yarra Valley, Australia.
   Google Scholar
• Lai, W. S., J. B. Huang, N. Ahuja, and M. H. Yang. 2019. “Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (11):2599–2613. doi: 10.1109/TPAMI.2018.2865304
   PubMed Web of Science ®Google Scholar
• Li, J., L. Yunfei, H. Lin, J. Chen, and A. Plaza. 2020a. “Spatio-temporal Fusion for Remote Sensing Data: An Overview and New Benchmark.” Science China Information Sciences 63: 4. doi:10.1007/s11432-019-2785-y.
   Web of Science ®Google Scholar
• Li, X., G. M. Foody, D. S. Boyd, G. Yong, Y. Zhang, D. Yun, and F. Ling. 2020b. “SFSDAF: An Enhanced FSDAF that Incorporates Sub-pixel Class Fraction Change Information for Spatio-temporal Image Fusion.” Remote Sensing of Environment 237. doi:10.1016/j.rse.2019.111537.
   Web of Science ®Google Scholar
• Lin, T., P. Dollar, R. Girshick, H. Kaiming, B. Hariharan, and S. Belongie. 2017. “Feature Pyramid Networks for Object Detection.” Paper presented at the computer vision and pattern recognition, Maryland, USA.
   Google Scholar
• Liu, X., C. Deng, J. Chanussot, D. Hong, and B. Zhao. 2019. “Stfnet: A Two-stream Convolutional Neural Network for Spatiotemporal Image Fusion.” IEEE Transactions on Geoscience Remote Sensing 57 (9): 6552–6564. doi:10.1109/TGRS.2019.2907310.
   Web of Science ®Google Scholar
• Liu, X., C. Deng, S. Wang, G.-B. Huang, B. Zhao, and P. Lauren. 2016. “Fast and Accurate Spatiotemporal Fusion Based upon Extreme Learning Machine.” IEEE Geoscience Remote Sensing Letters 13 (12): 2039–2043. doi:10.1109/LGRS.2016.2622726.
   Web of Science ®Google Scholar
• Ma, J., P. Liang, Y. Wei, C. Chen, X. Guo, W. Jia, and J. Jiang. 2020b. “Infrared and Visible Image Fusion via Detail Preserving Adversarial Learning.” Information Fusion 54: 85–98. doi:10.1016/j.inffus.2019.07.005.
   Web of Science ®Google Scholar
• Ma, J., X. Han, J. Jiang, X. Mei, and X. Zhang. 2020a. “DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion.” IEEE Transactions on Image Processing 29: 4980–4995. doi:10.1109/TIP.2020.2977573.
   Web of Science ®Google Scholar
• Ma, J., Y. Wei, P. Liang, L. Chang, and J. Jiang. 2019. “FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion.” Information Fusion 48: 11–26. doi:10.1016/j.inffus.2018.09.004.
   Web of Science ®Google Scholar
• Masi, G., D. Cozzolino, L. Verdoliva, and G. Scarpa. 2016. “Pansharpening by Convolutional Neural Networks.” Remote Sensing 8 (7): 594. doi:10.3390/rs8070594.
   Web of Science ®Google Scholar
• Nair, V., and G. E. Hinton. 2010. “Rectified Linear Units Improve Restricted Boltzmann Machines.” Paper presented at the Proceedings of the 27th international conference on machine learning (ICML-10), Haifa, Israel.
   Google Scholar
• Shao, Z., J. Cai, F. Peng, H. Leiqiu, and T. Liu. 2019. “Deep Learning-based Fusion of Landsat-8 and Sentinel-2 Images for a Harmonized Surface Reflectance Product.” Remote Sensing of Environment 235. doi:10.1016/j.rse.2019.111425.
   Web of Science ®Google Scholar
• Shen, H., W. Penghai, Y. Liu, A. Tinghua, Y. Wang, and X. Liu. 2013. “A Spatial and Temporal Reflectance Fusion Model considering Sensor Observation Differences.” International Journal of Remote Sensing 34 (12): 4367–4383. doi:10.1080/01431161.2013.777488.
   Web of Science ®Google Scholar
• Song, H., and B. Huang. 2012. “Spatiotemporal Satellite Image Fusion through One-pair Image Learning.” IEEE Transactions on Geoscience Remote Sensing 51 (4): 1883–1896. doi:10.1109/TGRS.2012.2213095.
   Web of Science ®Google Scholar
• Song, H., Q. Liu, G. Wang, R. Hang, and B. Huang. 2018. “Spatiotemporal Satellite Image Fusion Using Deep Convolutional Neural Networks.” IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing 11 (3): 821–829. doi:10.1109/JSTARS.2018.2797894.
   Web of Science ®Google Scholar
• Tan, Z., D. Liping, M. Zhang, L. Guo, and M. Gao. 2019. “An Enhanced Deep Convolutional Model for Spatiotemporal Image Fusion.” Remote Sensing 11: 24. doi:10.3390/rs11242898.
   Web of Science ®Google Scholar
• Tan, Z., P. Yue, D. Liping, and J. Tang. 2018. “Deriving High Spatiotemporal Remote Sensing Images Using Deep Convolutional Network.” Remote Sensing 10 (7): 1066. doi:10.3390/rs10071066.
   Web of Science ®Google Scholar
• Wang, F., M. Jiang, C. Qian, S. Yang, L. Cheng, H. Zhang, X. Wang, and X. Tang. 2017. “Residual Attention Network for Image Classification.” Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Maryland, USA.
   Google Scholar
• Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. “Image Quality Assessment: From Error Visibility to Structural Similarity.” IEEE Transactions on Image Processing 13 (4): 600–612. doi:10.1109/TIP.2003.819861.
   PubMed Web of Science ®Google Scholar
• Wang, Z., Y. Peng, K. Jiang, J. Jiang, Z. Han, L. Tao, and M. Jiayi. 2019. “Multi-Memory Convolutional Neural Network for Video Super-Resolution.” IEEE Transactions on Image Processing 28 (5): 2530–2544. doi:10.1109/TIP.2018.2887017.
   Web of Science ®Google Scholar
• Wei, J., L. Wang, P. Liu, X. Chen, L. Wei, and A. Y. Zomaya. 2017a. “Spatiotemporal Fusion of Modis and Landsat-7 Reflectance Images via Compressed Sensing.” IEEE Transactions on Geoscience Remote Sensing 55 (12): 7126–7139. doi:10.1109/TGRS.2017.2742529.
   Web of Science ®Google Scholar
• Wei, Y., Q. Yuan, H. Shen, and L. Zhang. 2017b. “Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network.” IEEE Geoscience Remote Sensing Letters 14 (10): 1795–1799. doi:10.1109/LGRS.2017.2736020.
   Web of Science ®Google Scholar
• Wu, B., B. Huang, and L. Zhang. 2015. “An Error-bound-regularized Sparse Coding for Spatiotemporal Reflectance Fusion.” IEEE Transactions on Geoscience Remote Sensing 53 (12): 6791–6803. doi:10.1109/TGRS.2015.2448100.
   Web of Science ®Google Scholar
• Wu, M., W. Huang, Z. Niu, and C. Wang. 2015. “Generating Daily Synthetic Landsat Imagery by Combining Landsat and MODIS Data.” Sensors 15 (9): 24002–24025. doi:10.3390/s150924002.
   PubMed Web of Science ®Google Scholar
• Wu, M., Z. Niu, C. Wang, W. Chaoyang, and L. Wang. 2012. “Use of MODIS and Landsat Time Series Data to Generate High-resolution Temporal Synthetic Landsat Data Using a Spatial and Temporal Reflectance Fusion Model.” Journal of Applied Remote Sensing 6 (1): 063507. doi:10.1117/1.JRS.6.063507.
   Web of Science ®Google Scholar
• Xue, J., Y. Leung, and T. Fung. 2019. “An Unmixing-Based Bayesian Model for Spatio-Temporal Satellite Image Fusion in Heterogeneous Landscapes.” Remote Sensing 11 (3): 324. doi:10.3390/rs11030324.
   Web of Science ®Google Scholar
• Yi, P., Z. Wang, K. Jiang, J. Jiang, and J. Ma. 2019a. “Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations.” In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 3106–3115, Seoul, Korea (South).
   Google Scholar
• Yi, P., Z. Wang, K. Jiang, Z. Shao, and M. Jiayi. 2019b. “Multi-Temporal Ultra Dense Memory Network For Video Super-Resolution.” IEEE Transactions on Circuits and Systems for Video Technology :1.
   Google Scholar
• Yu, F., and V. Koltun. 2015. “Multi-scale Context Aggregation by Dilated Convolutions.” arXiv preprint arXiv:.07122.
   Google Scholar
• Yuan, Q., H. Shen, L. Tongwen, L. Zhiwei, L. Shuwen, Y. Jiang, X. Hongzhang, et al. 2020. “Deep Learning in Environmental Remote Sensing: Achievements and Challenges.” Remote Sensing of Environment 241. doi:10.1016/j.rse.2020.111716.
   Web of Science ®Google Scholar
• Yuan, Q., Y. Wei, X. Meng, H. Shen, and L. Zhang. 2018. “A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-sharpening.” IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing 11 (3): 978–989. doi:10.1109/JSTARS.2018.2794888.
   Web of Science ®Google Scholar
• Zhang, Y., C. Liu, M. Sun, and O. Yangjun. 2019. “Pan-Sharpening Using an Efficient Bidirectional Pyramid Network.” IEEE Transactions on Geoscience Remote Sensing 57 (8): 5549–5563. doi:10.1109/TGRS.2019.2900419.
   Web of Science ®Google Scholar
• Zhang, Y., G. M. Foody, F. Ling, L. Xiaodong, G. Yong, D. Yun, and P. M. Atkinson. 2018. “Spatial-temporal Fraction Map Fusion with Multi-scale Remotely Sensed Images.” Remote Sensing of Environment 213: 162–181. doi:10.1016/j.rse.2018.05.010.
   Web of Science ®Google Scholar
• Zhu, X., D. Cheng, Z. Zhang, S. Lin, and J. Dai. 2019. “An Empirical Study of Spatial Attention Mechanisms in Deep Networks.” Paper presented at the Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea (South).
   Google Scholar
• Zhu, X., E. H. Helmer, F. Gao, D. Liu, J. Chen, and M. A. Lefsky. 2016. “A Flexible Spatiotemporal Method for Fusing Satellite Images with Different Resolutions.” Remote Sensing of Environment 172: 165–177. doi:10.1016/j.rse.2015.11.016.
   Web of Science ®Google Scholar
• Zhu, X., F. Cai, J. Tian, and T. Kay-Ann Williams. 2018. “Spatiotemporal Fusion of Multisource Remote Sensing Data: Literature Survey, Taxonomy, Principles, Applications, and Future Directions.” Remote Sensing 10 (4): 527.
   Web of Science ®Google Scholar
• Zhu, X., J. Chen, F. Gao, X. Chen, and J. G. Masek. 2010. “An Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model for Complex Heterogeneous Regions.” Remote Sensing of Environment 114 (11): 2610–2623. doi:10.1016/j.rse.2010.05.032.
   Web of Science ®Google Scholar