U-SeqNet: learning spatiotemporal mapping relationships for multimodal multitemporal cloud removal

References

• Bermudez, J. D., P. N. Happ, D. A. B. Oliveira, and R. Q. Feitosa. 2018. “SAR to Optical Image Synthesis for Cloud Removal with Generative Adversarial Networks.” Paper presented at the ISPRS TC I Mid-term Symposium on Innovative Sensing - From Sensors to Methods and Applications, Karlsruhe, GERMANY.
   Google Scholar
• Cheng, Q., H. Shen, L. Zhang, Q. Yuan, and C. Zeng. 2014. “Cloud Removal for Remotely Sensed Images by Similar Pixel Replacement Guided with a Spatio-Temporal MRF Model.” ISPRS Journal of Photogrammetry and Remote Sensing 92:54–19. https://doi.org/10.1016/j.isprsjprs.2014.02.015.
   Web of Science ®Google Scholar
• Cort, J. W., and M. Kenji. 2005. “Advantages of the Mean Absolute Error (MAE) Over the Root Mean Square Error (RMSE) in Assessing Average Model Performance.” Climate Research 30 (1): 79–82. https://doi.org/10.3354/cr030079.
   Web of Science ®Google Scholar
• Ding, H., F. Xie, Y. Zi, W. Liao, and X. Song. 2023. “Feedback Network for Compact Thin Cloud Removal.” Geoscience and Remote Sensing Letters, IEEE 20:1–5. https://doi.org/10.1109/LGRS.2023.3256416.
   Web of Science ®Google Scholar
• Ebel, P., V. S. Fare Garnot, M. Schmitt, J. D. Wegner, and X. Xiang Zhu. 2023. “UnCrtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series.” Paper presented at the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada.
   Google Scholar
• Ebel, P., M. Schmitt, and X. X. Zhu. 2021. “Internal Learning for Sequence-To-Sequence Cloud Removal via Synthetic Aperture Radar Prior Information.” Paper presented at the 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium.
   Google Scholar
• Ebel, P., Y. Xu, M. Schmitt, and X. Xiang Zhu. 2022. “Sen12MS-CR-TS: A Remote-Sensing Data Set for Multimodal Multitemporal Cloud Removal.” IEEE Trans. Geosci. Remote Sens. 60. https://doi.org/10.1109/tgrs.2022.3146246.
   Google Scholar
• Friedl, M. A., D. K. McIver, J. C. F. Hodges, X. Y. Zhang, D. Muchoney, A. H. Strahler, C. E. Woodcock, et al. 2002. “Global Land Cover Mapping from MODIS: Algorithms and Early Results.” Remote Sensing of Environment 83 (1–2): 287–302. https://doi.org/10.1016/s0034-4257(02)00078-0.
   Web of Science ®Google Scholar
• Ghamisi, P., B. Rasti, N. Yokoya, Q. Wang, B. Hoefle, L. Bruzzone, F. Bovolo, et al. 2019. “Multisource and Multitemporal Data Fusion in Remote Sensing a Comprehensive Review of the State of the Art.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 7 (1): 6–39. https://doi.org/10.1109/mgrs.2018.2890023.
   Web of Science ®Google Scholar
• Goodfellow, I. J., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. “Generative adversarial nets.” In Proceedings of the 27th International Conference on Neural Information Processing Systems, 2672–2680. Montreal, Canada: MIT Press.
   Google Scholar
• Hochreiter, S., and J. Schmidhuber. 1997. “Long Short-Term Memory.” Neural Computation 9 (8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
   PubMed Web of Science ®Google Scholar
• Jhin, S. Y., J. Lee, M. Jo, S. Kook, J. Jeon, J. Hyeong, J. Kim, and N. Park. 2022. “EXIT: Extrapolation and Interpolation-Based Neural Controlled Differential Equations for Time-Series Classification and Forecasting.” In Proceedings of the ACM Web Conference 2022, 3102–3112. Virtual Event, Lyon, France: Association for Computing Machinery.
   Google Scholar
• Ji, S., S. Wei, and M. Lu. 2019. “Fully Convolutional Networks for Multisource Building Extraction from an Open Aerial and Satellite Imagery Data Set.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 57 (1): 574–586. https://doi.org/10.1109/tgrs.2018.2858817.
   Web of Science ®Google Scholar
• Kaur, H., D. Koundal, and V. Kadyan. 2021. “Image Fusion Techniques: A Survey.” Archives of Computational Methods in Engineering 28 (7): 4425–4447. https://doi.org/10.1007/s11831-021-09540-7.
   PubMed Web of Science ®Google Scholar
• Kulkarni, S. C., and P. P. Rege. 2020. “Pixel level fusion techniques for SAR and optical images: A review.” Information Fusion 59:13–29. https://doi.org/10.1016/j.inffus.2020.01.003.
   Web of Science ®Google Scholar
• LeCun, Y., B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. “Backpropagation Applied to Handwritten Zip Code Recognition.” Neural Computation 1 (4): 541–551. https://doi.org/10.1162/neco.1989.1.4.541.
   Web of Science ®Google Scholar
• Li, X., Z. Du, Y. Huang, and Z. Tan. 2021. “A Deep Translation (GAN) Based Change Detection Network for Optical and SAR Remote Sensing Images.” ISPRS Journal of Photogrammetry and Remote Sensing 179:14–34. https://doi.org/10.1016/j.isprsjprs.2021.07.007.
   Web of Science ®Google Scholar
• Li, J., D. Hong, L. Gao, J. Yao, K. Zheng, B. Zhang, and J. Chanussot. 2022. “Deep Learning In Multimodal Remote Sensing Data Fusion: A Comprehensive Review.” International Journal of Applied Earth Observation and Geoinformation 112. https://doi.org/10.1016/j.jag.2022.102926.
   Google Scholar
• Lin, C.-H., P.-H. Tsai, K.-H. Lai, and J.-Y. Chen. 2013. “Cloud Removal from Multitemporal Satellite Images Using Information Cloning.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 51 (1): 232–241. https://doi.org/10.1109/tgrs.2012.2197682.
   Web of Science ®Google Scholar
• Liu, L., and B. Lei. 2018. “Can SAR Images and Optical Images Transfer with Each Other?.” Paper presented at the 38th IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Valencia, SPAIN.
   Google Scholar
• Liu, Z., H. Mao, C. Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. 2022. ”A ConvNet for the 2020s.” Paper presented at the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.
   Google Scholar
• Li, Y., F. Wei, Y. Zhang, W. Chen, and J. Ma. 2023. “HS2P: Hierarchical Spectral and Structure-Preserving Fusion Network for Multimodal Remote Sensing Image Cloud and Shadow Removal.” Information Fusion 94:215–228. https://doi.org/10.1016/j.inffus.2023.02.002.
   Web of Science ®Google Scholar
• Ma, L., Y. Liu, X. Zhang, Y. Ye, G. Yin, and B. Alan Johnson. 2019. “Deep Learning in Remote Sensing Applications: A Meta-Analysis and Review.” ISPRS Journal of Photogrammetry and Remote Sensing 152:166–177. https://doi.org/10.1016/j.isprsjprs.2019.04.015.
   Web of Science ®Google Scholar
• MañMañAs, O., A. Lacoste, X. Giró-I-Nieto, D. Vázquez, and P. Rodriguez. 2021. “Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data.” In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 9394–9403. IEEE Computer Society.
   Google Scholar
• Ma, D., R. Wu, D. Xiao, and B. Sui. 2023. “Cloud Removal from Satellite Images Using a Deep Learning Model with the Cloud-Matting Method.” Remote Sensing 15:4. https://doi.org/10.3390/rs15040904.
   Google Scholar
• Meraner, A., P. Ebel, X. Xiang Zhu, and M. Schmitt. 2020. “Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion.” ISPRS Journal of Photogrammetry and Remote Sensing 166:333–346. https://doi.org/10.1016/j.isprsjprs.2020.05.013.
   PubMed Web of Science ®Google Scholar
• Mittal, A., A. K. Moorthy, and A. C. Bovik. 2011. “Blind/Referenceless Image Spatial Quality Evaluator.” Paper presented at the 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), Pacific Grove, CA, USA.
   Google Scholar
• Mittal, A., A. K. Moorthy, and A. C. Bovik. 2012. “No-Reference Image Quality Assessment in the Spatial Domain.” IEEE Transactions on Image Processing 21 (12): 4695–4708. https://doi.org/10.1109/TIP.2012.2214050.
   PubMed Web of Science ®Google Scholar
• Mittal, A., R. Soundararajan, and A. C. Bovik. 2013. “Making a “Completely Blind” Image Quality Analyzer.” IEEE Signal Processing Letters 20 (3): 209–212. https://doi.org/10.1109/LSP.2012.2227726.
   Web of Science ®Google Scholar
• Papadomanolaki, M., M. Vakalopoulou, and K. Karantzalos. 2021. “A Deep Multitask Learning Framework Coupling Semantic Segmentation and Fully Convolutional LSTM Networks for Urban Change Detection.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 59 (9): 7651–7668. https://doi.org/10.1109/tgrs.2021.3055584.
   Web of Science ®Google Scholar
• Pekel, J.-F., A. Cottam, N. Gorelick, and A. S. Belward. 2016. “High-Resolution Mapping of Global Surface Water and Its Long-Term Changes.” Nature 540 (7633): 418±. https://doi.org/10.1038/nature20584.
   PubMed Web of Science ®Google Scholar
• Peng, T., M. Liu, X. Liu, Q. Zhang, L. Wu, and X. Zou. 2022. “Reconstruction of Optical Image Time Series with Unequal Lengths SAR Based on Improved Sequence-Sequence Model.” IEEE Trans. Geosci. Remote Sens. 60. https://doi.org/10.1109/tgrs.2022.3208926.
   Google Scholar
• Reyes, M. F., S. Auer, N. Merkle, C. Henry, and M. Schmitt. 2019. “SAR-To-Optical Image Translation Based on Conditional Generative Adversarial Networks-Optimization, Opportunities and Limits.” Remote Sensing 11 (17). https://doi.org/10.3390/rs11172067.
   Web of Science ®Google Scholar
• Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Paper presented at the Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham.
   Google Scholar
• Rumelhart, D. E., G. E. Hinton, and R. J. Williams. 1986. “Learning Representations by Back-Propagating Errors.” Nature 323 (6088): 533–536. https://doi.org/10.1038/323533a0.
   Web of Science ®Google Scholar
• Sangiorgio, M., and F. Dercole. 2020. “Robustness of LSTM Neural Networks for Multi-Step Forecasting of Chaotic Time Series.” Chaos, Solitons & Fractals 139. https://doi.org/10.1016/j.chaos.2020.110045.
   Web of Science ®Google Scholar
• Schmitt, M., L. Hughes, and X. Zhu. 2018. “The SEN1-2 Dataset For Deep Learning In Sar-optical Data Fusion.” ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences IV-1: 141–146. https://doi.org/10.5194/isprs-annals-IV-1-141-2018.
   Google Scholar
• Sebastianelli, A., E. Puglisi, M. Pia Del Rosso, J. Mifdal, A. Nowakowski, P. Philippe Mathieu, F. Pirri, and S. Liberata Ullo. 2022. “PLFM: Pixel-Level Merging of Intermediate Feature Maps by Disentangling and Fusing Spatial and Temporal Data for Cloud Removal.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 60:1–16. https://doi.org/10.1109/tgrs.2022.3208694.
   Web of Science ®Google Scholar
• Shen, H., X. Li, Q. Cheng, C. Zeng, G. Yang, H. Li, and L. Zhang. 2015. “Missing Information Reconstruction of Remote Sensing Data: A Technical Review.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 3 (3): 61–85. https://doi.org/10.1109/mgrs.2015.2441912.
   Web of Science ®Google Scholar
• Shi, X., Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-C. Woo. 2015. “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting.” Paper presented at the 29th Annual Conference on Neural Information Processing Systems (NIPS), Montreal, CANADA.
   Google Scholar
• Shi, C., Z. Zhang, W. Zhang, C. Zhang, and Q. Xu. 2022. “Learning Multiscale Temporal-Spatial-Spectral Features via a Multipath Convolutional LSTM Neural Network for Change Detection with Hyperspectral Images.” IEEE Trans. Geosci. Remote Sens. 60. https://doi.org/10.1109/tgrs.2022.3176642.
   Google Scholar
• Stucker, C., V. S. F. Garnot, and K. Schindler. 2023. “U-TILISE: A Sequence-to-Sequence Model for Cloud Removal in Optical Satellite Time Series.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 61:1–16. https://doi.org/10.1109/TGRS.2023.3333391.
   Web of Science ®Google Scholar
• Sun, X., Y. Tian, W. Lu, P. Wang, R. Niu, H. Yu, and K. Fu. 2023. “From Single- To Multi-modal Remote Sensing Imagery Interpretation: A Survey And Taxonomy.” Science China Information Sciences 66:4. https://doi.org/10.1007/s11432-022-3588-0.
   Web of Science ®Google Scholar
• Sutskever, I., O. Vinyals, and Q. V. Le. 2014. ”Sequence to Sequence Learning with Neural Networks.” Paper presented at the 28th Conference on Neural Information Processing Systems (NIPS), Montreal, CANADA.
   Google Scholar
• Venkatanath, N., D. Praneeth, B. M. Chandrasekhar, S. S. Channappayya, and S. S. Medasani. 2015. “Blind Image Quality Evaluation Using Perception Based Features.” Paper presented at the 2015 Twenty First National Conference on Communications (NCC), Mumbai, India.
   Google Scholar
• Verbesselt, J., R. Hyndman, G. Newnham, and D. Culvenor. 2010. “Detecting Trend and Seasonal Changes in Satellite Image Time Series.” Remote Sensing of Environment 114 (1): 106–115. https://doi.org/10.1016/j.rse.2009.08.014.
   Web of Science ®Google Scholar
• Wang, Z., and A. C. Bovik. 2009. “Mean Squared Error: Love it or Leave It? A New Look at Signal Fidelity Measures.” IEEE Signal Processing Magazine 26 (1): 98–117. https://doi.org/10.1109/MSP.2008.930649.
   Web of Science ®Google Scholar
• Williams, R. J., and D. Zipser. 1989. “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks.” Neural Computation 1 (2): 270–280. https://doi.org/10.1162/neco.1989.1.2.270.
   Web of Science ®Google Scholar
• Xia, M., and K. Jia. 2022. “Reconstructing Missing Information of Remote Sensing Data Contaminated by Large and Thick Clouds Based on an Improved Multitemporal Dictionary Learning Method.” IEEE Trans. Geosci. Remote Sens. 60. https://doi.org/10.1109/tgrs.2021.3095067.
   Google Scholar
• Xie, F., M. Shi, Z. Shi, J. Yin, and D. Zhao. 2017. “Multilevel Cloud Detection in Remote Sensing Images Based on Deep Learning.” Journal of Selected Topics in Applied Earth Observations and Remote Sensing 10 (8): 3631–3640. https://doi.org/10.1109/JSTARS.2017.2686488.
   Web of Science ®Google Scholar
• Xu, F., Y. Shi, P. Ebel, L. Yu, G.-S. Xia, W. Yang, and X. Xiang Zhu. 2022. “GLF-CR: SAR-Enhanced Cloud Removal with Global–Local Fusion.” ISPRS Journal of Photogrammetry and Remote Sensing 192:268–278. https://doi.org/10.1016/j.isprsjprs.2022.08.002.
   Web of Science ®Google Scholar
• Xu, R., H. Zhang, and H. Lin. 2017. “Urban Impervious Surfaces Estimation from Optical and SAR Imagery: A Comprehensive Comparison.” Journal of Selected Topics in Applied Earth Observations and Remote Sensing 10 (9): 4010–4021. https://doi.org/10.1109/JSTARS.2017.2706747.
   Web of Science ®Google Scholar
• Yang, X., J. Zhao, H. Zhang, C. Dai, L. Zhao, Z. Ji, and I. Ganchev. 2022. “Remote Sensing Image Detection Based on YOLOv4 Improvements.” Institute of Electrical and Electronics Engineers Access 10:95527–95538. https://doi.org/10.1109/access.2022.3204053.
   Google Scholar
• Yuan, Q., H. Shen, T. Li, Z. Li, S. Li, Y. Jiang, H. Xu, et al. 2020. “Deep Learning In Environmental Remote Sensing: Achievements And Challenges.” Remote Sensing of Environment 241. https://doi.org/10.1016/j.rse.2020.111716.
   Web of Science ®Google Scholar
• Zhang, Q., X. Liu, M. Liu, X. Zou, L. Zhu, and X. Ruan. 2021. “Comparative Analysis of Edge Information and Polarization on SAR-To-Optical Translation Based on Conditional Generative Adversarial Networks.” Remote Sensing 13:1. https://doi.org/10.3390/rs13010128.
   PubMed Web of Science ®Google Scholar
• Zhang, Z., Q. Liu, and Y. Wang. 2018. “Road Extraction by Deep Residual U-Net.” Geoscience and Remote Sensing Letters, IEEE 15 (5): 749–753. https://doi.org/10.1109/lgrs.2018.2802944.
   Web of Science ®Google Scholar
• Zhang, Q., Q. Yuan, C. Zeng, X. Li, and Y. Wei. 2018. “Missing Data Reconstruction in Remote Sensing Image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 56 (8): 4274–4288. https://doi.org/10.1109/tgrs.2018.2810208.
   Web of Science ®Google Scholar
• Zhang, L., L. Zhang, and B. Du. 2016. “Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 4 (2): 22–40. https://doi.org/10.1109/mgrs.2016.2540798.
   Web of Science ®Google Scholar
• Zhang, Y., H. Zhang, and H. Lin. 2014. “Improving the Impervious Surface Estimation with Combined Use of Optical and SAR Remote Sensing Images.” Remote Sensing of Environment 141:155–167. https://doi.org/10.1016/j.rse.2013.10.028.
   Web of Science ®Google Scholar
• Zhao, Y., S. Shen, J. Hu, Y. Li, and J. Pan. 2022. “Cloud Removal Using Multimodal GAN with Adversarial Consistency Loss.” IEEE Geosci. Remote. Sens. Lett. 19. https://doi.org/10.1109/lgrs.2021.3093887.
   Google Scholar
• Zhou, W., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. “Image Quality Assessment: From Error Visibility to Structural Similarity.” IEEE Transactions on Image Processing 13 (4): 600–612. https://doi.org/10.1109/TIP.2003.819861.
   PubMed Web of Science ®Google Scholar
• Zhou, G., W. Chen, X. Qin, J. Li, and L. Wang. 2023. “Lithological Unit Classification Based on Geological Knowledge-Guided Deep Learning Framework for Optical Stereo Mapping Satellite Imagery.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 61:1–16. https://doi.org/10.1109/TGRS.2023.3327774.
   Google Scholar
• Zhou, G., J. Xu, W. Chen, X. Li, J. Li, and L. Wang. 2023. “Deep Feature Enhancement Method for Land Cover with Irregular and Sparse Spatial Distribution Features: A Case Study on Open-Pit Mining.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 61:1–20. https://doi.org/10.1109/TGRS.2023.3241331.
   Web of Science ®Google Scholar
• Zhu, X. X., D. Tuia, L. Mou, G. S. Xia, L. Zhang, F. Xu, and F. Fraundorfer. 2017. “Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources.” IEEE Transactions on Geoscience and Remote Sensing: A Publication of the IEEE Geoscience and Remote Sensing Society 5 (4): 8–36. https://doi.org/10.1109/MGRS.2017.2762307.
   Web of Science ®Google Scholar
• Zhu, Z., S. Wang, and C. E. Woodcock. 2015. “Improvement and Expansion of the Fmask Algorithm: Cloud, Cloud Shadow, and Snow Detection for Landsats 4–7, 8, and Sentinel 2 Images.” Remote Sensing of Environment 159:269–277. https://doi.org/10.1016/j.rse.2014.12.014.
   Web of Science ®Google Scholar
• Zhu, Z., and C. E. Woodcock. 2012. “Object-Based Cloud and Cloud Shadow Detection in Landsat Imagery.” Remote Sensing of Environment 118:83–94. https://doi.org/10.1016/j.rse.2011.10.028.
   Web of Science ®Google Scholar