Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks

References

• Audebert, N., B. Le Saux, and S. Lefèvre. 2016. “Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-Scale Deep Networks.” In Asian Conference on Computer Vision, Taibei, China, 180–196. Springer.
   Google Scholar
• Audebert, N., B. Le Saux, and S. Lefèvre. 2017. “Joint Learning from Earth Observation and OpenStreetmap Data to Get Faster Better Semantic Maps.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hawaii, USA, 67–75.
   Google Scholar
• Audebert, N., B. Le Saux, and S. Lefèvre. 2018. “Beyond RGB: Very High Resolution Urban Remote Sensing with Multimodal Deep Networks.” ISPRS Journal of Photogrammetry and Remote Sensing 140: 20–32. doi:10.1016/j.isprsjprs.2017.11.011.
   Web of Science ®Google Scholar
• Audebert, N., B. Le Saux, and S. Lefèvre. 2019. “Deep Learning for Classification of Hyperspectral Data: A Comparative Review.” IEEE Geoscience and Remote Sensing Magazine 7 (2): 159–173. doi:10.1109/MGRS.2019.2912563.
   Web of Science ®Google Scholar
• Badrinarayanan, V., A. Kendall, and R. Cipolla. 2017. “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.” IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (12): 2481–2495. doi:10.1109/TPAMI.2016.2644615.
   PubMed Web of Science ®Google Scholar
• Bello, O. M. and Y. Adedoyin Aina. 2014. “Satellite Remote Sensing as a Tool in Disaster Management and Sustainable Development: Towards a Synergistic Approach.” Procedia-Social and Behavioral Sciences 120: 365–373. doi:10.1016/j.sbspro.2014.02.114.
   Google Scholar
• Buslaev, A., V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, and A. A. Kalinin. 2020. ”Albumentations: Fast and Flexible Image Augmentations”. Information 11 (2): 125. doi:10.3390/info11020125.
   Web of Science ®Google Scholar
• Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. 2018. “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected Crfs.” IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (4): 834–848. doi:10.1109/TPAMI.2017.2699184.
   PubMed Web of Science ®Google Scholar
• Chiu, M. T., X. Xu, K. Wang, J. Hobbs, N. Hovakimyan, T. S. Huang, and H. Shi, et al. 2020a. “The 1st Agriculture-Vision Challenge: Methods and Results.” In the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Virtual, 212–218.
   Google Scholar
• Chiu, M. T., X. Xu, Y. Wei, Z. Huang, A. G. Schwing, R. Brunner, and H. Khachatrian, et al. 2020b. “Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis.” In the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Virtual, June.
   Google Scholar
• Couprie, C., C. Farabet, L. Najman, and Y. LeCun. 2013. ”Learning Hierarchical Features for Scene Labeling.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8): 1915–1929. arXiv preprint arXiv:1301.3572. doi:10.1109/TPAMI.2012.231.
   PubMed Web of Science ®Google Scholar
• Diao, Q., Y. Dai, C. Zhang, Y. Wu, X. Feng, and F. Pan. 2022. “Superpixel- Based Attention Graph Neural Network for Semantic Segmentation in Aerial Images.” Remote Sensing 14 (2): 305. doi:10.3390/rs14020305.
   Web of Science ®Google Scholar
• Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, et al. 2021. ”An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ICLR.
   Google Scholar
• Fan, C., C. Zhang, A. Yahja, and A. Mostafavi. 2021. “Disaster City Digital Twin: A Vision for Integrating Artificial and Human Intelligence for Disaster Management.” International Journal of Information Management 56: 102049. doi:10.1016/j.ijinfomgt.2019.102049.
   Web of Science ®Google Scholar
• Feng, Q., D. Zhu, J. Yang, and B. Li. 2019. “Multisource Hyperspectral and Lidar Data Fusion for Urban Land-Use Mapping Based on a Modified Two-Branch Convolutional Neural Network.” ISPRS International Journal of Geo-Information 8 (1): 28. doi:10.3390/ijgi8010028.
   Web of Science ®Google Scholar
• Fu, J., J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu. 2019a. “Dual Attention Network for Scene Segmentation.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Long Beach, CA, 3141–3149.
   Google Scholar
• Ghosh, A., M. Ehrlich, S. Shah, L. Davis, and R. Chellappa. 2018. “Stacked U-Nets for Ground Material Segmentation in Remote Sensing Imagery.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, Utah, 252–2524.
   Google Scholar
• Gómez-Chova, L., D. Tuia, G. Moser, and G. Camps-Valls. 2015. “Multimodal Classification of Remote Sensing Images: A Review and Future Directions.” Proceedings of the IEEE 103 (9): 1560–1584. doi:10.1109/JPROC.2015.2449668.
   Web of Science ®Google Scholar
• Hazirbas, C., L. Ma, C. Domokos, and D. Cremers. 2016. “Fusenet: Incorporating Depth into Semantic Segmentation via Fusion-Based Cnn Architecture.” In Asian Conference on Computer Vision, Taipei, China, 213–228. Springer.
   Google Scholar
• Hong, D., L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang. 2020. “More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification.” IEEE Transactions on Geoscience and Remote Sensing, 4340–4354 .
   Web of Science ®Google Scholar
• Howard, A., M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, and W. Wang, et al. 2019. “Searching for Mobilenetv3.” In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 1314–1324.
   Google Scholar
• Hu, J., L. Shen, and G. Sun. 2018. “Squeeze-And-Excitation Networks.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, 7132–7141.
   Google Scholar
• Jiang, J., C. Lyu, Y. H. Siying Liu, and X. Hao. 2020. “Rwsnet: A Semantic Segmentation Network Based on SegNet Combined with Random Walk for Remote Sensing.” International Journal of Remote Sensing 41: 487–505. doi:10.1080/01431161.2019.1643937.
   Web of Science ®Google Scholar
• Kampffmeyer, M., A.-B. Salberg, and R. Jenssen. 2016. “Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, Nevada, 1–9.
   Google Scholar
• Kampffmeyer, M., A.-B. Salberg, and R. Jenssen. 2018. “Urban Land Cover Classification with Missing Data Modalities Using Deep Convolutional Neural Networks.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11 (6): 1758–1768. doi:10.1109/JSTARS.2018.2834961.
   Web of Science ®Google Scholar
• Kingma, D. P. and J. Ba. 2014. “Adam: A Method for Stochastic Optimization.” CoRR abs/1412.6980. http://arxiv.org/abs/1412.6980.
   Google Scholar
• Kipf, T. N. and M. Welling. 2016. ”Semi-Supervised Classification with Graph Convolutional Networks.” arXiv preprint arXiv:1609.02907.
   Google Scholar
• Li, X., L. Lei, Y. Sun, M. Li, and G. Kuang. 2020. “Multimodal Bilinear Fusion Network with Second-Order Attention-Based Channel Selection for Land Cover Classification.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13: 1011–1026. doi:10.1109/JSTARS.2020.2975252.
   Web of Science ®Google Scholar
• Lin, G., C. Shen, A. Van Den Hengel, and I. Reid. 2016. “Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Las Vegas, Nevada, 3194–3203.
   Google Scholar
• Lin, T.-Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017. “Feature Pyramid Networks for Object Detection.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, USA, 2117–2125.
   Google Scholar
• Liu, Y., S. Piramanayagam, S. T. Monteiro, and E. Saber. 2019. “Semantic Segmentation of Multisensor Remote Sensing Imagery with Deep ConvNets and Higher-Order Conditional Random Fields.” Journal of Applied Remote Sensing 13 (1): 16501. doi:10.1117/1.JRS.13.016501.
   Web of Science ®Google Scholar
• Liu, Q., M. Kampffmeyer, R. Jenssen, and A. B. Salberg. 2020. “Dense Dilated Convolutions’ Merging Network for Land Cover Classification.” IEEE Transactions on Geoscience and Remote Sensing 58 (9): 6309–6320. doi:10.1109/TGRS.2020.2976658.
   Web of Science ®Google Scholar
• Liu, Q., M. Kampffmeyer, R. Jenssen, and A.-B. Salberg. 2020a. “Self- Constructing Graph Convolutional Networks for Semantic Labeling.” In Proceedings of IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium, Virtual.
   Google Scholar
• Liu, Q., M. Kampffmeyer, R. Jenssen, and A.-B. Salberg. 2020b. “Multi-View Self-Constructing Graph Convolutional Networks with Adaptive Class Weighting Loss for Semantic Segmentation.” In the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Virtual, June.
   Google Scholar
• Long, J., E. Shelhamer, and T. Darrell. 2015. “Fully Convolutional Networks for Semantic Segmentation.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 3431–3440.
   Google Scholar
• Maggiori, E., Y. Tarabalka, G. Charpiat, and P. Alliez. 2017. “Can Semantic Labeling Methods Generalize to Any City? the Inria Aerial Image Labeling Benchmark.” In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Texas, USA, 3226–3229. IEEE.
   Google Scholar
• Marmanis, D., K. Schindler, J. Dirk Wegner, S. Galliani, M. Datcu, and U. Stilla. 2016. “Classification with an Edge: Improving Semantic Image Segmentation with Boundary Detection.” CoRr abs/1612.01337. http://arxiv.org/abs/1612.01337.
   Google Scholar
• Mohla, S., S. Pande, B. Banerjee, and S. Chaudhuri. 2020. “FusAtnet: Dual Attention Based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDar Classification.” In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Virtual, 416–425.
   Google Scholar
• Mou, L. and X. X. Zhu. 2019. “Learning to Pay Attention on Spectral Domain: A Spectral Attention Module-Based Convolutional Network for Hyperspectral Image Classification.” IEEE Transactions on Geoscience and Remote Sensing 58 (1): 110–122. doi:10.1109/TGRS.2019.2933609.
   Web of Science ®Google Scholar
• Paisitkriangkrai, S., J. Sherrah, P. Janney, and V.-D. Hengel, et al. 2015. “Effective Semantic Pixel Labelling with Convolutional Networks and Conditional Random Fields.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, USA, 36–43.
   Google Scholar
• Pashaei, M., H. Kamangir, M. J. Starek, and P. Tissot. 2020. “Review and Evaluation of Deep Learning Architectures for Efficient Land Cover Mapping with UAS Hyper-Spatial Imagery: A Case Study Over a Wetland.” Remote Sensing 12 (6): 959. doi:10.3390/rs12060959.
   Web of Science ®Google Scholar
• Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” In International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 234–241. Springer.
   Google Scholar
• Rottensteiner, F., G. Sohn, J. Jung, M. Gerke, C. Baillard, S. Benitez, and U. Breitkopf. 2012. ”The ISPRS Benchmark on Urban Object Classification and 3D Building Reconstruction.” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences I-3 (1): 293–298. Nr.1. doi:10.5194/isprsannals-I-3-293-2012.
   Google Scholar
• Salberg, A.-B. 2011. “Land Cover Classification of Cloud-Contaminated Multitemporal High-Resolution Images.” IEEE Transactions on Geoscience and Remote Sensing 49 (1): 377–387. doi:10.1109/TGRS.2010.2052464.
   Web of Science ®Google Scholar
• Salberg, A.-B., Ø. Rudjord, and A. H. Schistad Solberg. 2014. “Oil Spill Detection in Hybrid-Polarimetric SAR Images.” IEEE Transactions on Geoscience and Remote Sensing 52 (10): 6521–6533. doi:10.1109/TGRS.2013.2297193.
   Web of Science ®Google Scholar
• Sherrah, J. 2016. “Fully Convolutional Networks for Dense Semantic Labelling of High- Resolution Aerial Imagery.” CoRR abs/1606.02585. http://arxiv.org/abs/1606.02585.
   Google Scholar
• Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Ukasz Kaiser, and I. Polosukhin. 2017. ”Attention is All You Need”. In Advances in Neural Information Processing Systems 30, In edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, 5998–6008. San Jose, California: Curran Associates, Inc.
   Google Scholar
• Wambugu, N., Y. Chen, Z. Xiao, M. Wei, S. Aminu Bello, J. Marcato Junior, and J. Li. 2021. “A Hybrid Deep Convolutional Neural Network for Accurate Land Cover Classification.” International Journal of Applied Earth Observation and Geoinformation 103: 102515. doi:10.1016/j.jag.2021.102515.
   Web of Science ®Google Scholar
• Wang, H., Y. Wang, Q. Zhang, S. Xiang, and C. Pan. 2017. “Gated Convolutional Neural Network for Semantic Segmentation in High-Resolution Images.” Remote Sensing 9 (5): 446. doi:10.3390/rs9050446.
   Web of Science ®Google Scholar
• Xu, X., W. Li, Q. Ran, Q. Du, L. Gao, and B. Zhang. 2017. “Multisource Remote Sensing Data Classification Based on Convolutional Neural Network.” IEEE Transactions on Geoscience and Remote Sensing 56 (2): 937–949. doi:10.1109/TGRS.2017.2756851.
   Web of Science ®Google Scholar
• Xu, Y., B. Du, and L. Zhang. 2018. “Multi-Source Remote Sensing Data Classification via Fully Convolutional Networks and Post-Classification Processing.” In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 3852–3855.
   Google Scholar
• Xu, Y., B. Du, L. Zhang, D. Cerra, M. Pato, E. Carmona, S. Prasad, N. Yokoya, R. Hänsch, and B. Le Saux. 2019. “Advanced Multi-Sensor Optical Remote Sensing for Urban Land Use and Land Cover Classification: Outcome of the 2018 IEEE GRSS Data Fusion Contest.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (6): 1709–1724. doi:10.1109/JSTARS.2019.2911113.
   Web of Science ®Google Scholar
• Xu, Q., X. Yuan, C. Ouyang, and Y. Zeng. 2020. “Attention-Based Pyramid Network for Segmentation and Classification of High-Resolution and Hyperspectral Remote Sensing Images.” Remote Sensing 12 (21): 3501. doi:10.3390/rs12213501.
   Web of Science ®Google Scholar
• Xu, Q., X. Yuan, and C. Ouyang. 2020. “Class-Aware Domain Adaptation for Semantic Segmentation of Remote Sensing Images.” IEEE Transactions on Geoscience and Remote Sensing 60: 1–17. doi:10.1109/TGRS.2020.3036452.
   Web of Science ®Google Scholar
• Yuan, Y., X. Chen, and J. Wang. 2020. “Object-Contextual Representations for Semantic Segmentation.” In Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, United Kingdom, 173–190.
   Google Scholar
• Zhao, H., J. Shi, X. Qi, X. Wang, and J. Jia. 2017. “Pyramid Scene Parsing Network.” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, 6230–6239.
   Google Scholar
• Zhou, W., Y. Lv, J. Lei, and L. Yu. 2021. “Global and Local-Contrast Guides Content-Aware Fusion for RGB-D Saliency Prediction“. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51 3641–3649. doi:10.1109/TSMC.2019.2957386
   Web of Science ®Google Scholar
• Zhou, W., Q. Guo, J. Lei, L. Yu, and J.-N. Hwang. 2021a. “IRFR- Net: Interactive Recursive Feature-Reshaping Network for Detecting Salient Objects in RGB-D Images“. IEEE Transactions on Neural Networks and Learning Systems 1–13: . doi:10.1109/TNNLS.2021.3105484
   Web of Science ®Google Scholar
• Zhou, W., J. Liu, J. Lei, L. Yu, and J.-N. Hwang. 2021b. “Gmnet: Graded- Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation.” IEEE Transactions on Image Processing 30: 7790–7802. doi:10.1109/TIP.2021.3109518.
   PubMed Web of Science ®Google Scholar