292
Views
4
CrossRef citations to date
0
Altmetric
Research Article

Gradient Guided Pyramidal Convolution Residual Network with Interactive Connections for Pan-sharpening

, , &
Pages 5572-5602 | Received 11 Mar 2021, Accepted 09 Jun 2021, Published online: 09 Aug 2021

References

  • Ahn, N., B. Kang, and K.-A. Sohn. 2018. “Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network.” arXiv Preprint arXiv 08664: 1803.
  • Aiazzi, B., L. Alparone, S. Baronti, and A. Garzelli. 2002. “Context-driven Fusion of High Spatial and Spectral Resolution Images Based on Oversampled Multiresolution Analysis.” IEEE Transactions on Geoscience and Remote Sensing 40 (10): 2300–2312. doi:10.1109/TGRS.2002.803623.
  • Aiazzi, B., L. Alparone, S. Baronti, A. Garzelli, and M. Selva. 2003. “An MTF-based Spectral Distortion Minimizing Model for Pan-sharpening of Very High Resolution Multispectral Images of Urban Areas.” In 2003 2nd GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, 90–94. Berlin, Germany: IEEE.
  • Aiazzi, B., L. Alparone, S. Baronti, A. Garzelli, and M. Selva. 2006. “MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery.” Photogrammetric Engineering and Remote Sensing 72 (5): 591–596. doi:10.14358/PERS.72.5.591.
  • Alparone, L., B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva. 2008. “Multispectral and Panchromatic Data Fusion Assessment without Reference.” ASPRS Journal of Photogrammetric Engineering and Remote Sensing 74 (2): 193–200. doi:10.14358/PERS.74.2.193.
  • Burt, P. J., and E. H. Adelson. 1987. Martin Fischler and Oscar Firschein. “The Laplacian Pyramid as a Compact Image Code.” In Readings in Computer Vision, 671–679. Elsevier.
  • Carper, W. J. O. S. E. P. H., T. H. O. M. A. S. M. LILLESAND, and R. A. L. P. H. W. KIEFER. 1990. “The Use of Intensity-hue-saturation Transformations for Merging SPOT Panchromatic and Multispectral Image Data.” Photogrammetric Engineering and Remote Sensing 56 (4): 459–467.
  • Chavez, P., S. C. Sides, and J. A. Anderson. 1991. “Comparison of Three Different Methods to Merge Multiresolution and Multispectral Data- Landsat TM and SPOT Panchromatic.” Photogrammetric Engineering and Remote Sensing 57 (3): 295–303
  • Chavez, P. S., and A. Y. Kwarteng. 1989. “Extracting Spectral Contrast in Landsat Thematic Mapper Image Data Using Selective Principal Component Analysis.” Photogrammetric Engineering and Remote Sensing 55 (3): 339–348.
  • Chen, L., X. Yang, G. Jeon, M. Anisetti, and K. Liu. 2020. “A Trusted Medical Image Super-Resolution Method Based on Feedback Adaptive Weighted Dense Network.” Artificial Intelligence in Medicine 14 (4): 594–605.
  • Choi, J., Y. U. Kiyun, and K. I. M. Yongil. 2011. “A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement.” IEEE Transactions on Geoscience and Remote Sensing 49 (1): 295–309. doi:10.1109/TGRS.2010.2051674.
  • Dai, T., J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang. 2019. “Second-order Attention Network for Single Image Super-Resolution.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 11065–11074. Long Beach, CA, USA.
  • Denman, S., T. Lamb, C. Fookes, V. Chandran, and S. Sridharan. 2010. “Multi-spectral Fusion for Surveillance Systems.” Computers & Electrical Engineering 36 (4): 643–663. doi:10.1016/j.compeleceng.2008.11.011.
  • Do, M. N., and M. Vetterli. 2005. “The Contourlet Transform: An Efficient Directional Multiresolution Image Representation.” IEEE Transactions on Image Processing 14 (12): 2091–2106. doi:10.1109/TIP.2005.859376.
  • Dong, C., C. C. Loy, H. Kaiming, and X. Tang. 2014. “Learning a Deep Convolutional Network for Image Super-Resolution.” In Computer Vision – ECCV 2014, edited by D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, 184–199. Cham: Springer International Publishing.
  • Du, P., S. Liu, J. Xia, and Y. Zhao. 2013. “Information Fusion Techniques for Change Detection from Multi-temporal Remote Sensing Images.” Information Fusion 14 (1): 19–27. doi:10.1016/j.inffus.2012.05.003.
  • Duta, I. C., L. Liu, F. Zhu, and L. Shao. 2020. “Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition.” arXiv Preprint arXiv:2006.11538.
  • Fattal, R. 2007. Levoy, Marc. “Image Upsampling via Imposed Edge Statistics.” In ACM SIGGRAPH 2007 Papers, SIGGRAPH ’07, 95–es. New York, NY, USA: Association for Computing Machinery. doi:10.1145/1275808.1276496.
  • Fu, S., W. Meng, G. Jeon, A. Chehri, R. Zhang, and X. Yang. 2020. “Two-Path Network with Feedback Connections for Pan-Sharpening in Remote Sensing.” Remote Sensing 12 (10): 1674. doi:10.3390/rs12101674.
  • Fu, X., Z. Lin, Y. Huang, and X. Ding. 2019. “A Variational Pan-sharpening with Local Gradient Constraints.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10265–10274. Long Beach, CA, USA.
  • Garzelli, A., F. Nencini, and L. Capobianco. 2008. “Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images.” IEEE Transactions on Geoscience and Remote Sensing 46 (1): 228–236. doi:10.1109/TGRS.2007.907604.
  • Ghassemian, H. 2016. “A Review of Remote Sensing Image Fusion Methods.” Information Fusion 32: 75–89. doi:10.1016/j.inffus.2016.03.003.
  • Guo, Y., Y. Fei, and H. Gong. 2019. “Learning an Efficient Convolution Neural Network for Pansharpening.” Algorithms 12 (1): 16. doi:10.3390/a12010016.
  • Han, W., S. Chang, D. Liu, M. Yu, M. Witbrock, and T. S. Huang. 2018. “Image Super-resolution via Dual-state Recurrent Networks.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, USA.
  • He, L., Y. Rao, J. Li, J. Chanussot, A. Plaza, J. Zhu, and B. Li. 2019. “Pansharpening via Detail Injection Based Convolutional Neural Networks.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (4): 1188–1204. doi:10.1109/JSTARS.2019.2898574.
  • Hui, Z., J. Li, X. Gao, and X. Wang. 2021. “Progressive perception-oriented network for single image super-resolution.” Information Sciences 546: 769–786
  • Kim, J., J. K. Lee, and K. M. Lee. 2016a. “Accurate Image Super-Resolution Using Very Deep Convolutional Networks.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1646–1654. Las Vegas, Nevada, USA.
  • Kim, J., J. K. Lee, and K. M. Lee. 2016b. “Deeply-Recursive Convolutional Network for Image Super-Resolution.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1637–1645. Las Vegas, Nevada, USA.
  • Kingma, D. P., and B. Jimmy. 2014. “Adam: A Method for Stochastic Optimization.” arXiv preprint arXiv: 1412.6980
  • Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2017. “Imagenet Classification with Deep Convolutional Neural Networks.” Communications of the ACM 60 (6): 84–90. doi:10.1145/3065386.
  • Laben, C. A., and B. V. Brower. 2000. “Process for Enhancing the Spatial Resolution of Multispectral Imagery Using Pan-sharpening.” Jan. 4. US Patent 6,011,875.
  • Lai, W.-S., J.-B. Huang, N. Ahuja, and M.-H. Yang. 2017. “Deep Laplacian Pyramid Networks for Fast and Accurate Super-resolution.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 624–632. Honolulu, Hawaii, USA.
  • Lai, Z., L. Chen, G. Jeon, Z. Liu, R. Zhong, and X. Yang. 2021. “Real-time and Effective Pan-sharpening for Remote Sensing Using Multi-scale Fusion Network.” Journal of Real-Time Image Processing 1–17. https://doi.org/10.1007/s11554-021-01080-4
  • Lei, D., H. Chen, L. Zhang, and L. Weisheng. 2021. “NLRNet: An Efficient Nonlocal Attention ResNet for Pansharpening.” IEEE Transactions on Geoscience and Remote Sensing. DOI: 10.1109/TGRS.2021.3067097.
  • Lei, D., L. Shen, L. Zhang, and L. Weisheng. 2020. “Extraction-and-excitation Deep Neural Network for Pansharpening.” Concurrency and Computation: Practice and Experience e6098. https://doi.org/10.1002/cpe.6098.
  • Li, J., F. Fang, K. Mei, and G. Zhang. 2018. “Multi-scale Residual Network for Image Super-Resolution.” In Computer Vision – ECCV 2018, edited by V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, 527–542. Cham: Springer International Publishing.
  • Li, Z., J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wei 2019. “Feedback Network for Image Super-resolution.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3867–3876. Long Beach, CA, USA.
  • Lim, B., S. Son, H. Kim, S. Nah, and K. M. Lee. 2017. “Enhanced Deep Residual Networks for Single Image Super-Resolution.” In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July. Honolulu, Hawaii, USA.
  • Liu, C., Y. Zhang, S. Wang, M. Sun, O. Yangjun, Y. Wan, and X. Liu. 2020. “Band-Independent Encoder–Decoder Network for Pan-Sharpening of Remote Sensing Images.” IEEE Transactions on Geoscience and Remote Sensing 58 (7): 5208–5223. doi:10.1109/TGRS.2020.2975230.
  • Liu, D., B. Wen, Y. Fan, C. C. Loy, and T. S. Huang. 2018. “Non-Local Recurrent Network for Image Restoration.” In Advances in Neural Information Processing Systems, 1680–1689. Palais des Congrès de Montréal, Montréal CANADA.
  • Liu, X., Q. Liu, and Y. Wang. 2020. “Remote Sensing Image Fusion Based on Two-stream Fusion Network.” Information Fusion 55: 1–15. doi:10.1016/j.inffus.2019.07.010.
  • Liu, X., Y. Wang, and Q. Liu. 2018. “Psgan: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening.” In 2018 25th IEEE International Conference on Image Processing (ICIP), 873–877. Athens, Greece.
  • Ma, C., Y. Rao, Y. Cheng, C. Chen, L. Jiwen, and J. Zhou. 2020a “Structure-preserving Super Resolution with Gradient Guidance.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7769–7778.
  • Ma, J., Y. Wei, C. Chen, P. Liang, X. Guo, and J. Jiang. 2020b. “Pan-GAN: An Unsupervised Pan-sharpening Method for Remote Sensing Image Fusion.”.” Information Fusion 62: 110–120. doi:10.1016/j.inffus.2020.04.006.
  • Mallat, S. G. 1989. “A Theory for Multiresolution Signal Decomposition: The Wavelet Representation.” IEEE Transactions on Pattern Analysis and Machine Intelligence 11 (7): 674–693. doi:10.1109/34.192463.
  • Martin, D. B., T. Arnold, R. Leitner, G. McGunnigle, and R. Meester. 2010. Daniel J. Henry. “UAV-based Environmental Monitoring Using Multi-spectral Imaging.” In Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications VII. Vol. 7668, 766811. International Society for Optics and Photonics.
  • Masi, G., D. Cozzolino, L. Verdoliva, and G. Scarpa. 2016. “Pansharpening by Convolutional Neural Networks.” Remote Sensing 8 (7): 594. doi:10.3390/rs8070594.
  • Nason, G. P., and B. W. Silverman. 1995. Anestis Antoniadis and Georges Oppenheim. “The Stationary Wavelet Transform and Some Statistical Applications.” In Wavelets and Statistics, 281–299. Springer.
  • Nhamo, L., G. Y. Ebrahim, T. Mabhaudhi, S. Mpandeli, M. Magombeyi, M. Chitakira, J. Magidi, and M. Sibanda. 2020. “An Assessment of Groundwater Use in Irrigated Agriculture Using Multi-spectral Remote Sensing.” Physics and Chemistry of the Earth, Parts A/B/C 115: 102810. doi:10.1016/j.pce.2019.102810.
  • Ozcelik, F., U. Alganci, E. Sertel, and G. Unal. 2021. “Rethinking CNN-Based Pansharpening: Guided Colorization of Panchromatic Images via GANs.” IEEE Transactions on Geoscience and Remote Sensing 59 (4): 3486–3501. doi:10.1109/TGRS.2020.3010441.
  • Palsson, F., J. R. Sveinsson, and M. O. Ulfarsson. 2013. “A New Pansharpening Algorithm Based on Total Variation.” IEEE Geoscience and Remote Sensing Letters 11 (1): 318–322. doi:10.1109/LGRS.2013.2257669.
  • Palsson, F., M. O. Ulfarsson, and J. R. Sveinsson. 2019. “Model-Based Reduced-Rank Pansharpening.” IEEE Geoscience and Remote Sensing Letters 17 (4): 656–660. doi:10.1109/LGRS.2019.2926681.
  • Qu, Y., Q. Hairong, B. Ayhan, C. Kwan, and R. Kidd. 2017. “DOES Multispectral/hyperspectral Pansharpening Improve the Performance of Anomaly Detection?” In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 6130–6133. IEEE. Fort Worth, Texas, USA.
  • Shao, Z., L. Zexin, M. Ran, L. Fang, J. Zhou, and Y. Zhang. 2020. “Residual Encoder–Decoder Conditional Generative Adversarial Network for Pansharpening.” IEEE Geoscience and Remote Sensing Letters 17 (9): 1573–1577. doi:10.1109/LGRS.2019.2949745.
  • Shensa, M. J. 1992. “The Discrete Wavelet Transform: Wedding the a Trous and Mallat Algorithms.” IEEE Transactions on Signal Processing 40 (10): 2464–2482. doi:10.1109/78.157290
  • Shettigara, V. K. 1992. “A Generalized Component Substitution Technique for Spatial Enhancement of Multispectral Images Using A Higher Resolution Data Set.” Photogrammetric Engineering and Remote Sensing 58 (5): 561–567.
  • Starck, J.-L., E. J. Candès, and D. L. Donoho. 2002. “The Curvelet Transform for Image Denoising.” IEEE Transactions on Image Processing 11 (6): 670–684. doi:10.1109/TIP.2002.1014998.
  • Sun, J., X. Zongben, and H.-Y. Shum. 2010. “Gradient Profile Prior and Its Applications in Image Super-resolution and Enhancement.” IEEE Transactions on Image Processing 20 (6): 1529–1542. doi:10.1109/TIP.2010.2095871.
  • Tai, Y., J. Yang, and X. Liu. 2017. “Image Super-Resolution via Deep Recursive Residual Network.” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2790–2798. Honolulu, Hawaii, USA.
  • Tai, Y., J. Yang, X. Liu, and C. Xu. 2017. “MemNet: A Persistent Memory Network for Image Restoration.” In 2017 IEEE International Conference on Computer Vision (ICCV), 4549–4557. Venice, Italy.
  • Thomas, C., T. Ranchin, L. Wald, and J. Chanussot. 2008. “Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics.” IEEE Transactions on Geoscience and Remote Sensing 46 (5): 1301–1312. doi:10.1109/TGRS.2007.912448.
  • Tian, X., Y. Chen, C. Yang, X. Gao, and M. Jiayi. 2020. “A Variational Pansharpening Method Based on Gradient Sparse Representation.” IEEE Signal Processing Letters 27: 1180–1184. doi:10.1109/LSP.2020.3007325.
  • Tong, T., G. Li, X. Liu, and Q. Gao. 2017. “Image Super-Resolution Using Dense Skip Connections.” In 2017 IEEE International Conference on Computer Vision (ICCV), 4809–4817. Venice, Italy.
  • Vicinanza, M. R., R. Restaino, G. Vivone, M. D. Mura, and J. Chanussot. 2014. “A Pansharpening Method Based on the Sparse Representation of Injected Details.” IEEE Geoscience and Remote Sensing Letters 12 (1): 180–184. doi:10.1109/LGRS.2014.2331291.
  • Vivone, G., M. D. Mura, A. Garzelli, R. Restaino, G. Scarpa, M. O. Ulfarsson, L. Alparone, and J. Chanussot. 2021. “A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening with Classical and Emerging Pansharpening Methods.” IEEE Geoscience and Remote Sensing Magazine 9 (1): 53–81.
  • Vivone, G., L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, G. A. Licciardi, R. Restaino, and L. Wald. 2015. “A Critical Comparison Among Pansharpening Algorithms.” IEEE Transactions on Geoscience and Remote Sensing 53 (5): 2565–2586. doi:10.1109/TGRS.2014.2361734.
  • Vivone, G., M. Simões, M. D. Mura, R. Restaino, J. M. Bioucas-Dias, G. A. Licciardi, and J. Chanussot. 2014. “Pansharpening Based on Semiblind Deconvolution.” IEEE Transactions on Geoscience and Remote Sensing 53 (4): 1997–2010. doi:10.1109/TGRS.2014.2351754.
  • Wald, L. 2002. Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions. Presses de l’Ecole, Ecole des Mines de Paris, Paris, France. ISBN 2-911762-38-X. https://hal-mines-paristech.archives-ouvertes.fr/hal-00464703
  • Wald, L., T. Ranchin, and M. Mangolini. 1997. “Fusion of Satellite Images of Different Spatial Resolutions: Assessing the Quality of Resulting Images.” Photogrammetric Engineering and Remote Sensing 63 (6): 691–699. https://hal.archives-ouvertes.fr/hal-00365304.
  • Wang, Z., and A. C. Bovik. 2002. “A Universal Image Quality Index.” IEEE Signal Processing Letters 9 (3): 81–84. doi:10.1109/97.995823.
  • Wei, Y., Q. Yuan, H. Shen, and L. Zhang. 2017. “Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network.” IEEE Geoscience and Remote Sensing Letters 14 (10): 1795–1799. doi:10.1109/LGRS.2017.2736020.
  • Yang, J., X. FU, Y. Hu, Y. Huang, X. Ding, and J. Paisley. 2017a. “Deep Edge Guided Recurrent Residual Learning for Image Super-resolution.” In 2017 IEEE International Conference on Computer Vision (ICCV), 1753–1761. Venice, Italy.
  • Yang, W., J. Feng, J. Yang, F. Zhao, J. Liu, Z. Guo, and S. Yan. 2017b. “Deep Edge Guided Recurrent Residual Learning for Image Super-resolution.” IEEE Transactions on Image Processing 26 (12): 5895–5907. doi:10.1109/TIP.2017.2750403.
  • Yuan, Q., Y. Wei, X. Meng, H. Shen, and L. Zhang. 2018. “A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-sharpening.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11 (3): 978–989. doi:10.1109/JSTARS.2018.2794888.
  • Yuhas, R. H., F. H. G. Alexander, and J. W. Boardman. 1992. “Discrimination among Semi-arid Landscape Endmembers Using the Spectral Angle Mapper (SAM) Algorithm.” In Proceedings Summaries 3rd Annual JPL Airborne Geoscience Workshop, 147–149. Pasadena, California, USA.
  • Zhang, L., L. Weisheng, C. Zhang, and D. Lei. 2020b. “A Generative Adversarial Network with Structural Enhancement and Spectral Supplement for Pan-sharpening.” Neural Computing & Applications 32: 18347–18359.
  • Zhang, L., L. Weisheng, L. Shen, and D. Lei. 2020a. “Multilevel Dense Neural Network for Pan-sharpening.” International Journal of Remote Sensing 41 (18): 7217–7232. doi:10.1080/01431161.2020.1755474.
  • Zhang, Y., K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. 2018a. “Image Super-Resolution Using Very Deep Residual Channel Attention Networks.” In Proceedings of the European conference on computer vision (ECCV), 286–301. Munich, Germany.
  • Zhang, Y., Y. Tian, Y. Kong, B. Zhong, and Y. Fu. 2018b. “Residual Dense Network for Image Super-Resolution.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2472–2481. Salt Lake City, Utah, USA.
  • Zhang, Y., K. Li, K. Li, B. Zhong, and Y. Fu. 2019. “Residual Non-local Attention Networks for Image Restoration.” arXiv preprint arXiv.1903–10082
  • Zhang, Y., C. Liu, M. Sun, and Y. Ou. 2019a. “Pan-Sharpening Using an Efficient Bidirectional Pyramid Network.” IEEE Transactions on Geoscience and Remote Sensing 57 (8): 5549–5563. doi:10.1109/TGRS.2019.2900419.
  • Zhou, J., D. L. Civco, and J. A. Silander. 1998. “A Wavelet Transform Method to Merge Landsat TM and SPOT Panchromatic Data.” International Journal of Remote Sensing 19 (4): 743–757. doi:10.1080/014311698215973.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.