Image inpainting via Smooth Tucker decomposition and Low-rank Hankel constraint

Abstract

Image inpainting, aiming at exactly recovering missing pixels from partially observed entries, is typically an ill-posed problem. As a powerful constraint, low-rank priors have been widely applied in image inpainting to transform such problems into well-posed ones. However, the low-rank assumption of original visual data is only in an approximate mode, which in turn results in suboptimal recovery of fine-grained details, particularly when the missing rate is extremely high. Moreover, a single prior cannot faithfully capture the complex texture structure of an image. In this paper, we propose a joint usage of Smooth Tucker decomposition and Low-rank Hankel constraint (STLH) for image inpainting, which enables simultaneous capturing of the global low-rankness and local piecewise smoothness. Specifically, based on the Hankelization operation, the original image is mapped to a high-order structure for capturing more spatial and spectral information. By employing Tucker decomposition for optimizing the Hankel tensor and simultaneously applying Discrete Total Variation (DTV) to the Tucker factors, sharper edges are generated and better isotropic properties are enhanced. Moreover, to approximate the essential rank of the Tucker decomposition and avoid facing the uncertainty problem of the upper-rank limit, a reverse strategy is adopted to approximate the true rank of the Tucker decomposition. Finally, the overall image inpainting model is optimized by the well-known alternate least squares (ALS) algorithm. Extensive experiments show that the proposed method achieves state-of-the-art performance both quantitatively and qualitatively. Particularly, in the extreme case with 99% pixels missed, the results from STLH are averagely ahead of others at least 0.9dB in terms of PSNR values.

Keywords:

• Tucker decomposition
• low-rank
• discrete total variation
• image inpainting

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by National Natural Science Foundation of China [62276232].

Notes on contributors

Jing Cai

Jing Cai was born in Hangzhou, China, in 1985. He received the Ph.D. degree in control science and engineering from Zhejiang University of Technology. Currently, he is an associate professor at the Department of Forensic Science, Zhejiang Police College. His research interests are in the areas of machine learning and computer vision.

Jiawei Jiang

Jiawei Jiang received the B.E. degree from the China Jiliang University, Hangzhou, China, in 2018. He is currently working toward the Ph.D. degree with the College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou. His current research interests include remote sensing image process and its application.

Yibing Wang

Yibin Wang was born in Wenzhou, China, in 2001. Currently, he is an undergraduate in College of Computer Science and Technology from Zhejiang University of Technology. His research interests are in the areas of machine learning and computer vision.

Jianwei Zheng

Jianwei Zheng received the B.E. degree in electronic and computer engineering and the Ph.D. degree in control theory and control engineering from the Zhejiang University of Technology, Hangzhou, China, in 2005 and 2010, respectively. He is an associate professor with the College of Computer Science and Technology, Zhejiang University of Technology. His research interests include machine learning and data analysis. He has published more than 70 academic papers in reputable journals and conferences, including IEEE Trans. Image Processing, IEEE Trans. Neural Networks and Learning Systems, IEEE Trans. Geoscience and Remote Sensing, IEEE Trans. Industrial Informatics, Pattern Recognition, and so on.

Honghui Xu

Honghui Xu received the B.E. degree from Zhejiang University of Technology, Hangzhou, China, in 2018. He is currently pursuing the Ph.D. degree in the School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His current research interests include image processing and optimization algorithm.