Irregular mask image inpainting based on progressive generative adversarial networks

ABSTRACT

Existing image inpainting algorithms have achieved good results in regular mask inpainting tasks, but there are still many limitations for irregular mask, and the inpainting results lack edge consistency and semantic correctness. To address these problems, we designed a progressive generative adversarial network in this paper. The generative network mainly contains a progressive feature generation module and an adaptive consistent attention module, and the discriminator network uses SN-PatchGAN. The network performs recursive inpainting from the edge to the centre, and the repaired features are used as the conditions for the next feature generation, which makes the constraints of the central content gradually strengthened. In order to obtain distant information from the feature maps as well as to consider the possible problem of inconsistency between feature mappings under different recursions when using attention directly in the progressive inpainting network, we designed an adaptive consistent attention module in this paper, which can adaptively combine scores obtained from different recursions to capture more feature information. The discriminator used in this paper is SN-PatchGAN, which directly calculates the hinge loss of each point on the output graph and can focus on different locations as well as different semantics. Comparing this paper's method on CelebA dataset and CMP Facade dataset with the latest work, the experimental results show that this paper's method performs well in the irregular mask image inpainting task, and the inpainting results have better performance in terms of edge consistency, semantic correctness, and overall image structure.

KEYWORDS:

• Progressive generative adversarial network
• adaptive consistent attention module
• SN-PatchGAN
• semantic consistency
• style loss
• perceptual loss

1. Introduction

Digital image processing is the processing of image information to meet the needs of human visual psychology and practical applications [1]. With the continuous development of science and technology, people have more requirements for image quality, image inpainting has become an important branch in the field of image processing. The purpose of image inpainting is to fill in the missing pixels in a corrupted image with a given mask [2], in order to achieve consistency in the overall texture structure as well as semantic visual realism. This task has received extensive attention and become a valuable and popular research topic for decades [3–5]. High-quality image inpainting can be useful in a wide range of applications, such as inpainting of old photographs, object removal, image inpainting, image processing, image denoising, etc.[6–11]. Currently, image inpainting methods can be divided into traditional image inpainting methods and deep learning-based image inpainting methods.

Traditional methods for image inpainting by mathematical and physical means are divided into structure-based and texture-based methods. Structure-based approaches use geometric methods to recover missing parts of an image. Bertalmio et al. [12] used the BSCB algorithm based on Partial Differential Equation (PDE) for image inpainting for the first time. Chan and Shen proposed the Total Variation (TV) model [13] and the Curvature Driven Diffusions (CDD) model [14], inspired by Bertalmio et al. The TV inpainting model is suitable for repairing small areas of breakage and can effectively remove noise, but it does not connect the repaired part to the background well. The CDD model extends the TV model by better connecting the repaired area to the background, but the resulting interpolated segments are visually blurrier. Texture-based approaches use texture synthesis to restore the area of the image to be repaired. Bertalmio et al. [15] first proposed adding texture information to a structure-based image inpainting algorithm. Criminisi et al. [16] proposed a block-based texture synthesis algorithm, which can restore the texture information of the image well, but cannot maintain the structural consistency of the restored area with the background. Starck et al. [17] proposed an iterative inpainting algorithm based on an analytical dictionary, which achieves inpainting by making a sparse prior on the local smoothness of the image, but the inpainting of the texture and edges of the image is relatively poor. Traditional image inpainting algorithms are robust for small regions of missing images with simple textures, but the inpainting effect gradually deteriorates as the region to be restored becomes larger, and the inpainting results suffer from blurred images and incoherent contextual semantics.

In recent years, with the rapid development of artificial intelligence [18], image inpainting combined with deep learning has gradually become the mainstream direction of research, in which deep Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN) [19] are widely used. Pathak et al. [20] proposed the Context Encoder (CE), which uses a fully connected layer so that the network can learn the relationship between all feature locations to deepen the semantic understanding of the whole image. However, this method sometimes misses fine texture details, resulting in visual artefacts around broken areas, and cannot handle high-pixel images. Yang et al. [21] improved on CE by using a texture network to modify the prediction of missing parts to improve the visual quality of the filled image. However, it is difficult to achieve the expected results for image inpainting with a complex background structure. Yu et al. [5] proposed a two-stage image inpainting network and designed a contextual attention layer, but the method is too dependent on the correlation between unknown and known feature blocks and lacks an understanding of the high-level semantics of the image. To address the problem that the model cannot obtain a larger receptive field, Wang et al. [22] proposed a Generative Multi-column Convolutional Neural Networks (GMCNN), where the multi-column structure decomposes the image into components with different perceptual fields and feature resolutions, thus helping the model to obtain a larger receptive field. However, large convolution kernels lead to a large number of model parameters, which makes the computational cost larger and requires a large amount of computational resources. Hui et al. [23] improved on GMCNN by proposing a multi-scale feature fusion network that combines various sparse multi-scale features to obtain dense multi-scale features, so that the features of the generated image at each scale match the ground truth. Multi-scale feature fusion networks can effectively obtain multi-scale information at the expense of spatial detail, making it difficult to strike a balance between spatial detail and contextual information. Thanh et al. [24] proposed an adaptive image inpainting method based on the Eulerian elastic model, which uses structural tensor eigenvalues to estimate the parameters of the inpainting model and implements them using high-order discrete gradients, and the quality of the image inpainting method performs well.

The above methods improve the defects of traditional image inpainting and have a large improvement in inpainting results, but these models are trained on regular mask images and are therefore not suitable for the task of irregular mask image inpainting. To address this problem, Liu et al. [25] proposed a partial convolution method conditional only on valid pixels to handle irregular mask inpainting, which can control the information passed within the network. However, partial convolution makes the invalid pixels disappear gradually and also makes the rule mask information disappear, so it cannot achieve satisfactory inpainting results for images with sparse structures. Nazeri et al. [26] proposed a two-stage adversarial model EdgeConnect, where first an edge generator generates an edge hypothesis map, and then the image inpainting network uses the edge hypothesis map as a prior to fill in the missing regions, which can effectively solve the problem of deep learning that fails to reconstruct reasonable structures due to excessive smoothing. However, the edge hypothesis map generated by this network has not yet met the expectations, and the problem of generating reasonable edges for missing regions, for example, still needs to be addressed in the future. Yu et al. [27] proposed gated convolution, which learns a dynamic selection mechanism by adding an additional standard convolutional layer and a sigmoid activation function that learns a dynamic selection mechanism across all layers for each channel at each spatial location, significantly improving the colour consistency and picture quality of the Inpainting, but there is still room for improvement in terms of image realism, edge consistency, and global semantic structure. Abdulla et al. [28] proposed an example-based algorithm that consists of two phases; in the first phase, the entire image is searched and the most similar features are found and selected using Euclidean distance. In the second stage, the distance between the selected feature block location and the feature block location to be filled is measured. Ahmed et al. [29] proposed an improved search mechanism to select the most similar patches, and the algorithm adds the results of the first two stages and finally selects the patch that obtains the smallest distance value for filling. This method can obtain the best similar blocks to the maximum extent, which results in a higher quality of inpainting results. Mao et al. [30] proposed a self-attentive Wasserstein-based Generative Adversarial Network (WGAN) for single-image inpainting. This network has the ability to mine the global relevance of images by introducing WGAN to ensure the global consistency of the inpainting content and using self-attention to obtain the similarity of local features of images, thus performing well in image inpainting tasks.

For better inpainting of broken images with continuous masks, so that the inpainting results have edge consistency and semantic correctness, we design a Progressive Generative Adversarial Network (PGAN) for irregular masked image inpainting. The progressive feature generation approach is used to strengthen the constraints on the central region. Adaptive Consistent Attention Module (ACAM) is designed to obtain distant features and adaptively combine the scores to solve the inconsistent feature mapping problem between different recursions. To focus on irregular masks with diverse locations and shapes, SN-PatchGAN is used to calculate the hinge loss of each pixel point.

In summary, the main contributions of this paper are as follows:

(1)	The design of a progressive generative adversarial network (PGAN) that exploits the correlation between neighbouring pixels and progressive feature generation to strengthen the constraints of the central region.
(2)	The design of an adaptive consistent attention module (ACAM) which adaptively combines attention scores from different recursions to ensure consistency in feature mapping under different recursions to obtain better results.
(3)	The discriminator network uses SN-PatchGAN for irregular mask image repair, which can produce high-quality repair results simply and quickly.

In this paper, progressive generative adversarial network for irregular masked image inpainting is designed to strengthen the constraints of the image inpainting process. The remaining sections of this paper are organized as follows: in Section 2, the related work of this paper is presented. Section 3 presents the proposed method in this paper. Section 4 carries out the experimental details. Section 5 presents our conclusions.

2. Related work

2.1. Generative adversarial network

Generative adversarial networks are a deep learning model that is one of the most promising approaches to unsupervised learning on complex distributions in recent years. A GAN consists of a generative network and a discriminative network. The generative network takes as input a random sample from a noisy distribution, and its output needs to mimic as much as possible the real samples in a given training set. The discriminant network's input is the output of the generative network or the real sample, and its purpose is to distinguish the output of the generative network from the real sample as much as possible, while the generative network has to deceive the discriminant network as much as possible. The two networks confront each other and continuously optimize and adjust the parameters, with the ultimate goal of making the discriminant network unable to judge whether the output of the generative network is real or not. The structure of GAN is shown in Figure 1.

Figure 1. The basic structure of GAN.

The texts in this figure are: “Training set”, “Random noise”, “Generator”, “Fake image”, “Discriminator”, “Real” and “Fake”.

The training process of GAN is shown in Equation (1(1) $\begin{array}{rl}& \underset{\mathrm{D}}{\mathrm{m}\mathrm{a}\mathrm{x}}\underset{\mathrm{G}}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathrm{G}\mathrm{A}\mathrm{N}\left(\mathrm{D},\mathrm{G}\right)={\mathrm{E}}_{\mathrm{X}\sim {\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\left(\mathrm{x}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{D}\left(\mathrm{x}\right)\right]\\ & +{\mathrm{E}}_{\mathrm{z}\sim {\mathrm{P}}_{\mathrm{z}}\left(\mathrm{z}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\right)\right)\right)\right]\end{array}$(1) ). (1) $\begin{array}{rl}& \underset{\mathrm{D}}{\mathrm{m}\mathrm{a}\mathrm{x}}\underset{\mathrm{G}}{\mathrm{m}\mathrm{i}\mathrm{n}}\mathrm{G}\mathrm{A}\mathrm{N}\left(\mathrm{D},\mathrm{G}\right)={\mathrm{E}}_{\mathrm{X}\sim {\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\left(\mathrm{x}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{D}\left(\mathrm{x}\right)\right]\\ & +{\mathrm{E}}_{\mathrm{z}\sim {\mathrm{P}}_{\mathrm{z}}\left(\mathrm{z}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\right)\right)\right)\right]\end{array}$(1) Where E is the expectation, ${\mathrm{p}}_{data}\left(x\right)$ is the real sample, G is the generator network, and D is the discriminator network. The generator captures the distribution of the sample data and uses the noise z that follows a specific distribution to generate a sample similar to the real training data, and the discriminator is a binary classifier that estimates the probability that a sample is a ground truth.

Compared with other models, GAN can produce clearer, more realistic samples and is applied to scenarios such as super-resolution reconstruction, image editing, data generation [31–33], where complex loss functions can be designed and the two networks will spontaneously confront each other. However, because GAN is derived from the game theory, there is the problem that the gradients of the generator and discriminator cancel each other, which will lead to the difficulty of convergence of the network, and in addition, GAN is less stable and more difficult to train. So the attention module can make the GAN network discard the current irrelevant information and pay more attention to the target information, which can effectively reduce the complexity of training and improve the effect of repair.

2.2. Channel attention module

The Channel Attention Module (CAM) is a feature map compressed in spatial dimension to obtain a 1D vector. Since each channel of the feature map is considered as a feature detector, the CAM focuses on which content has a significant role. To efficiently compute the CAM, the spatial dimensionality method of compressing the input feature map is used, and CAM is shown in Figure 2.

Figure 2. Diagram of channel attention module.

CAM inputs the features into two parallel max-pooling layer and average pooling layer, changes the feature map size from C*H*W to C*1*1, and then goes through a shared fully connected layer, first compresses the number of channels to 1/r times the original number, and then expands to the original number of channels, and gets the two activated results after the ReLU activation function. For the output results, the CAM feature map is obtained after element-by-element summation by the sigmoid activation function.

The channel attention is calculated as shown in Equation (2(2) $\begin{array}{rll}{M}_c\left(F\right)& =\sigma \left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)& \\ & =\left(W_1\left(W_0\left(F_{avg}^c\right)\right)+W_1\left(W_0\left(F_{max}^c\right)\right)\right)\end{array}$(2) ). (2) $\begin{array}{rll}{M}_c\left(F\right)& =\sigma \left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)& \\ & =\left(W_1\left(W_0\left(F_{avg}^c\right)\right)+W_1\left(W_0\left(F_{max}^c\right)\right)\right)\end{array}$(2) Where, σ is the sigmoid function, $W_0\in R^{C/r\times C}$, $W_1\in R^{C\times C/r}$, $W_0$ and $W_1$ share the weights of the MLP.

2.3. Spatial attention module

The Spatial Attention Module (SAM) is a compression of channels, averaging pooling and max pooling in channel dimensions, respectively, and generating spatial attention maps using spatial relationships between features. Different from CAM, SAM focuses on which parts have important roles, and SAM is shown in Figure 3.

Figure 3. Diagram of spatial attention module.

SAM takes the output result of CAM as input, gets two 1*H*W feature maps by max-pooling and average pooling, performs stitching operation on the two feature maps, and then transforms them into 1-channel feature maps by 7*7 convolution, and finally gets the feature maps of SAM by a sigmoid activation function.

The spatial attention is calculated as shown in Equation (3(3) $\begin{array}{rll}{M}_s\left(F\right)& =\sigma \left(f^{7\times 7}\left(\left[AvgPool\left(F\right);MaxPool\left(F\right)\right]\right)\right)& \\ & =\sigma \left(f^{7\times 7}\left(\left[F_{avg}^s;F_{max}^s\right]\right)\right)\end{array}$(3) ). (3) $\begin{array}{rll}{M}_s\left(F\right)& =\sigma \left(f^{7\times 7}\left(\left[AvgPool\left(F\right);MaxPool\left(F\right)\right]\right)\right)& \\ & =\sigma \left(f^{7\times 7}\left(\left[F_{avg}^s;F_{max}^s\right]\right)\right)\end{array}$(3) Where, σ is the sigmoid function and $f^{7\times 7}$ is the convolution kernel of size $f^{7\times 7}$.

2.4. Perceptual loss

Perceptual loss was first proposed in 2016 by Johnson et al. [34] in the area of image style conversion for comparing two images that look similar but are different, such as the same photo but shifted by one pixel, or the same image across different resolutions. In these cases, the pixel-level loss function outputs a large error value despite the fact that the images are very similar, while the perceptual loss function compares high-level perceptual and semantic differences between the images. The perceptual loss is calculated as shown in Equation (4(4) $l_{feat}^{\phi ,j}(\hat{y},y)=\frac{1}{C_j{H}_j{W}_j}\Vert {\varphi }_j\left(\hat{y}\right)-{\varphi }_j\left(y\right){\Vert }_2^2$(4) ). (4) $l_{feat}^{\phi ,j}(\hat{y},y)=\frac{1}{C_j{H}_j{W}_j}\Vert {\varphi }_j\left(\hat{y}\right)-{\varphi }_j\left(y\right){\Vert }_2^2$(4) Where, y is the ground truth, $\hat{y}$ is the predicted image, $C_j$, $H_j$, $W_j$ denote the number of channels, length and width of the feature map, and ${\varphi }_j\left(y\right)$ is the activation of the VGG network in processing the image y at the layer j. The activation of the ground truth y and the predicted image $\hat{y}$ is compared by the shape normalization of the image using squared L2 loss.

2.5. Style loss

Initially used in the field of style transfer, style loss is now also widely used in tasks such as image inpainting and super-resolution reconstruction. The idea is to use the Gram matrix to perform the computation of feature map differences. A pre-trained model outputs different features in different layers of the feature map, the shallow features are more detailed and the deeper features are more abstract. And the Gram matrix is actually the calculation of the correlation between two vectors. After the Gram matrix calculation of different feature map vectors, the overall image style can better grasp. The style matrix formed by the ground truth is shown in Equation (5(5) $G_{k{k}^{\prime }}^{lS}=\sum _{i=1}^{n_H^l}\sum _{j=1}^{n_W^l}{\alpha }_{i,j,k}^{lS}{\alpha }_{i,j,k^{\prime }}^{lS}$(5) ). (5) $G_{k{k}^{\prime }}^{lS}=\sum _{i=1}^{n_H^l}\sum _{j=1}^{n_W^l}{\alpha }_{i,j,k}^{lS}{\alpha }_{i,j,k^{\prime }}^{lS}$(5) Where, S is the ground truth, i is the height, j is the width, k is the different channels, and ${\alpha }_{i,j,k}^l$ denotes the positions of i, j, and k in layer l. $G_{k{k}^{\prime }}^l$ is used to measure the correlation coefficient between the activation terms in the k and $k^{\prime }$ channels.

The style matrix formed by the generated images is shown in Equation (6(6) $G_{k{k}^{\prime }}^{lG}=\sum _{i=1}^{n_H^l}\sum _{j=1}^{n_W^l}{\alpha }_{i,j,k}^{lG}{\alpha }_{i,j,k^{\prime }}^{lG}$(6) ). (6) $G_{k{k}^{\prime }}^{lG}=\sum _{i=1}^{n_H^l}\sum _{j=1}^{n_W^l}{\alpha }_{i,j,k}^{lG}{\alpha }_{i,j,k^{\prime }}^{lG}$(6) Where, G is the generated image.

Two matrices were obtained from S and G, respectively, and the final error between these two matrices is shown in Equation (7(7) ${\mathrm{J}}_{style}^l\left(S,G\right)=\frac{1}{{\left(2n_H^l{n}_W^l{n}_C^l\right)}^2}\sum _k\sum _{k^{\prime }}\left(G_{k{k}^{\prime }}^{lS}-G_{k{k}^{\prime }}^{lG}\right)$(7) ). (7) ${\mathrm{J}}_{style}^l\left(S,G\right)=\frac{1}{{\left(2n_H^l{n}_W^l{n}_C^l\right)}^2}\sum _k\sum _{k^{\prime }}\left(G_{k{k}^{\prime }}^{lS}-G_{k{k}^{\prime }}^{lG}\right)$(7) Where, ${\mathrm{J}}_{style}^l\left(S,G\right)$ is the style loss function defined for layer l.

3. Proposed method

At present, image inpainting using deep neural networks can produce good results, but there are still problems of poor edge consistency and semantic consistency for the inpainting of irregularly masked images. Therefore, in this paper, we designed a progressive generative adversarial network, in which the network shares parameters during the inpainting process, gradually strengthens the constraints on the internally generated content, and performs this progressive process in the feature mapping space, which solves the limitation that the network input and output need to be represented in the same space. Meanwhile, adaptive consistent attention is designed to be used in the generative network, which effectively solves the problem of semantic inconsistency of the restored images.

3.1. Network structure

For the continuous mask with a large area, this paper designed a progressive generative adversarial network, and the network structure is shown in Figure 4. The input of the network is a mask image, and the mask image is encoded first and then restored using the progressive restore module. The restored feature map is passed through the decoder to obtain the restore result, and then the restored image and the ground truth are input to the discriminator for determination.

Figure 4. The framework of our method.

In the progressive generative network, we input the mask image and first restore it from the boundary region of the mask, using the progressive feature values to fill it, and then this part that has been filled as the known content, based on which the next smaller mask boundary region is restored until the mask region restore is completed. Traditional discriminator is mainly designed for single rectangular masks that fill regions with fixed locations. In this paper, we consider the task of repairing irregularly mask images, in which there may be multiple holes with arbitrary shapes at arbitrary locations, so we use a simple and effective GAN loss SN-PatchGAN for irregular image inpainting, whose network structure is shown in Figure 5.

Figure 5. SN-PatchGAN.

SN-PatchGAN consists entirely of convolutional layers and the input consists of generated image, ground truth and mask. The earliest discriminator was a direct output of 0/1 values, later PatchGAN used 2D tensor output, and in this paper we use 3D output with output size of $R^{\mathrm{h}\times \mathrm{w}\times \mathrm{c}}$, h, w and c are the height, width and number of channels, respectively. The network uses 6 convolutions with a kernel size of 5 and a step size of 2. The six convolutions are superimposed to capture the feature statistics of Markov blocks. Then, we apply the GAN directly to each feature element in this feature map, formulating a GAN of $h\times w\times c$ number, focusing on different positions and different semantics of the input image. By using SN-PatchGAN, our network has better performance in the irregular mask image inpainting task.

3.2. Adaptive consistent attention module

The attention mechanism can further enhance the ability of the progressive inpainting module to restore high-quality feature maps, and to enhance the semantic consistency of the images as well as the possibility of capturing distant features, the Convolutional Block Attention Module (CBAM) [35] is used in the feature generation phase, which consists of a channel attention module and a spatial attention module in tandem to extract feature information from both channel and spatial dimensions, but the direct use of the CBAM in progressive network does not work well because it does not consider the problem of consistency between feature mappings in different recursive cases, which may lead to inconsistent semantics of the generated images. To solve this problem, we designed an adaptive consistent attention module in this paper, which is different from previous attention mechanisms that calculate attention scores independently. This attention solves the problem of inconsistent feature mapping brought by considering only the current layer of attention scores by adaptively combining different recursive attention scores, and the repair model in this paper effectively improves the model consistency level with the assistance of adaptive consistent attention module. The attention module in this paper is shown in Figure 6.

Figure 6. Adaptive consistent attention module.

The adaptive consistent attention module in this paper first inputs the feature map and goes through CAM and SAM in turn to calculate the attention score. First, the feature map is input to the CAM for 2D convolution, and the convolution result is multiplied with the input feature map, and the CAM is calculated as $F_c^n=M_c\left(F\right)\otimes F$, where n is the number of current cycles, F is the input features, $F\in R^{C\times H\times W}$, and CAM 2D convolution. Then the output result of the channel attention is used as input for the 2D convolution of the SAM, and after the convolution, the output result is multiplied with the feature map obtained from the CAM to obtain the original attention score of the current cycle, which is calculated as $F_s^n=M_s\left(F_{\mathrm{c}}^n\right)\otimes F_c^n$. The final attention score is calculated from the original attention score of the current cycle.

To calculate the final attention score for the current cycle, it is necessary to determine whether each pixel on the feature map is a valid pixel, if a pixel is a known region or has been filled, it is a valid pixel, and if a pixel is a masked region and has not been filled, it is an invalid pixel. The final attention score of a pixel is calculated. If a pixel on the feature map was valid in the last cycle, then the final attention score of that pixel in the current cycle is equal to the weighted sum of the original attention score of the current cycle and the final attention score of the previous cycle, as shown in Equation (8(8) $Att\mathrm{\_}scor{e}_{i,j}^n=\lambda Att\mathrm{\_}scor{e}_{i,j}^{\prime }{}^n+\left(1-\lambda \right)Att\mathrm{\_}scor{e}_{i,j}^{n-1}$(8) ). (8) $Att\mathrm{\_}scor{e}_{i,j}^n=\lambda Att\mathrm{\_}scor{e}_{i,j}^{\prime }{}^n+\left(1-\lambda \right)Att\mathrm{\_}scor{e}_{i,j}^{n-1}$(8) where n is the number of current cycles, (i,j) is the coordinates of the pixel on the feature map, λ is the weight, $Att\mathrm{\_}scor{e}^{\prime }$ is the original attention score, and $Att\mathrm{\_}score$ is the final attention score.

If a pixel on the feature map was invalid in the last cycle, then the final attention score for that pixel in the current cycle is equal to the original attention score, as shown in Equation (9(9) $Att\mathrm{\_}scor{e}_{i,j}^n=Att\mathrm{\_}scor{e}_{i,j}^{\prime }{}^n$(9) ). (9) $Att\mathrm{\_}scor{e}_{i,j}^n=Att\mathrm{\_}scor{e}_{i,j}^{\prime }{}^n$(9) After calculating the final score, the reconstructed feature map is stitched with the original feature map, and then another pixel-by-pixel convolution is done to obtain the final updated feature map for this cycle.

3.3. Progressive generation network

The pixel correlation within the local area of the image to be repaired is strong, and the missing region pixels can be inferred more accurately from the surrounding pixel information. However, for continuous irregular masks, the pixel correlation is weakened due to its large missing range and the increased distance between known and unknown regions, which leads to insufficient constraints on the centre location of the missing regions. In this paper, we design progressive generative network to solve this problem, which is divided into three main modules: the region identification module (represented by the blue dashed frame in the figure), the feature generation module (represented by the yellow dashed frame in the figure) and the feature merging module (represented by the green dashed frame in the figure), and the network structure is shown in Figure 7. The region identification module is used to identify the mask boundary and treat the inner part of the boundary as the area to be patched. The feature generation module fills the content based on the results identified by the region identification module, which consists of a U-Net network and uses skip connection. The feature merging module is used to merge the features generated by the intermediate processes. The region identification and feature generation processes are alternated, and the final feature map is generated after the restore by feature merging.

Figure 7. Progressive generation network.

Partial convolution is the basic module for region recognition, the region identification module determines the region to be restored by identifying the mask boundaries. The module needs to cascade multiple partial convolution layers together to update the mask and feature mapping, and the partial convolution is used to identify the region to be updated in each recursion, we can obtain masks with a smaller range of updates after each partial convolution layer. The feature mapping is processed by the normalization layer and activation function after passing through the partial convolution layers and then passed to the feature generation module. The difference between the update mask and the input mask is then defined as the region to be generated in this recursion. The updated mask is retained throughout the recursion until the next recursion when it is reduced.

In the feature generation module, after the image to be restored has been identified in the region identification section as the region to be restored in the current round of recursion, the feature values within the region to be processed are estimated by the feature generation module, which aims at progressive feature values to fill the identified region. Progressive features not only produce better inpainting results but also facilitate subsequent inference. The feature generation module in this paper has a simple structure, consisting of an encoding layer and a decoding layer, which are combined together by a skip connection. After generating the feature values, the feature map is passed to the next recursion. In this paper, the region identification module and the feature generation module are alternated in the network, and the generation module is based on the previous generation of the feature map, which reduces the computation and saves a lot of computational cost. The attention module can be used in image inpainting to synthesize better-quality features, so this paper also designed an adaptive consistent attention module in this part of feature generation, which can adaptively combine different recursion scores to solve the problem of inconsistent feature mapping between different recursions, thus improving the semantic consistency of inpainting results.

When the mask image is completely filled, we use the feature merging module to merge the feature map. If we directly use the result of the last feature map as the output, the gradient will disappear and destroy the signal generated by the previous iteration. To solve this problem, an intermediate feature map merging is required. When the mask image inpainting is completed or the number of cycles has reached the upper limit, the average value of multiple feature maps output by multiple cycles is calculated to obtain the merged final feature map.

3.4. Loss functions

The loss function is an optimization target for the reconstruction accuracy of each pixel as well as the composition. The L1 loss function is used to calculate the absolute sum of the errors between the true and predicted values, using the average value to represent the overall loss value. First, we define the pixel losses in this paper, which are the L1 loss of the mask region and the valid region pixels on the network output, respectively. We input the masked image $I_{in}$, the initial binary mask M, the network output image $I_{out}$ and the real image $I_{gt}$, as shown in Equation (10(10) $\begin{array}{rll}& L_{hole}=\frac{1}{N_{I_{gt}}}{\Vert \left(1-M\right)\odot \left(I_{out}-I_{gt}\right)\Vert }_1& \\ & L_{valid}=\frac{1}{N_{I_{gt}}}{\Vert M\odot \left(I_{out}-I_{gt}\right)\Vert }_1\end{array}$(10) ). (10) $\begin{array}{rll}& L_{hole}=\frac{1}{N_{I_{gt}}}{\Vert \left(1-M\right)\odot \left(I_{out}-I_{gt}\right)\Vert }_1& \\ & L_{valid}=\frac{1}{N_{I_{gt}}}{\Vert M\odot \left(I_{out}-I_{gt}\right)\Vert }_1\end{array}$(10) Where, $N_{I_{gt}}$ is the number of elements in $I_{gt}$, $N_{I_{gt}}=C\ast H\ast W$, C, H and W are the number of channels, height and width, respectively.

For image generation learning, we use pre-trained and fixed VGG-16 with perceptual loss and style loss. The perceptual loss and style loss compare the difference between the depth feature map of the generated image and the ground truth. This loss function can effectively convey the structural and textural information of the image to the model, and the perceptual loss is calculated as shown in Equation (11(11) $L_{perceptual}=\sum _{i=1}^N\frac{1}{H_i{W}_i{C}_i}{\left|{\varphi }_{poo{l}_i}^{gt}-{\varphi }_{poo{l}_i}^{pred}\right|}_1$(11) ). (11) $L_{perceptual}=\sum _{i=1}^N\frac{1}{H_i{W}_i{C}_i}{\left|{\varphi }_{poo{l}_i}^{gt}-{\varphi }_{poo{l}_i}^{pred}\right|}_1$(11) Where, $H_i$, $W_i$ and $C_i$ are the height, weight and channel size of the $i_{th}$ feature map, respectively, and ${\varphi }_{poo{l}_i}$ is the feature map from the $i_{th}$ pooling layer in the fixed VGG-16.

The style loss uses the Gram matrix to perform the computation of feature map differences. Our pre-trained model, which outputs different layers of feature maps expressing different features, with the shallow layer used to compare details and the deep layer used to compare abstract content. The Gram matrix calculates the correlation of two vectors. By subjecting different feature map vectors to the Gram matrix calculation, we can better grasp the overall image style and make the restored image style more consistent. The style loss is calculated as shown in Equation (13(13) $L_{style}=\sum _{i=1}^N\frac{1}{C_i\times C_i}{\left|\frac{1}{H_i{W}_i{C}_i}\left({\varphi }_{poo{l}_i}^{styl{e}_{gt}}-{\varphi }_{poo{l}_i}^{styl{e}_{pred}}\right)\right|}_1$(13) ). (12) ${\varphi }_{poo{l}_i}^{style}={\varphi }_{poo{l}_i}{\varphi }_{poo{l}_i}^T$(12) (13) $L_{style}=\sum _{i=1}^N\frac{1}{C_i\times C_i}{\left|\frac{1}{H_i{W}_i{C}_i}\left({\varphi }_{poo{l}_i}^{styl{e}_{gt}}-{\varphi }_{poo{l}_i}^{styl{e}_{pred}}\right)\right|}_1$(13) The last item is Total Variation (TV) loss, whose role is mainly noise reduction. The difference in neighbouring pixel values in the image can be solved to some extent by reducing TV Loss, thus maintaining the smoothness of the image, as shown in Equation (14(14) $L_{tv}=\sum _{\left(i,j\right)\in R,\left(i,j+1\right)\in R}\frac{{\Vert I_{comp}^{i,j+1}-I_{comp}^{i,j}\Vert }_1}{N_{I_{comp}}}+\sum _{\left(i,j\right)\in R,\left(i+1,j\right)\in R}\frac{{\Vert I_{comp}^{i+1,j}-I_{comp}^{i,j}\Vert }_1}{N_{I_{comp}}}$(14) ). (14) $L_{tv}=\sum _{\left(i,j\right)\in R,\left(i,j+1\right)\in R}\frac{{\Vert I_{comp}^{i,j+1}-I_{comp}^{i,j}\Vert }_1}{N_{I_{comp}}}+\sum _{\left(i,j\right)\in R,\left(i+1,j\right)\in R}\frac{{\Vert I_{comp}^{i+1,j}-I_{comp}^{i,j}\Vert }_1}{N_{I_{comp}}}$(14) where $I_{comp}$ is the most original output image, $N_{I_{comp}}$ is the number of elements in $I_{comp}$, R is the mask region 1 pixel expansion region, and TV loss is a smoothing penalty on R, with the main purpose of maintaining the smoothness of the image and eliminating artefacts that may be brought by image inpainting.

In summary, the total loss function of our generative network is shown in Equation (15(15) $L_{\mathrm{G}}=\delta L_{hole}+\gamma L_{valid}+\chi L_{perceptual}+ϵL_{style}+\lambda L_{tv}$(15) ). (15) $L_{\mathrm{G}}=\delta L_{hole}+\gamma L_{valid}+\chi L_{perceptual}+ϵL_{style}+\lambda L_{tv}$(15) Where, δ, γ, χ, ε and λ are the parameters given to adjust the weights of each loss in the overall loss.

The discriminator uses spectral-normalization, which makes the GAN training more stable and converges faster. The final loss function of the discriminator is the hinge loss and SN-PatchGAN loss. The SN-PatchGAN loss function is calculated as shown in Equation (16(16) $L_{D^{sn}}=E_{x\sim p_{data\left(x\right)}}\left[\mathrm{Re}LU\left(1-D^{sn}\left(x\right)\right)\right]+E_{z\sim p_{z\left(z\right)}}\left[\mathrm{Re}LU\left(1+D^{sn}\left(x\right)\right)\right]$(16) ). (16) $L_{D^{sn}}=E_{x\sim p_{data\left(x\right)}}\left[\mathrm{Re}LU\left(1-D^{sn}\left(x\right)\right)\right]+E_{z\sim p_{z\left(z\right)}}\left[\mathrm{Re}LU\left(1+D^{sn}\left(x\right)\right)\right]$(16) Where, E is the expectation, ${\mathrm{p}}_{data}\left(x\right)$ is the true sample, and z is the noise.

4. Experiments

4.1. Experimental settings

All experimental data in this paper are done in the same experimental environment. The hardware devices used for the experiments are 64-bit Windows 10 operating system, Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz processor, and NIVIDIA GeForce RTX 2080Ti graphics card. For software, the framework chosen for this experiment is PyTorch v1.0.0, configured with cuda v10.0, Python v3.7.3, and the compiler used is PyCharm.

The number of training sessions is 160,000, and the pre-training network is trained using Adam optimizer with parameters set to ${\text{β}}_1=0.9$ and ${\text{β}}_2=0.999$. The model is trained using the learning rate of $1e^{-4}$, Batch_Size is 6, and then the model is fine-tuned using $1e^{-5}$. The loss function weights are set to $\delta =6$, $\gamma =1$, $\chi =0.05$, $ϵ=120$ and $\lambda =0.1$.

4.2. Datasets

In the experiments of this paper, two public image datasets and a mask dataset are used for the image inpainting task. And the training of network models and the comparative measurement of image inpainting effects are performed on this dataset.

CelebA Dataset [36]: the image dataset focusing on faces contains 202599 images of 10177 celebrity identities, and all are feature-labelled, this dataset is very good for face-related training tasks.

CMP Facade Dataset [37]: provides a dataset of frontal images collected at the Machine Perception Center, which includes 606 frontal images of buildings from different cities around the world and with different styles, which have been manually annotated.

NVIDIA Irregular Mask Dataset [25]: contains 12,000 masks and a total of six different hole-to-image area ratios, with each category containing 1000 masks with boundary constraints and 1000 masks without boundary restrictions.

4.3. Qualitative comparisons

In the same experimental setting, our method was compared with several classical, state-of-the-art methods, which include Liu et al. [25], Yu et al. [27], Hui et al. [23], Zhao et al. [10], and Li et al. [38]. On the CelebA Dataset [36] and CMP Facade Dataset [37] for the comparative experiments in this paper, and measured qualitative and quantitative results to compare our model with previous methods, and the results are shown in Figures 8 and 9.

Figure 8. Comparative experimental results on the CelebA Dataset.

Figure 9. Comparative experimental results on the CMP Facade Dataset.

Figures 8 and 9 show the results of the method in this paper compared with the five state-of-the-art methods on two datasets. In most cases, our model restore results are significantly better than the other state-of-the-art methods, especially in the case of large area breakage, where our algorithm generates semantically more reasonable results. All methods produce seemingly reasonable inpainting results on both datasets; however, the inpainting results of Liu et al. method produce visual artefacts and edge response in the transition region between the mask and the surrounding area, colour differences between the restored and known areas can be observed, and the inpainting results on larger mask images suffer from semantic discontinuities. Yu et al.'s method yielded more visually pleasing results without significant colour inconsistencies, but the restored images using the gated convolution method had flaws in edge processing and poor boundary consistency. Hui et al.'s method used multiscale dense fusion blocks and combines multiple losses, and the inpainting results perform well overall, but there are texture structure deficiencies. Zhao et al. used co-modulated generative adversarial networks, that is, given the conditions and introduced noise, to achieve good results in the large mask repair task, but the authenticity of the repair results needs further improvement. Li et al.'s method can produce realistic texture details, but the method is less robust in repairing large irregular masked images. Thanks to the progressive generative adversarial network and the well-designed adaptive consistent attention module, the method in this paper provides significant visual improvements in texture and structure, and our network can produce satisfactory image structure with significant improvements in global semantic continuity and boundary consistency of the image, and the image is free from artefact generation, which can be seen in the figure with smoother connections between the muscle textures and lines of the face, the inpainting of the building has more texture details.

4.4. Quantitative comparisons

We also quantitatively compared the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) of the model in this paper, where the SSIM index evaluates the generated image and the real image in terms of image structure. Quantitative evaluation of restoration results means objective evaluation of restoration results. Subjective evaluation information is influenced by personal opinions, so quantitative indexes are used to evaluate image restoration effects more objectively. Through quantitative analysis of the experimental data, it can be found that our method has better inpainting effect compared with other methods and our inpainting method has stable inpainting capability. Tables 1 and 2 show the results of the two datasets with different methods, and our method produces excellent results and the highest AVE SSIM and AVE PSNR on the CelebA and CMP Facade datasets.

Table 1. PSNR (dB)/SSIM (%) evaluation results of different methods on the CelebA Dataset.

Methods	MAX PSNR/SSIM	AVE PSNR/SSIM	MIN PSNR/SSIM
Liu et al.	32.79/0.911	28.89/0.782	26.92/0.613
Yu et al.	34.76/0.920	30.48/0.827	28.19/0.726
Hui et al.	34.07/0.939	31.56/0.879	28.30/0.774
Zhao et al.	33.68/0.960	31.93/0.907	29.29/0.701
Li et al.	36.68/0.939	31.39/0.894	27.42/0.586
Ours	34.93/0.981	32.57/0.934	30.16/0.813

Note: Bold font is the best value for each column.

Table 2. PSNR (dB)/SSIM (%) evaluation results of different methods on the CMP Facade Dataset.

Methods	MAX PSNR/SSIM	AVE PSNR/SSIM	MIN PSNR/SSIM
Liu et al.	31.49/0.911	28.57/0.762	26.74/0.627
Yu et al.	32.01/0.921	29.68/0.819	27.29/0.710
Hui et al.	32.68/0.982	30.48/0.895	28.47/0.783
Zhao et al.	33.29/0.939	31.15/0.890	29.16/0.729
Li et al.	35.47/0.963	30.67/0.873	26.71/0.659
Ours	33.82/0.993	31.71/0.952	29.95/0.862

Note: Bold font is the best value for each column.

Table 1 shows the analysis results of PSNR and SSIM metrics of our method and other advanced methods on the CelebA dataset, the numbers in bold in the table indicate the optimal values. The data shows that our method has the highest scores in AVE PSNR and AVE SSIM compared with other methods, our method improved by 3.69dB and 0.152, respectively, compared with Liu et al.'s method, compared with Yu et al.'s method improved by 2.09 dB and 0.107, compared with Hui et al.'s method improved by 1.01 dB and 0.055, compared with Zhao et al.'s method improved by 0.64 dB and 0.027, and compared with Li et al.'s method improved by 1.18 dB and 0.040, respectively.

Table 2 shows the analysis results of PSNR and SSIM indicators of this method and other advanced methods on the CMP Facade dataset, the numbers in bold in the table indicate the optimal values. The data shows that our method has the highest scores in AVE PSNR and AVE SSIM compared with other methods. Compared with the method of Liu et al., our method has improved 3.14dB and 0.190, respectively, compared with the method of Yu et al., 2.03dB and 0.133, respectively, and compared with the method of Hui et al., 1.23dB and 0.057, respectively, compared with Zhao et al.'s method, it is improved by 0.56dB and 0.062, respectively, and compared with Li et al.'s method, it is improved by 1.04dB and 0.079, respectively.

As can be seen from Tables 1 and 2, our model achieves better results compared with the other five advanced methods. The highest AVE PSNR and AVE SSIM are achieved in both datasets, which is a good indication that our method can obtain better results and a more stable model. The PGAN designed in this paper gradually repairs the masked regions from edge to centre and from easy to difficult, thus generating semantically true and edge-consistent results. PGAN contains ACAM and SN-PatchGAN, which can better ignore unimportant features and extract effective features, thus enhancing the model restore ability and thus generating semantically consistent and visually results with higher degree of realism. We can conclude from the experimental data that the image inpainting model in this paper has stronger and more stable inpainting ability and better inpainting results.

4.5. Ablation studies

In order to verify the effectiveness of each module in the proposed method, we did the ablation study in this paper under the same experimental environment, and the results are shown in Figure 10. The first column is the ground truth, the second column is the input image, and each other column corresponds to a module of this paper.

Figure 10. Results of ablation studies on CelebA and CMP Facade datasets.

From the ablation study results in Figure 10, it can be seen that the overall inpainting effect using only the progressive generative network method is poor, and the inpainting results will show semantic discontinuities and insufficient clarity with pixel-level defects. The use of progressive generative network and ACAM can effectively solve the problems of semantic discontinuity and low clarity because of the inclusion of attention mechanism, but the model convergence speed is slower and the generated results will show certain boundary artefacts. Using the full model without style loss, the inpainting results in poor overall style consistency compared to the real image. Finally, using full model(progressive generative network + ACAM + SN-PatchGAN+ style loss) gives the best results, generates more consistent and highest definition restored images, and performs best in terms of semantic content, boundary consistency, style consistency and texture.

Table 3 shows the analysis results of the PSNR and SSIM metrics of the ablation study of the method in this paper on the CelebA dataset, the numbers in bold in the table indicate the optimal values. The analysis of the experimental data shows that the full model has the best results and gets the highest values on both AVE PSNR and AVE SSIM results, compared with the dual progressive generative network by 2.20dB and 0.118, respectively, compared with progressive generative network and ACAM by 0.84dB and 0.077, respectively, and compared with progressive generative network and SN-PatchGAN by 0.41dB and 0.022, respectively. Compared with the full model without style loss is improved by 0.47dB and 0.024, respectively.

Table 3. PSNR(dB)/SSIM(%) evaluation results of the ablation study on the CelebA Dataset.

Models	MAX PSNR/SSIM	AVE PSNR/SSIM	MIN PSNR/SSIM
w/o ACAM and SN-PatchGAN	33.20/0.893	30.37/0.816	28.29/0.642
w/o SN-PatchGAN	33.79/0.910	31.73/0.857	28.84/0.690
w/o ACAM	34.31/0.942	32.16/0.912	29.63/0.749
w/o style loss	34.46/0.957	32.25/0.920	29.67/0.739
Full Model	34.93/0.981	32.57/0.934	30.16/0.813

Note: Bold font is the best value for each column.

Table 4 shows the analysis results of PSNR and SSIM metrics for the ablation study of the method in this paper on the Paris StreetView dataset, the numbers in bold in the table indicate the optimal values. And the analysis of the experimental data shows that the full model has the best results and gets the highest values on both AVE PSNR and AVE SSIM results, compared with the dual progressive generative network by 1.65dB and 0.125, respectively, compared with progressive generative network and ACAM by 1.15dB and 0.073, respectively, and compared with progressive generative network and SN-PatchGAN by 0.59dB and 0.045, respectively. Compared with the full model without style loss is improved by 0.22dB and 0.021, respectively.

Table 4. PSNR(dB)/SSIM(%) evaluation results of the ablation study on the CMP Facade Dataset.

Models	MAX PSNR/SSIM	AVE PSNR/SSIM	MIN PSNR/SSIM
w/o ACAM and SN-PatchGAN	34.76/0.920	30.06/0.827	28.19/0.726
w/o SN-PatchGAN	34.07/0.939	30.56/0.879	28.30/0.774
w/o ACAM	33.68/0.960	31.12/0.907	29.29/0.701
w/o style loss	33.60/0.972	31.30/0.926	29.37/0.793
Full Model	33.82/0.993	31.71/0.952	29.95/0.862

Note: Bold font is the best value for each column.

From Tables 3 and 4, it can be seen that this paper enhances the consistency between feature mappings by using the ACAM, the consistency between feature mappings is enhanced, the structural applicability of the model is improved, the model's ability to restore semantic information is increased, and the texture detail and edge consistency problems arising from the model restore results are mitigated. In addition, in this paper, by using PGAN and SN-PatchGAN, the model generates images with higher clarity, enhances the quality of the restored images, and improves the learning ability and convergence speed of the model. Based on the experimental data, we can conclude that the inpainting results of this paper using full model can achieve better results compared to using only some of the modules.

5. Conclusion

Image inpainting is an important research branch in the field of digital image processing, which aims to automatically recover the lost information based on the existing information of an image. In this paper, we designed an irregular mask image inpainting algorithm based on a progressive generative adversarial network, which uses correlation of neighbouring pixels and progressive feature generation to gradually enrich the information in the missing regions and give semantically clear and well-structured inpainting results. In addition, an adaptive consistent attention module is designed, which helps the generation process of the feature generation module in the progressive generative adversarial network, adaptively combines the attention scores of different recursions, ensures the feature mapping between different recursions, and makes the inpainting results more consistent. The discriminator network used SN-PatchGAN, which focuses on different locations and different semantics of the input images, allowing our network to have better performance in irregular mask image restoration tasks. In the experimental part, extensive quantitative (PSNR and SSIM) and qualitative comparison and ablation studies are conducted in this paper to demonstrate the advantages of our designed network in terms of performance and effectiveness.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The project was supported in part by the Natural Science Basis Research Plan in Shaanxi Province of China [grant number 2023-JC-YB-517] and the High-level Talent Introduction Project of Shaanxi Technical College of Finance and Economics [grant number 2022KY01].

Notes on contributors

Hong-an Li

Hong-an Li received the M.S. degree in 2009 and the Ph.D. degree in 2014 in computer science and technology from Northwest University, Shaanxi, China. From 2014 to 2023, he was an associate professor of the College of Computer Science and Technology, Xi’an University of Science and Technology. His research interests include computer graphics and computer-aided geometric design, virtual reality and image processing.

Liuqing Hu

Liuqing Hu received the bachelor’s degree in software engineering in 2021 from the College of Computer Science and Technology, Xi’an University of Science and Technology. She is currently studying for a master’s degree in the College of Computer Science and Technology, Xi’an University of Science and Technology. Her research interests include artificial intelligence, image processing and computer vision applications.

Jing Zhang

Jing Zhan received the M.S. degree in computer application technology from Northwest University of Shaanxi Province, China in 2013 and the Ph.D. degree in 2018. From 2018 to 2023, she served as a lecturer in the School of Computer Science and Technology of Xi'an University of Science and Technology. Her research interests include graphics and image processing, intelligent information processing and signal processing and perception.