Improving Face Recognition by Integrating Decision Forest into GAN

Figures & data

Figure 1. A brief overview of the TP-GAN model architecture. modifications have been made to the existing TP-GAN model to produce this architecture. The generator network that combines a two-pathway layout (local and global pathways) and a discriminator with a single deep neural structure, followed by a light-CNN model, determines the accuracy of identity-preserving properties.

Figure 2. Overview of our approach to modifying the TP-GAN $D$ network. where $d$and $\mathrm{\ell }$ represent the decision node and leaf node, respectively. each decision node/leaf is implemented by a deep neural network (DNN) structure. our forest consist of $16$ trees with $9$ depth levels (i.e. $T_1,\dots ,T_{16})$. arrows represent paths used to route information of the sample $x$along a tree to reach leaf ${\mathrm{\ell }}_2$, which has probability ${\mu }_{{\mathrm{\ell }}_2}=d_0\left(x\right){\bar{d}}_1\left(x\right){\bar{d}}_4\left(x\right)$.

Figure 3. Comparison of our method’s generated facial images with those generated by TP-GAN on the multi-PIE database. despite significant abnormalities in the faces image, our synthetic faces seem convincing. The dataset was downloaded from the TP-GAN GitHub repository at: https://github.com/HRLTY/TP-GAN.

Figure 4. Comparison of our method's generated facial images with those generated by TP-GAN on the FEI database. Our method consistently produced better texture detail for all face poses. We downloaded the dataset from the FEI official repository at: https://fei.edu.br/~cet/facedatabase.html.

Figure 5. Comparison of our method's generated facial images with those generated by TP-GAN on the CAS-PEAL database. Despite illumination variations such as grey faces, our method consistently produced better texture detail. We downloaded the dataset from CAS-PEAL official repository at: https://github.com/YuYin1/DA-GAN.

Figure 6. A comparison of our frontal-profile synthesis results with those from various methods in the Multi-PIE dataset, using 30° and 45° face poses. We downloaded the dataset from the TP-GAN GitHub repository at: https://github.com/HRLTY/TP-GAN.

Figure 7. A comparison of our frontal-profile synthesis results with those generated on Multi-PIE dataset, using various face poses. We downloaded the dataset from the TP-GAN GitHub repository at: https://github.com/HRLTY/TP-GAN.

Figure 8. A comparison of our frontal-profile synthesis results with those generated on Multi-PIE dataset, using various face poses. We downloaded the dataset from the TP-GAN GitHub repository at: https://github.com/HRLTY/TP-GAN.

Figure 9. Our frontal-profile synthesis results have been compared with those obtained from various methods using 15°, 30° and 45° face poses from the CAS-PEAL dataset. We downloaded the dataset from CAS-PEAL official repository at: https://github.com/YuYin1/DA-GAN.

Figure 10. Comparing our frontal-profile synthesis results with those from other methods in the FEI dataset, using 30, 75 and 90 degree face poses. The dataset was downloaded from FEI official repository at https://fei.edu.br/~cet/facedatabase.html.

Table 1. Comparing our approach recognition rate $\left(\mathrm{\%}\right)$ against various methods on multi-PIE dataset. Despite the large poses, rank-1 recognition was achieved in almost all of the face poses.

Methods Face pose	Our method	(Mahmood, Yea-Shuan, and Yi-An 2022)	(Xiaoguang et al. 2021)	(Yanfei and Junhua 2020)	(Rui et al. 2017)	(Xi et al. 2017)
$\pm {90}^{\mathrm{o}}$	74.30%	65.23%	70.20%	67.04%	64.03%	61.02%
$\pm {75}^{\mathrm{o}}$	90.78%	85.30%	85.31%	87.01%	84.10%	77.02%
$\pm {60}^{\mathrm{o}}$	95.96%	94.13%	91.81%	93.09%	92.93%	85.02%
$\pm {45}^{\mathrm{o}}$	99.54%	98.80%	98.05%	98.04%	98.58%	89.07%
$\pm {30}^{\mathrm{o}}$	99.88%	99.88%	99.82%	99.07%	99.85%	92.05%
$\pm {15}^{\mathrm{o}}$	99.87%	99.80%	99.83%	99.09%	99.78%	94.06%
Average ACC	93.38%	90.52%	90.83%	90.55%	89.87%	83.04%

Table 2. Comparing our approach recognition rate $\left(\mathrm{\%}\right)$ against various methods on CAS-PEAL dataset. Despite the large poses, rank-1 recognition was achieved in almost all of the face poses.

Methods	Our method	(Rui et al. 2017)	(Yu et al. 2020)	(Tian et al. 2018)
Face pose	Pitch at ($-{30}^0$)
${15}^{\mathrm{o}}$	99.86%	98.94%	83.91%	99.72%
${30}^{\mathrm{o}}$	99.55%	98.89%	83.17%	99.56%
${45}^{\mathrm{o}}$	98.89%	97.63%	80.83%	98.99%
Average ACC	99.43%	98.48%	82.63%	99.42%
Pitch at ($+{45}^0$)
${15}^{\mathrm{o}}$	98.92%	97.73%	89.44%	98.98%
${30}^{\mathrm{o}}$	99.00%	97.45%	87.95%	98.86%
${45}^{\mathrm{o}}$	98.56%	95.83%	83.90%	98.13%
Average ACC	98.82%	97.00%	87.09%	98.65%

Table 3. A comparison of deep learning frameworks that focus on systemic details. Each deep learning model was tested on the same computer, but with different environment settings.

Evaluated item	TP-GAN (Rui et al. 2017) (TensorFlow 0.12v, Keras 2.0.1v)	Our method (TensorFlow 1.12v, Keras 2.2.5v)
Overall running time “number of epoch”	30 epochs = 2.20 Hrs. 1 = 279.96 (sec per-epoch)	30 epochs = 2.00 Hrs. 1 = 240.00 (sec per-epoch)
GPU load	$12-14GB$	$8-12GB$
Memory utilization	$74-86\mathrm{\%}$ (step-per-epoch)	$67-77\mathrm{\%}$ (step-per-epoch)

Figure 11. An overview of the training procedure steps for decision trees and forests.

Table 4. The amount of data augmentation that TP-GAN model synthesizes for each dataset. During the entire process, three datasets are used: training and testing.

Dataset types	Face poses	A total of all available face poses iamges before data augmentation	A total of all available face poses image after data augmentation
Multi-PIE	$\pm {60}^0$ to $\pm {90}^0$	2010	6707
CAS-PEAL	$\pm {60}^0$ to $\pm {90}^0$	10000	32430
FEI	$\pm {60}^0$ to $\pm {90}^0$	1800	5280

Figure 12. Plots of the TP-GAN and our method loss curves on the multi-PIE, FEI and CAS-PEAL datasets. Pixel loss is shown on A, while generator loss is shown on B. An axis in the horizontal direction indicates the number of epochs, or the number of times that all of the training data has been trained. The vertical axis indicates the accuracy of the model after each epoch; the smaller the loss, the better it performs.

Table

Summary of Algorithm

Require: $\mathcal{T}$: training set, $nEpochs$ $\left(x,y\right)\in \mathcal{T}$ 1. random initialization of $\theta$ 2. for all $i\in \left\{1,...,nEpochs\right\}$ do Compute $\pi$ by iterating ${\pi }_{{\mathrm{\ell }}_y}^{\left(t+1\right)}=\frac{1}{P_{\mathrm{\ell }}^{\left(t\right)}}{\sum }_{\left(x,y\right)\in \beta }\frac{{ }_{{\pi }_{{\mathrm{\ell }}_y}^{\left(t\right)}{\mu }_{\mathrm{\ell }}\left(x|\theta \right)}}{Q(y|x,\theta ,{\pi }^{\left(t\right)})}$ 3. break $\mathcal{T}$ into a set of random mini-batches 4. for all $\beta$: mini-batch from $\mathcal{T}$ do 5. Update $\theta$ by SGD step in ${\theta }^{\left(t+1\right)}={\theta }^{\left(t\right)}-\frac{\eta }{\left|\beta \right|}{\sum }_{\left(x,y\right)\in \beta }\frac{\mathrm{\partial }L}{\mathrm{\partial }\theta }\left({\theta }^{\left(t\right)},\pi ;x,y\right)$ 6. end for 7. end for

Figure 13. Examples of illumination levels on the multi-PIE dataset. For instance, illumination is a combination of brightness, exposure, contrast, and shadows. Various effects of quality can also be observed, including sharpness, smoothness, and blurriness. Overall, those qualities can contribute to a low level of face recognition. We downloaded the dataset from the TP-GAN GitHub repository at https://github.com/HRLTY/TP-GAN.

Mahmood, A., H. Yea-Shuan, and C. Yi-An. 2022. 2D facial landmark localization method for multi-view face synthesis image using a two-pathway generative adversarial network approach. Journal of PeerJ Computer Science 1–28. doi:10.7717/peerj-cs.897.

 Google Scholar

Xiaoguang, T., Z. Jian, L. Qiankun, A. Wenjie, G. Guodong, L. Zhifeng, L. Wei, and F. Jiashi. 2021. Joint face image restoration and frontalization for recognition. IEEE Transactions on Circuits and Systems for Video Technology 32 (3):1–14. doi:10.1109/TCSVT.2021.3078517.

 Web of Science ®Google Scholar

Yanfei, L., and C. Junhua. 2020. Unsupervised face frontalization for pose-invariant face recognition. Image and Vision Computing 106:1–9. doi:10.1016/j.imavis.2020.104093.

 Web of Science ®Google Scholar

Rui, H., Z. Shu, L. Tianyu, H. Ran 2017. Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In: IEEE International Conference on Computer Vision (ICCV). Venice, Italy: 2439–48. doi: 10.1109/ICCV.2017.267.

 Google Scholar

Xi, Y., Y. Xiang, S. Kihyuk, L. Xiaoming, C. Manmohan 2017. Towards large-pose face frontalization in the wild. IEEE International Conference on Computer Vision (ICCV). Venice, Italy: 1–10. doi: 10.1109/ICCV.2017.430.

 Google Scholar

Yu, Y., J. Songyao, R. Joseph, F. Yun 2020. Dual-attention GAN for large-pose face frontalization. IEEE International Conference on Automatic Face and Gesture Recognition. Buenos Aires, Argentina: 1–8. doi: 10.1109/FG47880.2020.00004.

 Google Scholar

Tian, Y., X. Peng, L. Zhao, S. Zhang, D. -N. Metaxas 2018. CR-GAN: Learning complete representations for multi-view generation. IJCAI International Joint Conference on Artificial Intelligence: 1–7. doi: 10.48550/arXiv.1806.11191.

 Google Scholar