Analyzing the performances of squash functions in capsnets on complex images

Figures & data

Figure 1. A typical architecture of CapsNet encoder with an image from MNIST dataset.

Figure 2. A typical architecture of CapsNet decoder with an image from MNIST dataset.

Table

Algorithm 1. Routing Algorithm

1: $\mathit{begin}$ 2: $\mathit{for}\mathit{all}\mathit{capsule}\mathit{i}\mathit{in}\mathit{layer}\mathit{land}\mathit{capsule}\mathit{j}\mathit{in}\mathit{layer}$ $\left(\mathit{l}+1\right):b_{\mathit{ij}}\leftarrow 0$ 3: $\mathit{for}\mathit{r}\mathit{iterations}\mathit{do}$ 4: $\mathit{for}\mathit{all}\mathit{capusule}\mathit{i}\mathit{in}\mathit{layer}\mathit{l}:c_i\leftarrow \mathit{softmax}\left(b_i\right)$ 5: $\mathit{for}\mathit{all}\mathit{capusule}\mathit{j}\mathit{in}\mathit{layer}\left(\mathit{l}+1\right):s_j\leftarrow {\sum }_i{c}_{\mathit{ij}}{\hat{u}}_{\mathit{j}/\mathit{i}}$ 6: $\mathit{for}\mathit{all}\mathit{capusule}\mathit{j}\mathit{in}\mathit{layer}\left(\mathit{l}+1\right):v_j\leftarrow \mathit{squash}\left(s_j\right)$ 7: $\mathit{for}\mathit{all}\mathit{capusule}\mathit{i}\mathit{in}\mathit{layer}\mathit{land}\mathit{capsule}\mathit{j}\mathit{in}\mathit{layer}$ 8: $\mathit{layer}\left(\mathit{l}+1\right):b_{\mathit{ij}}\leftarrow b_{\mathit{ij}}+{\hat{u}}_{\mathit{j}/\mathit{i}}{v}_j$

Table 1. Some variants squash functions

Author	Classification type	Squash function
Sabour et al (Sara et al., 2017) Afriyie et al (Yaw et al., 2022b) Edgar et al (Edgar et al., 2017)	Image Image Image	${\mathit{v}}_{\mathit{j}}=\frac{\parallel s_j{\parallel }^2}{1+\parallel s_j{\parallel }^2}\frac{{\mathit{s}}_{\mathit{j}}}{\parallel s_j\parallel }\mathit{\hspace{0.17em}}$ ${\tilde{\mathit{v}}}_{\mathit{j}}=\frac{\parallel s_j{\parallel }^2}{1+\parallel s_j{\parallel }^2}\frac{{\mathit{s}}_{\mathit{j}}}{2\parallel s_j{\parallel }^2}$ $\mathit{f}\left(\mathit{x}\right)=\left(1-\frac{1}{{\mathit{e}}^{\parallel \mathit{x}\parallel }}\right)\frac{x}{\parallel \mathit{x}\parallel }$

Table 2. Properties of datasets

Dataset	# of Classes	Dimension	Train	Test	Total
FMNIST	10	28 x 28	60,000	10,000	70,000
CIFAR-10	10	32 x 32	50,000	10,000	60,000

Table 3. Performance metrics on FMNIST

Squash Function	Accuracy	Test Loss	# of Epochs
Edgar et al (Edgar et al., 2017)	92.78%	7.22%	200
Sabour et al (Sara et al., 2017) Afriyie et al (Yaw et al., 2022b)	92.49% 92.80%	7.51% 7.20%	200 200

Figure 3. Comparison between different squash functions.

Figure 4. Plot of accuracy graphs for FMNIST: (a) Afriyie et al (Yaw et al., 2022b) model(b) Sabour et al (Sara et al., 2017) model(c) Edgar et al (Edgar et al., 2017) model.

Figure 5. Plot of Loss graphs for FMNIST: (a) Afriyie et al (Yaw et al., 2022b) model(b) Sabour et al (Sara et al., 2017) model(c) Edgar et al (Edgar et al., 2017) model.

Figure 6. Plot of accuracy graphs for CIFAR10: (a) Afriyie et al (Yaw et al., 2022b) model(b) Sabour et al (Sara et al., 2017) model(c) Edgar et al (Edgar et al., 2017) model.

Table 4. Performance metrics on CIFAR-10

Squash Function	Accuracy	Test loss	# of Epochs
Edgar et al (Edgar et al., 2017)	85.63%	14.37%	100
Sabour et al (Sara et al., 2017) Afriyie et al (Yaw et al., 2022b)	84.57% 86.79%	15.43% 13.21%	100 100

Figure 7. Multi-class Receiver Operating Characteristic (ROC) curves for FMNIST: (a) Afriyie et al (Yaw et al., 2022b) model(b) Sabour et al (Sara et al., 2017) model(c) Edgar et al (Edgar et al., 2017) model.

Figure 8. Precision-Recall curves for FMNIST: (a) Afriyie et al (Yaw et al., 2022b) model(b) Sabour et al (Sara et al., 2017) model(c) Edgar et al (Edgar et al., 2017) model.

Figure 9. Multi-class Receiver Operating Characteristic (ROC) curves and Precision-Recall curves for CIFAR10. The (a), (b) and (c) represents the ROC curves for (a) Afriyie et al (Yaw et al., 2022b) model (b) Sabour et al (Sara et al., 2017) model. (c) Edgar et al (Edgar et al., 2017) model and the (d), (e) and (f) consists of the Precision-Recall curves of the respective models.

Figure 10. Visualization of the clusters formed at the: (a) FMNIST- caps-amp layer of the optimized model (b) FMNIST- caps-amp layer of the Sabour’s model (c) FMNIST- caps-amp layer of the Edgar’s model (d) CIFAR10- caps-amp layer of the optimized model (e) CIFAR10- caps-amp layer of the Sabour’s model (f) CIFAR10- caps-amp layer of the Edgar’s model.

Sara, S., Frosst, N., & Hinton, G. E. (2017). Dynamic routing between capsules. Advances in Neural Information Processing Systems, 30( 2017-Decem), 3857–3867.

 Google Scholar

Yaw, A., Weyori, B. A., & Opoku, A. A. (2022b). Classification of blood cells using optimized Capsule networks. Neural Processing Letters. https://doi.org/10.2139/ssrn.4073627/

 Web of Science ®Google Scholar

Edgar, X., Bing, S., & Jin, Y. (2017). Capsule Network Performance on Complex Data. arXiv preprint arXiv:171203480, 10707(Fall), 1–7. http://arxiv.org/abs/1712.03480

 Google Scholar