Deep Learning: Parameter Optimization Using Proposed Novel Hybrid Bees Bayesian Convolutional Neural Network

Figures & data

Table 1. Deep learning networks.

Deep Learning Network	Advantages	Limitations	Applications	References
Convolutional Neural Network (CNN)	Eliminating the needs for extracting the features manually, it is achieved by alternating and stacking pooling and convolutional layers that convolve with the input data to extract the most significant features Can be used with image data, text data, time series data and sequence input data Can be built from scratch or retrained enabling to build on the previous existing networks Has many pretrained architectures like AlexNet, GoogLeNet, ZFNet, RESNet and VGGNet Reducing the number of parameters	High computational complexity in model training	Image, pattern, and speech recognition Image and sentence classification Prediction problems Regression problems	(Singh et al. 2018) (Wang et al. 2018) (MathWorks-1) (Tch 2017)
Deep Recurrent Network (DRN)	Making directed cycle so that the outputs do not only depend on the immediate previous input Can be used with text data, speech data, time Series data, tabular data and generative models Allowing the information to persist in hidden layer and capturing the previous states and temporal correlation	Cannot be used with image data Difficult to memorize the long-term interaction in training	Speech and connected handwriting recognition Classification problems Regression prediction problems	(Wang et al. 2018) (Tch 2017) (Davydova 2017)
Sequence to Sequence Model (S2SM)	It is a model with two recurrent networks, including encoder to process the input and decoder to produce the output	It is limited to sequence problems	System for answering question and machine translation	(Tch 2017) (Davydova 2017)
Long Short-Term Memory (LSTM)	Modeling temporal sequences along with their long-term dependencies more accurate than traditional recursive neural network	There is no activation function and no modification for the stored values	Speech recognition	(Tch 2017) (Davydova 2017)
Gated Recurrent Network (GRN)	Modeling temporal sequences along with their long-term dependencies more accurate than traditional recursive neural network It is similar to long short-term memory but without output gate	There is no activation function and no modification for the stored values	Speech recognition	(Tch 2017)
Deep Belief Network (DBN)	Inferring the states of the unobserved variables Adjusting the interaction between variables to make the network generate observed data Can be trained without supervision Training using very fast algorithm to initialize the network and parameters	It is limited to binary inputs The parameters are fine-tuned using slow algorithm	Image recognition Face recognition Video sequence recognition Motion-capture data	(Wang et al. 2018) (Chandrayan 2017)
Restricted Boltzmann Machine (RBM)	Can quickly get unbiased sample from posterior distribution Can be trained without supervision Automatic training for the datasets avoiding the local minimum values Robust to ambiguous inputs	It is limited to binary inputs There is no output layer Restricted connection between hidden units	Filtering Prediction	(Wang et al. 2018) (Chandrayan 2017)
Recursive Network (RN)	It has the ability to make structured predictions	Used only when applying same weights recursively	Speech recognition	(Davydova 2017)
Auto Encoder Network (AEN)	When it is trained, the output is as close to input as possible Can be trained without supervision Eliminating the irrelevance in the input and preserve the meaningful information	Used only when output cells are equal to input cells and hidden cells is less than input cells Error propagation	Classification Clustering Feature compression	(Wang et al. 2018) (Tch 2017)
Generative Adversarial Network (GAN)	Very large family of double networks Leaning a map from an input to output image	Generator and discriminator fool each other	Image generation	(Tch 2017)

Figure 1. Inputs and hidden layers connection in CNN (Le 2015).

Figure 2. The way of working for CNN (MathWorks-1).

Figure 3. ANOVA Results.

Table 2. Hybrid CNN accuracy.

Hybrid Algorithm	Accuracy		Dataset	Reference
	Original CNN	Hybrid CNN
(EA-CNN)	-	98.88%	MNIST database for handwritten digits recognition	(Badan 2019)
	-	62.37%	CIFAR10 dataset for animals’ image classification
(GA-CNN)	71.69%	75.95%	Stock market fluctuation prediction	(Chung & Shin, 2019)
(LSTM-CNN)	82.1%	97%	Motion data and site video to recognize workers unsafe actions	(Ding et al. 2018)
(FL-CNN)	97.35%	99.10%	Handwritten digits recognition	(Popko and Weinstein 2016)

Table 3. CNN parameters information.

Variable	Benefit	Range
Section Depth (SD)	Control the depth of the network	1–3
Initial Learning Rate (ILR)	Allow for learning the features	1e^−2 – 1
Momentum (M)	Update the hyperparameters	0.8–0.98
Regularization (R)	Prevent overfitting	1e^−10 – 1e^−2

Table 4. Factors and levels definition.

Factors	Level 1	Level 2	Level 3
Section Depth	1	2	3
Initial Learning Rate	0.01	0.055	0.1
Momentum	0.88	0.9	0.92
Regularization	1e-10	2.5e-4	0.0005

Table 5. Taguchi orthogonal array (L9).

Exp #	SD	ILR	M	R	VA (%)
1	1	0.01	0.88	1e-10	67.96
2	1	0.055	0.9	2.5e-4	72.20
3	1	0.1	0.92	0.0005	70.90
4	2	0.01	0.9	0.0005	76.40
5	2	0.055	0.92	1e-10	78.44
6	2	0.1	0.88	2.5e-4	77.40
7	3	0.01	0.92	2.5e-4	77.74
8	3	0.055	0.88	0.0005	79.56
9	3	0.1	0.9	1e-10	80.34

Table 6. Classification accuracy and computational time for algorithms (cifar10DataDir dataset).

	Existing Original CNN	Existing BO-CNN	Novel Hybrid BA-BO-CNN
Training Accuracy	92.68%	90.27%	90.53%
Validation Accuracy	80.34%	80.72%	82.22%
Testing Accuracy	80.54%	80.69%	80.74%
Computational Time	42 Min 56 Sec	43 Min 24 Sec	41 Min 13 Sec

Table 7. Classification accuracy and computational time for algorithms (digits dataset).

	Existing Original CNN	Existing BO-CNN	Novel Hybrid BA-BO-CNN
Training Accuracy	99.94%	100%	100%
Validation Accuracy	98.80%	100%	100%
Testing Accuracy	99.20%	99.99%	99.99%
Computational Time	2 Min 35 Sec	6 Min 42 Sec	3 Min 30 Sec

Table 8. Classification Accuracy and computational time for algorithms (concrete cracks dataset).

	Existing Original CNN	Existing BO-CNN	Novel Hybrid BA-BO-CNN
Training Accuracy	99.96%	99.90%	99.95%
Validation Accuracy	99.50%	99.67%	99.67%
Testing Accuracy	99.83%	99.30%	99.35%
Computational Time	3 Min 20 Sec	2 Min 46 Sec	2 Min 50 Sec

Figure 4. The pseudo code for BA-BO-CNN.

Figure 5. Number of functions evaluations.

Figure 6. Optimal values for CNN parameters.

Figure 7. Optimal weight learning rate factors.

Figure 8. Training progress for BA-BO-CNN.

Figure 9. Training confusion matrix for BA-BO-CNN.

Figure 10. Validation confusion matrix for BA-BO-CNN.

Figure 11. Testing confusion matrix for BA-BO-CNN.

Singh, A. K., B. Ganapathysubramanian, S. Sarkar, and A. Singh. 2018. Deep learning for plant stress phenotyping: Trends and future perspectives. Trends in Plant Science 23 (10):883–98. doi:10.1016/j.tplants.2018.07.004.

 PubMed Web of Science ®Google Scholar

Wang, J., Y. Ma, L. Zhang, R. X. Gao, and D. Wu. 2018. Deep learning for smart manufacturing: Methods and applications. Journal of Manufacturing Systems 48:144–56. doi:10.1016/j.jmsy.2018.01.003.

 Web of Science ®Google Scholar

Tch, A. 2017. The mostly complete chart of neural networks, explained. Online Accessed October 7, 2019. https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464.

 Google Scholar

Davydova, O. 2017. 7 types of artificial neural networks for natural language. Online Accessed October 8, 2019. https://medium.com/@datamonsters/artificial-neural-networks-fornatural-language-processing-part-1-64ca9ebfa3b2

 Google Scholar

Chandrayan, P. 2017. Deep learning: deep belief network fundamentals. Online Accessed October 8, 2019. https://codeburst.io/deep-learning-deep-belief-network-fundamentals-d0dcfd80d7d4

 Google Scholar

Le, Q. V. 2015. A tutorial on deep learning part 2: Autoencoders, convolutional neural networks and recurrent neural networks. Google Brain 20: 1–20.

 Google Scholar

Badan, F. 2019. Evolutionary algorithms in convolutional neural network design. Online Accessed March 16, 2020. http://excel.fit.vutbr.cz/submissions/2019/033/33.pdf

 Google Scholar

Chung, H., & Shin, K. S. 2019. Genetic algorithm-optimized multi-channel convolutional neural network for stock market prediction. Neural Computing and Applications 1–18.

 PubMedGoogle Scholar

Ding, L., W. Fang, H. Luo, P. E. Love, B. Zhong, and X. Ouyang. 2018. A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory. Automation in Construction 86:118–24.

 Web of Science ®Google Scholar

Popko, E. A., and I. A. Weinstein. 2016. Fuzzy logic module of convolutional neural network for handwritten digits recognition. In Journal of Physics: Conference Series, Vol. 738, No. 1 IOP Publishing, Athens, Greece, August.

 Google Scholar