Comparative Investigation of Learning Algorithms for Image Classification with Small Dataset

Figures & data

Figure 1. Three common forms of machine learning, their subclasses and renowened algorihtms

Figure 2. Dataset for image classification task with ten classes: Calculator, Cigarette Pack, Fork, Glasses, Hook, Mug, Rubber Duck, Scissor, Stapler, and Toothbrush

Table 1. Eight learning algorithms with their learning rates, mini-batch sizes, number of layers, and number of epochs

Learning algorithms	Learning rates (η)	Mini-batch sizes (Mb)	Number of layers (Nl)	Number of epochs (Ne)
SGDNesterov	0.0009, 0.0015, 0.003	32 and 64	38 and 50	50 and 70
AdaGrad	0.0001, 0.0004, 0.0008
RMSProp	0.000009, 0.00001, 0.00003
AdaDelta	0.03, 0.06, 0.11
Adam	0.00001, 0.00005, 0.0001
AdaMax	0.00007, 0.00015, 0.0004
Nadam	0.00002, 0.00004, 0.00008
AMSGrad	0.0000099, 0.00003, 0.00009

Figure 3. The layout of first and second phases of the proposed algorithm using coarse to fine random sampling scheme

Figure 4. Accuracy and cost curves of eight learning algorithms on training and development sets. (a) Training accuracy, (b) Development accuracy, (c) Training cost, (d) Development cost

Figure 4. Continued

Figure 5. Precision-recall curves of eight learning algorithms on test set of ten object classes. (a) SGDNesterov optimizer, (b) AdaGrad optimizer, (c) RMSProp optimizer, (d) AdaDelta optimizer, (e) Adam optimizer, (f) AdaMax optimizer, (g) Nadam optimizer, (h) AMSGrad optimizer

Figure 5. Continued

Figure 5. Continued

Figure 5. Continued

Figure 6. Confusion matrices of eight learning algorithms on test set of ten object classes. (a) SGDNesterov optimizer, (b) AdaGrad optimizer, (c) RMSProp optimizer, (d) AdaDelta optimizer, (e) Adam optimizer, (f) AdaMax optimizer, (g) Nadam optimizer, (h) AMSGrad optimizer

Figure 6. Continued

Table 2. Training time, memory utilization, and accuracy on test set of eight learning algorithms. Bold font shows the best results

Learning algorithms	Training time (seconds)		Trained model memory (megabytes)		Accuracy on test set
	Minimum	Maximum	38 Layers architecture	50 Layers architecture	Average	Best
SGDNesterov AdaGrad RMSProp AdaDelta Adam AdaMax Nadam AMSGrad	166 157 161 175 168 164 176 173	365 362 374 418 395 382 418 409	135 135 135 203 203 203 203 270	180 180 180 271 271 271 271 361	0.871 0.889 0.847 0.877 0.882 0.864 0.855 0.857	0.938 0.959 0.917 0.938 0.927 0.941 0.955 0.934