A noise-based stabilizer for convolutional neural networks

References

• Goodfellow I, Bengio Y, Courville A. Deep learning. MIT Press; 2016; Available from: http://www.deeplearningbook.org. Accessed 20 December 2017.
   Google Scholar
• Russell S, Norvig P. Artificial intelligence: a modern approach. Upper saddle river. 3rd ed. NJ: Prentice Hall Press; 2009.
   Google Scholar
• Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge. Inter J Comput Vision (IJCV). 2015;115(3):211–252. doi: 10.1007/s11263-015-0816-y
   Web of Science ®Google Scholar
• LeCun Y, Cortes C. MNIST handwritten digit database; 2010. Available from: http://yannlecuncom/exdb/mnist/. Accessed 5 June 2017.
   Google Scholar
• Krizhevsky A, Nair V, Hinton G. Cifar-10 (canadian institute for advanced research); [cited 2018]. Available from: http://wwwcstorontoedu/∼kriz/cifarhtml
   Google Scholar
• Garofolo JS, Lamel LF, Fisher WM, et al. TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1. Web Download. Philadelphia: Linguistic Data Consortium; 1993. Available from: https://catalog.ldc.upenn.edu/LDC93S1. Accessed 5 June 2017.
   Google Scholar
• Soomro K, Zamir AR, Shah M. Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR. 2012; abs/1212.0402. Available from https://arxiv.org/abs/1212.0402. Accessed 5 June 2017.
   Google Scholar
• Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. CoRR. 2014; abs/1409.1556. Available from: http://arxiv.org/abs/1409.1556. Accessed 10 June 2017.
   Google Scholar
• Sutskever I, Martens J, Dahl G, et al. On the importance of initialization and momentum in deep learning. Proceedings of the 30th International Conference on International Conference on Machine Learning. Vol. 28, JMLR.org; 2013. p. III–1139–III–1147; ICML'13.
   Google Scholar
• Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011 Jul;12:2121–2159.
   Web of Science ®Google Scholar
• Zeiler MD. ADADELTA: an adaptive learning rate method. CoRR. 2012; abs/1212.5701.
   Google Scholar
• Kingma DP, Ba J. Adam: a method for stochastic optimization. CoRR. 2014; abs/1412.6980.
   Google Scholar
• Bach S, Binder A, Montavon G. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE. 2015;10(7):1–46. Available from: https://doi.org/10.1371/journal.pone.0130140 . Accessed 10 June 2017.
   Web of Science ®Google Scholar
• Girosi F, Jones M, Poggio T. Regularization theory and neural networks architectures. Neural Comput. 1995;7:219–269. doi: 10.1162/neco.1995.7.2.219
   Web of Science ®Google Scholar
• Krogh A, Hertz JA. A simple weight decay can improve generalization. In: Advances in Neural Information Processing Systems 4. Morgan Kaufmann; 1992. p. 950–957.
   Google Scholar
• Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc Ser B. 2005;67:301–320. doi: 10.1111/j.1467-9868.2005.00503.x
   Google Scholar
• Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014 Jan;15(1):1929–1958.
   Google Scholar
• Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach F, Blei D, editors. Proceedings of the 32nd International Conference on Machine Learning; (Proceedings of Machine Learning Research; Vol. 37); 07–09 Jul; Lille, France. PMLR; 2015. p. 448–456.
   Google Scholar
• Zhang C, Bengio S, Hardt M, et al. Understanding deep learning requires rethinking generalization. CoRR. 2016; abs/1611.03530.
   Google Scholar
• van Noord N, Postma E. Learning scale-variant and scale-invariant features for deep image classification. Pattern Recognit. 2017;61:583–592. doi: 10.1016/j.patcog.2016.06.005
   Web of Science ®Google Scholar
• Dieleman S, Willett KW, Dambre J. Rotation-invariant convolutional neural networks for galaxy morphology prediction. CoRR. 2015; abs/1503.07077.
   Google Scholar
• Dieleman S, Fauw JD, Kavukcuoglu K. Exploiting cyclic symmetry in convolutional neural networks. CoRR. 2016; abs/1602.02660.
   Google Scholar
• Cohen T, Welling M. Group equivariant convolutional networks. In: Balcan MF, Weinberger KQ, editors. Proceedings of The 33rd International Conference on Machine Learning; (Proceedings of Machine Learning Research; Vol. 48); 20–22 Jun; NY, USA. PMLR; 2016. p. 2990–2999.
   Google Scholar
• Lowe DG. Distinctive image features from scale-invariant keypoints. Inter J Comput Vision. 2004 Nov;60(2):91–110. doi: 10.1023/B:VISI.0000029664.99615.94
   Web of Science ®Google Scholar
• Galván IM, Valls JM, García M, et al. A lazy learning approach for building classification models. Int J Intell Syst. 2011;26:773–786. doi: 10.1002/int.20493
   Web of Science ®Google Scholar
• Valls JM, Galván IM, Isasi P. LRBNN: A lazy radial basis neural network model. AI Commun. 2007;20(2):71–86.
   Web of Science ®Google Scholar
• Galván I, Isasi P, Aler R, et al. A selective learning method to improve the generalization of multilayer feedforward neural networks. Int J Neural Syst. 2001 May;11:167–77. doi: 10.1142/S0129065701000588
   PubMedGoogle Scholar
• Zheng Z, Webb GI. Lazy learning of Bayesian rules. Mach Learn. 2000 Oct;41(1):53–84. doi: 10.1023/A:1007613203719
   Web of Science ®Google Scholar
• An G. The effects of adding noise during backpropagation training on a generalization performance. Neural Comput. 1996 Apr;8(3):643–674. doi: 10.1162/neco.1996.8.3.643
   Web of Science ®Google Scholar
• Audhkhasi K, Osoba O, Kosko B. Noise-enhanced convolutional neural networks. Neural Netw. 2016;78:15–23. doi: 10.1016/j.neunet.2015.09.014
   PubMed Web of Science ®Google Scholar
• Gülçehre Ç, Moczulski M, Denil M, et al. Noisy activation functions. CoRR. 2016; abs/1603.00391.
   Google Scholar
• Belharbi S, Chatelain C, Hérault R, et al. Neural networks regularization through class-wise invariant representation learning. CoRR. 2017; abs/1709.01867.
   Google Scholar
• Phan N, Wu X, Hu H, et al. Adaptive laplace mechanism: differential privacy preservation in deep learning. CoRR. 2017; abs/1709.05750.
   Google Scholar
• Xu Q, Zhang M, Gu Z, et al. Overfitting remedy by sparsifying regularization on fully-connected layers of cnns. Neurocomputing. 2018.
   Google Scholar
• Khan SH, Hayat M, Porikli F. Regularization of deep neural networks with spectral dropout. Neural Networks, Elsevier; 2018.
   Google Scholar
• Xie L, Wang J, Wei Z, et al. Disturblabel: regularizing cnn on the loss layer. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jun; 2016.
   Google Scholar
• Binder A, Montavon G, Bach S, et al. Layer-wise relevance propagation for neural networks with local renormalization layers. CoRR. 2016; abs/1604.00825.
   Google Scholar
• Spiegel MR. Laplace transforms. New Delhi: McGraw Hill; c1965.
   Google Scholar
• Bruna J, Mallat S. Invariant scattering convolution networks. IEEE Trans Pattern Anal Mach Intell. 2013 Aug;35(8):1872–1886. doi: 10.1109/TPAMI.2012.230
   PubMed Web of Science ®Google Scholar
• Bietti A, Mairal J. Invariance and stability of deep convolutional representations. In: Guyon I, Luxburg UV, Bengio S, et al., editors. Advances in neural information processing systems 30. Curran Associates, Inc.; 2017. p. 6210–6220.
   Google Scholar
• Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE; 1998. p. 2278–2324.
   Google Scholar
• Li X, Chen S, Hu X, et al. Understanding the disharmony between dropout and batch normalization by variance shift. CoRR. 2018; abs/1801.05134.
   Google Scholar
• Goodfellow I, Warde-Farley D, Mirza M, et al. Maxout networks. In: Dasgupta S, McAllester D, editors. Proceedings of the 30th International Conference on Machine Learning; (Proceedings of Machine Learning Research; Vol. 28); 17–19 Jun; Atlanta, Georgia, USA. PMLR; 2013. p. 1319–1327.
   Google Scholar
• Lee CY, Xie S, Gallagher P, et al. Deeply-Supervised Nets. In: Lebanon G, Vishwanathan SVN, editors. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics; (Proceedings of Machine Learning Research; Vol. 38); 09–12 May; San Diego, California, USA. PMLR; 2015. p. 562–570.
   Google Scholar
• Wan L, Zeiler M, Zhang S, et al. Regularization of neural networks using dropconnect. In: Dasgupta S, McAllester D, editors. Proceedings of the 30th International Conference on Machine Learning; (Proceedings of Machine Learning Research; Vol. 28); 17–19 Jun; Atlanta, Georgia, USA. PMLR; 2013. p. 1058–1066.
   Google Scholar
• deeplearningnet. Maxpool. Convolutional Neural Networks; [cited 2018]. Available from: http://deeplearning.net/tutorial/lenet.html
   Google Scholar
• Szegedy C, Zaremba W, Sutskever I. Intriguing properties of neural networks. International Conference on Learning Representations; 2014. Available from: http://arxiv.org/abs/1312.6199. Accessed 2 Feb 2018.
   Google Scholar
• Balan R, Singh M, Zou D. Lipschitz properties for deep convolutional networks. CoRR. 2017; abs/1701.05217.
   Google Scholar
• Yeh R, Hasegawa-Johnson M, Do MN. Stable and symmetric filter convolutional neural network. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); March; 2016. p. 2652–2656.
   Google Scholar
• MnistVariations. Variations on the mnist digits; [cited 2018]. Available from: http://wwwiroumontrealca
   Google Scholar