Weak and strong convergence analysis of Elman neural networks via weight decay regularization

References

• Rumelhart DE, Hinton GF, Williams RJ. Learning representations by back-propagation errors. Nature. 1986;323:533–536.
   Web of Science ®Google Scholar
• Ho SS, Schofield M, Wang N. Learning incentivization strategy for resource rebalancing in shared services with a budget constraint. J Appl Numer Optim. 2021;3:105–114.
   Google Scholar
• An NT, Dong PD, Qin X. Robust feature selection via nonconvex sparsity-based methods. J Nonlinear Var Anal. 2021;5:59–77.
   Web of Science ®Google Scholar
• Huang T, Li C, Duan S, et al. Robust exponential stability of uncertain delayed neural networks with stochastic perturbation and impulse effects. IEEE Trans Neural Netw Learn Syst. 2012;23:866–875.
   PubMed Web of Science ®Google Scholar
• Hu R, Wen S, Zeng Z, et al. A short-term power load forecasting model based on the generalized regression neural network with decreasing step fruit fly optimization. Neurocomputing. 2017;221:24–31.
   Web of Science ®Google Scholar
• Prieto A, Prieto B, Ortigosa EM, et al. Neural networks: an overview of early research, current frameworks and new challenges. Neurocomputing. 2016;214:242–268.
   Web of Science ®Google Scholar
• Han HG, Lu W, Hou Y, et al. An adaptive-PSO-based self-organizing RBF neural network. IEEE Trans Neural Netw Learn Syst. 2018;29:104–117.
   PubMed Web of Science ®Google Scholar
• Wang L, Meng Z, Sun Y, et al. Design and analysis of a novel chaotic diagonal recurrent neural network. Commun Nonlinear Sci Numer Simul. 2015;26(1–3):11–23.
   Web of Science ®Google Scholar
• Chen JC, Wang YM. Comparing activation functions in modeling shoreline variation using multilayer perceptron neural network. Water. 2020;12:1281.
   Web of Science ®Google Scholar
• Yen EC. Solubility and stability of recurrent neural networks with nonlinearity or time-varying delays. Commun Nonlinear Sci Numer Simul. 2011;16(1):509–521.
   Web of Science ®Google Scholar
• Elman JL. Finding structure in time. Cogn Sci. 1991;14:179–211.
   Web of Science ®Google Scholar
• Elman JL. Distributed representation simple recurrent networks and grammatical structure. Mach Learn. 1991;7:195–225.
   Web of Science ®Google Scholar
• Ooi SY, Tan SC, Cheah WP. Experimental study of elman network in temporal classification. Singapore: Springer; 2017.
   Google Scholar
• Tsoi AC, Back AD. Locally recurrent globally feedforward networks: a critical review of architectures. IEEE Trans Neural Networks. 1994;5:229–239.
   PubMed Web of Science ®Google Scholar
• Wei W, Shao H, Di Q. Strong convergence of gradient methods for BP networks training. International Conference on Neural Networks & Brain. IEEE; 2005.
   Google Scholar
• Tong X, Wang Z, Yu H. A research using hybrid RBF/Elman neural networks for intrusion detection system secure model. Comput Phys Commun. 2009;180:1795–1801.
   Web of Science ®Google Scholar
• Alkhasawneh MS, Tay LT. A hybrid intelligent system integrating the cascade forward neural network with Elman neural network. Arabian J Sci Eng. 2018;43:6737–6749.
   Web of Science ®Google Scholar
• Mehrgini B, Izadi H, Memarian H. Shear wave velocity prediction using Elman artificial neural network. Carbonates Evaporites. 2019;34:1281–1291.
   Web of Science ®Google Scholar
• Wang F, Wang MY, Tian FS, et al. Study on two-dimensional distribution of X-ray image based on improved Elman algorithm. Radiat Meas. 2015;77:1–4.
   Web of Science ®Google Scholar
• Ardalani-Farsa M, Zolfaghari S. Chaotic time series prediction with residual analysis method using hybrid Elman-NARX neural networks. Neurocomputing. 2010;73:2540–2553.
   Web of Science ®Google Scholar
• Liu H, Tian HQ, Liang XF, et al. Wind speed forecasting approach using secondary decomposition algorithm and Elman neural networks. Appl Energy. 2015;157:183–194.
   Web of Science ®Google Scholar
• Liou CY, Huang JC, Yang WC. Modeling word perception using the Elman network. Neurocomputing. 2008;71:3150–3157.
   Web of Science ®Google Scholar
• Alkhasawneh MS. Hybrid cascade forward neural network with Elman neural network for disease prediction. Arab J Sci Eng. 2019;44:9209–9220.
   Web of Science ®Google Scholar
• Xia CK, Su CL, Cao JT, et al. Multiboost with ENN-based ensemble fault diagnosis method and its application in complicated chemical process. J Cent South Univ. 2016;23(5):1183–1197.
   Web of Science ®Google Scholar
• Zhou Y, Tian L, Liu L. Improved extension neural network and its applications. Math Probl Eng. 2014;16:1–14.
   Google Scholar
• Ruiz LGB, Rueda R, Cullar MP, et al. Energy consumption forecasting based on Elman neural networks with evolutive optimization. Expert Syst Appl. 2018;92:380–389.
   Web of Science ®Google Scholar
• Massinaei M, Falaghi H, Izadi H. Optimisation of metallurgical performance of industrial flotation column using neural network and gravitational search algorithm. Can Metall Q. 2013;52:115–122.
   Web of Science ®Google Scholar
• Fan QW, Kang Q, Zurada JM. Convergence analysis for sigma-pi-sigma neural network based on some relaxed conditions. Inf Sci (Ny). 2022;585:70–88.
   Web of Science ®Google Scholar
• Guo C, Lu J, Tian Z, et al. Optimization of critical parameters of PEM fuel cell using TLBO-DE based on Elman neural network. Energy Convers Manage. 2019;183:149–158.
   Web of Science ®Google Scholar
• Xie K, Yi H, Hu G, et al. Short-Term power load forecasting based on Elman neural network with particle swarm optimization. Neurocomputing. 2020;416:136–142.
   Web of Science ®Google Scholar
• Jia W, Zhao D, Ding L, et al. A reliable small sample classification algorithm by Elman neural network based on PLS and GA. J Classif. 2019;36:306–321.
   Web of Science ®Google Scholar
• Ren G, Cao Y, Wen S, et al. A modified Elman neural network with a new learning rate scheme. Neurocomputing. 2018;286:11–18.
   Web of Science ®Google Scholar
• Fan QW, Zurada JM, Wu W. Convergence of online gradient method for feedforward neural networks with smoothing L1/2 regularization penalty. Neurocomputing. 2014;131:208–216.
   Web of Science ®Google Scholar
• Fan QW, Wu W, Zurada JM. Convergence of batch gradient learning with smoothing regularization and adaptive momentum for neural networks. SpringerPlus. 2016;5(1):295.
   PubMedGoogle Scholar
• Ludwig O, Nunes U, Araujo R. Eigenvalue decay:A new method for neural network regularization. Neurocomputing. 2014;124:33–42.
   Web of Science ®Google Scholar
• Schwarz G. Estimating the dimension of a model. Ann Stat. 1978;6(2):461–464.
   Web of Science ®Google Scholar
• Natarajan BK. Sparse approximate solutions to linear systems. SIAM J Comput. 1995;24:227–234.
   Web of Science ®Google Scholar
• Tibshirani RJ. Regression shrinkage and selection via the Lasso. J R Stat Soc Ser B: Methodol. 1996;73(1):273–282.
   Google Scholar
• Fan QW, Niu L, Kang Q. Regression and multiclass classification using sparse extreme learning machine via smoothing group L1/2 regularizer. IEEE Access. 2020;8:191482–191494.
   Web of Science ®Google Scholar
• Luo X, Chang XH, Ban XJ. Regression and classification using extreme learning machine based on L1-norm and L2-norm. Neurocomputing. 2016;174:179–186.
   Web of Science ®Google Scholar
• Fan QW, Peng J, Li H, et al. Convergence of a gradient-based learning algorithm with penalty for ridge polynomial neural networks. IEEE Access. 2021;9:28742–28752.
   Web of Science ®Google Scholar
• Saito K, Nakano R. Second-order learning algorithm with squared penalty term. Neural Comput. 2000;12:709–729.
   PubMed Web of Science ®Google Scholar
• Reed R. Pruning algorithms: a survey. IEEE Trans Neural Netw Learn Syst. 1993;4:740–747.
   PubMed Web of Science ®Google Scholar
• He HM, Peng JG, Li HY. Iterative approximation of fixed point problems and variational inequality problems on Hadamard manifolds. U P B Bull Ser A. 2022;84(1):25–36.
   Google Scholar
• He HM, Peng JG, Fan QW. An iterative viscosity approximation method for the split common fixed-point problem. Optimization. 2021;70(5–6):1261–1274.
   Web of Science ®Google Scholar
• Zhang H, Tang Y, Liu X. Batch gradient training method with smoothing L0 regularization for feedforward neural networks. Neural Comput Appl. 2014;26:383–390.
   Web of Science ®Google Scholar
• Zhang LQ, Wu W. Online gradient methods with a penalty term for neural networks with large training set. J Nonlinear Dyn Sci Technol. 2004;11:53–58.
   Google Scholar
• Wu W, Feng GR, Li ZX, et al. Convergence of an online gradient method for BP neural networks. IEEE Trans Neural Netw. 2005;16:533–540.
   PubMed Web of Science ®Google Scholar
• Heskes T, Wiegerinck W. A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning. IEEE Trans Neural Netw. 1996;7(4):919–925.
   PubMed Web of Science ®Google Scholar
• Wu W, Xu DP, Li Z. Convergence of gradient method for Elman networks. Appl Math Mech. 2008;29:1231–1238.
   Google Scholar
• Xu DP, Li ZX, Wu W. Convergence of approximated gradient method for Elman network. Neural Network World. 2008;18:171–180.
   Web of Science ®Google Scholar
• Kang Q, Fan QW, Zurada JM. Deterministic convergence analysis via smoothing group Lasso regularization and adaptive momentum for sigma-pi-sigma neural network. Inf Sci (Ny). 2021;553:66–82.
   Web of Science ®Google Scholar
• Gori M, Maggini M. Optimal convergence of on-line backpropagation. IEEE Trans Neural Networks. 1996;7(1):251–254.
   PubMed Web of Science ®Google Scholar
• Wang DL, Liu X, Ahalt SC. On temporal generalization of simple recurrent networks. Neural Netw. 1996;9(7):1099–1117.
   PubMed Web of Science ®Google Scholar