Research on three-step accelerated gradient algorithm in deep learning

Abstract

Gradient descent (GD) algorithm is the widely used optimisation method in training machine learning and deep learning models. In this paper, based on GD, Polyak's momentum (PM), and Nesterov accelerated gradient (NAG), we give the convergence of the algorithms from an initial value to the optimal value of an objective function in simple quadratic form. Based on the convergence property of the quadratic function, two sister sequences of NAG's iteration and parallel tangent methods in neural networks, the three-step accelerated gradient (TAG) algorithm is proposed, which has three sequences other than two sister sequences. To illustrate the performance of this algorithm, we compare the proposed algorithm with the three other algorithms in quadratic function, high-dimensional quadratic functions, and nonquadratic function. Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning. For conveniently facilitate the proposed algorithms, we rewite the R package ‘neuralnet’ and extend it to ‘supneuralnet’. All kinds of deep learning algorithms in this paper are included in ‘supneuralnet’ package. Finally, we show our algorithms are superior to other algorithms in four case studies.

Keywords:

• Accelerated algorithm
• backpropagation
• deep learning
• learning rate
• momentum
• stochastic gradient descent

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by National Natural Science Foundation of China (11271136, 81530086), Program of Shanghai Subject Chief Scientist (14XD1401600) and the 111 Project of China (No. B14019).

Notes on contributors

Yongqiang Lian

Yongqiang Lian is a Ph.D. candidate in Statistics at East China Normal University.

Yincai Tang

Yincai Tang is a Professor in School of Statistics at East China Normal University.

Shirong Zhou

Shirong Zhou is a Ph.D. student in Statistics at East China Normal University.