Advanced search
Publication Cover
Applied Mathematics in Science and Engineering Volume 31, 2023 - Issue 1
Submit an article Journal homepage
888
Views
1
CrossRef citations to date
0
Altmetric
Research Article

Learning rate selection in stochastic gradient methods based on line search strategies

Giorgia Franchinia Department of Physics, Informatics and Mathematics, University of Modena and Reggio Emilia, Modena, ItalyCorrespondence[email protected]
View further author information
,
Federica Portaa Department of Physics, Informatics and Mathematics, University of Modena and Reggio Emilia, Modena, ItalyView further author information
,
Valeria Ruggierob Department of Mathematics and Computer Science, University of Ferrara, Ferrara, ItalyView further author information
,
Ilaria Trombinib Department of Mathematics and Computer Science, University of Ferrara, Ferrara, Italy;c Department of Mathematical, Physical and Computer Sciences, University of Parma, Parma, ItalyView further author information
&
Luca Zannia Department of Physics, Informatics and Mathematics, University of Modena and Reggio Emilia, Modena, ItalyView further author information
Article: 2164000 | Received 27 Jul 2022, Accepted 26 Dec 2022, Published online: 09 Jan 2023

References

  • Robbins H, Monro S. A stochastic approximation method. Ann Math Stat. 1951;22(3):400–407.
  • Bottou L, Curtis FE, Nocedal J. Optimization methods for large-scale machine learning. SIAM REV. 2018;60(2):223–311.
  • Liang J, Xu Y, Bao C, et al. Barzilai–Borwein-based adaptive learning rate for deep learning. Pattern Recognit Lett. 2019;128:197–203.
  • Tan C, Ma S, Dai YH, et al. Barzilai–Borwein step size for stochastic gradient method. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, editors. Advances in neural information processing systems (NIPS2016), Barcelona, Spain; 2016. p. 29.
  • Byrd RH, Chin GM, Nocedal J, et al. Sample size selection in optimization methods for machine learning. Math Program. 2012;134(1):127–155.
  • Bollapragada R, Byrd R, Nocedal J. Adaptive sampling strategies for stochastic optimization. SIAM J Optim. 2018;28(4):3312–3343.
  • Franchini G, Porta F, Ruggiero V, et al. A line search based proximal stochastic gradient algorithm with dynamical variance reduction. Optimization online; 2022.
  • Barzilai J, Borwein JM. Two-point step size gradient methods. IMA J Numer Anal. 1988;8:141–148.
  • Dai YH, Liao LZ. R-linear convergence of the Barzilai and Borwein gradient method. IMA J Numer Anal. 2002;22(1):1–10.
  • Dai YH, Fletcher R. On the asymptotic behaviour of some new gradient methods. Math Program. 2005;103:541–559.
  • di Serafino D, Ruggiero V, Toraldo G, et al. On the asymptotic behaviour of some new gradient methods. Appl Math Comput. 2018;318:176–195.
  • Franchini G, Ruggiero V, Zanni L. Ritz-like values in steplength selections for stochastic gradient methods. Soft Comput. 2020;24:17573–17588.
  • Polyak BT. Introduction to optimization. New York: Optimization Software; 1987.
  • Poon C, Liang J, Schoenlieb C. Local convergence properties of SAGA/Prox-SVRG and acceleration. In: Dy J and Krause A, editors. Proceedings of the 35th International Conference on Machine Learning. PMLR; 2018. Vol. 80. p. 4124–4132.
  • Freund JE. Mathematical statistics. Englewood Cliffs, NJ, USA: Prentice-Hall; 1962.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.