A single timescale stochastic quasi-Newton method for stochastic optimization

References

• R. Bollapragada, R. Bydr, and J. Nocedal, Adaptive sampling strategies for stochastic optimization, SIAM J. Optim. 28(4) (2018), pp. 3312–3343.
   Web of Science ®Google Scholar
• D. Brownstone, D.S. Bunch, and K. Train, Joint mixed logit models of stated and revealed preferences for alternative-fuel vehicles, Transport Res. B. 34 (2000), pp. 315–338.
   Google Scholar
• F. Bastin, C. Cirillo, and P.L. Toint, Convergence theory for nonconvex stochastic programming with an application to mixed logit, Math. Program. 108 (2006), pp. 207–234.
   Web of Science ®Google Scholar
• J. Blanchet, D. Goldfarb, G. Iyengar, F. Li, and C. Zhou, Unbiased Simulation for Optimizing Stochastic Function Compositions, preprint, https://arxiv.org/abs/1711.07564, 2017.
   Google Scholar
• R.H. Byrd, S.L. Hansen, J. Nocedal, and Y. Singer, A stochastic quasi-Newton method for large-scale optimization, SIAM J. Optim. 26 (2016), pp. 1008–1031.
   Web of Science ®Google Scholar
• B. Dai, N. He, Y. Pan, B. Boots, and L. Song, Learning from conditional distributions via dual embeddings, Artif. Intell. Stat. 2 (2017), pp. 1458–1467.
   Google Scholar
• E.D. Dolan and J.J. More, Benchmarking optimization software with performance profiles, Math. Profiles 91 (2002), pp. 201–213.
   Web of Science ®Google Scholar
• D. Dentcheva, S. Penev, and A. Ruszczy n´ski, Statistical estimation of composite risk functionals and risk optimization problems, Ann. Inst. Stat. Math. 69 (2017), pp. 737–760.
   Web of Science ®Google Scholar
• Y. M. Ermoliev, Methods of Stochastic Programming, Nauka, Moscow, 1976.
   Google Scholar
• S. Ghadimi, A. Ruszczy n´ski, and M. Wang, A single timescale stochastic approximation method for nested stochastic optimization, SIAM J. Optim. 30 (2020), pp. 960–979.
   Web of Science ®Google Scholar
• R.M. Gower, D. Goldfarb, and P. Richt a´rik, Stochastic block BFGS: Squeezing more curvature out of data, in Proceedings of ICML, 2016.
   Google Scholar
• D.A. Hensher and W.H. Greene, The mixed logit model: the state of practice, Transportation 30 (2003), pp. 133–176.
   Web of Science ®Google Scholar
• Y. Lecun, C. Cortes, and C.J. Burges, The MNIST Database of Handwritten Digits, http://yann.lecun.com/exdb/mnist, 2010.
   Google Scholar
• M. Lichman, UCI Machine Learning Repository, http://archive.ics.uci.edu/ml, 2013.
   Google Scholar
• A. Lucchi, B. McWilliams, and T. Hofmann, A Variance Reduced Stochastic Newton Method, preprint, arXiv:1503.08316, 2015.
   Google Scholar
• J. Mairal, F. Bach, J. Ponce, and G. Sapiro, Online dictionary learning for sparse coding, in Proceedings of ICML, 2009.
   Google Scholar
• S. Msmussen and P.W. Glynn, Stochastic Simulation: Algorithm and Analysis, Springer, New York, 2007. Stoch. Model. Appl. Probab. 57.
   Google Scholar
• A. Mokhtari and A. Ribeiro, RES: regularized stochastic BFGS algorithm, IEEE Trans. Signal Process.62 (2014), pp. 6089–6104.
   Web of Science ®Google Scholar
• A. Mokhtari and A. Ribeiro, Global convergence of online limited memory BFGS, J. Mach. Learn. Res. 16 (2015), pp. 3151–3181.
   Web of Science ®Google Scholar
• P. Moritz, R. Nishihara, and M.I. Jordax, A linearly-convergence stochastic L-BFGS algorithm, in Proceedings of AISTATS, 2016, PP. 249–258.
   Google Scholar
• I. Mukherjee, K. Canini, R. Frongillo, and Y. Singer, Parallel boosting with momentum, in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Lecture Notes Comput. Sci. 8190, Springer, Berlin, 2013, pp. 17–32.
   Google Scholar
• A. Ruszczy n´ski, A linearization method for nonsmooth stochastic programming problems, Math. Oper. Res. 12 (1987), pp. 32–49.
   Web of Science ®Google Scholar
• S. Shalev-Shwartz and S. Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, Cambridge, UK, 2014.
   Google Scholar
• M. Wang, E.X. Fang, and B. Liu, Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions, Math. Program. 161 (2017), pp. 419–449.
   Web of Science ®Google Scholar
• M. Wang, J. Liu, and E.X. Fang, Accelerating stochastic composition optimization, J. Mach. Learn. Res. 18 (2017), pp. 1–23.
   Web of Science ®Google Scholar
• X. Wang, S. Ma, D. Gldfarb, and W. Liu, Stochastic quasi-Newton methods for nonconvex stochastic optimization, SIAM J. Optim. 27 (2017), pp. 927–956.
   Web of Science ®Google Scholar
• S. Yang, M. Wang, and E.X. Fang, Multilevel stochastic gradient methods for nested composition optimization, SIAM J. Optim. 29 (2019), pp. 616–659.
   Web of Science ®Google Scholar