Linear convergence of proximal incremental aggregated gradient method for nonconvex nonsmooth minimization problems

References

• Aytekin A, Feyzmahdavian HR, Johansson M. Analysis and implementation of an asynchronous optimization algorithm for the parameter server; 2016. Preprint arXiv:161005507.
   Google Scholar
• Zhang H. Linear convergence of the proximal incremental aggregated gradient method under quadratic growth condition; 2017. Preprint arXiv:170208166.
   Google Scholar
• Zhang H, Dai YH, Guo L. Proximal-like incremental aggregated gradient method with linear convergence under Bregman distance growth conditions. Math Oper Res. 2019. Published Online: https://doi.org/https://doi.org/10.1287/moor.2019.1047.
   Google Scholar
• Peng W, Zhang H, Zhang X. Nonconvex proximal incremental aggregated gradient method with linear convergence. J Optim Theory Appl. 2019;183:230–245.
   Web of Science ®Google Scholar
• Sun T, Sun Y, Li D, et al. General proximal incremental aggregated gradient algorithms: better and novel results under general scheme. Proceedings of 33th Conference on Neural Information Processing Systems (NIPS 2019), Vancouver, Canada, 2019, 996–1006.
   Google Scholar
• Vanli ND, Gurbuzbalaban M, Ozdaglar A. Global convergence rate of proximal incremental aggregated gradient methods. SIAM J Optim. 2018;28(2):1282–1300.
   Web of Science ®Google Scholar
• Zhang X, Peng W, Zhang H, et al. Inertial proximal incremental aggregated gradient method; 2017. Preprint arXiv:171200984.
   Google Scholar
• Tseng P, Yun S. A coordinate gradient descent method for nonsmooth separable minimization. Math Program. 2009;117(1–2):387–423.
   Web of Science ®Google Scholar
• Curtis FE, Scheinberg K. Optimization methods for supervised machine learning: from linear models to deep learning. Preprint arXiv:170610207;2017.
   Google Scholar
• Bottou L, Curtis FE, Nocedal J. Optimization Methods for Large-Scale Machine Learning. SIAM Rev. 2018;60(2):223–311.
   Google Scholar
• Fan J, Li R. Variable selection via nonconcave penalized likelihood and its Oracle properties. J Am Stat Assoc. 2001;96(456):1348–1360.
   Web of Science ®Google Scholar
• Zhang CH. Nearly unbiased variable selection under minimax concave penalty. Ann Stat. 2010;38(2):894–942.
   Web of Science ®Google Scholar
• Huang J, Horowitz JL, Ma S. Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann Stat. 2008;36(2):587–613.
   Web of Science ®Google Scholar
• Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol). 1996;58(1):267–288.
   Google Scholar
• Chambolle A, Pock T. An introduction to continuous optimization for imaging. Acta Numer. 2016;25:161–319.
   Web of Science ®Google Scholar
• Cui FY, Tang YC, Yang Y. An inertial three-operator splitting algorithm with applications to image inpainting. Appl Set-Valued Anal Optim. 2019;1(2):113–134.
   Google Scholar
• Barbu A, She Y, Ding L, et al. Feature selection with annealing for computer vision and big data learning. IEEE Trans Pattern Anal Mach Intell. 2017;39(2):272–286.
   PubMed Web of Science ®Google Scholar
• Masnadi-shirazi H, Vasconcelos N. On the design of loss functions for classification: theory, robustness to outliers, and savageboost. Proceedings of 22th Conference on Neural Information Processing Systems (NIPS 2008), Whistler, BC, 2008. 1049–1065.
   Google Scholar
• Wu Y, Liu Y. Robust truncated hinge loss support vector machines. J Am Stat Assoc. 2007;102(479):974–983.
   Web of Science ®Google Scholar
• Chapelle O, Do CB, Teo CH, et al. Tighter bounds for structured estimation. Proceedings of 22th Conference on Neural Information Processing Systems (NIPS 2008), Whistler, BC, 2008, 281–288.
   Google Scholar
• Candes EJ, Wakin MB, Boyd SP. Enhancing sparsity by reweighted l1 minimization; 2007. Preprint arXiv:07111612.
   Google Scholar
• Wang X, Ye J, Yuan X, et al. Perturbation techniques for convergence analysis of proximal gradient method and other first-order algorithms via variational analysis. Preprint arXiv:181010051; 2018.
   Google Scholar
• Rockafellar RT, Wets RJB. Variational analysis. New York: Springer; 2009.
   Google Scholar
• Pang JS. Error bounds in mathematical programming. Math Program. 1997;79(1–3):299–332.
   Web of Science ®Google Scholar
• Wen B, Chen X, Pong TK. Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J Optim. 2017;27(1):124–145.
   Web of Science ®Google Scholar
• Blatt D, Hero AO, Gauchman H. A convergent incremental gradient method with a constant step size. SIAM J Optim. 2007;18(1):29–51.
   Web of Science ®Google Scholar
• Gurbuzbalaban M, Ozdaglar A, Parrilo PA. On the convergence rate of incremental aggregated gradient algorithms. SIAM J Optim. 2017;27(2):1035–1048.
   Web of Science ®Google Scholar
• Wai HT, Shi W, Uribe CA, et al. Accelerating incremental gradient optimization with curvature information. Comput Optim Appl. 2020;76(2):347–380.
   Web of Science ®Google Scholar
• Rockafellar RT. Convex analysis. Princeton (NJ): Princeton University Press; 1970.
   Google Scholar
• Tibshirani R, Saunders M, Rosset S, et al. Sparsity and smoothness via the fused lasso. J R Stat Soc Ser B (Stat Methodol). 2005;67(1):91–108.
   Google Scholar
• Bondell HD, Reich BJ. Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR. Biometrics. 2008;64(1):115–123.
   PubMed Web of Science ®Google Scholar
• Ye JJ, Yuan X, Zeng S, et al. Variational analysis perspective on linear convergence of some first order methods for nonsmooth convex optimization problems. Preprint Optimization. 2018; Online. http://www.optimization-online.org/DB_HTML/2018/10/6881.html
   Google Scholar
• Yuan M, Lin Y. Model selection and estimation in regression with grouped variables. J R Stat Soc Ser (Stat Methodol). 2006;68(1):49–67.
   Web of Science ®Google Scholar
• Schmidt M, Roux NL, Bach F. Minimizing finite sums with the stochastic average gradient. Math Program. 2017;162(1):83–112.
   Web of Science ®Google Scholar
• Defazio A, Bach F, Lacoste-Julien S. Saga: a fast incremental gradient method with support for non-strongly convex composite objectives. Proceedings of 28th Conference on Neural Information Processing Systems (NIPS 2014), Montreal, QC, 2014, 1646–1654.
   Google Scholar
• Reddi SJ, Sra S, Poczos B, et al. Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Proceedings of 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 2016, 1145–1153.
   Google Scholar
• Xu Y, Rong J, Yang T. Non-asymptotic analysis of stochastic methods for non-smooth non-convex regularized problems. Proceedings of 33th Conference on Neural Information Processing Systems (NIPS 2019), Vancouver, Canada, 2019, 2630–2640.
   Google Scholar