Least squares auto-tuning

Abstract

Least squares auto-tuning automatically finds hyper-parameters in least squares problems that minimize another (true) objective. The least squares tuning optimization problem is non-convex, so it cannot be solved efficiently. This article presents a powerful proximal gradient method for least squares auto-tuning that can be used to find good, if not the best, hyper-parameters for least squares problems. The application of least squares auto-tuning to data fitting is discussed. Numerical experiments on a classification problem using the MNIST dataset demonstrate the effectiveness of the method; it is able to cut the test error of standard least squares in half.

Keywords:

• Least squares
• hyper-parameter optimization
• proximal gradient method

Disclosure statement

No potential conflict of interest was reported by the author(s).

ORCID

Shane T. Barratt  http://orcid.org/0000-0002-7127-0724

Additional information

Funding

This work was supported by the National Science Foundation Graduate Research Fellowship [Grant No. DGE-1656518].