Optimize to generalize in Gaussian processes: An alternative objective based on the Rényi divergence

Abstract

We introduce an alternative closed-form objective function α-ELBO for improved parameter estimation in the Gaussian process ($\mathcal{G}\mathcal{P}$) based on the Rényi α-divergence. We use a decreasing temperature parameter α to iteratively deform the objective function during optimization. Ultimately, our objective function converges to the exact log-marginal likelihood function of $\mathcal{G}\mathcal{P}.$ At early optimization stages, α-ELBO can be viewed as a regularizer that smoothes some unwanted critical points. At late stages, α-ELBO recovers the exact log-marginal likelihood function that guides the optimizer to solutions that best explain the observed data. Theoretically, we derive an upper bound of the Rényi divergence under the proposed objective and derive convergence rates for a class of smooth and non-smooth kernels. Case studies on a wide range of real-life engineering applications demonstrate that our proposed objective is a practical alternative that offers improved prediction performance over several state-of-the-art inference techniques.

Keywords:

• Gaussian process
• Rényi divergence
• convergence
• engineering applications

Additional information

Funding

The authors acknowledge the generous support from the NSF CAREER AWARD 2144147.

Notes on contributors

Xubo Yue

Xubo Yue is a PhD candidate in the Department of Industrial & Operations Engineering at the University of Michigan. His research focuses on federated and distributed data analytics. Currently, he is developing federated data analytics methods that rethink how both prescriptive and predictive analytics are achieved within IoT-enabled systems, specifically manufacturing and renewable energy. He has received several best paper awards from the Institute for Operations Research and the Management Sciences (INFORMS), the Institute of Industrial and Systems Engineers (IISE), and other renowned organizations.

Raed Al Kontar

Raed Al Kontar is an assistant professor in the Industrial & Operations Engineering Department at the University of Michigan and an affiliate with the Michigan Institute for Data Science. Raed’s research focuses on collaborative, distributed, and decentralized data science. Raed obtained an undergraduate degree in civil & environmental engineering and mathematics from the American University of Beirut in 2014 and a master’s degree in statistics in 2017 and a PhD degree in Industrial & System Engineering in 2018, both from the University of Wisconsin-Madison.