Active Learning for Deep Gaussian Process Surrogates

Abstract

Deep Gaussian processes (DGPs) are increasingly popular as predictive models in machine learning for their nonstationary flexibility and ability to cope with abrupt regime changes in training data. Here, we explore DGPs as surrogates for computer simulation experiments whose response surfaces exhibit similar characteristics. In particular, we transport a DGP’s automatic warping of the input space and full uncertainty quantification, via a novel elliptical slice sampling Bayesian posterior inferential scheme, through to active learning strategies that distribute runs nonuniformly in the input space—something an ordinary (stationary) GP could not do. Building up the design sequentially in this way allows smaller training sets, limiting both expensive evaluation of the simulator code and mitigating cubic costs of DGP inference. When training data sizes are kept small through careful acquisition, and with parsimonious layout of latent layers, the framework can be both effective and computationally tractable. Our methods are illustrated on simulation data and two real computer experiments of varying input dimensionality. We provide an open source implementation in the deepgp package on CRAN.

Keywords:

• Computer model
• Elliptical slice sampling
• Emulator
• Kriging
• Sequential design

Supplementary Material

Illustration of ALC surface for the 2d example, 3-dimensional computer experiment example, and derivations of IMSE (19) and ALC (20).

Acknowledgments

We thank referees for their valuable insights and suggestions that led to an improved article.

Additional information

Funding

This work was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through Advanced Computing (SciDAC) program under Award Number 0000231018.