A Scalable Method to Exploit Screening in Gaussian Process Models with Noise

Abstract

A common approach to approximating Gaussian log-likelihoods at scale exploits the fact that precision matrices can be well-approximated by sparse matrices in some circumstances. This strategy is motivated by the screening effect, which refers to the phenomenon in which the linear prediction of a process Z at a point ${\mathit{x}}_0$ depends primarily on measurements nearest to ${\mathit{x}}_0$. But simple perturbations, such as iid measurement noise, can significantly reduce the degree to which this exploitable phenomenon occurs. While strategies to cope with this issue already exist and are certainly improvements over ignoring the problem, in this work we present a new one based on the EM algorithm that offers several advantages. While in this work we focus on the application to Vecchia’s approximation (Vecchia), a particularly popular and powerful framework in which we can demonstrate true second-order optimization of M steps, the method can also be applied using entirely matrix-vector products, making it applicable to a very wide class of precision matrix-based approximation methods. Supplementary materials for this article are available online.

Keywords:

• EM algorithm
• Gaussian processes
• Stochastic trace estimation
• Vecchia’s approximation

Supplementary Materials

The supplemental materials provide proofs of Proposition 1 and Theorem 1 and a demonstration of the effect of varying the number of sampling vectors in the stochastic trace as discussed at the end of Section 2.

Acknowledgments

The authors thank Lydia Zoells for her careful copyediting.

Notes

1 Code for the EM procedure and a kernel-agnostic Vecchia likelihood, as well as all of the scripts used to generate the results in this work, are available as a Julia language package at https://github.com/cgeoga/Vecchia.jl.

Additional information

Funding

This material was based upon work supported by the US Department of Energy, Office of Science, Office of Advanced Scientific Computing Research (ASCR) under Contract DE-AC02-06CH11347.