Bayesian Robustness: A Nonasymptotic Viewpoint

Abstract

We study the problem of robustly estimating the posterior distribution for the setting where observed data can be contaminated with potentially adversarial outliers. We propose Rob-ULA, a robust variant of the Unadjusted Langevin Algorithm (ULA), and provide a finite-sample analysis of its sampling distribution. In particular, we show that after $T=\tilde{\mathcal{O}}(d/{\epsilon }_{\mathsf{acc}})$ iterations, we can sample from p_T such that $\text{dist}(p_T,p^{*})\le {\epsilon }_{\mathsf{acc}}+\tilde{\mathcal{O}}(ϵ)$, where ϵ is the fraction of corruptions and $\text{dist}$ represents the squared 2-Wasserstein distance metric. Our results for the class of posteriors $p^{*}$ which satisfy log-concavity and smoothness assumptions. We corroborate our theoretical analysis with experiments on both synthetic and real-world datasets for mean estimation, regression and binary classification. Supplementary materials for this article are available online.

KEYWORDS:

• Huber contamination
• MCMC methods
• Robust statistics

Supplementary Materials

The supplementary material for this paper contains the proofs of the main theorems along with additional experiment details.

Acknowledgments

We would like to thank members of SAIL, Stat-learning and InterACT labs at Berkeley for helpful discussions.

Disclosure Statement

The authors report there are no competing interests to declare.

Notes

1 For a pair of distributions p, q with smooth and continuous densities, the log-Sobolev inequality implies that $\text{KL}(p||q)\le \frac{1}{2\alpha }J(p||q)$ where $J(p||q)$ is the relative Fisher information of p with respect to q.

Additional information

Funding

This work is supported in part by the National Science Foundation grants NSF-SCALE MoDL(2134209) and NSF-CCF-2112665 (TILOS), the U.S. Department of Energy Office of Science, and the Facebook Research Award.