Bayesian Conditional Density Filtering

ABSTRACT

We propose a conditional density filtering (C-DF) algorithm for efficient online Bayesian inference. C-DF adapts MCMC sampling to the online setting, sampling from approximations to conditional posterior distributions obtained by propagating surrogate conditional sufficient statistics (a function of data and parameter estimates) as new data arrive. These quantities eliminate the need to store or process the entire dataset simultaneously and offer a number of desirable features. Often, these include a reduction in memory requirements and runtime and improved mixing, along with state-of-the-art parameter inference and prediction. These improvements are demonstrated through several illustrative examples including an application to high dimensional compressed regression. In the cases where dimension of the model parameter does not grow with time, we also establish sufficient conditions under which C-DF samples converge to the target posterior distribution asymptotically as sampling proceeds and more data arrive. Supplementary materials of C-DF are available online.

KEYWORDS:

• Approximate MCMC
• Density filtering
• Dimension reduction
• Sequential Monte Carlo
• Streaming data

Supplementary Materials

Included sample code:

• 1-samplecode_lm_cdf: Linear model example with C-DF, SMCMC

• 2-samplecode_anova_cdf: Anova example with C-DF, SMCMC

• 3-samplecode_probit_cdf: Binary regression example with C-DF, SMCMC, S-VB

• 4-samplecode_dlm_cdf: First-order time series model with C-DF, PL

Simulated datasets (several replications, e.g., 10) were used to report average summary statistics in Tables for each example. “Seed” values were not used, so new simulations may result in minor number differences.

For each example, code for inference methods (e.g., “C-DF”, “SMCMC” (batch Gibbs / MH)) can be run as-is, and we include sample code to generate kernel-density plots for inferential parameters of interest, and compute coverage and mean-square-error stats using stored (approximate) MCMC draws.

Notes

1 [θ_j|θ_{− j}, D] is the distribution of θ_j conditional on other model parameters θ_{− j} and a dataset D.

2 In cases where closed-form expression for the conditional mean is available, this may be used instead of Monte Carlo estimates.

3 The GDP prior has been demonstrated to have attractive shrinkage properties, with a carefully constructed hierarchical prior that induces Cauchy-like tails leading to better robustness (less bias due to over shrinkage) in estimating true signals, while also accommodating Laplace-like shrinkage near zero, leading to concentration around sparse coefficient vectors.