Bayesian Dynamic Feature Partitioning in High-Dimensional Regression With Big Data

Abstract

Bayesian computation of high-dimensional linear regression models using Markov chain Monte Carlo (MCMC) or its variants can be extremely slow or completely prohibitive since these methods perform costly computations at each iteration of the sampling chain. Furthermore, this computational cost cannot usually be efficiently divided across a parallel architecture. These problems are aggravated if the data size is large or data arrive sequentially over time (streaming or online settings). This article proposes a novel dynamic feature partitioned regression (DFP) for efficient online inference for high-dimensional linear regressions with large or streaming data. DFP constructs a pseudo posterior density of the parameters at every time point, and quickly updates the pseudo posterior when a new block of data (data shard) arrives. DFP updates the pseudo posterior at every time point suitably and partitions the set of parameters to exploit parallelization for efficient posterior computation. The proposed approach is applied to high-dimensional linear regression models with Gaussian scale mixture priors and spike-and-slab priors on large parameter spaces, along with large data, and is found to yield state-of-the-art inferential performance. The algorithm enjoys theoretical support with pseudoposterior densities over time being arbitrarily close to the full posterior as the data size grows, as shown in the supplementary material. Supplementary material also contains details of the DFP algorithm applied to different priors. Package to implement DFP is available in https://github.com/Rene-Gutierrez/DynParRegReg. The dataset is available in https://github.com/Rene-Gutierrez/DynParRegReg\_Implementation.

Keywords:

• Bayesian statistics
• Data shards
• High-dimensional regression
• Shrinkage prior
• Streaming data
• Sufficient statistics

Supplementary Material

Supplementary Material 1: This document contains proof of the convergence behavior for the DFP algorithm. It also contains details of the DFP algorithm when applied to the linear high-dimensional regression with Bayesian Lasso prior, Horseshoe prior and Spike & Lasso prior on coefficients.

TestScript.R: The package to implement DFP is available at https://github.com/Rene-Gutierrez/DynParRegReg. We also upload TestScript.R file which uses functions from the package to run the simulations.

Additional information

Funding

The research of Rajarshi Guhaniyogi is partially supported by grants from the Office of Naval Research (BAA N000141812741) and the National Science Foundation (DMS-1854662).