Adaptive Conditional Distribution Estimation with Bayesian Decision Tree Ensembles

Abstract

We present a Bayesian nonparametric model for conditional distribution estimation using Bayesian additive regression trees (BART). The generative model we use is based on rejection sampling from a base model. Like other BART models, our model is flexible, has a default prior specification, and is computationally convenient. To address the distinguished role of the response in our BART model, we introduce an approach to targeted smoothing of BART models which is of independent interest. We study the proposed model theoretically and provide sufficient conditions for the posterior distribution to concentrate at close to the minimax optimal rate adaptively over smoothness classes in the high-dimensional regime in which many predictors are irrelevant. To fit our model, we propose a data augmentation algorithm which allows for existing BART samplers to be extended with minimal effort. We illustrate the performance of our methodology on simulated data and use it to study the relationship between education and body mass index using data from the medical expenditure panel survey (MEPS). Supplementary materials for this article are available online.

Keywords:

• Bayesian nonparametrics
• Density estimation
• High dimensional
• Nonparametric methods
• Random forests

Supplementary Materials

In the Supplementary Material we provide proofs of all the results, give additional computational details and simulation results, and provide code for reproducing some of the results of our simulation study.

Additional information

Funding

This work was partially supported by grant SES-2046896 from the National Science Foundation.