Generalized Bayesian Additive Regression Trees Models: Beyond Conditional Conjugacy

Abstract

Bayesian additive regression trees have seen increased interest in recent years due to their ability to combine machine learning techniques with principled uncertainty quantification. The Bayesian backfitting algorithm used to fit BART models, however, limits their application to a small class of models for which conditional conjugacy exists. In this article, we greatly expand the domain of applicability of BART to arbitrary generalized BART models by introducing a very simple, tuning-parameter-free, reversible jump Markov chain Monte Carlo algorithm. Our algorithm requires only that the user be able to compute the likelihood and (optionally) its gradient and Fisher information. The potential applications are very broad; we consider examples in survival analysis, structured heteroscedastic regression, and gamma shape regression. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

KEYWORDS:

• Bayesian methods
• Bayesian nonparametrics
• Decision Trees
• Markov chain Monte Carlo
• Random forests
• Smoothing and nonparametric regression

Supplementary Materials

The online supplement contains additional illustrations, a heuristic justification of the algorithm, and proofs of all results.

Acknowledgments

The author thanks the reviewers for their helpful comments.

Disclosure Statement

The author reports no competing interest to declare.

Additional information

Funding

This work is partially supported by NSF award DMS-2144933.