Log-Linear Bayesian Additive Regression Trees for Multinomial Logistic and Count Regression Models

Abstract

We introduce Bayesian additive regression trees (BART) for log-linear models including multinomial logistic regression and count regression with zero-inflation and overdispersion. BART has been applied to nonparametric mean regression and binary classification problems in a range of settings. However, existing applications of BART have been mostly limited to models for Gaussian “data,” either observed or latent. This is primarily because efficient MCMC algorithms are available for Gaussian likelihoods. But while many useful models are naturally cast in terms of latent Gaussian variables, many others are not—including models considered in this article. We develop new data augmentation strategies and carefully specified prior distributions for these new models. Like the original BART prior, the new prior distributions are carefully constructed and calibrated to be flexible while guarding against overfitting. Together the new priors and data augmentation schemes allow us to implement an efficient MCMC sampler outside the context of Gaussian models. The utility of these new methods is illustrated with examples and an application to a previously published dataset. Supplementary materials for this article are available online.

Keywords:

• Multinomial logistic regression
• Negative binomial regression
• Nonparametric Bayes
• Poisson regression
• Zero inflation

Supplementary Materials

The supplemental material contains comparisons of logistic and probit models for binary outcomes, proofs of propositions, details of MCMC algorithms, additional results for the classification study, and details about heteroskedastic regression models with log-linear priors.

Acknowledgments

Thanks to P. Richard Hahn and Carlos Carvalho for helpful comments and suggestions on an early version of this article.

Notes

1 As pointed out by a reviewer, in some contexts it may be desirable to shrink toward some particular value for $1-\omega (\mathbf{x})$; this can be accomplished by setting $1-\omega ({\mathbf{x}}_i)=\frac{n_0{f}^{(0)}({\mathbf{x}}_i)}{n_0{f}^{(0)}({\mathbf{x}}_i)+n_1{f}^{(1)}({\mathbf{x}}_i)}$, which centers the prior at $n_0/(n_0+n_1)$ with increasing values of $n_0+n_1$ imply stronger shrinkage.

2 As noted by a referee, the normal approximation used to calibrate the prior might not hold as well with few trees—the symmetry and slightly heavier tails of the $P_{\lambda }$ generate a prior that puts somewhat less mass in the central interval. Given how close the $P_{\lambda }$ prior is to normal, we expect this prior is reasonable in any event.

3 We attempted to include Kindo,Wang, and Peña's (2016) multinomial probit BART, but the accompanying R package routinely crashed during simulations. We expect that it would perform similar to multinomial logistic BART in cross-validation, at substantially increased computational cost due to the need to update several latent Gaussian variables per covariate value as well as a latent covariance matrix, and to cross-validate the choice of reference category in addition to m and the parameters of the covariance matrix prior. (Kindo, Wang, and Peña (2016) proposed no default settings for reference category or prior on the covariance matrix.)

Additional information

Funding

The author gratefully acknowledges support from the National Science Foundation under grant numbers SES-1130706, SES-1631970, SES-1824555, and DMS-1043903. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies.