Regression Density Estimation With Variational Methods and Stochastic Approximation

Abstract

Regression density estimation is the problem of flexibly estimating a response distribution as a function of covariates. An important approach to regression density estimation uses finite mixture models and our article considers flexible mixtures of heteroscedastic regression (MHR) models where the response distribution is a normal mixture, with the component means, variances, and mixture weights all varying as a function of covariates. Our article develops fast variational approximation (VA) methods for inference. Our motivation is that alternative computationally intensive Markov chain Monte Carlo (MCMC) methods for fitting mixture models are difficult to apply when it is desired to fit models repeatedly in exploratory analysis and model choice. Our article makes three contributions. First, a VA for MHR models is described where the variational lower bound is in closed form. Second, the basic approximation can be improved by using stochastic approximation (SA) methods to perturb the initial solution to attain higher accuracy. Third, the advantages of our approach for model choice and evaluation compared with MCMC-based approaches are illustrated. These advantages are particularly compelling for time series data where repeated refitting for one-step-ahead prediction in model choice and diagnostics and in rolling-window computations is very common. Supplementary materials for the article are available online.

Key Words:

• Bayesian model selection
• Heteroscedasticity
• Mixtures of experts
• Stochastic approximation
• Variational Bayes

ACKNOWLEDGMENTS

The authors thank John Ormerod for comments and suggestions related to this work. The authors also thank Lucy Marshall for supplying the data for the example in Section 6.1. David J. Nott was supported by the Singapore Ministry of Education (MOE) grant R-155-000-068-133. Siew Li Tan was partly supported as part of the Singapore Delft Water Alliance’s (SDWA) tropical reservoir research program. Robert Kohn’s research was partially supported by the Australian Research Council (ARC) grant DP0988579.