Dynamic Mixture of Experts Models for Online Prediction

Abstract

A mixture of experts models the conditional density of a response variable using a mixture of regression models with covariate-dependent mixture weights. We extend the finite mixture of experts model by allowing the parameters in both the mixture components and the weights to evolve in time by following random walk processes. Inference for time-varying parameters in richly parameterized mixture of experts models is challenging. We propose a sequential Monte Carlo algorithm for online inference based on a tailored proposal distribution built on ideas from linear Bayes methods and the EM algorithm. The method gives a unified treatment for mixtures with time-varying parameters, including the special case of static parameters. We assess the properties of the method on simulated data and on industrial data where the aim is to predict software faults in a continuously upgraded large-scale software project.

KEYWORDS:

• Bayesian sequential inference
• Linear Bayes
• Mixture models
• Particle filtering
• Sequential Monte Carlo

Supplementary Materials

The electronic supplements include two files: (i) The appendices.pdf file provides details on the expressions of the gradient and the Hessian of the dynamic mixture of experts models and a discussion on the identifiability of the models applied in Section 4. (ii) The dmoe_code.zip includes source code files for reproducing the results in Section 5.

Acknowledgments

We thank the editor, the associate editor, and two referees for their valuable comments and suggestions that helped to improve the article. The author are grateful to the Data Insight team in Ericsson for providing the software fault data.

Disclosure Statement

The authors report that there are no competing interests to declare.