Dynamic Mixture of Experts Models for Online Prediction

References

• Andrieu, C., Doucet, A., and Holenstein, R. (2010), “Particle Markov Chain Monte Carlo Methods,” Journal of the Royal Statistical Society, Series B, 72, 269–342. DOI: 10.1111/j.1467-9868.2009.00736.x.
   Google Scholar
• Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018), “Automatic Differentiation in Machine Learning: A Survey,” Journal of Machine Learning Research, 18, 1–43.
   Web of Science ®Google Scholar
• Bishop, C. M. (2006), Pattern Recognition and Machine Learning, New York: Springer.
   Google Scholar
• Carpenter, J., Clifford, P., and Fearnhead, P. (1999), “Improved Particle Filter for Nonlinear Problems,” IEE Proceedings-Radar, Sonar and Navigation, 146, 2–7. DOI: 10.1049/ip-rsn:19990255.
   Web of Science ®Google Scholar
• Carvalho, A. X., and Tanner, M. A. (2005a), “Mixtures-of-Experts of Autoregressive Time Series: Asymptotic Normality and Model Specification,” IEEE Transactions on Neural Networks, 16, 39–56. DOI: 10.1109/TNN.2004.839356.
   PubMedGoogle Scholar
• Carvalho, A. X., and Tanner, M. A. (2005b), “Modeling Nonlinear Time Series with Local Mixtures of Generalized Linear Models,” Canadian Journal of Statistics, 33, 97–113. DOI: 10.1002/cjs.5540330108.
   Web of Science ®Google Scholar
• Carvalho, A. X., and Tanner, M. A. (2007), “Modelling Nonlinear Count Time Series with Local Mixtures of Poisson Autoregressions,” Computational Statistics & Data Analysis, 51, 5266–5294.
   Web of Science ®Google Scholar
• Carvalho, C. M., Lopes, H. F., Polson, N. G., and Taddy, M. A. (2010), “Particle Learning for General Mixtures,” Bayesian Analysis, 5, 709–740. DOI: 10.1214/10-BA525.
   Web of Science ®Google Scholar
• Celeux, G., Frühwirth-Schnatter, S., and Robert, C. P. (2019), “Model Selection for Mixture Models–Perspectives and Strategies,” in Handbook of Mixture Analysis, eds. S. Fruhwirth-Schnatter, G. Celeux, C. P. Robert, pp. 117–154, New York: Chapman and Hall/CRC.
   Google Scholar
• Chopin, N. (2004), “Central Limit Theorem for Sequential Monte Carlo Methods and its Application to Bayesian Inference,” The Annals of Statistics, 32, 2385–2411. DOI: 10.1214/009053604000000698.
   Web of Science ®Google Scholar
• Chopin, N., Jacob, P. E., and Papaspiliopoulos, O. (2013), “SMC2: An Efficient Algorithm for Sequential Analysis of State Space Models,” Journal of the Royal Statistical Society, Series B, 75, 397–426. DOI: 10.1111/j.1467-9868.2012.01046.x.
   Google Scholar
• Del Moral, P., Doucet, A., and Jasra, A. (2006), “Sequential Monte Carlo Samplers,” Journal of the Royal Statistical Society, Series B, 68, 411–436. DOI: 10.1111/j.1467-9868.2006.00553.x.
   Google Scholar
• Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977), “Maximum Likelihood from Incomplete Data via the EM Algorithm,” Journal of the Royal Statistical Society, Series B, 39, 1–22.
   Google Scholar
• Douc, R., and Cappé, O. (2005), “Comparison of Resampling Schemes for Particle Filtering,” in Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, ISPA 2005, pp. 64–69. IEEE. DOI: 10.1109/ISPA.2005.195385.
   Google Scholar
• Doucet, A., Briers, M., and Sénécal, S. (2006), “Efficient Block Sampling Strategies for Sequential Monte Carlo Methods,” Journal of Computational and Graphical Statistics, 15, 693–711. DOI: 10.1198/106186006X142744.
   Web of Science ®Google Scholar
• Doucet, A., De Freitas, N., and Gordon, N. J. (2001), Sequential Monte Carlo Methods in Practice (Vol. 1), Springer.
   Google Scholar
• Doucet, A., Godsill, S., and Andrieu, C. (2000), “On Sequential Monte Carlo Sampling Methods for Bayesian Filtering,” Statistics and Computing, 10, 197–208.
   Web of Science ®Google Scholar
• Doucet, A., and Johansen, A. M. (2009), “A Tutorial on Particle Filtering and Smoothing: Fifteen Years Later,” in Handbook of Nonlinear Filteringeds. D. Crisan and B. Rozovskiĭ, pp. 656–704, Oxford: Oxford University Press.
   Google Scholar
• Fahrmeir, L., and Kneib, T. (2011), Bayesian Smoothing and Regression for Longitudinal, Spatial and Event History Data, Oxford: Oxford University Press.
   Google Scholar
• Fahrmeir, L., Kneib, T., and Lang, S. (2004), “Penalized Structured Additive Regression for Space-Time Data: A Bayesian Perspective,” Statistica Sinica, 14, 731–761.
   Web of Science ®Google Scholar
• Famoye, F., and Singh, K. P. (2006), “Zero-Inflated Generalized Poisson Regression Model with an Application to Domestic Violence Data,” Journal of Data Science, 4, 117–130. DOI: 10.6339/JDS.2006.04(1).257.
   Google Scholar
• Fearnhead, P., and Clifford, P. (2003), “On-Line Inference for Hidden Markov Models via Particle Filters,” Journal of the Royal Statistical Society, Series B, 65, 887–899. DOI: 10.1111/1467-9868.00421.
   Google Scholar
• Gamerman, D. (1991), “Dynamic Bayesian Models for Survival Data,” Journal of the Royal Statistical Society, Series C, 40, 63–79. DOI: 10.2307/2347905.
   Web of Science ®Google Scholar
• Gamerman, D. (1998), “Markov Chain Monte Carlo for Dynamic Generalised Linear Models,” Biometrika, 85, 215–227.
   Web of Science ®Google Scholar
• Geweke, J. (1989), “Bayesian Inference in Econometric Models using Monte Carlo Integration,” Econometrica: Journal of the Econometric Society, 57, 1317–1339.
   Web of Science ®Google Scholar
• Geweke, J. (2007), “Interpretation and Inference in Mixture Models: Simple MCMC Works,” Computational Statistics & Data Analysis, 51, 3529–3550.
   Web of Science ®Google Scholar
• Geweke, J., and M. Keane (2007), “Smoothly Mixing Regressions,” Journal of Econometrics, 138, 252–290. DOI: 10.1016/j.jeconom.2006.05.022.
   Web of Science ®Google Scholar
• Goldstein, M., and Wooff, D. (2007), Bayes Linear Statistics: Theory and Methods (Vol. 716), Chichester: Wiley.
   Google Scholar
• Gordon, N. J., Salmond, D. J., and Smith, A. F. (1993), “Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation,” in IEE Proceedings F (Radar and Signal Processing) (Vol. 140), pp. 107–113. IET. DOI: 10.1049/ip-f-2.1993.0015.
   Google Scholar
• Gormley, I. C., and Frühwirth-Schnatter, S. (2018), “Mixtures of Experts Models,” arXiv preprint arXiv:1806.08200.
   Google Scholar
• Hastie, T., and Tibshirani, R. (1993), “Varying-Coefficient Models,” Journal of the Royal Statistical Society, Series B, 55, 757–779. DOI: 10.1111/j.2517-6161.1993.tb01939.x.
   Google Scholar
• Hjort, N. L., Holmes, C., Müller, P., and Walker, S. G. (2010), Bayesian nonparametrics (Vol. 28), Cambridge: Cambridge University Press.
   Google Scholar
• Hunter, D. R., and Young, D. S. (2012), “Semiparametric Mixtures of Regressions,” Journal of Nonparametric Statistics, 24, 19–38. DOI: 10.1080/10485252.2011.608430.
   Web of Science ®Google Scholar
• Jacobs, R. A., Peng, F., and Tanner, M. A. (1997), “A Bayesian Approach to Model Selection in Hierarchical Mixtures-of-Experts Architectures,” Neural Networks, 10, 231–241. DOI: 10.1016/S0893-6080(96)00050-0.
   PubMed Web of Science ®Google Scholar
• Jiang, W., and Tanner, M. A. (1999), “On the Identifiability of Mixtures-of-Experts,” Neural Networks, 12, 1253–1258. DOI: 10.1016/S0893-6080(99)00066-0.
   PubMed Web of Science ®Google Scholar
• Jordan, M. I., and Jacobs, R. A. (1994), “Hierarchical Mixtures of Experts and the EM Algorithm,” Neural Computation, 6, 181–214. DOI: 10.1162/neco.1994.6.2.181.
   Web of Science ®Google Scholar
• Klaas, M., De Freitas, N., and Doucet, A. (2005), “Toward Practical N2 Monte Carlo: The Marginal Particle Filter,” Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI2005).
   Google Scholar
• Kohlmorgen, J., Müller, K.-R., Rittweger, J., and Pawelzik, K. (2000), “Identification of Nonstationary Dynamics in Physiological Recordings,” Biological Cybernetics, 83, 73–84. DOI: 10.1007/s004220000144.
   PubMed Web of Science ®Google Scholar
• Lang, S., Fronk, E.-M., and Fahrmeir, L. (2002), “Function Estimation with Locally Adaptive Dynamic Models,” Computational Statistics, 17, 479–499. DOI: 10.1007/s001800200121.
   Web of Science ®Google Scholar
• Liehr, S., Pawelzik, K., Kohlmorgen, J., and Müller, K. R. (1999), “Hidden Markov Mixtures of Experts with an Application to EEG Recordings from Sleep,” Theory in Biosciences, 118, 246–260.
   Web of Science ®Google Scholar
• Liu, J., and West, M. (2001), “Combined Parameter and State Estimation in Simulation-based Filtering,” in Sequential Monte Carlo Methods in Practice, eds. A. Doucet, N. de Freitas, N. Gordon, pp. 197–223, New York: Springer.
   Google Scholar
• Liu, J. S., and Chen, R. (1998), “Sequential Monte Carlo Methods for Dynamic Systems,” Journal of the American Statistical Association, 93, 1032–1044. DOI: 10.1080/01621459.1998.10473765.
   Web of Science ®Google Scholar
• Malsiner-Walli, G., Frühwirth-Schnatter, S., and Grün, B. (2017), “Identifying Mixtures of Mixtures using Bayesian Estimation,” Journal of Computational and Graphical Statistics, 26, 285–295. DOI: 10.1080/10618600.2016.1200472.
   PubMed Web of Science ®Google Scholar
• Migon, H. S., Schmidt, A. M., Ravines, R. E., and Pereira, J. (2013), “An Efficient Sampling Scheme for Dynamic Generalized Models,” Computational Statistics, 28, 2267–2293. DOI: 10.1007/s00180-013-0406-9.
   Web of Science ®Google Scholar
• Munezero, P. (2021), “Efficient Particle Smoothing for Bayesian Inference in Dynamic Survival Models,” Computational Statistics, 37, 975–994. DOI: 10.1007/s00180-021-01155-7.
   Web of Science ®Google Scholar
• Muthén, B., and Shedden, K. (1999), “Finite Mixture Modeling with Mixture Outcomes using the EM Algorithm,” Biometrics, 55, 463–469. DOI: 10.1111/j.0006-341x.1999.00463.x.
   PubMed Web of Science ®Google Scholar
• Pitt, M. K., and Shephard, N. (1999), “Filtering via Simulation: Auxiliary Particle Filters,” Journal of the American statistical Association, 94, 590–599. DOI: 10.1080/01621459.1999.10474153.
   Web of Science ®Google Scholar
• Quiroz, M., and Villani, M. (2013), “Dynamic Mixture-of-Experts Models for Longitudinal and Discrete-Time Survival Data,” Manuscript. Available at https://github.com/mattiasvillani/Papers/raw/master/DynamicMixture.pdf
   Google Scholar
• Rasmussen, C. E., and Ghahramani, Z. (2002), “Infinite Mixtures of Gaussian Process Experts,” in Advances in Neural Information Processing Systems, pp. 881–888.
   Google Scholar
• Richardson, S., and Green, P. J. (1997), “On Bayesian Analysis of Mixtures with an Unknown Number of Components” (with discussion), Journal of the Royal Statistical Society, Series B, 59, 731–792. DOI: 10.1111/1467-9868.00095.
   Google Scholar
• Rue, H., and Held, L. (2005), Gaussian Markov Random Fields: Theory and Applications, Boca Raton, FL: CRC Press.
   Google Scholar
• Stephens, M. (2000), “Dealing with Label Switching in Mixture Models,” Journal of the Royal Statistical Society, Series B, 62, 795–809. DOI: 10.1111/1467-9868.00265.
   Google Scholar
• Villani, M., Kohn, R., and Giordani, P. (2009), “Regression Density Estimation using Smooth Adaptive Gaussian Mixtures,” Journal of Econometrics, 153, 155–173. DOI: 10.1016/j.jeconom.2009.05.004.
   Web of Science ®Google Scholar
• Villani, M., Kohn, R., and Nott, D. J. (2012), “Generalized Smooth Finite Mixtures,” Journal of Econometrics, 171, 121–133. DOI: 10.1016/j.jeconom.2012.06.012.
   Web of Science ®Google Scholar
• Wang, X., Whigham, P., Deng, D., and Purvis, M. (2003), “Time-Line Hidden Markov Experts for Time Series Prediction,” in Proceedings of the 2003 International Conference on Neural Networks and Signal Processing (Vol. 1), pp. 786–789. IEEE.
   Google Scholar
• West, M., Harrison, P. J., and Migon, H. S. (1985), “Dynamic Generalized Linear Models and Bayesian Forecasting,” Journal of the American Statistical Association, 80, 73–83. DOI: 10.1080/01621459.1985.10477131.
   Web of Science ®Google Scholar
• Wood, S. A., Jiang, W., and Tanner, M. (2002), “Bayesian Mixture of Splines for Spatially Adaptive Nonparametric Regression,” Biometrika, 89, 513–528. DOI: 10.1093/biomet/89.3.513.
   Web of Science ®Google Scholar
• Yuksel, S. E., Wilson, J. N., and Gader, P. D. (2012), “Twenty Years of Mixture of Experts,” IEEE Transactions on Neural Networks and Learning Systems, 23, 1177–1193. DOI: 10.1109/TNNLS.2012.2200299.
   PubMed Web of Science ®Google Scholar
• Zeevi, A., Meir, R., and Adler, R. (1996), “Time Series Prediction using Mixtures of Experts,” in Advances in Neural Information Processing Systems (Vol. 9).
   Google Scholar