An Overview of Deep Generative Models

REFERENCES

• T. S. Lee, and D. Mumford, “Hierarchical Bayesian inference in the visual cortex,” The Journal of the Optical Society of America A, Vol. 20, no. 7, pp. 1434–48, Jul. 2003.
   Google Scholar
• T. Serre, L. Wolf, and S. Bileschi, “Robust object recognition with cortex-like mechanisms,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 29, no. 3, pp. 411–26, Mar. 2007.
   PubMed Web of Science ®Google Scholar
• T. S. Lee, D. Mumford, and R. Romero, “The role of the primary visual cortex in higher level vision,” Vision Research, Vol. 38, no. 15, pp. 2429–54, Aug. 1998.
   PubMed Web of Science ®Google Scholar
• D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, Vol. 323, no. 7, pp. 533–6, Oct. 1986.
   Web of Science ®Google Scholar
• Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Advances in Neural Information Processing Systems, Vol. 19, B. Schölkopf, J. C. Platt and T. Hoffman, Eds. Cambridge, MA: MIT Press, 2006, pp. 153–60.
   Google Scholar
• H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin, “Exploring strategies for training deep neural networks,” Journal of Machine Learning Research, Vol. 1, pp. 1–40, Jan. 2009.
   Google Scholar
• P. J. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Boston: Harvard University, 1974.
   Google Scholar
• R. Hecht-Nielsen, “Replicator neural networks for universal optimal source coding,” Science, Vol. 269, pp. 1860–3, Sept. 1995.
   PubMed Web of Science ®Google Scholar
• G. Tesauro, “Practical issues in temporal difference learning,” Machine Learning, Vol. 8, no. 3–4, pp. 257–77, May 1992.
   Web of Science ®Google Scholar
• Y. Bengio, “Learning deep architectures for AI,” Foundations and trends® in Machine Learning, Vol. 2, no. 1, pp. 1–127, Jan. 2009.
   Google Scholar
• Y. Bengio, and Y. LeCun, “Scaling learning algorithms towards AI,” Large-Scale Kernel Machines, Vol. 34, pp. 1–41, Sept. 2007.
   Google Scholar
• G. E. Hinton, S. Osindero, and Y. W. The, “A learning algorithm for deep belief nets,” Neural Computation, Vol. 18, no. 7, pp. 1527–54, Jul. 2006.
   PubMed Web of Science ®Google Scholar
• B. Taskar, P. Abbeel, and D. Koller, “Discriminative probabilistic models for relational data,” in Proceedings of Conference on Uncertainty in Artificial Intelligence, Alberta, 2002, pp. 485–92.
   Google Scholar
• J. A. Lasserre, C. M. Bishop, and T. P. Minka, “Principled hybrids of generative and discriminative models,” in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New York, 2006, pp. 87–94.
   Google Scholar
• M. I. Jordan, Learning in Graphical Models. Dordrecht: Kluwer, 1998.
   Google Scholar
• P. Dayan, G. E. Hinton, R. Neal, and R. Zemel. “The Helmholtz machine,” Neural Computation, Vol. 7, no. 5, pp. 889–904, Sept. 1995.
   PubMed Web of Science ®Google Scholar
• G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal, “The “wake-sleep” algorithm for unsupervised neural network,” Science, Vol. 268, no. 5214, pp. 1558–161, May 1995.
   PubMed Web of Science ®Google Scholar
• L. K. Saul, T. Jaakkola, and M. I. Jordan, “Mean field theory for sigmoid belief networks,” Journal of Artificial Intelligence Research, Vol. 4, no. 1, pp. 61–76, Jan. 1996.
   Web of Science ®Google Scholar
• I. Titov, and J. Henderson, “Constituent parsing with incremental sigmoid belief networks,” in Proceedings of Meeting of Association for Computational Linguistics, Prague, 2007, pp. 632–9.
   Google Scholar
• P. Smolensky, “Information processing in dynamical systems: foundations of harmony theory,” Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, pp. 194–281, Feb. 1986.
   Google Scholar
• Y. Freund, and D. Haussler, “Unsupervised learning of distributions on binary vectors using two layer networks,” in Advances in Neural Information Processing Systems, Vol. 4, J. E. Moody, S. J. Hanson, and R.P. Lippmann, Eds. Denver, CO: Morgan Kaufmann, 1991, pp. 912–9.
   Google Scholar
• G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, Vol. 14, no. 8, pp. 1771–800, Aug. 2002.
   PubMed Web of Science ®Google Scholar
• M. Welling, M. Rosen-Zvi, and G. E. Hinton, “Exponential family harmoniums with an application to information retrieval,” in Advances in Neural Information Processing Systems, Vol. 17, L. K. Saul, Y. Weiss and L. Bottou, Eds. Cambridge, MA: MIT Press, 2004, pp. 1481–8.
   Google Scholar
• R. Salakhutdinov, and G. E. Hinton, “Deep Boltzmann machines,” in Proceedings of International Conference on Artificial Intelligence and Statistics, Florida, 2009, pp. 448–55.
   Google Scholar
• G. E. Hinton, and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, Vol. 313, no. 5786, pp. 504–7, May 2006.
   PubMed Web of Science ®Google Scholar
• R. Collobert, and J. Weston, “A unified architecture for natural language processing: Deep neural networks with multitask learning,” in Proceedings of International Conference on Machine learning, Helsinki, 2008, pp. 160–7.
   Google Scholar
• M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun, “Efficient learning of sparse representations with an energy-based model,” in Advances in Neural Information Processing Systems, Vol. 19, B. Schölkopf, J. C. Platt and T. Hoffman, Eds. Cambridge: MIT Press, 2006, pp. 1137–44.
   Google Scholar
• P. Y. Simard, D. Steinkraus, and J. C. Platt, “Best practices for convolutional neural networks applied to visual document analysis,” in Proceedings of the 7th International Conference on Document Analysis and Recognition, Washington DC, 2003, pp. 958–63.
   Google Scholar
• L. Deng, and D. Yu, “Deep learning for signal and information processing,” Microsoft Research Report, Redmond, 2013.
   Google Scholar
• K. H. Cho, T. Raiko, and A. Ilin, “Parallel tempering is efficient for learning restricted Boltzmann machines,” in Proceedings of the 2010 International Joint Conference on Neural Networks, Thessaloniki, 2010, pp. 1–8.
   Google Scholar
• N. Le Roux, and Y. Bengio, “Representational power of restricted Boltzmann machines and deep belief networks,” Neural Computation, Vol. 20, no. 6, pp. 1631–49, Jun. 2008.
   PubMed Web of Science ®Google Scholar
• A. Fischer, and C. Igel, “Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines,” in Proceedings of the 20th International Conference on Artificial Neural Networks, Thessaloniki, 2010, pp. 208–17.
   Google Scholar
• G. E. Hinton, “Products of experts,” in Proceedings of the 9th International Conference on Artificial Neural Networks, London, 1999, pp. 1–6.
   Google Scholar
• G. E. Hinton, “Learning multiple layers of representation,” Trends in Cognitive Sciences, Vol. 11, no. 10, pp. 428–34, Oct. 2007.
   PubMed Web of Science ®Google Scholar
• T. Tieleman, “Training restricted Boltzmann machines using approximations to the likelihood gradient,” in Proceedings of the 25th International Conference on Machine Learning, New York, 2008, pp. 1064–71.
   Google Scholar
• T. Tieleman, and G. Hinton, “Using fast weights to improve persistent contrastive divergence,” in Proceedings of the 26th Annual International Conference on Machine Learning, New York, 2009, pp. 1033–40.
   Google Scholar
• Y. Bengio, and O. Delalleau, “Justifying and generalizing contrastive divergence,” Neural Computation, Vol. 21, no. 6, pp. 1601–21, Jun. 2009.
   PubMed Web of Science ®Google Scholar
• A. Fischer, and C. Igel, “Empirical analysis of the divergence of Gibbs sampling based learning algorithms for restricted Boltzmann machines,” in Proceedings of the 20th International Conference on Artificial Neural Networks, Thessaloniki, 2010, pp. 208–17.
   Google Scholar
• D. J. Earl, and M. W. Deem, “Parallel tempering: theory, applications, and new perspectives,” Physical Chemistry Chemical Physics, Vol. 7, pp. 3910–6, Aug. 2005.
   PubMed Web of Science ®Google Scholar
• G. Desjardins, A. Courville, and Y. Bengio, “Parallel tempering for training of restricted Boltzmann machines,” in Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, New York, 2010, pp. 145–52.
   Google Scholar
• R. M. Neal, “Sampling from multimodal distributions using tempered transitions,” Statistics and Computing, Vol. 6, no. 4, pp. 353–66, Dec. 1996.
   Web of Science ®Google Scholar
• Y. Iba, “Extended ensemble monte carlo,” International Journal of Modern Physics, Vol. 12, no. 5, pp. 623–56, Jun. 2001.
   Web of Science ®Google Scholar
• J. Xu, H. Li, and S. Zhou, “Improving Mixing Rate with Tempered Transition for Learning Restricted Boltzmann Machines,” Neurocomputing, Vol. 139, pp. 328–35, Sept. 2014.
   Web of Science ®Google Scholar
• D. C. Plaut, and G. E. Hinton, “Learning sets of filters using back-propagation,” Computer, Speech and Language, Vol. 2, no. 1, pp. 35–61, Mar. 1987.
   Google Scholar
• D. DeMers, and G. Cottrell, “Non-linear dimension reduction,” in Advances in Neural Information Processing Systems, Vol. 5, S. J. Hanson, J. D. Cowan and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 580–7.
   Google Scholar
• R. Hecht-Nielsen, “Replicator neural networks for universal optimal source coding,” Science, Vol. 269, no. 5232, pp. 1860–3, Sept. 1995.
   PubMed Web of Science ®Google Scholar
• N. Kambhatla, and T. K. Leen, “Dimension reduction by local principal component analysis,” Neural Computation, Vol. 9, no. 7, pp. 1493–516, Oct. 1997.
   Web of Science ®Google Scholar
• R. Salakhutdinov, and G. E. Hinton, “Deep Boltzmann machines,” in Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, Clearwater Beach, 2009, pp. 448–55.
   Google Scholar
• R. Salakhutdinov, and G. Hinton, “A better way to pretrain deep Boltzmann machines,” Advances in Neural Information Processing Systems, Vol. 25, F. Pereira, C. J. C. Burges, L. Bottou and K. Q. Weinberger, Eds. Cambridge, MA: MIT Press, 2012, pp. 1–9.
   Google Scholar
• R. Salakhutdinov, “Learning deep generative models,” Ph.D. Dissertation, Graduate Department of Computer Science, Univ. Toronto, Toronto, 2009.
   Google Scholar
• A. Tanveer, J. Taskeed, and C. Ui-Pil, “Facial expression recognition using local transitional pattern on gabor filtered facial images”, IETE Technical Review, Vol. 30, no. 1, pp. 47–52, Jan. 2013.
   Web of Science ®Google Scholar
• G. E. Hinton, S. Osindero, and Y. W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, Vol. 18, no. 7, pp. 1527–54, Jul. 2006.
   PubMed Web of Science ®Google Scholar
• J. Luo, and A. Brodsky, “An EM-based multi-step piecewise surface regression learning algorithm,” in Proceedings of the 7th International Conference on Data Mining, Las Vegas, 2011, pp. 286–92.
   Google Scholar
• J. Luo, A. Brodsky, and Y. Li, “An EM-based ensemble learning algorithm on piecewise surface regression problem,” International Journal of Applied Mathematics and Statistics, Vol. 28, no. 4, pp. 59–74, Aug. 2012.
   Google Scholar
• V. Nair, and G.Hinton, “3-d object recognition with deep belief nets,” in Advances in Neural Information Processing Systems, Vol. 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams and A. Culotta, Eds. Cambridge, MA: MIT Press, 2009, pp. 1339–47.
   Google Scholar
• Y. Tang, and C. Eliasmith, “Deep networks for robust visual recognition,” in Proceedings of the 27th International Conference on Machine Learning, Haifa, 2010, pp. 1055–62.
   Google Scholar
• A. Taralba, R. Fergus, and Y. Weiss, “Small codes and large image databases for recognition,” in Proceedings of Computer Vision and Pattern Recognition, Anchorage, 2008, pp. 1–8.
   Google Scholar
• J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Ng, “Multimodal deep learning,” in Proceedings of the 28th International Conference on Machine Learning, Bellevue, 2011, pp. 689–96.
   Google Scholar
• N. Srivastava, and R. Salakhutdinov, “Multimodal learning with deep Boltzmann machines,”in Advances in Neural Information Processing Systems, Vol. 25, F. Pereira, C. J. C. Burges, L. Bottou and K. Q. Weinberger, Eds. Montreal, Canada: NIPS, 2012, pp. 2222–30.
   Google Scholar
• A. Mohamed, G. Dahl, and G. Hinton, “Deep belief networks for phone recognition,” in Proceedings of Neural Information Processing Systems 2009 Workshop on Deep Learning for Speech Recognition and Related Applications, Vancouver, 2009.
   Google Scholar
• G. Sivaram, and H. Hermansky, “Sparse multilayer perceptron for phoneme recognition,” IEEE Trans. Audio, Speech, & Language Processing, Vol. 20, no. 1, pp. 23–9, Jan. 2012.
   Web of Science ®Google Scholar
• A. Mohamed, G. Dahl, and G. Hinton, “Acoustic Modeling Using Deep Belief Networks,” IEEE Trans. Audio, Speech, & Language Processing, Vol. 20, no. 1, pp. 14–22, Jan. 2012.
   Web of Science ®Google Scholar
• A. Mohamed, G. Hinton, and G. Penn, “Understanding how deep belief networks perform acoustic modelling,” in Proceedings of the 37th International Conference on Acoustics, Speech, and Signal Processing, Kyoto, 2012, pp. 4273–76.
   Google Scholar
• A. Mohamed, D. Yu, and L. Deng, “Investigation of full-sequence training of deep belief networks for speech recognition,” in Proceedings of the 11th Annual Conference of the International Speech Communication Association, Makuhari, 2010, pp. 2846–9.
   Google Scholar
• D. Yu, F. Seide, G. Li, and L. Deng, “Exploiting sparseness in deep neural networks for large vocabulary speech recognition,” in Proceedings of the 37th International Conference on Acoustics, Speech, and Signal Processing, Kyoto, 2012, pp. 4409–12.
   Google Scholar
• D. Yu, S. Wang, Z. Karam, and L. Deng, “Language recognition using deep-structured conditional random fields,” in Proceedings of the 35th International Conference on Acoustics, Speech and Signal Processing, 2010, pp. 5030–3.
   Google Scholar
• F. Seide, G. Li, X. Chen, and D. Yu, “Feature engineering in context-dependent deep neural networks for conversational speech transcription,” in Proceedings of the 2011 IEEE Workshop on Automatic Speech Recognition and Understanding, Hawaii, 2011, pp. 24–9.
   Google Scholar
• G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent DBN-HMMs in large vocabulary continuous speech recognition,” in Proceedings of the 36th International Conference on Acoustics, Speech, and Signal Processing, Prague, 2011, pp. 4688–91.
   Google Scholar
• G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent, pre-trained deep neural networks for large vocabulary speech recognition,” IEEE Trans. Audio, Speech, & Language Proc., Vol. 20, no. 1, pp. 30–42, Jan. 2012.
   Web of Science ®Google Scholar
• Y. Kubo, T. Hori, and A. Nakamura, “Integrating deep neural networks into structural classification approach based on weighted finite-state transducers,” in Proceedings of the 13th Annual Conference of the International Speech Communication Association, Portland, 2012.
   Google Scholar
• L. Deng, J. Li, K. Huang, D. Yao, F. Yu, M. Seide, G. Seltzer, X. Zweig, J. He, Y. Williams, and A. Acero. “Recent advances in deep learning for speech research at Microsoft,” in Proceedings of International Conference on Acoustics, Speech and Signal Processing, Vancouver, 2013, pp. 8604–8.
   Google Scholar
• G. Hinton, and R. Salakhutdinov, “Discovering binary codes for documents by learning deep generative models,” Topics in Cognitive Science, Vol. 3, no. 1, pp. 74–91, Jan. 2011.
   PubMed Web of Science ®Google Scholar
• R. Salakhutdinov, and G. Hinton, “Semantic hashing,” International Journal of Approximate Reasoning, Vol. 50, no. 7, pp. 969–78, Jul. 2009.
   Web of Science ®Google Scholar
• N. Srivastava, R. Salakhutdinov, and G. E. Hinton, “Modeling documents with deep Boltzmann machines,” in Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, Bellevue, 2013, pp. 616–24.
   Google Scholar
• W. Fang, W. Pan, and Z. Cui, “View of MapReduce: Programming model, methods, and its applications”, IETE Technical Review, Vol. 29, no. 5, pp. 380–7, Sept. 2012.
   Web of Science ®Google Scholar