Empirical study on variational inference methods for topic models

Abstract

In topic modelling, the main computational problem is to approximate the posterior distribution given an observed collection. Commonly, we must resort to variational methods for approximations; however, we do not know which variational variant is the best choice under certain settings. In this paper, we focus on four topic modelling inference methods, including mean-field variation Bayesian, collapsed variational Bayesian, hybrid variational-Gibbs and expectation propagation, and aim to systematically compare them. We analyse them from two perspectives, i.e. the approximate posterior distribution and the type of -divergence; and then empirically compare them on various data-sets by two popular metrics. The empirical results are almost matching our analysis, where they indicate that CVB0 may be the best variational variant for topic models.

Keywords:

• Topic modelling
• variational methods
• variational distributions
• -divergence

Notes

No potential conflict of interest was reported by the authors.

1 http://www.cs.cmu.edu/~webkb/.

2 http://archive.ics.uci.edu/ml/datasets/Bag+of+Words.

3 http://qwone.com/~jason/20Newsgroups/.

4 On some data-sets, the test perplexities achieved by optimising are worse than those achieved by fixing hyperparameters. These results suggest that optimising does not always lead to a better solution (Asuncion et al., 2009).

Additional information

Funding

This work is supported by the National Natural Science Foundation of China [grant number 61602204], [grant number 61672261], [grant number 61472157].