Hidden Markov model with Pitman-Yor priors for probabilistic topic model

Abstract.

Empirical studies of natural language have demonstrated that word frequencies follow power law distributions. However, standard statistical models often fail to capture this property. The Pitman-Yor process (PYP), a Bayesian non parametric model capable of generating power law distributions, has been widely used in probabilistic topic models to handle data with an infinite number of components. However, existing PYP topic models rarely account for the relationships between topics. Hidden Markov models (HMMs) are popular models for modeling topic relationships. To address this limitation, we propose a probabilistic topic model that combines HMM with Pitman-Yor priors. The posterior inference was performed by using variational Bayes methods. We applied our method to text categorization and compared it with two related topic models: the hidden Markov topic model and hierarchical PYP topic model.

Keywords:

• Pitman-Yor process
• power law
• hidden Markov model
• variational Bayes
• text categorization

Acknowledgments

The authors thank the editor, the associate editor, and a referee for their constructive comments and suggestions that helped to improve the paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. http://www.davidlewis.com/resources/testcollections/Reuters21578/

2. http://www.qwone.com/Jason/20newsgroups/

Additional information

Funding

Jianjie Guo’s research was partially supported by Shanghai Philosophy and Social Science Planning Project (Grant No. 2023BGL014). Wenchao Xu’s research was partially supported by National Natural Science Foundation of China (Grant Nos. 12101591 and 12201006).