The Sichel model and the mixing and truncation order

Abstract

The analysis of word frequency count data can be very useful in authorship attribution problems. Zero-truncated generalized inverse Gaussian–Poisson mixture models are very helpful in the analysis of these kinds of data because their model-mixing density estimates can be used as estimates of the density of the word frequencies of the vocabulary. It is found that this model provides excellent fits for the word frequency counts of very long texts, where the truncated inverse Gaussian–Poisson special case fails because it does not allow for the large degree of over-dispersion in the data. The role played by the three parameters of this truncated GIG-Poisson model is also explored. Our second goal is to compare the fit of the truncated GIG-Poisson mixture model with the fit of the model that results from switching the order of the mixing and truncation stages. A heuristic interpretation of the mixing distribution estimates obtained under this alternative GIG-truncated Poisson mixture model is also provided.

Keywords:

• categorical data
• generalized inverse Gaussian
• mixture model
• Poisson mixture
• stylometry
• truncated model
• truncated mixture
• word frequency

Acknowledgements

This work was funded in part by Grant No. MTM2006-09920 of the Ministerio de Ciencia y Tecnologia of Spain. The authors are very grateful for the suggestions made by Marta Pérez-Casany and by an anonymous referee that helped in improving this manuscript to a great extent.