Lexical Richness and Text Length: An Entropy-based Perspective

ABSTRACT

Text length is a major concern in the measurement of lexical richness, and how lexical richness is affected by text length still remains open. The present study aims to explore the relation between text length and lexical richness from an entropy-based perspective. Results show a non-linear growth pattern of lexical richness by increasing text length. To be specific, lexical richness increases rapidly with shorter texts. It soon reaches a boundary point from which it stabilizes despite the continuous expansion of text length. The boundary point of the lexical richness by the Shannon estimation is around 1000 tokens and that by the Zhang estimation is lower and more varied, including 500, 800, and 1000 tokens. Such stability may be explained by the stabilization of word probability in the text.

Acknowledgments

This article is dedicated to Professor Fengxiang Fan and Professor Gabriel Altmann. The authors would like to thank the comments from the anonymous reviewers and the Editor. This work was supported by Double First Class University Plan, Huazhong University of Science and Technology.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental data

Supplemental data for this article can be accessed here.

Geolocation information

China

Additional information

Funding

This work was supported by Double First Class University Plan, Huazhong University of Science and Technology [N/A].