A Statistical Study on Chinese Word and Character Usage in Literatures from the Tang Dynasty to the Present

Abstract

In this paper, we carried out a statistical analysis on the Chinese corpus in the Tang, Song, Yuan, Ming and Qing Dynasties, as well as in the modern time. We found that character and word frequencies change over time so that the word frequency always abides by the Zipf-Mandelbrot law p(r) = C(r + r ₀)^−β, while the character frequency follows the Menzerath-Altmann law P(r) = Ae ^−ar r ^−b . In the case of the character frequency distribution, the exponential property increases and the power-law feature declines as time passes by. We also found that more and more compound words were created since the Tang Dynasty. Single-character words show up unevenly in the whole word frequency distribution, with more of them concentrating in the earlier period and decaying exponentially.

Acknowledgments

We thank the referee for the careful review and the valuable comments, which provided insights that helped improve the paper. We also thank Mr. Yang Wang for his contribution to the literatures' collections. This work was supported by National Natural Science Foundation of China under Grants No. 61174165 and the Fundamental Research Funds for the Central Universities of China.