Chinese Word Frequency Approximation Based on Multitype Corpora

Abstract

Due to the nature of Chinese, a perfect word-segmented Chinese corpus that is ideal for the task of word frequency estimation may never exist. Therefore, a reliable estimation for Chinese word frequencies remains a challenge. Currently, three types of corpora can be considered for this purpose: raw corpora, automatically word-segmented corpora, and manually word-segmented corpora. As each type has its own advantages and drawbacks, none of them is sufficient alone. In this article, we propose a hybrid scheme which utilizes existing corpora of different types for word frequency approximation. Experiments have been performed from statistical and application-oriented perspectives. We demonstrate that, compared with other schemes, the proposed scheme is the most effective one and leads to better word frequency approximation results.

Acknowledgements

This work is supported by the National Science Foundation of China under Grant No. 60873174, the National 863 High-Tech Project of China under Grant No. 2007AA01Z148 and the China–Germany (Tsinghua–Hamburg University) CINACS Program.

Notes

¹ICTCLAS 1.0: http://www.nlp.org.cn

² http://chasen.org/taku/software/CRF++/.

³The four types are Numbers, Dates (the Chinese characters for “day”, “month”, “year”, respectively), English letters and Others.