An automatic filtering method for field association words by deleting unnecessary words

Abstract

Document classification and summarization are very important for document text retrieval. Generally, humans can recognize fields such as ⟨Sports⟩ or ⟨Politics⟩ based on specific words called Field Association (FA) words in those document fields. The traditional method causes misleading redundant words (unnecessary words) to be registered because the quality of the resulting FA words depends on learning data pre-classified by hand. Therefore recall and precision of document classification are degraded if the classified fields classified by hand are ambiguous. We propose two criteria: deleting unnecessary words with low frequencies, and deleting unnecessary words using category information. Moreover, using the proposed criteria unnecessary words can be deleted from the FA words dictionary created by the traditional method. Experimental results showed that 25% of 38 372 FA word candidates were identified as unnecessary and deleted automatically when the presented method was used. Furthermore, precision and F-measure were improved by 26% and 15%, respectively, compared with the traditional method.

Keywords:

• Field association words
• Document classification
• Unnecessary words
• Precision
• F-measure