Assessing text mining algorithm outcomes

ABSTRACT

There is a surge in the development of decision-oriented analysis tools intended to extract actionable information from text. These tools integrate various text-mining methods that were performance tested in a manner that was often biased toward the new system. Those tests primarily utilised descriptive measurement criteria and test datasets that are inconsistent with most business corpora. We propose and test a user-oriented judgment approach that allows testing under controlled customer-oriented corpora and generates effect size measures.  To illustrate the approach, customer relations data was analysed by latent semantic analysis and latent Dirichlet analysis with results evaluated by prospective business analysts. Reporting includes comparisons of results with published literature. While the research centres on the context-region text-mining systems, literature comparisons include word-embedding methods. The analysis concludes that none of the systems reviewed possess a repeatable statistical advantage over the others. Instead, distribution attributes, algorithm configuration, and the evaluation task drive results.

KEYWORDS:

• Text mining
• algorithm testing
• model development
• latent semantic analysis
• latent Dirichlet allocation

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Kakkonen et al. (2005, 2006) data had 2 large effects and 11 medium effect sizes that are suspect because the samples were exceptionally small (n < 150).

2. Hofmann (2001) reports a version of 1999a.

3. Hofmann (1999a) also reports and compares to cos+tf and cos+tfidf baselines which are not included here. Further, comparisons among the pLSA variants is possible, but were not computed.

4. Xu et al. (2003) also report mutual information.

5. Note that while we applying (1) and report p-values, those results alone are inappropriate because the sample sizes are 7,803 and 9,494. If the sample is reduced to 750 and assuming the same accuracy scores, the number of significant p-values is reduced to 39.

6. Effect size computations are not affected by sample size.