Improving information-theoretic competitive learning by accentuated information maximization

Abstract

In this paper, we propose a new computational method for information-theoretic competitive learning. We have so far developed information-theoretic methods for competitive learning in which competitive processes can be simulated by maximizing mutual information between input patterns and competitive units. Though the methods have shown good performance, networks have had difficulty in increasing information content, and learning is very slow to attain reasonably high information. To overcome the shortcoming, we introduce the rth power of competitive unit activations used to accentuate actual competitive unit activations. Because of this accentuation, we call the new computational method “accentuated information maximization”. In this method, intermediate values are pushed toward extreme activation values, and we have a high possibility to maximize information content. We applied our method to a vowel–consonant classification problem in which connection weights obtained by our methods were similar to those obtained by standard competitive learning. The second experiment was to discover some features in a dipole problem. In this problem, we showed that as the parameter r increased, less clear representations could be obtained. For the third experiment of economic data analysis, much clearer representations were obtained by our method, compared with those obtained by the standard competitive learning method.

Keywords:

• Mutual information maximization
• Accentuated information maximization
• Competitive learning
• Winner-take-all
• r th power

Notes

Ryotaro Kamimura is a professor in Information Science Laboratory in Tokai University. He is currently working in the areas of neural networks and their application to problems in linguistics, cognitive science and multimedia.