Mutual information maximization: models of cortical self-organization

Abstract

Unsupervised learning procedures based on Hebbian principles have been successful at modelling low-level feature extraction, but are insufficient for learning to recognize higher- order features and complex objects. In this paper we explore a class of unsupervised learning algorithms called Imax (Becker and Hinton 1992 Nature 355 161–3) that are derived from information-theoretic principles. The Imax algorithms are based on the idea of maximizing the mutual information between the outputs of different network modules, and are capable of extracting higher-order features from data. They are therefore well suited to modelling intermediate-to-high-level perceptual processing stages. We substantiate this claim with some novel results for two signal classification problems, as well as by reviewing some previously published results and several related approaches. Finally, Imax is evaluated with respect to computational costs and biological plausibility.