Finding compact and sparse-distributed representations of visual images

Abstract

Some recent work has investigated the dichotomy between compact coding using dimensionality reduction and sparse-distributed coding in the context of understanding biological information processing. We introduce an artificial neural network which self-organizes on the basis of simple Hebbian learning and negative feedback of activation and show that it is capable both of forming compact codings of data distributions and of identifying filters most sensitive to sparse-distributed codes. The network is extremely simple and its biological relevance is investigated via its response to a set of images which are typical of everyday life. However, an analysis of the network's identification of the filter for sparse coding reveals that this coding may not be globally optimal and that there exists an innate limiting factor which cannot be transcended.