Information processing by a noisy binary channel

Abstract

We study the information processing properties of a binary channel receiving data from a Gaussian source. A systematic comparison with linear processing is made. A remarkable property of the binary sytem is that, as the ratio α between the number of output and input units increases, binary processing becomes equivalent to linear processing with a quantization output noise that depends on α. In this regime, which holds up to O(α⁻⁴), information processing occurs as if populations of α binary units cooperate to represent one α-bit output unit. Unsupervised learning of a noisy environment by optimization of the parameters of the binary channel is also considered.