The limitations of deterministic Boltzmann machine learning

Abstract

The stochastic Boltzmann machine (SBM) learning procedure allows a system of stochastic binary units at thermal equilibrium to model arbitrary probabilistic distributions of binary vectors, but the inefficiency inherent in stochastic simulations limits its usefulness. By employing mean field theory, the stochastic settling to thermal equilibrium can be replaced by efficient deterministic settling to a steady state. The analogous deterministic Boltzmann machine (DBM) learning rule performs steepest descent in an appropriately defined error measure under certain circumstances and has been empirically shown to solve a variety of non-trivial supervised, input-output problems.

However, by applying ‘naive’ mean field theory to a finite system with non-random interactions, the true stochastic system is not well described, and representational problems result that significantly limit the situations in which the DBM procedure can be successfully applied. It is shown that the independence assumption is unacceptably inaccurate in multiple hidden layer configurations, thus accounting for the empirically observed failure of DBM learning in such networks. Further restrictions in network architecture are suggested that maximize the utility of the supervised DBM procedure, but its inherent limitations are shown to be quite severe. An analogous system based on the TAP equations is also discussed.