Macroscopic properties of the cost function of a feed-forward neural network prior to training

Abstract

By making simple assumptions regarding the nodal potentials we have been able to obtain analytic expressions for the mean and standard deviation of the cost-function values of a feed-forward multilayer network, with continuous activation units, prior to training. We have also obtained means of the derivatives, with respect to the weights and biases, of the cost function. The expressions have been used to obtain systematic estimates of the learning rate required for backpropagation training. The results are exemplified using an 8-3-8 encoder network.