Removing potential flat spots on error surface of multilayer perceptron (MLP) neural networks

Abstract

As the derivative of the sigmoid activation function approaches zero, the back propagation algorithm involves flat spots on the error surface of multilayer perceptron (MLP) neural networks, which means the hidden neurons of MLP were saturated. Flat spots can slow down the gradient search and hamper convergence. In this paper, we propose a grading technique to gradually level off the potential flat spots to a sloping surface in a look-ahead mode; and thereby progressively renew the saturated hidden neurons. We introduce a criterion to measure the saturation level of MLP, and then we modify the error function by using a proposed piecewise error function that switches between two cases, regarding the level of MLP saturation. These two cases include the standard error function, when MLP is not saturated, and the modified error function, when MLP is saturated. We recorded considerable improvements, especially in convergence rate and generalization, on the tested benchmark problems.

Keywords:

• backpropagation
• error surface
• flat spot
• local minimum
• saturated hidden neuron
• piecewise error function

2000 AMS Subject Classifications :

• 68T05
• 68Q32

Acknowledgements

The authors would like to thank the anonymous reviewers of the paper for their valuable comments.