A statistical, self-organizing learning system with validation

Abstract

In this paper, a new learning system called statistical, self-organizing learning system (SSOLS), which combines functional-link neural networks, statistical hypothesis testing and self-organization of number of enhancement nodes, is introduced. Its structure consists of two stages, a mapping stage and a learning stage. The input training vectors are initially mapped to the enhancement vectors in the mapping stage by multiplying with a random matrix, followed by pointwise nonlinear transformation. Starting with only one enhancement node, the enhancement layer incrementally adds an extra node in each iteration. In the learning stage, both the input vectors and the enhancement vectors are fed into a least squares learning module to obtain the estimated output vectors. This is made possible by choosing the output layer linear. The optimum dimension of the enhancement layer is determined by testing against a separate validation set. In this way, the number of enhancement nodes is also learned automatically. Two inclusions to the mapping stage, the t-test and the gradient descent update algorithm, further reduce the number of enhancement nodes required, resulting in a more compact network. The method is simple, fast to compute, and very suitable for many real-world applications.

Keywords:

• Functional-link networks
• Hypothesis testing
• Validation
• Self-organization
• Least squares learning
• t-test

Notes

*[email protected]

Okan K. Ersoy is a professor of electrical and computer engineering at Purdue University, School of Electrical and Computer Engineering.. His current research interests include statistical and computational intelligence, digital signal/image processing and recognition, transform and time-frequency methods, imaging, remote sensing, bioinformatics, diffractive optics, phased array systems, and distant learning.

He has published approximately 220 papers in his areas of research. He also holds 3 patents. He is a fellow of IEEE, and a fellow of the Optical Society of America.

Hoi-Ming Chi was born in Hong Kong, China, on August 1, 1977. He received the B.S. degree with high distinction in Electrical Engineering from the University of Virginia, Charlottesville, VA, in May 1998, the M.S.E.E. and Ph.D. degrees from Purdue University, West Lafayette, IN, in December 1999 and December 2003, respectively. He is currently working as a Post-Doctoral Research Associate in the Krannert School of Management and School of Electrical and Computer Engineering at Purdue University. His research interests include statistical signal processing, pattern recognition, and computational intelligence using neural networks and support vector machines with applications in bioinformatics, remote sensing and Six Sigma principles.

He is the recipient of 2003 Juran Fellowship by the Carlson School of Management at the University of Minnesota, and he received the Second Runner-Up Best Paper Award in Theoretical Developments in Computational Intelligence, ANNIE 2002 Conference, St. Louis, Missouri in November 2002.