On the theory of flexible neural networks – Part I: a survey paper

ABSTRACT

Although flexible neural networks (FNNs) have been used more successfully than classical neural networks (CNNs) in many industrial applications, nothing is rigorously known about their properties. In fact they are not even well known to the systems and control community. In the first part of this paper, existing structures of and results on FNNs are surveyed. In the second part FNNs are examined in a theoretical framework. As a result, theoretical evidence is given for the superiority of FNNs over CNNs and further properties of the former are developed. More precisely, several fundamental properties of feedforward and recurrent FNNs are established. This includes the universal approximation capability, minimality, controllability, observability, and identifiability. In the broad sense, the results of this paper help that general use of FNNs in systems and control theory and applications be based on firm theoretical foundations. Theoretical analysis and synthesis of FNN-based systems thus become possible. The paper is concluded by a collection of topics for future work.

KEYWORDS:

• Controllability
• feedforward FNNs
• identifiability
• minimality
• observability
• recurrent FNNs
• universal approximation

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. The acronyms CNNs and FNNs have also been used to stand for cellular neural networks and feedforward neural networks, respectively. However, they seem superfluous and our usages sound better, since (i) they are most often referred to as recurrent neural networks (RNNs) and multi-perceptron/ multi-hidden-layer neural networks (MP/MHL NNs), respectively, and (ii) a bigger picture is presented in this work, see e.g. (Bavafa-Toosi, 2006). Nevertheless, if one wishes one can call CNNs e.g. ONNs, spelling out as ordinary neural networks – that is certainly a matter of taste.

2. Yet, there is another class of FNNs namely flexible bipolar radial basis neural networks (FBRBNNs) (Bavafa-Toosi & Ohmori, 2005) which will exclusively be studied elsewhere (see Section 5).

3. More precisely the output of a CNN is within ( − 1, 1). If it is needed that the output be outside this range, linear output scaling is performed. This is sometimes associated with linear input scaling – when the input data are also large. However, these tasks are automatically performed in a superior way – both linearly and nonlinearly – in FNNs through the adjustment of flexible parameters.

4. This is in sharp contrast with the existing literature on FNNs. So far only feedforward FNNs – not recurrent FNNs which are introduced and studied in this work – have been studied and this has been done only through simulation and the training method has been the common BP. This will be further clarified in the end of this section; read through.

5. The mathematical tools for the derivation of these results typically fall in five main categories: (a) Hahn–Banach Theorem, (b) Stone–Weierstrass theorem, (c) Radon transform, (d) Fourier analysis, and (e) Other techniques, like Spline Approximation and Staircase Functions (see Sanguineti, 2008 for more details).

6. It is noteworthy that the stability of the Tank–Hopfield equations is not yet fully resolved either, though many different conditions and characterizations do already exist. The interested reader is referred to Dhharani, Rakkiyappan, and Cao (2015; Wang, Zhang, & Shao, 2015, and the references therein).

7. Note that if $A_1\in R^{m_1\times n}$, $A_2\in R^{m_2\times n}$, and A = [A^T₁   A₂^T]^T, then $kerA=ker{A}_1\bigcap ker{A}_2$.

Additional information

Notes on contributors

Yazdan Bavafa-Toosi

Yazdan Bavafa-Toosi was born in Mashhad, Iran, on Sept. 22, 1974. He received B.Eng. and M.Eng. degrees in electrical power and control engineering from Ferdowsi University, Mashhad, and K.N. Toosi University of Technology, Tehran, Iran, in 1997 and 2000, respectively. He earned his Ph.D. degree in system design engineering (also known as systems and control) from Keio University, Yokohama, Japan, in 2006. His multi-disciplinary research spans systems and control theory and applications. Between and after his educations he has held various research and teaching positions in Germany, Japan, and Iran, and co-authored about 40 technical contributions. Dr Bavafa-Toosi has been a reviewer of some journals in the field of systems and control theory and applications in the past.