A brief survey on nonlinear control using adaptive dynamic programming under engineering-oriented complexities

ABSTRACT

Nonlinear dynamics is frequently encountered in practical applications. Adaptive dynamic programming (ADP), which is implemented via actor/critic neural networks with excellent approximation capabilities, is appropriate to be used in finding the solution for the control problem in the presence of known/unknown nonlinear dynamics. The objective of this paper is to introduce state-of-the-art ADP-based algorithms and survey the recent advances in the ADP-based control strategies for nonlinear systems with various engineering-oriented complexities. Firstly, the main motivation of the ADP-based algorithms is thoroughly discussed, and the way of implementing the ADP-based algorithms is highlighted. Then, the latest research results concerning ADP-based control policy design for nonlinear systems are reviewed in detail, Finally, we conclude the survey by outlining the challenges and possible research topics in the future.

KEYWORDS:

• Nonlinear dynamics
• adaptive dynamic programming
• nonlinear control
• neural networks
• engineering-oriented complexities

1. Introduction

In the past decades, optimal control has drawn particular research attention owing to its wide applications in various engineering fields (e,g., power systems, finance, ecology, and aerospace) (Ding et al., 2019, 2020; Wang et al., 2022). Optimal control aims at minimising (or maximising) a performance index function for the addressed system subject to some physical constraints such as some limitations on the states and the control inputs (Prokhorov et al., 1995). From the mathematical point of view, optimal control theory solves a class of functional extreme value problems with constraints (Lewis & Vrabie, 2009). It is well recognised that the optimal control policy is often parameterised by solving a Hamilton–Jacobi–Bellman (HJB) equation. For a linear system, the HJB equation can be simplified to one Riccati equation which can be easily solved. Unfortunately, when it comes to the optimal control problem for a nonlinear system, it is a challenging problem to develop effective strategies to solve the corresponding HJB equation subject to the effects of nonlinearities.

As is well known, reinforcement learning has been a promising research front in recent years (Deb et al., 2007). In reinforcement learning, an agent interacts with the external environment and improves the actions or control policies based on the reward/punishment stimulus from the external environment. One representative reinforcement learning approach is developed based on the actor/critic neural networks (Enns & Si, 2003; Xue et al. (2022)), and the implementation process can be divided into two steps: (1) policy evaluation: an actor neural network (ANN) exerts a control policy on the external environment, and then a critic neural network (CNN) evaluates the cost of this control policy (i.e. the response from the external environment); and (2) policy improvement: based on the policy evaluation, the control policy is improved with hope to achieve a better control performance compared with the previous control policy (Mohahegi et al., 2006). Such a two-step process is repeated until the cost of the control policy can satisfy specific standards (Beard & Saridis, 1998; Wang et al., 2009).

Inspired by reinforcement learning, the so-called adaptive dynamic programming (ADP), which was developed based on the reward/punishment mechanisms (Beard & Saridis, 1998), is regarded as an effective method to find solutions to optimal control problems for nonlinear systems. The ADP-based control scheme is implemented based on the actor/critic neural networks, and the core idea of the scheme is utilising two neural networks (i.e. CNN and ANN) to approximate the cost function and the optimal control policy, respectively (Liu et al., 2021; Vamvodakis & Lewis, 2010; Wang et al., 2009). To date, the ADP-related topics have received an increasing amount of research interest and the researchers have mainly focused on the following aspects: (1) the improvements of the ADP algorithms; (2) the design of the updating laws of neural network weights; (3) the convergence analysis of the ADP algorithms; and (4) the applications of the ADP algorithms to various practical control problems.

In engineering applications, complex external environments and physical constraints are often unavoidable, which makes the systems subject to various engineering-oriented complexities (Chen et al., 2022; Gao et al., 2021; Ma et al., 2022; Pang et al., 2021; Tian et al., 2010; Wei et al. 2022; Xu et al. 2020; Zhang et al., 2022; Zhang & Song, 2022; Zhang & Zhou, 2022). The engineering-oriented complexities can be generally categorised into network-oriented complexities and physics-oriented complexities (Ma et al., 2021; Shi et al., 2022). The network-oriented complexities, which are induced by the network-based transmission schemes, mainly contain communication protocols, event-triggered mechanisms, cyber-attacks, and so on Liu et al. (2022), Pinto et al. (2022), Zhang et al. (2022), and Zhu et al. (2022). The physics-oriented complexities are caused by the constraints of transmission networks and system components, and frequently-occurring physics-oriented complexities mainly include time delays, nonlinearities, sensor/actuator saturation constraints, faults, and so on L. Chen et al. (2021), Liu et al. (2022), Tan et al. (2022), Zhao et al. (2015), and Zhang et al. (2021). The engineering-oriented complexities may decrease the reliability of signal transmissions and complicate the system dynamics, and further lead to degradation or deterioration of the corresponding system performance (Liu et al., 2022; Qian et al., 2020; Wang et al., 2022b). Therefore, the effects of the underlying complexities should be fully taken into consideration in the controller design and performance analysis. So far, the control problem for systems with different engineering-oriented complexities have attracted considerable attention from both academic researchers and engineering practitioners (Ming et al., 2022; Xu et al., 2015).

Based on the above discussion, this paper aims to give a review of the existing results on the ADP-based control methods for nonlinear systems with various engineering-oriented complexities. The organisation of this paper is shown in Figure 1. In Section 2, the basic idea and the implementation process of the ADP-based algorithms are introduced in detail. In-depth overviews of the recent research advances related to the applications of the ADP-based algorithms under network-oriented complexities and physics-oriented complexities are given in Sections 3 and 4, respectively. Section 5 lists some challenging issues in the research field of ADP. Section 6 concludes the survey and points out some meaningful topics for future research.

Figure 1. The organisation structure of this survey.

2. ADP-Based algorithm

The ADP-based algorithm is a suboptimal control scheme proposed with actor/critic neural networks. In this section, to facilitate the understanding of the ADP-based algorithms, we will firstly elaborate the difficulties in dynamic programming for nonlinear systems, and then introduce the implementation process of the ADP-based algorithms.

2.1. The difficulties of dynamic programming for nonlinear systems

The key idea of dynamic programming is Bellman's optimality principle, i.e. the global optimum must be the local optimum (Lewis & Vrabie, 2009). By resorting to the principle, a multi-step optimisation problem can be transformed into many one-step optimisation problems, and in this way, the decision making (or control) process can be executed continuously. The details of dynamic programming are given as follows.

Consider a class of discrete-time systems with the following form: (1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) where $x(k)\in {\mathbb{R}}^n$, $u(k)\in {\mathbb{R}}^{\nu }$, and $y(k)\in {\mathbb{R}}^p$, respectively, represent the state vector, the control input, and the measurement output, $f(x(k))$ is a bounded nonlinear function, and $\omega (k)\in {\mathbb{R}}^q$ is the disturbance satisfying ${\omega }^T(k)\omega (k)\le {\overline{\psi }}^2$ and $\overline{\psi }$ is a known constant. A, B, C, D and E are known matrices with compatible dimensions. The cost function is defined as (2) $V(x(k))\triangleq \sum _{\iota =k}^{\mathrm{\infty }}z(x(\iota ),u(x(\iota ))),$(2) where $z(x(k),u(x(k)))$ denotes the utility function with $z(0,0)=0$, and $z(x(k),u(x(k)))$ is positive for any nonzero $x(k)$ and $u(x(k))$. According to the Bellman's optimality principle, the cost function (2(2) $V(x(k))\triangleq \sum _{\iota =k}^{\mathrm{\infty }}z(x(\iota ),u(x(\iota ))),$(2) ) can be rewritten as (3) $V(x(k))=z(x(k),u(x(k)))+V^{\ast }(x(k+1)).$(3) The target of optimal control is to minimise or maximise (3(3) $V(x(k))=z(x(k),u(x(k)))+V^{\ast }(x(k+1)).$(3) ) by designing a control policy. According to the HJB equation $\frac{\mathrm{\partial }z(x(k),u(x(k)))}{\mathrm{\partial }u(x(k))}+\frac{\mathrm{\partial }{V}^{\ast }(x(k+1))}{\mathrm{\partial }u(x(k))}=0,$we obtain the optimal cost function $V^{\ast }(x(k))=\underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}$and the optimal control policy (4) $\begin{array}{r}{u}^{\ast }(x(k))=\arg \underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}.\end{array}$(4) Based on the above discussion, it is obvious that finding the solution to the optimal control problem through dynamic programming follows a backward manner. Based on (4(4) $\begin{array}{r}{u}^{\ast }(x(k))=\arg \underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}.\end{array}$(4) ), $u(x(k+1))$ is a pre-condition to derive the optimal control policy $u(x(k))$, and the full knowledge of the system (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ) (e.g. A, B, and $f(x(k))$) is required. For a continuous-time system, the full knowledge of the system dynamics is necessary as well. Unfortunately, in engineering practice, it is difficult to obtain all the system information due to the complicated application environments, the limited economic budget, and the uncertainty of the system itself (Dong et al., 2008). It is often the case that the nonlinear function $f(x(k))$ is unknown. On the other hand, even if the nonlinear function $f(x(k))$ is known, the optimisation problem (4(4) $\begin{array}{r}{u}^{\ast }(x(k))=\arg \underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}.\end{array}$(4) ) needs to be solved in a backward manner, but the system state is calculated in a forward-in-time way, which leads to the well-known ‘curse of dimensionality’ and therefore poses huge challenges to the applications of dynamic programming in engineering practice (Yadav et al., 2007; Zhang et al., 2008). Considering the difficulties in implementing dynamic programming, the ADP, which is carried out in a forward-in-time manner, has been proposed as a proper approach to solve the optimal control problems for systems with nonlinear dynamics (Beard & Saridis, 1998; Mohahegi et al., 2006).

2.2. The implementation of the ADP-based algorithm

As shown in Figure 2, in light of the evaluation-improvement structure, two neural networks (i.e. the CNN and the ANN) are employed to approximate the cost function (2(2) $V(x(k))\triangleq \sum _{\iota =k}^{\mathrm{\infty }}z(x(\iota ),u(x(\iota ))),$(2) ) and the optimal control policy (4(4) $\begin{array}{r}{u}^{\ast }(x(k))=\arg \underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}.\end{array}$(4) ) in an online manner (Abu-Khalaf et al., 2006; Al-Tamimi et al., 2007, 2008; Balakrishnan & Biega, 1996; Cheng et al., 2007; Song et al., 2021). The weights of the two neural networks are updated adaptively, and in this way, the approximation errors of the two neural networks can be minimised (Havira & Lewis, 1972; Liu et al., 2005; Shervais et al., 2003; Tesauro, 1992).

Figure 2. The implementation of the ADP-based algorithm.

It is worth noting that there have been many researchers focusing on the improvement of ADP-based algorithms, and accordingly, many remarkable results have been reported (Ferrari et al., 2008; Kim & Lim, 2008; Padhi et al., 2006). The ADP-based algorithm to be introduced is proposed for the nonlinear discrete-time systems and belongs to the output feedback policy. The details of the algorithm are given as follows.

Consider a class of discrete-time systems in the form of (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ) and $f(x(k))$ denotes the bounded known/unknown nonlinear function.

With the help of the universal approximationstrategy, (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ) can be written as $\begin{array}{rl}x(k+1)& =Ax(k)+W_f{\delta }_f(x(k))+Bu(k)\\ & +E\omega (k)+{\xi }_f(k)\end{array}$where $W_f$, ${\delta }_f(x(k))$, and ${\xi }_f(k)$ denote the bounded optimal weight matrix, the activation function, and the approximation error of the nonlinear neural network, respectively.

For the purpose of generating desired state estimates, the following neural network-based observer is constructed: (5) $\begin{array}{rl}\hat{x}(k+1)& =A\hat{x}(k)+{\hat{W}}_f(k){\delta }_f(\hat{x}(k))+Bu(k)\\ & +L(y(k)-C\hat{x}(k))\end{array}$(5) where $\hat{x}(k)$ is the estimate of $x(k)$, ${\hat{W}}_f(k)$ represents the estimate of $W_f$ at the instant k, and L is the observer gain matrix to be designed.

In view of the gradient decent method, the updating rule of ${\hat{W}}_f(k)$ is chosen as (6) $\begin{array}{rl}& {\hat{W}}_f(k+1)\\ & ={\hat{W}}_f(k)+{\theta }_f((y(k+1)-C\hat{x}(k+1)){\delta }_f^T(\hat{x}(k)))\end{array}$(6) where ${\theta }_f$ is the tuning scalar of the nonlinear neural network.

The cost function (2(2) $V(x(k))\triangleq \sum _{\iota =k}^{\mathrm{\infty }}z(x(\iota ),u(x(\iota ))),$(2) ) and the control policy (4(4) $\begin{array}{r}{u}^{\ast }(x(k))=\arg \underset{u(x(k))}{min}\{z(x(k),u(x(k)))+V^{\ast }(x(k+1))\}.\end{array}$(4) ), which need to be approximated by the CNN and the ANN, respectively, are written as $V(x(k))=W_v^T{\delta }_v(x(k))+{\xi }_v(k)$and $u(k)=u(x(k))=W_u^T{\delta }_u(x(k))+{\xi }_u(k)$where $W_v$, ${\delta }_v(x(k))$ and ${\xi }_v(k)$ represent the desired weight matrix, the activation function, and the approximate error of the CNN, respectively. $W_u$, ${\delta }_u(x(k))$ and ${\xi }_u(k)$ denote the desired weight matrix, the activation function, and the approximate error of the ANN, respectively. The above-mentioned variables are all assumed to be bounded.

In light of the universal approximation property, we utilise neural networks to approximate $V(x(k))$ and $u(x(k))$, i.e. ${\hat{V}}_i(\hat{x}(k))={\hat{W}}_{v,i}^T(k){\delta }_v(\hat{x}(k))$and (7) ${\hat{u}}_i(k)={\hat{u}}_i(\hat{x}(k))={\hat{W}}_{u,i}^T(k){\delta }_u(\hat{x}(k))$(7) where i is the iteration index, and ${\hat{W}}_{v,i}(k)$ (${\hat{W}}_{u,i}(k)$, respectively) is the estimate of $W_v$ ($W_u$, respectively) at iteration i of time instant k. By resorting to the distinctive learning ability of neural networks, the cost function and the solution of the HJB equation are approximated in an iterative way, and the iteration will continue until ${\hat{V}}_{i+1}(\hat{x}(k))={\hat{V}}_i(\hat{x}(k))$.

Defining $r_{v,i}(k)$ as the residual error which is used to evaluate the approximation accuracy of the CNN, we have $\begin{array}{rl}{r}_{v,i}(k)& \triangleq z(\hat{x}(k),\hat{u}(k))+{\hat{W}}_{v,i}^T(k){\delta }_v(x(k+1))\\ & -{\hat{W}}_{v,i}^T(k){\delta }_v(\hat{x}(k))\end{array}$By minimising $\frac{1}{2}{r}_{v,i}^T(k)r_{v,i}(k)$ via the gradient decent approach, we can obtain the following updating law for the weight of the CNN (8) ${\hat{W}}_{v,i+1}(k)={\hat{W}}_{v,i}(k)+{\theta }_v({\delta }_v(\hat{x}(k))-{\delta }_v(x(k+1)))r_{v,i}^T(k)$(8) where ${\theta }_v$ is the CNN tuning scalar.

Next, according to the HJB equation, we have $\frac{\mathrm{\partial }z(\hat{x}(k),{\hat{u}}_i(k))}{\mathrm{\partial }{\hat{u}}_i(k)}=-\frac{\mathrm{\partial }{\hat{W}}_{v,i}^T(k){\delta }_v(x(k+1))}{\mathrm{\partial }{\hat{u}}_i(k)}$Define $h({\hat{u}}_i^{\ast }(k))\triangleq \frac{\mathrm{\partial }z(\hat{x}(k),{\hat{u}}_i(k))}{\mathrm{\partial }{\hat{u}}_i(k)}$ and assume $h({\hat{u}}_i^{\ast }(k))$ is an invertible function. Then, the suboptimal optimal policy can be expressed as $\begin{array}{rl}{\hat{u}}_i^{\ast }(k)& =h^{-1}\left(\frac{\mathrm{\partial }z(\hat{x}(k),{\hat{u}}_i(k))}{\mathrm{\partial }{\hat{u}}_i(k)}\right)\\ & =h^{-1}(-B^T\mathrm{\nabla }{\delta }_v^T(x(k+1)){\hat{W}}_{v,i}(k))\end{array}$where $\mathrm{\nabla }{\delta }_v(x(k+1))\triangleq \frac{\mathrm{\partial }{\delta }_v(x(k+1))}{\mathrm{\partial }{\hat{u}}_i(k)}$. Similarly, define $r_{u,i}(k)\triangleq {\hat{u}}_i(k)-{\hat{u}}_i^{\ast }(k)$ as the control input error. By minimising $\frac{1}{2}{r}_{u,i}^T(k)r_{u,i}(k)$, we can obtain the updating law for the weight of the ANN (9) ${\hat{W}}_{u,i+1}(k)={\hat{W}}_{u,i}(k)-{\theta }_u{\delta }_u(\hat{x}(k))r_{u,i}^T(k)$(9) where ${\theta }_u$ is the ANN tuning scalar.

The convergence of the introduced ADP-based algorithm has been proved in Al-Tamimi et al. (2008). Furthermore, in Ding et al. (2019), the stability of the nonlinear system (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ) has been analysed, and the boundedness has been proved with respect to the estimation error of the system state, the nonlinear neural network weights, and the CNN/ANN weights.

3. ADP-based algorithms under network-oriented complexities

In recent years, the network-based transmission technique has found successful applications in a large number of modern industrial systems due to its distinctive advantages in saving economic costs, reducing implementation difficulty, and decreasing hardwiring (Li et al., 2022). Nevertheless, compared with traditional point-to-point transmission strategies, the network-based transmission schemes give rise to some network-oriented complexities (e.g. communication protocols, event-triggered mechanisms, signal quantisation, packet losses, and cyber-attacks) owing to the unavoidable vulnerability and restricted bandwidth of transmission networks (Jiang et al., 2022; Wang et al., 2022a). The network-oriented complexities would inevitably degrade or even destroy the control/estimation performance if not handled properly. In this section, we focus on the summary of recent advances in ADP-based control policy design under various network-oriented complexities, and the considered network-oriented complexities mainly include communication protocols, event-triggered mechanisms, and cyber-attacks. To facilitate understanding, in each subsection, we will firstly give a brief introduction of the corresponding network-oriented complexities, and then present the relevant literature review.

3.1. ADP-Based algorithms under the scheduling of communication protocols

Network congestion, which is an inevitable phenomenon in many industrial applications, mainly results from the frequent signal transmissions between system components (i.e. sensors, estimators, and controllers) over a shared communication network with limited bandwidth (Dong et al., 2022; Li et al., 2022; Shen et al., 2022; Yao et al., 2022). Correspondingly, various communication protocols are often adopted to govern the signal transmission process to prevent collisions in the signal transmission process (Liu et al., 2022; Xu et al., 2022). Under the scheduling of communication protocols, only one sensor node is permitted to transmit the current signal at each time instant (Xu et al., 2021). In engineering practice, the widely adopted communication protocols include, but are not limited to, the Round-Robin communication protocol, the stochastic communication protocol, and the try-once-discard communication protocol (Xu et al., 2022).

Before proceeding further, let us utilise a simple example to illustrate the effects of the protocol scheduling. Define $\sigma (k)\in \{1,2,\dots ,N\}$ as the node accessing to the communication network selected by the protocol, $y_j(k)(j\in \{1,2,\dots ,N\})$ as the measurement signal of the jth sensor node, and $\overline{y}(k)$ as the signal collected at the receiving end. The scheduling behaviours of the communication protocols can be characterised by the following transmission model: (10) $\overline{y}(k)=\mathrm{\Theta }(k)y(k)+(I-\mathrm{\Theta }(k))\overline{y}(k-1)$(10) where $\begin{array}{rl}{\mathrm{\Theta }}_j(k)& \triangleq \theta (\sigma (k)-j)I,\\ \mathrm{\Theta }(k)& \triangleq \mathrm{diag}\{{\mathrm{\Theta }}_1(k),{\mathrm{\Theta }}_2(k),\dots ,{\mathrm{\Theta }}_N(k)\},\\ y(k)& \triangleq {\left[\begin{array}{cccc}{y}_1^T(k)& y_2^T(k)& \cdots & y_N^T(k)\end{array}\right]}^T,\end{array}$and $\theta (\sigma (k)-j)\in \{0,1\}$ denotes the Kronecker delta function with the following form $\theta (\sigma (k)-j)=\{\begin{array}{ll}1,& \mathrm{if}\sigma (k)=j\\ 0,& \mathrm{if}\sigma (k)\ne j\end{array}.$$\sigma (k)$ is a time-varying variable representing the scheduling behaviour of the considered communication protocols. The system under communication protocol scheduling can be regarded as a switched system, and the switching behaviour is dependent on $\sigma (k)$. Hence, to design the control strategy and analyse the performance for the nonlinear systems under communication protocols scheduling, we need to modify the traditional ADP-based methods introduced in Subsection 2.2. Obviously, the time-varying nature of $\sigma (k)$ complicates the system dynamics and poses significant challenges to the design of state estimation scheme (5(5) $\begin{array}{rl}\hat{x}(k+1)& =A\hat{x}(k)+{\hat{W}}_f(k){\delta }_f(\hat{x}(k))+Bu(k)\\ & +L(y(k)-C\hat{x}(k))\end{array}$(5) ), control strategy (7(7) ${\hat{u}}_i(k)={\hat{u}}_i(\hat{x}(k))={\hat{W}}_{u,i}^T(k){\delta }_u(\hat{x}(k))$(7) ), and the updating laws of neural network weights (i.e. (6(6) $\begin{array}{rl}& {\hat{W}}_f(k+1)\\ & ={\hat{W}}_f(k)+{\theta }_f((y(k+1)-C\hat{x}(k+1)){\delta }_f^T(\hat{x}(k)))\end{array}$(6) ), (8(8) ${\hat{W}}_{v,i+1}(k)={\hat{W}}_{v,i}(k)+{\theta }_v({\delta }_v(\hat{x}(k))-{\delta }_v(x(k+1)))r_{v,i}^T(k)$(8) ), and (9(9) ${\hat{W}}_{u,i+1}(k)={\hat{W}}_{u,i}(k)-{\theta }_u{\delta }_u(\hat{x}(k))r_{u,i}^T(k)$(9) )).

So far, the ADP-based control issues for nonlinear systems under communication protocol scheduling are still in their infancy despite their clear practical significance. Under the scheduling of the stochastic communication protocol, N sensor nodes are stochastically scheduled to transmit the measurement signals to the estimator. For example, the discrete-time Markov chain has been utilised to describe $\sigma (k)$ in Ding et al. (2019) and the system has been written as a stochastic parameter-switching system. Then, a novel control strategy has been proposed in Ding et al. (2019) for the nonlinear discrete-time systems under the stochastic communication protocol scheduling. The results in Ding et al. (2019) have been extended in Wang et al. (2021), where the ADP-based approach has been employed to address the optimal control issue for the nonlinear discrete-time systems with stochastic communication protocols and input constraints. Different from that of the stochastic communication protocol, the scheduling behaviour of the Round-Robin communication protocol can be modelled by a periodic function, and therefore the system can be seen as a periodic parameter-switching system ($\sigma (k)=j$ if $\mathrm{mod}(k-j,N)=0$, where $\mathrm{mod}(k-j,N)$ is the function calculating the non-negative reminder). In Ding et al. (2019), the ADP-based optimal output feedback controller has been designed for the nonlinear discrete-time systems with Round-Robin protocol.

3.2. ADP-Based algorithms under event-triggered mechanisms

In engineering practice, the sensors are usually equipped with energy-limited batteries, and signal transmissions between system components are usually the main cause of energy consumption (Cui et al., 2021; Gu et al., 2018; Shen et al., 2021; Wang & Wang, 2022; Xue et al. 2022; Zhu et al., 2022). For the purpose of energy saving, event-triggered mechanisms have been broadly adopted in engineering (Liu et al., 2022; Ma et al., 2021; Wang et al., 2022; Xie et al., 2017). Under event-triggered mechanisms, the signal transmissions between system components are activated only when the pre-specified triggering condition is satisfied. The triggering condition is dependent on the triggering threshold and the system state (Li et al., 2021). The triggering threshold determines when and how the triggering occurs. The triggering threshold should be determined to achieve the balance between the transmission-induced resource assumption and the control/estimation performance. Based on the characteristics of the triggering thresholds, the existing event-triggered mechanisms can be divided into two categories, i.e. the static event-triggered mechanisms (SETMs) with fixed triggering thresholds and the dynamic event-triggered mechanisms (DETMs) with dynamically adjusted thresholds.

Now, let us take a brief look on the SETMs. Define $k_t^e$ as the tth triggering instant and $\overline{h}(\pi ,\gamma (k))$ as the static event generator as follows: $\overline{h}(\pi ,\gamma (k))\triangleq {\gamma }^T(k)\gamma (k)-\pi$where $\gamma (k)\triangleq y(k_t^e)-y(k)(k_t^e\le k<k_{t+1}^e)$ and $\pi >0$ is a known constant. For $k_t^e\le k<k_{t+1}^e$, the scheduling behaviour is governed by the following model (11) $\overline{y}(k)=\overline{\mathrm{\Theta }}(k)y(k)+(1-\overline{\mathrm{\Theta }}(k))\overline{y}(k_t^e)$(11) where $\overline{\mathrm{\Theta }}(k)=\{\begin{array}{ll}1,& \mathrm{if}\overline{h}(\pi ,\gamma (k))>0\\ 0,& \mathrm{if}\overline{h}(\pi ,\gamma (k))\le 0\end{array}$In the static event-triggered case, owing to the introduction of $\gamma (k)$, the system dynamics becomes more complex than (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ), which presents significant challenges to the design of the estimators (5(5) $\begin{array}{rl}\hat{x}(k+1)& =A\hat{x}(k)+{\hat{W}}_f(k){\delta }_f(\hat{x}(k))+Bu(k)\\ & +L(y(k)-C\hat{x}(k))\end{array}$(5) ), the controllers (7(7) ${\hat{u}}_i(k)={\hat{u}}_i(\hat{x}(k))={\hat{W}}_{u,i}^T(k){\delta }_u(\hat{x}(k))$(7) ) as well as the updating law of the neural network weights (i.e. (6(6) $\begin{array}{rl}& {\hat{W}}_f(k+1)\\ & ={\hat{W}}_f(k)+{\theta }_f((y(k+1)-C\hat{x}(k+1)){\delta }_f^T(\hat{x}(k)))\end{array}$(6) ), (8(8) ${\hat{W}}_{v,i+1}(k)={\hat{W}}_{v,i}(k)+{\theta }_v({\delta }_v(\hat{x}(k))-{\delta }_v(x(k+1)))r_{v,i}^T(k)$(8) ), and (9(9) ${\hat{W}}_{u,i+1}(k)={\hat{W}}_{u,i}(k)-{\theta }_u{\delta }_u(\hat{x}(k))r_{u,i}^T(k)$(9) )).

Compared with the SETMs, the DETMs are more preferred in practical engineering since they can dynamically adjust the triggering thresholds and make an adequate trade-off between the energy consumption and the system performance. Define $\tilde{h}(\pi ,\gamma (k),\eta (k))$ as the dynamic event generator described by $\tilde{h}(\pi ,\gamma (k),\eta (k))\triangleq {\gamma }^T(k)\gamma (k)-\frac{1}{\zeta }\eta (k)-\pi ,$where $\zeta >0$ and $\pi >0$ are two known constants, and $\eta (k)$ denotes the internal dynamical variable in the following form $\eta (k+1)=\{\begin{array}{ll}\tau \eta (k)+\pi -{\gamma }^T(k)\gamma (k),& \mathrm{if}k>0\\ \eta (0),& \mathrm{if}k=0\end{array}$where $\tau \in (0,1)$ and $\eta (0)\ge 0$. The scheduling of DETMs can be represented as follows (12) $\overline{y}(k)=\tilde{\mathrm{\Theta }}(k)y(k)+(1-\tilde{\mathrm{\Theta }}(k))\overline{y}(k_t^e)$(12) where $\tilde{\mathrm{\Theta }}(k)=\{\begin{array}{ll}1,& \mathrm{if}\tilde{h}(\pi ,\gamma (k),\eta (k))>0\\ 0,& \mathrm{if}\tilde{h}(\pi ,\gamma (k),\eta (k))\le 0\end{array}.$Different from $\overline{h}(\pi ,\gamma (k))$, $\tilde{h}(\pi ,\gamma (k),\eta (k))$ is dependent on $\eta (k)$, which is an internal dynamic variable reflecting the adjustment of the triggering thresholds. In the dynamic event-triggered case, in addition to π and $\gamma (k)$, the parameter τ and the internal dynamic variable $\eta (k)$ should also be taken into consideration in the controller design and performance analysis.

To date, considerable research effort has been devoted to the ADP-based control policy design for the networked nonlinear systems with SETMs. By the ADP-based method, the optimal control problem has been handled in Zhu et al. (2017) for a class of nonlinear continuous-time systems, and a new static event-triggered condition has been proposed to reduce computational complexity. The event-triggered strategy in Zhu et al. (2017) has been extended to the nonlinear continuous-time system with input constraints in Song and Liu (2020). For the multi-player continuous-time systems with SETM, the optimal tracking control issue has been converted to the optimal regulation problem, which has been investigated by means of the ADP-based approach in Zhang et al. (2022). In Wu and Wang (2021), the ADP-based suboptimal tracking control approach has been proposed for a class of nonlinear continuous-time systems with partial known measurement signals and SETMs. For the path planning issue for unmanned vehicles, a SETM-based model predictive ADP has been implemented in Hu et al. (2021). For the nonlinear non-affine systems with constrained inputs, a SETM-based control method has been designed in Zhang et al. (2021). In the temperature field, for a special class of discrete-time systems with distributed parameters, the SETMs-based optimal control problem has been addressed by resorting to the ADP-based method in N. Chen et al. (2021). In Deng et al. (2022), an ADP-based control strategy has been proposed for the quarter-car electromagnetic active suspension systems under SETMs in both sensors-to-observer and controller-to-actuator channels.

When it comes to systems under DETMs, the optimal control problem in the ADP framework has also been well investigated. In Han et al. (2021), the DETM-based controller design issue has been considered for the switched affine discrete-time systems with constrained control inputs. Under the DETM, a novel adaptive learning structure has been proposed and the optimal control issue has been studied in Mu et al. (2022) for a class of nonlinear discrete-time systems with two controllers. Via the value-iterative ADP-based algorithm, the DETM-based tracking control strategy has been proposed in Wang et al. (2021) with application to the waste water treatment. The ADP-based optimal control problem has been studied in Wang et al. (2022) for a class of nonlinear discrete-time systems under cyber-attacks and DETMs. In Zhao et al. (2022), the near-optimal consensus tracking control issue has been discussed for the discrete-time multi-agent systems, and the dynamic event-triggered condition has been designed for each agent. By means of the ADP-based approach, the DETM-based stochastic optimal regulation issue has been solved in Ming et al. (2022) for the nonlinear networked continuous-time systems with delayed control inputs.

3.3. ADP-Based algorithms under cyber-attacks

As the communication technologies advances, the digital-network-based communication has been playing a more and more crucial role in the past several decades. Considering the inherent and longstanding openness of networks, the systems adopting network-based transmission schemes are especially vulnerable to cyber-attacks, which may lead to performance degradation or even instability (Chen et al., 2022; Niu et al., 2020; Wang et al., 2022). Generally speaking, the cyber-attacks for networked systems can be classified into three types, i.e. denial-of-service (DoS) attacks, deception attacks, and replay attacks (Gao et al., 2022; Tao et al., 2022; Xiao et al., 2021). DoS attacks block the signal transmissions between system components by congesting the transmission channels (Zhao et al., 2022). In the case of deception attacks, the measurements/control inputs are maliciously modified (Cui et al., 2021; He et al., 2020). Replay attacks aim to repeat or delay the transmitted signals (Zhu & Martinez, 2014).

Due to its opening-up characteristic, an NCS could be susceptible to cyber-attacks leading to performance degradation or even system instability. Accordingly, the cyber-security research has been gaining growing popularity as to how to guarantee satisfactory performance under the effects of possible cyber-attacks (Cheng et al., 2022; Yuan et al., 2020; Zhu et al., 2021).

Now let us take DoS attacks as an example to briefly introduce the effects of the cyber-attacks. Denoting $k_s^a$ as the sth attack instant for the networked nonlinear systems subject to DoS attacks, we have the following model (13) $\overline{y}(k)=\vartheta (k)y(k)+(1-\vartheta (k))\overline{y}(k-1)$(13) where $\vartheta (k)$ reflects the DoS attacks featured by $\vartheta (k)=\{\begin{array}{ll}1,& \mathrm{if}k\ne k_s^a,\\ 0,& \mathrm{if}k=k_s^a.\end{array}$In the existing literature, Bernoulli distributedsequence and Markov chain have usually been adopted to characterise the occurrence of the DoS attacks (Yuan et al., 2020). Obviously, the attack-induced switch parameter $\vartheta (k)$ complicates the system dynamics and thereby brings in difficulties to the design of the estimator (5(5) $\begin{array}{rl}\hat{x}(k+1)& =A\hat{x}(k)+{\hat{W}}_f(k){\delta }_f(\hat{x}(k))+Bu(k)\\ & +L(y(k)-C\hat{x}(k))\end{array}$(5) ), the control strategy (7(7) ${\hat{u}}_i(k)={\hat{u}}_i(\hat{x}(k))={\hat{W}}_{u,i}^T(k){\delta }_u(\hat{x}(k))$(7) ), and the selection of updating laws of the neural network weights (i.e. (6(6) $\begin{array}{rl}& {\hat{W}}_f(k+1)\\ & ={\hat{W}}_f(k)+{\theta }_f((y(k+1)-C\hat{x}(k+1)){\delta }_f^T(\hat{x}(k)))\end{array}$(6) ), (8(8) ${\hat{W}}_{v,i+1}(k)={\hat{W}}_{v,i}(k)+{\theta }_v({\delta }_v(\hat{x}(k))-{\delta }_v(x(k+1)))r_{v,i}^T(k)$(8) ), and (9(9) ${\hat{W}}_{u,i+1}(k)={\hat{W}}_{u,i}(k)-{\theta }_u{\delta }_u(\hat{x}(k))r_{u,i}^T(k)$(9) )).

So far, in the ADP framework, systems subject to cyber-attacks have just received some initial research attention. In Yang et al. (2018), a security control strategy has been developed for the nonlinear discrete-time systems in the presence of actuator saturation and DoS attacks. With ADP-based methods, the structure optimisation problem has been solved for the multi-machine power system under DoS attacks (Zhu et al., 2021). In Cheng et al. (2022), a distributed secondary control strategy has been proposed for the battery energy storage system subject to DoS attacks. Inspired by the basic idea of the two-player zero-sum game, the optimal control issue has been addressed for the nonlinear continuous-time systems in Lian et al. (2021), where three neural networks have been utilised to approximate the critic, the actor, and the deception attack, respectively. The ADP-based resilient control issue has been investigated in Huang and Dong (2021) for the nonlinear continuous-time systems, where the control signals have been transmitted via a communication link under deception attacks.

4. ADP-based algorithms under physics-oriented complexities

In engineering practice, physics-oriented complexities are frequently encountered due mainly to the complex external environments and the physical limitations of system components (Han et al., 2021). Representative physics-oriented complexities include faults, time delays, saturation constraints, and nonlinearities (Qu et al., 2021; Sun et al., 2022). If not fully considered in the design of the control strategies, the influences of the physics-oriented complexities may deteriorate the system performance or even result in instability (Sun & van Kampen, 2021; Zhang et al., 2021). In this section, recent research advances are reviewed with regard to the ADP-based control policy design under various physics-oriented complexities. The considered physics-oriented complexities include time delays and saturation constraints. In each subsection, the engineering backgrounds will be briefly introduced at first for the corresponding physics-oriented complexities, and then the relevant literature reviews will be given.

4.1. ADP-Based algorithms under time delays

Time delay, which results from the finite signal transmission speed and the vulnerability of the networks, is a ubiquitous network-induced phenomenon (Jiang et al., 2021; Ning et al., 2021; Qian et al., 2021). Time delays can be categorised into invariant time delays, time-varying delays, and so on Qian et al. (2019) and Zhang et al. (2021). The existence of time delays may degrade the control/estimation performance of the addressed system (Qian et al., 2021).

Now, let us briefly introduce the effects of the time delays, where the invariant time delay is taken as an example. The time-delayed systems are represented by the following model: (14) $\begin{array}{rl}x(k+1)& =Ax(k)+A_{d}x(k-d)+Bu(x(k))\\ & +f(x(k))+E\omega (k)\end{array}$(14) where d represents the known invariant time delay. Obviously, the delayed state $x(k-d)$ makes the system model more complex compared with traditional model (1(1) $\begin{array}{r}\{\begin{array}{rl}& x(k+1)=Ax(k)+f(x(k))+Bu(k)+E\omega (k)\\ & y(k)=Cx(k)+D\omega (k)\end{array}\end{array}$(1) ) of the nonlinear system. As a result, when the control/estimation problems need to be solved for (14(14) $\begin{array}{rl}x(k+1)& =Ax(k)+A_{d}x(k-d)+Bu(x(k))\\ & +f(x(k))+E\omega (k)\end{array}$(14) ), $x(k-d)$ should be fully taken into consideration. In this case, the ADP-based algorithm has been developed for the time-delayed systems in Ni et al. (2015).

So far, the ADP-based control methodologies have been thoroughly investigated for time-delayed systems. In Zhang et al. (2011), the initial research has been reported on the neural network-based optimal control problem for a class of nonlinear discrete-time systems with delayed inputs and delayed states. In Song et al. (2013), the multi-objective optimal control issue has been investigated for the discrete-time systems with invariant time delays. A dual iterative ADP-based algorithm has been proposed in Wei et al. (2013) for a special class of nonlinear discrete-time systems in consideration of distributed but multiple state/input delays. In Xu et al. (2014), for the nonlinear discrete-time systems with packet losses (in both sensor-to-controller and controller-to-actuator channels) and state delays, the neural network-based stochastic optimal control has been investigated in the framework of zero-sum games. In view of the ADP-based method, the distributed optimal consensus control problem has been studied in Zhang et al. (2018) for a class of multi-agent systems, where the considered time-varying input delays have been dependent on the sampling period. By employing the value-iterative ADP-based algorithm, the optimal control problem has been discussed for nonlinear continuous-time systems with invariant state and input delays in Shi et al. (2020). In Zhu et al. (2021), by adopting the ADP-based approach, the optimal regulation strategy has been established for a class of nonlinear continuous-time systems, where both state and input delays have been considered.

4.2. ADP-Based algorithms under saturation constraints

Because of the harsh external environment and limited economic budgets, system components (e.g. sensors and actuators) in engineering practice are usually subject to physical limitations and influences of protective equipment, which may lead to various saturation constraints (Bao et al., 2022; Chen et al., 2020; Hou et al., 2022; Yuan et al., 2019). The saturation constraints include input saturation, state saturation, etc (Chen & Wang, 2021).

Let us discuss the impacts of the saturation constraints with input saturation phenomenon as the example. Define $u_m(k)$ as the mth element of $u(k)$ and r as the saturation bound. The control input $u(k)$ belongs to ${\mathrm{\Omega }}_u=\{u(k)|u(k)={\left[\begin{array}{cccc}{u}_1^T(k)& u_2^T(k)& \cdots & u_{\nu }^T(k)\end{array}\right]}^T\in {\mathbb{R}}^{\nu }\}$where $u_m(k)\le r$. It is easy to see that the saturation phenomenon introduces an extra nonlinearity to the system dynamics, thereby making the analysis and synthesis of the addressed systems more difficult (He & Jagannathan, 2005; Liang et al., 2020; Rizvi & Lin, 2022).

The ADP-based control problems for the nonlinear systems with saturation constraints have been extensively studied, and much progress has been made on this topic. For the nonlinear continuous-time systems with uncertain interconnections and constrained control inputs, the distributed optimal control policy design problem has been addressed in Tan (2018) via the ADP-based method. Considering the influences of unknown input constraints, the stabilising control issue has been investigated in Zhao et al. (2018). For the mobile manipulation systems with input constraints, an ADP-based hierarchical motion/force control framework has been established in Zhao et al. (2019). By resorting to integral reinforcement learning techniques, the optimal control problem has been solved for the nonlinear continuous-time systems with multiple controllers under input constraints in Ren et al. (2020). In Khankalantary et al. (2020), the robust tracking control problem has been investigated for a class of nonlinear multi-agent systems with input constraints. For the wireless sensor networks with constrained inputs, the event-triggered adaptive control problem has been investigated in the framework of nonzero-sum games in Su et al. (2020). In Ding et al. (2020), the ADP-based consensus control problem has been investigated for the multi-agent systems with input saturation.

In the last two years, the design of ADP-based control strategy has received particularly considerable research attention for the systems with saturation constraints. The decentralised stabilisation issue has been studied for the nonlinear interconnected systems with input constraints in Yang et al. (2021). For the nonlinear systems with constrained inputs, an event-triggered adaptive critic learning scheme has been proposed with a discounted cost function in Yang and Wei (2021). In Liu et al. (2021), by adopting the basic idea of zero-sum games, an adaptive critic has been established. Via integral reinforcement learning techniques, the optimal tracking control problem has been tackled for the nonlinear continuous-time systems with input constraints (Mishra & Ghosh, 2022). The adaptive integrated guidance and control issues have been studied for the missile interception systems in Zhang and Yi (2021), where both multiple states and input constraints have been considered. An optimal controller has been designed for the nonlinear systems with state saturation in Xu et al. (2022), where the control barrier function has been considered in the selection of the utility function.

5. Some challenging problems

In the past few decades, significant progress has been made in the research related to the applications of ADP to various systems such as multi-agents, networked systems, nonlinear systems, and affine systems. Many remarkable research results have been published on the analysis, synthesis, and applications of ADP-based algorithms. In this section, some ADP-related challenging issues are listed as follows.

1. An underlying assumption in most ADPresearches is that the weights of the neural networks, the activation function of the neural networks, and the approximate error of the neural networks are bounded, which is of vital importance for the theoretical analysis of the ADP-based algorithms. Nevertheless, it is usually difficult to acquire the accurate bounds in practical applications, which greatly restricts the applications of the ADP-based approaches.

2. To guarantee the convergence of the ADP-based algorithm, the initial control input of the iterative ADP-based algorithm should be admissible. Unfortunately, it may be time-consuming to find a satisfying initial control input for a complex system, which brings significant difficulties to the applications of the ADP-based algorithms in engineering practice.

3. The online implementation of the ADP-based algorithm unavoidably introduces a heavy computing burden, especially when the dimension of the system state is high. It is important but challenging to develop an effective and efficient weight update algorithm in the ADP-based methods.

6. Conclusions and future works

An in-depth review has been presented with respect to the applications of the ADP-based algorithms to systems with various engineering-oriented complexities. Five typical engineering-oriented complexities have been reviewed in this paper, i.e. communication protocols, event-triggered mechanisms, cyber-attacks, time delays, and saturation constraints. Recent advances of the ADP-based control strategies have been reviewed for nonlinear systems with the above-mentioned complexities in great detail. According to the reviewed literature, some research topics which can be investigated in the future are provided as follows:

1. Nowadays, the ADP-related security control issue has stirred initial research attention, but the implementation of the ADP-based algorithm on large-scale networked control systems (such as complex networks and sensor networks) under deception attacks has not been investigated yet.

2. To guarantee the ultimate boundedness of the estimation error of neural network weight, linear matrix inequalities have been widely employed. Since the computational complexity and the conservatism are the major concerns of linear matrix inequalities, some other approaches should be developed with reduced computational complexity and conservatism.

3. In order to facilitate theoretical analysis of ADP-based methods, various assumptions have been made. To further broaden the application field of ADP, how to remove or slacken the strong assumptions is a very meaningful topic with considerable engineering significance.

4. Much research effort has been devoted to the improvement of the ADP-based algorithms. From the perspective of engineering practice, the comparisons between the performances of different algorithms have not yet attracted enough research attention.

5. Along with the development of the economy and the society, the application environment is more and more complicated. So far, the improvement of the ADP-based algorithm to systems in more complicated environments has not received enough research attention.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Date sharing is not applicable to this article as no new data were created or analysed in this study.

Additional information

Funding

This work was supported in part by the National Natural Science Foundation of China [grant number 62273087], the Shanghai Pujiang Program of China [grant number 22PJ1400400] and the Natural Science Foundation of Heilongjiang Province of China [grant number ZD2022F003].

Notes on contributors

Yuhan Zhang

Yuhan Zhang received the B.Eng. degree in electronic information science and technology from the Shandong University of Science and Technology, Qingdao, China, in 2016, and the M.Sc. degree in marketing from the University of Nottingham, Nottingham, UK, in 2017. She is currently pursuing the Ph.D. degree in control science and engineering from Shandong University of Science and Technology, Qingdao, China. Yuhan Zhang is a very active reviewer for many international journals.

Lei Zou

Lei Zou received the Ph.D. degree in control science and engineering in 2016 from Harbin Institute of Technology, Harbin, China. He is currently a Professor with the College of Information Science and Technology, Donghua University, Shanghai, China. From October 2013 to October 2015, he was a visiting Ph.D. student with the Department of Computer Science, Brunel University London, Uxbridge, U.K. His research interests include control and filtering of networked systems, moving-horizon estimation, state estimation subject to outliers, and secure state estimation. Professor Zou serves (or has served) as an Associate Editor for IEEE/CAA Journal of Automatica Sinica, Neurocomputing, International Journal of Systems Science, and International Journal of Control, Automation and Systems, a Senior Member of IEEE, a Member of Chinese Association of Automation, a Regular Reviewer of Mathematical Reviews, and a very active reviewer for many international journals.

Yang Liu

Yang Liu received the B.Sc. degree and the Ph.D. degree in the Department of Automation at Tsinghua University, Beijing, China, in 2010 and 2016, respectively.Since 2021, he has been working as a Research Associate in the Department of Aeronautical and Automotive Engineering, Loughborough University, U.K. His research interests include system security, complex networks as well as stochastic systems. He is a Member of IEEE and a Member of Chinese Association of Automation; and a very active reviewer for many international journals.

Derui Ding

Derui Ding received both the B.Sc. degree in Industry Engineering in 2004 and the M.Sc. degree in Detection Technology and Automation Equipment in 2007 from Anhui Polytechnic University, Wuhu, China, and the Ph.D. degree in Control Theory and Control Engineering in 2014 from Donghua University, Shanghai, China. He is currently a Senior Research Fellow with the School of Science, Computing and Engineering Technologies, Swinburne University of Technology, Melbourne, VIC, Australia. His research interests include nonlinear stochastic control and filtering, as well as distributed control, filtering and optimization. He received the 2021 Nobert Wiener Review Award from IEEE/CAA Journal of Automatica Sinica, the 2020 and 2022 Andrew P. Sage Best Transactions Paper Awards from the IEEE Systems, Man, and Cybernetics (SMC) Society, and the IET Premium Awards 2018. He is a Senior Member of the Institute of Electrical and Electronic Engineers (IEEE). He is serving as an Associate Editor for IEEE Transactions on Industrial Informatics, IEEE/CAA Journal of Automatica Sinica, Neurocomputing and IET Control Theory & Applications.

Jun Hu

Jun Hu received the B.Sc. degree in information and computation science and M.Sc. degree in applied mathematics from Harbin University of Science and Technology, Harbin, China, in 2006 and 2009, respectively, and the Ph.D. degree in control science and engineering from Harbin Institute of Technology, Harbin, China, in 2013. From September 2010 to September 2012, he was a Visiting Ph.D. Student in the Department of Information Systems and Computing, Brunel University, U.K. From May 2014 to April 2016, he was an Alexander von Humboldt research fellow at the University of Kaiserslautern, Kaiserslautern, Germany. From January 2018 to January 2021, he was a research fellow at the University of South Wales, Pontypridd, U.K. He is currently Professor in the Department of Mathematics, Harbin University of Science and Technology, Harbin 150080, China. His research interests include nonlinear control, filtering and fault estimation, time-varying systems and complex networks. He has published more than 80 papers in refereed international journals. Dr. Hu serves as a reviewer for Mathematical Reviews, as an editor for Neurocomputing, Journal of Intelligent and Fuzzy Systems, Neural Processing Letters, Systems Science and Control Engineering, and as a guest editor for International Journal of General Systems and Information Fusion.