Deep Recurrent Reinforcement Learning for Intercept Guidance Law under Partial Observability

ABSTRACT

Nowadays, the rapid development of hypersonic vehicles brings great challenges to the missile defense system. As achieving successful interception depends highly on terminal guidance laws, research on guidance laws for intercepting highly maneuvering targets has aroused increasing attention. Artificial intelligence technologies, such as deep reinforcement learning (DRL), have been widely applied to improve the performance of guidance laws. However, the existing DRL guidance laws rarely consider the partial observability problem of onboard sensors, resulting in the limitations of their engineering applications. In this paper, a deep recurrent reinforcement learning (DRRL)-based guidance method is investigated to address the intercept guidance problem against maneuvering targets under partial observability. The sequence consisting of previous state observations is utilized as the input of the policy network. A recurrent layer is introduced into the networks to extract hidden information behind the temporal sequence to support policy training. The guidance problem is formulated as a partially observable Markov decision process model, and then a range-weighted reward function that considers the line-of-sight rate and energy consumption is designed to guarantee convergence of policy training. The effectiveness of the proposed DRRL guidance law is validated by extensive numerical simulations.

Introduction

Missile attack and defense confrontations are integral components of modern wars that have been extensively researched. Missile penetration capabilities have rapidly advanced in recent decades, such as the development of hypersonic vehicles, which greatly enhance the maneuverability of attackers. Moreover, the increasing diversity of maneuvering penetration strategies has increased the complexity and unpredictability of attacker motion. As a result, to intercept highly maneuvering targets successfully and accurately, it is critical to develop effective terminal guidance laws for interceptors.

A variety of terminal guidance laws have been proposed for intercepting maneuvering targets, such as the variants of proportional navigation guidance (PNG) (Ha et al. 1990; Yuan and Hsu 1995; Zarchan 2012), differential game (Battistini and Shima 2014) and sliding model control (SMC) (Hu, Han, and Xin 2019; Zhang et al. 2018) methods. These approaches can hit maneuvering targets precisely when interceptors exhibit a maneuverability advantage over attackers. However, this condition is not always satisfied in practice, and once attackers perform higher maneuvers, the interception performance is greatly reduced. Moreover, as the information obtained from the sensors is only a partial observation of the system state, an observer is usually utilized in these methods to estimate the unobservable states before generating the guidance commands, which consumes considerable computing resources and causes estimation biases.

As a subfield of artificial intelligence, reinforcement learning (RL) (Sutton and Barto 1998) and deep reinforcement learning (DRL) algorithms (Fujimoto, Hoof, and Meger 2018; Hasselt, Guez, and Silver 2016; Hausknecht and Stone 2015; Lillicrap et al. 2015; Volodymyr et al. 2015) are powerful data-driven approaches for addressing complicated decision and control problems. Recently, RL and DRL algorithms have been widely applied in education (Luo and Zhang 2023), medicine (Hantous, Rejeb, and Hellali 2022) and unmanned vehicles (Chansuparp and Jitkajornwanich 2022; Tan 2023; Wang et al. 2021; Wang, Gao, and Zhang 2021). To overcome the shortcomings of traditional guidance methods, RL and DRL algorithms have also been utilized to design guidance laws. Based on the Q-learning algorithm, Zhang, Ao, and Zhang (2020) improved the PNG method with an optimal time-varying proportional coefficient, and He et al. (2020) developed a novel guidance law to intercept maneuvering targets by introducing the concept of zero-effort-miss. Gaudet and Furfaro (2012) proposed an RL guidance approach for the missile homing phase by using a policy optimization algorithm. Using the RL algorithm, Shalumov (2020) developed a cooperative online launch and guidance policy for target – missile – defender engagement, which was shown to be near-optimal. Hong, Kim, and Park (2020) developed a DDPG-based intercept guidance law using the interceptor’s velocity and line-of-sight angular rate to generate guidance commands. Liu et al. (2021) investigated the impact time control guidance problem by estimating the time-to-go via a neural network and then nullifying the impact time error by utilizing the RL approach. Other DRL algorithms, such as model-based DRL (Liang et al. 2019; Peng et al. 2022), approximate dynamic programming (Zhou and Xu 2022) and reinforcement meta-learning (Gaudet, Furfaro, and Linares 2020), have also been used to explore intercept guidance laws. Overall, the aforementioned DRL-based guidance laws can address the disadvantages of traditional methods and exhibit excellent performance.

However, the full state of missile-target engagement system is not always accessible to the missile in practice due to the limited observability of onboard sensors. For these existing works, the partial observability problem has not been resolved in the process of guidance law design. Some studies assume that the full system state is available when computing guidance commands (Zhang et al. 2018). Other papers directly utilize sensory information (Gaudet, Furfaro, and Linares 2020; He, Shin, and Tsourdos 2021; Hong, Kim, and Park 2020; Wu et al. 2022) to approximate guidance laws, but the required maneuverability of the interceptor is much greater than that of the attacker. Furthermore, to handle unobservable states, a neural network-based state estimator was built as an auxiliary to the agent’s decision-making process (Li, Zhu, and Zhao 2022), which shares the same shortcomings with traditional estimation methods, such as estimation biases.

As the partial observability problem is one of the primary issues that needs to be settled before implementing DRL guidance law in practice, a novel guidance framework based on deep recurrent reinforcement learning (DRRL) is proposed in this paper (see Figure 1). The main contributions are summarized as follows:

1. A novel deep recurrent reinforcement learning framework is proposed to address the partial observability problem. The historical observation sequence, which consists of several timesteps of previous observations, is utilized as the input of the policy network. A recurrent neural network is introduced to extract the information hidden behind the historical observation sequence to support policy training.

2. The proposed DRRL algorithm is utilized to address the intercept guidance problem against a maneuvering target under the constraints of partial observability and limited maneuverability. Considering the partial observability of onboard sensors, the guidance problem is first formulated as a partially observable Markov decision process (POMDP) model. An immediate reward function that weights the impacts of the line-of-sight rate and energy consumption is designed to solve the sparse reward problem and improve the training performance.

3. Compared with traditional methods that use the full system state and perfect target acceleration information, the proposed DRRL guidance law achieves a higher interception probability and smaller miss distance when intercepting highly maneuvering targets. Additionally, the effect of the length of the observation sequence on policy performance is studied through extensive simulations.

Figure 1. The proposed guidance framework based on deep recurrent reinforcement learning.

The remainder of this paper is organized as follows. Section 2 presents the deep recurrent reinforcement learning algorithm to address partial observability. Section 3 the planar intercept problem is formulated against a maneuvering target under the framework of the partially observable Markov decision process model and presents the DRRL-based guidance law. In Section 4, extensive simulations are performed to validate the performance of the proposed guidance law. The conclusions are given in Section 5.

Deep Recurrent Reinforcement Learning

MDP and POMDP

In model-free reinforcement learning, an agent learns a policy from scratch and completes a task via episodic interactions with the environment. This problem can be abstracted as a Markov decision process model and described by a 4-tuple $\left(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\right)$ (Sutton and Barto 1998). At each timestep $\mathit{t}$, the agent chooses an action $a_t\in \mathcal{A}$ depending on the state $s_t\in \mathcal{S}$ observed from the environment. This action is transferred to the next state $s_{\mathit{t}+1}$ according to the transition function $\mathcal{P}$. Then, a feedback signal that is known as the reward function and denoted as $r_t\sim \mathcal{R}\left(s_t,a_t\right)$ is conveyed to the agent.

However, instead of the full state of the environment, an incomplete and noisy state observation is usually provided to the agent in practice because of limited observability. One example is missile guidance, in which onboard sensors only measure the range and angle but not the velocity and angular rate, which are indispensable information for guidance command generation. The partially observable Markov decision process (POMDP) model is utilized to describe the partially observable properties of real-world environments by explicitly admitting that the information received by the agent is only a portion of the system state. Formally, a POMDP model can be described as a 6-tuple $\left(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathrm{\Omega },\mathcal{O}\right)$. $\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}$ are the state space, action space, transition probability and reward function, as defined in the MDP. The agent no longer has access to the full system state and instead receives an observation from the observation space $\mathrm{\Omega }$ according to the probability distribution $\mathit{o}\sim \mathcal{O}\left(\mathit{s}\right)$.

Twin Delayed Deep Deterministic Policy Gradient (TD3) Algorithm

As the goal of reinforcement learning is to find an optimal behavior strategy for the agent to obtain optimal rewards, the policy gradient method directly optimizes the policy function, which is usually modeled as a parameterized function ${\pi }_{\varphi }$, and the optimization objective is

(1) $\mathit{J}\left(\mathit{\pi }\right)={\mathbb{E}}_{\pi }\left[R_t\left|s_t\right.\right]={\mathbb{E}}_{\pi }\left[\sum _{\mathit{k}=1}^{\mathrm{\infty }}{\gamma }^{\mathit{k}-1}{r}_{\mathit{t}+\mathit{k}}\left|s_t\right.\right]$(1)

where $\mathit{\gamma }\in \left[0,1\right]$ is a discount factor that determines the priority of short-term rewards.

As shown in Figure 2, the twin delayed deep deterministic policy gradient (TD3) algorithm (Fujimoto, Hoof, and Meger 2018) employs a deterministic policy network ${\mu }_{\varphi }$ (named actor) that maps the state observation to the agent’s action directly and a group of state-action value functions (named critics) consisting of two independent networks $Q_{{\theta }_1},Q_{{\theta }_2}$. Furthermore, each network has a corresponding target network denoted as ${\mu }_{\mathit{\varphi }{\mathit{ }}^{\mathrm{\prime }}},Q_{{\mathit{\theta }{\mathit{ }}^{\mathrm{\prime }}}_1},Q_{{\mathit{\theta }{\mathit{ }}^{\mathrm{\prime }}}_2}$, which has the same structure as their original networks. The TD3 algorithm uses a replay buffer $\mathcal{B}$ with a finite size $N_{\mathcal{B}}$ to store the transitions sampled from the environment. At each timestep, the algorithm randomly samples $N_b$ transitions from the buffer and updates the parameters of the actor and critic. The target Q value is computed by

Figure 2. The workflow of the TD3 algorithm.

(2) $\begin{array}{rl}& Q^{\mathrm{tar}}=r_t+\mathit{\gamma }\underset{\mathit{k}=1,2}{min}{Q}_{{\mathit{\theta }{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{k}}}\left(s_{\mathit{t}+1},{\mu }_{\mathit{\varphi }{\mathit{ }}^{\mathrm{\prime }}}\left(s_{\mathit{t}+1}\right)+\mathit{\epsilon }\right)\\ & \mathit{\epsilon }:\mathrm{clip}\left(\mathcal{N}\left(0,\mathit{\sigma }\right),-\mathit{c},\mathit{c}\right)\end{array}$(2)

where a small amount of random noise $\mathit{\epsilon }$ is introduced into the target policy to realize smoothing regularization of the training process. Then, the pair of critics is updated to minimize the Q value estimation error:

(3) $\begin{array}{rl}& \mathit{L}\left({\theta }_k\right)=\mathbb{E}\left[{\left(Q_{{\theta }_k}(s_t,a_t)-Q^{\mathrm{tar}}\right)}^2\right]\\ & {\theta }_k^{i+1}={\theta }_k^i-{\alpha }_{\theta }{\mathrm{\nabla }}_{{\theta }_k}\mathit{L}\left({\theta }_k\right)\mathit{k}=1,2\end{array}$(3)

where ${\alpha }_{\theta }$ is the learning rate for critics. The actor network is updated less frequently than the critic network; that is, every $\mathit{d}$ times of critics’ training, the actor is updated to maximize the average reward of the policy:

(4) $\begin{array}{rl}& {\mathrm{\nabla }}_{\varphi }\mathit{J}\left({\mu }_{\varphi }\right)=\mathbb{E}\left[{\mathrm{\nabla }}_{\varphi }{\mu }_{\varphi }\left(s_t\right){\mathrm{\nabla }}_a{Q}_{{\theta }_1}\left(s_t,\mathit{a}\right)\left|{ }_{a={\mu }_{\varphi }\left(s_t\right)}\right.\right]\\ & {\varphi }^{\mathit{i}+1}={\varphi }^i+{\alpha }_{\varphi }{\mathrm{\nabla }}_{\varphi }\mathit{J}\left({\mu }_{\varphi }\right)\end{array}$(4)

where ${\alpha }_{\varphi }$ is the learning rate for the actor. The target networks are softly updated by slowly tracking their corresponding networks:

(5) $\begin{array}{cc}\begin{array}{c}\mathit{\varphi }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \varphi }+\left(1-\mathit{\kappa }\right)\mathit{\varphi }{ }^{\mathrm{\prime }}\\ \mathit{\theta }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \theta }+\left(1-\mathit{\kappa }\right)\mathit{\theta }{ }^{\mathrm{\prime }}\end{array}& \mathit{\kappa }\ll 1\end{array}$(5)

where $\mathit{\kappa }$ is the soft update rate.

Deep Recurrent Reinforcement Learning (DRRL) for POMDP

As mentioned above, the TD3 algorithm is proposed under the framework of the MDP, where the whole environment state is accessible to the agent. However, for practical tasks, the whole system state is not always available due to limited measurements, and the resulting partial observability would cause significant performance degradation of the RL policy. To address this problem, we propose a historical observation sequence-based deep recurrent reinforcement learning algorithm. As shown in Figure 3, by combining the previous $\mathit{m}$ timesteps of observation, the current state observation is extended as a temporal sequence

Figure 3. Deep Recurrent Reinforcement Learning.

(6) $y_t=\left[o_{\mathit{t}-\mathit{m}},\cdots ,o_{\mathit{t}-1},o_t\right]$(6)

Although the unobservable state is not provided explicitly in $y_t$, the extensive information hidden behind the historical observation sequence can be extracted via neural networks and then substituted for the whole system state to help train the agent’s policy.

To extract and memorize the dependencies of continuous-time data effectively, we utilize a gated recurrent unit (GRU) (Cho et al. 2014), which is a typical kind of recurrent neural network (RNN). As depicted in Figure 3(a), there are two gates in the GRU. The reset gate $r_t$ determines whether the previous hidden state should be ignored and is computed by

(7) $r_t=\mathit{sigmoid}\left(W_r\cdot \left[h_{\mathit{t}-1},x_t\right]\right)$(7)

where $\mathit{sigmoid}\left(\cdot \right)$ is the logistic sigmoid function, $W_r$ is the weight matrix, and $h_{\mathit{t}-1}$ and $x_t$ are the previous hidden state and input, respectively. The update gate $z_t$ determines whether the hidden state should be updated with a new state and is computed as

(8) $z_t=\mathit{sigmoid}\left(W_z\cdot \left[h_{\mathit{t}-1},x_t\right]\right)$(8)

Then, the hidden state $h_t$ is updated by

(9) $h_t=\left(1-z_t\right)\ast h_{\mathit{t}-1}+z_t\ast {\tilde{h}}_{\mathit{t}}$(9)

where ${\tilde{h}}_{\mathit{t}}=\mathit{tanh}\left(\mathit{W}\cdot \left[r_t\ast h_{\mathit{t}-1},x_t\right]\right)$ and $\mathit{tanh}\left(\cdot \right)$ is the hyperbolic tangent function. In this way, each GRU can capture dependencies of temporal sequences over different time scales.

As shown in Figures 3(b,c), we adopt a GRU layer in both the actor and critic networks as a preprocessing module, which takes the temporal observation sequence as input, and its output is then fed to the original fully connected layers so that the hidden information behind the input sequence is extracted effectively as a substitute for those unobservable system states.

Guidance Law Based Upon DRRL

Problem Formulation

As shown in Figure 4, we consider two-dimensional engagement, where T and M represent the target and the intercepting missile, respectively. An exo-atmospheric interception engagement is considered, and the effect of gravity is neglected for simplicity. It is assumed that the missile and target both travel at a constant velocity $\mathit{v}$ with a flight path angle $\mathit{\theta }$ and lateral acceleration $\mathit{a}$. Thus, the dynamics of the missile-target relative motion are modeled as

Figure 4. Engagement geometry.

(10) $\left\{\begin{array}{c}\dot{R}=-v_{\mathrm{M}}cos(\mathit{\lambda }-{\theta }_{\mathrm{M}})-v_{\mathrm{T}}cos(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ \mathit{R}\dot{\lambda }=v_{\mathrm{M}}sin(\mathit{\lambda }-{\theta }_{\mathrm{M}})+v_{\mathrm{T}}sin(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ {\dot{\theta }}_{\mathrm{T}}=a_{\mathrm{T}}/v_{\mathrm{T}}\\ {\dot{\theta }}_{\mathrm{M}}=a_{\mathrm{M}}/v_{\mathrm{M}}\end{array}\right.$(10)

where $\mathit{R}$ denotes the missile-target relative range and $\mathit{\lambda }$ is the line-of-sight (LOS) angle.

As the acceleration is generated according to the autopilot with a time delay, we model the autopilot dynamics as a first-order time lag system:

(11) $\dot{a}=-\frac{1}{\tau }\mathit{a}+\frac{1}{\tau }{a}^{\mathrm{C}}$(11)

where $\mathit{\tau }$ is the time constant and $a^{\mathrm{C}}$ is the acceleration command.

Here, we consider two kinds of practical target periodic maneuver policies (Zarchan 2012): the barrel roll $\mathit{a}(\mathit{t})=a_{{\mathit{ }}^{\mathrm{T}}}^{max}\cdot sin[\frac{2\mathit{\pi }}{T}(\mathit{t}-t_0)]$ and the vertical-S maneuver $\mathit{a}(\mathit{t})=a_{{\mathit{ }}^{\mathrm{T}}}^{max}\cdot \mathrm{sign}\left(sin[\frac{2\mathit{\pi }}{T}(\mathit{t}-t_0)]\right)$, where $a_{{\mathit{ }}^{\mathrm{T}}}^{max}$ is the maneuver amplitude denoting the maximum acceleration capability of the target, $\mathit{T}$ is the period, $t_0$ is the time to start maneuvering, and $\mathrm{sign}\left(\cdot \right)$ is the sign function.

POMDP Model

In this section, considering the partial observability constraint, we formulate the interception problem under the framework of the POMDP model.

Generally, the whole system state is denoted as

(12) $\mathit{s}={\left[R,\lambda ,\dot{R},\dot{\lambda }\right]}^{\mathit{T}}$(12)

including the missile-target relative range $\mathit{R}$, line-of-sight angle $\mathit{\lambda }$, relative velocity $\dot{R}$ and LOS angular rate $\dot{\lambda }$. In practice, however, $\dot{R}$ and $\dot{\lambda }$ cannot be measured directly by onboard sensors; therefore, the state observation is written as

(13) $\mathit{o}={\left[\mathit{R},\mathit{\lambda }\right]}^T$(13)

As mentioned above, this paper utilizes the historical observation sequence $o_{\mathit{t}-\mathit{m}},\cdots ,o_{\mathit{t}-1},o_t$ as the input of the policy network to generate guidance commands, which are chosen as the agent’s action:

(14) $\mathit{a}\in [-a_{\mathrm{M}}^{max},a_{\mathrm{M}}^{max}],\mathit{a}\in \mathbb{R}$(14)

where $a_{\mathrm{M}}^{max}$ is the maximum acceleration of the interceptor.

The reward function has two components: the inherent terminal reward and the designed immediate reward. When the sign of the missile-target closing velocity becomes positive, the agent has reached the terminal state, and the confrontation terminates. According to the terminal miss distance $R_{\mathrm{terminal}}$, a terminal reward function can be designed as

(15) $r_{{\mathit{ }}_{\mathrm{terminal}}}=\left\{\begin{array}{c}\begin{array}{ccc}{r}_0& \mathit{\hspace{0.17em}}& R_{\mathrm{terminal}}\le R_{\mathrm{hit}}\end{array}\\ \begin{array}{ccc}0& & R_{\mathrm{terminal}}>R_{\mathrm{hit}}\end{array}\end{array}\right.$(15)

where $r_0$ is a positive value and $R_{\mathrm{hit}}$ is a constant used to judge whether the target has been hit. This constant is chosen according to the kill radius of the interceptor. When $R_{\mathrm{terminal}}$ is equal to or less than $R_{\mathrm{hit}}$, the interception is successful, and the interceptor is rewarded with $r_0$. In contrast, there is no reward for the off-target case where $R_{\mathrm{terminal}}$ is larger than $R_{\mathrm{hit}}$. Furthermore, to improve the learning efficiency, an immediate reward $r_t$ that weights the impact of the LOS rate and energy consumption according to the missile-target relative range is designed for the case in which the state changes from $s_t$ to a nonterminal state $s_{\mathit{t}+1}$ via an action$a_t$

(16) $r_t=\left(1-{\bar{R}}_{\mathit{t}}\right)exp\left(-\mathit{\beta }{\dot{\lambda }}_{\mathit{t}}^2\right)-{\bar{R}}_{\mathit{t}}\cdot {\bar{a}}_{\mathit{t}}^2$(16)

where ${\bar{R}}_{\mathit{t}}=\frac{R_t}{R_{max}}$ is the normalized missile-target relative range at time $\mathit{t}$, ${\dot{\lambda }}_{\mathit{t}}^{\mathit{\hspace{0.17em}}}$ is the LOS rate, ${\bar{a}}_{\mathit{t}}$ is the normalized missile acceleration, and $\mathit{\beta }$ is a constant. The first term in Equation (16)(16) $r_t=\left(1-{\bar{R}}_{\mathit{t}}\right)exp\left(-\mathit{\beta }{\dot{\lambda }}_{\mathit{t}}^2\right)-{\bar{R}}_{\mathit{t}}\cdot {\bar{a}}_{\mathit{t}}^2$(16) is a Gaussian-like function of $\dot{\lambda }$ that encourages the interceptor to nullify the line-of-sight angular rate. This formulation is inspired by traditional guidance laws; if $\dot{\lambda }=0$ holds, the missile-target relative velocity points toward the target throughout the engagement, ensuring successful interception with the maneuvering target. This method is effective for intercepting a maneuvering target but consumes a considerable amount of energy when tracking the target. Therefore, we apply an adaptive coefficient $\bar{R}$ that encourages the interceptor to maintain a constant line-of-sight angle $\mathit{\lambda }$ only when the missile and target are sufficiently close. Correspondingly, the second term in Equation (16)(16) $r_t=\left(1-{\bar{R}}_{\mathit{t}}\right)exp\left(-\mathit{\beta }{\dot{\lambda }}_{\mathit{t}}^2\right)-{\bar{R}}_{\mathit{t}}\cdot {\bar{a}}_{\mathit{t}}^2$(16) is used to discourage unnecessary actions when the missile is far from the target.

Guidance Law Optimization

The detailed training procedure for the proposed DRRL guidance law is listed in Table 1.

Table 1. Training procedure of the proposed DRRL guidance law.

Training Procedure of DRRL Guidance Law

Parameters initialization: discount factor $\mathit{\gamma }$, replay buffer size $\left|\mathcal{D}\right|$, batch sampling size $\left|\mathcal{B}\right|$, learning rate for actor ${\alpha }_{\mu }$ and critic ${\alpha }_Q$, random exploration noise $\mathit{\xi }$, policy noise $\mathit{\sigma }$, clipped noise factor ${\epsilon }_{max}$, actor train frequency $\mathit{d}$, target network soft update rate $\mathit{\kappa }$ Initialize the network parameters ${\theta }_1,{\theta }_2,\mathit{\varphi }$, and copy them for the target networks ${\theta }_1,{\theta }_2,\mathit{\varphi }\to \mathit{\theta }{{\mathit{ }}^{\mathrm{\prime }}}_1,\mathit{\theta }{{\mathit{ }}^{\mathrm{\prime }}}_2,\mathit{\varphi }{ }^{\mathrm{\prime }}$ For n=1, 2, … ., maximum training episode do: Randomly initialize the environment with an initial state $s_0$ For t=0, 1, 2, … , maximum steps do: 1) Observe the current environment state, and obtain the observation sequence $y_t=\left[o_{\mathit{t}-\mathit{m}},\cdots ,o_{\mathit{t}-1},o_t\right]$ by incorporating previous observations of $\mathit{m}$ timesteps 2) choose an action with the random exploration noise $a_t={\pi }_{\varphi }\left(y_t\right)+\mathcal{N}$ and execute it 3) the environment transitions to the next state $s_{\mathit{t}+1}$ according to Equation (10)(10) $\left\{\begin{array}{c}\dot{R}=-v_{\mathrm{M}}cos(\mathit{\lambda }-{\theta }_{\mathrm{M}})-v_{\mathrm{T}}cos(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ \mathit{R}\dot{\lambda }=v_{\mathrm{M}}sin(\mathit{\lambda }-{\theta }_{\mathrm{M}})+v_{\mathrm{T}}sin(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ {\dot{\theta }}_{\mathrm{T}}=a_{\mathrm{T}}/v_{\mathrm{T}}\\ {\dot{\theta }}_{\mathrm{M}}=a_{\mathrm{M}}/v_{\mathrm{M}}\end{array}\right.$(10) and a reward $r_t$ is fed back to the agent according to Equation (15)(15) $r_{{\mathit{ }}_{\mathrm{terminal}}}=\left\{\begin{array}{c}\begin{array}{ccc}{r}_0& \mathit{\hspace{0.17em}}& R_{\mathrm{terminal}}\le R_{\mathrm{hit}}\end{array}\\ \begin{array}{ccc}0& & R_{\mathrm{terminal}}>R_{\mathrm{hit}}\end{array}\end{array}\right.$(15) and Equation (16)(16) $r_t=\left(1-{\bar{R}}_{\mathit{t}}\right)exp\left(-\mathit{\beta }{\dot{\lambda }}_{\mathit{t}}^2\right)-{\bar{R}}_{\mathit{t}}\cdot {\bar{a}}_{\mathit{t}}^2$(16) 4) store the transition $<y_t,a_t,r_t,y_{\mathit{t}+1}>$ to the replay buffer 5) a batch of $N_b$ samples are collected from the buffer, update the critic and actor $\begin{array}{rl}& {\theta }_k^{i+1}={\theta }_k^i+{\alpha }_Q\frac{1}{N_b}\sum _{\mathit{t}=1}^{N_b}\left(\begin{array}{c}{r}_t+\mathit{\gamma }\underset{\mathit{k}=1,2}{min}{Q}_{{\mathit{\theta }{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{k}}}\left(y_{\mathit{t}+1},{\pi }_{\mathit{\varphi }{\mathit{ }}^{\mathrm{\prime }}}\left(y_{\mathit{t}+1}\right)+{\mathcal{N}}_{\mathrm{tar}}\right)\\ -Q_{{\theta }_k}\left(y_t,a_t\right)\end{array}\right){\mathrm{\nabla }}_{{\theta }_k}{Q}_{{\theta }_k}\left(y_t,a_t\right)\mathit{k}=1,2\\ & {\varphi }^{\mathit{i}+1}={\varphi }^i+{\alpha }_{\pi }\frac{1}{N_b}\sum _{\mathit{t}=1}^{N_b}{\mathrm{\nabla }}_{\varphi }{\pi }_{\varphi }\left(y_t\right){\mathrm{\nabla }}_a{Q}_{{\theta }_1}\left(y_t,\mathit{a}\right)\left|{ }_{a={\pi }_{\varphi }\left(y_t\right)}\right.\end{array}$ 6) the target networks’ parameters are soft updated according to Equation (5)(5) $\begin{array}{cc}\begin{array}{c}\mathit{\varphi }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \varphi }+\left(1-\mathit{\kappa }\right)\mathit{\varphi }{ }^{\mathrm{\prime }}\\ \mathit{\theta }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \theta }+\left(1-\mathit{\kappa }\right)\mathit{\theta }{ }^{\mathrm{\prime }}\end{array}& \mathit{\kappa }\ll 1\end{array}$(5) 7) if the environment terminates, break End for End for

It is noted that exploration noise is generated by an Ornstein-Uhlenbeck (OU) process to achieve better exploration, which is written as

(17) $\mathit{d}{x}_t=\mathit{\theta }\left(\mathit{\mu }-x_t\right)\mathit{dt}+\mathit{\sigma d}{W}_t$(17)

The OU process includes a mean reversion that fluctuates around the mean value $\mathit{\mu }$ with a rate $\mathit{\theta }>0$ and a Brownian motion, denoted by $W_t$, with variance $\mathit{\sigma }>0$. As depicted in Figure 5, the OU process results in temporally correlated noise that oscillates near 0. For physical control problems with inertia, the gap between two adjacent actions should be small, as it is unrealistic to alter one action without experiencing a delay. Since the intercept guidance problem is a physical control problem, the OU process is adopted for high-efficiency exploration.

Figure 5. Ornstein–Uhlenbeck noise ($\mathit{\mu }=0,\mathit{\theta }=0.15,\mathit{\sigma }=0.2$).

Furthermore, considering the instability of the training process caused by the introduction of a recurrent layer, we consider a random sample from the replay buffer at each update and use a bootstrapped random update strategy in which updates begin at random points in one episode and proceed for only one timestep, with the GRU’s initial hidden state set to zero at the start of each update (Hausknecht and Stone 2015).

Experiments and Analysis

Environment and Algorithm Settings

The environment is initialized as follows. First, the minimum and maximum values of the engagement parameters listed in Table 2 are uniformly sampled. Then, the interceptor is placed on a collision triangle, as shown in Figure 4, with the velocity vector ${\mathit{v}}_{\mathrm{M}}^{\mathrm{P}\mathrm{I}\mathrm{P}}$ pointing in the direction of the predicted intercept point (PIP) and the leading angle $\mathit{L}={sin}^{-1}\frac{{\mathit{v}}_{\mathrm{T}}sin({\theta }_{\mathrm{T}}+\mathit{\lambda })}{v_{\mathrm{M}}}$. Furthermore, considering the potential biases when calculating the PIP in practice, a heading error $\mathit{\epsilon }$ is introduced into the interceptor’s initial flight path angle: ${\theta }_{\mathrm{M}}=\mathit{\lambda }+\mathit{L}+\mathit{\epsilon }$.

Table 2. Initial parameters for engagement.

Parameter	Definition (unit)	Min	Max
$\mathit{R}$	Relative distance (m)	30000	40000
$\mathit{\lambda }$	Line-of-sight angle (degree)	30	50
$v_{\mathrm{T}}$	Target velocity (m/s)	1000	1000
$v_{\mathrm{M}}$	Missile velocity (m/s)	1200	1500
${\theta }_{\mathrm{T}}$	Target path angle (degree)	5	10
$\mathit{\epsilon }$	Heading error (degree)	−5	5

In the training phase, the attacker’s maneuver mode is chosen as a barrel roll with a maximum acceleration of $a_{\mathrm{T}}^{max}=5\mathit{g}$ and a maneuver period of $\mathit{T}=10\mathrm{s}$. The attacker begins to maneuver randomly when the missile-target relative range is less than 20 km and more than 5 km. The maximum acceleration of the interceptor is set as $a_{\mathrm{M}}^{max}=5\mathit{g}$. The time constant of the autopilot is chosen as $\mathit{\tau }=0.1\mathrm{s}$. The system state is updated using the fourth-order Runge – Kutta method and variable step-size integration, where a timestep of 0.001 s is used for the final 100 m guidance and a timestep of 0.1 s is used otherwise. The number of previous observations in the input sequence is set to $\mathit{m}=5$. When the terminal state is reached, a terminal miss distance of less than $r_{\mathrm{hit}}=10\mathrm{m}$ is defined as successful interception, and the agent is given a positive reward of $r_0=20$ in this case. The constant in the immediate reward function is set to $\mathit{\beta }={10}^4$.

The reward at each step is discounted by the factor $\mathit{\gamma }=0.99$. The capacity of the experience replay buffer is $N_{\mathcal{B}}={10}^6$, and each update of a batch of $N_b=512$ samples is randomly chosen and fed into the networks. The exploration noise is generated by an OU process with parameters of $\mathit{\mu }=0$, $\mathit{\theta }=0.15$ and $\mathit{\sigma }=0.3$. The target policy noise is generated by a Gaussian process with parameters $\mathit{\mu }=0$ and $\mathit{\sigma }=0.2$, both of which are clipped with a maximum value of 0.5. For both the actor and critic networks, the number of hidden units is 128, 128, and 64, respectively, and their learning rates are set to ${\alpha }_{\varphi }={\alpha }_{\theta }=3\times {10}^{-4}$. The target networks are softly updated with a rate of $\mathit{\kappa }=5\times {10}^{-3}$. The actor train frequency is selected as $\mathit{d}=2$.

Training Results

The proposed DRRL guidance law is trained in the aforementioned stochastic scenario, and an ablation experiment using vanilla DRL is conducted for comparison, where a single observation layer and a fully connected layer are used. The learning curve for 60,000 episodes of training is shown in Figure 6. It is clear that in the first 20,000 episodes, the average cumulative reward increases gradually for DRRL, and after the 20,000^th episode, the curve becomes steady, indicating the convergence of the training policy. Nevertheless, the DRL approach exhibits inferior learning efficiency, and the average cumulative reward is much lower than that of DRRL.

Figure 6. Learning curves.

Furthermore, 1000 Monte Carlo simulation tests are performed for different initial conditions, in which the initial parameters vary in the range listed in Table 2. Two classic model-based approaches, augmented proportional navigation (APN) and SMC, are used for comparison; refer to Equations (18)(18) $a_{\mathrm{APN}}=N_{\mathrm{APN}}\dot{R}\dot{\lambda }+\frac{1}{2}{N}_{\mathrm{APN}}{\hat{a}}_{\mathrm{T}}$(18) and (19(19) $a_{\mathrm{SMC}}=N_{\mathrm{SMC}}\left|\dot{R}\right|\dot{\lambda }+{\bar{a}}_{\mathrm{T}}\frac{\dot{\lambda }}{\dot{\lambda }+\mathit{\delta }}$(19) ), respectively.

(18) $a_{\mathrm{APN}}=N_{\mathrm{APN}}\dot{R}\dot{\lambda }+\frac{1}{2}{N}_{\mathrm{APN}}{\hat{a}}_{\mathrm{T}}$(18)

(19) $a_{\mathrm{SMC}}=N_{\mathrm{SMC}}\left|\dot{R}\right|\dot{\lambda }+{\bar{a}}_{\mathrm{T}}\frac{\dot{\lambda }}{\dot{\lambda }+\mathit{\delta }}$(19)

where $N_{\mathrm{APN}}$ and $N_{\mathrm{SMC}}$ are coefficients for APN and SMC, respectively; ${\hat{a}}_{\mathrm{T}}$ is the estimation of target acceleration; ${\bar{a}}_{\mathrm{T}}$ is the upper bound of target acceleration; and $\mathit{\delta }$ is the coefficient of the limiter function. It is assumed that perfect information on the target maneuver is provided for the APN and SMC in the simulation. The results are shown in Table 3, and it is obvious that the proposed DRRL guidance law obtains a hit probability of 97.9%, which is 60% and 30% greater than those of APN and SMC, respectively. With the energy cost defined as ${\int }_{t_0}^{t_f}{a}_{\mathrm{M}}^2\mathrm{d}\mathit{t}$, the DRRL approach achieves the minimum average energy consumption. Despite the limited observability and maneuverability of interceptors, the DRRL guidance law can realize higher interception performance with less energy consumption, which is of practical engineering significance.

Table 3. Test results in the training environment.

Method	Hit probability (%)	Average Energy Cost (m^2/s^3)
APN	37.0	14580.33
SMC	60.3	16323.58
DRRL	97.9	12229.66

In Figure 7, a missile-target engagement trajectory is depicted, including the flight trajectories of the missile and target and the line-of-sight angular rate, target acceleration and missile acceleration curves. The target starts performing the barrel roll maneuver at approximately 13 s, and successful interception is achieved by the APN, SMC and DRRL. The terminal miss distances are 2.96 m, 1.85 m and 0.86 m, respectively, where DRRL realizes a more accurate attack on the maneuvering target.

Figure 7. Interception trajectories of the APN, SMC and DRRL methods.

Test Results

In this section, a well-trained DRRL policy is tested in diverse training scenarios to verify its adaptability and robustness. The test scenarios are designed as follows.

Scenario 1: the target performs a barrel roll maneuver with the same parameters as in the training scenario.

Scenario 2: the target performs a vertical-S maneuver with the same parameters as in the training scenario.

Scenario 3: the target performs a barrel roll maneuver, and its period is 5 s.

Scenario 4: the target performs a barrel roll maneuver, and its period is 15 s.

Scenario 5: the target performs a barrel roll maneuver with a maximum acceleration of 7 g.

For these five scenarios, 1000 Monte Carlo simulation tests are conducted with various maximum heading errors $\mathit{HE}=0^{\circ },5^{\circ },{10}^{\circ },{15}^{\circ },{20}^{\circ }$. The hit probability results for each case are shown in Figure 8. It is obvious that the proposed DRRL guidance law achieves a higher intercept probability than APN and SMC in all situations. As shown in Figure 8(a,b), as the target is always at maximum acceleration when performing a vertical-S maneuver, there is an obvious performance degradation for the three methods compared to the case of intercepting a barrel roll maneuvering target. Nevertheless, the hit probability of DRRL is approximately 30% greater than that of APN and SMC. According to Figure 8(c-f), the DRRL guidance law adapts to various uncertainties of the target maneuver, including the maneuver period and maximum acceleration. In addition, it is clear from Figure 8(a-e) that the hit probability of DRRL decreases slowly as the initial heading error increases but remains higher than that of the other two methods, while SMC exhibits susceptibility to heading error.

Figure 8. Performance comparison of APN, SMC and DRRL in test scenarios.

Discussion on the Observation Sequence

In this section, the sensitivity of the proposed DRRL guidance law to the observation sequence length is studied. The simulation is conducted with $\mathit{m}$ ranging from 1 to 7 in the training scenario (see Table 1), and the other hyperparameters remain unchanged.

The results are depicted in Figure 9 and Table 4. It is clear from Figure 9 that the efficiency of policy training is very low, and the average cumulative reward converges to a low value when the sequence input consists of only one timestep of the previous observation. When $\mathit{m}\ge 2$, there is a significant improvement in training efficiency, indicating that the inclusion of more information contributes to higher learning efficiency and effectiveness. However, Table 4 shows that the interception probability reaches the highest point of 99.7% when $\mathit{m}=6$ and does not further increase when $\mathit{m}>6$. The results indicate that there is no additional information that can be extracted from the sequence input even if many more previous observations are provided. As a result, it is suggested that 6 timesteps of previous observations be included in the observation sequence, and the resulting guidance law can well juggle the interception performance and computational complexity.

Figure 9. Comparison of learning curves when the model is trained with different values of $\mathit{m}$.

Table 4. Performance comparison with different lengths of observation sequence input.

m	1	2	3	4	5	6	7
Hit Probability (%)	5.9	76.9	68.2	85.1	97.9	99.7	98.4

Conclusion

In this work, a deep recurrent reinforcement learning algorithm is proposed to address the intercept guidance problem against maneuvering targets with constraints of partial observability and acceleration limits. By introducing historical observation sequence input and gated recurrent unit into the policy network, the information hidden behind the temporal sequence is extracted and used to support policy training. Numerical simulation results validate that the proposed DRRL guidance law can intercept highly maneuvering targets successfully and effectively when the interceptor has limited observation ability and maneuverability. Compared to the APN and SMC methods, the DRRL approach exhibits superior performance in terms of the hit rate, energy consumption and generalization capability. Moreover, sensitivity studies on the effect of observation sequence length were performed, and the best performance could be achieved when 6 timesteps of observation were included in the sequence input. This work provides insights for DRL guidance law design under constraints of partial observability and limited maneuverability.

However, there are still some limitations of the proposed DRRL guidance laws. First, although the learning efficiency of the DRRL method is enhanced compared to the vanilla DRL approach, it needs to be improved further to satisfy the requirement of engineering application. Additionally, future work might consider more diverse and realistic engagement scenarios, such as three-dimensional engagement and endo-atmospheric interception.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

The authors confirm that the data supporting the findings of this study are available within the article.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China under Grant Nos. 62203349 and 12302061.