Deep Recurrent Reinforcement Learning for Intercept Guidance Law under Partial Observability

Figures & data

Figure 1. The proposed guidance framework based on deep recurrent reinforcement learning.

Figure 2. The workflow of the TD3 algorithm.

Figure 3. Deep Recurrent Reinforcement Learning.

Figure 4. Engagement geometry.

Table 1. Training procedure of the proposed DRRL guidance law.

Training Procedure of DRRL Guidance Law

Parameters initialization: discount factor $\mathit{\gamma }$, replay buffer size $\left|\mathcal{D}\right|$, batch sampling size $\left|\mathcal{B}\right|$, learning rate for actor ${\alpha }_{\mu }$ and critic ${\alpha }_Q$, random exploration noise $\mathit{\xi }$, policy noise $\mathit{\sigma }$, clipped noise factor ${\epsilon }_{max}$, actor train frequency $\mathit{d}$, target network soft update rate $\mathit{\kappa }$ Initialize the network parameters ${\theta }_1,{\theta }_2,\mathit{\varphi }$, and copy them for the target networks ${\theta }_1,{\theta }_2,\mathit{\varphi }\to \mathit{\theta }{{\mathit{ }}^{\mathrm{\prime }}}_1,\mathit{\theta }{{\mathit{ }}^{\mathrm{\prime }}}_2,\mathit{\varphi }{ }^{\mathrm{\prime }}$ For n=1, 2, … ., maximum training episode do: Randomly initialize the environment with an initial state $s_0$ For t=0, 1, 2, … , maximum steps do: 1) Observe the current environment state, and obtain the observation sequence $y_t=\left[o_{\mathit{t}-\mathit{m}},\cdots ,o_{\mathit{t}-1},o_t\right]$ by incorporating previous observations of $\mathit{m}$ timesteps 2) choose an action with the random exploration noise $a_t={\pi }_{\varphi }\left(y_t\right)+\mathcal{N}$ and execute it 3) the environment transitions to the next state $s_{\mathit{t}+1}$ according to Equation (10)(10) $\left\{\begin{array}{c}\dot{R}=-v_{\mathrm{M}}cos(\mathit{\lambda }-{\theta }_{\mathrm{M}})-v_{\mathrm{T}}cos(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ \mathit{R}\dot{\lambda }=v_{\mathrm{M}}sin(\mathit{\lambda }-{\theta }_{\mathrm{M}})+v_{\mathrm{T}}sin(\mathit{\lambda }+{\theta }_{\mathrm{T}})\\ {\dot{\theta }}_{\mathrm{T}}=a_{\mathrm{T}}/v_{\mathrm{T}}\\ {\dot{\theta }}_{\mathrm{M}}=a_{\mathrm{M}}/v_{\mathrm{M}}\end{array}\right.$(10) and a reward $r_t$ is fed back to the agent according to Equation (15)(15) $r_{{\mathit{ }}_{\mathrm{terminal}}}=\left\{\begin{array}{c}\begin{array}{ccc}{r}_0& \mathit{\hspace{0.17em}}& R_{\mathrm{terminal}}\le R_{\mathrm{hit}}\end{array}\\ \begin{array}{ccc}0& & R_{\mathrm{terminal}}>R_{\mathrm{hit}}\end{array}\end{array}\right.$(15) and Equation (16)(16) $r_t=\left(1-{\bar{R}}_{\mathit{t}}\right)exp\left(-\mathit{\beta }{\dot{\lambda }}_{\mathit{t}}^2\right)-{\bar{R}}_{\mathit{t}}\cdot {\bar{a}}_{\mathit{t}}^2$(16) 4) store the transition $<y_t,a_t,r_t,y_{\mathit{t}+1}>$ to the replay buffer 5) a batch of $N_b$ samples are collected from the buffer, update the critic and actor $\begin{array}{rl}& {\theta }_k^{i+1}={\theta }_k^i+{\alpha }_Q\frac{1}{N_b}\sum _{\mathit{t}=1}^{N_b}\left(\begin{array}{c}{r}_t+\mathit{\gamma }\underset{\mathit{k}=1,2}{min}{Q}_{{\mathit{\theta }{\mathit{ }}^{\mathrm{\prime }}}_{\mathit{k}}}\left(y_{\mathit{t}+1},{\pi }_{\mathit{\varphi }{\mathit{ }}^{\mathrm{\prime }}}\left(y_{\mathit{t}+1}\right)+{\mathcal{N}}_{\mathrm{tar}}\right)\\ -Q_{{\theta }_k}\left(y_t,a_t\right)\end{array}\right){\mathrm{\nabla }}_{{\theta }_k}{Q}_{{\theta }_k}\left(y_t,a_t\right)\mathit{k}=1,2\\ & {\varphi }^{\mathit{i}+1}={\varphi }^i+{\alpha }_{\pi }\frac{1}{N_b}\sum _{\mathit{t}=1}^{N_b}{\mathrm{\nabla }}_{\varphi }{\pi }_{\varphi }\left(y_t\right){\mathrm{\nabla }}_a{Q}_{{\theta }_1}\left(y_t,\mathit{a}\right)\left|{ }_{a={\pi }_{\varphi }\left(y_t\right)}\right.\end{array}$ 6) the target networks’ parameters are soft updated according to Equation (5)(5) $\begin{array}{cc}\begin{array}{c}\mathit{\varphi }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \varphi }+\left(1-\mathit{\kappa }\right)\mathit{\varphi }{ }^{\mathrm{\prime }}\\ \mathit{\theta }{ }^{\mathrm{\prime }}\leftarrow \mathit{\kappa \theta }+\left(1-\mathit{\kappa }\right)\mathit{\theta }{ }^{\mathrm{\prime }}\end{array}& \mathit{\kappa }\ll 1\end{array}$(5) 7) if the environment terminates, break End for End for

Figure 5. Ornstein–Uhlenbeck noise ($\mathit{\mu }=0,\mathit{\theta }=0.15,\mathit{\sigma }=0.2$).

Table 2. Initial parameters for engagement.

Parameter	Definition (unit)	Min	Max
$\mathit{R}$	Relative distance (m)	30000	40000
$\mathit{\lambda }$	Line-of-sight angle (degree)	30	50
$v_{\mathrm{T}}$	Target velocity (m/s)	1000	1000
$v_{\mathrm{M}}$	Missile velocity (m/s)	1200	1500
${\theta }_{\mathrm{T}}$	Target path angle (degree)	5	10
$\mathit{\epsilon }$	Heading error (degree)	−5	5

Figure 6. Learning curves.

Table 3. Test results in the training environment.

Method	Hit probability (%)	Average Energy Cost (m^2/s^3)
APN	37.0	14580.33
SMC	60.3	16323.58
DRRL	97.9	12229.66

Figure 7. Interception trajectories of the APN, SMC and DRRL methods.

Figure 8. Performance comparison of APN, SMC and DRRL in test scenarios.

Figure 9. Comparison of learning curves when the model is trained with different values of $\mathit{m}$.

Table 4. Performance comparison with different lengths of observation sequence input.

m	1	2	3	4	5	6	7
Hit Probability (%)	5.9	76.9	68.2	85.1	97.9	99.7	98.4

Data Availability Statement

The authors confirm that the data supporting the findings of this study are available within the article.