A Novel Augmentative Backward Reward Function with Deep Reinforcement Learning for Autonomous UAV Navigation

Figures & data

Figure 2.1. UAV position and Euler angles in 3D simulation environment.

Figure 3.1. Simulation LiDAR sensor adopted in this work, Velodyne VLP-16 model.

Figure 3.2. Left: Truncated icosahedron structure, Middle: UAV in environment, Right: A UAV’s perception of surrounding objects.

Figure 3.3. Features extraction network structure. The parameters of one-dimensional convolutional layer (Conv1) denote its filters and its kernel size respectively and for the other layers, it denotes dimensionality of output. Each layer’s activation function is Rectified Linear Unit (ReLU).

Figure 3.4. The process flow of Augmentative Backward Reward function (ABR).

Figure 3.5. The process flow of proposed method in autonomous UAV navigation task. The UAV (agent) ask the actor network for next action and then the critic network will evaluate that selected action.

Table 1. Attributes and Hyper-parameters for this work.

No.	Name	Symbol	Value
1	UAV	-	Quadrotor
2	LiDAR sensor	-	Velodyne vlp
3	Environment	-	1200 $m^2$
4	Discount factor	$\gamma$	0.99
5	Soft update rate	$p_{tau}$	0.005
6	Noise bound	$c$	0.5
7	Action limit	$a_{low},a_{high}$	-2,2
8	Number of frames in one state	$p_{frames}$	3
9	Number of features extracted from one state	$p_{features}$	20
10	Similarity threshold	$p_{similarity}$	98%
11	Additional reward multiplier	$p_{addReward}$	0.1
12	Delay to slow the update of actor and target networks	$p_{updateDelay}$	2
13	Reward when collide	$r_{collided}$	-50
14	Appropriate altitude or Flight level	$p_{approAlt}$	3.5 m
15	Size of buffer	$p_{buffer}$	${10}^6$
16	Weight-parameters in reward function	$w_{1-4}$	8,1,1,0.5 respectively
17	Radius of Truncated icosahedron structure, Safe flying distance	$p_{radius}$	4 m
18	Maximum number of steps per episode	$p_{limitStep}$	300
19	Batch size	$N$	100
20	Traditional reward function: goal reward, collision punishment (In case “other”, Formula 3.5)	-	50,-50
21	Small reward given to all transitions in success episode of equal goal reward function	-	1

Figure 4.1. The 3D static simulation environment. Left: Top view of all whole map. Right: Main view of center of map. The white cylinder is goal point.

Figure 4.2. Experimental result in static environment. Left: Average success rate that UAV reach the goal point. Right: Average collision rate that UAV collide with environment surface. (The figure is designed for color version).

Figure 4.3. The 3D dynamic simulation environment. Left: Top view of all whole map. Right: Main view of center of map.

Figure 4.4. Experimental result in dynamic environment. Left: Average success rate. Right: Average collision rate. (The figure is designed for color version).

Figure 4.5. Experimental result of TD3 method and TD3+Seq+ABR method (Proposed) in other navigation task, BipedalWalkerHardcore. (The figure is designed for color version).