Monocular vision guided deep reinforcement learning UAV systems with representation learning perception

Figures & data

Table 1. Comparison of network structures used in 3 different continuous actions space DRL methods.

Model name	Number of hidden layers	Number of neurons in 1st hidden layer	Number of neurons in 2nd hidden layer
DDPG	2	400	300
TD3	2	256	256
SAC	2	256	256

Figure 1. TD3.

Figure 2. Single image model.

Figure 3. Convolutional variational autoencoder.

Figure 4. Structure of actor net and critic net in the model.

Figure 5. Continuous images model.

Figure 6. Structure of actor net and critic net in the model with LSTM layer.

Figure 7. Short-term memory state construction.

Figure 8. Short-term memory model.

Table 2. Environment Settings.

CPU	AMD Ryzen Threadripper 3970X 3.69GHz
GPU	2 x NVIDIA RTX3090
RAM	64 GB
Operating System	Windows 10
Program Language	Python 3.7
ML Library	Pytorch 1.9
CV Library	OpenCV 4.4
Simulator	Airsim 1.2.8
Game Engine	Unreal Engine 4.24.3

Figure 9. Simulation environment.

Table 3. Reward function.

Situation	Reward
Collision	−2.0
Action without collision	0.2

Figure 10. Reward comparison of single image model, continuous image model and short-term memory model during training.

Figure 11. Average moving distance comparison of single image model, continuous image model and short-term memory model during training.

Table 4. MobileNet V2 structure.

Input	Operator	t	c	n	s
${224}^2\times 3$	conv2d	–	32	1	2
${112}^2\times 32$	bottleneck	1	16	1	1
${112}^2\times 16$	bottleneck	6	24	2	2
${56}^2\times 24$	bottleneck	6	32	3	2
${28}^2\times 32$	bottleneck	6	64	4	2
${14}^2\times 64$	bottleneck	6	96	3	1
${14}^2\times 96$	bottleneck	6	160	3	2
$7^2\times 160$	bottleneck	6	320	1	1
$7^2\times 320$	conv2d $1\times 1$	–	1280	1	1
$7^2\times 1280$	avgpool 7$\times 7$	–	–	1	–
$1\times 1\times 1280$	conv2d $1\times 1$	–	k	–

Figure 12. Reward comparison of single image model and MobileNet V2 + TD3.

Figure 13. Average moving distance comparison of single image model and MobileNet V2 + TD3.

Table 5. Test results of short-term memory model.

Short-term Memory Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	100	100%
Obstacle 2	97	97%
Obstacle 3	93	95.0.9%
Obstacle 4	70	75.3%
Obstacle 5	62	88.6%
Obstacle 6	51	82.2%
Obstacle 7	39	76.5%
Obstacle 8	27	69.2%
Obstacle 9	25	92.6%
Average flight distance	22.6 metres
Average obstacle avoidance rate	86.4%

Table 6. Test results of continuous images model.

Continuous Images Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	96	96.0%
Obstacle 2	96	100%
Obstacle 3	63	65.7%
Obstacle 4	57	90.5%
Obstacle 5	53	93.0%
Obstacle 6	47	88.7%
Obstacle 7	31	66.0%
Obstacle 8	20	64.5%
Obstacle 9	14	70.0%
Average flight distance	19.8 metres
Average obstacle avoidance rate	81.6%

Table 7. Test results of single image model.

Single Image Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	98	98.0%
Obstacle 2	88	89.8%
Obstacle 3	57	64.8%
Obstacle 4	46	80.7%
Obstacle 5	42	91.3%
Obstacle 6	37	88.1%
Obstacle 7	24	64.9%
Obstacle 8	14	58.3%
Obstacle 9	0	28.6%
Average flight distance	14.7meters
Average obstacle avoidance rate	73.8%

Table 8. Test results of Mobile Net + TD3 model.

Mobile Net + TD3 Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	100	100%
Obstacle 2	57	57.0%
Obstacle 3	53	93.0%
Obstacle 4	33	62.3%
Obstacle 5	21	63.6%
Obstacle 6	18	85.7%
Obstacle 7	4	22.2%
Obstacle 8	0	0%
Obstacle 9	0	0%
Average flight distance	8.4 metres
Average obstacle avoidance rate	53.8%

Table 9. Test results in rearranged environment.

	Short-term Memory Model		Continuous Images Model
	Pass Times	Obstacle avoidance rate	Pass Times	Obstacle avoidance rate
Obstacle 7	98	98.0%	97	97.0%
Obstacle 5	87	88.8%	83	85.6%
Obstacle 8	84	96.6%	69	83.1%
Obstacle 9	41	48.8%	32	46.4%
Obstacle 2	28	68.3%	22	68.8%
Obstacle 1	23	82.1%	13	59.1%
Obstacle 6	13	52.2%	10	76.9%
Obstacle 4	8	61.5%	3	30.0%
Obstacle 3	3	37.5%	1	33.3%
Average flight distance	14.7 metres		12.6 metres
Average obstacle avoidance rate	70.9%		64.5%
	Single Image Model		Mobile Net + TD3 Model
	Pass Times	Obstacle avoidance rate	Pass Times	Obstacle avoidance rate
Obstacle 7	98	98.0%	80	80.0%
Obstacle 5	88	89.8%	65	81.3%
Obstacle 8	74	84.1%	30	46.2%
Obstacle 9	47	63.5%	18	60.0%
Obstacle 2	26	55.3%	14	77.8%
Obstacle 1	9	34.6%	12	85.7%
Obstacle 6	5	55.6%	7	58.3%
Obstacle 4	1	20.0%	1	14.3%
Obstacle 3	1	100%	0	0%
Average flight distance	10.7 metres		9.8 metres
Average obstacle avoidance rate	66.8%		55.9%

Table 10. Side collision count in 100 flights of 3 models.

Model Name	Short-term Memory Model	Continuous Images Model	Single Image Model	Mobile Net + TD3
Side Collision Count	13	26	24	26