Monocular vision guided deep reinforcement learning UAV systems with representation learning perception

Abstract

In recent years, numerous studies have applied deep reinforcement learning (DRL) algorithms to vision-guided unmanned aerial systems. However, DRL is not good at training deep networks in an end-to-end manner due to data inefficiency and lack of direct supervision signals. This paper provides a visual information dimension reduction scheme with representation learning as the visual perception module, which reduces the dimensions of high-dimensional visual information and retains its features related to UAV navigation. Combining such state representation learning with the DRL model can effectively reduce the complexity of the neural network required by DRL. Based on this scheme, we design three motion control models with a monocular camera as the main sensor and train them to control UAVs for obstacle avoidance tasks in a simulated environment. Experiments show that all these models achieve high obstacle avoidance ability after a certain period of training. In addition, one of them also enables the monocular vision guidance system to avoid obstacles in the blind spot of side vision.

KEYWORDS:

• Deep reinforcement learning
• vision guided robotic system
• monocular vision
• UAV
• VAE

1. Introduction

Vision guided robotic systems (VGR) have become a hot area of research in recent years (Singh et al., 2022). Among the methods employed in this research area, machine learning algorithms play an extremely important role. In the field of visual perception, machine learning has achieved remarkable results in obstacle detection (He et al., 2021), semantic segmentation (Kumar et al., 2021), road detection (Caltagirone et al., 2019) and many application scenarios. In the field of path planning, computer vision-based machine learning algorithms have also achieved good results (Elkholy et al., 2020). Machine learning also performs well when dealing with vision-based robot control tasks (Xiao et al., 2022).

In vision guided robotic systems, visual guidance of unmanned aerial vehicles (UAV) (Lu et al., 2018) has become a significant focus of research, due to its complex environmental changes and manipulation difficulties. Among them, research based on deep learning has achieved excellent results in various practical tasks. For example, (Lee et al., 2020) designed a model called Faster R-CNN for tree trunk detection. Thus, the UAV can perform monocular navigation in tree plantations. The authors of (Padhy et al., 2018) trained a CNN-based model as a classifier for indoor environments, making UAVs capable of monocular navigation in indoor corridors. Deep reinforcement learning has also produced many excellent results in the field of robot navigation in recent years (Nachum et al., 2018) (Zhou & Ho, 2022). UAVs often need to fly in partially or completely unknown environments that cannot be accurately modelled mathematically, technology that enables UAVs to learn paths and movements autonomously is critical. Therefore, application of deep reinforcement learning to UAVs’ vision guided systems has become a hot research direction (Azar et al., 2021). However, DRL has an important limitation when using visual information as a state. The use of deep neural networks in DRL, allows the agent to gain the ability to observe the environment directly in high dimensions. Nonetheless, the shortcomings of neural networks, namely, low sample efficiency, requirement of a large amount of data for training and testing, and the instabilities of learning are also being simulataneously inherited. Therefore, when faced with high-dimensional data such as visual input, DRL models are often caught in a dilemma: On the one hand, a large amount of stable data is not available for DRL to quickly fit complex image recognition networks such as Yolo (Redmon et al., 2016) and visual transformers (Dosovitskiy et al., 2021). On the other hand, shallow neural network structure cannot handle the task of extracting effective features from visual information. Some researchers are also working to address this dilemma. As in (Ota et al., 2020), in order to solve the poor performance of directly using wide and deep neural networks in reinforcement learning, Actor and Critic networks with DenseNet structure are used to deal with high-dimensional states.

One potential solution to the aforementioned issues is the utilisation of representation learning. This method involves the acquisition of low-dimensional representations of visual information, which preserve properties that are essential to the task at hand. These representations are then used to learn optimal policies. Representation learning, as stated in (Bengio et al., 2013), is often used as an auxiliary tool in the field of computer vision. It has shown impressive results in areas such as dimensionality reduction, compression, data transformation, and semantic segmentation of visual data (Ruiz-del-Solar et al., 2018).

In this paper, we introduce a novel approach for utilising representation learning to perform dimensionality reduction on visual data in order to enhance the performance of deep reinforcement learning for visual navigation tasks. While training the DRL model, we train a convolutional variational autoencoder (VAE) (Pham & Le, 2020) with visual data collected in the environment to complete the task of generating depth maps from monocular RGB images. As DRL training progresses, the generation accuracy of Conv-VAE gradually increases. We replace the original RGB image with the latent code generated by the encoder part of the trained Conv-VAE. Using latent code as a state to participate in DRL training can greatly reduce the complexity of the neural network. Based on this method, we have designed three deep reinforcement learning models for UAV obstacle avoidance tasks in a simulated environment: single image model, continuous images model, and short-term memory model.

Experiments with these three models prove that after using VAE for visual dimensionality reduction, the DRL model has efficient convergence ability when faced with visual information, time-series visual information, and time-series vision and IMU fusion information. Experiments with the short-term memory model also prove that the model can make the monocular vision DRL algorithm perceive obstacles in the blind area of the current field of vision.

The key contribution of this paper is the development and evaluation of the learning state representations with VAE to train DRL models over monocular vision guided tasks. Specifically, the study proposes:

1. Three deep reinforcement learning models for monocular UAV visual obstacle avoidance.

2. A new idea for end-to-end monocular visual-spatial perception via the short-term memory state DRL model.

3. A new vision-based navigation algorithm that provides new ideas for robot control applications in indoor and other signal-constrained scenarios.

The rest of this paper is organised as follows: Section 2 describes related works on visual based navigation. Section 3 focuses on the 3 DRL models we have designed in this research. Section 4 explains the experiments and results of our study in detail. Finally, section 5 summarises the research results along with their limitations and proposes future research plans.

2. Related works

Based on improvements to discrete action spaces such as DQN (Silver et al., 2016), Double DQN (Hasselt et al., 2016) and Duelling DQN (Wang et al., 2016), the emergence of DRL algorithms using Actor Critic structure (Sutton et al., 1999), such as DDPG (Lillicrap et al., 2019), TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2019), enables DRL algorithms to select actions with arbitrary values in the continuous action space, so that the precision meets the requirements of robot control. However, such DRL algorithms often require a lightweight Actor Critic network to achieve fast and high-quality convergence. Most of the neural networks used in continuous action space DRL are only 3–4 layers deep. The Actor Critic networks’ structure used in the original literatures for the three methods (DDPG, TD3 and SAC) mentioned above is shown in Table 1.

Table 1. Comparison of network structures used in 3 different continuous actions space DRL methods.

Model name	Number of hidden layers	Number of neurons in 1st hidden layer	Number of neurons in 2nd hidden layer
DDPG	2	400	300
TD3	2	256	256
SAC	2	256	256

However, when faced with the problem of visual navigation, such a simple network structure is unable to cope with complex visual data because these networks cannot expand enough feature space to extract rich semantic information. To this end, researchers have proposed many solutions.

2.1. Manual dimensionality reduction

When the original version of DQN was used to train agents to play Atari games, researchers tried to manually compress the pixels of the game screen, crop the size of the game screen, and convert three-channel RGB images to single-channel grayscale images. There are many similar image dimensionality reduction methods in robotic visual navigation DRL field. In (Yue et al., 2019), the authors first convert the image into a grayscale image and then compress the image, and arrange the four most recent compressed images into a continuous image state. Using the image information processed in this way as the state to train the DQN model enables the robot to avoid two carton obstacles without collision.

There is more than one way of manual state dimensionality reduction. In (Anderson et al., 2019), the authors subtly reduced the state space from the current state of every pixel in an image frame to a set of nine one-hot encoded values. The essence of this method is to transfer the image information from continuous state space to discrete state space, so that the model only needs to understand the limited state to train the UAV to complete the simple runway navigation task.

Manually pre-processing images work well for simple tasks. However, in a complex training environment, it is difficult to manually process images to ensure that the key information in the images is not lost. Therefore, image processing methods based on machine learning are more used in complex visual navigation tasks.

2.2. Transfer learning

As image networks become more and more mature, various CNN networks have achieved impressive performance in computer vision fields such as image classification and object detection. This has also led to the emergence of research on transferring pre-trained image networks to deep reinforcement learning. The feature extraction effect of pre-trained image networks is often much better than that of manually dimensionally reduced images. Therefore, visual navigation work using such methods tends to achieve better performance in more complex environments.

In (Zieliński & Markowska-Kaczmar, 2021), the authors pre-trained Yolo for obstacle detection, and then transferred the pre-trained network into the A2C (Wu et al., 2017) model to train the underwater robot to pass through the gate. In (Devo et al., 2020), the author used multiple Resnet50 to form an object localisation network, pre-trained the network, and then transplanted the network into the IMPALA (Espeholt et al., 2018) model. This study successfully trained the ground robot to have certain exploration ability in mazes with different structures. Some studies do not use the existing image network structure but design their own CNN network pre-training model for navigation tasks. In (Ma et al., 2018), the authors designed a CNN-based saliency detection network and transferred it to an actor-critic (AC) architecture deep reinforcement learning. The study achieves performance that exceeds the state-of-the-art on UAV obstacle avoidance tasks.

Although transfer learning and DRL combined methods have produced excellent results, the pre-training of visual perception image networks requires a lot of time and relies on a large amount of manually labelled data. This approach essentially discards the inherent advantage of DRL algorithms in that they do not rely on datasets.

2.3. Representation learning for visual perception

Compared with transfer learning, representation learning can directly use the data (camera and other sensor data) collected by the DRL model from the training environment to train itself. An example of training a VAE to represent the input state of a DRL can be found in the work of (Caselles-Dupré et al., 2018), where researchers use the generative power of a VAE to remember and reuse past knowledge. This enables DRL agents to learn in continuously changing environments. In the field of robotic control, (Finn et al., 2016) proposed a Deep Spatial Autoencoder to generate compact state representations from images, thereby enabling DRL algorithms to train robotic arms for precise manipulation.

The study (Zhou et al., 2019) modelled the hippocampus as a visual perception module through a combination of representation learning and self-organising learning. The visual perception module perceives the environment topology and the robot head orientation from the environment image. Hierarchical reinforcement learning (Zhang et al., 2019) is used to train the robot to perform navigation tasks through the output of the visual perception module. Research (Shin et al., 2020) drives U-net to generate special optical flow topologies from environment images by using the DRL algorithm and the rewards generated during training. The output of the U-net will participate in the training of DRL as the state that determines the action of the DRL algorithm. The authors used this model to give UAVs the ability to avoid simple obstacles in real-world environments.

Microsoft recently used a framework called Cross-modal Variational Autoencoder (CM-VAE) to generate tightly bridged representations to model reality gaps (Bonatti et al., 2020). The perception module of the system compresses the input image into the above-mentioned low-dimensional representation, and outputs the position information of the gate that the UAV needs to pass through and the speed of the UAV. Using this representation model labelled dataset to train a supervised model for UAV navigation achieves human-level performance. This proves that VAE has great value as a DRL visual perception module. The study (Lange & Riedmiller, 2010) also proved the auxiliary ability of VAE model for DRL algorithm.

However, representation learning for dimensionality reduction does not seem to have the advantage of extracting effective features when obstacles have more diverse textures, shapes, and depth information. Therefore, in this paper we try to train Conv-VAE to generate depth maps from RGB images to filter the information in visual states that is irrelevant to the obstacle avoidance task. Moreover, a short-term memory state model is proposed to improve the VAE + DRL model's perception of depth information.

3. DRL models

3.1. Twin delayed deep deterministic policy gradient algorithm (TD3)

TD3 is a deterministic policy reinforcement learning algorithm suitable for high-dimensional continuous action spaces. Its optimisation objective is simple: maximise Q (${\text{S}}_{\text{t}},{\text{A}}_{\text{t}}$). That is, to find the corresponding actions under different states, so that the score Q of the interaction (${\text{S}}_{\text{t}},{\text{A}}_{\text{t}}$) between the agent and the environment is the highest. The TD3 algorithm contains a total of four networks, namely: Actor network, Actor target network, Critic network, and Critic target network. Actor network is responsible for the iterative update of the policy network parameters θ and selecting the current action ${\text{A}}_{\text{t}}$ according to the current state ${\text{S}}_{\text{t}}$. Actor target network is responsible for selecting the optimal next action ${\text{A}}_{\text{t}+1}$ based on the next state ${\text{S}}_{\text{t}+1}$ sampled in the replay buffer based on experience. The network parameters θ’ are periodically copied from the θ. Critic network is responsible for the iterative update of the value network parameter w and computing the current Q value Q(${\text{S}}_{\text{t}},{\text{A}}_{\text{t}}$). The target Q value is computed as: (1) $y=R+\gamma Q^{\mathrm{\prime }}({\text{S}}_{\text{t}+1},{\text{A}}_{\text{t}+1})$(1) The Q'(${\text{S}}_{\text{t}+1}$, ${\text{A}}_{\text{t}+1}$) in Equation (1) obtained by critic target network is named the target Q value. The network parameters w’ are periodically copied from w. The overall structure of TD3 is shown in Figure 1.

Figure 1. TD3.

3.2. Single image model

Based on TD3, we designed the first single image model, which also serves as the basic algorithmic framework for all three of our models. We trained a VAE to generate depth maps from input RGB images concurrently with TD3 training. Then we used the encode network of VAE to reduce the dimensionality of the monocular RGB image to latent code. This latent code participates in the training of TD3 as the state ${\text{S}}_{\text{t}}$. The current state ${\text{S}}_{\text{t}}$ is input into the actor network, and the action ${\text{A}}_{\text{t}}$ is obtained. At the same time, the environment feedbacks a reward ${\text{R}}_{\text{t}}$. After executing ${\text{A}}_{\text{t}}$, the UAV reaches a new state ${\text{S}}_{\text{t}+1}$. The agent stores the data generated by each interaction in the replay buffer. After that, we sample (${\text{S}}_{\text{t}}$, ${\text{S}}_{\text{t}+1}$, ${\text{r}}_{\text{t}}$, ${\text{A}}_{\text{t}}$) from the replay buffer and use the critical network to evaluate its Q value. We use this Q-value to scale the actor network and update the actor network in the direction that maximises the Q-value. The entire workflow of the single image model is shown in Figure 2.

Figure 2. Single image model.

The Conv-VAE structure we used is shown in Figure 3. The input is a monocular RGB image, the size is $178\times 72$, and the number of channels is 3. The structure consists of an encoder and a decoder. The encoder consists of a four-layer convolutional network, and the convolution kernels in each convolutional layer are ($4\times 4$). The latent code generated by the encoder will be reconstructed into ($1024\times 1\times 3$) data as the input to the decoder. To successfully decode this data into a $178\times 72$ depth map with 1 channel, the decoder's convolution kernels are ($5\times 7$), ($6\times 8$), ($7\times 8$) and ($8\times 6$).

Figure 3. Convolutional variational autoencoder.

The encoder in Conv-VAE is trained to convert RGB images into 32 latent codes. These 32 variables will participate in the use and update of actors and critic networks as states. The actor network consists of four fully connected layers. Its input is the 32 variables output by the encoder. The output is two values in the range −1–1, representing the speed of the UAV in the y direction (left and right) and the z direction (up and down).

The critic network consists of three fully connected layers. Its role is to estimate the Q value that can be obtained when a certain action value is selected in a certain state. Therefore, its input is a combination of states and actions, resulting is 34 variables. Furthermore, in order to mitigate the issue of overestimation of the Q-value, the TD3 algorithm employs a dual output layer within the critic network. This layer generates two distinct Q-values, the smaller one being utilised for the update of the critic network. The structure of the actor and critic network is shown in Figure 4.

Figure 4. Structure of actor net and critic net in the model.

3.3. Continuous images model

The workflow for continuous images model is similar to the single image models. The difference is that the input to the actor and critic networks is no longer the single latent code converted from a single image. Rather, it is a combination of latent codes converted from four consecutive RGB images on the route the UAV travelled. That is, a 2D array of shape $4\times 32$. The entire workflow of a single image model is shown in Figure 5.

Figure 5. Continuous images model.

To enable the actor and critic network to understand the timing relationship between each latent code, we let the 2D state array first pass through a single-layer LSTM with a hidden size of 8. The output of the LSTM will be a 32-length code. The new structure of actor net and critic net is shown in Figure 6.

Figure 6. Structure of actor net and critic net in the model with LSTM layer.

3.4. Short-term memory model

Using only four consecutive image information as input can only enable the agent to understand the temporal relationship between images during the learning process. However, spatial relationships are also very important in visual navigation tasks. To this end, we design a new state construction method, called short-term memory state. To intuitively include the position and pose changes between each image in the state, we fuse the latent code produced by the encoder with the data produced by the inertial measurement unit (IMU). As a common hardware module of UAVs, IMU can obtain the orientation, velocity, and acceleration information of UAVs in real time. In this model, we use four consecutive frames to form a short-term memory state and obtain 10 pose data from the IMU while the camera obtains each image frame. The data includes the velocity and acceleration of the UAV on the x, y, and z axis of inertial frame of reference and a quaternion ($\cos \frac{O_{\omega }}{2}$, ($O_x$, $O_y$, $O_z$)$\sin \frac{O_{\omega }}{2}$) that represents the orientation. The $O_{\omega }$in this formula represents the rotation angle, and ($O_x$, $O_y$, $O_z$) is a three-dimensional vector representing the rotation axis. These 10 variables and the 32 variables compressed by VAE will form a code of length 42 to represent the current state of each step. The current state of the agent $S_t$ and the three successive states that have passed before will be combined into the short-term memory state ($S_t$, $S_{t-1}$, $S_{t-2}$, $S_{t-3}$). The complete state construction process is shown in Figure 7.

Figure 7. Short-term memory state construction.

The overall workflow of the short-term memory model is shown in Figure 8. It is also similar to the continuous images model. The difference is that the states result from the short-term memory state construction process above. Since the shape of the short-term memory state is 4 × 42, the input size of the LSTM layer of the actor and critic network will become 4 × 42.

Figure 8. Short-term memory model.

4. Experiments and results

The following experiments are designed to observe the training process of the three models: single image model, continuous images model and short-term memory model. We evaluate the three model training procedures in a simulation environment made by Airsim (Shah et al., 2017) and test the trained models in the same simulation environment and a rearranged environment.

4.1. Environment settings

The specific hardware and software settings of our experiment environment and work station are shown in Table 2.

Table 2. Environment Settings.

CPU	AMD Ryzen Threadripper 3970X 3.69GHz
GPU	2 x NVIDIA RTX3090
RAM	64 GB
Operating System	Windows 10
Program Language	Python 3.7
ML Library	Pytorch 1.9
CV Library	OpenCV 4.4
Simulator	Airsim 1.2.8
Game Engine	Unreal Engine 4.24.3

4.2. Simulation environment

We used UE4 and Airsim to construct a simulation environment for UAV flight training and testing. The simulation environment is a corridor that is 36 metres long, 5 metre wide and 5 metres high. Obstacles of different shapes appear in succession every 4 metres in the corridor as shown in Figure 9. The UAV takes off from the beginning of the corridor, maintains a forward speed of 1 m/s in the x direction, and is trained by our visual navigation models to control the speed in the y and z directions for obstacle avoidance training.

Figure 9. Simulation environment.

4.3. Reward shaping

To compare the performance of the three models, we use the same reward function for each model. Our reward function is very simple and falls into two cases as shown in Table 3. When a collision occurs after the action is taken, a penalty of −2.0 is fed back. When there is no collision after the action is taken, a reward of 0.2 is obtained.

Table 3. Reward function.

Situation	Reward
Collision	−2.0
Action without collision	0.2

4.4. Training processes

In the training phase, we used the three models introduced in Section 3 to train the UAV agent. The agent was trained to perform obstacle avoidance tasks from the same starting point in the same simulation environment. According to the reward shaping discussed in Section 4.3, if the agent can avoid more and more obstacles as the training progresses, the reward will increase with training. To verify that the models maximised the reward, we recorded the average reward of every 20 flights during the training. The reward growth trends of the three models are shown in Figure 10.

Figure 10. Reward comparison of single image model, continuous image model and short-term memory model during training.

As can be seen from Figure 10, the rewards obtained by the three models as the training progresses are generally on the rise and gradually stabilise within a certain range. After 7,000 updates, the average reward of a single image model has gradually stabilised, but it is still not completely stable. The average reward of the continuous image model has basically stabilised at around 0. In contrast, the short-term memory model far exceeds the previous two models in terms of both the growth rate of the reward and the final maximum reward level. We believe that this is because the IMU data in the short-term memory state accurately represents the position and pose relationship between continuous monocular visual information, thereby helping the model to better understand the environment through the state.

In order to more intuitively monitor the obstacle avoidance ability of the agent, the convergence of the models improves., we also recorded the average moving distance of the last 20 flights during the training process. Results are shown in Figure 11.

Figure 11. Average moving distance comparison of single image model, continuous image model and short-term memory model during training.

The training results show that after 7,000 updates in a 36-metre-long training environment with 9 obstacles, the short-term memory model can achieve an average flight distance of about 18 metres. In comparison, the continuous image model can achieve an average flight of about 14 metres, while the single image model can only reach an average flight distance of about 10 metres This proves that the short-term memory model can indeed significantly improve the obstacle avoidance ability of the agent after training. We think that this is because the short-term memory state can clearly allow the agent to learn the position and posture information of obstacles in the route passed in a short time, so that it is easier to avoid the side obstacles in the blind spot of the vision. To verify this conjecture, we designed a special experiment during model testing to record the number of times the three models collided with the side obstacles during 100 test flights. The details of this experiment will be shown in sub-section 4.7.

4.5. Validating the role of VAE

The purpose of this experiment is to verify that using VAE for visual perception can indeed achieve better performance than directly combining the image network and DRL. We chose the combination of MobileNet V2 (Sandler et al., 2018) and TD3 model as the control group and the single image model to compare. MobileNet V2 is a lightweight and expressive image recognition network proposed by Google in 2018. It achieves higher or similar mAP and faster training speed than SSD300 (Liu et al., 2016), SSD512 (Liu et al., 2016),Yolo V2 and MobileNet V1 (Howard et al., 2017) in object detection task with fewer parameters. Since Mobile Net V2 has a lightweight structure (a relatively shallow and narrow model in Image Nets) while having good performance. In image networks, this model is most suitable to be combined with the DRL model. Therefore, we use it as a representative of the DRL + image network model to compare with our models.

In this experiment we train the MobileNet V2 + TD3 model for 7,000 epochs in the same environment with the same reward function and simultaneously record the model's reward change and the average flight distance change. Here we use the structure of the original Mobile Net V2 in the paper, as shown in Table 4. We directly graft Mobile Net to the first layer of Actor and Critic of TD3. The size of the output k of the Mobile Net is set to 32, in order not to change the original structure of the Actor Critic network (the input size is 32). The Mobile Net will be updated as Actor and Critic Net are updated.

Table 4. MobileNet V2 structure.

Input	Operator	t	c	n	s
${224}^2\times 3$	conv2d	–	32	1	2
${112}^2\times 32$	bottleneck	1	16	1	1
${112}^2\times 16$	bottleneck	6	24	2	2
${56}^2\times 24$	bottleneck	6	32	3	2
${28}^2\times 32$	bottleneck	6	64	4	2
${14}^2\times 64$	bottleneck	6	96	3	1
${14}^2\times 96$	bottleneck	6	160	3	2
$7^2\times 160$	bottleneck	6	320	1	1
$7^2\times 320$	conv2d $1\times 1$	–	1280	1	1
$7^2\times 1280$	avgpool 7$\times 7$	–	–	1	–
$1\times 1\times 1280$	conv2d $1\times 1$	–	k	–

As can be seen from Figure 12 and Figure 13, after 7,000 updates, the rewards and average flight distance of the Mobile Net V2 + TD3 model still fluctuate greatly. It seems that the model does not converge very quickly. At the same time, the maximum average reward and moving distance it can achieve are only comparable to the single image model.

Figure 12. Reward comparison of single image model and MobileNet V2 + TD3.

Figure 13. Average moving distance comparison of single image model and MobileNet V2 + TD3.

4.6. Obstacle avoidance test

In order to test the obstacle avoidance ability of the three models after training, we use them to control the agent to fly 100 times in the simulation environment. During flights, we will record the number of times each obstacle is avoided and calculate the average flight distance and the average obstacle avoidance rate of all obstacles after 100 flights. We define the obstacle avoidance rate as the probability that the UAV can finally avoid an obstacle through a series of actions after detecting it. In the UAV test scene, the obstacles are arranged in the order $\{O_1,O_2\dots ..,O_n,O_{n+1}\dots \dots \}$. We assume that the number of times the UAV avoids the obstacle $O_n$ is $t_n$, and the number of times it avoids the obstacle $O_{n-1}$ is $t_{n-1}$, then the avoidance rate of the UAV for the $O_n$ is $R_n$ =  $t_n$/ $t_{n-1}$. We can use Equation (2) to calculate the average obstacle avoidance rate for the UAV to fly M times in an environment with N obstacles: (2) $R=\frac{\frac{t_1}{M}+{\sum }_{n=2}^N\frac{t_n}{t_{n-1}}}{N}$(2) Test results of the three models and the baseline model (Mobile Net V2 + TD3) are shown in Table 5, Table 6, Table 7 and Table 8.

Table 5. Test results of short-term memory model.

Short-term Memory Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	100	100%
Obstacle 2	97	97%
Obstacle 3	93	95.0.9%
Obstacle 4	70	75.3%
Obstacle 5	62	88.6%
Obstacle 6	51	82.2%
Obstacle 7	39	76.5%
Obstacle 8	27	69.2%
Obstacle 9	25	92.6%
Average flight distance	22.6 metres
Average obstacle avoidance rate	86.4%

Table 6. Test results of continuous images model.

Continuous Images Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	96	96.0%
Obstacle 2	96	100%
Obstacle 3	63	65.7%
Obstacle 4	57	90.5%
Obstacle 5	53	93.0%
Obstacle 6	47	88.7%
Obstacle 7	31	66.0%
Obstacle 8	20	64.5%
Obstacle 9	14	70.0%
Average flight distance	19.8 metres
Average obstacle avoidance rate	81.6%

Table 7. Test results of single image model.

Single Image Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	98	98.0%
Obstacle 2	88	89.8%
Obstacle 3	57	64.8%
Obstacle 4	46	80.7%
Obstacle 5	42	91.3%
Obstacle 6	37	88.1%
Obstacle 7	24	64.9%
Obstacle 8	14	58.3%
Obstacle 9	0	28.6%
Average flight distance	14.7meters
Average obstacle avoidance rate	73.8%

Table 8. Test results of Mobile Net + TD3 model.

Mobile Net + TD3 Model
	Pass Times	Obstacle avoidance rate
Obstacle 1	100	100%
Obstacle 2	57	57.0%
Obstacle 3	53	93.0%
Obstacle 4	33	62.3%
Obstacle 5	21	63.6%
Obstacle 6	18	85.7%
Obstacle 7	4	22.2%
Obstacle 8	0	0%
Obstacle 9	0	0%
Average flight distance	8.4 metres
Average obstacle avoidance rate	53.8%

The test results show that the trained short-term memory model achieves an average flight distance of 22.6 metres in 100 flights. In contrast, the average flight distances of the continuous images model and the single image model are 19.8 and 14.7 metres, respectively. In terms of the average obstacle avoidance rate, the short-term memory model reached 86.4%, the continuous image model reached 81.6% and the single image model reached 73.8%. The Mobile Net V2 + TD3 model can only fly an average of 8.4 metres and achieve an average obstacle avoidance rate of 53.8%. It can be seen that the obstacle avoidance ability of our three models is stronger than the traditional image network + DRL method. The performance of the short-term memory model is indeed better than the other two models. If we compare the obstacle avoidance rate of each obstacle, we find that the short-term memory model has a significantly higher obstacle avoidance rate at obstacle 3 and obstacle 9 than the other two models.

We speculated this is because when the UAV passes through such door frame-type obstacles, there are obstacles in the blind areas of both left and right sides, and the UAV has a greater probability of colliding with the side obstacles. The short-term memory model does increase the UAV's ability to avoid side obstacles.

To verify the generality of the trained models, we rearrange the obstacles in the training environment to construct a new test environment. In this environment, the order of obstacles becomes: 7-5-8-9-2-1-6-4-3. As shown in Table 9, the average obstacle avoidance rate and flight distance decreased for all three models. Only the short-term memory model still has a 70% obstacle avoidance rate in the new environment. This proves that our model has some degree of overfitting for training environment.\

Table 9. Test results in rearranged environment.

	Short-term Memory Model		Continuous Images Model
	Pass Times	Obstacle avoidance rate	Pass Times	Obstacle avoidance rate
Obstacle 7	98	98.0%	97	97.0%
Obstacle 5	87	88.8%	83	85.6%
Obstacle 8	84	96.6%	69	83.1%
Obstacle 9	41	48.8%	32	46.4%
Obstacle 2	28	68.3%	22	68.8%
Obstacle 1	23	82.1%	13	59.1%
Obstacle 6	13	52.2%	10	76.9%
Obstacle 4	8	61.5%	3	30.0%
Obstacle 3	3	37.5%	1	33.3%
Average flight distance	14.7 metres		12.6 metres
Average obstacle avoidance rate	70.9%		64.5%
	Single Image Model		Mobile Net + TD3 Model
	Pass Times	Obstacle avoidance rate	Pass Times	Obstacle avoidance rate
Obstacle 7	98	98.0%	80	80.0%
Obstacle 5	88	89.8%	65	81.3%
Obstacle 8	74	84.1%	30	46.2%
Obstacle 9	47	63.5%	18	60.0%
Obstacle 2	26	55.3%	14	77.8%
Obstacle 1	9	34.6%	12	85.7%
Obstacle 6	5	55.6%	7	58.3%
Obstacle 4	1	20.0%	1	14.3%
Obstacle 3	1	100%	0	0%
Average flight distance	10.7 metres		9.8 metres
Average obstacle avoidance rate	66.8%		55.9%

4.7. Validating perception of side obstacles

To verify the speculation of the previous experiment, we counted the number of times the UAV collided with obstacles in the blind area of the side view in 100 flying tests. When a collision occurs, we consider it a side collision if there is no obstacle close enough to the UAV's front-facing camera.

We can see from Table 10 that the short-term memory model does experience fewer side collisions over 100 flights. This also proves our speculation in the previous experiment.

Table 10. Side collision count in 100 flights of 3 models.

Model Name	Short-term Memory Model	Continuous Images Model	Single Image Model	Mobile Net + TD3
Side Collision Count	13	26	24	26

5. Conclusion

In this study, we aim to solve the dilemma that the deep reinforcement learning visual navigation system relies on the complex image network, while the complex network is not conducive to the convergence of the DRL model. For this, we use a method that uses Conv-VAE as the visual perception module and the DRL model to be trained simultaneously. Based on this method, a single image model, a continuous images model and a short-term memory model are proposed. Experiments show that the convergence and performance of these three models are better than DRL models that directly use image networks as actor and critic networks. Moreover, the short-term memory state including the spatiotemporal relationship can improve the ability of the DRL monocular vision navigation model to avoid side obstacles.

However, our study still has certain limitations. First, the single image model and continuous images model are less able to avoid particularly narrow door frame obstacles like obstacle 3 and 9. This situation is greatly improved when using the short-term memory model. We believe that this is because the short-term memory model has a stronger ability to perceive side collisions. However, even the short-term memory model did not achieve significantly better obstacle avoidance on obstacles 3 and 9 in the rearranged test environment. This implies our model has some degree of overfitting to the training environment. This is not only reflected in the short-term memory model's ability to avoid specific obstacles, but also in the degraded performance of the three models in the reconstructed test environment. In the future research, we plan to use environments that can randomly change for training, and allow the model to gain more diverse experience with obstacles during training. Besides, we also need to design a more reasonable noise scheme for visual navigation tasks. Secondly, the performance of the model when faced with obstacle 7 also has a significant drop. To this end, we examine Conv-VAE for the depth maps generated when facing each obstacle. It is found that the depth map generated by Conv-VAE cannot effectively display the outline of obstacle 7. We believe that this is because the surface of obstacle 7 is very smooth and has stronger light reflection ability, so that the perception module cannot effectively learn its depth information. In the future research, we will focus on learning state representations that have better perception capabilities for objects of different materials and textures.

Disclosure statement

No potential conflict of interest was reported by the author(s).