A Visual Navigation System for UAV under Diverse Illumination Conditions

ABSTRACT

The high-precision navigation and positioning ability of UAV (Unmanned Aerial Vehicles) is the key factor to reflect its degree of automation. Visual navigation based on image matching has become one of the important research fields for UAV to realize autonomous navigation, because of its low cost, strong anti-jamming ability, and good location result. However, the visual quality of images captured by UAV will be seriously affected by some factors, such as weak illumination conditions or insufficient performance of its sensors. Resolving a series of degradation of low-light images can improve the visual quality and enhance the performance of UAV visual navigation. In this paper, we propose a novel fully convolutional network based on the Retinex theory to solve the degradations of low-light images captured by UAV, which can improve the visual quality of the images and visual matching performance effectively. At the same time, a visual navigation system is designed based on the proposed network. Extensive experiments demonstrate that our method outperforms the existing methods by a large margin both quantitatively and qualitatively, and effectively improves the performance of the image matching algorithms. The visual navigation system can successfully realize the self-localization of UAV under different illumination conditions. Moreover, we also prove that our method is also effective in other practical tasks (e.g. autonomous driving).

Introduction

To realize long-term autonomous flight, UAV(Unmanned Aerial Vehicles) need an autonomous navigation and positioning system with high precision. At present, the navigation system of UAV mainly includes Global Navigation Satellite System (GNSS), Inertial Navigation System (INS), Radio Navigation System (RNS), and so on. The GNSS includes the GPS of America, the GLONASS of Russia, the Galileo of Europe, and the BDS of China(Dong (Dong and Department 2017)). The GNSS has a great advantage in location accuracy and the error will not accumulate over time. However, the disadvantage is poor autonomy, it is easy to be disturbed by external factors. Therefore, it can not meet the needs of high-precision positioning in some scenarios; once the UAV loses the signal due to external interference, it will not be able to accomplish the mission.

With the rapid development of computer vision and artificial intelligence technology, a new navigation method, visual navigation, has emerged. Visual navigation has the advantages of good autonomy, high reliability, and low cost. In recent years, with the continuous updating and improvement of the concepts, theories, and methods for visual navigation, it has become one of the important research fields of UAV autonomous navigation.

The research of visual navigation is initially inspired by the Terrain Contour Matching Guidance System in cruise missiles(Golden (Golden 1980)). In the following decades, there have been extensive visual navigation methods designed based on visual odometry(Comport (Comport, Malis, and Rives 2010); Strydom (Strydom, Thurrowgood, and Srinivasan 2014); Wang (Wang et al. 2018b); Depaola (Depaola et al. 2018)), simultaneous localization and mapping(SLAM) (Alpen et al. 2010), image matching (Xu et al. 2013), etc. Most visual navigation methods based on image matching use pre-stored reference images to match the image captured by the UAV with the reference image to obtain the real-time location information. However, the flying environment of the UAV is not necessarily an ideal environment with sufficient illumination. When the UAV flies on cloudy days or at night, the camera will capture low-light images due to the reduction of light, which will seriously affect the image quality, such as low contrast and low visibility. At the same time, due to the limitations of laws and regulations, manned aircraft fly much more frequently during the day than at night, which will greatly reduce the flight space of UAVs. Accordingly, the most suitable flight time for UAVs is the night with a small number of manned aircraft. Using the low-light image as the input of the high-level vision algorithm will also reduce the performance of the algorithms (e.g. image matching, object detection, etc.), as shown in Figure 1. Therefore, converting low-light aerial images into high-quality normal light images can not only improve the visual quality of images but also improve the performance of UAVs in some high-level visual tasks, such as object detection (Jiang et al. 2021), semantic segmentation (Ke et al. 2020), etc, it can also enable UAV to be used in a wider range of fields, such as emergency rescue, environmental monitoring, etc.

Figure 1. Comparison of matching results and detection results between low-light image and the enhanced image obtained by our method.

Over the past few decades, there have been a large number of methods to enhance degraded images captured by insufficient illumination conditions. HE-based methods improve the global contrast by changing the histogram distribution of the weakly illuminated image into a uniform distribution, such as Brightness Preserving Dynamic HE (Ibrahim 2007) (Ibrahim and Kong 2007), Contrast-limiting adaptive HE (Pisano 1998) (Pisano et al. 1998), etc. The methods based on Retinex theory (Land 1977) realize image enhancement by estimating the illumination and reflectance of the image and adjusting the dynamic range of the pixels of the illumination, such as MSRCR (Jobson 1997) (Jobson, Rahman, and Woodell 1997), SRIE (Fu 2016) (Fu et al. 2016), etc. The learning-based methods (Shen et al. 2017) achieve low-light image enhancement by designing efficient models. Although these methods have made great progress in improving image contrast, most methods can not properly enhance the image contrast, the results are often over-/under-enhanced or blurred. Therefore, there is a lot of room for low-light image enhancement.

In this paper, we propose a new fully convolutional network by combining the Retinex theory with a convolutional neural network to solve the degradations of low-light images captured by UAV and improve the performance of subsequent image matching algorithms. We design a visual navigation method based on the proposed low-light image enhancement network. Extensive experiments demonstrate that the proposed image enhancement method can better improve the contrast and visual quality of low-light images. Compared with other image enhancement methods, our method can improve the performance of image matching algorithms more effectively. It can realize the positioning requirements of UAV under diverse illumination conditions.

The Proposed Low-light Image Enhancement Method

In this section, we will introduce the details of our proposed image enhancement method, including the network structure, the loss function, the dataset for training, and the implementation details.

Network Architecture

We propose a new fully convolutional network based on the Retinex theory, which consists of two subnetworks, Decomposition-Net (Decom-Net) and Enhancement-Net (Enhance-Net). The proposed low-light image enhancement pipeline is shown in Figure 2. The Decom-Net decomposes the input image into an illumination and reflectance map and suppresses the noise in the reflectance at the same time. Enhance-Net takes the output of Decom-Net as the input to enhance the contrast and brightness of the illumination map. Therefore, the proposed method can improve the contrast and suppress the noise in the low-light image, and obtain the enhanced result with better visual quality.

Figure 2. The proposed low-light image enhancement pipeline. The Decom-Net decomposes the input image into an illumination map and reflectance map, and the Enhance-Net brightens up the illumination map. The reflectance map and illumination map of the low-light image are used as the input of Enhance-Net. The decompositions of normal-light images do not participate in the Enhance-Net training stage.

Decom-Net

Residual network (He 2016) (He et al. 2016) has been widely used in many computer vision tasks and achieved excellent results. Benefiting from the jump connection structure, the residual network can make the deep neural network easier to optimize during the training stage, and will not cause gradient disappearance. So, we use multiple residual components to form Decom-Net.

Decom-Net contains eight 3 × 3 convolutional layers with stride 2 and a Rectified Linear Unit(ReLU), the size of the input and output feature map of each convolutional layer are equal. By using the jump connection structure, the output result of the $i-1$ convolutional layer and the $i$ convolutional layer are combined as the input of the $i+1$ convolutional layer($i=1,\dots ,5$). The network architecture of Decom-Net is shown in Figure 3.

Figure 3. The network architecture of Decom-Net.

The input of Decom-Net are the low-light image and the normal-light image $S_{normal}$, and the output results are the illumination map($I_{low}$/$I_{normal}$) and the reflectance map($R_{low}$/$R_{normal}$) of the low-light image and normal-light image, respectively. But other methods based on the Retinex theory do not suppress the noise of the reflectance map in the process of decomposition. This will cause the final enhancement result to be affected by the noise in the reflectance. Benefit from the network architecture and the loss function, Decom-Net can perform decomposition and noise suppression at the same time, it can get better decomposition results.

Note that the illumination map and reflectance map of the normal-light image neither participate in the follow-up training, but only provide references for the decomposition of the low-light image. The illumination map and reflectance map of the low-light image are the input of Enhance-Net.

Enhance-Net

U-Net (Ronneberger 2015) (Ronneberger, Fischer, and Brox 2015) has achieved excellent results in a large number of computer vision tasks due to its excellent structural design. In the field of low-light image enhancement, a large number of networks have adopted the U-Net as the main architecture or a part of it. However, U-Net uses multiple max-pooling layers in the feature extraction stage, but the max-pooling layer will lead to the loss of a large amount of feature information. SoSpringenberg (2015) (Springenberg et al. 2014) replaces the max-pooling layers with stride convolutional layers, which will slightly increase the network parameters, but also improve the accuracy. Inspired by that, we use the 2 × 2 convolutional layers with stride 2 to replace the max-pooling layers in the U-Net, which is beneficial for the network to learn more feature information and maintain translation invariance. So that our network can better improve the contrast of the reflectance map.

We also utilize the multi-scale fusion, concatenate the output of each deconvolutional layer in the expansive stage to reduce the loss of feature information.

Enhance-Net contains seven convolutional blocks, each of them contains two 3 × 3 convolutional layers with stride 1 to maintain the same size of the front and the back feature maps; the first three convolutional blocks are followed by a stride convolution to perform the down-sampling; the last three convolutional blocks are followed by a deconvolutional layer to perform the up-sampling. Then, utilizing multi-scale fusion, the output of the seventh convolutional block and the output of the last three deconvolutional layers are concatenated as the input of the next convolutional layer, which can maximize the combination of context information and reduce the loss of feature information. Finally, the enhanced illumination map is obtained through a 3 × 3 convolutional layer, Each convolutional block in the Enhance-Net is followed by a ReLU. The network architecture of Enhance-Net is shown in Figure 4.

Figure 4. The network architecture of Enhance-Net.

After obtaining the decomposition result of Decom-Net and the enhancement result of Enhance-Net, the output of the two subnetworks is combined by element-by-element multiplication as the final result, which can be described as:

(1) $\hat{I}={\hat{I}}_{low}\circ R_{low}$(1)

Where $R_{low}$ is the reflectance map obtained by Decom-Net after decomposing the low-light image, ${\hat{I}}_{low}$ is the output of Enhance-Net, and $\circ$ represents the element-by-element multiplication operation. The decomposition result of the whole network for a low-light image is shown in Figure 5.

Figure 5. Example of the low-light image decomposition result. (a) is the input image, (b) is the reflectance map generated by Decom-Net, (c) is the illumination map generated by Enhance-Net.

Loss Function

In the training phase, Decom-Net and Enhance-Net are trained separately, so the whole loss function consists of two parts: the decomposition loss ${\mathcal{L}}_{Decom}$ and the enhancement loss ${\mathcal{L}}_{Enhance}$.

Decomposition Loss

The decomposition loss consists of three components: reconstruction loss ${\mathcal{L}}_{recon}$, invariable reflectance loss ${\mathcal{L}}_r$, and illumination smoothness loss ${\mathcal{L}}_i$:

(2) ${\mathcal{L}}_{Decom}={\mathcal{L}}_{recon}+{\lambda }_r{\mathcal{L}}_r+{\lambda }_i{\mathcal{L}}_i$(2)

Where ${\lambda }_r$ and ${\lambda }_i$ are the coefficients to balance the consistency of the reflectance map and the smoothness of the illumination map.

Since the reflectance map and illumination map decomposed by normal-light image can be used as references for low-light image decomposition, we use the Mean Square Error (MSE) loss to represent the reconstruction loss to better realize the decomposition of the low-light image.

(3) ${\mathcal{L}}_{recon}=\frac{1}{N}\sum _{i=1}^N\sum _{j=low,normal}{\lambda }_{re}{\left|\left|R_j\circ I_j-S_j\right|\right|}^2$(3)

We also use MSE loss to form the invariable reflectance loss to constrain the decomposition process and denoise the reflectance map;

(4) ${\mathcal{L}}_r=\frac{1}{N}\sum _{i=1}^N{\left|\left|R_{low}-R_{normal}\right|\right|}^2$(4)

To recover the structure information, we use the weighted TV loss in (Wei et al. 2018) as the illumination smoothness loss;

(5) ${\mathcal{L}}_i=\sum _{i=low,normal}\left|\left|\mathrm{\nabla }{I}_i\circ exp\left(-{\lambda }_{tv}\mathrm{\nabla }{R}_i\right)\right|\right|$(5)

Enhancement Loss

To make the result more realistic and natural, we also use reconstruction loss ${\mathcal{L}}_{recon}$, illumination smoothness loss ${\mathcal{L}}_i$, and structural similarity loss ${\mathcal{L}}_{ssim}$in this stage to obtain a better result.

(6) ${\mathcal{L}}_{Enhance}={\mathcal{L}}_{recon}+{\mathcal{L}}_i+{\mathcal{L}}_{ssim}$(6)

The illumination smoothing loss is consistent with the corresponding part of decomposition loss, and the reconstruction loss adopts the combination of MSE loss and L1 loss to obtain a better reconstruction result and further noise suppression.

(7) ${\mathcal{L}}_{recon}={\lambda }_{re1}\frac{1}{N}\sum _{i=1}^N{\left|\left|\hat{I}\circ R_{low}-S_{normal}\right|\right|}^2+{\lambda }_{re2}{\left|\left|\hat{I}\circ R_{low}-S_{normal}\right|\right|}^1$(7)

The structural similarity loss adopts SSIM loss to gain better detail expression ability.

(8) ${\mathcal{L}}_{ssim}=1-SSIM\left(\hat{I}\circ R_{low},S_{normal}\right)$(8)

Dataset and Implementation Details

It is impractical to collect a large-scale paired low/normal-light image dataset under real conditions, all the existing aerial image datasets contain normal-light images. As far as we know, there is no paired low/normal-light aerial image dataset. Therefore, we collect 6000 high-quality aerial images in the Visdrone-2018 dataset (Zhu 2018) (Zhu et al. 2018) and follow the previous research (Lore 2017) (Lore, Akintayo, and Sarkar 2017) to darken the images So that we build a large-scale paired low/normal-light aerial image dataset for training. All images are resized to 600 × 400 and convert to JPG format.

At the same time, we randomly select 120 images from the dataset proposed in (Long 2017) (Long et al. 2017) as the test images for subsequent experiments. The image size, format, and darkening method are consistent with the training data.

Low-Light Image Synthesis

The degradation of low-light image is mainly reflected in its low contrast. To better simulate the real weakly illuminated image, we apply Gamma Correct to each channel of the normal-light image to change the contrast. The whole process can be described as:

(9) $I_{\mathrm{l}\mathrm{o}\mathrm{w}}=\mathrm{A}\times I_{normal}^{\gamma }$(9)

Where $I_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}}$ represents a normal-light image, A is a constant determined by the maximum pixel intensity in the image, and γ obeys a uniform distribution $\mathrm{U}\left(3,5\right)$. An example of synthetic low-light images is shown in Figure 6.

Figure 6. Example of synthetic low-light image.

Implementation Details

Our implementation is done with Pytorch, the network is trained for 50 epochs on an Nvidia GTX1080Ti GPU with the proposed dataset. We use the ADAM (Kingma 2014) (Kingma and Ba 2014) optimizer with the parameters are $\beta 1$=0.9,$\beta 2$=0.999, and $lr$=0.0001. The batch size and patch size are set to 16 and 96, respectively. The $\lambda r$, $\lambda i$, and $\lambda re$ 0.01 in Eq.2(2) ${\mathcal{L}}_{Decom}={\mathcal{L}}_{recon}+{\lambda }_r{\mathcal{L}}_r+{\lambda }_i{\mathcal{L}}_i$(2) and Eq.3 are all set to 0.01, $\lambda tv$ in Eq.5(5) ${\mathcal{L}}_i=\sum _{i=low,normal}\left|\left|\mathrm{\nabla }{I}_i\circ exp\left(-{\lambda }_{tv}\mathrm{\nabla }{R}_i\right)\right|\right|$(5) is set to 10, $\lambda re1$ and $\lambda re2$ in Eq.7(7) ${\mathcal{L}}_{recon}={\lambda }_{re1}\frac{1}{N}\sum _{i=1}^N{\left|\left|\hat{I}\circ R_{low}-S_{normal}\right|\right|}^2+{\lambda }_{re2}{\left|\left|\hat{I}\circ R_{low}-S_{normal}\right|\right|}^1$(7) are set to 0.5 and 1.5, respectively.

Visual Navigation System

In this section, we will introduce the visual navigation system based on the proposed low-light image enhancement method.

Image Definition Evaluation

When carrying out visual navigation tasks, UAV may be affected by factors such as insufficient lighting conditions or lack of sensor performance, so it is necessary to enhance the low-light images captured. However, not all images need to be enhanced, so it has to judge whether the image has weak illumination conditions in the pre-processing stage. In this paper, we use the Laplacian gradient function to evaluate the image definition, which is defined as follows:

(10) $Df=\hspace{0.17em}\sum y\sum x\left|Gx,y\right|$(10)

Where $G\left(x,y\right)$ is the convolution of the Laplacian operator at the pixel $\left(x,y\right)$. For low-light images, the second-order partial derivative is small, so the value of the Laplacian gradient function will be small. For normal-light images, the second-order partial derivative is large, so the value of the Laplacian gradient function will be large, as shown in Figure 7. Therefore, the threshold can be set by calculating the Laplacian gradient function value of each image. If it is less than the threshold we set, the image needs to be enhanced, if it is larger than the threshold, the image does not need image enhancement processing.

Figure 7. The Laplacian gradient function values of images under different illumination conditions.

Visual Navigation Pipeline

To achieve accurate positioning in the visual navigation, reference images need to be stored in advance, the geographical location information of the reference image is inputted into the database as the positioning reference. In the process of visual navigation, UAV need to judge whether the captured real-time image is under weak illumination condition firstly. If there is no lack of illumination in the image, there is no need to enhance it. If there is a lack of light in the image, the proposed image enhancement method is used to enhance it. So that the enhanced result can achieve better visual quality, and more abundant feature information can be extracted for the subsequent image matching algorithm.

The enhanced result/normal-light image and the pre-stored reference image are used as the input of the high-performance image matching algorithm. After successfully realizing the matching between the enhanced result/normal-light image and the reference image through the image matching algorithm, the location of the UAV needs to be determined. By searching the image in the reference image database with the highest matching points to the enhanced result or normal-light image is used as the current position reference image, and extract the geographical location information of the reference image as the current UAV location information. In addition, the reference images we chose are the satellite images, the reference image have a larger visual range and larger size (7200 × 3600) than the aerial images, so we split the reference image into several small-size reference images and match the aerial image with small-size images in turn, which helps to reduce the search time in the matching process. The visual navigation pipeline is shown in Figure 8.

Figure 8. Visual navigation pipeline.

Experimental Evaluation

We compare the proposed method with the existing image enhancement methods through extensive experiments to verify the superiority of our method, including MF (Fu 2016a) (Fu et al. 2016a), Dong (Dong 2015) (Dong et al. 2015), NPE (Wang 2013) (Wang et al. 2013), SRIE, and Lighten-Net (Li 2018) (Li et al. 2018). Generally speaking, we have carried out three parts of experiments: 1) We have qualitatively and quantitatively compared the proposed method with existing methods on the synthetic test images. 2) We illustrate that our method can improve the performance of image matching algorithms. 3) We combine the proposed method with an existing image matching method to carry out the visual navigation experiment.

Low-light Image Enhancement Experiment

To verify the superiority of our method, we compare the proposed method with the existing image enhancement methods on the synthetic test images. The Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used to quantitatively compare our method with other methods. For fairness, we use the public codes of these methods and do not change any of the parameters. Quantitative comparison results are shown in Table 1.

Table 1. PSNR/SSIM values on the synthetic test images. Note that the red, blue, and green in the table represent the best, sub-optimal, third-place results, respectively

Method	PSNR	SSIM
Dong	13.9642	0.4251
MF	15.1901	0.4921
SRIE	11.2053	0.3412
NPE	15.2126	0.4883
LightenNet	12.8605	0.4254
Ours	21.0668	0.6882

Our method can not only achieve higher PSNR and SSIM than other methods, which demonstrates that our method can improve the visual quality of low-light images, but also outperform other methods in speed. The average time cost by our method to enhance per image is 0.11 seconds on the GPU faster than our competitor (Dong takes 0.61s, MF takes 1.63s, SRIE takes 58.02s, NPE takes 6.52s, and LightenNet takes 6.08s to enhance an image), which benefits from our smaller mode size (4.85MB) and better network design. The traditional methods are running on the CPU which will increase the inference time. Our results have better contrast, clearer details, and better noise suppression, as shown in Figure 9.

Figure 9. Visual comparison of the synthetic test images. Please zoom in for a better view.

Image Matching Experiment

To verify that our method can improve the performance of the image matching algorithms, we use the proposed method and other methods to enhance the synthetic test images. Due to the lack of rotation invariant ability of the existing image matching algorithms, image rotation will seriously affect the accuracy of the algorithm, but we want to simulate the heading angle change of the UAV. Therefore, we chose to rotate the aerial image by 180° to perform our experiments. Using SIFT (Lowe 2004) (Lowe 2004), SURF (Bay 2006) (Bay, Tuytelaars, and Gool 2006), and GMS (Bian 2017) (Bian et al. 2017) algorithm to match the pre-processed images with the reference images and calculating the average number of successful matching points. Moreover, We found that when the number of extracted feature points is too small, it will cause matching failure, and when the number of extracted points is too large, it will not lead to a great improvement in accuracy but will increase the matching time. We found that when the number of extracted feature points is set to 500, there will be a better tradeoff between accuracy and speed. The image matching results are shown in Table 2.

Table 2. Comparison of matching results on the synthetic test images

Method	SIFT		SURF		GMS
	MP	T (s)	MP	T (s)	MP	T (s)
Input	8.3	0.09	6.3	0.07	58.6	0.04
Dong	5.8	0.72	5.0	0.68	54.2	0.65
MF	25.5	1.75	34.5	1.70	84.8	1.68
SRIE	13.7	58.14	11.0	58.09	62.6	58.07
NPE	25.4	6.63	25.1	6.59	70.9	6.57
LightenNet	24.5	6.19	24.1	6.15	68.8	6.12
Ours	36.0	0.22	36.2	0.18	105.2	0.16

Notes: We rotated the aerial images by 180°. MP represents the average number of successful matching points on the test images. T represents the sum of the time cost by the image enhancement algorithm and the image matching algorithm. Note that the red, blue, and green in the table represent the best, sub-optimal, third-place results, respectively.

After enhancing the low-light images, the average matching points have been greatly improved, and our method has achieved the best promotion effect, which proves the superiority of our method. Moreover, when our method is combined with the three methods for image matching under low-light conditions, it only takes 0.22s, 0.18s, and 0.16s to complete the matching process, respectively, which is better than other enhancement methods.

Note that the performance of the image matching algorithms will be degraded when the visual angle changes greatly. This is because the existing image matching algorithms are very sensitive to image rotation. Due to this limitation, in the following visual navigation experiment, we do not consider the pose change of the UAV, but assume that the UAV is flying flat.

Visual Navigation Experiment

In the image matching experiment, we found that GMS has the highest accuracy and the fastest speed, so we chose GMS as the matching algorithm of our visual navigation system. Moreover, when the visual angle changes greatly, it will seriously affect the accuracy of image matching algorithms (Shetty and Gao 2019). So we assumed that the UAV is flying flat in the visual navigation experiment, and the pose change is not taken into account.

In the visual navigation experiment, we collected a dataset which contains 5 satellite images with large visual range, and each satellite image corresponds to 10 aerial images with a small range. We take the satellite images as reference images and carry out the experiment based on the visual navigation pipeline we proposed. Eq.9(9) $I_{\mathrm{l}\mathrm{o}\mathrm{w}}=\mathrm{A}\times I_{normal}^{\gamma }$(9) is used to darken half of the aerial images of each satellite image to simulate the low-light images, and the rest of them are used as normal light images for experiments. When the aerial images are low-light images, we use our proposed enhancement method to transform them into high-quality normal-light images and match them with the reference images.

Because the size of the satellite image we selected is too large (7200 × 3600), the matching time will be increased, so we divided the satellite image into 9 images with equal spacing, all of the small-size reference images are 2400 × 1200. During the matching process, the aerial images are matched with the small-size satellite images we obtained in turn, and the small-size images with the highest degree of matching are selected. According to the position relationship between the small-size image and the original satellite image, the aerial image is mapped to the original satellite image using homography estimation, and the longitude and latitude information of the corresponding position of the satellite image is extracted as the current position information of the UAV. Note that the geographical location of satellite images and aerial images is known.

The estimated location information we got may be different from the ground truth location information, so we use the Haversine formula to calculate the distance between the estimated position and the actual position for evaluation. The haversine formula is defined as follow:

(11) $\mathrm{D}=2\times \mathrm{R}\times \mathrm{a}\mathrm{c}\mathrm{r}\mathrm{s}\mathrm{i}\mathrm{n}(\sqrt{\mathrm{s}\mathrm{i}\mathrm{n}{\left(\frac{\mathrm{L}\mathrm{a}\mathrm{t}\mathrm{E}-\mathrm{L}\mathrm{a}\mathrm{t}\mathrm{R}}{2}\right)}^2+\mathrm{c}\mathrm{o}\mathrm{s}(\mathrm{L}\mathrm{a}\mathrm{t}\mathrm{E})\mathrm{c}\mathrm{o}\mathrm{s}(\mathrm{L}\mathrm{a}\mathrm{t}\mathrm{R})\mathrm{s}\mathrm{i}\mathrm{n}{\left(\frac{\mathrm{L}\mathrm{n}\mathrm{g}\mathrm{E}-\mathrm{L}\mathrm{n}\mathrm{g}\mathrm{R}}{2}\right)}^2})$(11)

Where (LngR, LatR) and (LngE, LatE) are location information of ground truth and our estimated results, respectively, and R is the radius of the earth. Some examples of experimental results are shown in Figure 10. The red box is the region where the aerial image is mapped to the satellite image, and the red point in the yellow box is the location where the center point of the aerial image is mapped to the satellite image, and the latitude and longitude information of this red position is the current position of the UAV estimated by our method.

Figure 10. Experimental results of visual navigation.

As can be seen from the experimental results, when using low-light images directly as input, the matching points are not enough to achieve self-localization, while using our method to enhance the low-light images, the image matching performance can be effectively improved and self-localization of UAV can be achieved successfully. It is proved that the visual navigation system we designed can solve the limitation of UAV visual navigation in low-light conditions and realize self-localization under diverse illumination conditions.

Autonomous Driving Experiment

In addition to applying our method to UAV visual matching navigation, we also want to further explore the performance of our method in other practical tasks, such as autonomous driving, etc. A large number of researchers have also tried to apply visual navigation to autonomous driving and collected many real-world datasets in different environments and different illumination conditions (e.g. 4Seasons (Wenze et al. 2020), Apollo Scape (Wang et al. 2018)).

The Apollo Scape dataset contains pairs of RGB images from different angles taken by two cameras in different streets, we use the Apollo Scape dataset for our autonomous driving experiment. We used 1121 paired images with different angles in the Apollo Scape dataset. We chose the images captured by the second camera as the reference images and use Eq.9(9) $I_{\mathrm{l}\mathrm{o}\mathrm{w}}=\mathrm{A}\times I_{normal}^{\gamma }$(9) to darken the image captured by camera 1 to simulate the low-light image as real-time images obtained by the vehicle. By using our proposed enhancement method, the low-light images are converted into high-quality normal-light images and then matched with 1121 reference images in turn. The reference image with the highest matching degree is selected, and the position of the reference image is extracted as the current position of the vehicle. Because the images captured by the two cameras are in pairs, we considered that the self-localization failed when the enhanced results can not match the corresponding reference image correctly. At the same time, we also calculate the average matching logarithm between the enhanced image and the reference image with the highest degree of matching in the matching process, and the results are shown in Table 3.

Table 3. Self-localization accuracy and average matching points of the autonomous driving experiment

	Low-light + GMS	Our method + GMS	Normal-light + GMS
Acc	0.388	0.727	0.801
MP	47.52	94.10	118.71

As can be seen from Table 3, our method can effectively improve the positioning accuracy of autonomous driving under low-light conditions and the accuracy of the matching algorithm. Some self-localization examples are shown in Figure 11.

Figure 11. Experimental results of autonomous driving.

Conclusion

In this paper, we propose a novel low-light image enhancement method to improve the positioning accuracy of UAV visual navigation under low-light conditions. And we design a visual navigation system based on the proposed method, which can meet the navigation and positioning requirements of UAV under diverse illumination conditions. Overall, the main contributions of our work are twofold: (1): To solve the degradations in low-light images captured by UAV, we propose a new fully convolutional network, which can effectively improve the contrast of low-light images and obtain high visual quality results. (2): We design a visual navigation system based on the proposed image enhancement method. Our visual navigation system can determine the location of the UAV successfully under different lighting conditions. Moreover, we also verify that our method can effectively improve the positioning accuracy of autonomous driving based on image matching under low-light environments.

The experimental results demonstrate that the enhanced results of our method have more feature information and can meet the requirements of image matching. The visual navigation system can realize the autonomous positioning of UAV under diverse illumination conditions. In addition, there are some problems that need to be improved in the future, (1):In this paper, the low-light images we used are synthetic, but in the real-world low-light condition, the UAV will extend the camera’s exposure time to avoid the image being too dark, but this may cause the image to become blurred. Therefore, in our follow-up work, we will design an efficient network that can solve the insufficient illumination and blur in the aerial images captured under low-light conditions at the same time. (2): Although our proposed low-light enhancement method is faster than our competitors, it still can not meet the high-speed requirements (e.g. 60fps). In the future, We will explore an efficient method with excellent speed and performance to realize the real-time self-localization of UAV. (3): Because the change of the image rotation angle will seriously affect the accuracy of the image matching algorithms, we do not consider the pose change of the UAV in our experiment but assume that the UAV is flying flat. However, the pose estimation of UAV is also very important in the process of navigation, so our follow-up work will take the pose estimation into account.

Disclosure Statement

The authors declare no competing interests.