Review of CNN in aerial image processing

ABSTRACT

In recent years, deep learning algorithm has been used in many applications mainly in image processing of object detection and classification. The use of image processing techniques is becoming more interesting with the existence of drone technology with the employ of deep learning in aerial view image processing because of the high resolution and heaps of images taken. This paper aims to review neural networks specifically on the aerial view image by drones and to discuss the work principles and classic architectures of convolutional neural networks, its latest research trend and typical models along with target detection in object detection, image classification and semantic segmentation. In addition, this study also provided a specific application in the aerial image. Finally, the limitations of the convolutional network and expected future development trends were also discussed. Based on the findings, the deep learning algorithm was observed to provide high accuracy, it outperformed other generally image processing-based techniques.

KEYWORDS:

• Aerial image
• review
• image processing
• drone technology
• object detection
• image classification
• deep learning algorithm
• convolutional neural networks

Introduction

The latest drone technology in acquiring aerial view images has made data analytic in image processing more interesting. The capabilities of the drone to capture high-resolution images contribute to the development of a new method in object classification and detection. Aerial view imaging is a key component in many applications, including building detection [1–3], crack damage [4,5] and earth observation classification [6,7]. Usually, aerial view images are taken using satellite-based instruments with hyperspectral, multispectral, near-infrared, light detection and ranging, synthetic aperture radar and thermal cameras, where the instruments utilize Near-infrared (NIR), Light Detection and Ranging (LIDAR), Synthetic aperture radar (SAR) and thermal cameras [8,9]. In practical applications, large-scale observation is implemented by unmanned aerial vehicles (UAV), aeroplanes and satellites, which provide wide-view snapshots of the earth. All these latest technologies of aerial view image data collection utilize the application of a wide diversity of sensor data, including medium resolution, high resolution, very high resolution (VHR), hyperspectral (HS) images and multispectral (MS) imagery. These advanced sensors possess a wide range of spectrum in order to enable them to be used in various sectors namely agriculture, environment, engineering and construction [10].

Applications such as river bank monitoring or estimating the yield of a plantation usually require information on the earth’s features, like the distribution of houses along the river bank or the number of plants in the plantation, where most of the required data collected through diversity sensing involve images [1,2,4,11,12]. Aerial imagery analysis is a challenging task due to their special feature characteristics like the high resolution and small objective of the images compared to the common images. The key technology in image analysis is extracting and understanding the feature of the image. In general, the ordinary methods employed for image analysis include linear polarization, machine learning (ML), regression analysis, vegetation indices (NDVI) and wavelet-based filtering [13]. The traditional image analysis approach mainly focuses on the design of feature extractor and classifier. However, the development of traditional image analysis methods is constrained due to the poor portability, relatively poor accuracy and the lack of generalization ability of the feature extracted by artificial feature extraction approach [14–17].

The introduction of the latest techniques in machine learning such as deep learning has greatly resolved problems pertaining to image processing in object detection and identification. Hinton proposed the notion of deep learning, and analysed that neural networks with a multiple hidden layer own superior ability of feature learning [16]. Deep learning algorithms like convolutional neural networks, deep belief networks, deep neural networks and recurrent neural networks have been widely exploited in urban planning [18], precision agriculture [19,20], military applications [13] and environmental management [21], wherein some cases they outperformed human experts. The most popular network in deep learning is convolutional neural network (CNN), which learns features from a large number of data, where the learned features describe the rich information inherent in the data. Furthermore, CNN combines feature extraction, feature selection and feature classification into an integrated model to conduct global optimization through end-to-end training, which enhances the distinguishing power of features making it significantly superior to traditional methods in classification and positioning accuracy [20].

This paper discusses and reviews the implementation of deep learning architecture CNN and its applications in aerial imaging, the potential problems for the current CNN algorithm and aerial image processing are also discussed. It’s expected to contribute to the development of deep learning in aerial imagery. The organization of the remaining sections are as follows, section 2 introduces the architectures and working principle of CNN, and also give a summary of the classical CNN architectures; section 3 presents the characteristics of aerial image and the aerial image dataset, and also summarizes the current Deep learning methods in aerial image application; section 4 discusses the problems of CNN and aerial image processing and section 5 gives a conclusion of this paper.

Architectures of CNN

Currently, the deep learning architectures employed in image recognition and analysis studies mainly include Deep Belief Network (DBN) [22], CNN [23] and Stacked Auto-Encoders (SAE) [24]. CNN is perceived to be superior to the others in accuracy detection and speed due to its powerful feature representations, as it is an in-depth research that has become a central issue in the area of image recognition. As shown in Figure 1, the typical CNN architecture includes input layer, convolutional layer followed by pooling layer and fully connected layer.

Figure 1. Convolutional neural network.

Working principle of CNN

The learning and training of the convolution kernel parameters of CNN are implemented using a supervised learning algorithm named the gradient backpropagation algorithm. It is assumed that the training of the network is not perfect, some errors are bound to occur between the training and the actual outputs. The errors propagate layer by layer in the gradient descent algorithm, whereby the network parameters are updated layer by layer in parallel.

The working principle of CNN can be divided into three phases: network model design, network training and network prediction. Taken the CNN in classification task as an example, the steps are shown below:

1. Network model design

In the classification task, the objective is to classify the data into several classes. According to the volume of the data and the characteristics of the data itself, the design of the network model needs to design network depth, the function of each layer of the network and the parameters in the network such as λ, η, etc. Among the many studies that conduct experiments on the model design of CNN, are the ones on model depth [25,26], the ride of the convolution [27,28], activation function [29], the parameter selection in the network, etc. [30].

1. Network training

Parameters are trained in the network through the backpropagation of residuals for CNNs. After the training, the parameters of the network are fixed and the feature of the data is extracted. However, during network training, the overfitting, as well as the vanishing and explosion of gradient problems, greatly affect the convergence performance of the training. Several effective improvements have been proposed to solve the problems, including random initialization network parameters based on Gaussian distribution [31], initialization with pre-trained network parameters [26], and independent and identically distributed initialization for parameters of different layers of CNNs [32]. According to recent research trends, the increased utilization of the CNN model has led to a more challenging complex network model training strategy.

1. Network prediction

In the classification task, the output of the network is the classification result and the accuracy. The CNN prediction process includes the following three steps. The first step is forward conduction through the input data, followed by the output of the feature map at each layer, and finally using the fully connected network to output the conditional probability distribution. Recent studies have highlighted the high-level features of the forward-conducting CNNs performing with the discriminative ability and generalization performance [33]. In addition, these features can also be applied to a wider range of areas through transfer learning, as the research result is of great significance for expanding the application field of CNNs.

Classical CNN architectures

Based on the 36 latest articles investigated in this paper on deep learning in aerial image processing, the classical network models used in practical applications were usually chosen as the backbone to construct the network, such as the original LeNet-5 network, the classical AlexNet, VGGNet, GoogLeNet and ResNet networks, the object detection framework like Faster R-CNN. Here we describe several commonly used classical networks.

Alexnet

In 2012, Krizhevsky et al. introduced the AlexNet network (Figure 2), a 5-layer convolutional network with approximately 650,000 neurons and 60 million trainable parameters [31]. These characteristics greatly exceed LeNet-5 in terms of network size [34]. In addition, by choosing the large image classification database ImageNet as the training dataset [35], AlexNet also utilized the dropout technique to reduce the network overfitting problem. In terms of hardware support, it used graphics processing unit (GPU) for training, which increased the training speed of the network by more than 10 times compared to traditional central processing unit (CPU) operations. Therefore, it won the championship in ImageNet’s 2012 image classification competition by achieving an 11% higher accuracy compared to the runner-up. This success has attracted academicians’ attention to utilizing AlexNet to the study of CNN once again.

Figure 2. AlexNet architecture [31].

VGGNet

Given the large convolution kernel problem of AlexNet, Simonyan et al. conducted experiments to discover that it is possible to replace large convolution kernels with several smaller convolution kernels, which obtained better experimental results [26]. In their experiments as shown in Figure 3, a 7 × 7 convolution kernel was considered as a superposition of 3 layers of 3 × 3 convolution kernels, which had 49 parameters to be learned while only one layer of feature image and one receptive field was provided. The 3 layers of 3 × 3 convolution kernel had 27 parameters, with 3 receptive fields and 3 layers of feature images at different scales. In addition, the network used the 3 × 3 convolution kernel and added a pooling layer after several convolutional layers instead of pooling operation immediately after each convolutional layer, which ensured the network depth. The structure of VGGNet further evidenced that increasing the depth of network is beneficial to enhance the accuracy of image classification. However, the depth of the network should be limited because an excessive number of layers will cause network degradation that will affect the test results [36]. Finally, VGGNet layer number is determined in both the 16 and 19 layers.

Figure 3. VGGNet architecture [26].

Googlenet

Different sizes of convolution kernels denote different sizes of receptive fields, whereby VGGNet had proven that different receptive fields help improve classification accuracy. Consequently, GoogLeNet added convolution kernels of different sizes in the same layer [37]. As shown in Figure 4, three sizes of convolution kernels 1 × 1, 3 × 3, 5 × 5 were inserted in the same layer, where a concatenation operation enables feature images of different scales to be fused together. In addition, a 1 × 1 convolution kernel is added in front of the 3 × 3 and 5 × 5 convolution kernel in order to decrease the dimension and the number of parameters [38]. Inception is an optimization step of CNN structure, where the number of parameters trained by GoogLeNet is only 1/12 than that of AlexNet and obtained more accurate results. GoogLeNet emerged as the champion of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)2014 classification competition with a network depth of 22 layers.

Figure 4. Inception module with dimension reduction [37].

Resnet

Since gradient exploration and vanish problems exist in network training, He et al. proposed a deep residual learning network [25]. As illustrated in Figure 5, the residual network directly maps the shallow layer feature map X to the upper layer through shortcut connections, where the output of the high layer becomes the aggregate of the underlying map and the original output. In addition, a priori of the shallow layer network is introduced in the training of the network. Considering the extreme situation, if X is already a near-perfect mapping, the gradient of the current layer will become very small in the subsequent training, and it will continue to affect the coefficient update of the previous layer during the back propagation. As the structure of residual is introduced, the overall gradient does not disappear due to the existence of the output X, and it ensures the performance of deeper networks.

Figure 5. Residual learning: a building block [25].

Deep residual network with a 152 layer network depth won the championship with a top-5 error of 3.57% in ILSVRC2015. This result from the residual network exceeded the human eye recognition rate (5.1%) [32]. However, the slimming down of CNN and the optimization of its structure will remain an important topic in the future as the deep residual network model is large and complicated.

Comparison of the classical network

Table 1 summarizes the aforementioned classical CNN networks in terms of the layers, the parameters, place and the Top-5 error rate when conducted the image classification task in the ILSVRC competition. The dataset is the public ImageNet dataset. The comparison in the table illustrates that with the optimization of the network model, the amount of the parameters decreased with the network depth increased. Meanwhile, the Top-5 error rate has decreased, where ResNet reached the lowest Top-5 error rate of 3.57% with 152 layers and only more than 2 million parameters. The transfer learning method saved much time during the network training and got excellent performance in image classification and object detection. Some applications used the pre-trained network by keeping the convolution layers’ architecture and parameters while changing the full connected layers. VGGNet is the most popular network used in the transfer learning task. AlexNet, GoogLeNet and ResNet also are used as the backbone of the CNN network in specific applications, which reduce the training time and get better results compared to the network trained from scratch.

Table 1. Classical CNN network.

CNN network	Layers	Parameters	Place	Top-5 error rate	Year
AlexNet	7	60 million	1st	15.3%	2012
GoogLeNet	19	4 million	1st	6.67%	2014
VGGNet	16	138 million	2nd	7.3%	2014
ResNet	152	2 + million	1st	3.57%	2015

Deep learning in aerial image processing

Characteristics of aerial image

The aerial images obtained from UAV or drone possess high resolution, where the interest targets with less number pixels are sparsely scattered, making it difficult to classify and detect these images. The attribute of the aerial image is summarized as the following: Small objects have less context information, lower resolution and cluttered background; The visual appearance of objects in aerial images differ largely due to the variation of viewpoint, illumination, shadow, occlusion and resolution [39]; High-computational problem due to a large number of heterogeneous images have different temporal resolutions, angular, spectral and spatial; The spectral resolution of VHR images are inherently lower, where the small-scale surface texture and small objects are visible, which lead to low inter-class differences and high intra-class variability of the image; Insufficient aerial image data to train a new CNN network.

In regards to the small object and cluttered background of the image, traditional image processing usually designs a feature extraction approach according to the characteristic of the specific object. However, the feature extraction approach for some objects cannot be used to other category objects, which lacks generalization ability. For deep learning-based method, one network model can be used to extract the feature of different objects, where the approach is not constrained by the characteristic of the object. When considering the high-computational problem, the transfer learning approach applies the parameter trained on public dataset like ImageNet to the new dataset which is similar to the public dataset. Here, the GPU unit is used as the data processing unit, because it is efficient in accelerating the training process. As for the issues of low inter-class dissimilarity and high intra-class variability, Cheng et al. put forward a discriminative CNNs (D-CNNs) model to enhance the accuracy in remote sensing image classification, where a metric learning regularization is used to extract more discriminative CNN features [40]. The experimental result exhibited that the method outperforms the existing three baseline methods with several public benchmark datasets. For the insufficient dataset problem, data augmentation approach is used to enlarge the dataset. Data augmentation increases the amount of data without changing the image category and also improves the generalization ability of the model. There are many ways to augment the data of natural images, such as horizontal flips, a certain degree of displacement, cropping and colour jitter.

Aerial image dataset

Generally, the aerial image is performed by means of UAV, aeroplanes and satellites, which provide wide-view snapshots of the earth. Table 2 shows several available public aerial image datasets, which are used for semantic segmentation, image classification and object detection. The image resolution for UC Merced Land Use, WHU-RS19, NWPU-RESISC45 and AID are no more than 600 × 600, which are used in the image classification task. WHU-RS19 contains 19 different scene classes and 50 sample sizes of 600 × 600 in each class, while UC Merced Land Use dataset has 2100 images which include 21 classes with 100 images of 256 × 256 sample size for each class. On the other hand, NWPU-RESISC45 dataset established by Northwestern Polytechnical University (NWPU), is a publicly available dataset for Remote Sensing Image Scene Classification (RESISC) which includes 31,500 images, covering 45 scene classes. Among these datasets, AID is the largest annotated aerial image dataset with the images collected from Google Earth imagery, where it contains 10,000 images within 30 scene classes and 200–400 samples in each class with 600 × 600 size. The AID dataset is closer to real aerial image classification tasks due to the large-scale dataset, smaller inter-class dissimilarity and higher intra-class variations images. The remaining datasets consist of high-resolution images, such as NWPU VHR-10, VEDAI, OIRDS and RIT-18 dataset, where NWPU VHR-10 with 10-class is a geospatial object detection dataset containing 800 VHR images collected from Vaihingen and Google Earth dataset, and it has been manually annotated. Besides, the VEDAI, OIRDS and Munich vehicle datasets are vehicle images, which are built for vehicle detection in aerial imagery, where VEDAI dataset contains 1240 images within 9 classes and the vehicles contained in the database displayed different variabilities such as occlusions or specularities, shadowing changes or lighting and multiple orientations. Meanwhile, OIRDS dataset contains 900 annotated images with 1800 targets identified. On the other hand, ISPRS Potsdam, Massachusetts Buildings and Vaihingen datasets consist of aerial imagery with buildings, with Vaihingen datasets and ISPRS Potsdam containing 33 and 38 patches with different sizes, respectively, where each consists of a real orthophoto (TOP) obtained from a broad TOP mosaic. RIT-18 is a high-resolution multispectral dataset collected using a UAV which was annotated with 18 object categories and is used for semantic segmentation.

Table 2. Public aerial image dataset.

Dataset name	Scene classes	Total images	Image size	Source
UC Merced Land Use [41]	21	2100	256 × 256	http://weegee.vision.ucmerced.edu/datasets/landuse.html
WHU-RS19 [42]	19	1005	600 × 600	http://www.tsi.enst.fr/~xia/satellite_image_project.html
NWPU-RESISC45 [43]	45	31,500	256 × 256	http://www.escience.cn/people/gongcheng/NWPU-RESISC45.html
AID [44]	30	10,000	600 × 600	https://pan.baidu.com/s/1mifOBv6#list/path=%2F
NWPU VHR-10 [45]	10	800	Very high resolution	http://www.escience.cn/people/gongcheng/NWPU-VHR-10.html
VEDAI [46]	9	1240	512 × 512 1024 × 1024	https://downloads.greyc.fr/vedai/
OIRDS [47]	1	900	High resolution	http://oirds.sourceforge.net/
ISPRS Potsdam [48]	38 patches		6000 × 6000	http://www.itc.nl/ISPRS_WGIII4/tests_datasets.html
Munich vehicle [49]	20 patches		5616 × 3744	http://www.dlr.de/eoc/desktopdefault.aspx/tabid-5431/9230_read-42467/
Massachusetts Buildings [50]	151 patches		1500 × 1500	http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-ofgeographic-information-massgis/
RIT-18 [51]	18 patches		Very-high resolution	https://github.com/rmkemker/RIT-18
Vaihingen[52]	33 patches		2494 × 2064	http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html

Deep learning in aerial image application

A total of 36 latest articles were investigated related to deep learning and their applications on aerial images focused on four categories which are object detection, image classification, image semantic segmentation and other applications. Table 3 represents the 36 identified relevant works in terms of the aerial image application area and the specific application with related reference. Of the 36 articles, 13 were related to object detection, 13 to classification, 8 to semantic segmentation, 1 to image denoizing and 1 to image matching. Several studies utilized deep learning for building detection [1,3], while a few used it for bottle detection [53], vehicle detection [39,54], etc. In addition to the classification, many studies used deep learning in aerial scene classification while some applied it in agriculture for crop classification and earth observation classification [6,11]. On the other hand, several studies focused on semantic segmentation [51,52,55,56] while some others used deep learning in building and vehicle segmentation [57–59]. Besides, one study focused on the image match issue which tried to find out the same object between two images [60], while another study referred to image denoizing which focused on removing noise in UAV images [61].

Table 3. Specific application of deep learning in the aerial image.

Application area	Specific application	References
Object Detection	Building detection for riverbank monitoring	[1–3]
	Bottle detection	[53]
	Vine diseases detection	[19]
	Crack damage detection	[4,5]
	Insulator detection	[7]
	Mammals detecting	[62]
	Tree species detection	[12]
	Vehicle detection	[39,54]
	Recognizing birds	[63]
Classification	Aerial scene classification	[40,64–67]
	Corn classification	[11]
	Infrastructure Quality Assessment	[68]
	Earth observation	[6]
	Vehicles classification	[69]
	Hyperspectral data classification	[70]
	Estimating plant centres	[71]
	Urban land use classification	[72,73]
Semantic Segmentation	Semantic segmentation	[51,52,55,56]
	Roof segmentation	[57]
	Building extraction	[58]
	Vehicle instance segmentation	[59]
Matching	Find the same object in two images	[60]
Denoizing	Remove noise in UAV images	[61]

Object detection

Object detection is a crucial research topic in image processing and computer vision, where it focused on detecting a certain class of semantic objects such as buildings, cars or humans in videos and digital images. In recent years, the accuracy of object detection has greatly increased due to the success of deep learning-based CNNs. Currently, deep learning methods used for object detection can be divided into two classes, namely one stage detection and two stage detection [74–77]. The latter involves two steps. First, the regions that potentially contain objects are proposed. Then, the proposed regions obtained in the first step will be classified into different categories. However, the regions proposal step is removed in one stage detection, whereby the object localization and classification are completed at one step, greatly improving the detection speed compared with two stage detection. Examples of well-researched fields for object detection contain pedestrian detection and face detection [39].

Table 4 enlists the overview of 13 investigations of deep learning-based algorithm used in object detection. It summarizes the investigation in terms of CNN framework used, dataset and other methods used along with the comparisons with other available techniques. Based on Table 4, only two investigations [5,62] used General CNN model to train their network while the rest of them employed well-known CNN architectures, such as LeNet-5, Faster R-CNN, SSD, YOLO, VGG-16 and Resnet. Reference [19] reached the highest accuracy at 95.8%, it focused on the vine diseases detection using the LeNet-5 as the CNN network, and the collected data was UAV images. In addition, 5 of the 13 studies utilized VGG architecture with 16–19 weight layers. Together with CNN model, other methods also were employed to improve the performance of detection, in which fine-tuning and transfer learning were the two common methods used in adjusting the pre-trained network in order to be suitable to a specific dataset. In a previous study, two Faster RCNN based models (Inception-Resnet and Resnet) and a one stage Focal Loss network named RetinaNet were used, where the detection accuracy of the models was estimated at 91.3%, 83.1% and 75.6%, respectively [3]. Kasthurirangan et al. and Hilal et al. employed the pretrained deep learning network VGG-16 and fine-tuned the network to a new application with less datasets [4,39]. The experiment implemented by Kasthurirangan et al. achieved 90% accuracy in determining the crack damage detection of civil infrastructure in realistic situations, while Hilal et al. utilized a uniform one stage model, where the proposed method was identified with two public datasets. The results from the latter study outperformed the current result in terms of mean average precision (MAP) and computation time. Recently, the depth of the proposed CNN networks has been increased in ResNet, but the deeper the network the harder it is to train. Therefore, the researchers utilized batch normalization to mitigate the time-consuming problem and avoid gradient vanish [1]. Diogo et al. also employed a multi-resolution feature fusion method, with the results indicating better accuracy and localization capabilities [5]. Besides, Zou et al. used VGG architecture, RAM, small target prior, fisher information and Bayesian statistics to improve the accuracy with the MAP increase of 1%-4% on LEVIR dataset compared to the medium-networks, tiny-networks (TINY-BASE) and VGG-f based networks [2]. Moreover, the multi-task loss function used in a previous study accomplished the improved detection accuracy in small-sized object detection and precise localization, the proposed method obtained the recall at 80.3% [54].

Table 4. Applications of object detection.

References	CNN framework	Dataset	Other method used	Comparisons	Accuracy
[1]	SegNet	Collect data with UAV	Fine-tuning Batch normalization	Valid the proposed method on the collected datasets	More than 90%.
[2]	VGG	LEVIR, NWPUVHR-10 TACDD, OIRDS	RAM, small target prior, Fisher information, Bayesian statistics	Tiny-networks, Medium-networks, VGG-f based networks	Improve mAP by 1%- 4%.
[3]	RetinaNet, Resnet, Inception-Resnet	Collect data with UAV	Focal loss mechanism, Fine-tuning	Faster RCNN with ResNet Faster RCNN with Inception-v4, RetinaNet	More than 75.6%.
[53]	Faster R-CNN, SSD, YOLOv2	Collect data with UAV	Arbitrary quadrilateral bounding boxes	Valid the proposed method on the collected datasets	Up to 90.3%, 90.1%, 77.4%, respectively.
[19]	LeNet-5	Collect data with UAV and RGB sensors	Sliding window segmentation, Post processing	Different patch size, filter size and vegetation indices	More than 95.8%.
[4]	VGG-16	Collect data with UAV	Fine-tuning, DNN Transfer learning	TL + LR, (L,5), TL + SVM, TL + ERT (300), TL + RF (300), (L1) TL + NN(AM)	Up to 90% accuracy.
[5]	General CNN	Collected data with WorldView-3, NWPU-RESISC45	Multi-resolution feature, Fusion approach, Batch-normalization	MR_a, MR_b, MR_c Baseline-ft, Baseline	More than 81.5%.
[7]	Faster R-CNN	Collect data with UAV	Convolutional network ZF	Valid the method on the collected datasets	More than 93.9%.
[62]	General CNN	Public dataset: MicroMappers	Evaluation protocol Curriculum learning	Baseline	Up to 90% Recall.
[12]	VGG-16	Collect data with Worldview-3	Patch-based Image Labeling	Random Forest (RF), Gradient Boosting (GB)	More than 92.13%.
[39]	VGG-16	Public dataset: NWPU VHR-10, RSOD dataset	Fine-tuning, Cross-entropy loss function	Transferred CNN, RICNN, SSD, R-P-Faster R-CNN, Rotation-Insensitive CNN, Deformable CNN	More than 92.13%.
[54]	VGG-16	Public dataset: VEDAI, Munich and Vehicle datasets	Multi-feature maps of different hierarchies and scales, anchor scheme, Multi-task loss function	Faster R-CNN (VGG-16), Faster R-CNN (Z&F), SLIC with Z&F, SLIC with VGG-16	More than 80.3% Recall.
[63]	Mask R-CNN and U-Net, ResNet-101	LBAI	Fine-tuning, Adam optimizer	YOLOv2, SSH, Tiny Face	For easy case, SSH got F1 score of 92.5%. For hard case, Tiny Face got F1 score of 80.2%.

Classification

Image classification refers to the task of classifying a number of images into categories in light of the characteristic of the images. It has been widely researched in past decades due to the extensive applications [40]. Recently, deep learning-based aerial image classification has been applied on various areas, including agriculture application like corn classification [11], earth observation classification [6], in estimating plant centres [71], civil infrastructure application like infrastructure quality assessment [68], and urban land use classification [72,73].

Table 5 represents an overview of image classification in terms of CNN framework used, dataset used, the number of classes, other methods used and comparisons. In total, 13 studies were investigated on the subject of image classification, in which 5 referred to aerial scene classification using publicly available dataset to validate the proposed method, with the UC Merced Dataset as the most commonly used dataset. Of the 5 studies, 2 studies trained their own CNN architecture with the training dataset [64,65] while the remaining 3 employed pre-trained CNN frameworks like AlexNet, VGGNet, GoogLeNet, OverFeat and CaffeNet [40,66,67]. Han et al. used a self-label co-training based SSGF method on AID dataset, NWPU-RESISC45, WHU-RS and UCM datasets [64]. The result indicated that the proposed method improved the classification accuracy compared to supervised learning, Selftraining, SSGA-E and Co-training methods. In addition, a metric learning regularization on CNN features was imposed, and the proposed D-CNN models addressed the problems of inter-class similarity and intra-class diversity by optimizing a new discriminative objective function, the mothed was validated on UC Merced dataset and WHU-RS Dataset with the classification more than 96.9% [67]. Two other studies collected their dataset using the UAV technique [11,71], while the former employed a customised X8-Skywalker frame to obtain data and LeNet to train their CNN architecture [11]. On the other hand, Oshri et al. used transfer learning, fine-tuning and Adam optimiser to improve the classification accuracy, and it was validated on Landsat 8 and Sentinel 1 dataset, and the accuracy reached 84% [71]. Besides, Cao et al. proposed a DNN-based data fusion method using SegNet, which integrate ground feature maps and aerial images together. The experiment on New York City land use maps dataset achieved high accuracy at 78.1% [72]. Similarly, Romero et al. combined sparse features learning, principal component analysis (PCA), greedy layerwise and unsupervised pre-training strategy together [73]. It outperformed PCA and kernel PCA (KPCA) methods when implemented on UC Merced dataset and AVIRIS Indiana’s Indian Pines test site.

Table 5. Applications of image classification.

References	CNN framework	Dataset	Classes	Other method used	Comparisons	Accuracy
[64]	General CNN	UC Merced Dataset	21 classes	Grid size reduction, factorization, hyper parameter tuning, residual units	CaffeNet from scratch, CaffeNet transfer learning, GoogLeNet from scratch, GoogLeNet transfer learning	Up to 78%.
[65]	General CNN	UCM dataset, WHU-RS, NWPU-RESISC45, AID datasets	21 classes 19 classes 45 classes 30 classes	SSGF co-training-based self-label method	Supervised learning, Self-training, Co-training SSGA-E	More than 91.35%.
[66]	ConvNets OverFeat CaffeNet	UC Merced dataset	21 classes	Fusing ConvNets, Descriptors based on bags of visual words (BoVW)	OverFeatS + OverFeatL OverFeatS + Caffe OverFeatL + Caffe	More than 90%.
[67]	AlexNet CaffeNet VGG	UC Merced dataset, WHU-RS Dataset	21 classes 19 classes	Transfer learning	AlexNet CaffeNet VGG-F VGG-M VGG-S VGG-VD16, VGG-VD19 PlacesNet OverFeat + CaffeNet	More than 96.9%.
[40]	AlexNet VGGNet GoogLeNet	NWPU RESISC45, AID data set, UC Merced dataset	21 classes 30 classes 45 classes	Metric learning, Regularization	Fine-tuned CNN Transferred AlexNet + SVM Fine-tuned AlexNet + SVM	More than 96.67%.
[11]	LeNet	Collect data with UAV	2 classes	Customized X8-Skywalker frame	Valid the propoed method on the collected dataset	Up to 86.8%.
[68]	ResNet	Landsat 8 dataset Sentinel 1 dataset	2 classes	Fine-tuning, Transfer Learning, Adam optimizer	Valid the propoed method on the public dataset	Up to 88.1%.
[6]	Overfeat model	UC Merced dataset	21 classes	Feature Fusion t-SNE	Random forest (RF), CNNs with RGB, CNNs with Overfeat	Up to 92.4%.
[69]	AlexNet VGG-16	VEDAI dataset	9 classes	SVM, Random Forest	Valid the proposed method on the public dataset	Up to 99%.
[70]	General CNN	AVIRIS and ROSIS,	20 classes 10 classes	Randomized PCA, MLP classifier	Linear kernel SVM, RBF kernel SVM, R-PCA RBF-SVM	More than 98.88%.
[71]	VGG	Collect data with UAV	2 classes	Connected components, Morphological operations	Valid the proposed method on the collected dataset	Up to 84%.
[72]	SegNet VGG-16	Publicly available dataset of New York City Land	12 classes	Multimodal data fusion, encoder-decoder, PCA Spatial Interpolation	Valid the proposed method on the public dataset	Up to 78.1%.
[73]	General CNN	UC Merced dataset, Indian Pines test site	21 classes	Sparse features learning, Greedy Layerwise, Unsupervised Pretraining	PCA, KPCA	Up to 84.53%.

Semantic segmentation

Image semantic segmentation is a classification at the pixel level, where pixels belonging to the same class are classified into one class, so semantic segmentation is to understand images from the pixel level. The objective of semantic segmentation is to change and simplify the representation of the image to make the image easier for further analyse. Generally, image semantic segmentation is used to locate objects in images. As a popular research domain in aerial image processing, segmentation of high-resolution images plays a crucial role in various applications like forestry and agriculture management, environmental monitoring, land inventory and urban planning. Deep learning algorithms-based CNNs have received an increased amount of attention, where it has been employed in semantic labelling of UAV imagery.

Table 6 summarizes seven articles that employed deep learning in image segmentation, in which four articles referred to semantic segmentation [51,52,55,56], one article aimed at roof segmentation [57] and another one focused on vehicle instance segmentation [59]. Based on the nature of optical image, the labels of nearby image pixels were strongly correlated for the spatial coherence of the objects, therefore, it was a challenging task to divide these areas effectively. Sun et al. presented a semantic segmentation approach that combined multi-resolution segmentation (MRS) and multi-filter CNN method together, where the multi-filter CNN method is used to aggregate the training data, the multimodal data fusion approach is used to label the imagery, while the MRS method was employed to delineate object boundaries to reduce salt-and-pepper artefacts [55]. In addition to the overfitting problem, dropout and FCRF post-processing methods were employed to improve segmentation performance [56]. With the network depth increasing, the accuracy gets saturated followed by rapid degradation, the Fully Convolutional Network (FCN) based framework with residual learning was used [60], where the experiment on ISPRS Potsdam dataset demonstrated it to outperform the other FCN-based methods. In line with the problem of the internal covariate shift of input layers, stacked convolutional autoencoders (SCAE) and multi-scale independent component analysis (MICA) were utilized to accelerate the training speed [51]. Besides, independent pixel-wise classification by ignoring the high-order regularities has always resulted in spatial inconsistency problems. In order to address the problem, a Building-A-Nets was presented where it successfully detected and rectified the spatial inconsistency on aerial images [58]. Given the complication of benchmark dataset, a new dataset of Aerial Imagery for Roof Segmentation (AIRS) was provided [57], in which several popular applied deep learning based semantic segmentation methods were implanted for the analysis of the proposed dataset and performance evaluation. The results indicated that the dataset served as the benchmark dataset for future study.

Table 6. Applications of image semantic segmentation.

Reference	CNN framework	Dataset	Other method used	Comparisons	Accuracy
[51]	ResNet-50, RefineNet	RIT-18 dataset	Batch normalization MICA, SCAE	FCN, KNN, SVM, MLP, MP, MICA, SCAE, MRS	Up to 59.8%.
[52]	VGG	Vaihingen dataset	Top-down spatial Regularization encoded	SegNet, PSPNet U-Net, AlexNet	More than 75.1%.
[55]	SegNet	high-resolution and LiDAR data optical imagery	Batch normalization, Multi-filter CNN, MRS based post-processing	SVM, Random Forest, G-FRNet, Multi-scale CNN, Multi-filter CNN	More than 87.88%.
[56]	FCN	ISPRS 2D semantic labeling contest	FCN-Pascal, drop-out, FCRF post-processing	FCN-Places, FCN-ImageNet + FCN-Pascal, FCN-ImageNet + FCN-Pascal, FCN-ImageNet + FCN-Pascal + FCN-Places	More than 80%.
[57]	ResNet-101	Land Information of New Zealand (LINZ)	Refinement of the open-source data	FPN, FPN + MSFF, PSPNet	The accuracy reached 96.3%, 95,8%, 96.1%, respectively.
[58]	FC-DensNet, GAN	Massachusetts, Inria aerial image labeling dataset	Deep adversarial network	Saito-multi-MA&CIS, Mnih-CNN + CRF, FC-DensNet, FCN MLP, SegNet, HF-FCN, U-Net	Up to 96.71%.
[59]	FCN, ResFCN	ISPRS Potsdam dataset	Residual learning	VGG-FCN, B-ResFCN Inception-FCN, B-inception-FCN, ResFCN	Up to 93.80%.

Other applications

Generally, all signal processing devices (both analogue and digital) are susceptible to noise, which can be random or white noise with an even frequency distribution or frequency-dependent noise introduced by a device’s mechanism or signal processing algorithms. Table 7 summarizes two articles that have applied deep learning in image denoizing and matching. In order to solve the noise problem, a generative adversarial learning framework-based deep neural network approach was proposed to track the relevance between noisy and clean images [61]. The proposed method outperformed the traditional method based on image matching on local features of the denoised UAV images along with corresponding clean UAV images. In the situation that the aerial images on ultra-wide baseline were collected by large camera rotations, the variation of the image exceeded the reach of RANSAC and SIFT methods. Altwaijry et al. introduced a deep learning-based method that sidestepped local correspondence by taking the issue as a classification operation, by incorporating an attention mechanism to generate a group of possible matches [60]. The experiment demonstrated that the proposed models approached human accuracy and outperformed the traditional ultra-wide baseline matching method.

Table 7. Applications in image denoizing and matching.

Reference	CNN framework	Dataset	Other method used	Comparisons
[60]	AlexNet	Aerial imagery through Google Maps	Fine-tuning, Spatial Transformer (ST)	A-SIFT, GIST Fisher Vectors, VLAD, Siamese Places CNN Siamese AlexNet
[61]	VGG	Collect data in Guiyang city	Generative adversarial learning (GAN), Batch-normalization, Residual blocks	Image matching and classification between denoised and original image

Discussion

Currently, CNNs based deep learning algorithm is being developed and incorporated rapidly into the field of computer vision, which has achieved amazing results. This paper mainly discussed the structure of CNNs and the research progress in the field of aerial image processing. In short, the success of deep learning depends on three factors, namely big data, large models and big calculations. A large amount of manual annotation data makes supervised training possible, whereby a deeper and larger model improves the recognition ability of the network and with the combination of GPU, the rapid development of computer hardware enables large-scale training time-saving and effective. However, some problems still exist in this field and need to be addressed in future:

1. It is difficult to determine the most suitable network structure for a specific task, especially the number of layers and neurons for each layer. Hence, it still requires a more detailed knowledge to choose reasonable values such as learning rate and regularization intensity.

2. The robustness of object detection is mainly affected by inter-class and intra-class diversity, which is reduced by the little inter-class difference and large intra-class diversity. Wang et al. used adversarial learning to address the image recognition problem, where they trained the detection network by generating occlusion and deformation image samples with adversarial network [78], which produced good results. Therefore, building a feature model with generalization capabilities remains an open research problem.

3. The current theory of CNNs lacks frame mathematical theory. The incorporation of many parameters for a specific model depends more on the experience of the researchers than theory. A complete mathematical explanation is an inevitable problem in the future development of CNNs.

Conclusion

In this systematic review, we conducted a review of deep learning-based research applications in aerial image processing. We investigated 36 relevant articles, examined the problems and particular areas, including the sources of data used, technical details of the network models and performance comparison between the existing techniques and deep learning. In summary, deep learning achieves state-of-the-art performance by outperforming other traditional image processing techniques. In future, we intend to utilize some well-known CNN networks to determine potential methods that can be used to address the problem described in Section 4.

This survey would help future researchers and engineers in conducting experiments with deep learning, and applying it to solve various aerial image processing problems involving object detection, classification, semantic segmentation and other area about image analysis or computer vision. Moreover, the review is expected to contribute to the development of deep learning in aerial imagery.

Acknowledgements

This work was supported by the Research Support Program of Xi’an University of Finance and Economics of China (22FCJH008), and the Social Science Foundation Program of Shaanxi of China (2022M005).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Xi’an University of Finance and Economics [grant number 22FCJH008]; Universiti Malaysia Pahang [grant number UIC191502; RDU1703228]; Social Science Foundation Program of Shaanxi [grant number 2022M005].

Notes on contributors

Xinni Liu

Xinni Liu is a senior lecturer at the school of information, Xi’an University of Finance and Economics. She received the PhD degree in electronic engineering and computer science at University Malaysia Pahang, Peken, Malaysia. Her research interest includes computer vision, image processing, machine learning and deep learning.

Kamarul Hawari Ghazali

Kamarul Hawari Ghazali is a professor of faculty of electrical and electronics engineering, University of Malaysia Pahang, Malaysia. His current research interests include Deep Learning, Machine Vision System, Image Processing, Signal Processing, Intelligent System, Vison Control, Computer Control System, Thermal Imaging Analysis. He is a Member of Board of Engineer Malaysia and a Senior Member of IEEE.

Fengrong Han

Fengrong Han is a senior lecturer of Baoji University of Arts and Sciences. Her research interest includes wireless sensor network, optimize algorithm.

Izzeldin Ibrahim Mohamed

Izzeldin Ibrahim Mohamed is a senior lecturer of faculty of electrical and electronics engineering, University of Malaysia Pahang, Malaysia. His research interest includes wireless sensor network, network security.