Review of CNN in aerial image processing

Figures & data

Figure 1. Convolutional neural network.

Figure 2. AlexNet architecture [31].

Figure 3. VGGNet architecture [26].

Figure 4. Inception module with dimension reduction [37].

Figure 5. Residual learning: a building block [25].

Table 1. Classical CNN network.

CNN network	Layers	Parameters	Place	Top-5 error rate	Year
AlexNet	7	60 million	1st	15.3%	2012
GoogLeNet	19	4 million	1st	6.67%	2014
VGGNet	16	138 million	2nd	7.3%	2014
ResNet	152	2 + million	1st	3.57%	2015

Table 2. Public aerial image dataset.

Dataset name	Scene classes	Total images	Image size	Source
UC Merced Land Use [41]	21	2100	256 × 256	http://weegee.vision.ucmerced.edu/datasets/landuse.html
WHU-RS19 [42]	19	1005	600 × 600	http://www.tsi.enst.fr/~xia/satellite_image_project.html
NWPU-RESISC45 [43]	45	31,500	256 × 256	http://www.escience.cn/people/gongcheng/NWPU-RESISC45.html
AID [44]	30	10,000	600 × 600	https://pan.baidu.com/s/1mifOBv6#list/path=%2F
NWPU VHR-10 [45]	10	800	Very high resolution	http://www.escience.cn/people/gongcheng/NWPU-VHR-10.html
VEDAI [46]	9	1240	512 × 512 1024 × 1024	https://downloads.greyc.fr/vedai/
OIRDS [47]	1	900	High resolution	http://oirds.sourceforge.net/
ISPRS Potsdam [48]	38 patches		6000 × 6000	http://www.itc.nl/ISPRS_WGIII4/tests_datasets.html
Munich vehicle [49]	20 patches		5616 × 3744	http://www.dlr.de/eoc/desktopdefault.aspx/tabid-5431/9230_read-42467/
Massachusetts Buildings [50]	151 patches		1500 × 1500	http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-ofgeographic-information-massgis/
RIT-18 [51]	18 patches		Very-high resolution	https://github.com/rmkemker/RIT-18
Vaihingen[52]	33 patches		2494 × 2064	http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html

Table 3. Specific application of deep learning in the aerial image.

Application area	Specific application	References
Object Detection	Building detection for riverbank monitoring	[1–3]
	Bottle detection	[53]
	Vine diseases detection	[19]
	Crack damage detection	[4,5]
	Insulator detection	[7]
	Mammals detecting	[62]
	Tree species detection	[12]
	Vehicle detection	[39,54]
	Recognizing birds	[63]
Classification	Aerial scene classification	[40,64–67]
	Corn classification	[11]
	Infrastructure Quality Assessment	[68]
	Earth observation	[6]
	Vehicles classification	[69]
	Hyperspectral data classification	[70]
	Estimating plant centres	[71]
	Urban land use classification	[72,73]
Semantic Segmentation	Semantic segmentation	[51,52,55,56]
	Roof segmentation	[57]
	Building extraction	[58]
	Vehicle instance segmentation	[59]
Matching	Find the same object in two images	[60]
Denoizing	Remove noise in UAV images	[61]

Table 4. Applications of object detection.

References	CNN framework	Dataset	Other method used	Comparisons	Accuracy
[1]	SegNet	Collect data with UAV	Fine-tuning Batch normalization	Valid the proposed method on the collected datasets	More than 90%.
[2]	VGG	LEVIR, NWPUVHR-10 TACDD, OIRDS	RAM, small target prior, Fisher information, Bayesian statistics	Tiny-networks, Medium-networks, VGG-f based networks	Improve mAP by 1%- 4%.
[3]	RetinaNet, Resnet, Inception-Resnet	Collect data with UAV	Focal loss mechanism, Fine-tuning	Faster RCNN with ResNet Faster RCNN with Inception-v4, RetinaNet	More than 75.6%.
[53]	Faster R-CNN, SSD, YOLOv2	Collect data with UAV	Arbitrary quadrilateral bounding boxes	Valid the proposed method on the collected datasets	Up to 90.3%, 90.1%, 77.4%, respectively.
[19]	LeNet-5	Collect data with UAV and RGB sensors	Sliding window segmentation, Post processing	Different patch size, filter size and vegetation indices	More than 95.8%.
[4]	VGG-16	Collect data with UAV	Fine-tuning, DNN Transfer learning	TL + LR, (L,5), TL + SVM, TL + ERT (300), TL + RF (300), (L1) TL + NN(AM)	Up to 90% accuracy.
[5]	General CNN	Collected data with WorldView-3, NWPU-RESISC45	Multi-resolution feature, Fusion approach, Batch-normalization	MR_a, MR_b, MR_c Baseline-ft, Baseline	More than 81.5%.
[7]	Faster R-CNN	Collect data with UAV	Convolutional network ZF	Valid the method on the collected datasets	More than 93.9%.
[62]	General CNN	Public dataset: MicroMappers	Evaluation protocol Curriculum learning	Baseline	Up to 90% Recall.
[12]	VGG-16	Collect data with Worldview-3	Patch-based Image Labeling	Random Forest (RF), Gradient Boosting (GB)	More than 92.13%.
[39]	VGG-16	Public dataset: NWPU VHR-10, RSOD dataset	Fine-tuning, Cross-entropy loss function	Transferred CNN, RICNN, SSD, R-P-Faster R-CNN, Rotation-Insensitive CNN, Deformable CNN	More than 92.13%.
[54]	VGG-16	Public dataset: VEDAI, Munich and Vehicle datasets	Multi-feature maps of different hierarchies and scales, anchor scheme, Multi-task loss function	Faster R-CNN (VGG-16), Faster R-CNN (Z&F), SLIC with Z&F, SLIC with VGG-16	More than 80.3% Recall.
[63]	Mask R-CNN and U-Net, ResNet-101	LBAI	Fine-tuning, Adam optimizer	YOLOv2, SSH, Tiny Face	For easy case, SSH got F1 score of 92.5%. For hard case, Tiny Face got F1 score of 80.2%.

Table 5. Applications of image classification.

References	CNN framework	Dataset	Classes	Other method used	Comparisons	Accuracy
[64]	General CNN	UC Merced Dataset	21 classes	Grid size reduction, factorization, hyper parameter tuning, residual units	CaffeNet from scratch, CaffeNet transfer learning, GoogLeNet from scratch, GoogLeNet transfer learning	Up to 78%.
[65]	General CNN	UCM dataset, WHU-RS, NWPU-RESISC45, AID datasets	21 classes 19 classes 45 classes 30 classes	SSGF co-training-based self-label method	Supervised learning, Self-training, Co-training SSGA-E	More than 91.35%.
[66]	ConvNets OverFeat CaffeNet	UC Merced dataset	21 classes	Fusing ConvNets, Descriptors based on bags of visual words (BoVW)	OverFeatS + OverFeatL OverFeatS + Caffe OverFeatL + Caffe	More than 90%.
[67]	AlexNet CaffeNet VGG	UC Merced dataset, WHU-RS Dataset	21 classes 19 classes	Transfer learning	AlexNet CaffeNet VGG-F VGG-M VGG-S VGG-VD16, VGG-VD19 PlacesNet OverFeat + CaffeNet	More than 96.9%.
[40]	AlexNet VGGNet GoogLeNet	NWPU RESISC45, AID data set, UC Merced dataset	21 classes 30 classes 45 classes	Metric learning, Regularization	Fine-tuned CNN Transferred AlexNet + SVM Fine-tuned AlexNet + SVM	More than 96.67%.
[11]	LeNet	Collect data with UAV	2 classes	Customized X8-Skywalker frame	Valid the propoed method on the collected dataset	Up to 86.8%.
[68]	ResNet	Landsat 8 dataset Sentinel 1 dataset	2 classes	Fine-tuning, Transfer Learning, Adam optimizer	Valid the propoed method on the public dataset	Up to 88.1%.
[6]	Overfeat model	UC Merced dataset	21 classes	Feature Fusion t-SNE	Random forest (RF), CNNs with RGB, CNNs with Overfeat	Up to 92.4%.
[69]	AlexNet VGG-16	VEDAI dataset	9 classes	SVM, Random Forest	Valid the proposed method on the public dataset	Up to 99%.
[70]	General CNN	AVIRIS and ROSIS,	20 classes 10 classes	Randomized PCA, MLP classifier	Linear kernel SVM, RBF kernel SVM, R-PCA RBF-SVM	More than 98.88%.
[71]	VGG	Collect data with UAV	2 classes	Connected components, Morphological operations	Valid the proposed method on the collected dataset	Up to 84%.
[72]	SegNet VGG-16	Publicly available dataset of New York City Land	12 classes	Multimodal data fusion, encoder-decoder, PCA Spatial Interpolation	Valid the proposed method on the public dataset	Up to 78.1%.
[73]	General CNN	UC Merced dataset, Indian Pines test site	21 classes	Sparse features learning, Greedy Layerwise, Unsupervised Pretraining	PCA, KPCA	Up to 84.53%.

Table 6. Applications of image semantic segmentation.

Reference	CNN framework	Dataset	Other method used	Comparisons	Accuracy
[51]	ResNet-50, RefineNet	RIT-18 dataset	Batch normalization MICA, SCAE	FCN, KNN, SVM, MLP, MP, MICA, SCAE, MRS	Up to 59.8%.
[52]	VGG	Vaihingen dataset	Top-down spatial Regularization encoded	SegNet, PSPNet U-Net, AlexNet	More than 75.1%.
[55]	SegNet	high-resolution and LiDAR data optical imagery	Batch normalization, Multi-filter CNN, MRS based post-processing	SVM, Random Forest, G-FRNet, Multi-scale CNN, Multi-filter CNN	More than 87.88%.
[56]	FCN	ISPRS 2D semantic labeling contest	FCN-Pascal, drop-out, FCRF post-processing	FCN-Places, FCN-ImageNet + FCN-Pascal, FCN-ImageNet + FCN-Pascal, FCN-ImageNet + FCN-Pascal + FCN-Places	More than 80%.
[57]	ResNet-101	Land Information of New Zealand (LINZ)	Refinement of the open-source data	FPN, FPN + MSFF, PSPNet	The accuracy reached 96.3%, 95,8%, 96.1%, respectively.
[58]	FC-DensNet, GAN	Massachusetts, Inria aerial image labeling dataset	Deep adversarial network	Saito-multi-MA&CIS, Mnih-CNN + CRF, FC-DensNet, FCN MLP, SegNet, HF-FCN, U-Net	Up to 96.71%.
[59]	FCN, ResFCN	ISPRS Potsdam dataset	Residual learning	VGG-FCN, B-ResFCN Inception-FCN, B-inception-FCN, ResFCN	Up to 93.80%.

Table 7. Applications in image denoizing and matching.

Reference	CNN framework	Dataset	Other method used	Comparisons
[60]	AlexNet	Aerial imagery through Google Maps	Fine-tuning, Spatial Transformer (ST)	A-SIFT, GIST Fisher Vectors, VLAD, Siamese Places CNN Siamese AlexNet
[61]	VGG	Collect data in Guiyang city	Generative adversarial learning (GAN), Batch-normalization, Residual blocks	Image matching and classification between denoised and original image

Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. Advances in Neural information Processing System; Lake Tahoe, Nevada, 1097–1105, 2012.

 Google Scholar

Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition, arXiv preprint, arXiv:1409.1556 (2014).

 Google Scholar

Szegedy C, Wei L, Yangqing J, et al. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–9, 2015.

 Google Scholar

He K, Xiangyu Z, Shaoqing R, et al. Deep residual learning for image recognition. IEEE Conference on Computer Vision Pattern Recognition (CVPR), Las Vegas, NV, USA, 770–778, 2016.

 Google Scholar

Marco C, Giovanni P, Carlo S, et al. Land use classification in remote sensing images by convolutional neural networks, arXiv preprint, arXiv:1508.00092 (2015).

 Google Scholar

Guofeng S, Wen Y, Tao X, et al. High-resolution satellite scene classification using a sparse coding based multiple feature combination. Int J Rem Sens. 2012;33(8):2395–2412.

 Web of Science ®Google Scholar

Gong C, Junwei H, Xiaoqiang L. Remote sensing image scene classification: benchmark and state of the art. Proc IEEE. 2017;105(10):1865–1883.

 Web of Science ®Google Scholar

GuiSong X, Jingwen H, et al. AID: a benchmark data set for performance evaluation of aerial scene classification. IEEE Trans Geosci Rem Sens Let. 2017;55(7):3965–3981.

 Web of Science ®Google Scholar

Gong C, Junwei H, Peicheng Z, et al. Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS J Photog Rem Sens Let. 2014;98:119–132.

 Web of Science ®Google Scholar

Razakarivonyb S, Jurie F. Vehicle detection in aerial imagery: a small target detection benchmark. J Vis Commu Ima Rep. 2016;34:187–203.

 Web of Science ®Google Scholar

Franklin T, Brian C, Craig P, et al. Overhead imagery research data set – an annotated data library & tools to aid in the development of computer vision algorithms. AIPR 2009, DC: Washington IEEE, 2010.

 Google Scholar

Rottensteiner F, Sohn G, Jung J, et al. The ISPRS benchmark on urban object classification and 3D building reconstruction. ISPRS Anna Photog, Rem Sens Spa Infor Sci. 2012;1(3):293–298.

 Google Scholar

Kang L, Gellert M. Fast multiclass vehicle detection on aerial images. IEEE Geosci Rem Sens Let. 2015;12(9):1938–1942.

 Web of Science ®Google Scholar

Mnih V. Machine learning for aerial image labeling. University of Toronto (Canada); 2013.

 Google Scholar

Kemker R, Salvaggio C, Kanan C. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning. ISPRS J Photog Re Sens Let. 2018;145:60–77.

 Web of Science ®Google Scholar

Volpi M, Tuia D. Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images. ISPRS J Photog Rem Sens Let. 2018;144:48–60.

 Web of Science ®Google Scholar

Jinwang W, Wei G, Pan T, et al.: Bottle detection in the wild using low-altitude unmanned aerial vehicles. International Confer. Infor. Fusion (FUSION), Cambridge, UK: IEEE, 2018.

 Google Scholar

Kerkech M, Hafiane A, Canals R. Deep leaning approach with colorimetric spaces and vegetation indices for vine diseases detection in UAV images. Comp Electr Agric. 2018;155:237–243.

 Web of Science ®Google Scholar

Kasthurirangan G, Hoda G, Akash V, et al. Crack damage detection in unmanned aerial vehicle images of civil infrastructure using pre-trained deep learning model. Int J Traffic Trans Eng. 2018;8(1):1–14.

 Google Scholar

Diogo D, Francesco N, Norman K, et al. Multi-resolution feature fusion for image classification of building damages with convolutional neural networks. Rem Sens. 2018;10(10):1636–1661.

 Google Scholar

Xiangtao Z, Yuan Y, Lu X. A deep scene representation for aerial scene classification. IEEE Trans: Geosci Rem Sens. 2019;57(7):4799–4809.

 Web of Science ®Google Scholar

Benjamin K, Diego M, Devis T. Detecting mammals in UAV images: best practices to address a substantially imbalanced dataset with deep learning. Rem Sens Enviro. 2018;216:139–153.

 Web of Science ®Google Scholar

Mohammad R, Yun Z, Rakesh M, et al. Using a VGG-16 network for individual tree species detection with an object-based approach. 10th IAPR Work. Pat. Rec. in Rem. Sensg (PRRS), IEEE, Beijing, China, 2018.

 Google Scholar

Tayara H, Chong KT. Object detection in very high-resolution aerial images using one-stage densely connected feature pyramid network. Sens. 2018;18(10):3341–3358.

 PubMed Web of Science ®Google Scholar

Jiandan Z, Tao L, Guangle Y. Robust vehicle detection in aerial images based on cascaded convolutional neural networks. Sens. 2017;17(12):2720–2737.

 PubMed Web of Science ®Google Scholar

Yang L, Peng S, Highsmith M R, et al. Performance comparison of deep learning techniques for recognizing birds in aerial images. IEEE Third International Conference on DSC; Guangzhou, China, 317–324, 2018.

 Google Scholar

Gong C, Ceyuan Y, Xiwen Y, et al. When deep learning meets metric learning: remote sensing image scene classification via learning discriminative CNNs. IEEE Trans Geosci Rem Sens. 2018;56(5):2811–2821.

 Web of Science ®Google Scholar

Fedra T, Andres F, Carlos S, et al. Corn classification using deep learning with UAV imagery. An operational proof of concept. IEEE 1st ColCACI, IEEE, Medellin, Colombia, 2018.

 Google Scholar

Barak O, Annie H, et al. Infrastructure quality assessment in Africa using Satellite Imagery and Deep Learning, arXiv preprint, arXiv:.00894 (2018).

 Google Scholar

Dimitrios M, Mihai D, Thomas E, et al. Deep learning earth observation classification using ImageNet pretrained networks. IEEE Geosci Rem Sens Let. 2016;13(1):105–109.

 Web of Science ®Google Scholar

Vysakh SM, Sowmya V, Soman KP. Deep neural networks as feature extractors for classification of vehicles in aerial imagery. International Conference on SPIN, Noida, India, 105–110, 2018.

 Google Scholar

Makantasis K, Karantzalos K, Anastasios D, et al. Deep supervised learning for hyperspectral data classification through convolutional neural networks. IEEE IGARSS; Milan, Italy, 4959–4962, 2015.

 Google Scholar

Yuhao C, Javier R, Edward JD. Estimating plant centers using a deep binary classifier. IEEE SSIAI, Las Vegas, NV, USA, IEEE, 2018

 Google Scholar

Rui C, Jiasong Z, Wei T, et al. Integrating aerial and street view images for urban land use classification. Rem Sens. 2018;10(10):1553–1575.

 Google Scholar

Adriana R, Carlo G, Gustau CV. Unsupervised deep feature extraction for remote sensing image classification. IEEE Trans Geosci Rem Sens. 2016;54(3):1349–1362.

 Web of Science ®Google Scholar

Ying S, Xinchang Z, Silvano G, et al. Developing a multi-filter convolutional neural network for semantic segmentation using high-resolution aerial imagery and LiDAR data. ISPRS J Photog Rem Sens Let. 2018;143:3–14.

 Web of Science ®Google Scholar

Marmanis D, Wegner JD, et al. Semantic segmentation of aerial images with an ensemble of CNNs. ISPRS Ann photog. Rem Sens Spa Infor Sci. 2016;3:473–480.

 Google Scholar

Qi C, Lei W, Yifan W, et al. Aerial imagery for roof segmentation: a large-scale dataset towards automatic mapping of buildings. ISPRS J Photog Rem Sens Let. 2019;147:42–55.

 Web of Science ®Google Scholar

Xiang L, Xiaojing Y, Yi F. Building-a-nets: robust building extraction from high-resolution remote sensing images with adversarial networks. IEEE J Sel Top App Earth Obs Rem Sens Let. 2018;11(10):3680–3687.

 Web of Science ®Google Scholar

Lichao M, Xiaoxiang Z. Vehicle instance segmentation from aerial image and video using a multi-task learning residual fully convolutional network. IEEE Trans Geosci Rem Sens. 2018;56(11):6699–6711.

 Web of Science ®Google Scholar

Hani A, Eduard T, et al. Learning to match aerial images with deep attentive architectures. IEEE Conference Computer Vision Pattern Recognition (CVPR), 3539–3547, 2016.

 Google Scholar

Ruihua W, Xionwu X, Bingxuan G, et al. An effective image denoising method for UAV images via improved generative adversarial networks. Sens. 2018;18(7):1985–2007.

 PubMed Web of Science ®Google Scholar

Wuttichai B, Yumin T, Yinghua Y, et al. A deep learning approach on building detection from unmanned aerial vehicle-based images in riverbank monitoring. Sensors. 2018;18(11):3921–3933.

 PubMed Web of Science ®Google Scholar

Zhengxia Z, Zhenwei S. Random access memories: a new paradigm for target detection in high resolution aerial remote sensing images. IEEE Trans Image Proc. 2018;27(3):1100–1111.

 PubMed Web of Science ®Google Scholar

Griffiths D, Boehm J. Rapid object detection systems, utilising deep learning and unmanned aerial systems (UAS) for civil engineering applications. ISPRS TC II Mid-term Sym, Riva del Garda, Italy, Proceedings of ISPRS 2018; 2018, 391–398.

 Google Scholar

Amin F, Mohammad E, Babak M. A deep residual neural network for low altitude remote sensing image classification. Ira. Joi. CFIS, IEEE, Kerman, Iran, 2018.

 Google Scholar

Wei H, Ruyi F, Lizhe W, et al. A semi-supervised generative framework with deep learning features for high-resolution remote sensing image scene classification. ISPRS J: Photog Rem Sens Let. 2018;145:23–43.

 Web of Science ®Google Scholar

Otavio AP, Keiller N, Jefersson AdS, et al. Do deep features generalize from everyday objects to remote sensing and aerial scenes domains. IEEE Conference Computer Vision Pattern Recognition (CVPR) Workshops, 44–51, 2015.

 Google Scholar

Fan H, GuiSong X, Jingwen H, et al. Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Rem Sens. 2015;7(11):14680–14707.

 Google Scholar