A survey of remote sensing image classification based on CNNs

Figures & data

Figure 1. Structure of CNN.

Table 1. Open datasets used for RS image classification.

Dataset			Bands	Pixel size (m)	Data source	Image size (pixels)	Categories	Num.
Scene classification	UC-Merced Land Use dataset (Yang & Newsam, 2010) ^1		R-G-B	0.3	United States Geological Survey National Map	256*256	21 (forest, beach harbor etc.)	2100
	Brazilian Cerrado-Savanna Scenes dataset (Nogueira, Santos, et al., 2017) ^2		R-G-NIR	5	the RapidEye satellite sensors	64*64	4 (agriculture, arboreal vegetation etc.)	1311
	Brazilian Coffee Scenes dataset (Penatti, Nogueira, & Santos, 2015)^3		R-G-NIR	10	SPOT 5 sensor	64*64	2 (coffee and non-coffee)	2876
	WHU-RS19 dataset (Xia et al., 2010) ^4		R-G-B	0.5	Google Earth	600*600	19 (airport, pond etc.)	1005
	RSSCN7 dataset (Zou, Ni, Zhang, & Wang, 2015) ^5		R-G-B		Google Earth	400*400	7 (forest, river etc.)	2800
	AID dataset (Xia et al., 2016) ^6		R-G-B		Google Earth	600*600	30 (forest, park etc.)	10000
	SIRI-WHU (Zhao, Zhong, Xia, & Zhang, 2016)		R-G-B	2	Google Earth	200*200	12 (park, agriculture etc.)	2400
Object detection	NWPU VHR-10 dataset (Cheng, Han, Zhou, & Guo, 2014) ^7		R-G-B	0.5–2	Google Earth		10 (airplane, tank etc.)	715
			R-G-NIR	0.08	Vaihigen data			85
	SZTAKI-INRIA building dataset (Benedek, Descombes, & Zerubia, 2011) ^8		R-G-B		Google Earth, IKONOS, QuickBird		2 (building, not building)	9
	UCAS-AOD dataset (Zhu et al., 2015) ^9		R-G-B		Google Earth		3 (airplane, car, background)	910
	RSOD-Dataset (Long, Gong, Xiao, & Liu, 2017) ^10		R-G-B		Google Earth		4 (aircraft, playground, overpass, oil-tank)	2326
Object segmentation	ISPRS 2D Semantic Labeling ^11	Vaihingen	R-G-B		airborne image		6 (building, tree, car etc.)	33
		Potsdam	R-G-NIR, R-G-B, R-G-B-NIR					38

Dataset websites: ¹ http://weegee.vision.ucmerced.edu/datasets/landuse.html; ² http://www.patreo.dcc.ufmg.br/2017/11/12/brazilian-coffee-scenes-dataset/; ³ http://captain.whu.edu.cn/repository.html; ⁴ http://weegee.vision.ucmerced.edu/datasets/landuse.html; ⁵ http://pan.baidu.com/s/1slSn6V; ⁶ http://www.lmars.whu.edu.cn/xia/AID-project.html; ⁷ http://pan.baidu.com/s/1hqwzXeG; ⁸ http://web.eee.sztaki.hu/remotesensing/building_benchmark.html; ⁹ http://www.ucassdl.cn/resource.asp; ¹⁰ https://github.com/RSIA-LIESMARS-WHU/RSOD-Dataset-; ¹¹ http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html;

Table 2. Summary of scene classification methods.

Method	Feature	Accuracy（%）
LPCNN (Zhong et al., 2016)	An improved CNN model with global average pooling layer, and trained with large patch sampling	89.9
MS-DCNN (Xu et al., 2016)	Multisource-driven and a multi-kernel SVM classifier	91.34
CNN with Overfeat pretrained modelm (Marmanis et al., 2016)	A pretrained network (Sermanet et al., 2013) is directly used as an extractor for remote sensing image features	92.4
CNN-ELM (Weng et al., 2017)	A pretrained network is directly used as an extractor for remote sensing image features and an extreme learning machine (ELM) classifier	95.62
Two-stage neural network ensemble model (Li, Fu, et al., 2017)	Combine pretrained networks and RBM retrained network as a two stage-training network and combine the classification results of several networks in testing stages	95.11
GCF-LOF CNN (Zeng, Chen, Chen, & Li, 2018)	a novel CNN by integrating the global-context features (GCFs) and local-object-level features (LOFs)	96
Deep-local-global feature fusion framework (DLGFF) (Zhu, Zhong, Liu, Zhang, & Li, 2018)	DLGFF establishes a framework integrating multi-level semantics from the global texture feature–based method, the bag-of-visual-words (BoVW) model, and a pre-trained CNN.	96.37
Deep random-scale stretched convolutional neural network (SRSCNN) (Liu, Zhong, Fei, Zhu, & Qin, 2018)	SRSCNN forces the CNN to extract features that are robust to the scale variation.	95.96

Table 3. Comparison of object detection methods.

Method	Sliding window methods for detection	Region proposal methods for detection
		R-CNN	Faster R-CNN
Input images for the CNN	Image area	Image area	Entire image (no fixed size is required)
Advantages	Relatively high object identification capability	Relatively high object identification capability; relatively few candidate regions (which can be controlled within thousands or hundreds of regions)	Relatively high object identification capability; end-to-end object detection; independent of a region proposal method; very high detection speed
Disadvantages	The method of exhaustion generates a large number of candidate regions, resulting in a significant volume of repeated calculations in the subsequent operations, and therefore has relatively low efficiency when used to process remote sensing images of large scenes.	Relatively long time to calculate candidate regions. Feature extraction will result in repeated calculations.	Relatively complex training process
Typical literature	Buildings (Li, Li, et al., 2017; Zhang, Wang, et al., 2016);	Multi-class targets (Cheng, Zhou, & Han, 2016);	Electrical devices detection in UAV images (Wang, Tian, Liu, Liu, & Li, 2017);
	Ships (Huang, Jiang, Zhang, & Yao, 2017);
	Palm trees (Li, Fu, et al., 2016);
	Golf courses (Ishii, Nakamura, Nakada, Mochizuki, and Ishikawa, 2015)	Aircraft (Wu, Zhang, Zhang, & Xu, 2015)	Aircraft (Deng et al., 2017; Zhang, Du, et al.,2016)
	Aircraft (Yao, Wang, Huo, and Fang, 2017)

Table 4. Comparison of object segmentation methods.

Method	Patch-based CNN	End-to-end CNN
Input image	Image patches	Entire images (no fixed size is required)
Advantages	Relatively good extraction results	Able to perform end-to-end object extraction and directly generate classification maps
Disadvantages	A large volume of redundant calculations; relatively complex processing process; generally requiring post-processing of the extraction results	Insufficiently fine classification results; lack of spatial homogeneity
Typical literature	Buildings (Li, Zhang, et al.,2016; Qu, Ren, Liu, & Li, 2017; Guo et al., 2016);	Buildings (Maggiori et al., 2017; Xu et al., 2018; Wu et al., 2018)
	Farmlands (Lu et al., 2015);
	Land cover (Chen, Fan, Wang, Wu, & Sun, 2017; Zhang, Zheng, Tang, & Zhao, 2017);
	Water bodies (He, Du, Chen, & Chen, 2017);
	Roads (Li, Zang, et al., 2016);
	Roads and buildings (Alshehhi, Marpu, Woon, & Mura, 2017; Saito & Aoki, 2015)

Yang, Y., & Newsam, S. (2010). Bag-of-visual-words and spatial extensions for land-use classification. Sigspatial International Conference on Advances in Geographic Information Systems, New York, NY.

 Google Scholar

Nogueira, K., Santos, J. A. D., Fornazari, T., Silva, T. S. F., Morellato, L. P., & Torres, R. D. S. (2017). In Towards vegetation species discrimination by using data-driven descriptors. 2016 9th IAPR Workshop on Pattern Recogniton in Remote Sensing (PRRS) (pp. 1–6), IEEE, Cancun, Mexico.

 Google Scholar

Penatti, O. A. B., Nogueira, K., & Santos, J. A. D. (2015). Do deep features generalize from everyday objects to remote sensing and aerial scenes domains. Computer Vision and Pattern Recognition Workshops 44–51.

 Google Scholar

Xia, G. S., Yang, W., Delon, J., Gousseau, Y., Sun, H., & Maitre, H. (2010). Structural high-resolution satellite image indexing. ISPRS TC VII Symposium - 100 Years ISPRS,  Vienna, Austria.

 Google Scholar

Zou, Q., Ni, L., Zhang, T., & Wang, Q. (2015). Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience & Remote Sensing Letters, 12, 2321–2325.

 Web of Science ®Google Scholar

Xia, G. S., Hu, J., Hu, F., Shi, B., Bai, X., Zhong, Y., … Lu, X. (2016). Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience & Remote Sensing, 55(7), 3965–3981.

 Web of Science ®Google Scholar

Zhao, B., Zhong, Y., Xia, G. S., & Zhang, L. (2016). Dirichlet-derived multiple topic scene classification model for high spatial resolution remote sensing imagery. IEEE Transactions on Geoscience & Remote Sensing, 54, 2108–2123.

 Web of Science ®Google Scholar

Cheng, G., Han, J., Zhou, P., & Guo, L. (2014). Multi-class geospatial object detection and geographic image classification based on collection of part detectors. Isprs Journal of Photogrammetry & Remote Sensing, 98, 119–132.

 Web of Science ®Google Scholar

Benedek, C., Descombes, X., & Zerubia, J. (2011). Building development monitoring in multitemporal remotely sensed image pairs with stochastic birth-death dynamics. IEEE Transactions on Pattern Analysis & Machine Intelligence, 34, 33–50.

 Web of Science ®Google Scholar

Zhu, H., Chen, X., Dai, W., Fu, K., Ye, Q., & Jiao, J. (2015). Orientation robust object detection in aerial images using deep convolutional neural network. IEEE International Conference on Image Processing (pp. 3735–3739), Quebec City, QC.

 Google Scholar

Long, Y., Gong, Y., Xiao, Z., & Liu, Q. (2017). Accurate object localization in remote sensing images based on convolutional neural networks. IEEE Transactions on Geoscience & Remote Sensing, 55, 2486–2498.

 Web of Science ®Google Scholar

Zhong, Y., Fei, F., & Zhang, L. (2016). Large patch convolutional neural networks for the scene classification of high spatial resolution imagery. Journal of Applied Remote Sensing, 10, 025006.

 Web of Science ®Google Scholar

Xu, S. H., Mu, X. D., Zhao, P., & Ma, J. (2016). Scene classification of remote sensing image based on multi-scale feature and deep neural network. Acta Geodaetica Et Cartographica Sinica, 45, 834–840.

 Google Scholar

Marmanis, D., Datcu, M., Esch, T., & Stilla, U. (2016). Deep learning earth observation classification using imagenet pretrained networks. IEEE Geoscience and Remote Sensing Letters, 13, 105–109.

 Web of Science ®Google Scholar

Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. unpublished paper. [Online]. Retrieved from http://arxiv.org/abs/1312.6229

 Google Scholar

Weng, Q., Mao, Z., Lin, J., & Guo, W. (2017). Land-use classification via extreme learning classifier based on deep convolutional features. IEEE Geoscience and Remote Sensing Letters, 14, 704–708.

 Web of Science ®Google Scholar

Li, H., Fu, K., Xu, G., Zheng, X., Ren, W., & Sun, X. (2017). Scene classification in remote sensing images using a two-stage neural network ensemble model. Remote Sensing Letters, 8, 557–566.

 Web of Science ®Google Scholar

Xu, Y., Wu, L., Xie, Z., & Chen, Z. (2018). Building extraction in very high resolution remote sensing imagery using deep learning and guided filters. Remote Sensing, 10, 144.

 Web of Science ®Google Scholar

Zhu, Q., Zhong, Y., Liu, Y., Zhang, L., & Li, D. (2018). A deep-local-global feature fusion framework for high spatial resolution imagery scene classification. Remote Sensing, 10, 568.

 Web of Science ®Google Scholar

Liu, Y., Zhong, Y., Fei, F., Zhu, Q., & Qin, Q. (2018). Scene classification based on a deep random-scale stretched convolutional neural network. Remote Sensing, 10, 444.

 Web of Science ®Google Scholar

Li, Z., Li, Y. S., Wu, X., Liu, G., Lu, H., & Tang, M. (2017). Hollow village building detection method using high resolution remote sensing image based on CNN. Transactions of the Chinese Society of Agricultural Machinery, 48, 160–165.

 Google Scholar

Zhang, Z. X., Wang, Y. H., Liu, Q. J., Li, L. L., & Wang, P. (2016). A CNN based functional zone classification method for aerial images. IGARSS,  Beijing, China.

 Google Scholar

Cheng, G., Zhou, P., & Han, J. (2016). Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 54, 7405–7415.

 Web of Science ®Google Scholar

Wang, W. G., Tian, B., Liu, Y., Liu, L., & Li, J. X. (2017). Study on the electrical devices in UAV images based on region based convolutional neural networks. Journal of Geo-information Science, 19, 256–263.

 Google Scholar

Huang, J., Jiang, Z. G., Zhang, H. P., & Yao, Y. (2017). Ship object detection in remote sensing images using convolutional neural networks. Journal of Beijing University of Aeronautics and Astronautics, 43, 1841–1849.

 Google Scholar

Li, W., Fu, H., Yu, L., & Cracknell, A. (2016). Deep learning based oil palm tree detection and counting for high-resolution remote sensing images. Remote Sensing, 9, 22.

 Web of Science ®Google Scholar

Ishii, T., Nakamura, R., Nakada, H., Mochizuki, Y., & Ishikawa, H. (2015). Surface object recognition with cnn and svm in landsat 8 images. 2015 14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, Japan.

 Google Scholar

Wu, H., Zhang, H., Zhang, J., & Xu, F. (2015). Typical target detection in satellite images based on convolutional neural networks. IEEE International Conference on Systems (pp. 2956–2961), Las Vegas, NV.

 Google Scholar

Deng, Z., Lei, L., Sun, H., Zou, H., Zhou, S., & Zhao, J. (2017). An enhanced deep convolutional neural network for densely packed objects detection in remote sensing images. 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP) (pp. 1–4), Shanghai, China.

 Google Scholar

Zhang, F., Du, B., Zhang, L., & Xu, M. (2016). Weakly supervised learning based on coupled convolutional neural networks for aircraft detection. IEEE Transactions on Geoscience and Remote Sensing, 54, 5553–5563.

 Web of Science ®Google Scholar

Yao, X. K., Wang, L. H., Huo, H., & Fang, T. (2017). Airplane object detection in high resolution remote sensing imagery based on multi-structure convolutional neural network. Computer Engineering, 43, 259–267.

 Google Scholar

Li, J., Zhang, R., & Li, Y. (2016). Multiscale convolutional neural network for the detection of built-up areas in high-resolution SAR images. Geoscience and Remote Sensing Symposium (IGARSS) (pp. 910–913),  Beijing, China.

 Google Scholar

Qu, C., Ren, Y. H., Liu, Y. L., & Li, Y. (2017). Functional classification of urban buildings in high resolution remote sensing images through POI-assisted analysis. Journal of Geo-information Science, 19, 831–837.

 Google Scholar

Guo, Z., Shao, X., Xu, Y., Miyazaki, H., Ohira, W., & Shibasaki, R. (2016). Identification of village building via google earth images and supervised machine learning methods. Remote Sensing, 8, 271.

 Web of Science ®Google Scholar

Maggiori, E., Tarabalka, Y., Charpiat, G., & Alliez, P. (2017). Convolutional neural networks for large-scale remote-sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 55, 645–657.

 Web of Science ®Google Scholar

Wu, G., Shao, X., Guo, Z., Chen, Q., Yuan, W., Shi, X., … Shibasaki, R. (2018). Automatic building segmentation of aerial imagery using multi-constraint fully convolutional networks. Remote Sensing, 10, 407.

 Web of Science ®Google Scholar

Lu, H., Fu, X., He, Y. N., Li, L. G., Zhuang, W. H., & Liu, T. G. (2015). Cultivated land information extraction from high resolution UAV images based on transfer learning. Transactions of the Chinese Society of Agricultural Machinery, 46, 274–279.

 Google Scholar

Chen, Y., Fan, R. S., Wang, J. X., Wu, Z. L., & Sun, R. X. (2017). High resolution Image Classification method combining with minimum noise fraction rotation and convolution neural network. Laser & Optoelectronics Progress, 54, 434–439.

 Google Scholar

Zhang, B. (2017). Current status and future prospects of remote sensing. Bulletin of Chinese Academy of Sciences, 32, 774–784.

 Google Scholar

He, H. Q., Du, J., Chen, T., & Chen, X. Y. (2017). Remote sensing image water body extraction combing NDWI with convolutional neural network. Remote Sensing Information, 32, 82–86.

 Google Scholar

Li, P., Zang, Y., Wang, C., Li, J., Cheng, M., Luo, L., & Yu, Y. (2016). Road network extraction via deep learning and line integral convolution. 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) (pp. 1599–1602), Beijing, China.

 Google Scholar

Alshehhi, R., Marpu, P. R., Woon, W. L., & Mura, M. D. (2017). Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks. ISPRS Journal of Photogrammetry and Remote Sensing, 130, 139–149.

 Web of Science ®Google Scholar

Saito, S., & Aoki, Y. (2015). Building and road detection from large aerial imagery. Image Processing: Machine Vision Applications VIII (Vol. 9405, p. 94050K), International Society for Optics and Photonics, San Diego, CA.

 Google Scholar

Data availability statement

Data sharing is not applicable to this article as no new data were created or analysed in this study.