Semantic segmentation of high spatial resolution images with deep neural networks

Figures & data

Figure 1. (a) A residual unit proposed in ResNet-101 (He et al. 2016a). (b) An improved residual unit proposed in ResNet-101-v2 (He et al. 2016b).

Figure 2. A building block with a bottleneck design for ResNet-101 (He et al. 2016a).

Figure 3. Overview of the PSPNet (Zhao et al. 2017a).

Figure 4. Pyramid pooling module (Zhao et al. 2017a).

Figure 5. Overview of our proposed deep neural network.

Table 1. The dimensions of 16 TOP tiles in the training set and 17 TOP tiles in the testing set.

Tiles in the training set			Tiles in the testing set
Patch Number	Column	Row	Patch Number	Column	Row
1	1919	2569	2	2428	2767
3	2006	3007	4	1887	2557
5	1887	2557	6	1887	2557
7	1887	2557	8	1887	2557
11	1893	2566	10	1887	2557
13	2818	2558	12	1922	2575
15	1919	2565	14	1919	2565
17	2336	1281	16	1919	2565
21	1903	2546	20	1866	2315
23	1903	2546	22	1903	2546
26	2995	1783	24	1903	2546
28	1917	2567	27	1917	3313
30	1934	2563	29	1917	2563
32	1980	2555	31	1980	2555
34	1388	2555	33	1581	2555
37	1996	1995	35	2805	1884
			38	3816	2550

Table 2. Experimental results with a different number of auxiliary losses. Loss 1, 2 and 3 are shown in Figure 5.

Method	OA(%)
	Vaihingen dataset	Potsdam dataset
ResNet101-v2(with Loss 3)	84.70	71.90
ResNet101-v2(with Loss 2 and 3)	85.70	87.80
ResNet101-v2(with Loss 1, 2 and 3)	85.60	88.00

Table 3. Evaluation of results in the Vaihingen dataset using the full reference set.

Methods		Impervious Surfaces	Buildings	Low Vegetation	Trees	Cars	Clutter
SP-SVL-3 (Gerke 2014)	Precision	81.00	89.90	76.10	79.60	63.20	30.50
	Recall	86.50	88.00	71.00	84.50	42.80	0.70
	F1	83.60	88.90	73.40	82.00	51.00	1.30
	OA	81.80
CNN-HAW (Tschannen et al. 2016)	Precision	84.70	87.90	77.80	80.20	50.00	68.80
	Recall	83.20	91.60	69.90	88.00	52.20	18.30
	F1	83.90	89.70	73.60	83.90	51.10	28.90
	OA	82.60
CNN-FPL (Volpi and Tuia 2017)	Precision	84.30	90.10	77.70	84.50	71.70	53.60
	Recall	88.40	90.80	78.40	83.30	38.70	2.10
	F1	86.30	90.50	78.00	83.90	50.30	4.10
	OA	84.30
Our method	Precision	86.50	90.80	79.50	84.40	83.00	97.20
	Recall	89.60	91.90	77.80	85.90	50.40	18.00
	F1	88.00	91.30	78.60	85.20	62.70	30.30
	OA	85.70

Table 4. OAs of results in the Vaihingen dataset using the reference set with eroded boundaries.

Methods	OA
SP-SVL-3 (Gerke 2014)	84.80
CNN-HAW (Tschannen et al. 2016)	85.90
CNN-FPL (Volpi and Tuia 2017)	87.30
SegNet-p (Marmanis et al. 2018)	84.80
Our method	88.70

Figure 6. Example results of test images in the Vaihingen dataset. (a) The original image, (b) the results of SP-SVL-3, (c) the results of CNN-HAW, (d) the results of CNN-FPL, (e) the results of PSPNet and (f) our results. White: impervious surfaces, Blue: buildings, Cyan: low vegetation, Green: trees, Yellow: cars, Red: clutter/background. (Best viewed in color version).

Table 5. Evaluation of results in the Potsdam dataset using the full reference set.

Methods		Impervious Surfaces	Buildings	Low Vegetation	Trees	Cars	Clutter
SP-SVL-3 (Gerke 2014)	Precision	77.80	90.00	65.10	69.00	79.60	36.70
	Recall	84.90	90.60	75.70	55.00	45.20	19.40
	F1	81.20	90.30	70.00	61.20	57.60	25.40
	OA	75.40
CNN-FPL (Volpi and Tuia 2017)	Precision	85.80	94.60	80.60	72.60	80.40	47.50
	Recall	88.70	93.70	77.60	83.70	82.30	19.60
	F1	87.20	94.10	79.00	77.70	81.30	27.70
	OA	83.40
DCNN(Kemker, Salvaggio, and Kanan 2017)	Precision	85.60	91.30	78.10	84.00	87.00	58.40
	Recall	89.50	91.50	84.70	78.80	84.40	31.00
	F1	87.60	91.40	81.20	81.30	85.70	40.50
	OA	84.40
Our method	Precision	89.00	93.70	82.10	87.40	91.60	74.80
	Recall	92.00	96.30	87.50	82.80	88.60	39.40
	F1	90.50	95.00	84.70	85.00	90.10	51.60
	OA	88.00

Table 6. OAs of results in the Potsdam dataset using the reference set with eroded boundaries.

Methods	OA
SP-SVL-3 (Gerke 2014)	77.80
CNN-FPL (Volpi and Tuia 2017)	85.80
DCNN (Kemker, Salvaggio, and Kanan 2017)	86.70
SegNet-p (Marmanis et al. 2018)	84.90
Our method	90.40

Figure 7. Example results of test images in the Potsdam dataset. (a) the original image, (b) the results of SP-SVL-3, (c) the results of CNN-HAW, (d) the results of CNN-FPL, (e) the results of PSPNet and (f) our results. White: impervious surfaces, Blue: buildings, Cyan: low vegetation, Green: trees, Yellow: cars, Red: clutter/background. (Best viewed in color version).

He, K., X. Zhang, S. Ren, and J. Sun. 2016a. “Deep Residual Learning for Image Recognition.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. Las Vegas, NV, USA: IEEE.

 Google Scholar

He, K., X. Zhang, S. Ren, and J. Sun. 2016b. “Identity Mappings in Deep Residual Networks.” In European Conference on Computer Vision, 630–645. Amsterdam, The Netherlands: Springer, Cham.

 Google Scholar

Zhao, H., J. Shi, Q. Xiaojuan, X. Wang, and J. Jia. 2017a. “Pyramid Scene Parsing Network.” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6230–6239. Honolulu, HI, USA: IEEE.

 Google Scholar

Gerke, M. 2014. “Use of the Stair Vision Library within the ISPRS 2D Semantic Labeling Benchmark (Vaihingen).” doi: 10.13140/2.1.5015.9683.

 Google Scholar

Tschannen, M., L. Cavigelli, F. Mentzer, T. Wiatowski, and L. Benini. 2016. “Deep Structured Features for Semantic Segmentation.” In 25th European Signal Processing Conference, 61–65. Kos, Greece: IEEE.

 Google Scholar

Volpi, M., and D. Tuia. 2017. “Dense Semantic Labeling of Subdecimeter Resolution Images with Convolutional Neural Networks.” IEEE Transactions on Geoscience and Remote Sensing 55 (2): 881–893. doi:10.1109/TGRS.2016.2616585.

 Web of Science ®Google Scholar

Marmanis, D., K. Schindler, J. D. Wegner, S. Galliani, M. Datcu, and U. Stilla. 2018. “Classification with an Edge: Improving Semantic Image Segmentation with Boundary Detection.” ISPRS Journal of Photogrammetry and Remote Sensing 135: 158–172. doi:10.1016/j.isprsjprs.2017.11.009.

 Web of Science ®Google Scholar

Kemker, R., C. Salvaggio, and C. Kanan. 2017. “Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery Using Deep Learning.” ISPRS Journal of Photogrammetry and Remote Sensing 145: 60–77. doi:10.1016/j.isprsjprs.2018.04.014.

 Web of Science ®Google Scholar