End-to-End Convolutional Neural Network Feature Extraction for Remote Sensed Images Classification

Figures & data

Table 1. Hyperparameter settings compared with earlier comparative studies.

DL Hyperparameters	Chosen values for each DL hyperparameters in both earlier and our studies
	(Helber et al. 2019)	CNN (Radamanthys Stivaktakis, Tsagkatakis, and Tsakalides 2019)	CNN-FE (Ours)
Optimizers	Stochastic gradient descent (SGD)	Adagrad	Adam
Batch size	16	10	64
Learning rate	le-3	-	0.0001
Iteration Epochs	120	300	100
Loss function	categorical_cross_entropy	binary_cross_entropy	categorical_cross_entropy
Activation functions	Relu	Relu, sigmoid	Relu, softmax
Dropout	-	0,0.25, 0.50, 0.75	0.5
Early stopping	-	-	Automatically stopping

Figure 1. Structure of CNN-FE model.

Figure 2. Sample images in each classes of the UCM dataset.

Figure 3. Sample images in each class of the SIRI-WHU dataset used for CNN-FE model checking.

Table 2. Summarization of the classification performance of CNN-FE for each class with performance measurement metrics in the UC-Merced dataset.

Class name	Precision	Recall	F1-score	Support
Agricultural	0.91	1.00	0.95	20
Airplane	0.94	0.80	0.86	20
Baseballdiamond	0.85	0.85	0.85	20
Beach	0.95	1.00	0.98	20
Buildings	0.79	0.75	0.77	20
Chaparral	1.00	0.95	0.97	20
Denseresidential	0.57	0.80	0.67	20
Forest	0.86	0.95	0.90	20
Freeway	0.95	0.90	0.92	20
Golfcourse	0.80	0.60	0.69	20
Harbor	0.91	1.00	0.95	20
Intersection	0.88	0.75	0.81	20
Mediumresidential	0.58	0.90	0.71	20
Mobilehomepark	0.92	0.55	0.69	20
Overpass	0.81	0.85	0.83	20
Parkinglot	1.00	0.85	0.92	20
River	0.78	0.90	0.84	20
Runway	0.95	0.90	0.92	20
Sparseresidential	0.75	0.90	0.82	20
Storagetanks	1.00	0.90	0.95	20
Tenniscourt	1.00	0.70	0.82	20

Table 3. Summarizations of the classification performance of VGG19 for each class in performance measurement metrics in the UC-Merced dataset.

Class name	Precision	Recall	F-score	Support
Agricultural	1.00	1.00	1.00	20
Airplane	0.95	0.90	0.92	20
Baseballdiamond	1.00	0.90	0.95	20
Beach	1.00	0.95	0.97	20
Buildings	0.82	0.70	0.76	20
Chaparral	1.00	1.00	1.00	20
Denseresidential	0.50	0.55	0.52	20
Forest	0.82	0.90	0.86	20
Freeway	1.00	0.85	0.92	20
Golfcourse	0.86	0.60	0.71	20
Harbor	1.00	1.00	1.00	20
Intersection	0.81	0.85	0.83	20
Mediumresidential	0.69	0.90	0.78	20
Mobilehomepark	0.61	0.55	0.58	20
Overpass	0.84	0.80	0.82	20
Parkinglot	0.95	0.95	0.95	20
River	0.76	0.95	0.84	20
Runway	0.87	1.00	0.93	20
Sparseresidential	0.90	0.90	0.90	20
Storagetanks	0.95	0.95	0.95	20
Tenniscourt	0.85	0.85	0.85	20

Table 4. Summarizations the classification performance of CNN-FE for each individual class with performance measurement metrics in SIRI-WHU dataset.

Class name	Precision	Recall	F1-score	Support
Agriculture	0.81	0.65	0.72	40
Commercial	0.94	0.82	0.88	40
Harbor	0.88	0.75	0.81	40
Idle_land	0.62	0.78	0.69	40
Industrial	0.90	0.90	0.90	40
Meadow	0.55	0.60	0.57	40
Overpass	0.92	0.90	0.91	40
Park	0.79	0.68	0.73	40
Pond	0.72	0.82	0.77	40
Residential	0.83	0.95	0.88	40
River	0.81	0.75	0.78	40
Water	0.93	1.00	0.96	40

Table 5. Summarizations of the classification performance of VGG19 for each class with performance measurement metrics in the SIRI-WHU dataset.

Class name	Precision	Recall	F1-score	Support
Agriculture	0.91	0.50	0.65	40
Commercial	0.72	0.95	0.82	40
Harbor	0.88	0.93	0.90	40
Idle_land	0.86	0.62	0.72	40
Industrial	0.85	0.88	0.86	40
Meadow	0.63	0.60	0.62	40
Overpass	0.80	0.97	0.88	40
Park	0.50	0.60	0.55	40
Pond	0.67	0.75	0.71	40
Residential	0.92	0.88	0.90	40
River	0.85	0.72	0.78	40
Water	0.98	1.00	0.99	40

Figure 4. Training and validation accuracies with and without applying early stopping technique. (a) Before applying early stopping. (b) After applying early stopping.

Figure 5. Training and validation accuracies in VGG19 with and without applying early in stopping technique in UC-Merced dataset. (a) Before applying early stopping. (b) After applying early stopping.

Figure 6. Training and validation accuracies of CNN-FE model in SIRI-WHU dataset with and without applying early stopping technique. (a) Before applying early stopping. (b) After applying early stopping.

Figure 7. Training and validation accuracies of VGG19 in the SIRI-WHU dataset with and without applying the early stopping technique. (a) Before applying early stopping. b) After applying early stopping.

Figure 8. Training and validation losses with and without applying early stopping technique. (a) Losses before applying early stopping. (b) Losses after applying early stopping.

Figure 9. Training and validation losses in VGG19 with and without applying the early stopping technique in the UC-Merced dataset. (a) Before applying early stopping. b) After applying early stopping.

Figure 10. Training and validation losses of CNN-FE model in SIRI-WHU dataset with and without applying early stopping technique. (a) Before applying early stopping. (b) After applying early stopping.

Figure 11. Training and validation losses of VGG19 in SIRI-WHU dataset with and without applying early stopping technique. (a) Before applying early stopping. (b) After applying early stopping.

Figure 12. CM performance results in the CNN-FE model for each labeled class.

Figure 13. CM performance results of VGG19 pre-trained for each labeled class.

Figure 14. CM performance results of CNN-FE for each class classification in SIRI-WHU.

Figure 15. CM performance results of VGG19 for each class classification in SIRI-WHU.

Table 6. Class comparisons in precision, recall, and F1-score (%) on the two models and datasets.

Dataset	Method	Precision performance		Recall performance		F1-score performance
		Perfect (1) classes	Lower (-) classes	Perfect (1) classes	Lower (-) classes	Perfect (1) classes	Lower (-) classes
UC-Merced	CNN-FE	Chaparral, parkinglot, storagetanks, and tenniscour	Denseresidential (0.57)	Agricultural, beach and harbor	Mobilehomepark (0.55) and golfcourse (0.60)	None	Denseresidential (0.67)
	VGG19	Agricultural, baseballdiamond, beach, charparral, freeway, and harbor	Denseresidential (0.50)	Agricultural, charparral, harbor and runway	Denseresidential (0.55)	Agricultural, charparral, and harbor	Denseresidential (0.52)
SIRI-WHU	CNN-FE	None	Meadow (0.55)	Water	Meadow (0.60)	None	Meadow (0.57)
	VGG19	None	Park (0.50)	Water	Agriculture	None	Park (0.55)

Table 7. Results of accuracy (%) performances at random early stopping technique.

Dataset	Methods	Stopped at epoch # out of 100	Accuracy performance results
			Before early stopping	After early stopping
UC-Merced	CNN-FE	26	85.95	89.76
	VGG19	42	85.00	85.95
SIRI-WHU	CNN-FE	22	78.67	80.00
	VGG19	41	76.88	78.33

Table 8. General comparison of the accuracy (%) with the state-of-the-arts in the UC-Merced target dataset.

Method	Dropout	Precision	Recall	F1-score	Accuracy
CNNs (Radamanthys Stivaktakis, Tsagkatakis, and Tsakalides 2019)	0.50	85.4	83.3	84.3	75.7
	0.75	85.5	85.8	85.7	77.7
CNN (Helber et al. 2019)	-	-	-	-	82.38
CNN-FE (Ours)	0.50	90.00	88.00	88.99	89.76
VGG19 (Ours)	0.50	88.00	86.00	86.99	85.95

Helber, P., B. Bischke, A. Dengel, and D. Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7):2217–26. doi:10.1109/JSTARS.2019.2918242.

 Web of Science ®Google Scholar

Stivaktakis, R., G. Tsagkatakis, and P. Tsakalides. 2019. Deep learning for multilabel land cover scene categorization using data augmentation. IEEE Geoscience and Remote Sensing Letters 16 (7):1031–1035. doi:10.1109/LGRS.2019.2893306.

 Web of Science ®Google Scholar