Transfer Learning Models for Land Cover and Land Use Classification in Remote Sensing Image

ABSTRACT

Land Cover or Land Use (LCLU) classification is an important, challenging problem in remote sensing (RS) images. RS image classification is a recent technology used to extract hidden information from remotely sensed images in the observed earth environment. This classification is essential for sustainable development in agricultural decisions and urban planning using deep learning (DL) methods. DL gets more attention for accuracy and performance improvements in large datasets. This paper is aimed to apply one of the DL methods called transfer learning (TL). TL is the recent research problem in machine learning and DL approaches for image classification. DL consumes much time for training when starting from scratch. This problem could be overcome in the TL modeling technique, which uses pre-trained models to build deep TL models efficiently. We applied the TL model using bottleneck feature extraction from the pre-trained models: InceptionV3, Resnet50V2, and VGG19 to LCLU classification in the UC Merced dataset. With these experiments, the TL model has been built the outdate performance of 92.46, 94.38, and 99.64 in Resnet50V2, InceptionV3, and VGG19, respectively.

Introduction

Land cover is variable and dynamic on the earth’s surface (Abdu 2019), whereas land use is the intervention of human activities on the earth. Land cover is the earth’s surface covered by physical features like forest, river, vegetation, or others. In contrast, land use is the ability of a human to use natural resources for various purposes (Cao, Dragićević, and Li 2019). Thus, land cover and land use (LCLU) describe the earth’s features and human interaction. Classification is needed in the land cover mapping (Fang et al. 2019; Stivaktakis, Tsagkatakis, and Tsakalides 2019; Tong et al. 2020) and land use resource management (Cao, Dragićević, and Songnian 2019; Castelluccio et al. 2015; Hung, Wu, and Tseng 2020). LCLU classification is an important, challenging task (Huang et al. 2021), and it contributes to agricultural decision-making and urban forecasting on the earth observation environment for sustainable development. This classification problem will be solved using transfer learning (TL) models in remote sensing (RS) images.

RS images are earth observation geospatial data and records of environmental information. RS data face the “big data” challenges and some new challenges for DL as they raise exceptional problems to new scientific questions (X. X. Zhu et al. 2017). RS imagery data classification is a significant problem in various domains (X. Liu et al. 2019; Deng et al. 2019; Abdu 2019; Yao et al. 2019; Cheng et al. 2020; W. Zhang, Tang, and Zhao 2019; Shabbir et al. 2021; D. Zhang, Liu, and Shi 2020; Alam et al. 2021). Thus, our consideration of classification was one of the major research problems in RS imagery data. Nowadays, researchers explored the application of DL to confrontation these challenges.

Deep learning (DL) gets more attention for the LCLU classification problem in RS images (Vali, Comai, and Matteucci 2020). The DL approaches could extract the earth’s features from remotely sensed imagery data to manage the earth’s environment by properly utilizing deep classification systems. DL algorithms are calling focuses on their automatic learning ability from large datasets (Côté-Allard et al. 2019; X. Liu et al. 2019; Li et al. 2019; Deng et al. 2019; Das et al. 2019; Xie et al. 2016; Bahri et al. 2020; X. Pan et al. 2018; W. Zhao et al. 2015; X. X. Zhu et al. 2017; Yao et al. 2019; Rashid et al. 2020; Weinstein et al. 2019; Cheng et al. 2020).

In recent studies, the DL methods, especially CNNs, are widely used in RS image classification for their outstanding performance and accuracy (Marmanis et al. 2016; Das et al. 2019; W. Zhang, Tang, and Zhao 2019; Vali, Comai, and Matteucci 2020; Rashid et al. 2020). However, DL algorithms could take more time and complexity, creating over-fitting (Das et al. 2019; B. Zhao, Huang, and Zhong 2017; Rostami et al. 2019; Hung, Hui-ching, and Tseng 2020; Zou and Zhong 2018) when training the DL models from scratch. TL, the innovative DL model in machine learning, could overtake this problem because TL is an optimization technique used to thrift the processing time, superior performance, or accuracy (Kumar, Naman, and Verma 2021).

Thus, TL could apply formerly learned techniques to resolve new problems efficiently (Chen, Zhang, and Ouyang 2018). Nowadays, TL has gotten increasing attention lately for reducing training time for large datasets (Das et al. 2019; Weiss, Khoshgoftaar, and Wang 2016; Scott et al. 2017; Lima and Marfurt Lima and Marfurt, 2020; Y. Zhu et al. 2011; X. Zhang, Guo, and Zhang 2020).

The TL models could be applied in various RS domains. For instance, it has been applied for forest variable estimation (Astola et al. 2021); for object (airplane) detection (Chen, Zhang, and Ouyang 2018; M. Zhu et al. 2019); for poverty mapping (Xie et al. 2016); for labeling the Synthetic Aperture Radar (SAR) (Rostami et al. 2019); for change analysis (Qian et al. 2020); and marsh vegetation classification (M. Liu et al. 2021).

Related work has been tried to investigate CNN-based TL model on the domain area. Few researchers (Mahdianpari et al. 2018; B. Zhao, Huang, and Zhong 2017; S. J. Pan and Yang 2010; Marmanis et al. 2016; Zou and Zhong 2018; X. Zhang, Guo, and Zhang 2020; Lima and Marfurt 2020; Shabbir et al. 2021) have investigated CNN-based models using pre-trained architectures on RS image classification. Using TL, the LCLU classification problem was investigated by (Naushad and Kaur 2021; D. Zhang, Liu, and Shi 2020) using TL. However, TL in RS has not been widely explored yet (Astola et al. 2021), especially in the LCLU classification. Thus, we applied deep neural network-based TL (Li et al. 2019) in LCLU classification using RS image.

Our motivation was to apply the deep TL model with pre-trained models for the LCLU classification in RS images and improve the performance efficiently. We have listed the related papers with their recommendations in our previous work (Alem and Kumar 2020). Therefore, we also were motivated to investigate the recommended pre-trained networks suggested by (Marmanis et al. 2016; Stivaktakis, Tsagkatakis, and Tsakalides 2019).

Our objective in this paper was to apply the deep TL models and improve their performance efficiently for LCLU classification in RS images. To achieve this objective, we followed the procedures: preprocessed the UCM imagery data, extracted the image features using the bottleneck feature extraction technique, and modeled the TL with four sequential layers (flatten, dense, two activations (ReLu and softmax) and dropout layers), and evaluated using a confusion matrix.

Our contributions in this paper were:

• Applying deep TL method on RS imagery data for LCLU classification problem;

• performed LCLU classification problem in RS using three TL architectures (ResNet50V2, VGG19, and InceptionV3) with a bottleneck feature extraction technique on the UCM dataset,

• evaluated and improved the performance of the TL models efficiently

Methods

This study proposed the Deep TL method, which is a deep CNN technique, for efficient time-consuming. Building the model for better performance uses various parameters, such as pre-trained models, learning rate, early stopping, dropout, optimizer, loss, and activation functions.

Pre-trained models have become recent applications in RS image classification problems (Risojevic and Stojnic 2021; Rashid et al. 2020; D. Zhang, Liu, and Shi 2020; Marmanis et al. 2016; Stivaktakis, Tsagkatakis, and Tsakalides 2019). The pre-trained CNN based (Kumar, Naman, and Verma 2021) TL models used in this paper included ResNet50V2 (He et al. 2016; Shabbir et al. 2021), VGG19 (Mateen et al. 2019; Simonyan and Zisserman 2015; Xiao et al. 2020), and InceptionV3 (Szegedy et al. 2015a, 2015b). These pre-trained architectures are the deep CNN pre-trained models used to design a new TL model from the existing problem.

Learning rate (LR) was used to facilitate the TL model learning from the UCM dataset. It has various values such as 0.01, 0.001, and 0.0001. However, if the larger TR is used, the fluctuation of training and learning could happen (Naushad and Kaur 2021). Therefore, the smaller TR value is advisable to be used in building DL models. So, we used the TR of 0.0001 in this paper to optimize our model.

Reducing over-fitting in the DL method is dynamic. Dropout and early stopping are the major parameters used for reducing overfitting when training data. The percentage values for dropout expressed in decimal forms are usually recommended to use 0.2, 0.3, 0.4, and 0.5. We used 0.5 (i.e.50%) to reduce the training over-fitting since higher dropout could perform better than lower values (Stivaktakis, Tsagkatakis, and Tsakalides 2019). Early stopping is a deep CNN regularization technique used to stop the training after random epochs when the model performance could not improve (Naushad and Kaur 2021).

In DL modeling techniques, classification loss functions are widely used. This classification loss could be binary cross-entropy or multi-class cross-entropy. We preferred the multi-class entropy loss function since our class is multi-classes of the RS images.

Activation functions could be used afterward for each convolutional layer to raise the neural network capability (Yao et al. 2019). ReLu (Nair and Hinton 2010) and softmax (Mahdianpari et al. 2018) activation functions were applied in this paper because they are more advantageous than other conventional nonlinear functions, such as tanh and sigmoid functions. ReLu and softmax are better in their easily propagating errors; multiple layers of the neurons have been activated and simpler mathematical operation than that of tanh and sigmoid functions.

ReLu generates an output x if x > 0 or 0 if x < 0 as observed in (equation 1(1) $f\left(x\right)=max\left(0,x\right)=\left\{\begin{array}{c}{x}_i,ifx>0\\ 0,ifx<0\end{array}\right.$(1) and Figure 1a). This output implies that the neurons with the negative values are not activated except those with positive values. The slope of the gradient (derivative) value of ReLu is constant, i.e., ether 1 ∀x, x ≥ 0 or 0 ∀x, x < 0 (equation 2(2) $f^{\prime }\left(x\right)=\left\{\begin{array}{c}1,ifx\ge 0\\ 0,ifx<0\end{array}\right.$(2) and Figure 2b)

Figure1. ReLu function graphical representation. a) Forward propagation b) gradient descent propagation (derivative).

Figure2. Sample input images feeding into pre-processing.

.

(1) $f\left(x\right)=max\left(0,x\right)=\left\{\begin{array}{c}{x}_i,ifx>0\\ 0,ifx<0\end{array}\right.$(1)

(2) $f^{\prime }\left(x\right)=\left\{\begin{array}{c}1,ifx\ge 0\\ 0,ifx<0\end{array}\right.$(2)

Softmax (softargmax) is used to predict the class having the highest probability in multi-class classification problems for the input labels. We also used this function since our RS imagery data is a multi-class classification problem. The softmax output is between 0 and 1, and the sum of each class probability is 1.0. If some N elements of the input vector are N <0 or N > 1, they would be between (0, 1) after using the softmax function. Its equation $\mathit{f}\left({\text{b}z}_{\mathit{i},\mathit{j}}\right)$ over N classes is computed in the equation (3)(3) $s=f\left({\mathrm{z}}_{i,j}\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{z}{}_i\right)}{{\sum }_{j=1}^{N}exp\left({\mathrm{z}}_j\right)}=\frac{{\mathrm{e}}^{{\mathrm{z}}_i}}{{\sum }_{j=1}^N{\mathrm{e}}^{{\mathrm{z}}_j}}$(3) given.

(3) $s=f\left({\mathrm{z}}_{i,j}\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{z}{}_i\right)}{{\sum }_{j=1}^{N}exp\left({\mathrm{z}}_j\right)}=\frac{{\mathrm{e}}^{{\mathrm{z}}_i}}{{\sum }_{j=1}^N{\mathrm{e}}^{{\mathrm{z}}_j}}$(3)

In summary of the method, the parameters such as networks and weights were trained in the pre-trained InceptionV3, Resnet50V2, and VGG19 models. We used the bottleneck feature extraction method to extract image features from these pre-trained models. A fully connected network for pre-trained models was removed, and then a new model was built, and its weights were also removed. The bottleneck features, become the inputs for FC, are trained for UCM images (Figure 2). The bottleneck feature extracted the features of shape (1264, 6, 6, 2048) in training bottleneck prediction and shape of the features (420, 6, 6, 2048) on validation and testing bottleneck predictions extracted for each pre-trained model.

Experiments and Performance Evaluations

The experiments were performed on a laptop with the Intel Core i3-4000 M CPU 2.40 GHz 2.40 GHz RAM = 4GB and in Colaboratory with its GPU in Keras and Tensorflow software packages.

Experimental Datasets Setting

The University of California Merced (UCM) data set is used to solve the problem of LCLU classification. The UCM Land Use data set was manually collected and introduced by (Yang and Newsam 2010) from the USGS National Map Urban Area Imagery. This dataset consists of 21 land use and land cover classes containing 100 images each, measuring 256 × 256 pixels with a spatial resolution of about 30 cm/pixel. The dataset was divided into 60:20:20 ratio for training samples, validation samples, and tasting samples for each class, respectively (Table 1).

Table 1. Parameter setting for UCM dataset

Classes	Training Samples	Validation Samples	Test Samples	Total
Agricultural	60	20	20	100
Airplane	60	20	20	100
Baseball diamond	60	20	20	100
Beach	60	20	20	100
Buildings	60	20	20	100
Chaparral	60	20	20	100
Dense residential	60	20	20	100
Forest	60	20	20	100
Freeway	60	20	20	100
Golf course	60	20	20	100
Harbor	60	20	20	100
Intersection	60	20	20	100
Medium residential	60	20	20	100
Mobile home park	60	20	20	100
Overpass	60	20	20	100
Parking lot	60	20	20	100
River	60	20	20	100
Runway	60	20	20	100
Sparse residential	60	20	20	100
Storage tanks	60	20	20	100
Tennis court	60	20	20	100
Total	1260	420	420	21,000

Experimental Setting

As we discussed in section 2 earlier, various hyperparameters do have important inspirations for classification problems. So, we have used some of the important parameters in our experiment listed in Table 2. In addition to using the dropout (0.5) layer, we used the early stopping technique to reduce the over-fitting.

Table 2. Hyperparameters setting for training data

Hyperparameter	Parameter values used
Optimizer	Adam
Activation functions	ReLu and Softmax
Loss function	categorical cross entropy
Batch-size	64
Epochs	100
Learning rate	0.0001

Performance Evaluation Measurements

The model’s performance is evaluated using a confusion matrix (CM) and overall accuracy (OA). CM analyzes errors and confusion between the column with the occurrences in a predicted class and the row with the occurrences in an actual class (W. Zhang, Tang, and Zhao 2019). So, CM could also be named as an error matrix since it classifies the error classification.

The errors could be type I error and type II error (Table 3). A type I error is an outcome where the model incorrectly predicts the positive class when it is the actual negative value. In contrast, a type II error is an outcome where the model incorrectly predicts the negative class when it is the actual positive value. We combined the training and validation trained data after validating the model during the process and then evaluating the model’s performance with 20% of testing data. CM measures the performance of the TL model, whether it is classified correctly or incorrectly.

Table 3. Confusion matrix in summarized view

		Actual Values
		Positive	Negative
Predictive Values	Positive	True Positive(TP)	False Positive(FP)/Type I Error	Positive Prediction for Precision
	Negative	False Negative(FN)/Type II Error	True Negative(TN)	NegativePrediction

We used the classification metrics to calculate the model’s performance: accuracy, recall, precision, and F1 measures.

Accuracy is the measure of predictions that the model classified correctly.

(4) $\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{\#}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}.\mathrm{\#}\mathrm{o}\mathrm{f}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$(4)

Precision computes a positive predictive value, i.e., a quantity of the relevant classes among the retrieved classes. It evaluates what amount of positive identifications was actually correct.

(5) $\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{\#}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}.\mathrm{\#}\mathrm{o}\mathrm{f}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{s}}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$(5)

A recall is used to identify all actual correct relevant classes retrieved from the dataset.

(6) $\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{\#}\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{A}\mathrm{c}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}.\mathrm{\#}\mathrm{o}\mathrm{f}\mathrm{A}\mathrm{c}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s}}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$(6)

F1 score is the harmonic mean of precision and recall. Its score becomes 1 when both precision and recall are perfect and becomes 0 when either precision or recall result becomes 0. F1 score measures the preciseness and robustness of the classification model.

(7) $\mathrm{F}1\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\frac{2\left(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\right)}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}=\frac{2\left(\left(\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}\right)\ast \left(\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}\right)\right)}{\left(\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}\right)+\left(\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}\right)}=\frac{2\mathrm{T}\mathrm{P}}{2\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$(7)

TL Models Evaluation Using Measurements

There are N (N = 21) classes with an integer labeled 0 to N-1. The generated records were transformed into a confusion matrix (Figure 3). The three TL models generated the class label records for 21 classes ranging from 0 to 20 while testing each class with 20 samples. For instance, in the Inception_v3 model in Figure 3a, the first-class labeled with 0, among 20 testing samples, 18 classes are correctly classified, but the other two classes, i.e., the actual classes 3 and 18, are predicted as class 1.

Figure 3. Confusion matrix of each model on UC merced dataset. a) Inception_v3 model b) Resnet50v2 model c) VGG19 model.

Based on the confusion matrix (Figure 3a) records, the performance of TL with the Inception_v3 model has been calculated and recorded in Table 4. Similarly, the performance of TL with the resnet50v2 and VGG19 models have been measured in Tables 5 and 6 based on the confusion matrix (Figure 3b and 3c), respectively.

Table 4. Inception_v3 model for class performances in precision, recall and F1-score

Class Name	Class Label	Precision	Recall	F1-score	Support
Agricultural	0	1.00	0.90	0.95	20
Airplane	1	1.00	1.00	1.00	20
Baseball diamond	2	0.86	0.90	0.88	20
Beach	3	0.95	0.95	0.95	20
Buildings	4	0.72	0.90	0.80	20
Chaparral	5	1.00	1.00	1.00	20
Dense residential	6	0.71	0.50	0.59	20
Forest	7	0.86	0.95	0.90	20
Freeway	8	1.00	0.95	0.97	20
Golf course	9	0.75	0.45	0.56	20
Harbor	10	0.95	1.00	0.98	20
Intersection	11	0.95	0.90	0.92	20
Medium residential	12	0.64	0.80	0.71	20
Mobile home park	13	0.78	0.70	0.74	20
Overpass	14	1.00	0.95	0.97	20
Parking lot	15	1.00	1.00	1.00	20
River	16	0.70	0.95	0.81	20
Runway	17	0.87	1.00	0.93	20
Sparse residential	18	0.85	0.85	0.85	20
Storage tanks	19	1.00	0.95	0.97	20
Tennis court	20	0.83	0.75	0.79	20
Average Accuracy in each Mesures		0.88	0.87	0.87	420

Table 5. Resnet50v2 model for class performances in precision, recall, and F1-score

Class Name	Class Label	Precision	Recall	F1-score	Support
Agricultural	0	1.00	1.00	1.00	20
Airplane	1	1.00	1.00	1.00	20
Baseball diamond	2	1.00	0.95	0.97	20
Beach	3	0.83	0.95	0.88	20
Buildings	4	0.75	0.75	0.75	20
Chaparral	5	1.00	1.00	1.00	20
Dense residential	6	0.56	0.70	0.62	20
Forest	7	0.86	0.90	0.88	20
Freeway	8	0.83	1.00	0.91	20
Golf course	9	0.85	0.55	0.67	20
Harbor	10	1.00	1.00	1.00	20
Intersection	11	0.84	0.80	0.82	20
Medium residential	12	0.65	0.85	0.74	20
Mobile home park	13	0.71	0.60	0.65	20
Overpass	14	1.00	0.80	0.89	20
Parking lot	15	1.00	0.95	0.97	20
River	16	0.71	0.85	0.77	20
Runway	17	1.00	1.00	1.00	20
Sparse residential	18	0.81	0.85	0.83	20
Storage tanks	19	1.00	0.90	0.95	20
Tennis court	20	1.00	0.75	0.86	20
Average Accuracy in each Mesures		0.88	0.86	0.86	420

Table 6. VGG19 model for class performances in precision, recall, and F1-score

Class Name	Class label	Precision	Recall	F1-score	Support
Agricultural	0	1.00	1.00	1.00	20
Airplane	1	1.00	0.90	0.95	20
Baseball diamond	2	1.00	0.85	0.92	20
Beach	3	0.95	0.95	0.95	20
Buildings	4	0.94	0.85	0.89	20
Chaparral	5	1.00	1.00	1.00	20
Dense residential	6	0.52	0.85	0.64	20
Forest	7	0.86	0.90	0.88	20
Freeway	8	1.00	0.90	0.95	20
Golf course	9	0.83	0.50	0.62	20
Harbor	10	1.00	1.00	1.00	20
Intersection	11	0.82	0.90	0.86	20
Medium residential	12	0.85	0.85	0.85	20
Mobile home park	13	0.82	0.45	0.58	20
Overpass	14	0.85	0.85	0.85	20
Parking lot	15	1.00	1.00	1.00	20
River	16	0.61	0.95	0.75	20
Runway	17	0.87	1.00	0.93	20
Sparse residential	18	1.00	0.95	0.97	20
Storage tanks	19	1.00	1.00	1.00	20
Tennis court	20	1.00	0.85	0.92	20
Average Accuracy in each Mesures		0.90	0.88	0.88	420

The accuracies of the three TL models for training and validation data are shown in Figure 4

Figure4. Accuracies in Training vs. Validation for TL classification models. a) Accuracy of resnet50V2 b) Accuracy of inceptionV3 c) Accuracy in VGG19.

, and the overall accuracies are recorded in (Table 7). The accuracy of F1-Score is the preciseness and robustness of the classification model since it computes the harmonic mean of precision and recall.

Table 7. Number of early stopping at epoch #, time is taken for training and overall accuracy in the three models

Architecture	#of layers	#of layers	Early stopping at epoch#	Time (s)	OA
Resnet50V2	152	3*3	19	6	92.46
InceptionV3	22	5*5	18	9	94.36
VGG19	16	3*3	95	5	99.64

The categorical-cross-entropy loss function was used while compiling the model. For the correct class, the value of the loss function becomes closer to 0 (Figure 5).

Figure 5. Losses in Training vs. Validation for TL classification models. a) Loss in resnet50V2 b) Loss in inceptionV3 c) Loss in VGG19.

Discussions

Discussions on Results, Methods, and TL Performances

The experiments resulted in section 3 prove that the pre-trained models are applicable in LCLU classification in RS images. The accuracy is also our aim, and we got better results in each model and most individual classes. We evaluated each class’s accuracies using precision, recall, and F1-score measurements for each model. Precision is outperformed, i.e., 88%, 88%, and 90% for all the three models (Tables 4, 5 and 6). That means the relevant classes were retrieved and predicted correctly. If F1-score is perfect (1), i.e., 100% accurate for certain classes, precision, and recall are also perfect for all classes.

As seen in Tables 4 and 8 for the Inception_v3 model, the classes such as agricultural, airplane, chaparral, freeway, overpass, and parking were best-performed precision. In contrast, medium residential was performed the poorest result, i.e., 64% precision result. Airplane, chaparral, harbor, parking lot, and runway classes were best-performed recall while golf course and dense residential were performed the worst results, i.e., 45% and 50% recall result, respectively. Moreover, the classes such as airplane, chaparral, and parking lot are best accurate in f1-score while dense golf course (56%) and residential (59%) are the poorest. For similar situations, we grouped classes in their best or worst/poorest value under each Resenet50v2 and VGG19 model measurement in Table 8.

Table 8. The best and worst class accuracies in precision, recall, and F1-score measurements

Model	Precision		Recall		F1-score
	Perfect (1) class	Worst class/value	Perfect (1) class	Worst class/value	Perfect (1) class	Worst class/value
Inception_v3 (Table 4)	Agricultural, Airplane, Chaparral, Freeway, Overpass, Parking	Medium residential/ 0.64	Airplane, Chaparral, Harbor, Parking lot, Runway	Golf course/0.45 and Dense residential/0.50	Airplane, Chaparral, Parking lot	Golf course /0.56 and Dense residential /0.59
Resnet50v2 (Table 5)	Agricultural, Airplane, Baseball diamond, Chaparral, Harbor, Overpass, Parking lot, Runway, Storage tanks, and Tennis court	Dense residential /0.56 and Medium residential /0.65	Agricultural, Airplane, Chaparral, Freeway, Harbor, and Runway	Golf course /0.55	Agricultural, Airplane, Chaparral, Harbor, and Runway	Dense residential /0.62 and Golf course /0.67
VGG19 (Table 6)	Agricultural, Baseball diamond, Chaparral, Freeway, Harbor, Parking lot, Sparse residential, Storage tanks, and Tennis court	Dense residential /0.52	Agricultural, Chaparral, Harbor, Parking lot, Runway and Storage tanks	Mobile home park /0.45 and Golf course /0.50	Agricultural, Chaparral, Harbor, Parking lot, and Storage tanks	Mobile home park /0.58

The poorest precision results in the medium residential and dense residential classes and the poorest recall result in the golf course are observed in all three models. The results of the poorest accuracy could be the cause of the image variant similarity and resolution differences.

By applying the hyperparameters listed in Table 2, the TL model has been modeled using the Adam optimizer with TR of 0.0001 and compiled with the loss function of categorical-cross-entropy. The loss function predicts an integer value for each class N assigned from 0 to N-1, where N =21 classes in the UCM dataset. The cross-entropy loss has become lower and lower for the deeper network training process to identify the correct class. A correct cross-entropy value is 0 for a correct class. The value of the cross-entropy loss function increases for misclassified classes, and the trained network fails to find the correct class (Bahri et al. 2020). In Figure 5, the training loss graph with blue color is closer to 0. So, the trained network is good for predicting the correct class in TL.

In addition to dropout, we used the early stopping technique to reduce overfitting and improve performance. The training was stopped early either the performance of the validation loss stopped decreasing even though the performance of the training loss decreased, or the performance of the validation accuracy stopped increasing although the performance of the training accuracy increased. We assigned the epoch value 100, and the early stopping stopped at epoch 19, 18, and 95 randomly when validation loss stopped decreasing for Resnet50v2, InceptionV3, and VGG19, respectively. Mostly we observed that the larger early stopping resulted, the larger the accuracy, for instance, VGG19 in this study.

Therefore, VGG19 has superior TL model performance. As seen from Table 7, the VGG19 model achieved the superior accuracy of 99.64% for a 70% training ratio among all methods.

Discussions on Similar Studies

In this paper, we utilize the Resnet50V2, Inception‐V3, and VGG‐19 as our baselines. We compared the classification performance accuracy of this paper with the UCM dataset’s state-of-the-art classification studies (Table 9). According to Table 9, all the proposed TL models achieved superior accuracy among the state-of-the-arts. We used the Adaptive optimizer with the smallest LR value, i.e., 0.0001 while the others used stochastic gradient descent (SGD) with various parameters. Most of the researchers listed in Table 9 have used the epoch number 50, but we have used 100 epochs and early stopping while validating the model.

Table 9. Comparative state-of-the-art classification methods and OA (in %) on the UCM dataset

Authors	Methods	Dataset	Accuracy on UCM/our model	Optimizer
(Naushad and Kaur 2021)	Resnet50,VGG‐16	EuroSAT	99.04, 98.14	Not specified
(D. Zhang, Liu, and Shi 2020)	ResNet50,VGG16, Inception-v4	UCM, AID, NWPU	95.95,92.50,91.73	SGD
(Hung, Hui-ching, and Tseng 2020)	RSSCNet	UCM, RSSCN7, WHU-RS19	99.81,97.41,99.46	SGD
(Lima and Marfurt 2020)	Inception‐V3, VGG‐19	UCM, AID, PatternNet	91.00,94.3	SGD, Adam and Adamax
(Yao et al. 2019)	VGG-16: with multiple pyramid pooling	UCM, NWPU	93.24, 88.62	SGD
(W. Zhang, Tang, and Zhao 2019)	VGG‐16- CapsNet and Inception‐V3- CapsNet	AID,UCM, NWPU	98.81 99.02	SGD
(Yang and Newsam 2010)	bag-of-visual-words (BOVW)	UCM	81.19	No
Proposed	Resnet50V2, Inception‐V3 and VGG‐19	UCM	92.46, 94.36 99.64	Adam (Adaptive)

Therefore, the proposed VGG19 achieved the superior accuracy of 99.64% for a 70% training ratio among all methods we used. The Resnet50v2 model results in lower performance than the other two methods. The results in the three pre-trained models demonstrate that the TL model can prove its availability on RS images.

Conclusions

In this paper, we addressed the problem of LCLU classification in RS images using deep TL models with the bottleneck feature extraction. Our objective was to apply the TL model and improve the classification performance for LCLU classification in RS images. The training time of TL is more efficient (trained in seconds) than the other deep CNN models (trained in days when they trained from scratch), as observed in the state-of-the-arts study by(Mahdianpari et al. 2018; Rashid et al. 2020). We used the bottleneck feature extraction method from pre-trained models to improve the model’s training speed and accuracy.

The model’s performance is also prominent in all models, i.e., 92.46%, 94.36%, 99.64% accuracy results for Resnet50V2, Inception‐V3, and VGG‐19, respectively. However, the superior accuracy is profound in the VGG19 model with efficient time. Most of the classes’ performance is prominent accuracies except for some classes such as the medium residential, dense residential classes, and golf course, which have the poorest accuracies when evaluated by precision, recall, and F1-score.

LCLU classification in RS image contributes significant values (Alshari and Gawali 2021; Lima and Marfurt Lima and Marfurt, 2020) to decision-making and planning in rural and urban areas. Our contribution is applying the deep TL using bottleneck feature extraction for the LCLU classification problem using RS image. This contribution directs environmental resource management and sustainable development for agricultural and urban planning. In addition to this contribution, we evaluated and improved the performance of the TL models and proved their availability on the LCLU classification in RS images.

For further investigation, we will use more recent deep neural networks with various pre-trained architectures as our baseline to substantiate the effectiveness of TL pre-trained models on various datasets and parameters.

Acknowledgments

The Ethiopian Ministry of Education (MOSHE) co-op with Debre Tabor University has financially supported this research. The authors acknowledged these institutions heartily.

The authors would like to give special thanks to the reviewers, editors, and editorial assistant of the Applied Artificial Intelligence journal for their precious, pertinent, and constructive reviews and comments to increase the manuscript quality.

Disclosure Statement

No potential conflict of interest was reported by the author(s).