Evaluation of different machine learning methods for land cover mapping of a Mediterranean area using multi-seasonal Landsat images and Digital Terrain Models

Figures & data

Figure 1. Location of the study area.

Table 1. Land cover classification scheme.

Class ID	Class name
1	Urban
2	Poplar grove
3	Conifers
4	Greenhouses
5	Shrublands
6	Olive grove
7	Grasslands
8	Oak grove
9	Herbaceous irrigated
10	Ligneous irrigated
11	Herbaceous dry
12	Bare soils
13	Tropical crops
14	Water

Table 2. Summary of the overall and per categories mapping accuracy obtained by the different classification methods.

	CT			RF			ANN			SVM
	OA (0.86)		K (0.85)	OA (0.92)		K (0.92)	OA (0.91)		K (0.90)	OA (0.92)		K (0.91)
Class Id	PA	UA	K	PA	UA	K	PA	UA	K	PA	UA	K
1	0.74	0.88	0.72	0.88	0.90	0.87	0.86	0.84	0.85	0.94	0.84	0.93
2	0.98	0.98	0.98	1.00	1.00	1.00	0.98	1.00	0.98	1.00	1.00	1.00
3	0.72	0.80	0.70	0.92	0.88	0.91	0.88	0.92	0.87	0.90	0.90	0.89
4	0.96	1.00	0.96	1.00	1.00	1.00	0.96	1.00	0.96	1.00	1.00	1.00
5	0.86	0.66	0.85	0.80	0.83	0.79	0.88	0.86	0.87	0.88	0.81	0.87
6	0.84	0.93	0.83	0.98	0.91	0.98	0.94	0.85	0.93	0.92	0.90	0.91
7	0.80	0.95	0.79	0.84	1.00	0.83	0.86	0.93	0.85	0.86	1.00	0.85
8	0.72	0.67	0.70	0.86	0.84	0.85	0.92	0.84	0.91	0.88	0.86	0.87
9	0.92	0.96	0.91	1.00	0.98	1.00	1	0.91	1.00	0.92	0.96	0.91
10	0.86	0.90	0.85	0.86	0.98	0.85	0.9	0.96	0.89	0.90	0.98	0.89
11	0.94	0.84	0.93	1.00	0.89	1.00	0.98	0.91	0.98	1.00	0.91	1.00
12	0.74	0.66	0.72	0.82	0.84	0.81	0.66	0.77	0.64	0.68	0.83	0.66
13	0.94	0.90	0.94	0.98	0.91	0.98	0.9	0.94	0.89	0.96	0.87	0.96
14	0.98	1.00	0.98	1.00	1.00	1.00	1	1.00	1.00	1.00	1.00	1.00

OA, overall accuracy; K, kappa; PA, producer's accuracy; UA, user's accuracy.

Table 3. Results of the evaluation of the statistical significance (Z) of the differences in kappa coefficients of the thematic maps classified by the different machine learning algorithms.

	CT	RF	ANN	SVM Lin.	SVM Pol.	SVM Rad.
RF	4.96	–	–	–	–	–
ANN	3.63	1.48	–	–	–	–
SVM Lin.	3.62	1.73	0.14	–	–	–
SVM Pol.	3.75	1.6	0	0.26	–	–
SVM Rad.	4.4	0.93	0.7	1.3	1.13	–
SVM Sig.	0.48	4.61	3.64	4.23	4.63	4.65

Figure 2. Effect of adding noise in training data on the mapping accuracy. SVM is less noise sensitive than the rest of classifiers, especially for noise proportions over 50%.

Table 4. Z-score values obtained for data classified from training data with different noise proportions with respect to the original results.

% Noise	5	10	15	20	25	30	35	40	45
CT	2.36	2.29	2.32	3.20	4.90	4.24	5.61	6.49	6.73
RF	0.33	0.28	1.15	1.98	2.69	2.78	3.54	3.68	4.30
ANN	1.84	2.01	3.50	4.39	5.18	5.84	7.13	7.54	9.17
SVM (RBF)	0.00	1.22	1.35	1.10	2.20	3.04	4.13	3.83	4.18
% Noise	50	55	60	65	70	75	80	85	90
CT	6.86	8.99	7.49	9.35	10.52	11.99	18.45	20.09	22.07
RF	5.67	8.64	9.10	11.46	13.38	16.04	18.39	21.83	23.37
ANN	8.59	10.94	12.42	13.03	14.63	17.36	18.63	19.65	22.41
SVM (RBF)	5.17	6.20	7.46	14.63	9.34	10.61	12.92	18.29	22.90

Figure 3. Effect of reducing training data on the mapping accuracy. RF and SVM show a similar behaviour with relation to the reduction of training data. However, the ANN and CT underwent a more noticeable decrease of mapping accuracy, especially for high reduction values. This may mean a higher need for training data of these algorithms.

Table 5. Z-score values obtained for data classified from reduced-size training datasets with respect to the original results.

% Reduction	5	10	15	20	25	30	35	40	45
CT	0.22	0.56	1.95	1.17	2.10	2.71	1.86	1.48	1.69
RF	0.00	1.39	0.00	0.45	0.77	0.58	0.69	1.46	1.62
ANN	0.28	0.46	1.24	0.62	1.18	1.02	1.73	2.03	2.03
SVM (RBF)	0.26	0.00	0.22	0.76	0.77	0.89	0.60	0.78	0.38
% Reduction	50	55	60	65	70	75	80	85	90
CT	1.35	3.11	3.21	3.94	4.31	5.96	6.98	7.54	7.03
RF	2.40	3.09	1.86	2.87	3.16	4.44	4.64	5.66	6.48
ANN	2.32	2.42	3.97	4.13	3.49	5.80	7.69	8.40	9.51
SVM (RBF)	0.35	1.62	0.85	0.93	1.57	1.90	2.99	4.45	5.97