A decision-based multi-sensor classification system using thermal hyperspectral and visible data in urban area

ABSTRACT

Multi-sensor data fusion has become more and more popular for classification applications. The fusion of multisource remote-sensing data can provide more information about the same observed site results in a superior comprehension of the scene. In this field of study, a combination of very high-resolution data collected by a digital color camera and a new coarse resolution hyperspectral data in the long-wave infrared range for urban land-cover classification has been extensively enticed much consideration and turned into a research hot spot in image analysis and data fusion research community. In this paper, a decision-based multi-sensor classification system is proposed to completely use the advantages of both sensors to attain enhanced land-cover classification results. In this context, spectral, textural and spatial features are extracted for the proposed multilevel classification. Then, a land-cover separability preprocessing is employed to identify how the proposed method can fully utilize the sensor advantages. Next, a support vector machine is applied to classify road classes by using thermal hyperspectral image data; plants, roofs and bare soils are classified by the joint use of sensors via Dempster–Shafer classifier fusion. Finally, an object-based post-processing is employed to improve the classification results. Experiments carried out on the dataset of 2014 IEEE GRSS data fusion contest indicate the superiority of the proposed methodology for the potentialities and possibilities of the joint utilization of sensors and refine the classification outcomes when evaluated against single sensor data. Meanwhile, the obtained classification accuracy can be a competitor against the results issued by the 2014 IEEE GRSS data fusion contest.

KEYWORDS:

• Decision-level fusion
• land-cover classification
• multi-sensor fusion
• support vector machine
• thermal hyperspectral

Introduction

With recent technological advances in remote-sensing systems and the accessibility of multi-sensor information, the research community has motivated an increasing utilization of well-defined features measured by various sensors to obtain an improved classification accuracy of remotely sensed data (S. Bigdeli, Samadzadegan, & Reinartz, 2013; Li, Wu, Wan, & Zhu, 2011; Lu, Zhang, Li, & Zhang, 2015). Fusion of multi-sensor data provides complementary data from the same observed site results in a superior comprehension of the scene which is impossible with single sensor data (Bigdeli, Samadzadegan, & Reinartz, 2014; Du, Liu, Xia, & Zhao, 2013; Lu et al., 2015). In this regard, image analysis and data fusion play several roles in image pansharpening (Guo, Zhang, Li, Zhang, & Shen, 2014; Thomas, Ranchin, Wald, & Chanussot, 2008; Wald, 1999), classification (Camps-Valls, Tuia, Bruzzone, & Benediktsson, 2014; Fauvel, Tarabalka, Benediktsson, Chanussot, & Tilton, 2013; Plaza et al., 2009; Huang & Zhang, 2012a), change detection (Bruzzone & Bovolo, 2013; Huang, Zhang, & Zhu, 2014; Tian & Reinartz, 2011), large-scale processing (Blanchart, Ferecatu, Cui, & Datcu, 2014; Espinoza-Molina & Datcu, 2013), multiple resolution (Voisin, Krylov, Moser, Serpico, & Zerubia, 2014; Wemmert, Puissant, Forestier, & Gancarski, 2009), domain adaption (Bruzzone & Marconcini, 2009; Persello & Bruzzone, 2012; Tuia, Volpi, Trolliet, & Camps-Valls, 2014), interactive systems (Crawford, Tuia, & Yang, 2013; Tuia, Volpi, Copa, Kanevski, & Munoz-Mari, 2011) and signal modalities fusion with different meaning and features (Moser, Serpico, & Benediktsson, 2013). Among the mentioned fusion tasks, urban area classification is one of the most challenging issues due to land-cover kinds’ complexity and diversity of man-made objects and therefore attracts a lot of research interests (Lu et al., 2015; J. Li et al., 2015). Remotely sensed data fusion can be conducted at signal, pixel, feature and decision levels. Signal level fusion combines various signal modalities to make an enhanced signal with a superior signal-to-noise ratio (SNR). At pixel level fusion, pixel information is fused to refine image analysis potentialities. Feature level fusion integrates distinctly extracted descriptors into a concatenated feature vector to be used by a conventional classifier. At decision level fusion, numerous classifiers are fused regardless of data types and data misalignment errors (Du et al., 2013).

With recent technological advances in remote-sensing systems, fusion of very high-resolution data collected by a digital color camera and a new coarse resolution hyperspectral data in the long-wave infrared (LWIR) range for urban land-cover classification has been extensively enticed much consideration and turned into a research hot spot in image analysis and data fusion research community (Liao et al., 2015; Lu et al., 2015; Eslami & Mohammadzadeh, 2015; J. Li et al., 2015). In this context, thermal infrared hyperspectral (TIR HS) data represent extremely challenging remotely sensed data with numerous potentialities in target recognition and material classification irrespective of illumination conditions (Liao et al., 2015); these data show very high potentialities in many circumstances, e.g. for remote-sensing and non-destructive technologies which provide an exhaustive discrimination of similar ground entities (Lu et al., 2015; Wang, Wu, Nerry, Li, & Li, 2011). Nevertheless, low energy, low SNR, high inter-band correlation, spectral variation and ambiguous object boundaries are the most challenging problems which can seriously affect the classification efficiency (S. Li et al., 2011; Rodríguez-Galiano, Ghimire, Pardo-Igúzquiza, Chica-Olmo, & Congalton, 2012 ; Miliaresis, 2014; J. Li et al., 2015). On the other hand, visible image data provide detailed spatial features and clarity. However, low distinctive spectral features result in an inability to distinguish homogeneous spectral objects.

The physical background of the LWIR HS data is the basic spectral absorption features of silicate minerals, which are the most constituent of the terrestrial surface and man-made construction objects. The silicon–oxygen bonds of the silicate minerals (Si–O) cannot exhibit spectral features in the visible-to-shortwave infrared region of the spectrum, while the Si–O bonds’ stretching vibrations expand strong significant features in the LWIR spectral wavelengths. Man-made objects additionally emit a greater extent polarized infrared radiation than naturally derived background materials (i.e. tree, soil and vegetation), because they have relatively smooth surface features compared to most naturally occurring surfaces. In this context, the emissivity can parametrically suffice if surface irregularities are large compared to the emitted radiation’s wavelength. However, if surface irregularities are small compared to the emission wavelength, the surface may be more specular and an observable induced polarization occurs in the emitted thermal radiation. The basic principles can be employed for the development of spectral-based urban classification/un-mixing of man-made objects by LWIR HS data (Liao et al., 2015).

In this paper, a decision-based multi-sensor classification system is proposed to completely use the advantages of both sensors to attain enhanced land-cover classification results. In this context, spectral, textural and spatial (STS) features are extracted for the proposed multilevel classification. Then, a land-cover separability preprocessing is employed to identify how the proposed method can fully utilize the sensor advantages. Next, a support vector machine (SVM) is applied to classify road classes by using TIR HS image data; plants, roofs and bare soils are classified by the joint use of sensors via Dempster–Shafer (D–S) classifier fusion. Finally, an object-based post-processing (OBPP) is employed to improve the classification results.

The rest of the paper is organized as follows: After a literature review, the concept of the proposed decision-based multi-sensor fusion system is presented, followed by experiments, obtained results, discussion and conclusion.

Literature review

The image analysis and data fusion technical committee of the geoscience and remote-sensing society (GRSS) is an international network of scientists who are active in multi-temporal, multisource, multi-resolution and multimodal remote-sensing image analysis fields, released two airborne datasets collected at various spectral and spatial modalities with a concise temporal interval to deal with two open image analysis and data fusion research community problems comprising of handling multiple source and multiple resolution data in two parallel identical validity tracks. The classification contest concentrated on classification performance outcome at the highest spatial modalities with recent kinds of sensors, while the paper contest was regarded as new ideas of the multi-resolution data processing and analysis of the new TIR HS imagery. In this context, the classification contest’s winning manuscript was focused on maximizing the land-cover mapping’s accuracy for a particular dataset. In this study, both datasets were resampled to 0.5-m spatial resolution. A principal component analysis (PCA) was employed on TIR HS image data to reduce redundancy and computation time for image classification. Then, textural features, vegetation index and morphological building index were extracted to identify successively required classes using a binary SVM (Huang & Zhang, 2012b). Finally, the obtained pixel-based land-cover classification map was refined by majority voting (MV), adaptive mean shift segmentation and multiple semantic rule. On the other side, the paper contest’s winning manuscript was focused on the novel development for mutually taking profits from both datasets. In this respect, morphological features were extracted from the visible imagery; the visible image was utilized as a part of a guided filtering scheme to increase the LWIR image’s spatial separability in the PCA domain. Then, the extracted features and enhanced LWIR data were integrated by using a graph-based method. Finally, the feature combination was used to generate the final land-cover classification map with an SVM classifier. As the last point, the classification contest’s winner modified the primary land-cover classification map using impressive production-like points and obtained maximal classification accuracy, while the paper contest winner’s focus turned into the novel development with less emphasis on the land-cover classification map’s geometrical precision (Liao et al., 2015).

After the contest, the data remain publicly available for further experimental analysis;¹ Lu et al. (2015) proposed a synergetic decision-based classification method to estimate a thematic classification map for the mentioned datasets. First, a set of preprocessing steps was carried out on both datasets. Next, a semi-supervised local discriminant analysis was applied to identify distinguish descriptors for an SVM classifier. A combination of texture and spectral features was used for visible image classification. Finally, the outcomes of both SVM classifiers were fused to estimate the thematic classification map. The obtained results of the study confirmed an enhancement of the proposed synergic decision-based classification method against the standard classifiers or any single sensor’s classification outcome.

Eslami and Mohammadzadeh (2015) proposed an integration method to classify urban objects for the 2014 IEEE GRSS data fusion contest datasets. First, TIR HS image’s atmospheric effects were removed by the in-scene atmospheric compensation as described in (Winter, 2004). Then, the sequential parametric projection pursuit dimension reduction (DR) operator was used to achieve multispectral TIR image data at 20-cm spatial resolution. An SVM classifier was employed to classify integrated visible and multispectral TIR images. Finally, an object rule-based post-processing was applied to generate the final classification map. The obtained results of this study proved the advantages of the proposed method against the standard classifiers or any single sensor’s classification outcome.

Li et al. (2015) proposed another fusion approach for the mentioned datasets to attain enhanced urban land-cover classification map. In this study, the proposed method was composed of data preprocessing, road extraction and remaining classes classification. In the preprocessing step, TIR HS data were de-noised by using a low-rank matrix recovery. In parallel, visible data gaps were predicted by utilizing the mapping relationship between the visible and LWIR HS data in a supervised fashion. In the road extraction step, a linear SVM classifier was employed to classify road pixels by the use of TIR HS imagery. A mean shift algorithm was applied to segment the visible image; further, a MV approach was utilized to achieve object-oriented fusion results guided by the clusters of the segmentation map. Finally, the morphological dilation operation was conducted to refine extracted road pixels of the TIR HS image. In the remaining class classification step, the fine spatial resolution visible data were used to be classified using a linear SVM classifier. After the classification, the segmentation map was utilized as an object-based decision fusion step to conduct the post-classification process. The obtained results of this study demonstrated the advantages of the proposed method.

In the above-described papers, different ways were proposed to combine the 2014 IEEE GRSS data fusion contest datasets; they considered fascinating innovations in terms of development novelty or urban land-cover mapping applications. The significant challenge was the contrivance of a combined classification architecture that is a trade-off between accuracy enhancement and land-cover classification solution’s reliability, complexity reduction and processing proficiency optimization. In addition, the architecture was extremely constrained by the demand to integrate data from multiple source and at multiple resolution. The interpretation of this new dataset combination remains quite challenging and therefore is still a focus of research activities. The most important challenges that should be considered in the integration architecture are as follows (J. Li et al., 2015): (a) the TIR HS image’s low energy and low SNR can hinder the extraction of distinctive descriptors; (b) excessive inter-band correlation of TIR HS image indicates considerable descriptor redundancy and a very time-consuming image classification process (S. Li et al., 2011); (c) various descriptors for identical land-cover objects collected by LWIR HS at different times are possible (Miliaresis, 2014); (d) TIR HS ambiguous object boundaries can severely affect the classification accuracy at the highest spatial resolution (Rodríguez-Galiano et al., 2012) and (e) visible data interpretation can also be difficult due to severe interclass variations (Liao et al., 2015).

In order to tackle the abovementioned challenges, a decision-based multi-sensor classification system is proposed to completely use the advantages of both sensors to attain enhanced urban land-cover classification results.

Proposed method

The proposed methodology is a combined architecture which can be valued as a trade-off between accuracy enhancement and land-cover classification solution’s reliability, complexity reduction and processing proficiency optimization (Figure 1); preprocessing, feature extraction, multilevel SVM classification (MLSC) and OBPP are employed as shown in the following sections.

Figure 1. The proposed workflow for sensor fusion.

Preprocessing

The first step of the multilevel classification process is to analyze the land-cover spectral separability of the training dataset, as it can be used as a foundation to enhance the classification accuracy. In this context, the Jeffries–Matusita distance (JMD) and the transformed divergence (TD) indices, the most widely used discriminability evaluation indices, are estimated to identify how the proposed method can fully utilize the advantages of the 2014 IEEE GRSS data fusion contest datasets (Table 1). From Table 1, first, it can be observed that the road pixels can be easily classified by utilizing the TIR HS imagery due to the strong separability of the road and other land-cover classes. Second, plant (tree/vegetation), roof (red/gray/concrete roofs) and bare soil classes are discriminated by the fusion of both visible and TIR HS datasets. For the TIR HS image data, it can be observed that the internal plant classes (tree and vegetation) show weak separability, and similar observation can be made for the internal roof classes (red, gray and concrete roofs) as marked in Table 1; in this context, tree and vegetation pixels are separated within the plant pixels by utilizing the visible image data and the same operation is employed to classify red, gray and concrete roofs within the roof pixels. As a summary, the proposed multilevel classification framework contains the following operations: (a) road pixels’ classification by utilizing the TIR HS imagery; (b) plant, roof and bare soil pixels’ discrimination by the fusion of both visible and TIR HS imageries and (c) land-cover pixels’ separation by utilizing the visible imagery. Table 2 illustrates the procedure of utilizing the 2014 IEEE GRSS data fusion contest datasets to obtain maximum classification accuracy based on the spectral separability analysis.

Table 1. JMD/TD discriminability evaluation indices for the 2014 IEEE GRSS data fusion contest datasets.

Land cover		Road		Tree		Red roof		Gray roof		Concrete roof		Vegetation		Bare soil
		VIS	TIR	VIS	TIR	VIS	TIR	VIS	TIR	VIS	TIR	VIS	TIR	VIS	TIR
Road	JMD	0.0	0.0
	TD	0.0	0.0
Tree	JMD	2.0	2.0	0.0	0.0
	TD	2.0	2.0	0.0	0.0
Red roof	JMD	1.7	1.8	2.0	1.4	0.0	0.0
	TD	2.0	1.9	2.0	1.7	0.0	0.0
Gray roof	JMD	0.9	1.7	1.9	1.6	1.4	0.2	0.0	0.0
	TD	1.3	1.8	2.0	1.8	1.7	0.3	0.0	0.0
Concrete roof	JMD	2.0	1.9	2.0	0.9	2.0	1.0	2.0	1.0	0.0	0.0
	TD	2.0	2.0	2.0	1.1	2.0	1.1	2.0	1.1	0.0	0.0
Vegetation	JMD	2.0	2.0	1.2	0.2	2.0	1.5	2.0	1.7	2.0	1.1	0.0	0.0
	TD	2.0	2.0	1.4	0.2	2.0	1.9	2.0	1.9	2.0	1.3	0.0	0.0
Bare soil	JMD	2.0	2.0	2.0	1.4	1.4	1.9	2.0	1.9	2.0	1.4	1.9	1.3	0.0	0.0
	TD	2.0	2.0	2.0	2.0	1.6	2.0	2.0	2.0	2.0	2.0	1.9	2.0	0.0	0.0

JMD: Jeffries–Matusita distance; TD: transformed divergence.

Table 2. The procedure of utilizing the 2014 IEEE GRSS data fusion contest datasets.

Land cover			VIS	TIR
Road			-	x
Plant			x	x
Tree		Vegetation	x	-
Roof			x	x
Red	Gray	Concrete	x	-
Bare soil			x	x

Feature extraction

The next phase of the classification procedure for the multi-sensor data consists of extracting appropriate descriptors. The extracted features should comprise distinct descriptors to separate several objects (Table 3). Hyperspectral remote-sensing images consist of extremely narrow spectral bands that result in high inter-band correlation and time-consuming image analysis operations; excessive descriptors can lead to the curse of dimensionality problem, also called Hughes phenomenon, in case of using standard classifiers (S. Bigdeli et al., 2013; Li et al., 2011). In this context, DR is used to transform the data volume into a reduced dimensionality form with distinct descriptor information to overcome the mentioned phenomenon (Hasanlou, Samadzadegan, & Homayouni, 2015). In this regard, PCA, the most widely known linear technique for data volume reduction, is applied to reduce the high dimensionality of TIR HS imagery and the first five principal components (PCs) are extracted as spectral features on the TIR HS data. Furthermore, the gray value of the image and its spatial distribution in a local window can be used as spectral and textural descriptors of the visible data (Haralick & Shanmugam et al., 1973). After extraction of the above features, a multilevel classification framework is performed by considering the above-described land-cover spectral separability analysis.

Table 3. Basic mathematics of STS descriptors.

Data	Name			Formula	Description
TIR HS	Spectral features	PCA		First five PCs	G is the number of gray levels ρ(i) is the approximate probability density of occurrence of the intensity levels ρ(i,j) is the relative frequency of two pixels occur within a given neighborhood
TIR HS/VIS	Spectral features	Vegetation index
VIS	Textural features	Central moments	Mean
			Variance
			Skewness
			Kurtosis
		Co-occurrence matrix	Mean
			Variance
			Homogeneity
			Contrast
			Dissimilarity
			Entropy
			Second moment
			Correlation

TIR HS: thermal infrared hyperspectral; VIS: visible; PCA: principal component analysis.

Multi-level SVM classification

In the proposed multilevel classification strategy, a progressive multiple procedure classification model is applied on the extracted features in order to be a trade-off between accuracy enhancement and land-cover classification reliability, complexity reduction and processing proficiency optimization; a progressive process composed of multiple step as described in the following sections.

In the first step, the road pixels are classified by using the first five PCs due to the strong separability of the road and other land-cover classes as described in the preprocessing step. Among the various supervised classifiers, an SVM is an encouraging and well-documented methodology because of its simple utilization, supreme efficiency and ability to handle different issues (Abe, 2010; Lu et al., 2015). An SVM maximizes the predefined classes’ discrepancy using optimal separating hyperplane estimation. A linear decision function is applied to transform nonlinearly separable data into a higher dimensional space using SVM kernel functions. In this paper, the most well-known SVM kernel function, radial basis function, is used to estimate inner products among any sample pairs in the feature space. As the SVM parameters, composed of regularization parameter (C), defines a trade-off between the training error and the model complexity minimization, and the kernel bandwidth parameters (γ), have strong effects on its classification efficiency, grid search is employed to automatically estimate the optimum regularization and kernel parameters (Chang & Lin, 2011).

In the next step, plants, roofs and bare soils are discriminated by the fusion of both data source information. In this paper, a multi-SVM system is applied to classify each data source features and then a multi-classifier system is taken on the decision level to integrate the SVM classification results. Decision level fusion is commonly determined as the procedure of combining single source data results in an enhanced classifier in comparison with any single classifiers that create the ensemble (Kuncheva, 2004). In this context, the proposed multiple SVM system uses one SVM classifier for each data source features which is adjusted according to the corresponding data information, while standard methods employ one SVM classifier for the series combination of the whole data source features that cannot be adjusted to the entire data. Among the various decision level fusion techniques, D–S theory can demonstrate and combine uncertain data, as it may integrate objective evidence for a hypothesis by the framework’s expectation of the significance of that evidence to the hypothesis (Lu et al., 2015). The following steps are carried out to illustrate how the D–S algorithm combines c individual classifiers (Kuncheva, 2004; Rogova, 1994):

• The “proximity” Φ is calculated between DTⁱ and the classifier’s output D_i for the input x as

(1)

DTⁱ is the ith decision template’s row DT_j.

D_i(x) is the ith decision profile’s row DP(x).

• The following belief degrees are determined for each class, j = 1,…,c and for each classifier, i = 1,…,L as

(2)

• The final degree of support is estimated as

(3)

K is a normalizing constant.

In the final step, land-cover classes are separated by using the visible data source features as described in the preprocessing step. In this context, an SVM classifier is applied to classify tree versus vegetation from the extracted plant pixels. The same procedure is used to classify red, gray and concrete roofs from the extracted roof pixels. Meanwhile, grid search is employed before the mentioned procedure to automatically estimate the optimum regularization and kernel SVM parameters.

Object-based post-processing

There can be several outlier pixels (wrongly classified pixels) and a spatial coherency deficiency caused by the problem of excessive heterogeneity in the standard pixel-based classification techniques. Image segmentation is used to eliminate the salt and pepper noise or points commonly created by “the same object but different spectrum” or “the same spectrum but different objects”. It makes utilization of the mentioned STS descriptors to split an image into spatially uninterrupted disjoint and un-overlapping alike zones (Lu et al., 2015). In this paper, multi-resolution segmentation technique is applied to segment the data into regions. The multi-resolution segmentation algorithm starts with one pixel as single image objects and frequently combines an image objects pair into larger single entities. The combination decision is based on a local homogeneity criterion, defining the likeness among contiguous image objects (Baatz and Schäpe 2000). After performing segmentation, a MV should be employed on each of the segmented regions to make the final label outputs decision (Kuncheva, 2004).

In the final step, the spatial relationship between classes is investigated and the following semantic rules are employed to refine the land-cover classification results (Table 4).

Table 4. Classification map refinement (CMR) semantic rules.

Land Cover	Rule
Road	If relative border of Road class = 1 then merge it as Road class
	For Road class, if length/width ≤ Tr and existence of Road class = 0, then merge it with the neighboring object that shares the most of its boundary
Plant	For Tree object, if length/width ≥ Tr, then merge it as Vegetation class
	For Vegetation object, if elliptic fit ≥ Tr, then merge it as Tree class
Roof	If length/width ≤ Tr and relative border of Roof class = 1, then merge it as Roof class
	For Roof object, if length/width ≥ Tr and existence of Road class = 1, then merge it as Road class
	For Roof object, if size ≤ Tr and size ≥ Tr, then merge it with the neighboring object that shares the most of its boundary

Experiments and results

The proposed decision-based multi-sensor fusion system is evaluated using the released 2014 IEEE GRSS data fusion contest datasets comprising two airborne datasets collected at two spectral and spatial modalities with a concise temporal interval: (a) TIR HS imagery with approximately 1-m resolution; (b) visual data with approximately 0.1-m resolution that was spatially down-sampled into 0.2-m resolution to optimize multiple resolution ratio (Figure 2). The contest datasets were acquired by two different fixed-wing aircrafts at approximately 800-m flight height on 21 May 2013 by Telops Inc., Québec, Canada.² Furthermore, the flights were performed over an urban area at a short distance away from Thetford Mines in Québec, Canada, containing commercial and residential structures, gardens and roads.

Figure 2. The 2014 IEEE GRSS data fusion contest dataset: first row illustrates the subset of the data prepared for training: (a) LWIR image data (1-m resolution); (b) visible image data (20-cm resolution); (c) training labels. The second row denotes the data published for testing: (d) LWIR image data; (e) visible image data and (f) ground truth.

The TIR HS image was collected by utilizing the recent airborne LWIR HS imager “Hyper-Cam” which is a Fourier transform spectrometer (FTS) comprising 84 spectral narrow bands in the 7.8–11.5-µm wavelength range. The visible data were composed of un-calibrated digital data at the highest spatial resolution with sparse ground coverage over the same area as the LWIR HS imagery. The visible data were georeferenced to be aligned into the LWIR HS image’s coordinate system (Figure 2).

The provided multi-sensor dataset presents some challenging problems which can seriously affect the classification efficiency, including (a) low energy, low SNR, high inter-band correlation, spectral variation and ambiguous object boundaries in TIR HS imagery; (b) spatial gaps in visible data.

• As shown in Figure 2(c), the training samples represent obvious spatial correlation which reveal significant spatial redundancy causing over-fitting issues.

• As can be seen in Figure 2(d), the ambiguous boundaries of land objects can affect thermal image interpretation at a fine spatial resolution.

• The existing spatial gaps between the sequentially acquired visible data strips can decrease the spatial descriptors efficiency (Figure 2(e)).

  Furthermore, there are several outlier pixels and a spatial coherency deficiency caused by the problem of excessive heterogeneity due to “the same object but different spectrum” or “the same spectrum but different objects”.

• Figure 3(a) shows thermal mean spectra of each class where the horizontal axis indicates the number of thermal bands and the vertical axis represents the radiant energy. It can be seen that the maximum vertical value is less than 0.12, which illustrates extremely bounded radiant energy.

  The excessive inter-band correlation indicates considerable descriptor redundancy in the thermal data.

• As shown in Figure 3(b), there is a radiant energy discrepancy across the flight direction between the sequentially acquired thermal data strips due to environmental changes, while the visible data remain relatively stable as shown in Figure 2(e).

In the first step of the presented methodology, JMD/TD indices were estimated using Envi software to analyze the land-cover spectral separability of the training dataset (Table 1). Following that, STS features (Table 3) were employed on both data to determine the feature space (Figure 4).

Figure 3. The 2014 IEEE GRSS data fusion contest dataset: (a) thermal mean spectrum; (b) the 80th band of thermal data and (c) the 82th band of thermal data.

Figure 4. STS features on the 2014 IEEE GRSS data fusion contest dataset: (a) PCA-LWIR; (b) VI; (c) variance and (d) homogeneity.

After determination of abovementioned feature space, the proposed progressive processing model was used by considering the land-cover spectral separability analysis. Then, the multi-resolution segmentation algorithm was performed using eCognition software with the values 25, 0.3 and 0.5 for the scale, compactness and shape parameters. After that, MV was employed on each of the segmented regions to make the final label outputs decision. Finally, the spatial relationship among land covers (Table 4) was considered to refine the classification results. In this section, the efficiency of the proposed method is assessed via a single experiment. The following objectives of the methodology are considered:

• the effectiveness of STS features to enhance the classification accuracy,

• the effectiveness of the multilevel classification method to utilize the pros of multi-sensor data,

• the effectiveness of the knowledge-based system to tackle the common challenges of traditional pixel-based classification methods, and

• the comparison of the obtained results with the methods evaluated in the 2014 IEEE GRSS data fusion contest.

First, the effect of the extracted visible STS features is investigated through the visible data classification results (Table 5 and Figure 5). It can be seen that the classification performance coefficients are still identical (OA/kappa: 0.82/0.75) while the accuracies of most classes have been increased effectively; STS features can obtain much better accuracies of the plant (tree/vegetation: 0.85/0.82) and gray roof (0.75) classes via the generated distinguishing descriptors. Table 6 presents the confusion matrix of the experiment using the extracted visible STS features to enhance land-cover classification.

Table 5. SVM classification accuracy.

Desc.			C	γ	OA	Kappa	Road	Tree	Red roof	Gray roof	Concrete roof	Vegetation	Bare soil
Standardmethod	RGB		64	8	0.82	0.75	0.82	0.75	0.93	0.70	0.93	0.79	0.93
	STS		64	4	0.82	0.75	0.79	0.85	0.93	0.75	0.94	0.82	0.93
	LWIR		64	1	0.64	0.46	0.88	0.01	0.34	0.55	0.39	0.35	0.21
	PCA-LWIR		64	8	0.69	0.53	0.94	0.00	0.46	0.56	0.42	0.37	0.21
	D–S fusion		–		0.87	0.81	0.86	0.85	0.94	0.85	0.94	0.79	0.94
Proposedmethod	MLSC		–		0.91	0.87	0.93	0.85	0.94	0.90	0.92	0.78	0.94
	OBPP	MV	–		0.93	0.89	0.94	0.87	0.97	0.94	0.93	0.81	0.97
		CMR	–		0.96	0.93	0.98	0.88	0.98	0.93	0.93	0.86	0.98
Train							112,457	27,700	46,578	53,520	97,826	185,329	44,738
Test							809,098	100,749	136,697	142,868	109,539	103,583	49,212

RGB: red, green and blue; STS: spectral, textural and spatial; LWIR: long-wave infrared; PCA: principal component analysis; D-S: Dempster–Shafer; MLSC: multilevel support vector machine classification; OBPP: object-based post-processing; MV: majority voting; CMR: classification map refinement.

Table 6. Confusion matrix of the STS-based classification – visible.

	Road	Tree	Red roof	Gray roof	Concrete roof	Vegetation	Bare soil
Road	641,547	0	111	62,569	95,258	253	9360
Tree	36	85,199	0	85	0	15,409	20
Red roof	2109	533	127,363	605	215	485	5387
Gray roof	25,836	313	1166	107,044	6547	295	1667
Concrete roof	1632	0	24	4554	103,285	14	30
Vegetation	131	18,818	23	104	0	84,438	69
Bare soil	379	0	2539	0	54	676	45,564

Figure 5. SVM classification accuracy.

After this step, the effect of the extracted LWIR STS features is investigated by the TIR HS data classification results (Table 5 and Figure 5). It can be summarized that the PCA-LWIR classification method improves the classification coefficients (OA/kappa: 0.69/0.53) via inter-band correlation reduction. Also, it can be observed that the accuracies of all classes have been increased effectively. Table 7 displays the confusion matrix of the test by utilizing the PCA-LWIR features.

Table 7. Confusion matrix of the STS-based classification – PCA-LWIR.

	Road	Tree	Red roof	Gray roof	Concrete roof	Vegetation	Bare soil
Road	762,018	0	6014	11,427	15,401	14,049	189
Tree	1991	0	1745	10,196	33,648	53,091	78
Red roof	4750	0	62,579	57,553	7243	4572	0
Gray roof	5129	0	45,344	79,768	6240	6387	0
Concrete roof	5912	0	18,698	22,950	46,297	13,419	2263
Vegetation	5766	9	1016	7968	50,790	37,956	78
Bare soil	574	0	9	684	14,137	23,361	10,447

Second, the effect of the MLSC method is investigated through the D–S combining STS-based classification results (Table 5 and Figure 5). From the classification accuracy viewpoint, the two strategies resulted in satisfactory accuracies when compared with individual classification results. In more detail, the overall results clearly demonstrate that the proposed progressive processing model outperforms the STS-based data fusion method in terms of classification accuracy; the MLSC strategy represents the best classification performance coefficients (OA/kappa: 0.91/0.87) that was caused by accuracy improvement of D–S fusion strategy with up to 4% for OA and 6% for kappa coefficient. The proposed MLSC method can improve the classification accuracies of the road (0.93) and gray roof (0.90) classes by considering the land-cover spectral separability analysis (Table 1). Table 8 displays the confusion matrix of the experiment using the MLSC method.

Table 8. Confusion matrix of the MLSC method.

	Road	Tree	Red roof	Gray roof	Concrete roof	Vegetation	Bare soil
Road	752,173	0	128	47,836	8257	26	678
Tree	1353	85,232	17	151	0	13,994	2
Red roof	3009	142	128,422	1391	298	96	3339
Gray roof	3536	9	4479	128,553	4574	187	1530
Concrete roof	3111	1	41	5865	100,383	30	108
Vegetation	4954	17,150	174	65	0	81,198	42
Bare soil	501	27	1677	36	72	693	46,206

Third, the effect of the knowledge-based system is perused to investigate how the proposed strategy tackles the common challenges of traditional pixel-based classification methods (Table 5 and Figure 5). MV strategy leads to an even better classification accuracy (OA/kappa: 0.93/0.89) that represents an accuracy improvement of the MLSC strategy by up to 2% for the classification performance coefficients. Furthermore, it can be seen that the accuracies of all classes have been enhanced effectively via MV on multi-resolution segmented regions. As a consequence, the semantic rules can improve the classification accuracy (OA/kappa: 0.96/0.93) which still enhance MV strategy by up to 4% for the kappa coefficient. Hereby, the accuracies of all classes have been enhanced effectively via considering the spatial relationship between extracted classes. Table 9 demonstrates the confusion matrix of the experiment using the OBPP.

Table 9. Confusion matrix of the OBPP method.

	Road	Tree	Red roof	Gray roof	Concrete roof	Vegetation	Bare soil
Road	794,015	0	0	14,653	318	18	94
Tree	1240	88,278	5	198	8	11,020	0
Red roof	1142	146	133,290	2110	0	9	0
Gray roof	3521	9	4342	133,209	1786	1	0
Concrete roof	2469	2	1	4771	102,296	0	0
Vegetation	4232	10,238	19	102	0	88,992	0
Bare soil	688	0	0	0	0	122	48,402

The obtained results confirm that the proposed decision-based multi-sensor classification system exhibits a superior performance compared to the conventional classification methods or any individual classification result. Figure 6 illustrates the obtained classification maps of the 2014 IEEE GRSS data fusion contest datasets.

Figure 6. SVM classification maps: (a) STS; (b) PCA-LWIR; (c) D–S fusion; (d) MLSC – roads; (e) MLSC – plants; (f) MLSC – roofs; (g) MLSC – bare soils; (h) MLSC; (i) MV and (j) CMR.

Fourth, the proposed classification method is in a higher rank place than the majority of the participating teams regarding the comparison of the obtained results with the top 10 papers presented in the 2014 IEEE GRSS data fusion contest. This comparison is performed under the same condition, i.e. same training and testing datasets prepared by Telops Inc. (Figure 7).

Figure 7. Kappa comparison of the proposed method with the top 10 techniques evaluated in the 2014 IEEE GRSS data fusion contest.

Conclusion

This paper represents a decision-based multi-sensor classification system for LWIR HS and visible images to produce a classification map at the spatial resolution of the visible data. In the proposed method, a combination architecture is used to be a trade-off between accuracy enhancement and land-cover classification solution’s reliability, complexity reduction and processing proficiency optimization. In this context, STS features are extracted for the proposed multilevel classification. Then, a land-cover separability preprocessing is employed to identify how the proposed method can fully utilize the advantages of both sensors. Next, an SVM is applied to classify road classes by using thermal hyperspectral image data; plants, roofs and bare soils are classified by the joint use of both sensors via D–S classifier fusion. Finally, an OBPP is employed to improve the classification results. The proposed method is evaluated with respect to the potentiality of STS features to enhance the classification accuracy, the effectiveness of the multilevel classification methodology to utilize the pros of multi-sensor data, the effectiveness of the knowledge-based system to tackle the common challenges of traditional pixel-based classification methods and the comparison of the obtained results to the methods presented in the 2014 IEEE GRSS data fusion contest. As a conclusion, the decision-based multi-sensor fusion system yields a higher classification performance coefficient against single source images and is indicated to be an encouraging method against the top 10 techniques evaluated in the 2014 IEEE GRSS data fusion contest. Furthermore, the land-cover classification map shows a superior objective result and turns out to be more reliable toward human perception. Future studies will focus on the context-aware decision level fusion.

Acknowledgments

The authors would like to thank Telops Inc. (Québec, Canada) for acquiring and providing the data used in this study, the IEEE GRSS Image Analysis and Data Fusion Technical Committee and Dr. M. Shimoni (Signal and Image Centre, Royal Military Academy, Belgium) for organizing the 2014 Data Fusion Contest, the Centre de Recherche Public Gabriel Lippmann (CRPGL, Luxembourg) and Dr. M. Schlerf (CRPGL) for their contribution of the Hyper-Cam LWIR sensor and Dr. M. De Martino (University of Genoa, Italy) for her contribution in data preparation.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ Available: http://cucciolo.dibe.unige.it/IPRS/IEEE_GRSS_IADFTC_2014_Data_Fusion_Contest.htm.

² Available: http://www.telops.com.