A comparative data-fusion analysis of multi-sensor satellite images

Abstract

Remote-sensing data play an important role in extracting information with the help of various sensors having different spectral, spatial and temporal resolutions. Therefore, data fusion, which merges images of different spatial and spectral resolutions, plays an important role in information extraction. This research investigates quality-assessment methods of multisensor (synthetic aperture radar [SAR] and optical) data fusion. In the analysis, three SAR data-sets from different sensors (RADARSAT-1, ALOS-PALSAR and ENVISAT-ASAR) and optical data from SPOT-2 were used. Although the PALSAR and the RADARSAT-1 images have the same resolutions and polarisations, images are gathered in different frequencies (L and C bands, respectively). The ASAR sensor also has C-band radar, but with lower (25 m) resolution. Since the frequency is a key factor for penetration depth, it is thought that the use of different SAR data might give interesting results as an output. This study describes a comparative study of multisensor fusion methods, namely the intensity-hue-saturation, Ehlers, and Brovey techniques, by using different statistical analysis techniques, namely the bias of mean, correlation coefficient, standard deviation difference and universal image quality index methods. The results reveal that Ehlers' method is superior to the others in terms of spectral and statistical fidelity.

Keywords:

• image fusion
• SAR
• multisensor
• image processing
• accuracy

1. Introduction

Multisensor satellite data fusion is used to increase the quantitative analysis (i.e. to facilitate image interpretation) due to complementary information of different spectral and spatial multisensor data characteristics (Cetin and Musaoglu 2009; Chitade and Katiyar 2012; Han, Li, and Gu 2008; Jalan and Sokhi 2012; Kumar, Mukhopadhyay, and Ramachandra 2009; Kumar et al. 2011; Li and Wang 2001; Tsai 2004, Yuhendra et al. 2012; Zhang 1999; Zhou, Civco, and Silander 1998). Today, there are many algorithms for image fusion, and integration of multisensor data is of the essence for many applications.

In recent years, the launch of new synthetic aperture radar (SAR) satellites such as ALOS, TerraSAR-X, RADARSAT-2 and COSMO-SkyMed have opened a new era for remote-sensing applications. Previous studies proved that the combination of optical and microwave data provides more accurate identification when compared with the results obtained with individual sensors (Aschbacher and Lichtenegger 1990; Rahman, Sumantyo, and Sadek 2010). Because the response of radar is more a function of geometrical properties and the structure of the surface, the use of multiple types of sensors for image fusion increases the information content of images in comparison with surface reflection as in optical images. The following definition explains the data-fusion framework in remote sensing rather than the tools and algorithms used in fusion processes. ‘Data fusion is a formal framework in which the means and tools for the alliance of data originating from different sources are expressed. It aims at obtaining information of greater quality; the exact definition of “greater quality” will depend upon the application’ (Wald 1999). In general, the image-fusion techniques can be classified into three levels: pixel (iconic) fusion, feature-level (symbolic) fusion and knowledge/decision-level fusion (Pohl and van Genderen 1998). Among them, pixel-based fusion has the best potential for keeping the original information of the input images in the merged output data.

Among the various image-fusion algorithms in the literature, intensity-hue-saturation (IHS) transformation is one of the most widely used methods for merging complementary multisensor data-sets. Although the IHS technique originally uses only three band images in the fusion process, it has been used for many applications and upgraded by the Ehlers method (Balik-Sanli, Kurucu, and Esetlili 2008; Ehlers 2005; Sunar and Musaoglu 1998). Other common techniques are the Brovey transformation, which is an arithmetic combination method (Binh et al. 2006), and the principal-component analysis (PCA) method, which has the ability to reduce the dimensionality of the original data from n (number of input bands) to two or three principal components containing the majority of information (Amarsaikhan and Douglas 2004; Pohl and van Genderen 1998). The description and use of various methods and tools can be found in Aiazzi, Baronti, Alparone et al. (2006), Aiazzi, Baronti, Selva et al. (2006), Bethune, Muller, and Donnay (1998), Jin, Ruliang, and Ruohong (2006), Liu (2000) and Shi et al. (2005).

Several studies compare fusion techniques such as IHS, PCA, discrete-wavelet-transformation (DWT) and Brovey to achieve the best spectral and spatial quality (Colditz et al. 2006; Shi et al. 2005; Zhang and Hong 2005; Zhou, Civco, and Silander 1998). Generally, image parameters such as mean value (i.e. average intensity of an image), standard deviation, entropy information, and profile-intensity curves are used for assessing the quality of the fused images (Zhang 1999; Zhang et al. 2005). In addition to these techniques, bias of mean (BM) and correlation-coefficient (CC) analyses are other statistical tools used for measuring the distortion between the original image and the fused image in terms of spectral information. Still other well-known measurements to assess the quality of the fusion process are the estimation of the universal-image-quality index (UIQI) (Wang 2002) and the relative dimensionless global error in synthesis (ERGAS, Erreur Relative Globale Adimensionnelle de Synthèse) (Wald 2002).

The contribution of SAR images to optical images is also investigated in some research studies that use IHS image fusion (Kurucu et al. 2009) and DWT fusion (Huang et al. 2005). In these studies, a comparative analysis was described using L-band JERS SAR and Landsat TM data with different fusion techniques such as IHS, DWT, PCA and high-pass-philtre (HPF), and the mean, standard deviation, correlation coefficient and entropy factors were computed to calculate the quality of the fused images (Rokhmatuloh et al. 2003; Shi et al. 2005). Pal, Majumdar, and Bhattacharya (2007) fused four bands of IRS-1C with ERS-2 using PCA techniques to enhance the detection of geological information.

In this study, a horizontal-horizontal (HH) polarised L band PALSAR, an HH polarised C band RADARSAT-1, five vertical-vertical polarised ENVISAT-ASAR images, and a SPOT-2 XS were used for image fusion. The quality-assessment analyses were used to assess the fusion results of each red-green-blue (RGB) image composition, and the results were compared both visually and statistically. Based on the literature review, BM, CC, standard deviation difference (SDD) and UIQI statistical analyses were performed for all fused images (Binh et al. 2006; Colditz et al. 2006; Rokhmatuloh et al. 2003; Shi et al. 2005; Wald 2002).

2. Study area and data used

In this research, the Menemen plain in the western part of the Gediz Basin in the Aegean region of Turkey was selected as study area. Its central geographical coordinates are at longitude 27° 04′ east and latitude 38° 36′ north. The Aegean Sea lies to the west of the study area, and Manisa Province lies to the north (Figure 1). The study area is about 400 km², which includes both residential and agricultural areas, with a smooth micro relief. In the SAR images, the surface roughness affects geometry and backscatter values. To eliminate the distortions caused by the relief, the area having micro relief is preferred as in many SAR applications. Hence, an area having a smooth micro relief with an approximate slope of 1–2% (i.e. flat), as well as linear features such as field borders, channels, etc., was chosen as the study area.

Figure 1. Map of study area.

In a large part of the study area, where the fields were prepared for cotton and corn farming, the actual planting (seeding) started at the beginning of May 2006 – i.e. the study area was unplanted except for winter crops, which are wheat and barley. Hence, the soil in the area was compacted to prevent moisture loss. The surface roughness of the study area is homogeneous. In this season, since the area had not received enough rain until the beginning of May, the soil moisture levels varied according to its water-holding capacity. In the view of the study area (from the false-colour composite image derived from SPOT-2), red colours imply land cover of wheat and barley, green colours imply ploughed soil surfaces with different moisture contents and reddish black colours imply swampy areas with natural plantation (Figure 1).

In the analysis, three SAR data-sets from different SAR sensors, namely RADARSAT-1, PALSAR and ASAR were used. Table 1 summarises the main characteristics of the data in detail.

Table 1. Specifications of the data used.

	RADARSAT–1	PALSAR	ENVISAT	ENVISAT	ENVISAT	ENVISAT	ENVISAT	SPOT-XS
Date	28.05.2006	10.06.2006	23.05.2006	08.06.2006	11.06.2006	01.08.2006	17.08.2006	14.05.2006
Sensor	SAR Fine 1	PALSAR/FBS	ASAR/IMP	ASAR/IM	ASAR/IM	ASAR/IM	ASAR/IM	HRV/HRG
Pixel spacing	6.25 m	6.25 m	12.5 m	12.5 m	12.5 m	12.5 m	12.5 m	20 m
Flight direction	Ascending	Ascending	Ascending	Ascending	Ascending	Ascending	Ascending
Polarisation	H/H	H/H	V/V	V/V	V/V	V/V	V/V
Swath	50 km	80 km	IS2-105	IS1-105	IS3-82	IS2-105	IS1-105	60 km
Incidence angle	37–40	41.5	19.2–26.7	15–22.9	26–31.4	19.2–26.7	15–22.9	L29.6
Azimuth×range	1×1	2×1	4×1 looks	4×1 looks	4×1 looks	4×1 looks	4×1 looks
Size	9319 P×8026 L	12837 P×12426 L	8450 P×8167 L	8539 P×8186 L	6662 P×8152 L	8445 P×8167 L	8535 P×8188 L	4666 P×3811 L

3. Methodology

3.1. Pre-processing

As a first step, the SAR images were filtered to reduce the inherent speckle noise. For this purpose, various filters such as mean, median, Frost, Lee, and Gamma-Map were tested with different window sizes and over two passes on the data-sets. According to the visual interpretations, the best results were obtained by two passes using median and mean filters. The first filtering was applied using a 3×3 median filter, and the second iteration used a 5×5 mean filter.

Since the image fusion of different data-sets will be conducted at the pixel level, spatial registration accuracies should be in the sub-pixel range to avoid the combination of unrelated data. Therefore, in fusion applications, geometric correction is very important for registration of the images. First, the SPOT XS image was rectified using cadastral maps at 1/5000 scale. However, the map includes only the cadastral details of the area such as field borders. The study area covers agricultural fields, and it was difficult to find good ground control points (GCPs) in the area covered by a 1/5000-scaled cadastral map sheet. Therefore, topographic maps at 1/25,000 scale were also used as an additional data source to collect GCPs such as road intersections, water channels, bridges, and so on from the study area and its surroundings. Second, the filtered SAR images were registered to the rectified SPOT image using an image-to-image registration method with a root mean square error of less than one pixel. Images were registered using first-order polynomial transformation, and they were re-sampled using bilinear interpolation.

The RADARSAT and ALOS PALSAR images were re-sampled to 8 m, and the ENVISAT ASAR images were re-sampled to 25 m using the nearest-neighbour algorithm in order to retain the multi-temporal information content of the radar data. The nominal resolution of a RADARSAT fine-beam image is 8 m (Toutin 2001). In Toutin (2001), the pixel spacing of RADARSAT-1 fine-beam modes F1 and F5 is 6.25×6.25 m, whereas ground resolution is 9.1×8.4 m and 7.8×8.4 m, respectively. In this context, while rectifying RADARSAT-1 images they are generally re-sampled to 8×8 m resolution (Gauthier et al. 2006). In this paper, RADARSAT-1 images were rectified to a Turkish uniform coordinate system by re-sampling to an average resolution of 8 m. With regard to the ALOS PALSAR images, as shown in the ERSDAC PALSAR User Guide (ERSDAC 2006), the pixel spacing of the PALSAR image is 6.25×6.25 m, and the spatial resolution is approximately 9.5 m in range×10-m azimuth. It is also declared that the highest ground resolution is 7 m. Thus, the PALSAR images were rectified to a Turkish uniform coordinate system by re-sampling to an average resolution of 8 m. In the original (i.e. raw) ENVISAT ASAR images, the pixel size is 12.5 m and the geometric resolution is 22–37 m, depending on the image swath type and incidence angle. On average, the resolution is re-sampled at around 25 m (Beran 1994; Meadows and Wright 2002). Therefore, the ENVISAT images used in the study were rectified to a Turkish Uniform Coordinate System (i.e. UTM ED50, northing and easting in metres) with 25-m pixel spacing. The average of five ENVISAT ASAR images, which gives more detailed information especially for the linear features than does a single image, was calculated, and the output image was also considered in the analysis (Engdahl and Hyyppa 2000; Gunzl and Selige 1998; Wang et al. 2001).

3.2. Image-fusion methods used

In this study, the three most widely used data-fusion methods, namely IHS, Ehlers and Brovey were applied, and their results were examined. Since the late 1980s, various algorithms have been developed for fusing or pansharpening to combine the panchromatic and multispectral images to produce an enhanced multispectral image of high spatial resolution (e.g. Cakir and Khorram 2003; Chavez, Sides, and Anderson 1991; Chen, Hepner, and Forster 2003; Cliché, Bonn, and Teillet 1985; Ehlers 1991; Li, Kwok, and Wang 2002; Price 1987; Shettigara 1992; Welch and Ehlers 1987; Yesou, Besnus, and Rolet 1993; Zhang 1999; Zhang 2002; Zhang and Hong 2005; Zhou, Civco, and Silander 1998). According to a comprehensive review of multi-sensor image fusion in remote sensing done by Pohl and Van Genderen (1998), the existing image-fusion techniques are grouped into two classes: ‘colour-related techniques’ such as IHS and hue-saturation-value fusion methods and ‘statistical/numerical methods’ such as PCA, HPF, Brovey transform, regression variable substitution and wavelet methods (Ling et al. 2008). According to this classification, we took care to choose methods from both classes, and consequently we chose the IHS and Ehlers methods as the colour-related techniques and the Brovey Transform as the statistical/numerical method. These methods are also available in commonly used image-processing software because they are accepted as successful algorithms in the literature.

The IHS method separates the intensity, hue and saturation components of a three-band multispectral image by converting from RGB colour to IHS colour. Spatial frequency related to the intensity component is replaced with the high-resolution image and transformed back to the RGB domain using an inverse IHS transform (Pohl and van Genderen 1998).

The Brovey transform uses a ratio algorithm for combining the images. It can use all spectral bands for the input layer, which is different from the IHS method. The algorithm calculates the ratio of each image band by the sum of the chosen bands, followed by a multiplication with the high-resolution image (Ranchin and Wald 2000).

Ehlers fusion uses IHS transformation with subsequent Fourier domain filtering. First, it transforms the selected bands as RGB-to-IHS images, then intensity and high-resolution images are transformed into the frequency domain by a fast Fourier transformation (FFT). A low-pass filter and an inverse HPF are used for the intensity and high-resolution pan images, respectively. After an inverse FFT is applied to these images, the high-pass–filtered high-resolution image and the low-pass–filtered intensity image are added to form an enhanced intensity component. An inverse IHS transform is applied for the fused output image as RGB (Ehlers 2005). This process can be sequentially repeated until all bands are exhausted (Ehlers et al. 2010).

3.3. Quality assessment

For quantitative and qualitative analyses of the methods used, various composite images were visually evaluated to analyse the content of structural information. The following five false-colour composites (as R/G/B) were selected for the analysis (Figures 2 and 3):

1. Image I: ENVISAT 08/06(R)/RADARSAT 28/05(G)/ENVISAT 11/06(B)

2. Image II: ENVISAT 08/06(R)/RADARSAT 28/05(G)/ENVISAT average(B)

3. Image III: ENVISAT 08/06(R)/RADARSAT 28/05(G)/SPOT NIR(B)

4. Image IV: ENVISAT 08/06(R)/ENVISAT 11/06(G)/SPOT NIR(B)

5. Image V: ENVISAT 08/06(R)/ENVISAT average(G)/SPOT NIR(B)

Figure 2. Fusion results for selected images (image I. image II. image III).

Figure 3. Fusion results for selected images (image IV. image V).

Each composite image was fused with PALSAR data (Figure 4) with three different techniques, and as a result fifteen new images were generated (Figures 2 and 3). In general, we used C-band SAR data for RGB composites and L-band SAR data as a high-resolution substitute. For the different RGB SAR combinations, we tried as much as possible to use the radar data that had different polarizations and that was acquired in closer dates. However, to include all of the information available in all ENVISAT data gathered in a slightly larger time period, we also took the average of them into two composites. With regard to considering the average of ENVISAT ASAR images (i.e. the average data of 5 ENVISAT images), previous studies demonstrated that the simple technique known as temporal averaging has the advantage of reducing the speckle without losing spatial resolution (Engdahl and Hyyppa 2000; Gunzl and Selige 1998; Wang et al. 2001). As a result, fields become more homogeneous, and their boundaries are better defined.

Figure 4. Different information contents in (a) PALSAR image and (b) SPOT-XS (NIR) band.

In addition, we used two different approaches, namely the ‘SAR-optical’ and the ‘SAR-SAR’ data-fusion approaches. The SPOT data were used in two combinations, but for the other three combinations the radar data were preferred. Since we chose a study area with agricultural features as a land cover/use (i.e. vegetation), we chose only the near-infrared band of SPOT data due to the fact that it is the best spectral region in which to discriminate among the different vegetated fields. Therefore, the other SPOT bands were not taken into consideration.

In this study, the main idea was to produce new hybrid images for extracting information with the help of various sensors having different spectral, spatial and temporal resolutions (Figure 4). The main focus was not on extracting a certain object or phenomena from the output data; rather it was on seeing whether using different data-sets would improve the quality of the newly produced images. As seen in the Figures 2 and 3, the different information contents available in the two data-sets were the main reason they were used in the data fusion.

3.3.1. Visual comparison

In order to evaluate the spectral quality of the fused images, visual interpretation was done by comparing the fused output images with the selected original five RGB composites. The increase in the spatial resolution and its effects for the visual interpretation were analysed qualitatively by comparisons of the specific features such as field borders, roads and buildings in the original data-set.

3.3.2. Statistical evaluation

For quantitative analysis, image-fusion quality was evaluated using the following four statistical tools: BM, CC, SDD and UIQI.

3.3.2.1. Bias of mean

BM is the difference between the means of the original colour image and the fused output image (Bethune, Muller, and Donnay 1998). The value is normalised with respect to the mean value of the original image. The ideal value is zero.

3.3.2.2. Correlation coefficient

This measures the correlation between the original and the fused output images. The higher the correlation between the fused and original images, the better is the estimation of the spectral values. The ideal value of the CC is one.

3.3.2.3. Standard deviation difference

SDD is the difference between the standard deviations of the original colour image and the fused output image. Standard deviation reflects the average amount of the variation from the ‘average’ (mean) of the image. A low standard deviation indicates that the image pixels tend to be very close to the mean.

3.3.2.4. Universal-image-quality index

This is designed by modelling any image distortion as a combination of three factors: loss of correlation, luminance distortion and contrast distortion.

The image-fusion techniques aim to inject the spatial detail into the multispectral imagery while keeping the original spectral values. Today, the standard image-fusion methods used are often successful, but most of them distort the colour (spectral) information in the process. Therefore, the comparison between the methods was done to evaluate whether or not the spectral information in the input image was preserved in the output image. Within the fusion literature, there are many studies discussing why it is important to include sensor spectral consistency in image-fusion methods – e.g. Ehlers (2004, 2006), Ehlers et al. (2010), Klonus (2008) and Klonus and Ehlers (2007).

4. Application results

The spectral quality of the fused images was evaluated by visual interpretation – i.e. comparison with the original colour composite images and spectral profile plots. For that purpose, the profile shown in Figure 5 was taken into consideration, and changes in spectral values were evaluated by comparing the resulting profiles with those of the originals (Figure 6).

Figure 5. Spectral profile for analysing spectral similarities/changes after the fusion process.

Figure 6. Spectral profile analysis of the fused output images.

As can be seen in Figure 6, there was a high correlation between the output grey values of the Ehlers method and the original RGB images. Usually, CC and UIQI values obtained from the Ehlers method were quite high for image bands 2 and 3, and this inference is also valid for the statistical values for BM and SDD. The BM and differences in standard deviation were lower for image bands 2 and 3, whereas they were higher for image band one.

All fused output images were compared statistically with original composite images, and the results are presented in Table 2. Among the three methods, the Ehlers fusion yielded the best statistical results. In other words, the BM was the smallest, CC was the highest of all, SDD was the smallest, and UIQI values are were the largest. The other methods IHS and Brovey fusion performed worse with similar statistical results. For band one of all fused images, all three fusion methods showed poor performance, especially for the CC. However, in general, it was apparent that the poorest results were from the Brovey transform (i.e. higher biases of mean and SDDs), and CC and UIQI values for IHS and Brovey were lower than those obtained from the Ehlers method. CC and UIQI values were almost similar for IHS and Brovey.

Table 2. Statistical evaluation results (each band of RGB images compared with the corresponding bands of fused images).

		BROVEY				IHS				EHLERS
		CC	UIQI	BM	SDD	CC	UIQI	BM	SDD	CC	UIQI	BM	SDD
Image I	Band 1	0.092	0.126	0.874	0.092	0.193	0.248	0.752	−0.053	0.246	0.496	0.504	−0.024
	Band 2	0.726	0.240	0.760	0.113	0.651	0.313	0.687	0.027	0.986	0.807	0.193	0
	Band 3	0.586	0.386	0.614	0.049	0.563	0.310	0.690	0.008	0.990	0.798	0.202	−0.014
Image II	Band 1	0.092	0.126	0.874	0.092	0.202	0.344	0.656	−0.119	0.224	0.726	0.274	−0.025
	Band 2	0.726	0.240	0.760	0.113	0.612	0.432	0.568	−0.028	0.977	0.896	0.104	0.012
	Band 3	0.347	0.680	0.320	−0.126	0.358	0.421	0.579	−0.017	0.976	0.739	0.261	−0.017
Image III	Band 1	0.092	0.126	0.874	0.092	0.255	0.178	0.822	−0.028	0.225	0.679	0.321	−0.034
	Band 2	0.726	0.240	0.760	0.113	0.597	0.220	0.780	0.049	0.978	0.804	0.196	−0.011
	Band 3	0.663	0.207	0.793	0.123	0.107	0.209	0.791	0.090	0.982	0.854	0.146	−0.022
Image IV	Band 1	0.092	0.126	0.874	0.092	0.241	0.173	0.827	−0.003	0.223	0.668	0.332	−0.029
	Band 2	0.586	0.386	0.614	0.049	0.666	0.218	0.782	0.031	0.980	0.794	0.206	−0.016
	Band 3	0.663	0.207	0.793	0.123	0.126	0.206	0.794	0.100	0.985	0.853	0.147	−0.021
Image V	Band 1	0.091	0.126	0.874	0.092	0.240	0.183	0.817	−0.054	0.220	0.728	0.272	−0.036
	Band 2	0.347	0.697	0.303	−0.122	0.401	0.217	0.783	0.021	0.978	0.875	0.125	−0.003
	Band 3	0.660	0.211	0.789	0.122	0.107	0.213	0.787	0.079	0.981	0.849	0.151	−0.025

5. Conclusions

With the advances in satellite technology, there are now various kinds of image data-sets having different capabilities, including multi-sensor, multi-temporal, multi-resolution, and multi-frequency data, which are gathered from operational Earth-observation satellites and are widely used in many applications. Today, the fusion of image data has become a valuable tool in remote-sensing image evaluation to integrate the best characteristics of each type of sensor. This research investigates the quality assessment of multi-sensor (RADARSAT-1, ALOS-PALSAR, ENVISAT-ASAR and optical SPOT-2) image-fusion output, as well as the techniques used, namely IHS, Ehlers and Brovey transforms. Statistical analysis techniques used to investigate the fusion quality are BM, CC, SDD and UIQI.

The spectral quality of the fused images is evaluated by visual interpretation – i.e. comparison with the original colour composite images and the spectral profile plots. The comparison of the original colour composites with the outputs generated with the IHS, Ehlers and Brovey methods indicated that the IHS and Brovey fusion methods, although they distorted the colours, yielded much sharper images (i.e. had better appearance in the same resolution) compared with the original images. On the other hand, as can be seen from the output images, the Ehlers method generally retained spectral consistency more than the other methods, even for data having different spectral-response information (e.g. band centre, bandwidth). Therefore, this study indicated that the Ehlers method is more convenient for studies in which the spectral information content is the most important (i.e. thematic classification). Also, the Ehlers method has been accepted as the best method for radar/optical fusion for the three algorithms applied, with IHS in second place. However, it was also observed that averaging the radar data-set leads to additional sharpening in the Ehlers method, as can be seen in Figure 3, image 5.

Statistically, only four common evaluation criteria (BM, CC, SSD and UIQI) were applied in this study, and overall the Ehlers fusion method proved best in terms of spectral and statistical fidelity with the original RGB images among the three algorithms applied. Although these four criteria cover most image-evaluation aspects, our future work will involve adopting more indices to assess image-fusion results, with an emphasis on the effects of image fusion on classification accuracy.