Multivariate statistical analysis of measures for assessing the quality of image fusion

Abstract

Various measures are available for assessing image fusion quality. Some measures are from traditional image quality assessment and some are specially designed for image fusion evaluation. It has been found from a survey that there is a total of 27 measures in common use. It can be imagined that some of them are more reliable than others for certain applications and some of them may be quite highly correlated. Therefore, a thorough mathematical analysis of these measures is desirable to understand what measures should be adopted for a given application. This article describes a multivariate statistical analysis of these measures to reduce redundancy and find comparatively independent measures for assessing the quality of fused images. First, correlation coefficients are calculated for the 27 measures; then, factor analysis using principal components is performed based on correlation matrix; and finally, hierarchical clustering is carried out on the factors to obtain finer clusters and to find representative measures. Experiments are carried out on 144 fused images. Based on the results, the 27 measures are classified into five categories: difference-based, noise-based, similarity-based, information-clarity-based and overall-based. Further, the most representative measure is selected from each category as a recommendation.

Keywords:

• image fusion
• quantitative quality assessment
• multivariate statistical analysis
• principal component
• factor analysis
• hierarchical cluster analysis

1. Introduction

Image fusion is a process for combining two or more different images to form a new image by using certain algorithms. In remote sensing image fusion, the aim is to integrate spatial information (such as edges and textures) from a panchromatic (PAN) image (with higher spatial resolution) and spectral information (such as colours) from a multispectral (MS) image (with higher spectral resolution) to obtain an MS image with higher spatial resolution.

In the past two decades, many image fusion methods have been developed at three different levels, i.e. pixel, feature and decision level (Pohl and Van Genderen 1998). This leads to an increasing need for evaluation of the performance of image fusion algorithms (Angell 2005). Quality assessment is one of such evaluations. Two types of quality assessment are possible: direct and indirect. The former is to assess the quality of the fused images and the latter is to assess the quality of the products extracted from the fused images. This article will deal with the issues in the direct approach.

To assess the quality of fused images, some quality measures are required. The measures could be either qualitative (e.g. excellent, good, bad) or quantitative (see Section 2 for a review). Qualitative measures are normally used for the results of visual inspection due to the limitation of human eyes, although quantitative results can also be converted into qualitative terms. However, in mathematical modelling, quantitative measures are more desirable. As can be seen from a review in Section 2, a total of 27 quantitative measures are in common use. Naturally, one would ask ‘should I use all of these 27 measures every time when assessing the quality of fused images?’, or ‘which measure should be adopted in a given application?’. To answer these questions, a thorough mathematical evaluation is required. Indeed, in this project, a multivariate statistical analysis is employed to reduce the redundancy among these measures, i.e. find comparatively independent measures for assessing the quality of fused images.

Section 2 reviews existing measures for image fusion quality evaluation. Section 3 presents the strategies and methodology for multivariate statistical analysis of these measures. Section 4 reports experiments and results. Conclusions are made in Section 5.

2. Measures for evaluating image fusion quality

There are three approaches for assessing the quality of fused images according to the level of using reference images (Wang and Bovik 2006): no reference (NR), full reference (FR) and reduced reference (RR).

In NR quality assessment, as the name implies, NR image is used. A number of statistical parameters are computed from the test image. In FR quality assessment, a reference image is used as benchmark for comparison. From such a comparison, a number of parameters can be computed. In RR quality assessment, certain features are extracted from a reference image for assisting the evaluation of the quality of the test image.

The FR or RR quality assessment methods, as the names imply, employ reference images as benchmarks. However, in practice, no MS images at the same spatial resolution as PAN images are available for use as benchmarks (reference images) for comparison. On the other hand, for convenience, the resolutions of the images for comparison must be identical. To meet this requirement, two solutions are available. The first is to degrade the fused image to the same spatial resolution as that of the original MS image. The other solution is to degrade the original PAN image to the resolution of MS images and degrade MS images to a lower spatial resolution (normally by the ratio between MS resolution and PAN resolution), and then fuse the degraded images to obtain a new image (Wald et al. 1997). In this case, the original MS image can be used as reference image.

It has been found that 27 measures have been in use for assessment of fused images. These measures are expressed mathematically and explained in Table 1. It should be noted that the list shown in Table 1 is not complete and only the commonly used measures are included. For example, mean error and absolute mean error are omitted. It should be noted that some of the measures listed in Table 1 are for single images only (i.e. for NR approach). They are included because sometimes we are also interested in the quality of the fused image in an absolute sense, i.e. not necessarily comparative.

Table 1. The 27 measures for assessing the quality of fused images.

No.	Equation	Description
V1		Average gradient: reflecting the clarity of an image (Choi et al. 2003, Yang et al. 2007)
V2		Correlation coefficient: measuring the degree of correlation between the fused and the reference images (Karathanassi et al. 2007, Yang et al. 2007)
V3		Cross entropy: reflecting the information difference between the fused and the reference image (Sasikala and Kumaravel 2007)
V4		Entropy: measuring the richness of information in fused image (Vijayaraj et al. 2004, Chen and Blum 2005, Karathanassi et al. 2007)
V5		Relative dimensionless global error in synthesis: offering a global picture of the quality of the fused product (Alparone et al. 2004)
V6		Information fidelity criterion: measuring the information shared between the fused and the reference images using natural scene statistics (Sheikh et al. 2005)
V7		Mutual information: measuring the information shared between the fused image and the reference image using histograms (Qu et al. 2002, Canga et al. 2005, Chen and Blum 2005, Alparone et al. 2006, Sasikala and Kumaravel 2007)
V8		Mean square error: measuring the spectral distortion introduced by the fusion process (Karathanassi et al. 2007)
V9		Normalised least square error: indicating the normalised difference between the fused and the reference images (Sasikala and Kumaravel 2007)
V10		Peak signal-to-noise ratio: indicating the ratio between maximum information and noise of the fused image (Damera-Venkata et al. 2000, Canga et al. 2005)
V11		Quaternion theory-based quality index: measuring the radiometric, spectral and correlation between the fused and reference images using quaternion theory (Alparone et al. 2004)
V12		Objective image fusion performance measure: evaluating the success of gradient information transferred from PAN and MS images to the fused image (Xydeas and Petrovic 2000)
V13		Quality index based on local variance: dealing with the global statistics of the local variances of the images (Aja-Fernandez et al. 2006)
V14		Quality without reference: measuring spectral and spatial quality jointly (Aiazzi et al. 2006)
V15		Relative average spectral error: measuring the average performance of a method in the considered spectral bands (Ranchin and Wald 2000)
V16		Relative shift in mean: indicating the amount of information added or lost during fusion (Parcharidis and Kazi-Tani 2000, Vijayaraj et al. 2004, Yang et al. 2007)
V17		Root of mean square error metric: measuring the standard error of fused images (Beaulieu et al. 2003, Karathanassi et al. 2007)
V18		RR image quality assessment: representing the overall distortion between the fused and reference images using a wavelet-domain natural image statistic model (Wang and Simoncelli 2005)
V19		Spectral angle mapper: measuring global spectral distortion (Alparone et al. 2007).
V20		Standard deviation: reflecting the contrast of image (Chen and Blum 2005, Karathanassi et al. 2007, Sasikala and Kumaravel 2007)
V21		Spatial frequency: measuring the clarity of fused image. RF and CF are row frequency and column frequency, respectively (Eskicioglu and Fisher 1995)
V22		Signal-to-noise ratio: measuring the ratio between information and noise of the fused image (Damera-Venkata et al. 2000)
V23		Structure similarity index metric: measuring structural similarity between the fused and the reference images (Wang et al. 2004)
V24		Union entropy: reflecting the information shared between the fused and the reference images (Qu et al. 2002)
V25		Universal image quality index: measuring linear correlation, mean luminance and contrast between the fused and the reference images (Wang and Bovik 2002, Karathanassi et al. 2007)
V26		Warping degree: representing the level of optical spectral distortion (Eskicioglu and Fisher 1995)
V27		Weighted signal-to-noise ratio: measuring the weighted ratio between information and noise of fused image according to contrast sensitivity function (Kite et al. 2000)

The measures in Table 1 are listed in alphanumeric order and they look haphazard. On the other hand, it is believed that, at the end of the analysis conducted in this project, they will be nicely categorised automatically, that is they are unclassified by intention.

In Table 1, σ denotes the standard deviation; μ denotes the mean value; f is the fused image with a dimension of M × N; r is the reference image with the same dimension as f; (x, y) is the location of pixel in the image, ; K is the number of MS image bands and h denotes the normalised histogram of the image.

3. Strategies and methodology

3.1 A strategy for this study

As discussed in Section 2, there are a total of 27 measures found in literature commonly used for assessing image quality. It is impractical to make use of all these measures to assess the quality of image fusion each time. Therefore, the numbers must be reduced. To reduce the number, the first consideration is to remove redundant measures, that is, one representative measure may be selected from those, which are lightly correlated. To achieve this, the following strategy is employed in this study:

• select some images with different natures for fusion;

• compute the quality of fused images quantitatively using all these 27 measures;

• calculate the correlation coefficients of these measures;

• carry out factor analysis (FA) using principal component for the correlation matrix;

• conduct a hierarchical clustering on the factors to obtain finer clusters and

• recommend a representative measure for each cluster.

3.2 Pearson correlation coefficient

A number of solutions are available to obtain the correlation coefficient (Rodgers and Nicewander 1988). In multivariate statistical analysis, e.g. FA, the Pearson correlation coefficient is one of most commonly used measures of the correlation between two variables. It reflects the degree of linear relationship between two variables, which ranges from +1 to −1. A correlation coefficient of ‘+1’ means a perfect positive linear relationship between two variables. At the other extreme, a correlation coefficient of ‘−1’ means a perfect negative linear relationship between two variables. In the middle, a correlation coefficient of 0 means no linear relationship between the two variables. The definition of correlation coefficient is given in V2 in Table 1.

A correlation coefficient matrix can be built using the correlation coefficients between two variables for multivariate statistical analysis. This represents the normalised measure of the strength of linear relationship between variables.

3.3 FA using principal component

Principal component analysis (PCA) is a multivariate statistical technique used for data reduction and for deciphering patterns within large sets of data (Wold et al. 1987, Chen et al. 2007). Principal components are eigenvectors of a variance, covariance or a correlation matrix of the original data matrix. These eigenvectors may provide significant insight into the structure of the matrix not available at the first glance.

FA is a statistical method used to describe variability among observed variables in terms of fewer unobserved variables called factors. The observed variables are modelled as linear combinations of the factors, plus ‘error’ terms. The information gained about the interdependencies can be used later to reduce the set of variables in a dataset. The classical model of FA (Lawley and Maxwell 1962) is given by

where is the vector of the observed variables, is the vector of the latent common factors and is the vector of the latent specific factors. The matrix is the so-called loading matrix.

The main purpose of FA is to reduce the contribution of less significant variables to simplify the component structure of the results from PCA. This goal can be achieved by rotating the axes defined by PCA. Here, rotation is used to reorient the factor loadings so that the factors are more interpretable. The rotation of factors is carried out as a way of transforming the factors so as to obtain the apparent factor structure. The simplest case of rotation is an orthogonal (varimax) rotation in which the angle between the reference axes of factors is maintained at 90°. This type of rotation is used with FA using principal component.

3.4 Hierarchical cluster analysis

Hierarchical cluster analysis (HCA) is a method for finding the underlying structure of objects through an iterative process that associates (using agglomerative methods) or dissociates (using divisive methods) objects one by one, and that is halted when all objects have been processed (Almeida et al. 2007). The resulting clusters of objects should exhibit high internal (within cluster) homogeneity and high external (between clusters) heterogeneity (McKenna 2003).

Given N vectors , a hierarchical clustering algorithm attempts to establish a dendrogram between the sets of clusters according to the minimum distance rule:

1. starts with N clusters, each consisting of exactly one point (or vector);

2. find the most similar clusters C_i and C_j , then merge C_i and C_j into one cluster and

3. repeat step (2) for a total of N−1 times.

Any valid metric may be used as a measure of similarity between pairs of observations. The choice of which clusters to merge or split is determined by a linkage criteria, which is a function of the pairwise distances between observations. The commonly used linkage criteria are complete linkage (or furthest neighbour), single linkage (or nearest neighbour) and average linkage (between groups and within groups), centroid linkage and Ward's linkage. The use of different linkage criteria may lead to different clustering results. It has been found by Milligan (1980) that the hierarchical clustering with average linkage is preferred for the data sets possessing the features of internal cohesion and external isolation. This method is thus adopted in this study and squared Euclidean distance is used as a measure of similarity.

Average linkage treats the distance between two clusters as the average distance between all pairs of objects where one object of a pair belongs to each cluster. Cluster membership is determined by the minimum of

The quantity d_ik is the distance between object i in cluster U and object k in cluster W, and N_U and N_W are the numbers of objects in the two clusters.

4. Experiments

4.1 Test images

To make the number of samples large enough for statistical analysis, a total of 12 pairs of QuickBird PAN and MS images with different land covers are selected as original images for fusion. These covers include urban, rural or urban and rural mixed areas. They are shown in Figure 1. Also, a total of 12 fusion methods are performed on each of the image pairs. These fusion methods are average of PAN and MS images (Average), computationally efficient pixel level image fusion (CEPIF), contrast pyramid (Contrast), filter subtract decimate pyramid (FSD), gradient pyramid (Gradient), Laplace pyramid (Laplace), morphological difference pyramid (Morph.), ratio pyramid (Ratio), region-based (Region), Haar wavelet transform (Haar), discrete wavelet transform (DWT) and unsampling Contourlet transform (Contourlet) fusions. As a result, totally 144 test images are obtained as samples.

Figure 1. Original PAN and MS images for fusion.

Due to page limits, not all of the fused images are presented here. Figure 2 shows the 12 fused images from pair 1 using 12 different fusion methods mentioned above, and Figure 3 shows the fused images from the 12 pairs in Figure 1 using the same fusion method-contrast pyramid fusion.

Figure 2. A set of 12 fused images from pair 1 in Figure 1.

Figure 3. A set of 12 fused images from the 12 pairs in Figure 1 using the same fusion method.

4.2 Correlation of measures

All 27 measures are used to assess the quality of the fused images. In the end, a matrix of 27 × 144 is obtained as quality statistics. The correlation coefficients of measures are given in Table 2. Here, the degrees of freedom (df) is , where n is the number of samples and m is the number of variances. It can be found from a statistical table for the critical value of correlation coefficient, when df is 117 and the confidence level is 1%, the critical value is 0.228, that is when the correlation coefficient is larger than 0.228, two variables will be deemed to have good correlation.

Table 2. Pearson correlation coefficients of the measures listed in Table 1.

	V1	V2	V3	V4	V5	V6	V7	V8	V9	V10	V11	V12	V13	V14	V15	V16	V17	V18	V19	V20	V21	V22	V23	V24	V25	V26	V27
V1	1
V2	0.48	1
V3	−0.42	−0.58	1
V4	0.82	0.63	−0.36	1
V5	0.23	−0.04	0.60	0.37	1
V6	0.62	0.81	−0.53	0.60	−0.12	1
V7	0.43	0.77	−0.38	0.61	0.15	0.51	1
V8	0.14	0.00	0.55	0.35	0.93	−0.07	0.15	1
V9	0.30	0.06	0.47	0.42	0.94	−0.03	0.19	0.88	1
V10	−0.24	−0.08	−0.50	−0.48	−0.93	−0.01	−0.22	−0.94	−0.84	1
V11	0.57	0.60	−0.74	0.37	−0.50	0.76	0.34	−0.52	−0.36	0.44	1
V12	0.41	0.11	−0.23	0.35	−0.10	0.35	0.11	−0.01	−0.11	0.03	0.14	1
V13	−0.75	−0.48	0.31	−0.80	−0.36	−0.38	−0.58	−0.33	−0.36	0.43	−0.24	−0.36	1
V14	−0.15	−0.12	0.52	−0.01	0.56	−0.05	0.08	0.61	0.43	−0.62	−0.51	0.10	−0.03	1
V15	0.24	0.02	0.55	0.39	0.98	−0.05	0.19	0.96	0.92	−0.94	−0.47	−0.03	−0.41	0.59	1
V16	0.19	−0.02	0.60	0.35	0.99	−0.09	0.15	0.94	0.93	−0.94	−0.49	−0.13	−0.34	0.59	0.98	1
V17	0.19	0.04	0.53	0.42	0.95	−0.03	0.19	0.98	0.88	−0.98	−0.49	−0.03	−0.38	0.62	0.97	0.96	1
V18	−0.57	−0.75	0.55	−0.59	0.08	−0.67	−0.67	0.08	−0.01	0.04	−0.69	−0.01	0.46	0.23	0.05	0.06	0.03	1
V19	0.02	−0.10	0.68	0.18	0.92	−0.13	0.05	0.93	0.82	−0.92	−0.59	−0.09	−0.18	0.71	0.94	0.94	0.94	0.16	1
V20	0.79	0.57	−0.29	0.91	0.48	0.42	0.63	0.41	0.54	−0.48	0.25	0.24	−0.84	−0.08	0.49	0.44	0.46	−0.51	0.22	1
V21	0.99	0.52	−0.42	0.83	0.26	0.60	0.50	0.17	0.33	−0.27	0.54	0.40	−0.81	−0.15	0.28	0.22	0.22	−0.61	0.04	0.83	1
V22	−0.28	−0.06	−0.51	−0.47	−0.96	0.02	−0.23	−0.93	−0.88	0.99	0.44	0.04	0.47	−0.59	−0.97	−0.97	−0.97	0.02	−0.92	−0.52	−0.32	1
V23	0.64	0.75	−0.46	0.62	−0.04	0.95	0.43	0.00	0.06	−0.11	0.76	0.20	−0.34	−0.03	0.02	0.00	0.05	−0.68	−0.05	0.41	0.60	−0.07	1
V24	0.82	0.60	−0.40	0.95	0.25	0.66	0.53	0.25	0.31	−0.41	0.51	0.28	−0.72	−0.05	0.27	0.25	0.33	−0.67	0.11	0.78	0.82	−0.38	0.73	1
V25	0.63	0.75	−0.47	0.61	−0.05	0.94	0.44	−0.01	0.05	−0.10	0.77	0.16	−0.33	−0.04	0.00	−0.01	0.04	−0.71	−0.07	0.40	0.60	−0.06	1.00	0.73	1
V26	0.10	−0.01	0.60	0.33	0.94	−0.07	0.12	0.98	0.85	−0.97	−0.54	−0.07	−0.28	0.67	0.96	0.96	0.99	0.09	0.97	0.35	0.12	−0.95	0.02	0.25	0.01	1
V27	−0.14	0.00	−0.56	−0.37	−0.93	0.07	−0.16	−0.92	−0.83	0.98	0.52	0.05	0.37	−0.67	−0.94	−0.94	−0.97	−0.08	−0.93	−0.40	−0.17	0.97	−0.02	−0.29	0.00	−0.96	1

Table 2 is the correlation matrix of the 27 measures. From this table, it can be seen that many of the measures are highly correlated with each other. V3 is correlated with all the others. It means that CE is the most representative measure for assessing image fusion quality. However, it is not enough to consider only one measure. As it is difficult to detect finer structures in the relationships between measures from this correlation table, the correlation matrix is further evaluated using FA.

4.3 Factor analysis

The correlation coefficient matrix is analysed by FA using principal components. To verify the appropriateness of FA for this study, the Kaiser–Meyer–Olkin (KMO) measurement of sample adequacy and Bartlett test of sphericity are initially performed on the correlation matrix. In such a test, the sample size should be five times larger than the number of variables. Under this condition, when the KMO is greater than 0.5 and Bartlett's test of sphericity significance is less than 0.05, the sample is considered adequate for FA (MacCallum 1983).

As shown in Table 3, for this experiment, the KMO measure of sampling adequacy is 0.82 and Bartlett's test of sphericity significance (sig.) = 0.00 < 0.05. This means that the sample is adequate for FA. The preliminary results of FA are shown in Table 4.

Table 3. KMO and Bartlett's test of sample in this experiment.

Kaiser–Meyer–Olkin measure of sampling adequacy		0.82
Bartlett's test of sphericity	Approximate Chi-square	11200.80
	df	351
	Sig.	0.00
Note: df–degree of freedom; Sig.–significance.

Table 4. The initial eigenvalues of components by FA.

Component	Initial eigenvalues
	Total	Percentage of variance	Cumulative percentage
1	12.26	45.39	45.39
2	9.01	33.37	78.76
3	1.71	6.32	85.08
4	1.23	4.55	89.63
5	0.96	3.55	93.18
6	0.40	1.48	94.65
7	0.38	1.41	96.07
8	0.26	0.97	97.03
9	0.21	0.78	97.81
10	0.19	0.70	98.50
11	0.10	0.37	98.87
12	0.08	0.30	99.17
13	0.06	0.21	99.38
14	0.05	0.17	99.55
15	0.04	0.13	99.68
16	0.03	0.11	99.79
17	0.02	0.07	99.86
18	0.01	0.04	99.90
19	0.01	0.03	99.93
20	0.01	0.03	99.95
21	0.01	0.02	99.97
22	0.00	0.01	99.98
23	0.00	0.01	99.99
24	0.00	0.00	99.99
25	0.00	0.00	100.00
26	0.00	0.00	100.00
27	0.00	0.00	100.00

An eigenvalue gives a measure of the significance for the component. The highest eigenvalue means the most significant component. Eigenvalues of 1.0 or greater are considered significant (Kim and Mueller 1978). It can be seen from Table 4 that there are four principal components with the eigenvalues values larger than 1 and the accumulated variance of these four principal factors is 89.63%. This indicates that these four components can well reflect the information in the original data matrix. A scatter plot is shown in Figure 4, which clearly indicates which component should be retained (the eigenvalues of the factors retained should be larger than 1).

Figure 4. The scatter plot of components.

An orthogonal rotation using varimax method is then applied to simplify the principal component and render it easier to interpret.

Table 5 shows the component loading matrix after rotation. The loading is the correlation coefficient of measures and components. For each measure, the higher the loading value on the component, the more important the measure on this component. The component with the highest loading value for a measure is used to classify the measure into categories (factors). From Table 5, it can be seen that all the measures of Warp, SAM, RMSE, RM, WSNR, RASE, PSNR, MSE, SNR, ERGAS, NLSE, QNR and CE have the highest loading in component 1 and thus are classified into one category. As described in Table 1, these measures represent the spectral distortion, or signal-to-noise ratio or error of the fused image using the reference image. As the distortion (or the noise or the error) is calculated by the difference between the fused and the reference images; this category can thus be interpreted as difference-based. All the measures of UIQI, SSIM, IFC, CC, Q4, RRIQA and MI have the highest loading value in the second component and they are also classified into one category. These measures represent the structural similarity, correlation or information shared between the fused and the reference images. Thus, this category can be interpreted as similarity-based. All the measures of SD, QILV, SF, AG, Entropy and UE have the highest value in the third component and they are classified into a category. As described in Table 1, SD, SF and AG measure the clarity of fused image. At the same time, QILV and UE measure the information of the fused image. Thus, this category can be interpreted as information-clarity-based. The measure of has the highest value of loading in component 4 and it becomes a category by itself. As measures the gradient information transferred from PAN and MS images to the fused image, this category can be interpreted as overall-based.

Table 5. Rotated component loading matrix.

Measure	Component
	1	2	3	4
Warp	0.99	0.01	0.04	0.00
SAM	0.98	−0.05	−0.07	0.02
RMSE	0.98	0.01	0.17	−0.01
RM	0.97	−0.05	0.16	−0.08
WSNR	−0.97	0.02	−0.12	−0.01
RASE	0.96	−0.04	0.21	−0.02
PSNR	−0.96	−0.07	−0.20	0.01
MSE	0.96	−0.03	0.13	0.01
SNR	−0.95	−0.01	−0.26	0.03
ERGAS	0.95	−0.10	0.22	−0.07
NLSE	0.87	−0.02	0.28	−0.13
QNR	0.72	0.06	−0.34	0.29
CE	0.65	−0.43	−0.43	0.02
UIQI	0.00	0.96	0.19	0.10
SSIM	0.01	0.95	0.20	0.14
IFC	−0.08	0.92	0.23	0.21
CC	−0.04	0.79	0.37	−0.23
Q4	−0.55	0.71	0.30	0.00
RRIQA	0.12	−0.71	−0.44	0.30
MI	0.11	0.50	0.49	−0.30
SD	0.31	0.25	0.89	−0.07
QILV	−0.25	−0.19	−0.85	−0.10
SF	0.10	0.41	0.84	0.19
AG	0.08	0.43	0.79	0.25
Entropy	0.28	0.47	0.78	0.09
UE	0.20	0.59	0.68	0.11
	−0.09	0.08	0.33	0.85

4.4 Hierarchical clustering analysis

Hierarchical clustering creates a hierarchy of clusters which may be represented in a tree structure called a dendrogram. The root of the tree consists of a single cluster containing all measures and the leaves correspond to individual measures. The number of clusters can be found from the crossing of vertical cut-off line and dendrogram. The optimal cut-off line in the dendrogram is at the largest difference of distance between two successive combination stages (Beaulieu et al. 2008). The number of the stages equals the number of measures minus 1. The last stage is reached when the cluster U and cluster W include all the measures to be tested.

The combination schedule and the dendrogram for the category with highest loading value in component 1 (i.e. the difference-based category) are shown in Table 6 and Figure 5, respectively. The number of measures in this category is 13, thus the number of combination stages is 12. From Table 6, it can be seen that the largest distance difference between two successive stages is 409.3. These two successive stages are stage 11 and stage 12. Thus, in Figure 5, the optimal cut-off line is at a distance between 123.6 and 532.9. In this way, two sub-categories are obtained, the first of which consists of Warp, RMSE, MSE, RM, ERGAS, RASE, SAM, NLSE, QNR and CE, and the second of which consists of PSNR, SNR and WSNR. Indeed, it is quite understandable that the first sub-category measures the difference between fused image and reference image, but the second sub-category measures the ratio of signal to noise. Of course, how to select the optimal number of clusters is determined by the actual requirement of quality assessment. The first sub-category can also be further sub-divided according to the different position of the cut-off line. Similarly, the second sub-category can also be sub-divided.

Figure 5. Dendrogram for difference-based category.

Table 6. Combination schedule of measures in difference-based category.

Stage	Combination of two clusters		dU W	Difference of two successive dU W
	Cluster U	Cluster W
1	V17	V26	2.5	0
2	V5	V16	3.3	0.8
3	V10	V22	4.2	0.9
4	V8	(V17, V26)	5.9	1.7
5	(V5, V16)	V15	6.1	0.2
6	(V10, V22)	V27	7.4	1.3
7	(V5, V16, V15)	(V8, V17, V26)	13.9	6.5
8	(V5, V8, V15, V16, V17, V26)	V19	17.4	3.5
9	(V5, V8, V15, V16, V17, V19, V26)	V9	31.8	14.4
10	(V5, V8, V9, V15, V16, V17, V19, V26)	V14	114.7	83.0
11	V3	(V5, V8, V9, V14, V15, V16, V17, V19, V26)	123.6	8.9
12	(V3, V5, V8, V9, V14, V15, V16, V17, V19, V26)	(V10, V22, V27)	532.9	409.3

Similarly, the clustering sequence for the category with highest loading value in component 2 is shown in Table 7 and the resultant dendrogram is shown in Figure 6. It can be seen that RRIQA is the last measure to be combined in the cluster. It means that RRIQA is independent of other measures in this category and thus is a good representative to measure the similarity between fused and reference images. When this category is further sub-divided, for example, the cut-off line is at distance between 67.3 and 120.3, three sub-categories are obtained. The first sub-category consists of UIQI, SSIM, IFC and Q4; the second sub-category, CC and MI; and the third sub-category, RRIQA by itself.

Figure 6. Dendrogram for similarity-based category.

Table 7. Combination schedule of measures in similarity-based category.

Stage	Combination of two clusters		dU W	Difference of two successive dU W
	Cluster U	Cluster W
1	V25	V23	0.6	0
2	(V25, V23)	V6	14.5	13.9
3	V2	V7	65.4	50.9
4	(V25, V23, V6)	V11	67.3	1.9
5	(V25, V23, V6, V11)	(V2, V7)	120.3	52.9
6	(V25, V23, V6, V11, V2, V7)	V18	484.5	364.2

The process of clustering sequence for the category with highest loading in component 3 is shown in Table 8 and the resultant of clustering is shown in Figure 7. From Figure 7, it can be seen that QILV is the last measure to be combined and thus is representative of this category. When the cut-off line is at different distances, different sub-divided categories can be obtained.

Figure 7. Dendrogram for information-clarity-based category.

Table 8. Combination schedule of measures in information-clarity-based category.

Stage	Combination of two clusters		dU W	Difference of two successive dU W
	Cluster U	Cluster W
1	V21	V1	3.3	0
2	V4	V24	14.2	10.9
3	V20	(V4, V24)	43.3	29.1
4	(V20, V4, V24)	(V21, V1)	52.4	9.2
5	(V20, V21, V1, V4, V24)	V13	510.5	458.0

5. Conclusions

In this study, a comprehensive analysis of 27 existing measures for assessing the quality of fused images has been conducted. A total of 12 image pairs with different types of land covers are used for the testing. Correlation analysis, FA using principal components and hierarchical analysis have been employed as mathematical tools. Through such a comparative analysis, the 27 measures are classified into five categories. A representative measure is identified for each category as follows (with underline):

1. difference-based: Warp, SAM, RMSE, RM, RASE, MSE, ERGAS, NLSE, QNR and CE;

2. noise-based: SNR, PSNR and WSNR;

3. similarity-based: UIQI, SSIM, IFC, CC, Q4, RRIQA and MI;

4. information- and clarity-based: SD, QILV, SF, AG, Entropy and UE; and

5. overall-based: .

That is to say, the representative measures for assessing image fusion quality are CE for difference-based, WSNR for noise-based, RRIQA for similarity-based, QILV for information-clarity-based and for overall-based. It is therefore recommended that

1. when evaluating the difference between fused and original MS image, users should make use of difference-based measures;

2. when evaluating the noise of the fused image compared to the original MS image, users should make use of optimal noise-based measures;

3. when evaluating the similarity of structure or the shared information between the fused image and the original MS image, users should make use of the similarity-based measures;

4. when evaluating the information or clarity of the fused image, users should make use of optimal information-clarity-based measures; and

5. when evaluating the overall quality of the fused image comparing to the original MS and PAN images, they should make use of the overall-based measures.

Of course, more categories and representative measures can be obtained when the distance of cut-off line in the hierarchical clustering is smaller, if there is a need. This will certainly depend on the actual requirement.

Acknowledgement

This research was supported by the State 973 project (2006CB701304) in China and a project funded by The Hong Kong Polytechnic University (G-U633).