Remote sensing image fusion: an update in the context of Digital Earth

Abstract

Remote sensing image fusion has come a long way from research experiments to an operational image processing technology. Having established a framework for image fusion at the end of the 90s, we now provide an overview on the advances in image fusion during the past 15 years. Assembling information about new remote sensing image fusion techniques, recent technical developments and their influence on image fusion, international societies and working groups, and new journals and publications, we provide insight into new trends. It becomes clear that image fusion facilitates remote sensing image exploitation. It aims at achieving better and more reliable information to better understand complex Earth systems. The numerous publications during the last decade show that remote sensing image fusion is a well-established research field. The experiences gained foster other technological developments in terms of sensor configuration and data exploitation. Multi-modal data usage enables the implementation of the concept of Digital Earth. In order to advance in this respect, we recommend that updated guidelines and a set of commonly accepted quality assessment criteria are needed in image fusion.

Keywords:

• remote sensing
• image fusion
• algorithms
• quality assessment
• needs

1. Introduction

Since the publication of an extended remote sensing image fusion review paper in 1998 (Pohl and van Genderen 1998), there have been many technological advances in the field of remote sensing image fusion, an enormous increase in computer power, speed and data storage, as well as new algorithms and applications. In the same year that the above mentioned review was published, Al Gore gave his famous speech on ‘Digital Earth: Understanding our planet in the 21^st Century’ (Gore 1998). His statement that ‘… we need a digital Earth. A multi-resolution, three-dimensional representation of the planet, into which we can embed vast quantities of geo-referenced data’ has stimulated much new research into interoperability and data integration of which remote sensing image fusion forms a part. In addition, numerous new satellites with much higher spatial and spectral resolutions have been launched since that date, which enabled many other developments in this increasingly useful research field and brought new challenges to the scientists. Hence this paper aims to inform the reader of some of these new developments, indicating the progress that has been made and highlighting future research needs.

In the late 90s, image fusion started to be accepted and applied in the remote sensing community. Since then, it has become part of commercial software packages, product lists of satellite image providers, and an important part of processing imagery for Digital Earth research. Fusion of multi-source images is considered to be a prime solution to optimize information extraction from remote sensing data (Zhang 2010; Fang et al. 2013). The challenges that researchers face nowadays are more related to the enormous increase in spatial and spectral resolutions of the images provided plus the development of standardized quality evaluation criteria (Thomas and Wald 2004; Thomas et al. 2008; Zhang 2008a; Bovolo et al. 2010).

The second section of this paper concentrates on new image fusion approaches followed by quality assessment methods. New developments in terms of remote sensing technology and computing hardware are presented in section four. Then we provide a description of new working groups, societies, and publications. The paper ends with conclusions to address new research fields.

2. New algorithms

The methods used to fuse remote sensing images have developed from generic algorithms towards adaptive and context-oriented methods. With the growing number of sensors and bands and the increase in resolution, this was a logical way to go to account for spatial and spectral integrity of the fused images. Researchers in the field have significantly improved the results of image fusion by considering the sensor as well as the study area characteristics along with the anticipated information that they are looking for. This led to more sophisticated approaches that give flexibility to the user to fuse three multispectral bands with one panchromatic band. The number of bands is not limited anymore, and the resulting fused image is adapted towards selected image content criteria, for example, spectral preservation or increased spatial enhancement. The following paragraphs discuss trends in image fusion, the challenges that users face today, and provide an overview on recently published techniques to fuse remotely sensed images.

2.1. Trends in image fusion

Back in the late 90s/early 2000, one of the main research areas in remote sensing image fusion was in the field of pansharpening, by combining high- and low-resolution images from the same satellite system. Examples are the combination of 10-meter panchromatic images with the 20-meter multispectral images from the Satellite Pour l'Observation de la Terre (SPOT) satellite (e.g. Vrabel 1996). Other commonly used image combinations applied optical data from different platforms, such as SPOT 10-meter panchromatic with Landsat multispectral images at 30-meter spatial resolution (e.g. Chavez, Sides, and Anderson 1991; Price 1999; Ranchin and Wald 2000). Other processing approaches focused on the use of optical and microwave remote sensing images due to applications that required input from synthetic aperture radar (SAR) (e.g. Harris, Murray, and Hirose 1990; Yesou et al. 1993). In the meantime image fusion has become well established. Pansharpening still enjoys high popularity (Ehlers et al. 2010; Guo et al. 2010; Massip, Blanc, and Wald 2012). A comprehensive review including an overview on the different applications and recommendations has been prepared by Zhang (2008b). The importance of pansharpening is also shown by a patent on the Gram-Schmidt (GS) pansharpening algorithm that has been filed in the United States (e.g. No. 6011875 by Laben, Bernard, and Brower 2000) and commercially used algorithms (e.g. Zhang 2004; Fuze Go™ 2013). Now, fusion algorithms have become standard functions in most commercial software packages such as ERDAS, PCI, ENVI, etc. Satellite and airborne sensors with higher spatial and spectral resolution are increasingly available. As a result, remote sensing image fusion is now also feasible between satellite imagery and airborne data-sets such as hyperspectral and Light Detection and Ranging (LiDAR) data (Huang, Zhang, and Yu 2012; Berger et al. 2013).

2.2. New challenges for the user

Taking into account the increased demand for details and parameters to be considered, researchers are facing new challenges:

• Integration of more than three spectral bands;

• High-resolution images require higher measures in geometric accuracy prior to image fusion;

• Observation angles impose different shadow features in different images in high-resolution data creating artifacts in fused images;

• Changes in the range of spectral bands of new-generation sensors require particular consideration in matching the panchromatic channel;

• The high spatial resolution imposes new features being disturbed (e.g. blurring of building edges or moving cars) that were not even noticeable before;

• Higher spectral and spatial resolutions put higher demands on the quality of fused images and therefore on the fusion technique itself;

• The geometric and radiometric requirements for fused imagery that is of good quality put a high demand on the operator's knowledge and processing complexity.

2.3. New sensors provide new potential

In terms of data sources being fused, there are new sensors being taken into account, for example, fusing airborne hyperspectral imagery with high-resolution satellite data (Borel and Spencer 2009; Cetin and Musaoglu 2009; Zhang 2012; Zhang et al. 2013), optical data with LiDAR data (Dalponte et al. 2008; Goodenough et al. 2008; Roberts, van Aardt, and Ahmed 2011; Buckley et al. 2013; Buddenbaum, Seeling, and Hill 2013), and high-resolution SAR images from satellite systems such as the 1-meter TerraSAR with high-resolution optical systems (Chen, Hepner, and Forster 2003; Amarsaikhan et al. 2010; Berger 2010; Lisini et al. 2011; Berger et al. 2013) for regular monitoring in cloud-covered regions of the world.

2.4. Development of new image fusion techniques

With the advancement in sensor technology and the availability of new, very diverse imagery available, the further development of commonly used fusion techniques became a prerequisite for success. Not only was it necessary to open the techniques to more than three bands, but also the amount of detail in the images required adaptive and context-oriented approaches. As a result, researchers derived new versions of popular fusion techniques, such as Intensity Hue Saturation (IHS), principal component analysis (PCA), Brovey transform (BT), wavelet-based techniques, and others and created so-called hybrid techniques (Zhang 2010) that consist of a combination of known techniques. A good summary and description of the adapted and amended methods can be found in Alparone et al. (2007) apart from the listing of methods provided in the next paragraph.

In terms of new algorithms, many researchers are concerned with various uses of the wavelet transform (Ranchin et al. 2003; Mercer et al. 2005; Gamba et al. 2012; Xiao and He 2013). Another development is the curvelet transform. To separate the image into unconnected scales, curvelets use multi-scale ridgelets with a bandpass filtering. In comparison to wavelets it has the advantage of integrating directional elements (Nencini et al. 2007). Interesting enough, more and more techniques are used in an integrated manner. Wavelets started to be combined with the traditional method of IHS (Amolins, Zhang, and Dare 2007; Roberts, van Aardt, and Ahmed 2011). Others apply adapted versions of substitution techniques, such as General Intensity Hue Saturation (GIHS) with genetic algorithm (GIHS-GA), IHS in combination with the Fast fourier transform (FFT), called Ehlers fusion (Ehlers et al. 2010). The approach by Ehlers extracts the spatial detail in the frequency domain and incorporates this information into the multispectral data by the IHS transform.

Another new technique is the GS approach, also existing as adaptive approach (GSA) and GSA using a local model (GSA-CA) (Aiazzi, Baronti, and Selva 2007; Choi et al. 2013), just to name a few. Others tackle the fusion process at a higher level using a unified framework by establishing correspondences between structures within high spatial resolution and high temporal but low spatial resolution images (Huang and Song 2012). All recent techniques aim at taking into account sensor-specific characteristics, image band fit, and the local context in the imagery (Park and Kang 2004).

A commercial algorithm that just entered the market is the Fuze Go™ (2013). It is based on the University of New Brunswick (UNB) algorithm, operating with regression analysis to produce weight factors, creating a synthetic image by multiplication and addition. The adaptive and context-specific approach preserves spectral integrity while adding spatial detail. Depending on the anticipated application, the user would choose different adjustments of the technique depending on the parameters needed for interpretation (spectral integrity or high spatial detail). The commercial offer is currently being extended to a visible and infrared (VIR)/SAR fusion algorithm called Fuze Go SAR.

Another interesting new look at image fusion is the approach published by Huang et al. (2013) called spatial and spectral fusion model (SASFM) using spectral unmixing and spectral libraries. It is a two-stage algorithm where the first step establishes an optimal set of atoms from a spectral library, followed by step two where the spectrum of each pixel is reconstructed to produce the high-resolution multispectral image. In summary, it can be stated the fusion algorithms started to become ‘intelligent’ meaning that they are not generic anymore but considering sensor, context, application, and other knowledge about parameters in the process.

3. Quality assessment

There is value in performing image fusion only if the resulting fused image contains information of greater quality than the individual input images alone. This becomes obvious when we look at the accepted definition of image fusion proposed by Pohl and van Genderen (1998). Thomas and Wald (2004) contributed very valuable insights to the quality assessment aspect in image and data fusion. They refer to the issue of defining quality gain and using a genuine reference to obtain a quality statement. As a result, it is very important to distinguish between relative and absolute quality. Relative quality refers to the closeness of the fused image to the original image. Absolute quality corresponds to the similarity between the fused and a reference image. The latter is produced synthetically due to the fact that there is no access to the image that has the quality parameters for comparison (Thomas and Wald 2007; Alparone et al. 2007).

Most commonly, users of image fusion use two different ways to assess their achievements. The first, a qualitative approach, is the visible inspection of the results, comparing the outcome to the original input data. The second, a quantitative approach, uses statistics and other assessment methods to provide comparable quality measures, such as quality indexes. An excellent overview on quality metrics is compiled in Li, Li, and Gong (2010). Others measure the outcome of an application of fused images, for example, in the context of land cover mapping and classification (Amarsaikhan et al. 2012; Colditz et al. 2006; Pohl and Hashim 2013).

A visual evaluation of the results to assess fused image quality is done in almost all published studies in the field. However, the research community has come up with a variety of established quantitative evaluation criteria that can be applied even though till today there is no unity on which criteria should be used as standard (Thomas and Wald 2004; Zhang 2008a).

A selection of commonly used indexes and assessment methods are listed in Table 1. This table is not meant as review, nor does it list all published indexes. It illustrates the variety of quality measures image fusion users have access to and the difficulty of choosing the ‘right’ one.

Table 1. Examples of commonly used fusion quality assessment indexes.

Quality index	Meaning of abbreviation	Main quality assessed
CC	Correlation Coefficient	Spatial or spectral
DM	Difference of Means in radiance	Spectral
RMSE	Root Mean Square Error	General
SAM	Spectral Angle Mapper	Spectral
RASE	Relative Average Spectral Error	Global spectral quality
SID	Spatial Information Divergence	Spectral
ERGAS	Erreur Relative Global Adimensionelle de Synthèse	Global spectral quality
UIQI (Q-average)	Universal Image Quality Index	Global spectral and spatial
QPS	Quality PanSharpening	Global spectral & spatial
CM	Convolution Mask	Spatial
MSSIM	Mean Structure Similarity Index Measure	Luminance, contract & structure
SSQM	Structural Similarity Quality Metric	Spatial
ED	Edge Detector	Spatial
SPD	Spectral Discrepancy	Spectral
AG	Average Gradient Index	Spatial

There is plenty of literature explaining the various indexes and applying them in image fusion studies (Gonzalez-Audicana et al. 2004; Wang et al. 2004; Alparone et al. 2007; Ling et al. 2007; Li, Li, and Gong 2010; Meng, Borders, and Madden 2010; Padwick et al. 2010; Fonseca et al. 2011). An interesting study is described in a recent paper by Abdikan et al. (2012) focusing on quality considerations in the case of optical and radar image fusion.

What we can also see from Table 1 is the fact that there is neither consensus in the literature on a uniform quality assessment approach, nor do we have a consistent naming for the various indexes. Looking at the Mean Structure Similarity Index Measure (MSSIM) and the Structural Similarity Quality Metric (SSQM), for example, the different authors use similar approaches but the individual researchers keep producing their own terminology (Wang et al. 2004; Zheng and Qin 2009). In the context of Digital Earth this is not very useful. To achieve a multi-dimensional, multi-scale, multi-temporal, and multi-layer information facility for a better understanding of the complex Earth systems, we need unified approaches and terms. This may be a role that the International Society of Digital Earth could take up via one of its Working Groups (ISDE 2013).

4. New developments in technology

Since 1998, computer performance has greatly increased: from about 2000 million instructions per second (MIPS) at 600 MHz in the end of the 90s to about 180,000 MIPS at 3 GHz in 2011. Data storage has increased tenfold. Instead of talking about megabyte capacity we are now dealing with Tera- or even Petabytes (Wikipedia 2013). As image fusion requires not only the input images but also many intermediate stages, such as geometrically and radiometrically corrected images, enhancement or adaptation pre-processed images plus the fused output image, these developments have made it much easier today to perform image fusion than 15 years ago. Hence current researchers are able to fuse very large image data-sets from different sensors at high resolutions even using sophisticated fusion techniques in a timely manner. This enables many more applications to be carried out efficiently but also introduces new problems and issues to be considered as already mentioned in section 2.2.

The imagery provided by the many launched satellites since 1998 also has improved in quality, with regard to looking at the spatial resolution and the number of spectral bands available. There are high-resolution optical (spatial and spectral resolution) as well as microwave (spatial resolution and polarization) sensors. The high-resolution data have become more and more publically available due to the launch of many commercial satellites in contrast to the many military satellites before. In the 90s spatial resolutions ranged between 10 and 80 meters, now we look at 0.5 to 1 meter resolutions introducing a whole lot of new challenges (e.g. matching accuracy, shadow, distortions, etc.). The number of spectral bands that we were concerned with in the 1990s was up to seven (Landsat TM, later ETM + ); now hyperspectral sensors deliver hundreds of bands. We have access to stereo capabilities, and the sensors cover the entire range of the electromagnetic spectrum. In image fusion there are new aspects to be considered due to the fact that the spectral range of panchromatic bands has changed (Padwick et al. 2010). In addition, previously working algorithms, such as the very popular IHS or the BT, are not working anymore because they are limited to three multispectral bands. Multi-look capabilities introduce different shadow effects between different image acquisition dates that have to be accounted for. In radar remote sensing, the multi-polarization aspect provides a whole lot of new information to the user to be dealt with (Goodenough et al. 2008), not even talking about the amount of data to be handled.

5. Image fusion working groups and societies

From experimental stages in the very beginning of remote sensing image fusion, for example by Welch and Ehlers (1987), to commercial fused image production the technology has come a long way. It is only natural that professional societies recognized this and set up new mechanisms to advance knowledge in this field.

5.1. International Society of Digital Earth (ISDE)

In 1999, the Chinese Academy of Sciences organized the first International Symposium on Digital Earth in Beijing. That led to the establishment of the International Society of Digital Earth (ISDE), which organizes biannual Symposia and Digital Earth Summits. These events plus the Society's journal, the International Journal of Digital Earth, have shown how image fusion techniques have enabled highly accurate 3-D models, integration of multi-source, multi-resolution, and multi-temporal global earth observation data-sets to be produced (see also below) (Liu et al. 2012; Abdikan et al. 2012; Wu et al. 2013; Yu et al. 2013).

5.2. International Society of Photogrammetry and Remote Sensing (ISPRS)

In 2004, ISPRS set up a new Technical Commission VII entitled: Thematic Processing, Modeling and Analysis of Remotely Sensed Data. This Commission established a Working Group (VII-6) on Remote Sensing Image Fusion, chaired by Zhang Jixian (ISPRS 2013). During the past nine years, this Working Group has organized several international remote sensing image fusion symposia, such as on optical-SAR image fusion. In August 2013, the third International Workshop on Image and Data Fusion was held in Jilin, China. Major issues discussed in this workshop included the monitoring of changes and improvement of our understanding of the dynamics of processes related to food and water security, sustainability, climate change, and disasters. The researchers presented new fusion techniques; issues in the use of SAR, Interferometric SAR (InSAR), and Unmanned Aerial Vehicles (UAVs); as well as more reliable and efficient ways to produce and update spatial data to feed Digital Earth.

5.3. IEEE: Geoscience and Remote Sensing Society (GRSS)

Another activity that boosted research in remote sensing image fusion was the IEEE's GRSS Working Group on image fusion. The IEEE GRSS Data Fusion Technical Committee (DFTC) serves as a global, multi-disciplinary network for geospatial data fusion, connecting people and resources ( http://www.grss-ieee.org/community/technical-committees/data-fusion/ ). A very interesting and advancement-fostering activity of DFTC is the annual Data Fusion Contest (http://www.grss-ieee.org/community/technical-committees/data-fusion/). It has been run since 2006. Here, a well calibrated data-set is ;provided to all interested researchers to show their fusion algorithm for fusing and classifying the two different remote sensing data-sets. The first contest was on pansharpening (Alparone et al. 2007), the results of 2007 were published in Pacifici et al. (2008) dealing with urban mapping, while in 2012 on very high spatial resolution multi-modal/multi-temporal image fusion using multispectral, SAR, and LiDAR data (Berger et al. 2013), and this year (2013) the contest participants received a hyperspectral data-set along with a LiDAR image. Throughout the years the interest in this contest increased to more than 1000 researchers.

5.4. The International Society of Information Fusion (ISIF)

The International Society of Information Fusion (ISIF) (http://www.isif.org) is a large society focusing on all aspects of information fusion. This is of course much broader than remote sensing image fusion. There is a paper on their website by Bostrom et al. (2007) giving an interesting perspective on what information fusion is as a research discipline. The society holds an Annual International Conference, which produces relevant Proceedings. The ISIF also publishes a journal, namely the Journal of Advances in Information Fusion. They have produced several interesting Special Issues. The Society has three Working groups, of which the one on ‘Fusion Process Model and Framework’ is of most relevance to remote sensing researchers.

6. Publications

As a result of all the efforts being taken in remote sensing image fusion, a variety of new journals, textbooks, and research articles have been published in the meantime.

6.1. New journals

One of the most significant events in this field was the launch of a dedicated new international peer-reviewed journal entitled: International Journal of Image and Data Fusion (IJIDF). The first issue came out in the beginning of 2010. Next year (2014), that journal will apply for Science Citation indexing with Thomson-Reuters. The journal is published by the same publisher as this article. The IJIDF is not limited to but more than 90% of the published papers to date are on remote sensing image fusion (http://www.tandfonline.com/tidf).

Another new journal since 1998 is the Elsevier Journal ‘Information Fusion’. As has been mentioned above, the ISIF also has a journal, which at times contains useful papers for remote sensing researchers working on image fusion. These journals, although they include papers on remote sensing image fusion from time to time, are much broader in their aims and scope, and most papers are from non-remote sensing fields. However, their Special Issues are useful for remote sensing researchers.

Last but not least the International Journal of Digital Earth launched in 2008 has published many papers on various aspects of image and data fusion in relation to Digital Earth (Shupeng and van Genderen 2008; Abdikan et al. 2012; Wu et al. 2013; Yu et al. 2013). Due to the integrative nature of Digital Earth, the journal contributes significantly to the dissemination of advances in image and data fusion.

6.2. New textbooks

In 1998 there were no dedicated textbooks on remote sensing image fusion. In fact, even now, in 2013 there is not a good textbook about this subject yet. Some of the better books available back in 1998 were those by Hall (1992), on mathematical techniques in multi-sensor data fusion, and that by Bar-Shalom in 1995 that was updated in 2012 (Bar-Shalom, Willett, and Tian 2011). This shows the increasing emphasis on image and data fusion, as the new version devotes four chapters (instead of just one) to image and data fusion (more than 250 pages). As of today, there are many up-to-date excellent books on data fusion, but fewer on image fusion, and even less on remote sensing image fusion. Books on data fusion that ought to be mentioned are those by Klein (1999), Hall and McMullen (2004), Liggins II, Hall, and Llinas (2009), and Mitchell (2012). One of the earliest books on remote sensing image fusion was that by Wald in 2002. It focused on pansharpening of SPOT images (panchromatic with multispectral images). Of course, as both sensors are on the same platform, with the same orbital characteristics, this is much more straightforward than fusing images from different satellite systems, with different orbit inclinations, different sensor geometry, completely different spectral characteristics, etc. There are some recent books on Remote Sensing Image Fusion, such as those by Mitchell (2010), Dong et al. (2011), Stathaki (2011), Zheng (2011), and that by Zhao (2012). But most of these are edited books consisting of a collection of papers on the subject by multiple authors. What is really needed is a good textbook for students, so that they understand the concepts, issues, techniques, and applications, as well as the challenges of fusing remote sensing images from disparate sources.

6.3. New research papers

Since 1998 more than 1000 papers on the subject of remote sensing image fusion have been identified by the authors, both in peer-reviewed international remote sensing journals, as well as many in remote sensing conference proceedings. Some of the most interesting recent review papers that have appeared on the subject of image fusion are those by Karathanassi, Kolokousis, and Ioannidou (2007), Zhang (2010), and Foo and Ng (2013). An excellent review on the broader subject of data fusion can be found in Khaleghi et al. (2013). As a matter of fact these authors tackle issues that also play an important role in remote sensing image fusion, such as data correlation and data registration.

In the many remote sensing image fusion papers that have appeared recently, one can observe the trend towards much more quantitative evaluation of the fusion algorithm performance. There has been a marked shift from visually comparing the fused outputs to a more systematic quantitative evaluation (Wang and Bovik 2002; Li, Li, and Gong 2010; Carvalho and Chang 2012; Gamba et al. 2012; Han et al. 2013). In terms of applications, urban analyses, fusing either high and low spatial resolution images or fusing optical and SAR data-sets (Amarsaikhan et al. 2010; Wurm et al. 2011; Brook, Ben-Dor, and Richter 2013; Dahiya, Garg, and Jat 2013), and land cover changes/change detection are other major application areas (Zeng et al. 2010; Du et al. 2013). Several journals have devoted ‘Special Issues’ on the subject of image and data fusion. Here one should be mentioned in particular: the IEEE's Transactions on Geoscience and Remote Sensing, Special Issue on Data Fusion (2008). It contains a diverse number of papers on remote sensing image fusion for many different applications. Another application area that has received increasing attention from the remote sensing image fusion community is that of geology and mineral exploration (He and Zhao 2010; West and Resmini 2009; Beiranvand Pour and Hashim 2013). As a last example of a rapidly new field of research is the area of natural disasters (Llinas 2002; Cleary et al. 2012).

7. Conclusions

This update on the developments in remote sensing image fusion shows that there has been an enormous increase in the amount of research in this field over the past 15 years. At the time of the previous review paper, the main techniques focused on straightforward pixel-based fusion existing at that time and its applications. Today, the achievements of international research resulted in adapted, focused, and optimized techniques that take into account sensor and image characteristics as well as the intended application. It is obvious that higher resolutions in terms of temporal, spatial, and spectral characteristics require context dependent and locally operating algorithms and assessment tools. Furthermore, standardized assessment criteria are needed in order to evaluate the quality of fused products and the possibility to exchange information and apply fused images in the context of Digital Earth. Based on the efforts and achievements of Wald and his team, a fundamental framework has already been established (Wald 2000; Thomas and Wald 2004, 2007). Future research should concentrate on common image fusion strategies and optimization aspects to minimize efforts taken in fusion experiments in new application areas and the use of new sensor images along with standardization of quality assessment criteria for compatibility reasons.

Acknowledgements

The authors acknowledge the valuable comments of the reviewers to improve the original manuscript. They also acknowledge the support of the Research Alliance for Sustainability of the Universiti Technologi Malaysia.