Validation of EO-derived information for crisis management: a Digital Earth perspective in the VALgEO expert community

New generation remote sensing technologies today open new application areas and demonstrate their effectiveness providing geo-information in support of disaster management. At the same time, new challenges can be identified and among them the validation of geo-information under the new specific constraints dictated by the disaster management application area has a crucial role. In the frame of the Digital Earth concept, geo-information is a sharable resource, and a resource is truly shareable not only if we know the formal code used to make it – and then we can read it – but also – and in particular during disaster – if we know the reliability of this information. This is crucial in order to avoid information overflow and time wasting during the decision-making process in support of disaster management. The ‘International workshop on validation of geo-information products for crisis management,’ VALgEO 2009, was held at the Joint Research Centre, in Ispra, Italy on 21–23 November 2009. The purpose of this first workshop was to initiate a dialog between the main actors of Emergency Response Services around validation and to raise awareness of the necessity for an independent formal assessment of geo-information for disaster management. This special issue includes a collection of invited papers presented at the workshop. It gathers interdisciplinary initiatives where the majority of the contributions either directly or implicitly address different aspects of validation of geo-information generated following a disaster, ranging from flood information to forest fires and earthquake damage assessments.

1. Earth Observation (EO) for disaster management

The tremendous development of space technologies in the last decade has opened new possibilities to use remotely sensed image data in support of new application areas including disaster management. Improved spatial, temporal, and spectral sensors are available today. The meter or sub-meter spatial resolution available now in remote sensors allows the observation, recognition, and reporting about targets and phenomena present on the ground that were impossible to detect a few years ago with a spatial resolution of about 5–10 meters or more. Moreover, constellations of these new Earth Observation (EO) platforms are assuring an increased temporal resolution, with increased spectral capabilities including more visible and infrared bands in the optical domain, and recently also including microwave (SAR) very high resolution data.

These new generation EO image data can represent a precious source of information in support of the whole cycle of crisis management activities, including risk assessment, disaster preparedness, warning, alerting, damage assessment, needs assessment, emergency response, rehabilitation and reconstruction, and development (Al-Khudhairy 2010). In particular, while the hazard or disaster impact area (for example, wind, flood, fire, earthquake) is often also observable using moderate resolution satellite data eventually combined with modeling techniques, the evaluation of the impact on the population goes necessarily through the extraction of information related to the status of human settlements (built-up structures and infrastructures), and this information is detectable only using meter and sub-meter resolution image data (Pesaresi and Ehrlich 2009).

Unfortunately, the development of the sensor technology is not fully aligned with the development of the methodology used for information extraction from these new EO imageries and for the validation of this EO-derived geo-information. This fact may generate the paradoxical and unexpected effect of the decrease of the capability to generate and use the EO-derived information with the increase of the quality of the image source, namely the spatial, temporal, and spectral resolution of the sensor.

The main technical challenges to be faced can be regrouped into two items: (1) issues related to the image data preparation and (2) issues related to the proper information extraction. Regarding the data preparation, we point the attention to the systematic insufficient available information, digital elevation model (DEM), and ground control points (GCP) for effective geo-coding of the image data. This fact generates the effect of image data having systematically a displacement error greater than the pixel size, leading to unavoidable spatial inconsistency of the image datasets that have to be handled by the subsequent image processing steps involving multi-temporal image information comparison.

From the point of view of the information extraction, we may spot the attention on (1) the dramatic increase of the data volume and complexity, including the necessity to handle 3D geometry (parallax, panoramic distortion) and spatial imprecision; (2) the increasing of complication and then of instability of the inferential models used for automatic information extraction with a general effect of decreasing of robustness and generality; and then (3) the geometric increase of the computational capacity needed for effective image processing (Pesaresi et al. 2010).

The technological challenges listed above are related to the nature of the new satellite image data source and they interact significantly with a set of constraints that are typical of the ‘crisis management’ application area. These constraints can be summarized as follows: (1) the need for a globally and synoptic consistent approach (action anywhere, anytime, multi-scale); (2) the need for multi-temporal analysis (pre-/post-event, analysis of trends); (3) the need for rapid or time-critical assessment (hours, days) on sometime large areas; and (4) the fact that very often basic information is missing, making some basic image processing steps (geo-coding, building the image interpretation keys, validation) difficult.

In short, new EO image data are more suitable for providing information support to disaster management, but are producing new challenges regarding the consistency and the control of the quality of the extracted information, in the specific disaster management operational constraints.

2. Digital Earth (DE), disaster management, and validation role

The Digital Earth (DE) concept has been successfully linked to disaster management application areas for a long time (Qisheng et al. 1999), and EO-derived geo-information for disaster management is included as a key application area in the Beijing Declaration on Digital Earth (2009). The way to collect, process, and deliver geo-information has improved dramatically in recent years. New generations of EO platforms and sensors and a new generation of DE systems are able to access, process, and distribute geo-information and have demonstrated their effectiveness in providing support to the whole disaster management cycle including prevention, preparedness, response, and recovery. The growing effort to use geo-information for disaster management produced relevant initiatives, e.g. the Global Earth Observation System of Systems (GEOSS), the Global Monitoring for Environment and Security (GMES), the International Charter Space and Major Disasters, and the United Nations Platform for Space-based Information for Disaster Management and Emergency Response (UN-SPIDER). Moreover, one can observe that the capacity of the recent DE systems (in accessing and distributing image and geo-information) is relatively more developed than the capacity to extract information from new generation EO image data (Pesaresi et al. 2010), and consequently to control the reliability of this information. This fact can lead today to the unexpected effect of fast and effective distribution of uncontrolled, non-validated geo-information derived from new EO sensors, eventually producing information overflow, contradictory asse ssments, and other negative impacts on the decisional process.

The original DE vision (Gore 1998) can be viewed as a shared and accessible repository of geo-knowledge and geo-information for intelligent actions that in our case includes decisional support in the event of a disaster. In this perspective, it is important to understand that DE frame cannot be reduced to the notion of shared code, agreed format of the geo-information and communication protocol and metadata so that it can be queried, read, and understood by the distributed users and DE systems. This is necessary but not sufficient for real interoperability, or in other words to interoperate these geo-information in real disaster scenarios in support of decision-making processes. The real use of any kind of information in support of the decisional process means to control the reliability and quality of the information ingested in the local system. The real major disaster scenarios are characterized by the existence of multiple and concurrent producers of assessments from possibly non-consistent sets of data input. Moreover, these geo-information producers may have a different level of structure from the governmental institution, to the industry and the volunteer NGO, having radically different approaches regarding quality control and delivering protocols, standards, and documentation. In these scenarios, it becomes crucial to be able to activate an independent mechanism able to assess and label the quality of the geo-information flowing between the global DE system and the local problem-driven DE systems implemented in case of a disaster. In other words, to design and implement efficient and independent validation is crucial for interoperating geo-information in real complex disaster management scenarios using DE technologies. The cost of the absence of validation would be to decrease dramatically the possibility of use of this information in support of the decisional process.

3. Five years of emergency services provision: the current situation and the recent advent of validation

In parallel to the growing complexity of information content, the end-users of crisis maps derived from EO data are also evolving and becoming increasingly demanding.

The statement of ‘something better than nothing’ is no longer valid. The need for robustness, quality, and relevance of the information is gradually becoming recognized as a major component of satellite-derived geo-products. In the specific case of GMES, advances have already been made within Emergency Response Services. A very strong network of service providers has been established during the last 5 years. Throughout this period, service delivery to end-users, has been running without any validation, or with only some elements of validation. After 5 years of service delivery, the users are now asking not only for the information but also for its accuracy.

The need for an independent formal assessment of these products to provide operational services at homogeneous and reliable standards has recently become recognized as an integral component of service development. Validation is intended to help end-users decide how much to trust geo-information products (maps, spatial dataset) and, combined with quality assurance, can help to identify improvements.

The Joint Research Centre1 of the European Commission is providing scientific support to EU external relations policies, in particular those concerned with international crisis management and mainly the external security dimension of GMES policy. It is in charge of validation activities in GMES projects (e.g. SAFER, G-MOSAIC…) and endeavoring to develop a framework for the validation of emergency response geo-information products.

In that context, a dedicated effort has been made in the development and implementation of a comprehensive, product-oriented validation protocol (Broglia et al. 2010) tailored to crisis maps and dataset products. The purpose of this protocol is to ensure that the services are delivered with similar quality, agreed standards, and degrees of certainty and within the specifications derived from the users’ expectations. The protocol considers different aspects within the process of geo-information generation and its delivery to the end-users: (1) the reliability of the information content; (2) the consistency of the information support; and (3) the usability of the product.

Recently, the JRC validation team has applied a subset of the validation protocol to assess the quality of a sample of 250 crisis maps produced by several Service Providers in the period starting from the Indonesian Tsunami of December 2004, and ending with the Haiti earthquake of January 2010 (Carrion et al. 2010). The chosen subset of the protocol comprises parameters that can be considered as essential information to allow a clear understanding of the maps and their integration with local Geographic Information Systems. Besides only elements that could be evaluated visually, without any reference data, have been taken into account (such as the presence and the completeness of the title, the presence of the scale, the readability of the legend, and the presence of metadata). Out of 25 parameters that have been checked, 80% are fulfilled by 15% of maps and only 50% of parameters are fulfilled by almost all the maps (95%). This result shows that a lot of work can be done to improve the readability of crisis maps and their possibility to be integrated with other data as an effective support in crisis management.

4. Scope of VALgEO workshop

As part of its activities related to validation, JRC organized VALgEO 2009, the ‘International workshop on validation of geo-information products for crisis management’ that attempted to initiate a dialog between the main actors of Emergency Response Services. The main objectives of the workshop were to:

•	share and compare expectations, needs, experiences, points of view with respect to scientific, technical, or policy-related validation issues;
•	discuss and compare methodologies, practical experiences, and operational procedures that can be used for the validation of maps and spatial datasets;
•	establish validation criteria that can help to assess the quality of geo-information with respect to users’ expectations;
•	draft recommendations for a generic protocol for the validation of geo-information for crisis management; and
•	start a professional network.

The workshop brought together participants with different professional competencies:

•	policy experts from international organizations involved in the disaster cycle management and interested in the role of geo-information;
•	academic, scientific, and technical people with expertise in the validation of geo-information;
•	people with sector-dependent professional competencies applied to specific disasters, e.g. forest fires, floods, earthquakes, etc.;
•	users and experts in field operations; and
•	people from geo-information industry and emergency service providers.

The detailed information of the workshop can be found at http://isferea.jrc.ec.europa.eu/Workshops/2009-11-VALgEO/Pages/default.aspx

5. Content of this special issue

This special issue is the result of a joint effort between the organizers of the workshop, the participants, and the editors of the International Journal of Digital Earth. It includes a collection of invited papers covering a wide range of topics in relation to validation of geo-information.

This special issue faces the validation of crisis maps, starting from its basic principles, discussed in the following three papers.

In the paper ‘Development and implementation of a validation protocol for crisis maps: reliability and consistency assessment of burnt area maps,’ Corbane et al. present the validation protocol developed at JRC in the framework of the Services and Applications For Emergency Response (SAFER) project and the main principles that it takes into account, such as the Reliability of the information content, the Consistency of the information support, and the Usability of the product. Then the authors show the main results of the application of the protocol to a study case on burnt area maps, highlighting the critical issues.

In the paper ‘Generalisation, symbol specification and map evaluation: feedback from research done at COGIT Laboratory, IGN France’ Duchêne et al., the results of the research conducted on generalization and symbols specification are discussed. Many indications are given for the evaluation of the whole generalization process and of the symbol spe cifications. Moreover, the main problems regarding the generalization in the framework of crisis mapping are analyzed.

Gallego in ‘Validation of GIS layers in the EU: getting adapted to available reference data’ discusses how to deal with the lack of ground-truth information to validate thematic maps, using data available at the European Union level. The CORINE Land Cover 2000 (CLC2000), soil sealing layer (SSL), and Land Use/Cover Area-frame Survey (LUCAS) data are considered. Validation and cross-validation examples are presented and advantages and limitations of systematic sampling are analyzed.

Following the first part, which dealt with the issues related to the validation methodologies, a few examples of applications are presented in the next five papers.

In the paper ‘The independent service validation in GMES RESPOND: the flood validation exercise,’ Shimoni et al. present the validation exercise carried out by the independent service validation group of the GMES RESPOND project on flood damage mapping. For this exercise six service providers, who based their work on EO information, have been involved. The products have been compared both with ground truth and among them, to evaluate accuracy and consistency of data.

In ‘Quality control, validation and user feedback of the European Flood Alert System (EFAS),’ De Roo et al. present validation and quality control of the European Flood Alert System (EFAS). It is shown how the performance of the flood forecast service has been checked continuously over time thanks to the users’ feedback and to the development of specific methods of verification.

In ‘Monitoring changes in the Menik Farm IDP camps in Sri Lanka using multi-temporal very high resolution satellite data,’ Kemper et al. present the automatic extraction internally displaced persons (IDP) camps from very high resolution optical satellite images thanks to morphological analysis and the validation of the results. The EO acquired information is crucial for the location and size estimate of IDP camps, since no field data are usually available.

Tenerelli and Ehrilch, in their ‘Analysis of built-up spatial pattern at different scales: can scattering affect map accuracy?’ investigate the relationship between spatial allocation of error and built-up distribution patterns. They found a positive exponential relationship between the Mean Scattering Index, which is related to the spatial pattern of buildings, and the overestimation error of the built-up extracted from VHR satellite imagery. They also affirm that the effect on overestimation errors decreases when the spatial resolution of images increases.

In their paper ‘Mapping urban building stocks for vulnerability assessment – preliminary results,’ Saito and Spence apply a methodology, based on a family of Gabor filters and on unsupervised clustering, that allow detecting urban areas on very high resolution optical satellite images. The authors have also found a correlation between the clusters singled out and the distribution of structure types, in particular reinforced concrete and unreinforced masonry. This brings the possibility to estimate the number of buildings of each typology given the building inventory at least for a sample of each cluster.

Taken together, the papers published in this special issue give new insights into the meaning, the purpose of validation and its added-value. They can be regarded as a starting point for the establishment of a strategy and of best practices guidelines for achieving validation tasks of crisis geo-information products.

Acknowledgements to referees

All papers in this special issue were double peer-reviewed. We wish to acknowledge publicly the valuable services provided by the reviewers. Reviewers are usually not identified to the authors, but we take this opportunity to identify and thank them as a group. On behalf of the associate editors, the editorial advisory board, authors and readers of the International Journal of Digital Earth, we wish to thank the reviewers for their time, hard work, and dedication in reading and drafting critiques of each paper submitted to this special issue.

List of referees

Aksoy Selim, Bilkent University, Turkey

Baghdadi Nicolas, CEMAGREF, France

Biorgo Einar, UNITAR, Switzerland

Brunner Dominik, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, Germany

Weber Christiane, CNRS, France

From the Joint Research Centre

Broglia Marco

De Groeve Tom

Gueguen Lionel

Kauffmann Mayeul

Lemoine Guido

Guest Editors

C. Corbane, M. Pesaresi and D. Carrion

European Commission, Joint Research Centre, Italy

Email: [email protected]

Notes

1. ISFEREA Action, GlobeSec Unit, Institute for the Protection, and Security of the Citizen.