Development and implementation of a validation protocol for crisis maps: reliability and consistency assessment of burnt area maps

Abstract

Validation is an emerging topic in the field of geospatial information for emergency response. In the framework of the Global Monitoring for Environment and Security initiative, the Services and Applications for Emergency Response project seeks to implement and validate the pre-operational version of Emergency Response Core Services. In that context, a dedicated effort has been made in the development and implementation of a comprehensive, product-oriented validation protocol tailored to crisis maps and data sets. This paper presents the approach for validation of crisis maps and its application in the context of rapid mapping of forest fires in Corsica during the summer of 2009. It also discusses the main challenges and the critical aspects of validation of geospatial information in emergency response services.

Keywords:

• validation protocol
• crisis mapping
• remote sensing
• forest fires

1. Introduction

Europe is making a significant effort to develop geo-information services for crisis management as part of the Global Monitoring for Environment and Security (GMES) program. The Services and Applications for Emergency Response (SAFER) project implements the ‘fast track’ Emergency Response Core Services (ERCS) as one of the priority developments in GMES. A number of services are developed and implemented by different European industries and research institutes, using a range of data sources, tools, and methodologies. In crisis situations resulting from a disaster, a major part of the developed services rely on satellite imagery as a main source of geo-information used to conduct rapid-mapping tasks. This is because, in recent years, the amount, timeliness, and availability of satellite imagery covering a certain crisis situation have improved substantially. Maps, geospatial information, and thematic analysis derived from satellite imagery can support decision making and situation awareness especially for large disaster scenarios and remote areas (Voigt et al. 2007). Service providers face a series of challenges related to the provision of timely geospatial information to the end users. In order to respond as quickly as possible, they have to rely on the available geo-information and to analyze it rapidly as well; the map production time should preferably be within 24 hours of the event impact and the major constraints are due to the satellite images acquisition. Since in emergency scenarios timeliness is often the most critical issue, the end user can decide to ingest the products even without having the possibility to assess their quality and accuracy. The lack of a proper assessment of the quality and soundness of the generated information was revealed to be a crucial shortfall when it came to information sharing across different users and its integration within Geographic Information Systems.

The main concern is that error propagation may result in flawed decision making, with consequences to local population and assets. Therefore, the need for an independent and formal assessment of the delivered products represents a critical issue in emergency response services.

In the SAFER project, a dedicated effort has been made in the development and implementation of a comprehensive, product-oriented validation protocol tailored to crisis maps and dataset products. The purpose of this protocol is to ensure that the services are delivered with similar quality, agreed standards and degrees of certainty, and within the specifications derived from the users’ expectations.

The objectives of this paper are: (1) to highlight the key concepts of the validation protocol for crisis maps and (2) to present and discuss the main outcomes of a test case in which the validation protocol was applied to rapid maps of forest fires in the South of Corsica during the summer of 2009.

2. The proposed approach to validation

Validation can be described as ‘a producer-independent process generating documented evidence to which degree the object subject to validation reaches pre-determined requirements.’

This synthetic definition deserves detailed comments:

•	Validation is producer-independent: validation provides documented evidence that a product and a service meet not only pre-determined levels of technical content and accuracy, but also most importantly fulfill users’ needs and expectations. In order to avoid being a self-referential activity, validation should be producer-independent. In that sense it differs from quality control, which is an internal producer process carried out by a quality management team without the involvement of the users.
•	Validation is a process: this implies that a systematic reproducible approach is defined to perform validation activities.
•	Documented evidence: this implies the definition of parameters, as much as possible quantitative, i.e. metrics that can be reported and analyzed in order to assess the performance of a product/service.
•	Reaches pre-determined requirements: validation is not an ‘absolute’ check. It is a relative check; it compares the object under validation to a reference level, in order to (1) justify the cost of it and verify that the characteristics of the product/service acquired are delivered as promised at the moment of the transaction and (2) minimize use problems (safety, misuse, etc.).

This means that clear statements, covering all user needs and requirements, are crucial for good validation practices. Requirements should be defined prior to the conduction of validation activities. Whenever requirements are missing or are not sufficiently detailed, reference standards, professional experience (best practice), and expert knowledge shall be used. The pre-determined requirements also imply that acceptable validity ranges and tolerance should be defined prior to the conduction of validation activities.

Validation is performed at the interface between the service provider and the end user. Both entities normally prefer that validation is performed by an entity that is independent from the interests of both the service provider and the end user. Validation requires in-depth knowledge of the service set-up, which may be commer cially confidential. Therefore, a validating entity should have no commercial interest in similar service development. At the same time, the validation entity must have a good understanding of the intended end-user service take-up, to ensure that service specification has been sufficiently detailed to meet the stated requirements.

3. Overview of the validation protocol

The aim of the protocol is to provide validation experts with a tool giving an ensemble of methods and procedures for verifying that the products meet users’ requirements. The protocol has an extensive approach that is based on the principles defined above. It was designed in the form of a generic mechanism that can be applied to geo-information products for emergency response independently of the nature of the disaster (Broglia et al. 2010).

The protocol considers different aspects within the process of geo-information generation and its delivery to the end users: (1) the reliability of the information content; (2) the consistency of the information support; and (3) the usability of the product. For each of these three aspects a dedicated approach has been developed.

3.1. Reliability of the information content

Reliability regards:

1.	The soundness of the methodology and of the sources used to generate the service. The assessment of the methodology can be seen as a science-driven verification process, which aims to evaluate the scientific feasibility of information production taking into account the proposed algorithms, methods or models, and the characteristics of the input data.
2.	The degree to which the information contained in a product is similar to a reference (also called ‘ground truth’), dependable or repeatable (stability), i.e. the degree to which it is free of errors, logical, and complete.

Reliability includes several attributes, e.g. thematic accuracy/consistency, spatial accuracy, spatial detail, and semantic definition. Depending on the type of reference data available, it is possible to assess either the accuracy, when the reference data are assumed to be more correct than the map (Congalton 2009), or the consistency, when only inter-comparison with another map is performed.

3.2. Consistency of the information support

A consistent product is the one that does not contain contradictions. While the ‘reliability of the information content’ looks at the degree of accuracy with respect to a reference, the ‘consistency of the information support’ focuses on internal contradictions of the product, between different components of the product (documentation included) or with respect to specifications.

The consistency of the information support comprises several attributes such as the relative positional consistency across different feature sets in the same product, the consistency between map and legend symbols, and repeatability across similar events.

3.3. Usability of the product

The usability of a product is contingent upon its appropriate use; that is through avoidance of misuse or erroneous use. It is directly related to the communication of the geo-information to others. This issue centers on the following questions: will the users find the geo-information product useful and informative? Is the data exchange format used for delivering the product fitting the specific use context? Using a geo-information product includes reading, interpreting, analyzing, and eventually integrating the information contained in the product. Therefore, it is crucial to eliminate any misunderstanding and ambiguities. Ensuring the usability and avoiding the problems of misunderstanding and misuse of a product require a collaborative approach between the user and the producer.

The first complete version of the SAFER validation protocol has been released in June 2009 and has been applied to a series of rapid maps produced for different crisis events, e.g. forest fires in Corsica (July 2009), floods in Turkey (September 2009), earthquake in Haiti (January 2010). In the following sections, an example detailing the application of the validation protocol to crisis maps of Corsica fires is presented and discussed.

4. Validation of Corsica burnt area maps

4.1. Presentation of the maps under validation

Between the 24 and 25 July 2009 three fires occurred in the region of Sartène (Southern Corsica, France, center coordinates: 41°46′18″N and 9°4′48″E) destroying 4200 hectares of vegetation and almost touching the touristic village of Aullène. The SAFER project was activated triggering the acquisition of a high-resolution satellite image (ALOS AVNIR-2; Advanced Visible and Near Infrared Radiometer type 2) 3 days after the disaster. Several satellite-derived geo-information products were generated and delivered to the users on 29 July 2009. Among the different rapid crisis maps generated for this event, the validation protocol was applied to two rapid crisis maps considered to contain the most relevant information: a map of affected urban areas and a map of affected forest cover. The authors wish to keep the Service Provider anonymous, since the aim of this paper is to discuss issues regarding the validation methodology. For this reason only subsets of the maps will be shown.

4.1.1. Map of affected urban areas (printing scale 1:15,000)

This rapid map (Figure 1a) has been produced by means of semi-automatic change detection and a comparison of a pre-event ALOS AVNIR-2 acquired on the 19 June 2009 and the post-event ALOS AVNIR-2 acquired on 28 July 2009. This enabled extracting the burnt areas, which are represented in red on the map overlaying the background pre-event satellite image. On the top left of the map, there is an enlargement on the Aullène village area. The data are represented in the metric French coordinate reference system (Lambert II étendu projection – Datum NTF) at 1:15,000 scale.

Figure 1.  (a) Subset of the map of affected urban areas (1:15,000); (b) subset of the map of affected forest cover (1:25,000). North direction is upward in both maps.

4.1.2. Map of affected forest cover (printing scale 1:25,000)

This map has been produced (Figure 1b), like the previous one, using the pre- and post-event ALOS AVNIR-2 satellite images. The perimeter of the burnt area is shown in red and, with two different shades of green, the separation between forest and maquis covers (typical Mediterranean vegetation) is represented. According to the map, the extents of affected forest and maquis covers are 18.7 km² and 9.3 km², respectively. The data are also represented in the metric French coordinate reference system (Lambert II étendu projection – Datum NTF) at 1:25,000 scale.

4.2. Validation methodologies and the related measurements

In this exercise, a subset of the validation protocol has been implemented. The focus was on reliability and consistency assessment with a particular consideration given to the evaluation of thematic accuracy and consistency and of positional consistency. The workflow implemented for the validation of the two crisis maps is illustrated in Figure 2. For each map under study a dedicated validation methodology has been employed with different reference data sources.

•	For the map of affected urban areas, the thematic accuracy and consistency have been evaluated following three different methodologies each one using a different type of input reference data including remote-sensing data (SPOT-4 post-event image) and in-situ data (perimeter of burnt areas and field observations/GPS locations). Since we are dealing with surface information (i.e. the burnt area), the error matrix and its derived metrics (overall agreement, omission, commission errors, etc.) have been used for measuring the thematic accuracy and consistency. Besides, simple comparisons of the total burnt areas have also been adopted for evaluating the thematic accuracy of this map.
•	Due to the lack of in-situ reference data, only the relative positional consistency of the map of affected urban areas has been evaluated. The commonly used horizontal root mean square (RMSE) has been computed as a measure of the relative positional consistency. Some general observations have also been made on the positional accuracy based on the resolution of the input images and on known cartographic standards.
•	The map of affected forest cover has been assessed with respect to its thematic consistency only, using one single type of reference data (manually digitized forest cover obtained from natural color ortho-photography) and a simple comparison of the total areas of forest cover as a validation methodology.

Figure 2.  Workflow of the validation methodology used for reliability and consistency assessments of the two crisis maps.

The following section gives more insights into the various types of reference data used, their processing steps, and their use within the different validation methodologies.

4.3. Reference data collection and processing

Different sources of reference data were used in this case study. They are summarized in Table 1 for each of the two maps under validation. The data set comprises remote-sensing images, a manually digitized forest cover map, in-situ data collected during the fire event by the Office National des Forêts (ONF), and post-event in-situ data collected during a dedicated field mission to Aullène from 21 to 24 September 2009.

Table 1. Reference data used for the validation of the two rapid crisis maps.

Reference data	Map of affected urban areas	Map of affected forest cover
Remote-sensing data
ALOS AVNIR-2 pre- and post-event images	X
SPOT-4 post-event image	X
Manually digitized forest cover
Two different layers performed by two analysts		X
In-situ data
Perimeter of burnt area	X
Field observations/GPS locations	X

4.3.1. Remote-sensing data

Two ALOS AVNIR-2 images were used by the service provider. The AVNIR-2 instrument is a multi-spectral sensor operating in four spectral bands in the Visible and Near Infrared bands, with 10 m spatial resolution and a ground swath of 70 km at nadir. For the validation exercise and more precisely for assessing the relative positional consistency of the delivered products, we used as input data, these two ALOS images, comprising the pre- and the post-event image.

A SPOT-4 image was also acquired on the 28 July 2009 and was made available for validation purposes. The High Resolution Visible and Infrared (HRVIR) instrument of SPOT-4 satellite has four channels operating at a spatial resolution of 10 m. The first three channels are in the Green, Red, and NIR ranges. The fourth channel lies in the spectral range of 1.58–1.78 mm, which is the short-wave infrared (SWIR) region of the spectrum, suitable for monitoring of forest fires.

The availability of a SPOT-4 image acquired on the same date as the ALOS post-event image allows the inter-comparison of burnt areas extracted from two sensors having the same spatial resolution but covering different spectral ranges. For the purpose of inter-comparison, the SPOT-4 image is first co-registered to the ALOS image. Then, to discriminate burnt from non-burnt areas, we implemented a semi-automatic approach; it is based on a combination of Support Vector Machines (SVM; Vapnik 2005), which is a supervised learning technique, and the unsupervised K-means classification algorithm. The method can be briefly described as follows:

1.	The K-means algorithm is initially used to cluster the pixels.
2.	The K sets are then eroded leaving all the pixels which are close to the center of the connected pixels sets. The latter represent the training sets used for SVM classification.
3.	The SVM with a Gaussian Kernel is then run with the training set chosen automatically by the K-means algorithm.
4.	The operator labels the resulting clusters as burnt and non-burnt.
5.	The mask of burnt area is then derived from the result of SVM classification using simple manual thresholding.

4.3.2. Digitized forest cover

For the validation of the map of affected forest cover, we focused on the forest cover included in the perimeter of burnt areas. The reference data, in that case, were obtained from on-screen digitization of the forest cover, generated separately by two image analysts using Google Earth. The satellite imagery displayed on Google Earth corresponds to a natural color 0.5 m resolution aerial ortho-photography from IGN France of 5 March 2007. The map of the affected forest cover has been geo-referenced over the Google imagery, in order to insure a minimum displacement with respect to the manually digitized forest cover. This allowed deriving two estimates of the total area of burnt forest cover and to perform a comparison with the amount of burnt forest cover declared on the map under validation.

4.3.3. In-situ data

Thematic accuracy can be evaluated comparing data to an alternative information source that can be considered as ‘the truth,’ or at least a reliable reference (Congalton 2001). Actual ground visits are usually considered as the most reliable method for reference data collection. For this validation exercise, in-situ data, gathered from two different sources, were used: (1) a mask of burnt areas provided by ONF and (2) a series of observations/GPS locations collected from field sampling.

ONF participated to the fire brigade during the active fire with the purpose of performing ‘real-time’ cartography (Duplaa 2009). This resulted in a rough emergency assessment of the perimeter of the global burnt area based on the foresters’ experience, their knowledge of the area, and the site accessibility.

Between 21 and 24 September 2009, a field campaign to the Aullène village was organized by the validation team. The purpose of this mission was to collect field data for checking the thematic accuracy of the rapid maps. This mission was also an occasion for the validation team to meet with the users of the rapid crisis maps (i.e. ONF in Corsica) and to get their feedback on the maps as well as their expectations with regard to the quality and usefulness of the delivered maps.

The challenge, during this field mission, was to design a cost-effective sampling scheme suitable for investigating a relatively large burnt area (around 28 km² according to the map estimates).

For thematic accuracy assessment, a stratified sampling design was adopted as it allows sampling densely some areas of interest taking into account field constraints, such as the accessibility to point locations. Accordingly, three strata with the following characteristics were manually selected on the basis of a topographic map (1:25,000 scale):

Stratum 1: combines several characteristics on a relatively small area (e.g. presence of rocks, of forest, and hilly terrain), besides most of this stratum can be easily accessible by road.

Stratum 2: combines an urban area with forest and shrub areas. This stratum is particularly interesting for validating the impact of the fires on the urban area of Aullène.

Stratum 3: corresponds to the remaining burnt area.

For the sample unit, a cluster of 5×5 pixels (corresponding to an area of 50×50 m) was selected, given the need to balance between thematic and positional accuracy (see Section 5.1).

As for the selection of the sample size, since we are facing a binary problem (burnt and non-burnt), it was necessary to equally sample these two classes. Besides, given that we are mainly interested in Strata 1 and 2, it was decided to concentrate the sampling effort in these two areas by sampling them more densely. Hence, taking into account both statistical constraints for thematic accuracy assessment and time constraints, a minimum of 20 sample units were randomly selected per strata leading to a total of 60 samples for the whole study area.

5. Results of the validation

5.1. Positional accuracy and consistency assessment

In this section, the validation results for the map of affected urban areas are considered, since it contains the most detailed information (1:15,000 scale with respect to 1:25,000).

Regarding the positional accuracy some fundamental considerations need to be mentioned. According to scientific literature (Amhar and Ade Komara 2009, Uchiyama et al. 2008) ALOS PRISM/AVNIR images are designed for the mapping and revision of 1:25,000 scale topographic maps, in particular thanks to the PRISM panchromatic images with a 2.5 m spatial resolution. In this case study, AVNIR 10 m resolution images were the only available images and it is straightforward to infer that this spatial resolution does not allow meeting the accuracy of 3 m required by cartographic standards for the 1:15,000 map scale (American Society for Photogrammetry and Remote Sensing (ASPRS) Specifications Standards Committee 1990).

In addition to the check of positional accuracy, the relative positional consistency has been considered. It refers to the consistency between different layers inside the map. In this case, the layers that have been taken into account are the background pre-event image and the polygon of burnt areas that was extracted from the post-event image. For the assessment of the relative positional consistency, the identification of tie points was made on the pair of pre- and post-event ALOS images. A total of 30 points have been considered around the burnt area. These points have been extracted partially by an automatic algorithm (using ENVI® software) and partially by photo-interpretation. The computed RMSE was 47 m. This low relative positional consistency raises a question on the inconsistency between the procedure adopted for geo-coding and the declared nominal scale of the output. It highlights the problem of availability and access to accurate and up to date spatial data for geometric correction of high-resolution satellite data, especially in high relief areas as in the Aullène valley.

5.2. Thematic accuracy and consistency assessment

Thematic accuracy and consistency were assessed for the two rapid crisis maps. We first present the results for the map of affected forest cover analyzed with respect to the manually digitized forest cover. Then, we present the results of accuracy/consistency assessment of affected urban areas analyzed using two different reference sources, including a SPOT-4 image and in-situ data.

5.2.1. Map of affected forest cover

Table 2 allows the comparison of the total area of forest cover declared on the map to the reference areas obtained by image analysts #1 and #2 using photo-interpretation. The map under test does not declare the applied definition of forest cover. We tried to cope with the lack of a semantic definition by asking the two analysts to apply a different definition of forest cover: for both the forest cover corresponds to a land with tree crown cover greater than 30% (Di Gregorio and Jansen 2005), then analyst #1 was asked to include low and dense vegetation in the forest cover, whereas analyst #2 was asked to exclude it from his visual interpretation. The value reported on the map (18.7 km²) is in between the two analysts estimates (20.0 and 18.0 km²). We may then conclude that the map provides a good approximation of the total area of forest cover although the latter was derived from a 10 m resolution ALOS image and in a rapid-mapping context.

Table 2. Assessment of the total area of forest cover based on visual interpretation of Google Earth data and performed by two analysts; comparison of the total burnt areas obtained from remote-sensing data to the total burnt area estimated by ONF.

Total area of affected forest cover (Km^2)
Map	18.7
Analyst #1	20.0
Analyst #2	18.0
Total burnt area (Km^2)
ALOS AVNIR-2	28.0
SPOT-4	30.0
ONF	30.0

5.2.2. Map of affected urban areas

Both thematic accuracy and consistency assessments of the map of affected urban areas were undertaken following different approaches depending on the type of reference data.

5.2.2.1. Inter-comparison with reference to SPOT-4 image

As previously described in Section 2.4, the SPOT-4 post-event image was used to extract a mask of burnt areas. This allows performing inter-comparison of burnt areas extracted from ALOS and SPOT-4 data. Product inter-comparison only allows the examination of consistency (or relative differences) among similar products, but cannot tell which one is the better one. The error matrix was computed using the two masks of burnt areas. Table 3 shows the results of the inter-comparison and the values of consistency measures obtained from the error matrix, considering SPOT-4 results as a reference.

Table 3. Inter-comparison of the masks of burnt areas derived from SPOT-4 (reference) and ALOS images.

Inter-comparison of burnt areas extracted from SPOT-4 (reference) and ALOS (%)
Overall agreement	Commission error	Omission error	Producer accuracy	User accuracy
86.89	0	13.1	86.9	100.0

The consistency measures and Figure 3 show a good agreement between the two products (overall agreement of 87% considering the SPOT-4 as a reference) and demonstrate that ALOS AVNIR-2 instrument is suitable for delimiting the burnt areas even in the absence of the SWIR band. Bearing in mind that the rapid map was produced in a very short time frame (5 hours after image re ception), we may conclude that this product is suitable for a fast estimation of the affected areas.

Figure 3.  Perimeter of burnt areas derived from SPOT-4 (yellow outline) and ALOS AVNIR-2 images (red outline).

5.2.2.2. Comparison with reference to Office National des Forêts (ONF) perimeter of burnt area

In addition to the error matrix, a comparison of the total amount of burnt areas obtained from SPOT-4 and ALOS was also performed, using the estimates made by ONF as a reference. Table 2 shows the closeness of the estimated values from remote-sensing data (28 km² from ALOS against 30 km² from SPOT-4) and, most of all, the matching between the total burnt area, obtained from SPOT-4, and the reference amount provided by ONF (30 km²).

5.2.2.3. Comparison with field observations/GPS locations

The error matrix was also computed based on reference in-situ data. Both burnt and non-burnt thematic classes were considered for accuracy assessment. Unfortunately due to time and accessibility constraints, only 38 sample units were checked on the field instead of the 60 planned ones. We remind that each sampling unit corresponds to an area of 50×50 m which is equivalent to a square of 5×5 pixels on the 10 m resolution image. Table 4 represents the result of thematic accuracy assessment with the pixel-based error matrix. Both the overall accuracy value (87%) and Kappa (0.7) indicate that the burnt areas extracted from ALOS AVNIR-2 image correctly capture the real burnt areas observed in the field. We can note that all the non-burnt areas were identified as such on the rapid map. The main confusion was between some areas identified as non-burnt on the map whereas they actually corresponded to burnt areas. The latter presented some ambiguities when checked in the field; this is because the understory vegetation and part of the tree trunks were burnt while the tree crowns remained green as visible in Figure 4. According to the foresters from ONF, these areas are considered as burnt because the affected trees may not survive until the next season. The ambiguities in detecting partially burnt areas could be related to: (1) the sensors characteristics and (2) the classification approach.

1.	Near vertical looking satellite sensors (both ALOS and SPOT-4) will not detect partial damage in densely vegetated areas.
2.	The adopted classification approach takes into account only burnt or non-burnt areas. Partially burnt areas are not taken into consideration even though they might be detectable in case of sparse vegetation.

Figure 4.  Ground photographs taken in situ and highlighting the limitations of nadir looking remote-sensing data for the detection of understory burns.

Table 4. Error matrix computed with the 38 surveyed sample units. Each unit is representative of 50×50 m² (or 5×5 pixels).

Reference	Burnt	Non-burnt	Total
Map
Burnt	23	0	23
Non-burnt	5	10	15
Total	28	10	38
Overall accuracy	86.84%
Kappa	0.7

6. Discussion

This case study illustrates the application of the validation protocol on rapid maps derived from remote-sensing data and allows highlighting the main challenges related to the validation of geo-information for crisis management. These challenges can be summarized in three points.

6.1. Availability and suitability of reference data

Validation of crisis maps should be typically performed as soon as the maps are released, this means, e.g. in the first week after the disaster. This allows a fast revision of the products, their update, and the release of early revised versions to the end users. Timely acquisition of reference data for validation is as important as the fast acquisition of input satellite data for rapid mapping. For some types of events, like in the occurrence of floods, the timing of reference data collection is crucial in determining the flood extent and should be as close as possible to the overpass time of the satellite. Inversely, in the case of damage assessment maps following an earthquake or a forest fire, reference data can be collected once the initial rescue and relief efforts are over and when access to the affected areas is restored. Besides, in complex emergency situations, it is often impossible to access the affected zone because of obvious security reasons. Independent data collection may be problematic. In that case, national or local authorities can play an important role in attempting to collect valuable information that can feed into the independent validation process. In this case study, we benefited from the availability of a wide range of reference data including in-situ and satellite data. This is not very common, since in most validation cases, a practical choice must be made between the use of ground visits or very high-resolution remote-sensing data (spatial resolution below 5 m) as a reference source for validation. Both photo-interpretation and automatic information extraction methods were used for deriving the reference geo-information from remote-sensing data. However, the level of accuracy of these approaches is limited, because photo-interpretation may contain errors depending on the skills of the photo-interpreter and the automatic extraction can generate systematic errors as well. Therefore, actual ground visits remain the most reliable method for validation despite the problems that may be encountered during the field data collection as in the case of Corsica fires:

•	difficulties in accessing sampling areas due, e.g. to the highly steep terrain or to the thick vegetation cover;
•	difficulties in classifying some areas into burnt or non-burnt without the help of an expert; and
•	difficulties in systematically checking the perimeter of burnt areas, because of accessibility constraints and the large extent of the affected area.

As a suggestion for future validation exercises, for increasing the efficiency of field data collection, in addition to GPS devices, other technical solutions can be used such as a laser distance meter connected to the GPS for the acquisition of coordinates of remote targets.

6.2. Availability of service/product specifications and user requirements

In ideal cases both service/product specifications and user requirements should be clearly defined and documented to allow validation deciding on the product's fitness for purpose. However, in most cases they are not available or they are incomplete. A solution in the case that the information is not complete is to refer to standards, for example cartographic standards, or to scientific references or to common practice.

In this case study, neither specifications nor user requirements were available. Instead, it was possible to collect feedback on the delivered products from a single end user (ONF) and only after the event. For future validation exercises, it is preferable to contact a wider set of potential end users to define, beforehand, both general and specific requirements. Such a definition should be made from the perspective of the end user's field of expertise, rather than from a satellite capacity and service definition perspective. For example, in the case of burnt area mapping, the collected end-user requirements may cover those from the fire brigades (e.g. fire status reports and fire fuel maps), civil protection (e.g. risk to settlements, road obstruction, and routing), forest service (reforestation needs), and crisis rooms (coordination and communication). Each of those end users may have varying requirements with respect to accuracies, update frequency, and product formats. Value-added products from Earth Observation resources can then be better tailored to the various (partial) needs of those organizations. Furthermore, such a joint analysis would help define the limits of applicability of current satellite remote-sensing capacities, and contribute to discussion on future options for space assets.

6.3. Cartographic standards, the map accuracy, and usability

Following this exercise it was clear that cartographic specifications (Section 5.1; American Society for Photogrammetry and Remote Sensing (ASPRS) Specifications Standards Committee 1990) are sometimes ignored in rapid crisis maps. In fact, according to these standards, with the 10 m spatial resolution image used to produce the ‘Map of affected urban areas,’ the expected scale could have been 1:50,000 instead of 1:15,000 scale declared on the map. Feedback from the ONF end users on the delivered map showed that they do not require an accuracy of better than 100 m, and that they were satisfied with the actual positional accuracy of the map. Although this may not be crucial for ONF, an incorrectly declared scale and consequently overestimated map accuracy can increase the cost of the ingestion of this information in integrated GIS environments used in crisis rooms in support to disaster management.

It is therefore recommended to give more attention to the map specifications, mainly the scale declaration in order to avoid confusion or possible misuse of the product when used in a different context and in more complex situations requiring integration of heterogeneous layers of data. A practical suggestion is to declare the printing scale and the nominal scale, i.e. the scale of processed data.

6.4. Validation ranges

The possible disagreement between cartographic specifications and users’ needs raises a problem about validation of crisis maps. Knowing that the ultimate purpose of validation is to guarantee that the user receives a product fitting his needs, it is sometimes necessary to soften and combine by multi-criteria approach the existing rigid standard specifications in the final assessment of a product. One way to do so is to provide the validation expert with an instrument which allows him to concentrate on what is crucial for the user. This is the idea behind ‘validation ranges.’ This concept helps to formally rank the products by assigning to each validation parameter an acceptance range, based on the trade-off between quality and timeliness or other trade-offs selected by the user, i.e. between cost and benefit. Together with other criteria, validation ranges can help in identifying critical problems in a crisis product and in suggesting recommendations for improvement of some aspects of a rapid-mapping service. Users benefit from validation results through the enhanced trust in products or reliance on services that successfully pass validation and the gradual improvement in the service evolution. This would, however, require an a priori definition of products’ specifications and users’ requirements. Validation ranges are currently being developed based on several consultations with the end users (e.g. EU civil protection, agencies of the United Nations, etc.). It is planned to integrate them in the next release of the validation protocol.

7. Conclusion and outlook

This paper has presented an overview of the validation protocol designed for rapid crisis maps. It also summarized the methodology and the results related to the application of this protocol to the case of forest fires. The focus was on the assessment of reliability and consistency of the maps due to their importance for decision making in disaster management. The two maps under validation showed a good overall quality, with minor detected problems. Taking into account time constraints, the extent of the affected area and the inherent limitations of remote-sensing data, the thematic accuracy and consistency of the final products are high. Inversely, spatial accuracy and spatial consistency were problematic from a cartographic point of view resulting in potential increase of cost for data integration.

The extensive approach to validation, applied in this case study, allowed experimenting with the main aspects of the validation of crisis maps, including image processing, photo-interpretation, field campaign, and assessment of users’ needs. From this example, it is possible to point out some critical aspects related to the validation of geo-information products for crisis management: (1) validation approaches, including timing, input reference data, and methodologies, need to be adapted to the type of disaster; (2) validation is a sensitive task since it involves different stakeholders including service providers, end users, and validation experts; (3) validation requires a good number of professional competencies, e.g. EO, GIS, image processing, photo-interpretation, cartography, statistics, eventually field data collection, etc. In addition to these, sector-dependent professional competencies, e.g. forest science, hydrology, etc. could be needed as well; and (4) the validation protocol is a useful tool for the definition of rules to ensure unbiased and independent validation. The existing protocol is a live document that is being regularly updated and consolidated based on various practical experiences.

Notes on contributors

Christina Corbane received a PhD degree from the University of Montpellier II (France) in 2006. Her research work was conducted at the Laboratory for the study of Interactions between Soils, Agrosystems, Hydrosystems (LISAH) Joint Research Unit on automatic detection of soil surface characteristics by remote sensing at Very High Spatial Resolution. She received two post-doctoral fellowships from the Institute of Research for Development for conducting research in the framework of two EU projects SOWAMED and LIMES/GMES. Besides her interests in the application of Remote Sensing to natural resources management, she worked on algorithm development for maritime surveillance and for urban areas monitoring by satellite imagery. Currently she is working at the EC Joint Research Center on the development of validation methodologies for crisis mapping in the framework of SAFER/GMES project.

Daniela Carrion received in 2003 a PhD degree on ‘Surveying and Geodesy’ from Politecnico di Milano, where she has worked, as research fellow, until August 2009. The PhD Thesis was focused on the automatic classification of remotely sensed images and during that period she was involved in two projects financed by the Italian Space Agency (ASI) about the production and assessment of thematic maps form satellite imagery. Then her research field enlarged toward Cartography and GIS. In particular, she was involved in projects co-financed by the Italian University Ministry concerning HR images registration and GIS quality standards. She also contributed to the GIS prototype planning and implementation of some local municipalities and to the validation of local base maps. She contributed to the computation of the Italian geoid (Italgeo05) and also to the computation of some other Countries’ geoids with the International Geoid Service (IGeS) organization. In 2009, she joined the EC Joint Research Center, in the framework of SAFER/GMES project, for crisis-mapping validation.

Marco Broglia is an electronic engineer and obtained his first-class honors degree from Politecnico di Milano in 1995. He pursued research activities in the same university, involving environmental and transportation modeling, decision support systems, computer architectures, and combinatorial optimization. He entered the GIS office of Provincia di Milano in 1998, working on GIS data design, analysis, and quality. In 2000, he was appointed as the responsible of the GIS office, and led several projects involving databases and maps production, topographic database production, decision support system design, and development. He was also responsible of coo rdinated GIS activities with other administrations. Occasional lecturer and business tutor with Politecnico di Milano and LIUC Castellanza. Since November 2008, he works at the EC Joint Research Center, Institute for Protection and Security of the Citizen (IPSC), Global Security and Crisis Management, ISFEREA team, focusing on GIS and SDI.

Acknowledgements

The Authors wish to thank: SERTIT and ONF (Office National des Forêts) for their helpfulness and their contribution to the field work; Lauren Rizzo for the photo-interpretation and Andrea Gerhardinger for helping to prepare the field campaign. This work has been supported with a grant from the European Commission's FP7-SPACE research and development project SAFER (grant no. 218802).