The independent service validation in GMES RESPOND: the flood validation exercise

Abstract

This article is aimed at providing a detailed description of the Flood Validation Exercise organised by the Independent Service Validation Group of the Global Monitoring for Environment and Security (GMES) RESPOND project. The aims of the validation exercise were: (1) providing a practical example of validation procedures in the frame of the GMES Emergency Response services; (2) executing a full-scale validation exercise able to cope with the requirements of an emergency service; and (3) better understanding the performances and limitations of Earth observation services for Flood Damage Mapping. This validation exercise is a first step of the main task to define the whole validation process for GMES services. When this is achieved, there will be knowledge concerning how well services meet the service specifications derived from the user needs. The present exercise has the purpose of gathering this knowledge. The output of this validation exercise can be used to characterise and qualify the performance and timeliness of Crisis and Damage Mapping Services. This paper summarises the methodology for the flood exercise validation and the results of product validation and inter-comparison.

Keywords:

• flood mapping
• validation exercise
• product validation
• inter-comparison

Introduction

RESPOND is a Global Monitoring for Environment and Security (GMES) Service Element (GSE) project, which addresses Earth observation (EO) services development in support to emergency and crisis response, for humanitarian emergencies and natural disasters response (RESPOND 2010). The Independent Service Validation Group (ISVG) is an initiative to design a global validation framework and methodology based on independent experts for Emergency Response services, including the realisation of validation exercises. A Flood Validation Exercise was designed and conducted by the ISVG, in collaboration with Service Providers (SPs) from Europe and the United Nations (UN) with the goal of increasing the trust and acceptability of EO-derived products at hand by assessing the performances of the delivered products and the timeliness of product delivery. The validation exercise involved a two-fold technical validation:

•	Product validation, which involves the comparison of the product of a given SP against ground truth or reference data.
•	Product inter-comparison, which involves the cross comparison of the products coming from different SPs, without involving the use of ground truth. This can be used to check the degree of consistency between products generated over the same event.

This paper summarises the methodology for the flood exercise validation, the product validation results and product inter-comparison results.

Approach for the Flood Validation Exercise

This section describes the most important components of the Flood Validation Exercise, the required products/services, the selection of the site and event to be analysed, the inputs and outputs of the validation exercise, and the validation and inter-comparison procedures.

Required products/services

Starting from the RESPOND service specifications there are six types of services for flooding. At the beginning of the exercise, five possible types of crisis mapping products were defined: water body extent (at flood observation time); flood water extent (observed flooded waters); temporal evolution of flood water extent (multiple observation times); detailed impact assessment; and flood traces extent (comprehensive flood traces). For the scope of the full-scale exercise, two of them have been selected:

•	Crisis mapping/water body extent: hazard map, which typically comprises a single information layer called ‘water surfaces’, where both flooded areas and other in-land water bodies (for instance rivers, lakes, etc.) are mapped. This quick product does not differentiate flood waters from existing water bodies and provides no additional information about pre-crisis water levels.
•	Crisis mapping/flood water extent: hazard map corresponding to a given observation time, providing the extent of the observable flooded areas. They are typically space maps of the affected area at the observation time (Wood et al. 2002, Silander et al. 2006).

Selection of the site and event to be analysed: a full-scale historical event

To perform a successful validation exercise it is particularly important to pay attention to the selection of an appropriate case study (van Oosterom et al. 2005). The criteria for the selection were as follows:

1.	It should be a past and real ‘historical’ event.
2.	It should be an emergency event that is within the scope of GMES Emergency Response services.
3.	At least one good reference data set (ground truth) should be available to the ISVG and should be ‘undisclosed’ to the SPs.
4.	It must be an event for which EO-based services should prove useful. The mapping/damage mapping services must be feasible from a technical viewpoint.
5.	None of the SPs involved in the Validation Exercise should have been involved with the concerned event.

The 1997 Red-River flood event that took place in the ‘Red River of the North’, in North Dakota and Minnesota (USA) and Manitoba (Canada) during the period 3–30 April 1997, was selected for the validation exercise. The 1997 flood arose from a number of factors: record number of snowfalls during the winter; heavy early-spring precipitation; and coincidence of tributary peaks with main stem peaks. The emergency dikes in Grand Forks, North Dakota, and Ste Agathe, Manitoba, failed causing widespread damage. The inundation covered in total an area of 25,000 km² and caused 3.5 billion US$ of damaged cost. At least 103,000 people, 75,000 in the USA and 28,000 in Canada, were evacuated (Rust Environment and Infrastructure, Inc. 1999).

The chosen area has interesting characteristics from the point of view of remote sensing based flood monitoring: the area of interest (AOI) is quite large (116,500 km²), it is covered by vegetation, ice and urban areas and some part of the area topography causes shadowing and layover artefacts in the SAR scenes. However, this is not a typical site of flooding in dense urban land.

Inputs and outputs of validation exercise

In order to run the exercise, the required products, the EO data sets to generate them and their delivery times were specified. The flood mapping products were requested for two T-dates: the 24 April 1997 (T1); and the 28 April 1997 (T2). For each T-date, the product types required were as follows:

•	Water surface areas: ArcGIS shapefile and ancillary files containing water surface areas as polygon features.
•	Flood coarse crisis mapping: vector files of the flooded areas at times T1&2 and for Area of Interest AOI-T1 and AOI-T2 (ArcGIS shapefile and ancillary files containing flood surface areas as polygon features). In addition, a raster file that is the superimposition of a background EO image (geo-referenced SAR image) and the class ‘flooded areas’ was requested.

The EO data set to run the exercise were assembled to meet the requirements for the flood impact mapping. Two pairs of RADARSAT-1 SAR data (Figure 1; crisis image and pre-event data) and a series of pre-event optical data (SPOT XS, PAN and Landsat TM) were collected (see Figure 2)The flood extent maps were derived by exploiting the contrast in backscattering between land and water features of SAR data sets; while landcover information was derived using optical data as SPOT and TM images. For the exercise, the following time constraints had to be kept: the T0 (time at kick-off) at 12 October 2009, 13:00 h, the crisis window time at from T0 to T0 + 5 days, and the delivery of SP final operations report at T0 + 7 days. The flood mapping products for T1 and T2 were expected as soon as possible, within a few hours and, as a target, within 48 h from T0.

Figure 1.  RADARSAT-1 images of: (a) pre-event AOI-T1 16 June 1996; (b) flood event AOI-T1 24 April 1997; (c) pre-event AOI-T2 13 June 1996; and (d) flood event AOI-T2 28 April 1997.

Figure 2.  Footprints of the EO data sets and AOIs available for the exercise.

For the exercise a website was available for uploading the requested products and for downloading the EO data set, relevant documents and templates. Six SPs participated in the validation exercise: five (public or private) value-adding organisations from Europe and one from the UN. It was agreed with the UN SP that they would take part on a best effort basis, and the constraints on service timeliness would not apply to them.

Validation procedure

Ground truth data sets consisted of inundation extents produced by the Hydraulic Engineering Department of the US Army Corps of Engineers, St. Paul District. The validation inundation extents were based on air-photos surveys that were taken over the region of interest on the 24 April 1997 and the 28 April 1997 (approximately at mid-day time). The flood photos were georeferenced to Digital Orthophoto Quadrangles (DOQs) obtained from USDA Aerial Photography Field Office. A minimum of six reference points per photo were selected for georeferencing with a maximum error of 0.2 m. The inundation extent was digitised by photo-interpreter and includes all the areas that are covered by water excluded permanent water bodies (lakes, rivers, surface waters of swamps, marshes, bogs and fens).

The validation involved three main steps:

1.	Confusion matrix, computing in particular the overall accuracy (AC), false positive (FP), false negative (FN), true positive (TP), true negative (TN), precision (P) and Kappa (Kohavi and Provost 1998, Cohen 2001).
2.	Missing and extra areas – in this work the Extra areas are the pixels that were mapped as flooded in the delivered product and not flooded in the validation data set. The Missing areas are the pixels that were mapped as not flooded in the delivered product and flooded in the validation data set. The calculation was made using ArcInfo and a design MATLAB code.
3.	Geometric offsets – determination of the geometrical linear shift (offset) between the flood products delivered by the SPs and the ground truth. The ground truth consists of two ortho-photo mosaics composed from air photos taken during the flooding. Based on the offsets in X and Y direction, basic statistics have been calculated.

Inter-comparison procedure

The inter-comparison analysis was performed over the same parameters analysed in the validation procedure as described above. Three inter-comparison analyses were performed:

•	The inter-comparison can be used to check if the products generated over the same event are consistent. Even though one cannot pretend to have identical products, one may expect that the differences between them are within given limits. This is important for a project like RESPOND, where the user may potentially get the same service from different providers.
•	Identification of possible weak points in the products from one or more providers.
•	Computation of inter-comparison statistics, which can be a valuable source of complementary information with respect to the outcomes of product validation.

Validation results – flood extent map service

In the following sections the validation results for the flood extent products (named T1.2 and T2.2), provided by the SPs for AOI-T1 (24 April 1997) and AOI-T2 (28 April 1997) are reported. The results shown in this paper are anonymous and thus the SPs are mentioned using arbitrary names: SPC; SPH; SPL; SPO; SPR; and SPZ. The georeferencing of the products delivered by the SPs was mainly obtained using GIS available layers and semi-automatic methods, while the flood extent maps were produced using automatic methods.

Thematic validation – confusion matrix and timeliness

The overall accuracy results for the flood extent maps T1.2 and T2.2 are shown in Tables 1 and 2, respectively. The overall accuracy obtained for flood extent products ranges from 80.8 to 87.6%: there is a good agreement between the delivered products and the ground truth for the two considered AOI. This result is in line with the results of other RADARSAT flood mapping validation studies published in the literature (Pietroniro and Martz 2001, Townsend 2001, Suwanwerakamtorn and Salakkham 2006).

Table 1. Confusion matrix results for T1.2 product – flood mapping product for the 24 April 1997.

	SPC	SPH	SPL	SPO	SPR	SPZ
AC	80.8	83.2	83.3	81.7	81.8	81.0
TP	77.1	81.3	82.1	79.6	77.5	80.3
FP	11.8	13.0	14.4	14.1	9.6	17.4
TN	88.2	87.0	85.6	85.9	90.4	82.6
FN	22.9	18.7	17.9	20.4	22.5	19.7
P	63.6	65.1	65.7	65.0	63.1	66.0
Kappa	0.60	0.64	0.64	0.61	0.62	0.60

Table 2. Confusion matrix results for T2.2 product – flood coarse mapping for the 28 April 1997.

	SPC	SPH	SPL	SPO	SPR	SPZ
AC	87.3	84.6	86.8	87.6	85.8	86.3
TP	86.6	83.3	90.2	85.6	85.3	84.5
FP	11.6	13.6	18.2	9.4	13.5	11.0
TN	88.4	86.4	81.8	90.6	86.5	89.0
FN	13.4	16.7	9.8	14.4	14.7	15.5
P	58.9	58.5	61.7	58.0	59.1	58.1
Kappa	0.74	0.69	0.72	0.75	0.71	0.72

Figures 3 and 4 show the timeliness and accuracy for products T1.2 and T2.2. Over the first test area (T1.2) 66% of the products were delivered in l ess than 10 h from T0 (one at 5.5 h, three at 7.5 h and one at 11 h), and all products were delivered within 28 h. Similar performances were achieved in the second test area (T2.2): 50% of the products were delivered in less than 12 h, and all of them within 28 h. The entire AOIs of T1.2 and T2.2 were covered with flood maps at best within 9 h and at the worst within 49 h. This is in agreement with the publication of SERTIT in the study of more than 70 activations of the International Charter Space and Major Disasters (Allenbach et al. 2005, Bally and Boubila 2009).

Figure 3.  Accuracy (%) and Kappa vs. delivery time (in hours) for T1.2 product – flood coarse mapping for the 24 April 1997.

Figure 4.  Accuracy (%) and Kappa vs. delivery time (in hours) for T2.2 product – flood coarse mapping for the 28 April 1997.

Thematic validation – extra and missing areas

From Tables 3 and 4 (T1.2 and T2.2, respectively) one may notice that extra areas are smaller in comparison to the missing ones. The locations of missing and extra areas are shown in Figures 5 and 6. The figures present similar patterns for all SPs: the missing areas are mainly located along the Red-River stream and its tributaries, while the extra areas are mainly in the east and west borders of the image and along the emergency channels.

Figure 5.  Missing (red) and extra (blue) areas mapping for product T1.2 – flood mapping for the 24 April 1997.

Figure 6.  Missing (red) and extra (blue) areas mapping for product T2.2 – flood mapping for the 28 April 1997.

Table 3. Missing and extra areas for T1.2 product – flood mapping for the 24 April 1997.

	SPC	SPH	SPL	SPO	SPR	SPZ
Extra area (%)	3.9	4.3	4.8	4.7	3.2	5.8
Extra area (km^2)	28.476	31.367	34.837	34.114	23.273	42.064
Missing area (%)	15.3	12.4	11.9	13.6	14.8	13.1
Missing area (km^2)	110.219	89.910	86.224	98.149	107.184	94.969

Table 4. Missing and extra areas for T2.2 product – flood mapping for the 28 April 1997.

	SPC	SPH	SPL	SPO	SPR	SPZ
Extra area (%)	4.7	5.5	7.4	3.8	5.5	4.5
Extra area (km^2)	35.111	40.988	54.973	28.491	40.914	33.252
Missing area (%)	8.0	9.9	5.8	8.6	8.8	9.2
Missing area (km^2)	59.139	73.793	43.145	63.825	65.090	68.438

The regularity in the patterns of the missing and extra areas suggests that the reason behind these mapping errors is related to the scattering behaviour of the SAR data. A large aerial photo view of the 1997 flood (Figure 7) in the area of Drayton reveals that all along the meanders of the Red River there are forest areas and their canopies are out of the water. Verification of the presence of forests along the Red River and its tributaries was done also using the aerial photos (Figure 8b) collected by the US army during the flood period. Zooming at the actual resolution of the air-photos (2 m) reveals that the forest areas were flooded (Figure 8d). From the above results it is clear that systematic errors in mapping extra and missing areas are related to limitation of the EO-SAR data. In order to overcome this limitation, state-of-the-art studies suggest improving the SAR-based flood mapping by combining the SAR data with opti cal and/or DEM data (Imhoff 1997, Townsend and Walsh 1998, Alsdorf 2001, Schumann et al. 2007). The ISVG tried to assess the usability of SRTM DEM data to further improving the flood mapping performances. Figure 9 shows the correction of one of the flood results, obtained by combining the SAR data and a SRTM DEM. One can see in Figure 9a that the flood plane area along the meanders of the Red River was classified by Service Provider ‘C’ (SPC) as flood. This area, as other areas along the meanders, is covered by forest (see Figure 8b and 8d). By comparing 9a and the DEM classification in Figure 9e, one can see that areas with altitude between 227 and 239 m were classified as flooded. The SRTM DEM data (Figure 9d) reveal that there are some hills along the Red River and its tributaries; however, areas lower than 239 m (Figure 9e), which were classified as flooded (Figure 9a) are also located near the river and its tributaries. This means that the limitation of the SAR data in forest areas might be only partially corrected using DEM data.

Figure 7.  Aerial photo view of the 1997 flood.

Figure 8.  Mapping of missing areas due to forest presence along the Red River: (a) Radarsat-1 product of 24 April 1997; (b) aerial view; (c) mapping of the same zone as missing areas; and (d) enlarged view of the high-resolution image shows that the forest surfaces were covered by water.

Figure 9.  Correction using combined SAR and DEM: (a) SPC flood product; (b) validation product based on ground-truth data; (c) SRTM DEM (2D); (D) SRTM DEM (3D); (e) SRTM DEM classification; (f) correction of SPC flood.

Geometric validation – geometric offset

By comparing the geospatial flood products (T1.2 and T2.2) with the reference ortho-mosaic, potential geometrical shifts reflecting positional errors (Guptill and Morrison 1995) were investigated. The geometrical agreement of the vector and raster products was measured using 18 validation points for each AOI, estimating the offsets in the X and Y directions. Some basic statistics and the spatial distribution of the shifts were analysed. As a rough estimation, given the spatial resolution of the original SAR images (12.5 m), a positional accuracy between 10 and 15 m was expected. Table 5shows the statistics of the geometrical shifts for the vector products. In general, there are considerable variations among the results obtained by the different SPs, both in magnitude and patterns of shifts. By evaluating the spatial distribution of the shift, no specific directions or patterns were found, e.g. see Figure 10. Table 6 shows the same type of results for the raster data products. Again, one can observe large variations between the results obtained from the different SPs. In general, mean-x and mean-y shifts are larger in comparison to the vector results.

Figure 10.  Rotational shift pattern observable for T1.2 of SPO (in metres); the average shift (especially in Y-direction) is larger than the expected value range of 10–15 m.

Table 5. Mean and Std-Dev for X and Y shifts for T1 and T2 flooded areas (Vector) products against referenced orthomosaic (units: metres).

SP	T2.2 Mean X-Shift	T2.2 Mean Y-Shift	T2.2 Std-Dev X-Shift	T2.2 Std-Dev Y-Shift	T1.2 Mean X-Shift	T1.2 Mean Y-Shift	T1.2 Std-Dev X-Shift	T1.2 Std-Dev Y-Shift
SP-C	2.13	4.49	25.62	34.85	15.68	28.64	33.64	35.58
SP-H	−215.77	−25.45	30.28	37.70	−226.6	13.85	23.09	28.61
SP-L	−16.48	−8.34	21.23	38.11	1.00	9.11	15.82	29.11
SP-O	−3.83	−21.49	16.21	42.94	6.70	6.75	13.25	26.99
SP-R	9.14	−41.83	49.44	43.13	0.37	1.67	23.17	21.85
SP-Z	−21.36	1.99	56.27	41.65	54.46	−18.50	61.59	56.78

Table 6. Mean and Std-Dev for X and Y shifts for T1 and T2 flooded areas (raster) products against referenced orthomosaic.

SP	T2.2 Mean X-Shift	T2.2 Mean Y-Shift	T2.2 Std-Dev X-Shift	T2.2 Std-Dev Y-Shift	T1.2 Mean X-Shift	T1.2 Mean Y-Shift	T1.2 Std-Dev X-Shift	T1.2 Std-Dev Y-Shift
SP-C	25.24	−22.75	37.15	61.12	28.07	7.70	40.10	30.70
SP-H	−220.93	−44.29	29.78	32.04	−230.63	3.93	51.56	32.43
SP-L	−1.58	−22.80	30.70	38.04	17.96	−1.83	36.30	27.40
SP-O	−2.91	−19.39	18.05	26.89	2.96	3.50	11.35	25.42
SP-R	−8.33	−44.44	47.59	43.08	−11.37	22.28	26.07	27.62
SP-Z	−2.65	−6.87	50.70	38.66	59.32	−38.48	77.52	49.84

Inter-comparison results

The total area for AOI-T1 is 2508 km² and for AOI-T2 is 2127 km². The inter-comparison procedure involved the same processing steps used for the product validation, but were done on pairs of the same products coming from different SPs. The accuracy and the kappa results for T1.2 and T2.2 products are shown in Tables 7 and 8. It is worth emphasising that in the context of inter-comparison the terms ‘overall accuracy’ and ‘kappa’ have a particular meaning. In fact, they are computed exactly like in any validation analysis (i.e. building a confusion matrix) but without having ground truth values: what are compared are the classification values coming from pairs of SPs. The results obtained for flood extent maps T2.2, are higher than those produced for T1.2. The highest accuracy (89.7%) and agreement (κ = 0.78) was found between Service Providers ‘H’ (SPH) and ‘R’ (SPR). In general one can state that there is a relatively good consistence between the flood extent maps delivered by the SPs. However, despite the fact of coming from exactly the same EO data, the SP results are far from being very similar: there is a clear impact of the different used processing procedures.

Table 7. Inter comparison – confusion matrix results for product T1.2, flood coarse mapping 24 April 1997.

	Overall accuracy (AC, %)
	SPC	SPH	SPL	SPO	SPR	SPZ
SPC		80.3	78.1	86.0	85.0	80.8
SPH			82.2	81.5	86.1	80.9
SPL				78.7	81.8	77.0
SPO					87.4	82.8
SPR						85.8
SPZ
	Kappa (κ)
	SPC	SPH	SPL	SPO	SPR	SPZ
SPC		0.577	0.535	0.701	0.673	0.597
SPH			0.626	0.610	0.700	0.601
SPL				0.554	0.614	0.524
SPO					0.731	0.643
SPR						0.701
SPZ

Table 8. Inter comparison – confusion matrix results for product T2.2, flood coarse mapping 28 April 1997.

	Inter comparison product T2.2 – flood mapping 28 April 1997
	Overall accuracy (AC, %)
	SPC	SPH	SPL	SPO	SPR	SPZ
SPC		87.0	83.0	84.3	85.9	83.9
SPH			84.6	86.7	89.7	86.5
SPL				88.5	86.2	87.0
SPO					89.2	88.9
SPR						87.7
SPZ
Kappa (κ)
	SPC	SPH	SPL	SPO	SPR	SPZ
SPC		0.726	0.650	0.671	0.704	0.663
SPH			0.682	0.719	0.781	0.714
SPL				0.763	0.715	0.732
SPO					0.770	0.764
SPR						0.740
SPZ

The impact of the study

The results of this project were presented and discussed with the SPs in a workshop that took place in January 2010. The discussion raised the following elements related to the validation results:

1.	The results of the analysis of the geometric quality indicate the necessity, at least for some SP, to improve the geo-referencing methods and the control of the geo-referencing quality before product delivery.
2.	A major limitation to achieve good geocoding performances is the lack of suitable data (Ground Control Points and features that can be identified in the EO imagery) in the required geographic or cartographic coordinates. This aspect should be addressed in future validation exercises.
3.	The missing areas could be sensibly improved by performing a relatively straightforward analysis based on the SAR data and a DEM of the area.

In RESPOND and in the International Charter about 65% of the activations are for flood emergencies. Given the frequency and importance of flood events it is important to further improve flood services. The validation exercises should be run more regularly during the lifetime of any project dedicated to flood services based on EO data. Each and all of the Crisis Mapping and Damage Assessment services need to go through a proper technical validation as done in this exercise. The dialogue between validation experts and providers concerning the quality of products and services should continue in the short-term. This may require re-allocating resources within ongoing projects. Following the discussions with the SPs it was concluded that future validation exercises should address the following aspects, which were not addressed so far:

•	Increase the involvement of SPs in the validation exercise preparation. This could help in choosing products that are closer to those usually delivered in practice, e.g. delivering maps in addition to ‘layers’.
•	This exercise was focused on plain flooding and HR C-Band SAR, HH polarisation. Additional technical validation should consider: (a) HR SAR with VV polarisation (ERS SAR, ENVISAT ASAR); (b) VHR SAR (C-Band, X-Band); and (c) optical imagery (HRO and VHRO in case of dense vegetation and snow).
•	The quality of other flood mapping products delivered by RESPOND could be addresses, e.g. the reference and asset mapping, the dynamic flood mapping, the detailed impact assessment in urban areas, etc.

Conclusions

The product validation and inter-comparison analysis results obtained in the RESPOND flood exercise have been summarised in this paper. The most important conclusions derived from the exercise are listed below. They refer to the main product, i.e. the flood extent map service (flood extent product), which provides the flood impact observation at the same time and at the same resolution of the input EO data.

•	Thematic accuracy. The overall accuracy obtained by the six SPs ranges from 81 to 87%. This shows a good global agreement between the delivered products and the ground truth. This result is in line with the results of other RADARSAT flood mapping validation studies published in the literature.
•	Inter-comparison. The ‘inter-comparison overall accuracy’, computed by comparing pairs of products, indicating how consistent the products from different providers are, ranges from 77 to 89.7%. In principle one should expect that these values, which come from the comparison of products based on the same source (EO data), are higher than those from the product validation results which compares data of different nature: EO-based products vs. ground truth. This is not the case. Possible causes can be the above mentioned spatial referencing errors (though there is not a clear correlation between shifts and low inter-comparison overall accuracy) and the use of different flood estimation procedures.
•	Timeliness (value-adding time only). For the flood extent products, 66% of T1.2 products were delivered in less than 10 h. The entire AOIs of T1.2 and T2.2 were covered with flood maps at best within 9 h and at the worst within 49 h. This is in agreement with the performance of more than 70 activations of the International Charter Space & Major Disasters, which is described in Allenbach et al. (2005). It is worth emphasising that this exercise solely considers the value-adding timeliness. The timeliness of the whole flood mapping service is affected by the performance of the EO data supply service: mobilisation, satellite tasking and EO-data production.
•	Geometric accuracy. The magnitude of the geometric shifts (geocoding errors) varies from one SP to another. Some SPs show negligible shifts, while some others have considerably larger geo-referencing errors. In addition, in some cases there are non-negligible geo-referencing inconsistencies between the same vector and raster products generated by the same SP.
•	Missing and extra areas. Extra areas range from 3.2 to 7.4%, while the missing areas range from 5.8 to 15.3%. The missing and extra areas have systematic patterns: the missing areas are mainly along the Red River and its attributors, while extra areas are mainly in the east and west borders of the image and along the emergency channels. These systematic errors are due to the intrinsic characteristics of the EO data.
•	An in-depth analysis made by the validation team, which was based on the combination of SAR and a DEM, showed that the missing areas could be improved by exploiting the DEM information. Using a SRTM DEM over the two test areas the overall accuracy could be improved by 5–6.2% and the Kappa by 17–18%.

Following a workshop addressing the results of this study; it was concluded with the SPs that there is a necessity to improve the geo-referencing methods and the control of the geo-referencing quality before product delivery. Moreover, it was emphasised that future validation exercises should be run more regularly during the lifetime of any project dedicated to flood services based on EO data. Each and all of the Crisis Mapping and Damage Assessment services need to go through a proper technical validation as done in this exercise. The dialogue between validation experts and SPs concerning the quality of products and services should be a continuous process.

Notes on contributors

Michal Shimoni received a MSc in Physical Geography in the subject of thermal remote sensing at Tel Aviv University, Israel in 1996. In 1999 she received a Diplôme d'Etude Approfondie (DEA) in Civil Engineering in the subject of SAR interferometry for hydrological applications at the University of Gembloux, Belgium. Since 1999 she has been employed as a researcher in the Signal & Image Centre of the Belgian Royal Military Academy. Her research interests include imaging spectroscopy, PolSAR, PolInSAR and multi sensor fusion.

Michele Crosetto is a Civil Engineer from the Politecnico of Turin (1993). After 2 years of specialisation in geomatics in Switzerland (EPF Lausanne and ETH Zurich) he got his Ph.D. in Geodesy in 1998 at the Politecnico of Milan, with a thesis on interferometric SAR DEM generation. Since 1998 he has been working as a researcher at the EU Joint Research Centre of ISPRA (Italy), the Cartographic Institute of Catalonia, Barcelona (Spain) and the Institute of Geomatics, Castelldefels (Spain). Since 2002 he has been the head of the Active Remote Sensing Unit of the Institute of Geomatics. His main research fields include deformation measurement techniques, including spaceborne and terrestrial SAR data and Persistent Scatterers Interferometry techniques.

Stefan Lang is a Geographer and GIS specialist. He obtained his Ph.D. from Salzburg University in 2005. Since 2006 he is a Senior Researcher and Research Coordinator at the Centre for Geoinformatics (Z_GIS), University of Salzburg. Between 2007 and 2009 he was deputy professor and research fellow at the Technical University of Berlin, Department of Geoinformation Processing for Landscape and Environmental Planning. His main research interests are object-based image analysis (OBIA), spatial modelling, EO-based monitoring and spatial indicators.

Philippe Bally is an EO Applications Specialist working with the European Space Agency. He is the technical officer of the GMES RESPOND project and represents ESA at the Executive Secretariat of the International Charter Space and Major Disasters.

François Boubila is an Earth Observation Engineer working with Serco at the ESRIN centre of ESA. He is a project Manager with the International Charter Space and Major Disasters. He has previously worked alongside the Bulgarian authorities in a cooperation project aiming at developing the integration of satellite imagery in the operational procedures of hazard management in Bulgaria.

Acknowledgements

The flood products analysed in this paper were generated by six Service Providers: E-Geos (Italy), Metria (Sweden), Sertit (France), DLR ZKI (Germany), Infoterra GmbH (Germany) and UNITAR/UNOSAT (UN, based in Geneva). The authors would like to thank U.S. Army Corps of Engineers, St. Paul District, and especially Mr. Evans Kreig and Mrs. Keith Le Claire for the Red-River flood validation data sets. Moreover, the ISVG would like to thank the Canadian Space Agency and especially Mr. Mahmood Ahmed and Mr. Robert Saint-Jean for the assistance in the selection and the provision of the RADARSAT-1 data.