Service-oriented infrastructure for flood mapping using optical and SAR satellite data

Abstract

In this paper, we present the service-oriented infrastructure within the Wide Area Grid project that was carried out within the Working Group on Information Systems and Services of the Committee on Earth Observation Satellites. The developed infrastructure integrates services and computational resources of several regional and national Grid systems: Ukrainian Academician Grid (with satellite data processing Grid segment, UASpaceGrid) and Grid system at the Center on Earth Observation and Digital Earth of Chinese Academy of Sciences. The study focuses on integrating geo-information services on flood mapping provided by Ukrainian and Chinese entities to benefit from information acquired from multiple sources. We also describe services for workflow automation and management in Grid environment and provide an example of workflow automation for generating flood maps from optical and synthetic-aperture radar satellite imagery. We also discuss issues of enabling trust for the infrastructure using certificates and reputation-based model. Applications of utilizing the developed infrastructure for operational flood mapping in Ukraine and China are given as well.

Keywords:

• earth observation
• geoinformatics
• geospatial data integration
• image processing
• remote sensing

1. Introduction

Nowadays, earth observation (EO) data are very important for disaster monitoring and forecasting. Satellite data are used for rapid mapping in case of emergencies (Gallego et al. 2012, Dellepiane, Angiati, and Vernazza 2010, Di Baldassarre and Uhlenbrook 2012, Di Baldassarre et al. 2011, Kravchenko et al. 2008, Kussul, Shelestov, and Skakun 2008, Kussul et al. 2010a, 2012a, 2012b, Schumann, Di Baldassarre, and Bates 2009, Skakun et al. 2007) and are integrated into models to predict them (Bates et al. 1997, Horritt and Bates 2002, Horritt 2006, Schumann et al. 2009, Di Baldassarre et al. 2010, Prestininzi et al. 2011, Stephens et al. 2012). There are a lot of international initiatives aimed at utilization of satellite data for disaster management. The most important among them are the Global Earth Observation System of Systems (GEOSS), International Charter ‘Space and Major Disasters’, Global Monitoring for Environment and Security (GMES), UN Platform for Space-based Information for Disaster Management and Emergency Response (UN-SPIDER) and Sentinel Asia.

A system of systems is required in order to enable solution of these problems at both regional and global scales. Such an approach envisages the interoperable use of satellite data and corresponding products and services, and integration of existing regional and international satellite monitoring systems (Zyl, Simonis, and McFerren 2009, Percivall 2010, Kussul et al. 2010a, 2012a, 2012b). It is particularly important for the area of disaster management where different space agencies collaborate in order to timely provide necessary data and services in case of emergency (Guo 2010).

A considerable need therefore exists for an appropriate infrastructure that will enable the integrated and operational use of multi-source data for different applications domain. From technological point of view, Grids can provide solutions to the above-mentioned problems. In this case, a Grid environment can be considered not only for providing high-performance computations but also facilitate interactions between different actors by providing a standard infrastructure and a collaborative framework to share data, algorithms, storage resources and processing capabilities (Lecca et al. 2011, Kussul, Shelestov, and Skakun 2008, 2009, Kussul et al. 2012a, 2012b, Cossu et al. 2009, Fusco, Cossu, and Retscher 2007, Shelestov, Kussul, and Skakun 2006).

There have been several studies on evaluating and using Grids for the Earth observation (EO) domain. In particular, we may refer to the GENESI-DR project (www.genesi-dr.eu) that aims at building a Grid-based infrastructure supporting the GEOSS architecture. The successor of this project, GENESI – Digital Earth Community, aims to establish open data and services access, allowing end-users to seamlessly access, produce and share data, information, products and knowledge.

The European Space Agency (ESA) and the European Space Research Institute developed the Earth Observation Grid Processing on-Demand infrastructure (G-POD) (Cossu et al. 2009, Fusco, Cossu, and Retscher 2007). Fast Access to Imagery for Rapid mapping Exploitation (FAIRE) is another Grid-based application that is operationally used by ESA in the context of flood mapping. The application takes advantage of Grid technology for near real-time data access, calibration, orthorectification, map projection and co-registration. The application is operationally used in the context of the International Charter ‘Space and Major Events’ (http://www.disasterscharter.org).

Dissemination and Exploitation of GRids in Earth science (DEGREE) project was a European-funded project that aimed to build a bridge linking the Earth Science and Grid communities throughout Europe (Cossu et al. 2010). Grid was considered to be the appropriate platform for integration of heterogeneous data resources, processing tools, models, algorithms and so on. The following applied problems were within the scope of DEGREE: earthquake analysis, floods modelling and forecasting, and influence of climate changes on agriculture.

Xue et al. (2008) discuss the use of Grid technology for digital earth applications. They are describing core-middleware for EO data processing and applications for the Digital Earth Prototype System.

Huang et al. (2010) describe an integrated framework, comprising the geosensor network layer, the Grid layer and the application layer to address the problem of collecting physical data dynamically and modelling a real-time virtual world.

A Grid services for EO image data processing framework exploits Grid computing and service-oriented architecture approaches to provide a data processing environment, where users can discover the data and process, based on their requirements (Dube, Ramakrishnan, and Dasgupta 2013).

In this paper, we present a service-oriented infrastructure for flood mapping using optical and synthetic-aperture radar (SAR) satellite data. This infrastructure was developed within the Wide Area Grid (WAG) project that was carried out within the Working Group on Information Systems and Services (WGISS, http://www.ceos.org/wgiss) of the Committee on Earth Observation Satellites (CEOS). The developed infrastructure integrates services and computational resources of several regional and national Grid systems: Ukrainian Academician Grid (with satellite data processing Grid segment, UASpaceGrid) and Grid system at the Center on Earth Observation and Digital Earth of Chinese Academy of Sciences (CEODE-CAS). The study focuses on integrating geo-information services on flood mapping provided by Ukrainian and Chinese entities to benefit from information acquired from multiple sources. It is particularly important for the area of flood monitoring and prediction where daily flood extent maps during emergency are usually required. We also describe services for workflow automation and management in Grid environment and provide an example of workflow automation for generating flood maps from optical and SAR satellite imagery. We also discuss issues of enabling trust for the infrastructure using certificates and reputation-based model. Applications of utilizing the developed infrastructure for operational flood mapping are given as well.

2. The WAG project architecture

The WAG project was initiated by the French Space Agency (Centre national d'études spatiales [CNES]) within the CEOS-WGISS activities with main contributions from Space Research Institute (SRI) NASU-NSAU (Ukraine) and CEODE-CAS (China). The WAG project is considered CEOS-WGISS contribution to GEOSS. The project objective was to develop the ‘horizontal’ infrastructure in order to integrate computational, human, intellectual and informational resources of the space agencies within a distributed system. In order to create such an infrastructure, Grid technologies were used. A domain application that was addressed within the project is flood monitoring and prediction. In particular, optical and SAR satellite data are used for flood mapping.

In order to enable fast acquisition, processing and delivery of geo-information products in case of emergency, corresponding services were developed within the infrastructure. Figure 1 shows services of the geo-information infrastructure for automated workflow of satellite data processing.

Figure 1. System architecture.

User interface of the system is implemented through a portal (Figure 2). It provides a transparent user interaction with resources at different levels. The portal is directly connected with the following high-level services (Williams et al. 2008):

• security subsystem services including reputation management services;

• data catalogue services;

• metadata services;

• services for automatic workflow generation;

• services for data aggregation, selection and visualization.

Figure 2. Portal enabling flood mapping from satellite SAR imagery.

High-level services that require high-performance computations interact with low-level services that are implemented within the Grid infrastructure (Kussul, Shelestov, and Skakun 2009, Kussul 2012b, Cossu et al. 2009). The Grid infrastructure has the following components:

• Grid security infrastructure (GSI);

• data transfer protocols within Grid (Grid file transfer protocol, GridFTP);

• global resource allocation manager (GRAM);

• credential management service (MyProxy);

• replica location service (RLS);

• open Grid services architecture – database access and integration (OGSA-DAI) services.

At the lowest level are data access services:

• remote data storage;

• operational data storage;

• computational systems.

The use of standardized interfaces and compatible components at the different levels provides scalability and controllability of the system. All the complexity of subsystems interactions is hidden from the end-user.

Currently, the system is composed of the following resources:

• SRI NASU-NSAU (Ukraine) with deployed computing and storage nodes based on the Globus Toolkit (GT) 4 and gLite 3 middleware, access to geospatial data and a Grid portal.

• Institute of Cybernetics of NASU (IC NASU, Ukraine) with deployed computing and storage nodes based on GT 4 middleware and access to computing resources (SCIT-1/2/3 clusters, more than 650 processors).

• CEODE-CAS (China) with deployed computing nodes based on gLite 3 middleware and access to geospatial data (approximately 16 processors).

Satellite data are distributed over the Grid environment. The Envisat/Advanced Synthetic Aperture Radar (ASAR) data are stored on the ESA's rolling archive and routinely downloaded for the Ukrainian territory. Then, they are stored at the SRI NASU-NSAU archive that is accessible via the Grid. Moderate Resolution Imaging Spectroradiometer (MODIS) data from Terra and Aqua satellites that are used in flood and agriculture applications are routinely downloaded from the U.S. Geological Survey (USGS) archives and stored at the SRI NASU-NSAU, IC NASU and CEODE-CAS.

3. High-level services

This level includes security subsystem, catalogue services, metadata services (description and access), automatic workflow generation services, and data aggregation, sub-setting and visualization services. These services are connected to the Grid services at the lower level.

3.1. Data catalogue services

Satellite data are stored at the storage resources of the Grid system. Corresponding catalogue services were developed in order to search and getting access to data.

3.2. Metadata services

Metadata store necessary information on data that enables data search through the portal of the system.

3.3. Services for automatic workflow generation

Satellite data processing is a complex workflow composed of many steps. Corresponding services were developed in order to automatically generate processing workflow based on user inputs. The workflow is further submitted and executed in the Grid. The workflow of the data processing steps in the Grid is managed by a Karajan engine (www.gridworkflow.org/snips/gridworkflow/space/Karajan).

To benefit from the use of the Grid, a parallel version of the method for flood mapping from satellite SAR imagery has been developed (Skakun 2010). The parallelization of the image processing was implemented in the following way: a SAR image is split into the uniform parts that are processed on different nodes using the OpenMP Application Program Interface (www.openmp.org).

3.4. Services for data aggregation, selection and visualization

The results of data processing are further converted to the format required by the user, for example Keyhole Markup Language (KML), WMS, Environmental Systems Research Institute (ESRI) shape. The user can also select a region of interest from the generated map and integrated it with other information.

4. Grid services

These services provide access to the shared resources of the Grid system, access to credentials, file transfer, job submission and management.

4.1. GSI

In order to enable security trust between different parties in the Grid system, a public key infrastructure is traditionally applied. X.509 is the most widely used format which is supported by most of the existing software. To get access to resources of the Grid system, a user should make a request to a Certificate Authority (CA) which is always a known third party. The CA validates the information about the user and then signs the user certificate by the CA's private key. The certificate can thus be used to authenticate the user to grant access to the system. To provide a single sign on and delegation capabilities, the user can use the certificate and his private key to create a proxy certificate. This certificate is signed not by CA but rather the user himself. The proxy certificate contains information about the user's identity and a special time stamp after which the certificate will no longer be accepted. Therefore, the user can get access to the services within the system. At present, the user can only access predefined services or workflow for satellite data processing. In order to make customizable workflow composed of several services defined by the user, corresponding privilege information should be further mapped onto local resources. This can be done using a generalization of the X.509 certificates with the X.509 attribute certificates (ACs) and Virtual Organization Membership Service (VOMS) (Alfieri et al. 2005), and Local Credential Mapping Service (LCMAPS). Currently, we have created a virtual organization Earth Observation Grid (http://eo-grid.ikd.kiev.ua), and established a VOMS server. According to the VO policy and membership, user privileges are stored inside the X.509 AC by the VOMS server, signed by the VOMS server, and then embedded into the user's proxy certificate to be exploited in the Grid (Alfieri et al. 2005). In order to translate Grid credentials to credentials local to the site the LCMAPS service is used.

In order to enable Grid system integration with different middleware installed and security mechanisms and policies used, the following solutions were tested:

1. To create our own CAs and to enable the trust between them.

2. To obtain certificates from a well-known CA, for example, the European Policy Management Authority for Grid Authentication (http://www.eugridpma.org).

3. To use a combined approach in which some of the Grid nodes accept only certificates from the local CA and others accept certificates from a well-known third-party CAs.

Within the integration of the UASpaceGrid and the CEODE-CAS Grid, the second and the third approaches were verified. In such a case, the UASpaceGrid accepted the certificates issued by the local CA that was established using the TinyCA, and certificates issued by the UGRID CA (https://ca.ugrid.org).

It is worth mentioning that GT v.4 and gLite v.3 middleware implement the same standard for the certificates, but different standards for describing the certificate policies. That is why it is necessary to use two different standards for describing the CA's identity in a policy description file.

In order to enable trust between different parties, we also focused on estimating resources reputation based on quality of service (QoS) metrics. In particular, we extended the existing utility-based reputation model for virtual organizations in Grids (Arenas, Aziz, and Silaghi 2008) to address threats scenarios in the area of trust and reputation management (Kussul and Skakun 2004, Skakun, Kussul, and Lobunets 2005, Skakun and Kussul 2006, Shelestov and Kussul 2008, Kussul, Kussul, and Skakun 2013). These modifications included: assigning initial reputation to a new entity in VO, capturing alliance between consumer and resource, defining a time decay function and score function. The proposed reputation model was analysed against the existing security threat scenarios and how the model overcomes them. In particular, the model was effective in countering such threats as individual malicious peers, malicious collectives, driving down the reputation of a reliable peer, while there were some limitations in countering malicious collectives with camouflage.

The proposed model was evaluated not only in terms of countering security threats but also resource management in service-oriented systems. A problem of online job scheduling was considered, and a general procedure for integrating reputation into the scheduler using a non-linear trade-off scheme was presented. The advantage of using such a scheme is that it provides a Pareto-optimal solution partially satisfying criteria with corresponding weights. The experiments showed that the scheduler with knowledge of reputation using a non-linear trade-off scheme outperformed a scheduler without knowledge of reputation on average 45% for all performance metrics used in the study. A non-linear trade-off scheme for incorporating reputation into the scheduler of resource management system was compared to a multiplication scheme, and showed better performance with a factor of 2. Currently, we are evaluating the use of such a model for the GEOSS infrastructure.

4.2. GridFTP

The GridFTP protocol (Allcock et al. 2005) was chosen to provide data transfer between the Grid systems. GridFTP is an extension of the standard FTP with the following additional capabilities: integration with the GSI enabling the support of various security mechanisms; improved performance of data transfer using parallel streams to minimize bottlenecks; multicasting by doing one-source-to-many-destinations transfers. In order to test file transfers between different platforms used at the UASpaceGrid and the CEODE-CAS Grid, the GridFTP version with the Grid Mapfile authorization was utilized. File transfers were successfully completed in both directions between two Grids with configured client and server roles.

4.3. GRAM

This service (Feller, Foster, and Martin 2007) was used to execute jobs on the Grid resources.

4.4. MyProxy Credential Management Service

An open-source software MyProxy (Basney, Humphrey, and Welch 2005) was used to delegate credentials to services acting on behalf of the users.

4.5. RLS

RLS of the Globus Toolkit (GT) is used to provide data replication management in Grids. The RLS services provide further abstraction layer for dealing with data by mapping information from logical names for data items to target names.

4.6. OGSA-DAI

This framework provides uniform interfaces to access heterogeneous data. This framework allows the creation of high-level interfaces to data abstraction layer hiding the details of data formats and representation schemas. Most of the problems such as delegation, reliable file transfer, data flow between different sources are handled automatically in the OGSA-DAI framework. The OGSA-DAI containers are easily extendable and embeddable. One can achieve distributed geospatial data processing such as OGSA-DAI in the Grid environment through some extension implementations (Huang et al. 2011). But comparing to the Web Services Resource Framework (WSRF) basic functionality, the implementation of an OGSA-DAI extension is much more complicated. Moreover, the OGSA-DAI framework requires a preliminary deployment of additional software components.

4.7. Data storage

Data are stored on storage resource with operational and long-term data archives as well as back-up subsystems.

5. The use of service-oriented infrastructure for flood monitoring

5.1. Flood mapping workflow from satellite imagery

This subsection describes the workflow for satellite data processing required to produce flood maps.

The processing of satellite data can be viewed as a complex distributed workflow that composed of many steps (Kussul, Shelestov, and Skakun 2008, 2009). To process a SAR satellite imagery, corresponding workflow is created. For this purpose, we use an Extensible Markup Language-based Karajan language. Karajan provides services for data transfer from remote sources using GridFTP, and submission of local and remote tasks through the GT2 GRAM and GT4 WSRF-GRAM services. Karajan also allows presentation of the workflow in the form of the direct acyclic graph.

For flood mapping from satellite data, the workflow consists of the following stages: (1) data transfer upon user request from the remote ESA rolling archives to the local or Grid resources. Currently, this is done via the use of standard ftp server. (2) Orthorectification and radiometric correction of the SAR image using digital elevation model. (3) Processing of SAR image using a neural network classifier (Bakan and Kussul 1996, Kussul et al. 2007, Kussul, Shelestov, and Skakun 2008, 2009, Skakun 2010). The neural network assigns to each pixel of the output image a binary value corresponding to one of two classes (‘Water’ and ‘No water’) taking as input a moving window of image pixel intensities. For this stage, a parallel version of the classification algorithm was developed using OpenMP API that can be distributed on difference processing nodes. (4) Removal of those regions that were detected as shadow/layover at the stage No. 2, since the pixel values in these regions are not related to physical conditions of the environment. (5) Geocoding, i.e. transformation to geographical projection using ground-control points incorporated in the SAR image. (6) Visualization of the results using ESRI shape files, KML files and Open Geospatial Consortium (OGC)-Web Map Service (WMS).

5.2. Integration of Ukrainian and Chinese flood mapping services

In order to benefit from data of different nature (e.g. optical and radar) and provide integration of different products in case of emergency, Ukrainian flood mapping service was integrated with the flood mapping services provided by the CEODE-CAS. The latter service is based on the use of optical data acquired by MODIS instrument onboard Terra and Aqua satellites (Xie et al. 2010). Figure 3 shows the architecture of the China–Ukrainian service-oriented system for disaster management.

Figure 3. Architecture of China–Ukrainian service-oriented system for disaster monitoring.

The integration of the Ukrainian and Chinese systems is done at the level of services. The portals of SRI NASU-NSAU and CEODE-CAS are operated independently and communicate with corresponding brokers that provide interfaces to the flood mapping services. These brokers process requests from both local and trusted remote sites. We use the POST request method for making requests. Currently, we are implementing web services for making and processing requests. To provide a flood mapping product using SAR data, the CEODE-CAS portal generates the corresponding search request to the broker at the SRI NASU-NSAU side based on user search parameters. This request is processed by the broker and the search results are displayed at the CEODE-CAS portal. The user selects the SAR image file to be processed, and the request is submitted to the SRI NASU-NSAU broker which generates and executes the workflow in the Grid, and delivers the flood maps to the CEODE-CAS portal. We use the OGC-WMS and KML standards for displaying flood maps. The flood maps can also be delivered as a GeoTIFF file, and further vectorized into the shapefile. The same applies to the broker operated at the CEODE-CAS side that provides flood mapping services using optical satellite data. In order to get access to the portal, the user should have a valid certificate. The SRI NASU-NSAU runs the VOMS server to manage with this issue.

6. Case study applications

6.1. Flood monitoring and risk assessment for Ukraine

The severe 2010 and 2012 winters in Ukraine that were characterized by large amount of snow and ice on rivers posed a high threat of floods during the melting period. In order to provide flood risk assessment (Kussul et al. 2010b), both SAR and optical satellite imagery were used.

In particular, we acquired more than 30 Envisat/ASAR scenes during the 2010 winter–spring period, and more than 50 archived scenes during autumn 2009 to monitor the snow cover. Satellite optical images in 2010 and 2012 were acquired using the EO-1 Sensor Web System (Figure 4). Images from Ukrainian remote sensing satellite Sich-2 were acquired for 2012 as well (Figure 5). These images were processed using Grid resources.

Figure 4. Fragment of EO-1/ALI image over the Kyiv area acquired on 17 March 2012. Spatial resolution is 30 m. Data courtesy of the NASA EO-1 mission operated by the Goddard Space Flight Center.

Figure 5. Fragment of Sich-2 image over the Kyiv area acquired on 17 March 2012. Spatial resolution is 8 m. Data courtesy of the State Space Agency of Ukraine.

The OGC Sensor Planning Service was used for tasking the EO-1 satellite (Liping 2010, Moe et al. 2008, Mandl et al. 2006). Through the UN-SPIDER RSO in Ukraine, a request was made from local authorities to acquire satellite images over the Kyiv city area due to high risk of flood in 2010 and 2012. The use of the Sensor Web and the Grid ensured a timely delivery of products to end-users. It took less than 12 h after image acquisition to generate geospatial products that were delivered to the Ukrainian Ministry of Emergency Situations, the Council of National Security and Defence and the Ukrainian Hydrometeorological Centre. The information on river extent that was derived from the EO-1 image was used to calibrate and validate hydrological models to produce various scenarios of water extent for flood risk assessment.

6.2. Flood mapping for China

Two case study areas were considered within the developed infrastructure: river Huaihe and Poyang lake.

In July 2007, following the flooding on the river Huaihe (China), SRI NASU-NSAU and CEODE-CAS processed Envisat/ASAR and Radarsat-1 images that were acquired 15 and 19 July 2007, respectively. The images were processed using the Grid resources (Xie et al. 2010). The resulting flood maps were delivered using OGC-WMS services. The example of flood extent map is given in Figure 6.

Figure 6. RADARSAT-1 image (a) and the resulting flood extent shown with white colour (b) for the river Huaihe, China (©Canadian Space Agency [CSA] 2007).

To test the flood monitoring algorithms and WAG testbed, Poyang lake in China is selected as one of the study areas. The longitude of the lake is between 110.40° and 113.10°, and the latitude is between 28.30° and 30.20°. In the experiment of the testbed, the latest MODIS data obtained by TERRA 6 September 2010 were used to detect the current water body extent. The ASAR data acquired in Wide Swath Mode at 150-m resolution from ESA of 27 August 2010 were used to detect the water body extent and cross-validate the results (Figure 7).

Figure 7. Water extent visualized in Google Earth: (a) MODIS data and water boundary result visualize in KML (MODIS data acquisition time: 6 September 2010.); (b) ASAR data and water boundary result visualized in KML (ASAR data acquisition time: 27 August 2010).

7. Results and lessons learned

The presented infrastructure for flood mapping from satellite imagery takes advantage of the service-oriented approach and Grid computing. All the interaction within the system is done via services. From technological point, it allows a seamless integration of services and data provided by different entities (Ukrainian and Chinese organizations) in case of emergency. Also, it allows integration with third-party systems such as GEOSS, GMES, UN-SPIDER, International Charter ‘Space and Major Disasters’ to be provided to facilitate and automate delivery of geospatial services to the end-users. The use of Grid allowed us to enable a collaborative framework for integrating resources and services of geographically distributed organizations. Within the Grid, two components important to the EO domain were provided: workflow management and access to high-performance computing. However, it is worth mentioning that to reduce time required for transferring large amounts of satellite imagery, a ‘code-to-data’ paradigm is a feasible solution.

From application point, the proposed infrastructure allows integration and exploitation of thematic services (flood maps) produced using data of different nature with different characteristics. In particular, the Ukrainian side provides geo-information products based on cloud-free SAR satellite imagery (Envisat/ASAR), whereas Chinese side provides daily product-based optical satellite imagery. These datasets complement each other in the repetition frequency, spatial resolution, resistance to cloud cover (with the use of SAR) to provide daily products in case of flooding.

Overall, the use of the proposed approach and technologies made it possible to improve quality of services, in particular to reduce the time required for delivering a flood map from 24 h to 12 h.

8. Conclusions

In this paper, we presented results that were achieved within the WAG project. In order to benefit from using data from multiple sources, integration of the Ukrainian and Chinese flood mapping services that use radar and optical satellite data was carried out. The particular advantages of such integration are as follows: the system will benefit from the use of services that are provided by different partners within the system; data resources are distributed within the system at different sites (Ukrainian and Chinese), so this allows us to improve the quality of services delivered within the system; the use of Grids enables the use of high-performance computing resources that in turn will decrease the time required for satellite data processing and delivery of service which is critical for disaster management domain; the use of unified Grid and OGC services provides scalability of the system while adding new components and services; such collaboration provides valuable contribution and impact to the on-going large-scale international activities such as creation of the GEOSS infrastructure.