An open compute and data federation as an alternative to monolithic infrastructures for big Earth data analytics

Figures & data

Figure 1. Basic functional framework for the C-SCALE compute and data federation.

Table 1. EO product redistribution software used by relevant parties, indicating currently supported protocols for data access and lookup. Those are the two most important features to be unified across the federation.

Software stack (sites)	Product access model	Query interface
Data Hub Software (CESNET^a, GRNET^b, ESA^c)	HTTPs, basic authentication, self-registration, OIDC authentication introduced in early 2022	OData^d + OpenSearch^e (both HTTPs, basic authentication, OIDC authentication introduced in early 2022)
EODC Data Access (EODC^f)	FTP, open access	CSW^g (over HTTPs, open access)
TerraCatalogue (VITO^h)	HTTPs, OIDC authentication (through keycloak)	OpenSearch (over HTTPs, open access)
CREODIAS^i (CloudFerro)	HTTPs, OIDC authentication (through keycloak)	CSW + OpenSearch (both HTTPs, open access)
NextGEOSS Data Hub^j (NextEOS, CESNET)	N/A	OpenSearch (over HTTPs, open access)

^ahttps://www.cesnet.cz/.

^bhttps://grnet.gr/en/.

^chttps://www.esa.int/.

^dhttps://scihub.copernicus.eu/userguide/ODataAPI.

^ehttps://opensearch.org/.

^fhttps://eodc.eu/.

^ghttps://www.ogc.org/standards/cat.

^hhttps://vito.be/en.

ⁱhttps://creodias.eu/.

^jhttps://catalogue.nextgeoss.eu/.

Table 2. Pros and cons of OpenSearch and STAC-API.

OpenSearch	STAC-API
+ Stable Search API	+ Active community, tools (server, client), support
+ Simple and well-known query interface (SciHub)	+ Active development, new features, bug fixes
+ Supported by most project partners, although there are different flavours which need to be mapped	+ Small/structured extensions
	+ Simpler metadata structure
- No active development of community	- Currently no stable release, stable base parts but large changes in query extension
- XML based: more verbose, larger payload, slower data transfer speeds compared to JSON.

Figure 2. Metadata query service architecture.

Figure 3. Federated access in C-SCALE is achieved via EGI Check-in and SRAM. Both serve as community Authentication and Authorisation Infrastructures (Community AAI in the above) and are compliant with the AARC blueprint architecture. The above figure was taken from the EOSC-hub project documentation¹⁸.

Figure 4. Federated software distribution in C-SCALE.

Figure 5. Basic use case sequence diagram.

Table 3. Summary of use cases and the challenges they are trying to overcome.

Use case	Scope of challenge
Land surface change monitoring	This use case compares the performance of accessing and processing Sentinel-2 data on remote object storage and local object storage.
Coastal hydrodynamic and water quality modelling	This use case tests the interoperability, scalability and performance of combining cloud and HPC compute and data resources in its workflow.
Seasonal river discharge ensemble forecasting	This use case compares the performance and scalability of ensemble forecasting on HTC and cloud compute and data resources.
Monitoring tropical forest recovery capacity	This use case explores methodologies to create interoperable and scalable analysis ready data cubes for Landsat and Sentinel-1, including assessing the quality of the Sentinel-1 GRD application ready data to detect magnitudes and recovery times of the signal disturbances.
Real-time reservoir surface water area monitoring	In a real-time application, this use case quantifies the redistribution delays (latency) of Sentinel data offered via the ESA DataHub Relays that are part of the federated infrastructure.
Wetland Water Stress Analysis	This use case tests the performance of connecting data cubes hosted on distributed infrastructures in a single workflow compared to using data cubes hosted on a single infrastructure.

Data availability statement

The data that support the findings of this study are openly available in https://zenodo.org/ at https://zenodo.org/communities/c-scale/.