Innovative Analysis Ready Data (ARD) product and process requirements, software system design, algorithms and implementation at the midstream as necessary-but-not-sufficient precondition of the downstream in a new notion of Space Economy 4.0 - Part 2: Software developments

Figures & data

Figure 45. National Oceanic and Atmospheric Administration (NOAA)’s Geostationary Operational Environmental Satellites (GOES)-16 Advanced Baseline Imager (ABI), 16-band spectral resolution. Temporal resolution: 5–15 minutes. Spatial resolution: Bands 1, 500 m (0.5 km), Band 2 to 5: 1000 m (1 km), Bands 6–16: 2000 m (2 km). Table legend: Visible green/red channels: G, R. Near/Middle Infrared channel: NIR, MIR. Far/Thermal Infrared: TIR, see Figure 7 in the Part 1. Top-of-atmosphere reflectance, in range [0.0, 1.0]: TOARF. Kelvin degrees, in range ≥ 0: K°.

Figure 46. Active fire land cover (LC) class-specific (LC class-conditional) sample of 311 pixels selected world-wide, National Oceanic and Atmospheric Administration (NOAA)’s Geostationary Operational Environmental Satellites (GOES)-16 imaging sensor’s bands 1 to 6 (see Figure 45), radiometrically calibrated into top-of-atmosphere (TOA) reflectance (TOARF) values, belonging to the physical range of change [0.0, 1.0] (Rocha de Carvalho, 2019). (a) Active fire LC class-specific family (envelope) of spectral signatures, including Band 4 – Cirrus band, consisting of, first, a multivariate shape information component and, second, a multivariate intensity information component, see Figure 30 in the Part 1. To be modeled as a hyperpolyhedron, belonging to a multi-spectral (MS), specifically, a 6-dimensional, color data hypercube in TOARF values, see Figure 29 in the Part 1. (b) Same as (a), without Band 4 – Cirrus band.

Figure 47. Active fire land cover (LC) class-specific (LC class-conditional) sample of 311 pixels selected world-wide, National Oceanic and Atmospheric Administration (NOAA)’s Geostationary Operational Environmental Satellites (GOES)-16 imaging sensor’s bands 7 to 16 (see Figure 45), radiometrically calibrated into top-of-atmosphere (TOA) Temperature (TOAT) values, where the adopted physical unit of measure is the Kelvin degree, whose domain of variation is ≥ 0 (Rocha de Carvalho, 2019). This Active fire LC class-specific family (envelope) of spectral signatures consists of, first, a multivariate shape information component and, second, a multivariate intensity information component, see Figure 30 in the Part 1. To be modeled as a hyperpolyhedron, belonging to a multi-spectral (MS), specifically, a 10-dimensional, color data hypercube in Kelvin degree values, see Figure 29 in the Part 1.

Figure 48. (a) Moderate Resolution Imaging Spectroradiometer (MODIS) image acquired on August 23, 2007, at 9.35 (CEST), covering Greece, depicted in false colors (monitor-typical channel R = MODIS band 6 in the Middle Infrared, channel G = MODIS band 2 in the Near Infrared, channel B = MODIS band 3 in the Visible Blue), spatial resolution: 1 km, radiometrically calibrated into top-of-atmosphere reflectance (TOARF) values, where an Environment for Visualizing Images (ENVI, by L3Harris Geospatial) standard histogram stretching was applied for visualization purposes. (b) Satellite Image Automatic Mapper (SIAM™)’s map of color names (Baraldi, 2017, 2019a; Baraldi et al., 2018a, 2018b, 2006; Baraldi & Tiede, 2018a, 2018b), generated from the MODIS image shown in (a), consisting of 83 spectral categories, depicted in pseudo-colors, refer to the SIAM map legend shown in Table 4 and Figure 50.

Figure 49. At left, portion of the image shown in Figure 48, radiometrically calibrated into top-of-atmosphere reflectance (TOARF) values. Moderate Resolution Imaging Spectroradiometer (MODIS) image acquired on August 23, 2007, at 9.35 (CEST), covering Greece, depicted in false colors (monitor-typical channel R = MODIS band 6 in the Middle Infrared, channel G = MODIS band 2 in the Near Infrared, channel B = MODIS band 3 in the Visible Blue), spatial resolution: 1 km, radiometrically calibrated into top-of-atmosphere reflectance (TOARF) values, where an Environment for Visualizing Images (ENVI, by L3Harris Geospatial) standard histogram stretching was applied for visualization purposes. At right, Map Legend – Red: fire pixel detected in both the traditional MODIS Fire Detection (MOFID) algorithm and the SOIL MAPPER-based Fire Detection (SOMAFID) algorithm, an expert system for thermal anomalies detection (Pellegrini, Natali, & Baraldi, 2008). White: fire pixel detected by SOMAFID, exclusively. Green: fire pixel detected by MOFID, exclusively.

Figure 50. Legend (vocabulary) of hyperspectral color names adopted by the prior knowledge-based Landsat-like Satellite Image Automatic Mapper™ (L-SIAM™, release 88 version 7, see Table 4) lightweight computer program for multi-spectral (MS) reflectance space hyperpolyhedralization (see Figures 29 and 30 in the Part 1), superpixel detection (see Figure 31 in the Part 1) and object-mean view (piecewise-constant input image approximation) quality assessment (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b). Noteworthy, hyperspectral color names do not exist in human languages. Since humans employ a visible RGB imaging sensor as visual data source, human languages employ eleven basic color (BC) names, investigated by linguistics (Berlin & Kay, 1969), to partition an RGB data space into polyhedra, which are intuitive to think of and easy to visualize, see Figure 29 in the Part 1. On the contrary, hyperspectral color names (see Figure 30 in the Part 1) must be made up with (invented as) new words, not-yet existing in human languages, to be community-agreed upon for correct (conventional) interpretation before use by members of a community (refer to Subsection 4.2 in the Part 1). For the sake of representation compactness, pseudo-colors associated with the 96 color names/spectral categories, corresponding to a partition of the MS reflectance hyperspace into a discrete and finite ensemble of mutually exclusive and totally exhaustive hyperpolyhedra, equivalent to 96 envelopes/families of spectral signatures (see Figures 29 and 30 in the Part 1), are gathered along the same raw if they share the same parent spectral category (parent hyperpolyhedron) in the prior knowledge-based (static, non-adaptive to data) SIAM decision tree, e.g. “strong” vegetation, equivalent to a spectral end-member (Adams et al., 1995). The pseudo-color of a spectral category (color name) is chosen to mimic natural RGB colors of pixels belonging to that spectral category (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b).

Figure 51. (a) False color quick-look Red-Green-Blue (RGB) Sentinel-2A Multi-Spectral Instrument (MSI) image Level-1C, calibrated into top-of-atmosphere reflectance (TOARF) values, depicting a surface area in SE Australia, acquired on 2019-12-31. In particular: R = Middle Infrared (MIR) = Band 11, G = Near Infrared (NIR) = Band 8, B = Visible Blue = Band 2. Spatial resolution: 10 m. No histogram stretching is applied for visualization purposes. (b) Same as Figure (a), followed by an Environment for Visualizing Images (ENVI, by L3Harris Geospatial) standard histogram stretching, applied for visualization purposes. (c) SIAM’s map of color names generated from Figure (a), consisting of 96 spectral categories (color names) depicted in pseudo-colors. Map of color names legend:

, shown in Figure 50. (d) Zoom-in of Figure (a), where no histogram stretching is applied for visualization purposes. (e) Zoom-in of Figure (b), where an ENVI standard histogram stretching is applied for visualization purposes. (f) Zoom-in of the SIAM’s map of color names shown in Figure (c), consisting of 96 spectral categories (color names) depicted in pseudo-colors, see Figure 50.

Figure 52. Multi-spectral (MS) signature in top-of-atmosphere reflectance (TOARF) values in range [0, 1], byte-coded into range {0, 255}, such that TOARF_byte = BYTE(TOARF_float * 255.0 + 0.5), where TOARF_byte ∈ {0, 255} is affected by a discretization (quantization) error = (TOARF_Max – TOARF_Min)/255 bins/2 (due to rounding to the closest integer, either above or below) = (1.0–0.0)/255.0/2.0 = 0.002 = 0.2%, to be considered negligible (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b) (refer to Subsection 3.3.2 in the Part 1). (a) Active fire samples (region of interest, ROI), MS signature in TOARF_byte values in range {0, 255}, Sentinel-2A Multi-Spectral Instrument (MSI) image Level-1C, of Australia, acquired on 2019-12-31 and shown in Figure 51, with bands 1 to 6 equivalent to Landsat-7 ETM+ bands 1 to 5 and 7, respectively. No thermal channel is available in Sentinel-2 MSI imagery, equivalent to the Landsat 7 ETM+ channels 61 and/or 62. (b) For comparison purposes with Figure (a), active fire samples, whose MS signature in TOARF_byte values belongs to range {0, 255}, are selected from a Landsat 7 ETM+ image of Senegal, Path: 203, Row: 051, acquisition date: 2001-01-11. Thermal band ETM62 in kelvin degrees in interval [−100, 155], linearly shifted into range {0, 255}. The two sensor-specific active fire spectral signatures in TOARF values shown in Figure (a) and Figure (b) should be compared with those collected by the geostationary GOES-16 ABI imaging sensor, shown in Figures 46 and 47.

Figure 53. Ideal Earth observation (EO) optical sensory data-derived Level 2/Analysis Ready Data (ARD) product generation system design as a hierarchical alternating sequence of: (A) hybrid (combined deductive and inductive) class-conditional radiometric enhancement of EO Level 1 multi-spectral (MS) top-of-atmosphere reflectance (TOARF) values into EO Level 2/ARD surface reflectance (SURF) 1-of-3, SURF 2-of-3, SURF 3-of-3 and surface albedo values (EC – European Commission, 2020; Li et al., 2012; Malenovsky et al., 2007; Schaepman-Strub et al., 2006; Shuai et al., 2020) corrected in sequence for (1) atmospheric, (5) topographic (6) adjacency and (7) bidirectional reflectance distribution function (BRDF) effects (Egorov, Roy, Zhang, Hansen, & Kommareddy, 2018), and (B) hybrid (combined deductive and inductive) classification of TOARF, SURF 1-of-3 to SURF 3-of-3 and surface albedo values into a stepwise sequence of EO Level 2/ARD scene classification maps (SCMs), whose legend (taxonomy) of community-agreed land cover (LC) class names, in addition to quality layers Cloud and Cloud–shadow, increases stepwise in mapping accuracy and/or in semantics, i.e. stepwise, it reaches deeper semantic levels/finer semantic granularities in a hierarchical LC class taxonomy, see Figure 3 in the Part 1. An instantiation of this EO image pre-processing system design for EO Level 2/ARD symbolic and subsymbolic co-products generation is depicted in Figure 54. In comparison with this desirable system design, the existing ESA Sen2Cor software system design (see Figure 38 in the Part 1) adopts no hierarchical alternating approach between MS image classification and MS image radiometric enhancement. In more detail, it accomplishes, first, one SCM generation from TOARF values based on a per-pixel (2D spatial context-insensitive) prior spectral knowledge-based decision-tree classifier (synonym for static/non-adaptive-to-data decision-tree for MS color naming, see Figures 29 and 30 in the Part 1). Next, in the ESA Sen2Cor workflow, a stratified/class-conditional MS image radiometric enhancement of TOARF into SURF 1-of-3 up to SURF 3-of-3 values corrected for atmospheric, topographic and adjacency effects is accomplished in sequence, stratified (class-conditioned) by (the haze map and cirrus map of) the same SCM product generated at first stage from TOARF values. In summary, the ESA Sen2Cor SCM co-product is TOARF-derived; hence, it is not “aligned” with data in the ESA Sen2Cor output MS image co-product, consisting of TOARF values radiometrically corrected into SURF 3-of-3 values (refer to Subsection 3.3.2 in the Part 1), where, typically, SURF ≠ TOARF holds, see Equation (9) in the Part 1.

Figure 54. The well-known engineering principles of modularity, hierarchy and regularity, recommended for system scalability (Lipson, 2007), characterize a single-date EO optical image processing system design for state-of-the-art multi-sensor EO data-derived Level 2/Analysis Ready Data (ARD) product generation, encompassing an ARD-specific symbolic co-product, known as Scene Classification Map (SCM) (refer to Subsection 8.1.1), and an ARD-specific numerical co-product (refer to Subsection 8.1.2) to be estimated alternately and hierarchically, see Figure 53. Stage 1: Absolute radiometric calibration (Cal) of dimensionless Digital Numbers (DNs) into top-of-atmosphere (TOA) radiance (TOARD) values ≥ 0. Stage 2: Cal of TOARD into TOA reflectance (TOARF) values ∈ [0.0, 1.0]. Stage 3: EO image classification by an automatic computer vision (CV) system, based on a convergence of spatial with colorimetric evidence (refer to Subsection 4.1 in the Part 1). Stage 4: Class-conditional/Stratified atmospheric correction of TOARF into surface reflectance (SURF) 1-of-3 values ∈ [0, 1]. Stage 5: Class-conditional/Stratified Topographic Correction (STRATCOR) of SURF 1-of-3 into SURF 2-of-3 values. Stage 6: Class-conditional/Stratified adjacency effect correction of SURF 2-of-3 into SURF 3-of-3 values. Stage 7: Class-conditional/Stratified bidirectional reflectance distribution function (BRDF) effect correction of SURF 3-of-3 values into surface albedo values (Bilal et al., 2019; EC – European Commission, 2020; Egorov, Roy, Zhang, Hansen, & Kommareddy, 2018; Franch et al., 2019; Li et al., 2012; Malenovsky et al., 2007; Qiu et al., 2019; Schaepman-Strub et al., 2006; Shuai et al., 2020). This original ARD system design is alternative to, for example, the EO image processing system design and implementation proposed in (Qiu et al., 2019) where, to augment the temporal consistency of USGS Landsat ARD imagery, neither topographic correction nor BRDF effect correction is land cover (LC) class-conditional. Worth mentioning, both SCM (referred to as land cover) and surface albedo (referred to as albedo) are included in the list of terrestrial Essential Climate Variables (ECVs) defined by the World Climate Organization (WCO) (Bojinski et al., 2014) (see Table 2 in the Part 1), which complies with the Group on Earth Observations (GEO)’s second implementation plan for years 2016–2025 of a new Global Earth Observation System of (component) Systems (GEOSS) as expert EO data-derived information and knowledge system (GEO – Group on Earth Observations, 2015; Nativi et al., 2015, 2020; Santoro et al., 2017), see Figure 1 in the Part 1.

Figure 55. Top left: Red-Green-Blue (RGB) image, source: Akiyoshi Kitaoka @AkiyoshiKitaoka, web page: http://nymag.com/selectall/2017/02/strawberries-look-red-without-red-pixels-color-constancy.html. Strawberries appear to be reddish, though the pixels are not, refer to the monitor-typical RGB input-output histograms shown at bottom left. No histogram stretching is applied for visualization purposes, see the monitor-typical RGB input-output histograms shown at bottom left. Top right: Output of the self-organizing statistical model-based color constancy algorithm, as reported in (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b; Baraldi et al., 2017; Vo et al., 2016), input with the image shown top left. No histogram stretching is applied for visualization purposes, see the monitor-typical RGB input-output histograms shown at bottom right.

Figure 56. Fully automated two-stage stratified (class-conditional) topographic correction (STRATCOR) (Baraldi et al., 2010). (a) Zoomed area of a Landsat 7 ETM+ image of Colorado, USA (path: 128, row: 021, acquisition date: 2000–08-09), depicted in false colors (R: band ETM5, G: band ETM4, B: band ETM1), 30 m resolution, radiometrically calibrated into TOARF values. (b) STRATCOR applied to the Landsat image shown at left, with data stratification based on the Shuttle Radar Topography Mission (SRTM) Digital Elevation Model (DEM) and a 18-class preliminary spectral map generated at coarse granularity by the 7-band Landsat-like Satellite Image Automatic Mapper (L-SIAM) software toolbox (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b), see Table 4.

Figure 57. (a) Meteosat Second Generation (MSG) Spinning Enhanced Visible and InfraRed Imager (SEVIRI) image acquired on 2012-05-30, radiometrically calibrated into TOARF values and depicted in false colors (R: band MIR, G: band NIR, B: band Blue), spatial resolution: 3 km. No histogram stretching is applied for visualization purposes. (b). Advanced Very High Resolution Radiometer (AVHRR)-like SIAM (AV-SIAM™, release 88 version 7) hyperpolyhedralization of the MS reflectance hyperspace and prior knowledge-based mapping of the input MS image into a vocabulary of hypercolor names. The AV-SIAM map legend, consisting of 83 spectral categories (see Table 4), is depicted in pseudo-colors, similar to those shown in Figure 50:

(c) To visualize contours of image-segments automatically detected in the SIAM’s multi-level map of color names by a deterministic two-pass connected-component multi-level image labeling algorithm (Dillencourt et al., 1992; Sonka et al., 1994), an automatic 4- or 8-adjacency cross-aura measure is estimated in linear time (see Figure 31 in the Part 1). (d) Segmentation map deterministically generated from the SIAM multi-level output map shown in (b). Each segment is identified by a monotonically increasing (from top to bottom) integer number. (e). Vegetation binary mask, automatically generated from the SIAM multi-level output map shown in (b). On the left, pixel candidates for vegetation belong to spectral category: vegetation in shadow. (f) Piecewise-constant image reconstruction, where each pixel is replaced by the mean reflectance value of the segment that pixel belongs to (known as “object-mean view” in the GEOBIA literature (Blaschke et al., 2014)). If the color quantization (vector quantization, VQ) error (equal to the per-pixel absolute difference between the input image and the piecewise-constant image reconstruction) is “low”, then the quality of the prior knowledge-based SIAM’s color space partitioning is “high”. No histogram stretching is applied for visualization purposes.

Figure 58. (a) Airborne 10 cm resolution true-color Red-Green-Blue (RGB) orthophoto of Trento, Italy, 4017 × 4096 pixels in size x 3 bands, acquired in 2014 and provided with no radiometric calibration metadata file. No histogram stretching is applied for visualization purposes. (b) Same RGB orthophoto, subject to self-organizing statistical color constancy (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b; Baraldi et al., 2017; Vo et al., 2016). No histogram stretching is applied for visualization purposes. (c) RGBIAM’s polyhedralization of the RGB color space and prior knowledge-based map of RGB color names generated from the RGB image, pre-processed by a color constancy algorithm (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b; Baraldi et al., 2017; Vo et al., 2016). The RGBIAM legend of RGB color names, consisting of 50 spectral categories at fine discretization granularity, is depicted in pseudo-colors. Map legend, shown in Figure 59:

. Input hyperparameters (to be user-defined): none. Processing time (one-pass, RGBIAM software implementation in the Interactive Data Language (IDL) by L3Harris Geospatial) = 2 min. (d) To visualize contours of image-segments automatically detected in the RGBIAM multi-level color map-domain by a deterministic two-pass connected-component multi-level image labeling algorithm (Dillencourt et al., 1992; Sonka et al., 1994), an automatic 4-adjacency and/or 8-adjacency cross-aura measure is estimated in linear time, see Figure 31 in the Part 1.

Figure 59. Legend (vocabulary) of RGB color names adopted by the prior knowledge-based Red-Green-Blue (RGB) Image Automatic Mapper™ (RGBIAM™, release 6 version 2) lightweight computer program for RGB data space polyhedralization (see Figure 29 in the Part 1), superpixel detection (see Figure 31 in the Part 1) and object-mean view (piecewise-constant input image approximation) quality assessment (Baraldi, 2017, 2019a; Baraldi et al., 2006; 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b). Before running RGBIAM to map onto a deterministic RGB color name each pixel value of an input 3-band RGB image, encoded in either true- or false-colors, the RGB image should be pre-processed (enhanced) for normalization/ harmonization/ calibration (Cal) purposes, e.g. by means of a color constancy algorithm (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b; Baraldi et al., 2017; Boynton, 1990; Finlayson et al., 2001; Gevers et al., 2012; Gijsenij et al., 2010; Vo et al., 2016), see Figures 55 and 58. The RGBIAM implementation (release 6 version 2) adopts 12 basic color (BC) names (polyhedra) as coarse RGB color space partitioning, consisting of the eleven BC names adopted by human languages (Berlin & Kay, 1969), specifically, black, white, gray, red, orange, yellow, green, blue, purple, pink and brown (refer to Subsection 4.2 in the Part 1), plus category “Unknowns” (refer to Section 2 in the Part 1), and 50 color names as fine color quantization (discretization) granularity, featuring parent-child relationships from the coarse to the fine quantization level. For the sake of representation compactness, pseudo-colors associated with the 50 color names (spectral categories, corresponding to a mutually exclusive and totally exhaustive partition of the RGB data space into polyhedra, see Figure 29 in the Part 1) are gathered along the same raw if they share the same parent spectral category (parent polyhedron) in the prior knowledge-based (static, non-adaptive to data) RGBIAM decision tree. The pseudo-color of a spectral category (color name) is chosen to mimic natural RGB colors of pixels belonging to that spectral category (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b; Baraldi et al., 2017; Vo et al., 2016).

Figure 60. First test image of a Cambodia site (Baraldi & Tiede, 2018b). Final 3-level Cloud/Cloud–shadow/Others maps generated by the three algorithms under comparison, specifically, single-date AutoCloud+, the single-date AutoCloud+ Baraldi & Tiede, 2018a, 2018b), the single-date Sen2Cor (DLR - Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies, 2011; ESA - European Space Agency, 2015) and the multi-date MAJA (Hagolle et al., 2017; Main-Knorn et al., 2018), where class “Others” is overlaid with the input Sentinel-2 A Multi-Spectral Instrument (MSI) Level 1C image, radiometrically calibrated into top-of-atmosphere reflectance (TOARF) values and depicted in false-colors: monitor-typical Red-Green-Blue (RGB) channels are selected as R = Near Infrared (NIR) channel, G = Visible Red channel, B = Visible Green channel. Histogram stretching is applied for visualization purposes. Output class Cloud is shown in a green pseudo-color, class Cloud–shadow in a yellow pseudo-color.

Figure 61. First test image of a Cambodia site (Baraldi & Tiede, 2018b). Zoom-in of the final 3-level Cloud/Cloud–shadow/Others maps generated by the three algorithms under comparison, where class “others” is overlaid with the input Sentinel-2 A Multi-Spectral Instrument (MSI) Level 1C image, radiometrically calibrated into top-of-atmosphere (TOA) reflectance (TOARF) values and depicted in false-colors: monitor-typical RGB channels are selected as R = Near InfraRed (NIR) channel, G = Visible Red channel, B = Visible Green channel. Histogram stretching is applied for visualization purposes. Output class Cloud is shown in a green pseudo-color, class Cloud–shadow in a yellow pseudo-color. Based on qualitative photointerpetation, the single-date Sen2Cor algorithm (DLR - Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies, 2011; ESA - European Space Agency, 2015) appears to underestimate Cloud–shadows, although some Water areas are misclassified as Cloud–shadow. In addition, some River/river beds are misclassified as Cloud. These two cases of Cloud false positives and Cloud–shadow false positives are highlighted in blue circles. The multi-date MAJA algorithm (Hagolle et al., 2017; Main-Knorn et al., 2018), overlooks some Cloud instances small in size (in relative terms), as highlighted in red circles.

Figure 62. First test image of a Cambodia site (Baraldi & Tiede, 2018b). Extra zoom-in of the final 3-level Cloud/Cloud–shadow/Others maps generated by the three algorithms under comparison, where class “Others” is overlaid with the input Sentinel-2 A Multi-Spectral Instrument (MSI) Level 1C image, radiometrically calibrated into top-of-atmosphere (TOA) reflectance (TOARF) values and depicted in false-colors: monitor-typical RGB channels are selected as R = Near InfraRed (NIR) channel, G = Visible Red channel, B = Visible Green channel. Histogram stretching is applied for visualization purposes. Output class Cloud is shown in a green pseudo-color, class Cloud–shadow in a yellow pseudo-color. Based on qualitative photointerpetation, the single-date Sen2Cor algorithm (DLR - Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies, 2011; ESA - European Space Agency, 2015) appears to underestimate class Cloud–shadow, although some detected Cloud–shadow instances are false positives because of misclassified Water areas. In addition, some River/river beds are misclassified as Cloud. To reduce false positives in Cloud–shadow detection, MAJA adopts a multi-date approach. Nevertheless, the multi-date MAJA algorithm (Hagolle et al., 2017; Main-Knorn et al., 2018), misses some instances of Cloud-over-water. Overall, MAJA’s Cloud and Cloud–shadow results look more “blocky” (affected by artifacts in localizing true boundaries of target image-objects).

Figure 63. Typical values of a minimally dependent maximally informative (mDMI) set of outcome and process quantitative quality indicators (OP-Q²Is) featured by a pixel-based (2D spatial context-insensitive) 1D image analysis algorithm (see Figure 18 in the Part 1) for Cloud and Cloud-shadow detection in MS imagery (see Figure 22 in the Part 1), such as Fmask (Zhu et al., 2015), ESA Sen2Cor (DLR – Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies, 2011; ESA – European Space Agency, 2015) and CNES-DLR MAJA (Hagolle et al., 2017; Main-Knorn et al., 2018) (see Table 5), considered a mandatory EO image understanding (classification) task for quality layers detection in Analysis Ready Data (ARD) workflows.

Figure 64. Typical values of a minimally dependent maximally informative (mDMI) set of outcome and process quantitative quality indicators (OP-Q²Is) featured by a deep convolutional neural network (DCNN) (Cimpoi et al., 2014) involved with Cloud and Cloud-shadow detection in MS imagery (Bartoš, 2017; EOportal, 2020; Wieland et al., 2019) (see Table 5), considered a mandatory EO image understanding (classification) task for quality layers detection in Analysis Ready Data (ARD) workflows.

Figure 65. Expected values of a minimally dependent maximally informative (mDMI) set of outcome and process quantitative quality indicators (OP-Q²Is) featured by a “universal” multi-sensor automatic AutoCloud+ algorithm for Cloud and Cloud-shadow detection in multi-spectral (MS) imagery (Baraldi & Tiede, 2018a, 2018b) (see Table 5), considered a mandatory EO image understanding (classification) task for quality layers detection in Analysis Ready Data (ARD) workflows. By scoring “high” in each indicator of an mDMI set of OP-Q²Is, a “universal” multi-sensor automated AutoCloud+ algorithm for Cloud and Cloud-shadow quality layers detection in MS imagery is expected to be considered in operational mode (Baraldi, 2017; Baraldi & Tiede, 2018a, 2018b), as necessary-but-not-sufficient precondition of ARD workflows.

Table 4. The Satellite Image Automatic Mapper™ (SIAM™) lightweight computer program (release 88 version 7) for multi-spectral (MS) reflectance space hyperpolyhedralization (see Figures 29 and 30 in the Part 1), superpixel detection (see Figure 31 in the Part 1) and object-mean view quality assessment (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b). SIAM is an Earth observation (EO) system of systems scalable to any past, present or future MS imaging sensor, provided with radiometric calibration metadata parameters for radiometric calibration (Cal) of digital numbers into top-of-atmosphere reflectance (TOARF), surface reflectance (SURF) or surface albedo values, where relationship ‘TOARF ⊇ SURF ⊇ Surface albedo’ = Equation (8) in the Part 1 holds. SIAM is an expert system (deductive/ top-down/ prior knowledge-based static decision tree, non-adaptive to input data) for mapping each color value in the MS reflectance space into a color name belonging to the SIAM vocabulary of hypercolor names. It consists of the following subsystems. (i) 7-band Landsat-like SIAM™ (L-SIAM™), with Landsat-like input channels Blue (B) = Blue_0.45÷0.50, Green (G) = Green_0.54÷0.58, Red (R) = Red_0.65÷0.68, Near Infrared (NIR) = NIR_0.78÷0.90, Middle Infrared 1 (MIR1) = MIR_1.57÷1.65, Middle Infrared 2 (MIR2) = MIR_2.08÷2.35, and Thermal Infrared (TIR) = TIR_10.4–12.5, see Figure 7 and Table 3 in the Part 1. (ii) 4-band (channels G, R, NIR, MIR1) SPOT-like SIAM™ (S-SIAM™). (iii) 4-band (channels R, NIR, MIR1, and TIR) Advanced Very High Resolution Radiometer (AVHRR)-like SIAM™ (AV-SIAM™). (iv) 4-band (channels B, G, R, and NIR) QuickBird-like SIAM™ (Q-SIAM™) (Baraldi, 2017, 2019a; Baraldi et al., 2006, 2010a, 2010b, 2018a, 2018b; Baraldi & Tiede, 2018a, 2018b).

SIAM, r88v7	Input bands	Number of output spectral categories = Cardinality of an a priori knowledge-based legend/ vocabulary/ taxonomy of multi-spectral (MS) color names, corresponding to a discrete and finite set of mutually exclusive and totally exhaustive (hyper)polyhedra in a MS data (hyper)space
		Fine discretization levels (fine hyperpolyhedralization of the MS data space, see Figures 29 and 30 in the Part 1, featuring parent-child hyperpolyhedra relationships, from intermediate to fine discretization levels)	Intermediate discretization levels (featuring parent-child hyperpolyhedra relationships, from coarse to intermediate discretization levels)	Coarse discretization levels (featuring parent hyperpolyhedra, exclusively)	Inter-sensor discretization levels (*)
L-SIAM	7 – B, G, R, NIR, MIR1, MIR2, TIR	96	48	18	33 (*): employed for inter-sensor post-classification change/no-change detection
S-SIAM	4 – G, R, NIR, MIR1	68	40	15
AV-SIAM	4 – R, NIR, MIR1, TIR	83	43	17
Q-SIAM	4 – B, G, R, NIR	61	28	12

Note: Landsat-like spectral channels: B = Blue_0.45÷0.50, G = Green_0.54÷0.58, R = Red_0.65÷0.68, Near Infrared = NIR = NIR_0.78÷0.90, Middle Infrared 1 = MIR1 = MIR_1.57÷1.65, Middle Infrared 2 = MIR2 = MIR_2.08÷2.35, Thermal Infrared = TIR = TIR_10.4–12.5. See Figure 7 and Table 3 in the Part 1.

Table 5. Conceptual/methodological comparison of existing algorithms for Cloud and Cloud-shadow quality layers detection in spaceborne MS imagery for EO Level 2/ARD-specific symbolic output co-product generation.

Existing Cloud and Cloud-shadow quality layers detector	2D (retinotopic)/1D (non-retinotopic) image analysis, either pixel-based (2D spatial context-insensitive) or local window-based	Deductive (model-driven)/ Inductive (data-driven)/ Hybrid (combined inductive and deductive)	Single-date/ Multi-temporal	Feedback connections/ End-to-end Feedforward	Modularity, Hierarchy, Regularity (MHR) design criteria (recommended for scalability and interpretability)/ Black-box system (affected by low interpretatbility)	Expected mDMI set of OP-Q^2Is
Fmask (Zhu et al., 2015)	1D, pixel-based (spatial context – insensitive)	Deductive	Single-date and multi-temporal versions	Feedforward	MHR	See Figure 63
ESA Sen2Cor (DLR – Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies, 2011; ESA – European Space Agency, 2015)	1D, pixel-based	Deductive	Single-date	Feedforward	MHR
CNES-DLR MAJA (Hagolle et al., 2017)	1D, pixel-based	Deductive	Multi-temporal	Feedforward	MHR
DCNN (Bartoš, 2017)	2D spatial context-sensitive/ local window-based, but spatial topology non-preserving (non-retinotopic)	Inductive	Single-date and multi-temporal versions	Feedforward	Black box	See Figure 64
AutoCloud+ (Baraldi & Tiede, 2018a, 2018b)	2D (retinotopic)	Hybrid	Single-date	Feedback	MHR	See Figure 65

Rocha de Carvalho, L. F. (2019). An Automatic Spectral Rule-Based System for Real-Time Thermal Anomalies Detection Using GOES-16 ABI Data. Master’s thesis. Sapienza University of Rome, Italy.

 Google Scholar

Baraldi, A. (2017). Pre-processing, classification and semantic querying of large-scale Earth observation spaceborne/airborne/terrestrial image databases: Process and product innovations. Ph.D. dissertation in Agricultural and Food Sciences, University of Naples “Federico II”, Department of Agricultural Sciences, Italy. Ph.D. defense: 16 May 2017. Accessed 12 Oct. 2017. Retrieved from http://www.fedoa.unina.it/11724/1/TesiDottorato_Unina_XXIX_AB_EO-IU4SQ_v4_RGB_10April2017.pdf

 Google Scholar

Baraldi, A. (2019a). Satellite Image Automatic Mapper™ (SIAM™) as a Service (SIAMaaS) Demonstrator installed onto the Serco’s ONDA Data and Information Access Services (DIAS) Marketplace. Accessed 9 Jan 2020. Retrieved from https://indoors2001.zgis.at/portal/apps/MapJournal/index.html?appid=1ab1e4a63ce24da2b69e5cfa3e4f1907

 Google Scholar

Baraldi, A., Humber, M. L., Tiede, D., & Lang, S. (2018a). GEO-CEOS stage 4 validation of the satellite image automatic mapper lightweight computer program for esa earth observation level 2 product generation – Part 1: Theory. Cogent Geoscience, 4(1). doi:10.1080/23312041.2018.1467357

 Web of Science ®Google Scholar

Baraldi, A., Humber, M. L., Tiede, D., & Lang, S. (2018b). GEO-CEOS stage 4 validation of the satellite image automatic mapper lightweight computer program for esa earth observation level 2 product generation – Part 2: Validation. Cogent Geoscience, 4(1). doi:10.1080/23312041.2018.1467254

 Web of Science ®Google Scholar

Baraldi, A., Puzzolo, V., Blonda, P., Bruzzone, L., & Tarantino, C. (2006). Automatic spectral rule-based preliminary mapping of calibrated Landsat TM and ETM+ images. IEEE Trans. Geosci. Remote Sens, 44(9), 2563–2586. doi:10.1109/TGRS.2006.874140

 Web of Science ®Google Scholar

Baraldi, A., & Tiede, D. (2018a). AutoCloud+, a “universal” physical and statistical model-based 2D spatial topology-preserving software for cloud/cloud–shadow detection in multi-sensor single-date earth observation multi-spectral imagery – Part 1: Systematic ESA EO Level 2 product generation at the ground segment as broad context. ISPRS Int. J. Geo-Inf, 7(12), 457. doi:10.3390/ijgi7120457. Accessed 8 Jan. 2020. Retrieved from http://www.mdpi.com/2220-9964/7/12/457/pdf

 Web of Science ®Google Scholar

Baraldi, A., & Tiede, D. (2018b). AutoCloud+, a “universal” physical and statistical model-based 2D spatial topology-preserving software for cloud/cloud–shadow detection in multi-sensor single-date earth observation multi-spectral imagery – Part 2: Outcome and process requirements specification, information/ knowledge representation, system design, algorithm, implementation and preliminary experimental results. ISPRS Int. J. Geo-Inf, 7(12), 457. doi:10.3390/ijgi7120457. Accessed 8 Jan. 2020. Retrieved from https://www.mdpi.com/2220-9964/7/12/457#supplementary

 Web of Science ®Google Scholar

Pellegrini, L., Natali, S., & Baraldi, A. (2008). SOMAFID: A second level stratified image processing system for thermal anomalies identification in (A)ATSR imagery. In 2nd MERIS-(A)ATSR Workshop, 22-26 Sep. 2008, Frascati, Italy. Accessed 13 Mar. 2021. Retrieved from https://www.researchgate.net/publication/290557347_SOMAFID_A_second_level_stratified_image_processing_system_for_thermal_anomalies_identification_in_AATSR_imagery

 Google Scholar

Baraldi, A., Durieux, L., Simonetti, D., Conchedda, G., Holecz, F., & Blonda, P. (2010a). Automatic spectral rule-based preliminary classification of radiometrically calibrated SPOT-4/-5/IRS, AVHRR/MSG, AATSR, IKONOS/QuickBird/OrbView/GeoEye and DMC/SPOT-1/-2 imagery – Part I: System design and implementation. IEEE Trans. Geosci. Remote Sens, 48(3), 1299–1325. doi:10.1109/TGRS.2009.2032457

 Web of Science ®Google Scholar

Baraldi, A., Durieux, L., Simonetti, D., Conchedda, G., Holecz, F., & Blonda, P. (2010b). Automatic spectral rule-based preliminary classification of radiometrically calibrated SPOT-4/-5/IRS, AVHRR/MSG, AATSR, IKONOS/QuickBird/OrbView/GeoEye and DMC/SPOT-1/-2 imagery – Part II: Classification accuracy assessment. IEEE Trans. Geosci. Remote Sens, 48(3), 1326–1354. doi:10.1109/TGRS.2009.2032064

 Web of Science ®Google Scholar

Berlin, B., & Kay, P. (1969). Basic Color terms: Their universality and evolution. Berkeley, CA, USA: University of California.

 Google Scholar

Adams, J. B., Donald, E. S., Kapos, V., Almeida Filho, R., Roberts, D. A., Smith, M. O., & Gillespie, A. R. (1995). Classification of multispectral images based on fractions of endmembers: Application to land-cover change in the Brazilian Amazon. Remote Sens. Environ, 52(2), 137–154. doi:10.1016/0034-4257(94)00098-8

 Web of Science ®Google Scholar

EC - European Commission. (2020). Copernicus Global Land Service - Providing bio-geophysical products of global land surface, Energy Products. Accessed 25 Apr. 2020. Retrieved from https://land.copernicus.eu/global/themes/Energy

 Google Scholar

Li, F.Q., Jupp, D.L.B., Thankappan, M., Lymburner, L., Mueller, N., Lewis, A., & Held, A. (2012). A physics-based atmospheric and BRDF correction for landsat data over mountainous terrain. Remote Sens. Environ, 124, 756–770. doi:10.1016/j.rse.2012.06.018

 Web of Science ®Google Scholar

Malenovsky, Z., Bartholomeus, H. M., Acerbi-Junior, F. W., Schopfer, J. T., Painter, T. H., Epema, G. F., & Bregt, A. K. (2007). Scaling dimensions in spectroscopy of soil and vegetation. Int. J. Applied Earth Obs. And Geoinfo, 9(2), 137–164. doi:10.1016/j.jag.2006.08.003

 Web of Science ®Google Scholar

Schaepman-Strub, G., Schaepman, M. E., Painter, T. H., Dangel, S., & Martonchik, J. V. (2006). Reflectance quantities in optical remote sensing—Definitions and case studies. Remote Sens. Environ, 103(1), 27–42. doi:10.1016/j.rse.2006.03.002

 Web of Science ®Google Scholar

Shuai, Y., Tuerhanjiang, L., Shao, C., Gao, F., Zhou, Y., Xie, D., … Chu, N. (2020). Re-understanding of land surface albedo and related terms in satellite-based retrievals. Big Earth Data, 4(1), 45–67. doi:10.1080/20964471.2020.1716561

 Web of Science ®Google Scholar

Main-Knorn, M., Louis, J., Hagolle, O., Müller-Wilm, U., & Alonso, K. (2018). The Sen2Cor and MAJA cloud masks and classification products. In Proc. of the 2nd Sentinel-2 Validation Team Meeting. Rome, Italy: ESA-ESRIN, Frascati. 29–31 January 2018. Accessed 4 Feb. 2021. Retrieved from https://elib.dlr.de/119324/

 Google Scholar

Lipson, H. (2007). Principles of modularity, regularity, and hierarchy for scalable systems. J. Biol. Phys. Chem, 7(4), 125–128. doi:10.4024/40701.jbpc.07.04

 Google Scholar

Bilal, M., Nazeer, M., Nichol, J. E., Bleiweiss, M. P., Qiu, Z., Jäkel, E., … Lolli, S. (2019). A simplified and robust surface reflectance estimation method (SREM) for use over diverse land surfaces using multi-sensor data. Remote Sens, 11(1344), 1–24. doi:10.3390/rs11111344

 Google Scholar

Franch, B., Vermote, E., Skakun, S., Roger, J., Masek, J., Junchang, J., … Santamaria-Artigas, A. (2019). A method for landsat and sentinel 2 (HLS) BRDF normalization. Remote Sens, 11(632), 1–18. doi:10.3390/rs11060632

 Google Scholar

Qiu, S., Lin, Y., Shang, R., Zhang, J., Ma, L., & Zhu, Z. (2019). Making Landsat time series consistent: Evaluating and improving landsat analysis ready data. Remote Sens, 11(51), 1–21.

 Google Scholar

Bojinski, S., Verstraete, M., Peterson, T. C., Richter, C., Simmons, A., & Zemp, M. (2014). The concept of essential climate variables in support of climate research, applications and policy. American Meteorological Society, 9(9), 1431–1443. doi:10.1175/BAMS-D-13-00047.1

 Google Scholar

GEO - Group on Earth Observations. (2015). GEO Strategic Plan 2016-2025: Implementing GEOSS. Accessed 5 May 2021. Retrieved from https://earthobservations.org/documents/open_eo_data/GEO_Strategic_Plan_2016_2025_Implementing_GEOSS.pdf

 Google Scholar

Nativi, S., Mazzetti, P., Santoro, M., Papeschi, F., Craglia, M., & Ochiai, O. (2015). Big data challenges in building the global earth observation system of systems. Environmental Modelling and Software, 68, 1–26. doi:10.1016/j.envsoft.2015.01.017

 Web of Science ®Google Scholar

Nativi, S., Santoro, M., Giuliani, G., & Mazzetti, P. (2020). Towards a knowledge base to support global change policy goals. Int. J. Digital Earth, 13(2), 188–216. doi:10.1080/17538947.2018.1559367

 Web of Science ®Google Scholar

Santoro, M., Nativi, S., Maso, J., & Jirka, S. (2017). Observation Inventory Description and Results Report. Deliverable D4.2 of H2020 project ConnectinGEO. Accessed 22 Feb. 2019. Retrieved from https://ddd.uab.cat/record/146891

 Google Scholar

Baraldi, A., Tiede, D., & Lang, S. (2017). Automated linear-time detection and quality assessment of superpixels in uncalibrated true- or false-color RGB images. arXiv: 1701.01940. Accessed 4 January 2020. Retrieved from https://arxiv.org/ftp/arxiv/papers/1701/1701.01940.pdf

 Google Scholar

Vo, A.-V., Truong-Hong, L., Laefer, D. F., Tiede, D., d’Oleire-Oltmanns, S., Baraldi, A., & Shimoni, M. (2016). Processing of extremely high resolution LiDAR and optical data: Outcome of the 2015 IEEE GRSS data fusion contest. part–B: 3D contest. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens, 9(12), 5560–5575. doi:10.1109/JSTARS.2016.2581843

 Google Scholar

Baraldi, A., Gironda, M., & Simonetti, D. (2010). Operational two-stage stratified topographic correction of spaceborne multi-spectral imagery employing an automatic spectral rule-based decision-tree preliminary classifier. IEEE Trans. Geosci. Remote Sens, 48(1), 112–146. doi:10.1109/TGRS.2009.2028017

 Web of Science ®Google Scholar

Dillencourt, M. B., Samet, H., & Tamminen, M. (1992). A general approach to connected component labeling for arbitrary image representations. J. Assoc. Computing Machinery, 39(2), 253–280. doi:10.1145/128749.128750

 Web of Science ®Google Scholar

Sonka, M., Hlavac, V., & Boyle, R. (1994). Image Processing, Analysis and Machine Vision. London, UK: Chapman and Hall.

 Google Scholar

Blaschke, T., Hay, G. J., Kelly, M., Lang, S., Hofmann, P., Addink, E., … Tiede, D. (2014). Geographic object-based image analysis - towards a new paradigm. ISPRS J. Photogram. Remote Sens, 87, 180–191. doi:10.1016/j.isprsjprs.2013.09.014

 PubMed Web of Science ®Google Scholar

Boynton, R. M. (1990). Human color perception. In K. N. Leibovic (Ed.), Science of Vision (pp. 211–253). New York, NY, USA: Springer.

 Google Scholar

Finlayson, G. D., Hordley, S. D., & Hubel, P. M. (2001). Color by correlation: A simple, unifying framework for color constancy. IEEE Trans. Pattern Anal. Mach. Intell, 23(11), 1209–1221. doi:10.1109/34.969113

 Web of Science ®Google Scholar

Gevers, T., Gijsenij, A., van de Weijer, J., & Geusebroek, J.-M. (2012). Color in Computer Vision. Hoboken, NJ, USA: Wiley.

 Google Scholar

Gijsenij, A., Gevers, T., & van de Weijer, J. (2010). Computational color constancy: Survey and experiments. IEEE Trans. Image Proc., 20, 2475–2489.

 Google Scholar

DLR - Deutsches Zentrum für Luft-und Raumfahrt e.V. and VEGA Technologies. (2011). Sentinel-2 MSI–Level 2A Products Algorithm Theoretical Basis Document. Document S2PAD-ATBD-0001. Paris, France: European Space Agency.

 Google Scholar

ESA - European Space Agency (2015). Sentinel-2 User Handbook. Standard Document, European Space Agency: Paris, France. Accessed 3 Feb. 2016. Retrieved from https://sentinels.copernicus.eu/documents/247904/685211/Sentinel-2_User_Handbook.pdf/8869acdf-fd84-43ec-ae8c-3e80a436a16c?t=1438278087000

 Google Scholar

Hagolle, O., Huc, M., Desjardins, C., Auer, S., & Richter, R. (2017). MAJA Algorithm Theoretical Basis Document. CNES-CESBIO and DLR, December 7. Accessed 16 Jan. 2018. Retrieved from https://zenodo.org/record/1209633#.W2ffFNIzZaQ

 Google Scholar

Zhu, Z., Wang, S., & Woodcock, C. E. (2015). Improvement and expansion of the fmask algorithm: cloud, cloud-shadow, and snow detection for landsats 4–7, 8, and sentinel 2 images. Remote Sens. Environ, 159, 269–277. doi:10.1016/j.rse.2014.12.014

 Web of Science ®Google Scholar

Cimpoi, M., Maji, S., Kokkinos, I., & Vedaldi, A. (2014). Deep filter banks for texture recognition, description, and segmentation. Int. J. Comput. Vis, 113(1), 3–18. doi:10.1007/s11263-014-0790-9

 PubMedGoogle Scholar

Bartoš, M. (2017). Cloud and shadow detection in satellite imagery. Master’s Thesis, Computer Vision and Image Processing, Faculty of Electrical Engineering, Department of Cybernetics, Czech Technical University, Prague, Czech Republic.

 Google Scholar

EOportal. (2020). PhiSat-1 Nanosatellite Mission. Accessed 23 Sept. 2020. Retrieved from https://directory.eoportal.org/web/eoportal/satellite-missions/p/phisat-1

 Google Scholar

Wieland, M., Li, Y., & Martinis, S. (2019). Multi-sensor cloud and cloud shadow segmentation with a convolutional neural network. Remote Sens. Environment, 230, 111203. doi:10.1016/j.rse.2019.05.022

 Web of Science ®Google Scholar

Data availability statement

Data sharing is not applicable to this article as no new data were created or analyzed in this study.