Cloud restoration of optical satellite imagery using time-series spectral similarity group

Figures & data

Table 1. Band specifications of Landsat 8–9 OLI/TIRS sensors.

Bands	Wavelength ($\mathit{\mu m}$)	Resolution ($\mathit{m}$)
Band 1 – Coastal Aerosol	0.43–0.45	30
Band 2 – Blue	0.45–0.51	30
Band 3 – Green	0.53–0.59	30
Band 4 – Red	0.64–0.67	30
Band 5 – Near InfraRed (NIR)	0.85–0.88	30
Band 6 – Short Wavelength InfraRed (SWIR) 1	1.57–1.65	30
Band 7 – Short Wavelength InfraRed (SWIR) 2	2.11–2.29	30
Band 8 – Panchromatic	0.50–0.68	15
Band 9 – Cirrus	1.36–1.38	30
Band 10 – Thermal InfraRed (TIRS) 1	10.6–11.19	100
Band 11 – Thermal InfraRed (TIRS) 2	11.50–12.51	100

Figure 1. Experimental areas with diverse land cover types in South Korea and the United States (images shown from May 7, 2020, and July 4, 2022, respectively).

Figure 2. Workflow of the cloud detection and TSSG restoration algorithm.

Table 2. Pixel quality classification types from Landsat QA band.

Bit	Flag Description	Values
0	Fill	0 for image data 1 for fill data
1	Dilated cloud	0 for cloud is not dilated or no cloud 1 for cloud dilation
2	Cirrus	0 for cirrus confidence: no confidence level set or low confidence 1 for high confidence cirrus
3	Cloud	0 for cloud confidence is not high 1 for high confidence cloud
4	Cloud shadow	1 for cloud shadow confidence is not high 1 for high confidence cloud shadow
5	Snow	0 for snow/ice confidence is not high 1 for high confidence snow cover
6	Clear	0 if cloud or dilated cloud bits are set 1 if cloud and dilated cloud bits are not set
7	Water	0 for land or cloud 1 for water
8–9	Cloud confidence	00 for no confidence level set 01 low confidence 10 medium confidence 11 high confidence
10–11	Cloud shadow confidence	00 for no confidence level set 01 low confidence 10 reserved 11 high confidence
12–13	Snow/Ice confidence	00 for no confidence level set 01 low confidence 10 reserved 11 high confidence
14–15	Cirrus confidence	00 for no confidence level set 01 low confidence 10 reserved 11 high confidence

Figure 3. Sample images of cloud maps according to cloud confidence from QA image: (a) cloud RGB image acquired on April 4, 2020, in path/row 115/35, (b) cloud map A, (c) cloud map B, (d) cloud map C, (e) cloud map D, and (f) cloud map E.

Table 3. Cloud maps based on the level of cloud confidence from QA image.

Cloud Map	Attributes
A	All attributes except “Clear”
B	“Cloud-high,” “Cloud-medium,” “Cloud-low,” “Cirrus-high,” “Cirrus-low”
C	“Cloud-high,” “Cloud-medium,” “Cirrus-high”
D	“Cloud-high,” “Cirrus-high”
E	“Cloud-high”

Figure 4. Proposed TSSG cloud regions restoration technique.

Table

Algorithm 1. TSSG cloud regions restoration method

Require: Pre-processed time-series satellite imagery dataset D. Select the target cloud image $\mathit{I}$ and number of $\mathit{N}$ reference images $R_n$ $\left(\mathit{n}=1,\dots ,\mathit{N}\right)$ from D. Main algorithm 1: Select one cloud pixel $\mathit{i}$ in $\mathit{I}$ to be restored. 2: Generate difference image between pixel value $\mathit{i}$ in reference image and entire reference image by Equation (3)(3) $\mathrm{\Delta }\mathit{I}\left(\mathit{i}\right)=\left[{\mathrm{\Delta }}_1\left(\mathit{i}\right),{\mathrm{\Delta }}_2\left(\mathit{i}\right),\dots {\mathrm{\Delta }}_n\left(\mathit{i}\right),\dots ,{\mathrm{\Delta }}_N\left(\mathit{i}\right)\right],$(3) , (4(4) ${\mathrm{\Delta }}_n\left(\mathit{i}\right)=R_n-R_n\left(\mathit{i}\right),$(4) ). 3: Generate integrated difference value image to extract similarity group using Equation (5)(5) $\overline{\mathrm{\Delta }I\left(i\right)}=\sqrt{\frac{{\sum }_{n=1}^N{\left({\mathrm{\Delta }}_n\left(\mathit{i}\right)\right)}^2}{\mathit{N}}},$(5) . 4: Extract time-series spectral similarity group pixels using integrated difference image. 5: Restore the pixel value of $\mathit{i}$ using similarity group pixels located in non-cloud region by Equation (6)(6) $\widehat{\mathit{I}\left(\mathit{\iota }\right)}=\frac{{\sum }_{m=1}^M\mathit{TSS}{G}_m}{\mathit{M}},$(6) . 6: Repeat for all cloud pixels and all bands of the target image. Output: Image that restored cloud regions

Table 4. Equation of indices for evaluating the accuracy of cloud region restoration.

Index	Equation
NRMSE (Normalized Root Mean Square Error)	$\mathit{NRMSE}\left(I_r,I_p\right)=\frac{1}{\overline{I_r}}\sqrt{\frac{{\sum }_{I_r{I}_p}{\left(I_r-I_p\right)}^2}{I_r{I}_p}}$
MAPE (Mean Absolute Percentage Error)	$\mathrm{MAPE}($$I_r,I_p)=\frac{{\sum }_{{\mathit{I}}_{\mathit{width}},I_{\mathit{height}}}\left|\frac{I_r-I_p}{I_r}\right|}{I_{\mathit{width}}{I}_{\mathit{height}}}\times 100$
PSNR (Peak Signal-to-Noise Ratio)	$\mathit{PSNR}\left(I_r,I_p\right)=10{log}_{10}\frac{{\left(2^n-1\right)}^2}{\mathit{MSE}\left(I_r,I_p\right)}$ $\mathit{MSE}\left(I_r,I_p\right)=\frac{{\sum }_{I_r{I}_p}{\left(I_r-I_p\right)}^2}{I_r{I}_p}$
SSIM (Structural Similarity Index Measure)	$\mathit{SSIM}\left(I_r,I_p\right)=\mathit{l}\left(I_r,I_p\right)\mathit{c}\left(I_r,I_p\right)\mathit{s}\left(I_r,I_p\right)$ $\mathit{l}\left(I_r,I_p\right)=\frac{2{\mu }_{I_r}{\mu }_{I_p}+C_1}{{\mu }_{I_r}^2+{\mu }_{I_p}^2+C_1},\mathit{c}\left(I_r,I_p\right)=\frac{2{\sigma }_{I_r}{\sigma }_{I_p}+C_2}{{\sigma }_{I_r}^2+{\sigma }_{I_p}^2+C_2},\mathit{s}\left(I_r,I_p\right)=\frac{{\sigma }_{I_r{I}_p}+C_3}{{\sigma }_{I_r}{\sigma }_{I_p}+C_3}$

$I_r$= reference image, $I_p$= reconstruction image, $I_{\mathit{width}}$= width of image(s), $I_{\mathit{height}}$= height of image(s)

$C_1$, $C_2$, and $C_3$= stabilizing constants, $\mathit{n}$ = bit-depth of the image

Table 5. SSIM, PSNR, NRMSE, and MAPE values of SSG and TSSG algorithms-based reconstruction images at two sites.

	Site 1 (South Korea)				Site 2 (USA)
	SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Cloud image	0.1536	28.4584	0.2727	6.2482	0.0457	23.7896	0.1738	13.5678
SSG	0.5241	40.7478	0.0682	1.2664	0.4485	28.6493	0.1275	18.9369
TSSG	0.8927	52.1979	0.0184	0.2249	0.8859	40.3423	0.0313	4.3484

Figure 5. Visual comparison examples of true-color restoration results in site 1, South Korea. The second and third lines of each figure depict enlarged images of the red rectangle regions: (a) Landsat 8 original cloud-free images on March 20, 2020, (b) cloud simulation images, (c) restored images using SSG, and (d) restored images using proposed TSSG (Landsat 8 on November 29, 2019 and April 5, 2020 were used for restoration as reference data).

Figure 6a. Visual comparison examples of true-color restoration results in site 2, USA. The second and third lines of each figure depict enlarged images of the red rectangle regions: (a) Landsat 8 original cloud-free images on September 25, 2023, (b) cloud simulation images, (c) restored images using SSG, and (d) restored images using proposed TSSG (Landsat 9 images on August 29, 2022 and October 19, 2023 were used for restoration as reference data).

Figure 6b. (Continued)

Figure 7a. Visual comparison examples of restoration results. The second and third lines of each figure depict enlarged images of the yellow rectangle region: (a) Landsat 8 images on March 4, 2020, South Korea, (b) Landsat 8 images on December 17, 2020, South Korea, (c) Landsat 9 images on April 3, 2022, USA.

Figure 7b. (Continued)

Figure 8. Experimental true-color images from Landsat 8 with abrupt land cover change by wildfire in South Korea. The second lines of each figure depict enlarged images of the red rectangle regions: (a) cloud simulation image on 31 December 2019, (b) first reference image on 5 April 2020, (c) second reference image on 8 June 2020 including regions affected by wildfires.

Table 6. SSIM, PSNR, NRMSE, and MAPE values of TSSG reconstructed images according to the number of reference images in two sites.

		Site 1				Site 2
		SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Number of reference images for TSSG	1	0.6634	46.9554	0.0344	0.5949	0.6390	31.6004	0.0539	7.7915
	2	0.8927	52.1979	0.0184	0.2249	0.8859	40.3423	0.0313	4.3484
	3	0.8913	52.0338	0.0187	0.2253	0.8844	39.9858	0.0326	4.345
	4	0.8864	51.8289	0.0192	0.2337	0.8632	38.8170	0.0378	4.8906
	5	0.8791	51.4813	0.0199	0.2453	0.8543	38.4666	0.0393	4.9886
	6	0.8809	51.6476	0.0195	0.2405	0.8463	37.9409	0.0415	5.1762
	7	0.8717	51.2304	0.0204	0.2524	0.8519	38.1547	0.0406	5.1852
	8	0.8717	51.2246	0.0204	0.2524	0.8350	36.3171	0.0481	5.6986

Table 7. SSIM, PSNR, NRMSE, and MAPE values of TSSG reconstructed images according to the cloud ratio of reference images in two sites.

		Site 1				Site 2
		SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Cloud ratio of reference image for TSSG	1%	0.8724	51.2573	0.0204	0.2641	0.8859	40.3423	0.0313	4.3484
	5%	0.4592	44.6118	0.0479	0.9966	0.7083	35.0059	0.0582	7.2984
	10%	0.6425	45.5739	0.0390	0.5668	0.7835	35.7543	0.0536	7.3260
	20%	0.5992	46.6018	0.0351	0.6327	0.5569	32.9797	0.0725	9.8077
	30%	0.4041	43.1996	0.0533	0.9431	0.2642	29.6447	0.1068	15.4000
	40%	0.4225	39.8299	0.0772	1.6024	0.2256	29.8215	0.1043	15.5360
	50%	0.2967	40.8756	0.0682	1.4250	0.2319	28.9130	0.1167	16.8713

Table 8. SSIM, PSNR, NRMSE, and MAPE values of TSSG reconstructed images according to the similarity group pixel ratios in two sites.

		Site 1				Site 2
		SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Similarity group pixel ratios	0.05%	0.8646	52.0377	0.0186	0.2510	0.8684	40.8139	0.0296	3.9661
	0.1%	0.8717	52.0316	0.0186	0.2455	0.8759	40.8387	0.0295	3.9900
	0.2%	0.8753	51.9726	0.0186	0.2434	0.8808	40.7260	0.0299	4.0851
	0.3%	0.8766	52.0682	0.0184	0.2437	0.8831	40.5831	0.0304	4.1844
	0.5%	0.8772	51.9823	0.0185	0.2461	0.8859	40.3423	0.0313	4.3484
	1%	0.8764	51.5318	0.0194	0.2527	0.8893	39.9709	0.0327	4.6027
	2%	0.8733	50.8080	0.0211	0.2653	0.8913	39.6146	0.0340	4.8452
	3%	0.8699	50.2854	0.0224	0.2828	0.8865	39.4214	0.0398	4.9981

Table 9. Comparisons of SSIM, PSNR, NRMSE, and MAPE between SSG and TSSG cloud restoration results for site 1 (South Korea).

Site 1 (South Korea)	SSG				TSSG
	SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Coastal aerosol	0.5780	43.7221	0.0483	0.7713	0.9005	57.0121	0.0105	0.0884
Blue	0.5331	42.4572	0.0647	1.0117	0.8979	55.4625	0.0145	0.1467
Green	0.5181	41.5463	0.0834	1.2069	0.8997	54.4963	0.0188	0.1897
Red	0.5619	40.7601	0.0887	1.2007	0.9002	52.9768	0.0217	0.2475
NIR	0.4472	40.6704	0.0480	1.1754	0.8731	49.6443	0.0171	0.2750
SWIR1	0.4654	38.2528	0.0584	1.6079	0.8842	47.5507	0.0200	0.2878
SWIR2	0.5650	37.8260	0.0859	1.8906	0.8936	48.2429	0.0259	0.3394

Table 10. Comparisons of SSIM, PSNR, NRMSE, and MAPE between SSG and TSSG cloud restoration results for site 2 (USA).

Site 2 (USA)	SSG				TSSG
	SSIM	PSNR	NRMSE	MAPE	SSIM	PSNR	NRMSE	MAPE
Coastal aerosol	0.4650	29.7950	0.2184	20.5951	0.8722	44.4286	0.0405	3.4776
Blue	0.5828	29.8142	0.1802	22.7051	0.9045	45.0347	0.0312	3.534
Green	0.6481	29.5820	0.1154	19.1172	0.8958	42.6592	0.0256	3.8571
Red	0.3957	30.7042	0.0885	11.6117	0.8929	41.5162	0.0255	4.6072
NIR	0.2707	28.8399	0.0659	13.4713	0.8685	36.2856	0.0280	4.7754
SWIR1	0.4810	25.4316	0.1133	18.9369	0.8720	35.1798	0.0369	4.3484
SWIR2	0.2963	26.3788	0.1109	26.1215	0.8954	37.2922	0.0316	5.8395

Figure 9. The performance of TSSG using wildfire-affected image as supplementary material. The second and third lines of each figure depict enlarged images of the red rectangle regions: (a) cloud simulation images, (b) original cloud-free images, (c) restored images using proposed TSSG.

Data availability statement

The dataset will be open to public via https://github.com/YerinYun17/TSSG.