Multi-temporal urban semantic understanding based on GF-2 remote sensing imagery: from tri-temporal datasets to multi-task mapping

ABSTRACT

High resolution satellite images are becoming increasingly available for urban multi-temporal semantic understanding. However, few datasets can be used for land-use/land-cover (LULC) classification, binary change detection (BCD) and semantic change detection (SCD) simultaneously because classification datasets always have one time phase and BCD datasets focus only on the changed location, ignoring the changed classes. Public SCD datasets are rare but much needed. To solve the above problems, a tri-temporal SCD dataset made up of Gaofen-2 (GF-2) remote sensing imagery (with 11 LULC classes and 60 change directions) was built in this study, namely, the Wuhan Urban Semantic Understanding (WUSU) dataset. Popular deep learning based methods for LULC classification, BCD and SCD are tested to verify the reliability of WUSU. A Siamese-based multi-task joint framework with a multi-task joint loss (MJ loss) named ChangeMJ is proposed to restore the object boundaries and obtains the best results in LULC classification, BCD and SCD, compared to the state-of-the-art (SOTA) methods. Finally, a large spatial-scale mapping for Wuhan central urban area is carried out to verify that the WUSU dataset and the ChangeMJ framework have good application values.

KEYWORDS:

• GF-2 remote sensing imagery
• multi-temporal satellite datasets
• urban LULC mapping
• binary and semantic change detection
• multi-task framework

1. Introduction

The urbanization process reflects the symbiosis between human activities and the ecological environment (Manoli et al. 2019). The rapid urbanization in the twenty-first century has resulted in several environmental problems, such as the degradation of arable land and forest land (Tong et al. 2020) and the expansion of cities (Reba and Seto 2020). Remote sensing semantic understanding, including land-use/land-cover (LULC) extraction and change analysis is emerging as an effective way to better characterize urban environments, which is conducive to promoting the sustainable development of human society (Huang et al. 2020; Smith et al. 2016). Semantic change detection (SCD), cannot only locate change regions but also identify change directions related to LULC types. Therefore, SCD is usually regarded as an integration of binary change detection (BCD) which answers ‘where changes occur’ (Singh 1989), and LULC classification which aims to designate each pixel in an image with a predefined LULC label (Zhao et al. 2016).

After decades of methodological development, the era of deep learning for remote sensing big data has started since the 2010s, allowing the exploration of deeper spectral-spatial features (Hua, Mou, and Zhu 2019; Xing et al. 2020). Deep learning based semantic segmentation models, as an important basis for LULC classification and change detection, nowadays have two main categories. One is based on convolutional neural network (CNN) architecture and the other is based on transformer architecture. CNN-based models, including U-Net (Ronneberger, Fischer, and Brox 2015), the DeepLab series of networks (Chen et al. 2014, 2018), the feature pyramid network (FPN) (Lin et al. 2017), and HRNet (Sun et al. 2019), reduce the image resolution by pooling and convolution with a fixed-size convolutional kernel. While transformer-based methods, including the Shifted Window (Swin) Transformer (Liu et al. 2021) and SegFormer (Xie et al. 2021), require more computational resources, albeit with powerful modeling ability.

The siamese architecture has been widely used in change detection by extracting features from two independent inputs with a shared structure, except for the early fusion (EF) strategy, where different images are treated as concatenated input channels and fed together into a network (Daudt, Le Saux, and Boulch 2018). Recently, the attention mechanism has also been introduced into Siamese frameworks to extract more discriminative features. Bitemporal image transformer (BIT) introduces transformer into BCD (Chen, Qi, and Shi 2021). ChangeFormer puts a transformer encoder into a Siamese framework so that details and robust semantic features can be reserved (Bandara and Patel 2022).

Recently, deep learning based Siamese frameworks have tried to combine BCD and LULC classification for SCD such as the spatially and semantically enhanced Siamese network (SSESN) (Zhao et al. 2022) and the Bi-temporal Semantic Reasoning Network (Bi-SRNet) (Ding et al. 2022). A weight-shared encoder is used to extract bitemporal features and multiple headers support mapping for multiple tasks. This solves the problems of the traditional SCD methods, where the direct comparison (DC) strategy that treats SCD as a multi-temporal LULC classification requires a large number of ‘from-to’ labels, while the post-classification comparison (PCC) strategy that obtains semantic changes by comparing co-registered LULC classification maps, requires more parameters and results in error accumulation. However, Siamese-based SCD frameworks are difficult to balance on three tasks, focusing on improving the contribution of BCD to LULC classification and SCD and ignoring the influence of LULC mapping quality on change detection mapping.

All deep learning based methods are data driven. Well-annotated datasets are one of the most crucial factors facilitating training models with the generalization capability. However, single-temporal LULC datasets are incapable of training models to learn about change information. BCD datasets are unable to capture the variety of changed LULC types because only the range of changes are labeled. Although several SCD datasets have been released recently, they still have some limitations. For instance, the High Resolution Semantic Change Detection (HRSCD) dataset (Daudt et al. 2019) lacks the subdivision of artificial surfaces, thus limiting the potential for analyzing urban structural changes. The popular SECOND dataset (Yang et al. 2022), does not annotate unchanged areas, leading to the same problem of neglecting LULC mapping for unchanged regions. What’s more, most datasets are collected in the form of unified patches, without geographic continuity over a large area.

To address the above problems, a multi-temporal large-scale geographically continuous urban semantic dataset named the Wuhan Urban Semantic Understanding (WUSU) dataset made up of multi-band Gaofen-2 (GF-2) remote sensing imagery with a 1-m spatial resolution was built in this study. The WUSU dataset was annotated with a refined class system of 11 LULC classes over the entire study area, covering three time phases, and 60 change directions were generated. Global-wide annotations are useful for LULC mapping and global assessments.

In order to verify the reliability of WUSU and sufficiently exploit the relationship between LULC classification and BCD to better complete the SCD mapping, a multi-task joint change detection framework (abbreviated by ChangeMJ) was proposed. A multi-task joint loss (MJ loss) was embedded in ChangeMJ to recover the building boundaries. In order to evaluate the effectiveness of ChangeMJ for building extraction more rationally, a multi-task joint F₁ score (MJF₁) was proposed.

The two major contributions of this paper are summarized as follows:

1. A multi-temporal multi-task dataset for urban semantic understanding named the WUSU dataset is presented. The WUSU dataset made up of GF-2 images with 1-m spatial resolution has the most LULC classes and time phases among the public SCD datasets, and the semantic annotations cover the whole study area with large-scale geographic continuity. The WUSU dataset has the finest class system for urban construction among public change detection datasets.

2. The popular deep learning networks for LULC classification, BCD and SCD are tested to provide a benchmark for further studies. A Siamese framework named ChangeMJ framework is proposed for LULC mapping, BCD mapping and SCD mapping simultaneously and its generalization and validity is confirmed on WUSU and another SCD dataset. The MJ loss is introduced in ChangeMJ to obtain refined mapping results by recovering building boundary information. The MJF₁ is proposed for result evaluation, considering the building refinement.

The rest of the paper is organized as follows. Section 2 presents the related public datasets. The proposed WUSU dataset is introduced in Section 3. Section 4 describes the proposed ChangeMJ framework with a MJ loss, and a metric MJF1 for refined mapping. Section 5 gives the experimental analysis. The generalization and reliability of ChangeMJ, and the benefits of WUSU are discussed in Section 6. Finally, the conclusions are drawn in Section 7.

2. Public datasets for land-use/land-cover classification and change detection

Several public datasets with spatial resolution better than 10 m for LULC classification, BCD and SCD are reviewed and listed in chronological order in Tables 1 and 2 (More information can be found in https://github.com/AngieNikki/openWUSU).

Table 1. The publicly available remote sensing based LULC datasets.

Dataset name	Bands	Spatial resolution (m)	Class number	Data source
ISPRS 2D Semantic Labeling Contest dataset (Rottensteiner et al. 2012)	3/4	0.09/0.05	6	Aerial
Zurich Summer v1.0 dataset (Volpi and Ferrari 2015)	4	0.0062	8	QuickBird
Aerial Image Segmentation dataset (Kaiser et al. 2017)	3	0.091/0.098/0.101/0.111	3	Google Earth
EvLab-SS dataset (Zhang et al. 2017)	3	0.2	11	World-View-2
		0.5		GeoEye
		2		QuickBird
		1		GF-2
		0.25/0.1		Aerial
DeepGlobe Land Cover Classification Challenge dataset (Demir et al. 2018)	3	0.5	7	Satellite
Dense Labeling Remote Sensing Dataset (DLRSD) (Shao, Yang, and Zhou 2018)	3	0.3	17	Aerial
Gaofen Image Dataset (GID) (Tong et al. 2020)	4	4	5/15	GF-2
Wuhan Dense Labeling Dataset (WHDLD) (Shao et al. 2020)	3	2	6	Aerial
Land-cOVEr Domain Adaptive semantic segmentation (LoveDA) dataset (Wang et al. 2021)	3	0.3	7	Google Earth
WHU-OPT-SAR dataset (Li, Zhang, et al. 2022)	4	2	7	GF-1
	1	5		GF-3 fine strip II

Table 2. The publicly available remote sensing based CD datasets

Dataset name	Bands	Spatial resolution (m)	Class number	Years	Data source
BCD datasets
SZTAKI Air Change dataset (Benedek and Sziranyi 2009)	3	1.5	2	2	Aerial
ABCD dataset (Fujita et al. 2017)	3	0.4	2 (building)	2	Aerial
WHU building dataset (Ji, Wei, and Lu 2019)	3	0.2	2 (building)	2	Aerial
CDD Dataset (Lebedev et al. 2018)	3	0.03–1	2	2	Google Earth
LEVIR-CD dataset (Chen and Shi 2020)	3	0.5	2 (building)	2	Google Earth
Google dataset (Peng et al. 2021)	3	0.55	2	2	Google Earth
LEVIR-CD + dataset (Shen et al. 2021)	3	0.5	2 (building)	2	Google Earth
S2Looking dataset (Shen et al. 2021)	3	0.5–0.8	2 (building)	2	GF, SuperView and BeiJing-2
Satellite-UAV heterogeneous image change detection (HTCD) dataset (Shao et al. 2021)	3	0.5971	2	2	Google Earth
	3	0.07465			Open Aerial Map
Sun Yat-Sen University (SYSU)-CD dataset (Shi et al. 2022)	3	0.5	2	2	Aerial
Cropland change detection (CLCD) dataset (Liu et al. 2022)	3	0.5–2	2	2	GF-2
Multisource built-up change (MSBC) datasets (Li, Zhu, et al. 2022)	4	2	2	2	GF-2
	VV, VH				Sentinel-1
	7				Sentinel-2
SCD datasets
HRSCD dataset (Daudt et al. 2019)	3	0.5	5	2	Aerial
Hi-UCD mini dataset (Tian et al. 2020)	3	0.1	9	3	Aerial
SECOND dataset (Yang et al. 2022)	3	0.5–3	6	2	(Multiple)
Hi-UCD dataset (Tian et al. 2022)	3	0.1	9	2	Aerial
WUSU dataset (proposed)	4	1	11	3	GF-2

As shown in Table 1, the LULC classification dataset with the largest area is GID, which has five classes labeled at the pixel level and lacks spatial details due to the 4-m spatial resolution. Some datasets made up of high resolution images and with a refined class system, such as DLRSD and LoveDA, have a small field of view that neglects the large-scale spatial dependence. In addition, the images are mostly of only three bands, which ignores the significance of other bands for feature extraction, especially the near-infrared band. Most of all, all the LULC datasets have only one time phase.

Both temporal information and LULC information are important for analyzing urban development. However, change detection datasets mostly focus on BCD. The SECOND dataset contains temporal and LULC information, but has no LULC annotations in unchanged areas. The Hi-UCD mini dataset, which is the only public tri-temporal SCD dataset equipped with global-wide LULC annotations, has a small receptive field in each patch, and the geographical area is about only 30 km².

Therefore, a multi-temporal dataset with global-wide semantic annotation based on multi-band high resolution imagery is urgently needed to support the technological development for LULC classification, BCD and SCD tasks.

3. Study areas and the proposed WUSU dataset

For better integrating spectral and spatial information, GF-2 images with an effective spatial resolution of 1 m in panchromatic mode and 4 m in multispectral mode were adopted to construct the dataset. Two Chinese urban regions with great potential and speed of development were selected as the study area and a detailed class system of urban LULC was then established.

3.1. Study areas

Wuhan is the main city of the Yangtze River Economic Belt. During the past ‘13th Five-Year Plan’, Jiang’an District has been committed to building a new spatial development pattern and Hongshan District has been focusing on improving the urban functional quality. These two central urban districts are still key development areas in the ‘14th Five-Year Plan’. Therefore, the two districts covering a total geographic area of nearly 80 km² were chosen as the study areas (Figure 1).

Figure 1. The study areas in the city of Wuhan, Hubei province, China.

3.2. WUSU dataset

The construction of the WUSU dataset is shown in Figure 2 and the specific procedures are detailed as following.

Figure 2. The construction workflow of the WUSU dataset.

3.2.1. Image pre-processing for the WUSU dataset

First, the available GF-2 images were screened for cloud cover less than 5% in an image. Candidate images then underwent pre-processing, including PAN image orthographic correction, MSS image orthographic correction and image pan sharpening. Multi-temporal image registration was performed for change detection tasks, requiring the root-mean-square error (RMSE) of less than 1 pixel. Finally, through image cropping, images for Hongshan District with $6358\times 6382$ pixels were obtained for April 22, 2015, September 1, 2016, and November 23, 2018 (Figure 3a–c), and images for Jiang’an District with $7025\times 5500$ pixels were obtained for April 22, 2015, December 8, 2016, and November 23, 2018 (Figure 3d–f). Hence, 4-band fusion satellite images with 1-m spatial resolution were formed for the construction of the WUSU dataset.

Figure 3. The WUSU dataset. GT represents the ground truth. (a)–(f) Images of Hongshan District and Jiang’an District taken in 2015, 2016 and 2018. (g)–(l) LULC_GT annotations for images (a)–(f). (m)–(p) BCD_GT annotations for the two districts between 2015 and 2016 and between 2016 and 2018. (q)–(t) SCD_GT annotations for the two districts between 2015 and 2016 and between 2016 and 2018. (u) Legends where the abbreviation in the legend for SCD_GT is the first two letters of each LULC class name.

3.2.2. Label generation for the WUSU dataset

Class system definition. The WUSU class system refers to the Chinese Land Use Classification Criteria (GB/T21010-2017) and the Contents and Indicators of the Chinese Geographical Conditions Census (GDPJ 01-2013). As shown in Table 3, the first-level classes in the two documents are listed and their correspondences are indicated by arrows. The classes of natural surfaces in the two documents are the same, such as ‘Arable land’ and ‘Woodland’, so they were inherited in WUSU. There are some intersections between the classes of artificial surfaces of the two documents. For example, ‘Structure’ or ‘Excavation’ can occur in various land use areas such as ‘Commercial land’ or ‘Residential land’. Therefore, the classes of artificial surfaces in two documents were adjusted according to the characteristics of the study area. Eleven classes are consequently included in the class system of WUSU: 1-Road, 2-Low building, 3-High building, 4-Arable land, 5-Woodland, 6-Grassland, 7-River, 8-Lake, 9-Structure, 10-Excavation, and 11-Bare surface. Areas not containing any of the above 11 classes are labeled as ‘unclassified’, and are marked as 0. Examples and detailed descriptions of all the classes are provided in Figure 4. The distinction of ‘River’ (referring specifically to the Yangtze River in WUSU) and ‘Lake’ mainly considers the special geographical location of Wuhan. The 11 LULC classes lead to 60 change directions which can be find in Figure 3.

Figure 4. Examples and descriptions of each class in the WUSU dataset.

Table 3. The class systems used in Chinese public documents.

	Contents and Indicators of Chinese Geographical Conditions Census (GDPJ 01-2013)	Chinese Land Use Classification Criteria (GB/T21010-2017)
First-level classes

Label annotation. The building outlines were derived from the open-source building vectors and registered to the co-registered images. Other classes were manually annotated by the expert group RSIDEA to ensure high accuracy. High buildings have different tilt angles in multi-temporal images, so building annotations are snapped to the building footprints. Building annotations were then modified to match images at different time phases. As a result, WUSU is unique in that the building classes are fully annotated with accurate boundaries while other classes are partially annotated due to the challenges of shadow obscuration for low impervious surfaces and indistinct boundaries for natural surfaces. A cross-temporal inspection was performed on the LULC annotations to ensure that annotations for unchanged areas were not shifted. Finally, the proportion of annotated pixels is 54.31%, 56.75%, and 56.14% for the images from 2015, 2016, and 2018 respectively.

3.2.3. Characteristics and challenges of the WUSU dataset

The WUSU dataset has the most LULC classes, bands, and time phases among the publicly available change detection datasets, as shown in Table 2 and Figure 5. WUSU contains richer urban structure information than HRSCD and has twice the geographical area of Hi-UCD mini. LULC annotations are provided in the unchanged areas, providing more training data. The notable characteristics of WUSU are as follows:

Figure 5. The characteristics of WUSU compared with popular SCD datasets. The closer to the pole, the higher the ranking.

Geographic-temporal continuity. The tri-temporal high resolution WUSU dataset has a large spatial scale, and is not composed of small patches, allowing model training to explore spatial–temporal correlations over a continuous geographic area.

Refined class system. The WUSU dataset has the most LULC classes and change directions, compared to the public SCD datasets. The proportions of annotated pixels for different LULC classes in three years are shown in Figure 6.

Figure 6. Pixel statistics for the LULC distribution of WUSU in 2015, 2016, and 2018.

Urbanization process. The WUSU dataset focuses on urban areas, so ‘Structure’ and ‘Excavation’ areas with a high potential for change are annotated. For urban development, building reconstruction has become a major trend in recent years, so ‘High building’ and ‘Low building’ are distinguished.

4. A multi-task joint change detection framework for refined mapping

For Siamese-based SCD framework, the problem of modeling the temporal dependency remains when the semantic and change features are introduced into the decoder structure simultaneously. Commonly used temporal dependency representations are concatenation and difference of temporal features. However, there are problems of temporal order restriction and weak feature representation separately. ChangeMask (Zheng et al. 2022) introduced a temporal-symmetric transformer (TST) into a Siamese-based framework to guarantee the change representation with temporal symmetry and strong discrimination. TST, consisting of spatial–temporal interaction and symmetric fusion, computes the temporal relationship embedded in the encoded features layer by layer.

To further optimize semantic features, a multi-task joint change detection framework named ChangeMJ (Figure 7) was proposed based on ChangeMask. ChangeMJ is an end-to-end weight-shared Siamese-based framework where any SOTA deep network can be chosen as the semantic encoder. The semantic decoder and the change decoder use the decoder proposed in U-Net.

Figure 7. The flowchart of the proposed ChangeMJ framework.

Cross Entropy loss (CE loss) is usually chosen for semantic segmentation. But the unclassified pixels in WUSU can blur the object boundaries as shown in Figure 8. Unclassified pixels surrounding the buildings are easily classified as buildings, which is not consistent with the true land surfaces. Therefore, a multi-task joint loss (MJ loss) was put forward for the semantic branch of ChangeMJ to obtain refined LULC and change detection maps.

Figure 8. The problem of fuzzy boundaries.

Finally, Binary Cross Entropy loss (BCE loss) for the change branch and MJ loss for the semantic branch constitute the total loss. The expression of total loss is shown in Equation (1). (1) $L_{total}=L_{MJ}^{t1}+L_{MJ}^{t2}+L_{BCE}$(1) where $L_{MJ}^{t1}$ and $L_{MJ}^{t2}$ separately represents the MJ loss to supervise semantic branches for the two time phases. $L_{BCE}$ is BCE loss to supervise the change branch.

MJ loss effectively restored the shape of the buildings. A multi-task joint F₁ score (MJF₁) was consequently proposed to quantitatively evaluated the mapping refinement.

4.1. The multi-task joint loss function (MJ loss)

The MJ loss distinguishes ‘Low building’ and ‘High building’ from other LULC classes which are partially labeled. As seen in Figure 7, the MJ loss is the integration of CE loss for all classes and BCE loss for two building-related classes. In the part of CE loss for all classes, the unlabeled pixels are ignored. The BCE loss for ‘Low building’ and ‘High building’ is calculated in Equation (2). For example, when calculating the BCE loss for ‘Low building’, ‘High building’ and other classes are treated as background so that the building boundaries can be defined. (2) $L_{MJ}=-\sum _{i=1}^C{y}_i\log (x_i)-{\displaystyle \frac{1}{n}\sum _{i\in L}(-({\stackrel{\wedge }{y}}_i\log (x_i)+(1-{\stackrel{\wedge }{y}}_i)\log (1-x_i)))}$(2) where $C$ is the number of classes, $y_i$ and $x_i$ is respectively the target and predict value, $L$ is the set of class codes for which the BCE loss is to be calculated separately. Here, $L=\{2,3\}$ where the correspondence between the class code and class name can refer to Figure 4. $n$ is the length of set $L$. $y$ is the one-hot encoding of all the $C$ classes excluding the ‘unclassified’. $\stackrel{\wedge }{y_i}$ is the binary map, where pixels of class $i$ ($i\in L$) are marked as 1 and pixels of other classes including ‘unclassified’ are marked as 0.

4.2. The multi-task joint F₁ score (MJF₁)

Considering the characteristics of WUSU that the buildings are fully labeled with accurate boundaries, a multi-task joint F₁ score (MJF₁) was proposed to evaluate the semantic branch. MJF₁ which is more suitable for urban areas, aims to evaluate the extraction accuracy of building boundaries. MJF₁ is a separated F₁ score where one part is designed for classes other than the two building classes and the other part is calculated for ‘Low building’ and ‘High building’. Therefore, the MJF₁ is defined as Equation (3). (3) $MJ{F}_1={\displaystyle \frac{1}{C}\left(\sum _{\genfrac{}{}{0ex}{}{i=1}{i\notin L}}^C\left(2q_{ii}/\left(\sum _{j=1}^C{q}_{ji}+\sum _{j=1}^C{q}_{ij}\right)\right)+\sum _{i\in L}^{}\left(2q_{ii}/\left(\sum _{j=0}^C{q}_{ji}+\sum _{j=0}^C{q}_{ij}\right)\right)\right)}$(3) where $q_{ij}$ is the element of the $i$th column and the $j$th row in the confusion matrix $Q$. For classes except the building classes, unclassified pixels are ignored when evaluating. But the other classes including ‘unclassified’ are considered as the negative class when evaluating for ‘Low building’ or ‘High building’.

5. Experiments and results

5.1. Experimental settings and accuracy assessment

Representative methods of three tasks were tested on WUSU (Table 4). Considering similarity in parameter quantity and balance in simplicity and computational efficiency, ResNet-50 (He et al. 2016), HRNet-w32, a tiny Swin Transformer Swin-T (Liu et al. 2021) and Mix Transformer Mit-B2 (Xie et al. 2021) were chosen as the backbones. For ChangeMask and ChangeMJ, HRNet-w32 and ResNet-50 were chosen as backbones for contrast experiments. The PCC strategy is based on FPN and HRNet due to their good performance. All backbones are pre-trained with ImageNet. The top halves of the six images were used as the testing set, and the bottom halves were combined to form the training set. The parameters for the tested models are set as Table 5.

Table 4. The comparison methods for LULC classification, BCD and SCD.

Methods		Architectures	Backbone
LULC classification methods	CNN-based	U-Net (Ronneberger, Fischer, and Brox 2015)	ResNet-50
		PSPNet (Zhao et al. 2017)	ResNet-50
		FPNs (Lin et al. 2017)	ResNet-50
		Deeplabv3+ (Chen et al. 2018)	ResNet-50
		HRNet (Sun et al. 2019)	HRNet-w32
	Transformer-based	UPerNet (Xiao et al. 2018)	Swin-T (Liu et al. 2021)
		Segformer (Xie et al. 2021)	MiT-B2
BCD methods	CNN-based	FC-EF (Daudt, Le Saux, and Boulch 2018)	–
		FC-Siam-conc (Daudt, Le Saux, and Boulch 2018)	–
		FC-Siam-diff (Daudt, Le Saux, and Boulch 2018)	–
	Attention-based	STANet (Chen and Shi 2020)	ResNet-50
	CNN-Transformer-based	BIT (Chen, Qi, and Shi 2021)	ResNet-50
	Transformer-based	ChangeFormer (Bandara and Patel 2022)	–
SCD methods	CNN-based	FPN (PCC, PCC_mask)	ResNet-50
		HRNet (PCC, PCC_mask)	HRNet-w32
	CNN-based (Multi-task)	SSESN (Zhao et al. 2022)	ResNet-50
		Bi-SRNet (Ding et al. 2022)	ResNet-50
		ChangeMask (Zheng et al. 2022)	ResNet-50/HRNet-w32
		ChangeMJ (proposed)	ResNet-50/HRNet-w32

Table 5. Parameter settings for deep learning models.

	LULC classification Models	Change detection Models
Input size	256 × 256	256 × 256
Batch size	32	16
Loss	CE	BCE for BCD and the change branch in SCD; CE for the semantic branch in SCD
Optimizer	Stochastic Gradient Descent	Stochastic Gradient Descent
Initial learning rate	0.01	0.01

Evaluation metrics introduced for different tasks are shown in Table 6. The missed alarms (MA), false alarms (FA) and changed F1-score (cF1) for BCD treated ‘Change’ as the positive class. The mean intersection over union (mIoU) gives the same weights for ‘No change’ and the rest class(es) in both BCD and SCD tasks. The mIoU for SCD and the separated kappa (SeK) coefficient (Yang et al. 2022) aim to evaluate SCD results by escaping the impact of class imbalance.

Table 6. Metrics for the quantitative analysis.

Tasks	Metrics	Formulas
LULC classification	Overall Accuracy (OA)	$OA=\sum _{i=1}^C{q}_{ii}/\sum _{i=1}^C\sum _{j=1}^C{q}_{ij}$
	Kappa	$Kappa=(OA-\eta )/(1-\eta )$, $\eta =\sum _{j=1}^C(q_{j+}{q}_{+j})/{\left(\sum _{i=1}^C\sum _{j=1}^C{q}_{ij}\right)}^2$
	MJF1	$MJ{F}_1={\displaystyle \frac{1}{C}\left(\sum _{\genfrac{}{}{0ex}{}{i=1}{i\notin L}}^C\left(2q_{ii}/\left(\sum _{j=1}^C{q}_{ji}+\sum _{j=1}^C{q}_{ij}\right)\right)+\sum _{i\in L}^{}\left(2q_{ii}/\left(\sum _{j=0}^C{q}_{ji}+\sum _{j=0}^C{q}_{ij}\right)\right)\right)}$
	F1_Low	$F_1\mathrm{\_}Low=2q_{22}/\left(\sum _{j=0}^C{q}_{j2}+\sum _{j=0}^C{q}_{2j}\right)$
	F1_High	$F_1\mathrm{\_}High=2q_{33}/\left(\sum _{j=0}^C{q}_{j3}+\sum _{j=0}^C{q}_{3j}\right)$
BCD	OA	$OA=\sum _{i=1}^2{q}_{ii}/\sum _{i=1}^2\sum _{j=1}^2{q}_{ij}$
	Kappa	$Kappa=(OA-\eta )/(1-\eta )$, $\eta =\sum _{j=1}^2(q_{j+}{q}_{+j})/{\left(\sum _{i=1}^2\sum _{j=1}^2{q}_{ij}\right)}^2$
	mIoU	$mIoU={\displaystyle \frac{1}{2}\sum _{i=1}^2\left(q_{ii}/\left(\sum _{i=1}^2{q}_{ij}+\sum _{j=1}^2{q}_{ij}-q_{ii}\right)\right)}$
	cF1	$cF1=2TP/(2TP+FP+FN)$
	MA	$MA=FN/(FN+TP)$
	FA	$FA=FP/(FP+TN)$
SCD	OA	$OA=\sum _{i=1}^M{q}_{ii}/\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}$ (where $M$ is the number of SCD classes)
	Kappa	$Kappa=(OA-\eta )/(1-\eta )$, $\eta =\sum _{j=1}^M(q_{j+}{q}_{+j})/{\left(\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}\right)}^2$
	mIoU	$mIoU={\displaystyle \frac{1}{2}\left(q_{11}/\left(\sum _{i=1}^M{q}_{i1}+\sum _{j=1}^M{q}_{1j}-q_{11}\right)+\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}/\left(\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}-q_{11}\right)\right)}$
	SeK	$SeK=e^{(Io{U}_2-1)}(\stackrel{\wedge }{\rho }-\stackrel{\wedge }{\eta })/(1-\stackrel{\wedge }{\eta })$, $Io{U}_2=\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}/\left(\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}-q_{11}\right)$ $\stackrel{\wedge }{\rho }=\sum _{i=2}^M{q}_{ii}/(\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}-q_{11})$, $\stackrel{\wedge }{\eta }=\sum _{j=1}^M(\stackrel{\wedge }{q_{j+}}\stackrel{\wedge }{q_{+j}})/{\left(\sum _{i=1}^M\sum _{j=1}^M{q}_{ij}-q_{11}\right)}^2$

5.2. Experimental results for land-use/land-cover classification

The LULC mapping results and details are shown in Figures 9 and 10 separately. Transformer-based methods balance the completeness and detail compared with CNN-based methods. Siamese-based frameworks show better visualization than single-task segmentation models due to the temporal interaction. SSESN and Bi-SRNet best differentiate ‘River’ and ‘Lake’ due to their additional modules to enhance the semantic features using change features but such strategies also introduce the overfitting problem, which will be shown in the next section. For most comparison methods, each building is well distinguished if there are other class annotations around the buildings (yellow boxes in Figure 10), otherwise the buildings stick together (cyan boxes in Figure 10). ChangeMJ restores the building shapes, so that other classes, such as ‘Structure’, can be extracted more accurately in ChangeMJ results.

Figure 9. LULC maps for Hongshan and Jiang’an Districts. Notes (R) or (H) separately means setting ResNet-50 or HRNet-w32 as the backbone.

Figure 10. Detailed 2016 LULC maps for WUSU.

The quantitative results (Table 7) are consistent with the above conclusions. FPN achieves the best scores among the CNN-based methods with ResNet-50 as the backbone. HRNet shows abundant details and a higher continuity for ‘Road’ even than SegFormer and UPerNet, proving that the high spatial resolution is mantained in HRNet.

Table 7. Quantitative results of LULC classification for WUSU.

LULC classification method	OA (%)	Kappa (%)	MJF1 (%)	F1_Low (%)	F1_high (%)
PSPNet	70.88	66.41	51.07	38.06	47.59
DeepLabv3+	73.26	69.09	52.70	37.47	45.99
U-Net	74.04	69.98	53.38	38.28	47.48
FPN	74.29	70.24	53.50	37.88	46.30
SegFormer	74.45	70.66	54.47	42.91	50.18
HRNet	75.68	71.97	55.33	41.17	49.85
UPerNet	76.29	72.67	55.94	42.84	49.78
SSESN	82.62	79.89	61.13	43.72	52.44
Bi-SRNet	79.35	76.06	58.20	40.70	43.67
ChangeMask(R)	76.59	72.95	55.55	39.75	48.71
ChangeMask(H)	77.73	74.39	56.99	42.23	50.28
ChangeMJ (R)	76.64	73.15	58.00	50.88	63.43
ChangeMJ (H)	78.28	75.05	59.34	51.61	63.56

5.3. Experimental results for change detection

The BCD and SCD results are separately shown in Figures 11 and 12, and the quantitative results are listed in Tables 8 and 9. Details of BCD and SCD from 2016 to 2018 are shown in Figures 13 and 14.

Figure 11. BCD maps for Hongshan and Jiang’an Districts.

Figure 12. SCD maps for Hongshan and Jiang’an Districts.

Figure 13. Detailed BCD maps from 2016 to 2018 for the WUSU dataset.

Figure 14. Detailed SCD maps from 2016 to 2018 for the WUSU dataset.

Table 8. Quantitative results of BCD for WUSU.

BCD methods	OA (%)	Kappa (%)	mIoU (%)	cF1 (%)	MA (%)	FA (%)
FC-EF	90.45	52.08	64.78	56.89	18.13	8.83
FC-Siam-conc	92.75	57.55	68.31	61.40	25.06	5.76
FC-Siam-diff	93.48	58.89	69.23	62.41	29.66	4.59
STANet	92.34	47.04	63.21	51.19	47.78	4.32
BIT	91.95	54.27	66.41	58.51	26.17	6.54
ChangeFormer	91.06	51.74	64.88	56.37	24.89	7.61
SSESN	94.21	58.19	69.07	61.32	40.37	2.91
Bi-SRNet	94.40	55.36	67.66	58.30	49.10	1.98
ChangeMask(R)	95.03	63.41	72.06	66.08	37.05	2.30
ChangeMask(H)	95.41	65.04	73.05	67.48	38.11	1.79
ChangeMJ(R)	95.20	63.75	72.28	66.31	38.58	1.99
ChangeMJ(H)	95.49	65.77	73.49	68.17	37.20	1.79

Table 9. Quantitative results of SCD for WUSU.

SCD methods	mask by FC-Siam-diff	MJ loss	OA (%)	kappa (%)	mIoU (%)	SeK (%)
FPN			71.92	18.37	47.18	1.73
	√		91.94	42.82	70.48	6.10
	√	√	92.05	43.63	70.52	6.85
HRNet			72.65	20.15	47.66	2.61
	√		92.56	46.60	71.66	9.25
	√	√	92.77	47.40	71.76	9.97
SSESN			92.53	45.19	68.26	8.18
Bi-SRNet			92.87	42.72	67.79	6.94
ChangeMask(R)			92.90	47.51	71.89	9.89
ChangeMask(H)			93.31	49.78	72.84	12.08
ChangeMJ(R)		√	93.36	48.87	71.70	11.38
ChangeMJ(H)		√	93.44	50.62	73.26	12.85

For BCD, stacking images in FC-EF brings a high FA. STANet eliminates the noise in unchanged areas but results in a high MA. BIT and ChangeFormer also bring high FA. FC-Siam-diff obtains the best results and keeps a good balance between FA and MA, indicating that a simple network may be appropriate for WUSU. Overdrawing the semantic features of different time phases in SSESN leads to a lot of MA in BCD results. Bi-SRNet has the similar problem to SSESN and brings the highest MA.

ChangeMask and ChangeMJ have significant advantages over the other methods due to a balance between the use of semantic and change information. HRNet-w32 that leads to better LULC classification performance than ResNet-50 also leads to a better BCD result in ChangeMask and ChangeMJ. This is further evidence of the importance of semantic branches for BCD. It can be found that using MJ loss not only improves the LULC classification accuracy but also benefits BCD by comparing the results of ChangeMJ and ChangeMask with the same backbone.

Masked by the best BCD maps, the SCD results based on PCC have significant improvement (Figure 14). Introducing the MJ loss to PCC-mask based methods, the visualization results and the metric scores have further advanced. SSESN and Bi-SRNet overfit the semantic features and ignore the change branch, which also results in poor SCD results. A similar conclusion still comes that better semantic features promote to better BCD and SCD results by comparing ChangeMask and ChangeMJ. Therefore, annotations in unchanged areas are proved to have a positive influence for the three tasks.

To better demonstrate the effectiveness of the MJ loss, the detailed maps of two regions are demonstrated in Figure 15. By using the MJ loss, the mapping quality and precision are improved and the building classes especially the stand-alone buildings are separated from each other. The LULC maps from ChangeMJ show that the buildings are surrounded by ‘Woodland’, reflecting the high degree of urbanization to some extent. Whether in the LULC maps or in the change detection maps, ChangeMJ mantains more refined boundaries than ChangeMask.

Figure 15. Detailed LULC, BCD, SCD mapping results for WUSU. (a)-(h) Image, LULC_GT, ChangeMask result and ChangeMJ result in 2016 and 2018. (i)-(p) Corresponding BCD/SCD ChangeMask result, ChangeMJ result and GT in 2016 and 2018.

6. Discussion

6.1. Validity of the proposed ChangeMJ framework on Hi-UCD mini dataset

Experiments on Hi-UCD mini were conducted to assess the generalization of ChangeMJ. Hi-UCD mini uses aerial images with 0.1 m to construct tri-temporal semantic changes in urban area of Tallinn, the capital of Estonia. The data division of training and testing is consistent with the scheme provided by the authors, with 2017 and 2018 used as training. The experimental settings are the same as those on WUSU. The quantitative results for three tasks are separately listed in Tables 10–12.

Table 10. Quantitative results of LULC classification for Hi-UCD mini.

LULC classification method	OA (%)	Kappa(%)	MJF1(%)	F1_building(%)
PSPNet	81.47	75.76	53.77	79.43
DeepLabv3+	83.91	78.98	58.91	79.10
U-Net	83.55	78.51	58.02	80.29
FPN	83.22	78.04	61.38	80.06
SegFormer	84.53	79.78	60.50	82.21
HRNet	83.82	78.85	58.20	81.69
UPerNet	84.30	79.46	59.22	81.14
SSESN	84.44	79.70	61.70	80.39
Bi-SRNet	82.60	77.17	59.72	75.78
ChangeMask(R)	83.51	78.40	49.39	78.25
ChangeMask(H)	83.99	79.06	62.70	78.40
ChangeMJ (R)	83.37	78.26	50.32	88.03
ChangeMJ (H)	84.03	79.10	64.13	89.26

Table 11. Quantitative results of BCD for Hi-UCD mini.

	OA (%)	Kappa (%)	mIoU (%)	cF1 (%)	MA (%)	FA (%)
FC-EF	88.54	52.85	64.63	58.90	18.01	10.74
FC-Siam-conc	88.51	52.61	64.51	58.68	18.58	10.70
FC-Siam-diff	88.57	53.20	64.80	59.22	17.18	10.79
STANet	88.72	53.56	65.03	59.52	17.20	10.63
BIT	88.36	52.55	64.42	58.68	17.61	10.98
ChangeFormer	88.22	52.09	64.15	58.29	17.88	11.10
SSESN	94.77	67.01	74.02	69.80	39.63	1.41
Bi-SRNet	94.55	67.38	74.22	70.36	35.42	2.12
ChangeMask(R)	95.28	71.79	77.06	74.37	31.59	1.73
ChangeMask(H)	95.33	72.41	77.47	74.97	30.20	1.83
ChangeMJ(R)	95.40	73.23	78.01	75.76	28.29	1.96
ChangeMJ(H)	95.58	73.41	78.15	75.81	30.83	1.48

Table 12. Quantitative results of SCD for Hi-UCD mini.

SCD methods	MJ loss	OA (%)	kappa (%)	mIoU (%)	SeK (%)
SSESN		92.84	54.68	73.03	17.70
Bi-SRNet		92.79	55.36	73.27	17.95
ChangeMask(R)		93.49	60.86	76.76	23.51
ChangeMask(H)		93.46	61.61	77.11	24.16
ChangeMJ(R)	√	93.54	61.96	77.25	24.64
ChangeMJ(H)	√	93.76	62.79	77.61	25.69

Although the spatial resolution of Hi-UCD mini is extremely higher than that of WUSU, ChangeMJ still achieves the best results. ChangeMJ outperforms most single-task LULC classification models and all BCD models. ChangeMJ achieves superior BCD and SCD accuracies, surpassing SSESN and Bi-SRNet, while maintaining a balanced performance across different tasks.

In terms of visualization, ChangeMJ can find building instances and their changes more accurately and in finer detail than other methods (Figure 16 left). In other classes, ChangeMJ also learns features with generalization better and the LULC classification, BCD and SCD results are more accurate (Figure 16 right).

Figure 16. The LULC classification, BCD and SCD results for the test set of Hi-UCD mini. (a) Images. (b) SSESN. (c) Bi-SRNet. (d) ChangeMask(H). (e) ChangeMJ(H). (f) GT. (g) Legend for LULC_GT and SCD_GT. (h) Legend for BCD_GT.

6.2. Comparison of multi-task frameworks

The results of these multi-task frameworks on LULC classification, BCD and SCD for WUSU and Hi-UCD mini datasets are aggregated to better demonstrate the characteristics of each framework. MJF₁, cF₁, SeK was separately selected for three tasks to create a bar chart to represent the effects of these frameworks (Figure 17).

Figure 17. Accuracy comparison chart of multi-task frameworks.

According to the Average Kappa in Table 13, all frameworks achieved higher values on Hi-UCD mini with higher spatial resolution and less classes, which illustrates the challenges of WUSU. SSESN and Bi-SRNet present slightly better results than the other frameworks for LULC classification and significantly worse results for BCD and SCD on both datasets, suggesting that they overfit semantic features of two branches, resulting in MA of changes. ChangeMJ(H) achieved optimal results on all indicators except the LULC classification index of WUSU. It can also be seen from Figure 17 that SSESN and Bi-SRNet optimize the results of LULC to some extent, but are inferior to ChangeMJ (H) in terms of generalization and reliability.

Table 13. Quantitative results of different Siamese-based frameworks on three tasks for WUSU and Hi-UCD mini datasets.

Methods	Dataset		LULC classification		BCD		SCD		Average Kappa (%)
	WUSU	Hi-UCD mini	Kappa (%)	MJF1(%)	Kappa (%)	cF1 (%)	kappa (%)	SeK (%)
SSESN	√		79.89	61.13	58.19	61.32	45.19	8.18	61.09
		√	79.70	61.70	67.01	69.80	54.68	17.70	67.13
Bi-SRNet	√		76.06	58.20	55.36	58.30	42.72	6.94	58.05
		√	77.17	59.72	67.38	70.36	55.36	17.95	66.64
ChangeMask(R)	√		72.95	55.55	63.14	66.08	47.51	9.89	61.20
		√	78.40	49.39	71.79	74.37	60.86	23.51	70.35
ChangeMask(H)	√		74.39	56.99	65.04	67.48	49.78	12.08	63.07
		√	79.06	62.70	72.41	74.97	61.61	24.16	71.03
ChangeMJ(R)	√		73.15	58.00	63.75	66.31	48.87	11.38	61.92
		√	78.26	50.32	73.23	75.76	61.96	24.64	71.15
ChangeMJ(H)	√		75.05	59.34	65.77	68.17	50.62	12.85	63.81
		√	79.10	64.13	73.41	75.81	62.79	25.69	71.77

6.3. Change analysis for Wuhan central urban area

By training ChangeMJ with the whole WUSU dataset, the mapping in the central urban area of Wuhan has been realized. The LULC maps for 2016 and 2018 and the BCD and SCD maps are shown in Figure 18. Some local regions are enlarged to show more details. The mapping results show that WUSU with refined annotations can be used to achieve a wide range of multi-task mapping and works in a larger spatial and temporal dimension.

Figure 18. The LULC classification, BCD and SCD maps for Wuhan central urban area in 2016 and 2018.

As shown in Figure 19, the data flow shown in the Sankey map for Wuhan central urban area represents the change directions. Many changes are from ‘Low building’ to ‘Excavation’ and from ‘Grassland’ or ‘Woodland’ to ‘Excavation’, indicating a high speed of urbanization. The ‘High building’ and ‘Structure’ are increasing from year to year, reflecting the acceleration of urbanization. The inflows of ‘Structure’ and ‘Road’ are greater than the outflows respectively, which is closely related to Wuhan's increased investment in urban construction and transportation construction at that time, indicating that the urban landscape and transportation conditions are improving.

Figure 19. The Sankey map of the change from 2016 to 2018 in Wuhan central urban area.

6.4. Further uses and benefits of the WUSU dataset

WUSU is the first publicly available SCD dataset in Chinese urban areas with the most fine-grained feature classes and annotations. Researchers can use WUSU to train and evaluate algorithms, discover new associations and trends, and drive advances in remote sensing academia and industry.

WUSU can be used for a higher level of functional zoning in the urban scene dimension. By analyzing WUSU, differences of land use structure in different urban functional zones can be identified, which is of great significance for urban management and planning due to the rapid urban change (Feng et al. 2021; Huang et al. 2020). As shown in Figure 20, vegetation and buildings have different densities in different functional zones. Educational land uses appear to have more structures such as playgrounds. Vacant land uses are often covered by excavations and vegetation. WUSU can also further depict the residential lands.

Figure 20. Schematic of the WUSU dataset guiding the perception of urban functional areas.

According to the Chinese Code for Classification of Urban Land Use and Planning Standards of Development Land (GB 50137-2011), residential lands can be classified into three levels based on factors such as the environment, facilities and architectural characteristics. These elements can be depicted using WUSU, as illustrated in Figure 21.

Figure 21. Examples of the WUSU dataset guiding the more detailed classification of ‘Residential Land’.

These analyses combined WUSU and urban functional zones are helpful for urban planning departments to monitor and make decisions on urban development to improve the quality of urban development, and will be one of our future research.

7. Conclusion

The WUSU dataset is a tri-temporal remote sensing dataset made up of GF-2 remote sensing imagery with high spatial resolution, which is appropriate for LULC classification, BCD, and SCD. WUSU has distinctive characteristics, i.e. geographic-temporal continuity, a refined class system, and rich urban structure information. In this study, 7 public deep learning based LULC classification methods, 6 BCD methods and 3 multi-task frameworks are selected to tested for WUSU. To complete refined LULC classification maps as well as more accurate change detection maps, MJ loss was introduced to the proposed ChangeMJ framework to recover the building boundaries. MJF₁ was therefore proposed to evaluate the degree of mapping refinement and boundary retention. The experimental results showed that the ChangeMJ framework achieved the best results in all the three tasks and confirmed the importance of multi-temporal semantic features for change detection tasks. The generalization and validity of ChangeMJ was also confirmed on another SCD datasets. The WUSU dataset is the only tri-temporal semantic annotation dataset for China based on high resolution GF-2 images. In our future work, the WUSU dataset will be expanded and applied to other tasks.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The proposed WUSU dataset and other mentioned public datasets can be accessible at: https://github.com/AngieNikki/openWUSU.

Additional information

Funding

This work was supported by National Key Research and Development Program of China under grant number 2022YFB3903404, National Natural Science Foundation of China under grant number 42325105, 42071350, and LIESMARS Special Research Funding.