Multi-class change detection of remote sensing images based on class rebalancing

ABSTRACT

Multi-class change detection can make various ground monitoring projects more efficient and convenient. With the development of deep learning, the multi-class change detection methods have introduced Deep Neural Network (DNN) to improve the accuracy and efficiency of traditional methods. The class imbalance in the image will affect the feature extraction effect of DNN. Existing deep learning methods rarely consider the impact of data on DNN. To solve this problem, this paper proposes a class rebalancing algorithm based on data distribution. The algorithm iteratively trains the SSL model, obtains the distribution of classes in the data, then expands the original dataset according to the distribution of classes, and finally trains the baseline SSL model using the expanded dataset. The trained semantic segmentation model is used to detect multi-class changes in two-phase images. This paper is the first time to introduce the image class balancing method in the multi-class change detection task, so a control experiment is designed to verify the effectiveness and superiority of this method for the unbalanced data. The mIoU of the class rebalancing algorithm in this paper reaches 0.4615, which indicates that the proposed method can effectively detect ground changes and accurately distinguish the types of ground changes.

KEYWORDS:

• Multi-class change detection
• remote sensing
• class rebalancing
• semantic segmentation

1. Introduction

In recent years, due to the implementation of earth observation projects such as Copernicus, ZiYuan, Gaofen, and WorldView, various sensors provide a great number of remote sensing images for the analysis of the human living environment (Migas-Mazur et al. 2021; Panuju, Paull, and Griffin 2020). Therefore, it is extremely urgent to develop efficient and accurate techniques for mining more meaningful knowledge from remote sensing images. In all kinds of remote sensing image processing technology, multi-class change detection technology is a very popular and widely used direction. Change detection technology is designed to detect the specific regions that have changed in the image pairs of the same region or sequence of images acquired on different dates (Hussain et al. 2013; Hou, Wang, and Liu 2016; Peng et al. 2020). This means that change detection will play a great role in human settlement environment monitoring (Asokan and Anitha 2019; Chen et al. 2012), natural resources monitoring (Hegazy and Kaloop 2015; Lu et al. 2004), and natural disaster monitoring (Brisco et al. 2013; Singh et al. 2008).

Change detection technology develops with the development of image processing and deep learning (Santana et al. 2019). Since the development of imaging technology was relatively slow in the early days, sensors could only generate remote sensing images of medium and low resolution, so the main target of change detection research at that time was remote sensing images of medium and low-resolution (Hussain et al. 2013). The medium and low-resolution remote sensing images give birth to many classical remote sensing image change detection methods. Most of these algorithms use the pixel relationship between two-time phases to judge whether the ground changes (Munyati 2000). Due to the rapid development of the imaging capability of sensors, a large number of high-resolution remote sensing images have been presented to remote sensing data scientists in recent years. For example, UHN-DCVA initializes the untrained model to distinguish changed and unchanged pixels, and then implements change detection by clustering method (Saha, Bovolo, and Bruzzone 2019; Saha et al. 2021).

The development of imaging capability has made the target of change detection more complicated, and now the more needed detection result is what happened to the ground, that is, multi-class change detection (Mountrakis, Im, and Ogole 2011). The most commonly used multi-class change detection task is the deep neural network, which can learn image features with a broad perspective and strong image understanding ability (Guo et al. 2021). Most of the current methods rely on the deep neural network. For example, Multitask learning mainly carries out multi-class change detection through a large amount of data (Daudt et al. 2019), and ChangeMask uses a difference graph to make a trade-down of semantic segmentation maps to obtain multi-class change detection results (Zhuo et al. 2022). These methods use a large amount of data to train neural networks. However, training deep neural networks requires a large amount of data. In reality, it is difficult for us to find a large amount of data in the region of interest in time. Semi-supervised deep learning can only rely on a small amount of data to learn the strong coupling and highly nonlinear relationship among data (Ding, Nasrabadi, and Fu 2018). The multi-class change detection task is mainly to find the changing relationship between highly nonlinear and strongly coupled data. Therefore, semi-supervised learning is very suitable for solving the multi-class change detection problem.

The problem of multi-class change detection can be solved from multiple angles (Chen et al. 2012; Guo et al. 2021; Nemmour and Chibani 2010; Walter 2004), and this study uses semi-supervised learning to achieve it. Firstly, a pre-trained deep neural network is used for semantic segmentation of the original two-phase remote sensing image, and then the resulting figure of semantic segmentation is used for multi-class change detection. This requires the precision of semantic segmentation to be high enough. If the precision of semantic segmentation is not high enough, the final change detection results will be poor. The results of semantic segmentation depend on the segmentation accuracy of deep pre-trained neural networks, but the accuracy of deep neural networks is often very poor due to the imbalanced distribution of data features (Wang et al. 2016; Yan et al. 2015).

At present, many researchers have proposed some methods to solve the problem of the unbalanced data class distribution, such as resampling (Yu et al. 2018), re-weighting (Fernando and Tsokos 2021), and two-stage (Guzmán-Ponce et al. 2021), but these methods all rely on the pseudo-labels generated by the deep neural network to balance the unbalanced data class (Wei et al. 2021).

These methods are highly dependent on the truthfulness of pseudo-labels. SeK inhibits the effect of label imbalance during training and testing (Yang et al. 2022). However, deep neural networks that have completed pre-training tend to have a certain preference for categories. The pre-training process and data set selection process of the deep neural network need further study. Figure 1 shows the class distribution of the SECOND dataset (Yang et al. 2022). The data volume of classes 6, 7, and 8 is much smaller than that of other classes, generally referred to as ‘the tail data’. DNN models are more likely to predict most classes and their ability to predict tail data will be reduced.

Figure 1. The percentage of each class in the dataset.

To overcome the impact of unbalanced data class distribution on deep neural networks and solve the multi-class change detection problem of remote sensing images with unbalanced class distribution, the following solutions are proposed in this paper.

1. For change detection, we solve the problem from the perspective of semantic segmentation by using a trained model to perform semantic segmentation on two-phase images, and then further process the semantic map to obtain change detection results.

2. Aiming at the problem of class imbalance of general data, we propose an algorithm based on class rebalancing, which can effectively reduce the influence of class imbalance on a deep neural network.

3. For multi-class change detection, this paper obtains a binary map by subtracting a semantic segmentation map and then selects a semantic segmentation map by using a binary map to obtain a multi-class change detection result.

The rest of this paper is organized as follows: section Ⅱ discusses the related work, section Ⅲ introduces the method we proposed in detail, section Ⅳ is the experiment, section Ⅴ further discusses the method, and finally, section Ⅵ summarizes the paper.

2. Previous work

This section first briefly reviews the development of semantic segmentation, then gives a summary of class rebalancing methods, and finally gives a brief description of multi-class change detection methods.

2.1. Semantic segmentation

In recent years, due to the development of imaging technology and data transmission capabilities, it has been very easy to obtain image data (Champagne et al. 2014). Accordingly, driven by data, many methods have been used for semantic segmentation of remote sensing images (Li et al. 2021). However, due to the strong ability of a deep neural network for image feature extraction, and its more flexible structure, it is more conducive to semantic segmentation under different task backgrounds, so the current semantic segmentation method based on the deep neural network still occupies a dominant position.

In (Papandreou et al. 2015), the author uses a deep convolutional neural network based on weakly-supervised learning for semantic segmentation and proposes a solution for the combination of weakly labeled data, well-labeled data, and improperly labeled data. In (Long, Shelhamer, and Darrell 2015), the full-convolutional neural network is used to obtain images of arbitrary size and generate outputs of corresponding spatial dimensions. In (Jégou et al. 2017), DenseNet architecture is extended to complete convolutional networks for semantic segmentation. All of these methods achieve a high level of semantic segmentation accuracy in their problems, but they are prone to fall into the trap of simply pursuing network depth and superimposing computer computing power.

In (Ronneberger, Fischer, and Brox 2015), data enhancement is performed applying elastic deformation to available data, which can achieve segmentation of medical images with a small amount of data. In (Wu et al. 2019), a video-based method is proposed to expand the training set by synthesizing new training samples, aiming to reduce the problems existing in the data itself by utilize the relationship between adjacent frames. ResNet network is a milestone event in the history of deep neural network image processing (Ren et al. 2015). ResNet has made great achievements in many images segmentation and target detection contests and is widely used in current semantic segmentation tasks on account of its classical residual structure that can reduce a great number of network parameters.

2.2. Class rebalancing

Although at present, all deep neural networks need pre-training on a large amount of data. Due to the different degree of class equilibrium between the training data and the target data, almost all the deep neural networks completed by training will show some preferences, that is, it is particularly easy to judge the uncertain classes like the ones that appear most frequently in the training data (Wang, Wang, and Zhao 2020). The decision preference problem of a neural network can be solved by balancing the classification features in the original data. The common classification balancing methods include resampling, re-weighting, and two-stage methods.

The class rebalancing of the segmentation task is different from that of image classification. The common methods of class rebalancing in image classification task are pseudo-label class rebalancing (Tantithamthavorn, Hassan, and Matsumoto 2018). Because there are many classes in each image in the segmentation task, it is not easy to select data samples directly through the sample selection method, so the class rebalancing method required by the segmentation task is more complex. Many studies use pre-trained deep neural networks to add false labels to data for data class rebalancing, and some studies use data flipping and rotation to achieve partial class balancing (Houborg and McCabe 2018).

In terms of methodology, the segmentation problem can learn from other research methods. Class imbalance is common in deep neural networks that require a lot of data training. Traditional data-level solutions include resampling and undersampling, etc., while algorithmically based solutions mainly include cost-sensitive learning (Ling and Sheng 2008). From the data level solution, simply over-sampling or under-sampling all data will lead to more unbalanced classes, while from the algorithm level solution, it is difficult to determine the penalty cost of each data class. Class rebalancing algorithm mainly solves the problem of image classification and balances the raw data by using the decision preference of deep neural network model, but it needs to be improved for image segmentation task.

2.3. Multi-class change detection

Single-class change detection can usually obtain a binary image of the ground. Generally, white represents the changing area and black represents the unchanged area. Before the appearance of a deep neural network, change detection is generally carried out by pixel matching method, which can almost only achieve single-class change detection (Chang and Ho 2016). After the appearance of a deep neural network, many studies use the powerful feature extraction ability of a deep neural network to extract image features, and then further processing to obtain a single class change detection figure (Snyder et al. 2018).

With the development of computing power and sensors, single-class change detection can no longer meet the needs of land monitoring, disaster relief, environmental monitoring, and other tasks. Multi-class change detection is to obtain the specific land types before and after the changes in the image change area. In (Zheng et al. 2021), the author used the data of external data sets to carry out knowledge transfer, which to obtain better multi-class change detection results. In (Chen and Shi 2020), the author used the attention network to extract the changing areas, thus obtaining more accurate results.

Multi-class change detection can be directly accomplished through the existing semantic segmentation network. In (Yang et al. 2021), the researcher used two semantic segmentation networks to process remote sensing images of two phases respectively, and then directly subtracted them, and then used the subtracted binary graph to make trade-offs for the original semantic segmentation results, and finally obtained multi-class change detection results. This is a simple and easy method, but the conventional multi-class change detection dataset usually only marks the class of the changing region, and is not interested in the class of the unchanged region. Therefore, this paper designs an ingenious method to reduce the influence of unchanged regions on semantic segmentation.

3. Proposed method

To solve the problems existing in multi-class change detection, this paper proposes a segmentation-based multi-class change detection method for remote sensing images, and the main process is shown in Figure 2.

Figure 2. Flowchart of the proposed method.

The method consists of two parts. The first part is the class rebalancing training algorithm part, which iterates the training baseline SSL model by using the class rebalancing training algorithm, and obtains a semantic segmentation model completed by training.

The second part is the multi-class change detection part, which obtains the multi-class change detection results by generating semantic segmentation graph and further processing.

3.1. Semantic segmentation based on weight sharing

The most common detection methods in the field of change detection mainly use two semantic segmentation networks with shared weights to perform semantic segmentation for images of different phases. Without losing generality, this paper compares the results of various semantic segmentation backbones. In this paper, feature extraction and semantic segmentation of remote sensing images are carried out using a semantic segmentation network based on weight sharing.

Many excellent research achievements have been published in the field of remote sensing image semantic segmentation, such as Unet (Dey et al. 2021), ResNet, MobileNet (García, Donini, and Bovolo 2021), and so on. Without losing generality, this paper compares the results of various semantic segmentation backbone.

The weight-sharing semantic segmentation network adopted in this paper is shown in the figure. The semantic segmentation feature extraction networks we choose mainly include VGG16, ResNet, HRNet (Wang et al. 2020), etc. The commonly used classification loss function is mainly the cross-entropy loss function (De Boer et al. 2005), which is often used in classification problems, especially when neural networks do semantic segmentation, also often used as the cross-entropy loss function. To verify the effectiveness of the proposed method, this paper does not improve the cross-entropy loss function, but directly uses the cross-entropy loss function to train the cross-entropy loss function of the multi-classification semantic segmentation network, as shown in Formula 1: (1) $\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}=-\sum _{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{y}}_{\mathrm{i}}\log \left({\mathrm{p}}_{\mathrm{i}}\right)$(1) Where N is the number of classes, ${\mathrm{y}}_{\mathrm{i}}$ is the class label, and ${\mathrm{p}}_{\mathrm{i}}$ is the class probability output by the neural network.

3.2. Class rebalancing algorithm

The neural network has very limited feature extraction ability for unbalanced data. To solve this problem, we introduce a class rebalancing training algorithm. The original class rebalancing algorithm is for the classification task. This paper improves it for the multi-class change detection task.

The main process of the class rebalancing algorithm is as follows: in the process of training a semi-supervised model, the tail data predicted by the model is obtained, and then the tail data is added to the original training set to continue training the semi-supervised model. This algorithm has two main advantages, one is to make the model can be further trained, the other is to expand the original dataset so that the class distribution of the data set is more uniform. However, the original class rebalancing algorithm is also inadequate, and it is used for image classification. There are many differences between segmentation and classification tasks, so this paper proposes a class rebalancing algorithm for semantic segmentation.

The main process of the class rebalancing algorithm for semantic segmentation is shown in Figure 3.

Figure 3. Class rebalancing algorithm.

First, we use dataset A to train A simple semantic segmentation model, then make predictions on dataset B based on the trained model, and make statistics on the predicted results. Finally, we extend the data of the tail class to the original dataset B. There are mainly selective expansion methods and data distribution-based expansion methods. This paper will discuss the specific effects of these two methods in the experimental part.

In the simple expansion method, this paper chooses to fill all the minority types of data into the original data set according to the data expansion method. The distribution-based class rebalancing algorithm first counts the network prediction results, and then adds the data with tail classes to the original dataset B according to the distribution of results to form $\hat{\mathrm{B}}$. $\hat{\mathrm{B}}$ is an extended set of B. The data expansion method used in this paper is shown in Figure 4, which is mainly to rotate, flip, stretch, and other operations after extracting key areas.

Figure 4. Data expansion method.

The data expansion method is described as: define a set, including vertical flip, horizontal flip, rotation, center clipping, etc., and then select a random method in the set to transform image data when data needs to be filled.

3.3. Multi-class change detection method

The objective of multi-type change detection is to detect changes in two remote sensing images of the same areas at a different time and determine the types of changes. To make full use of the prediction results of the semantic segmentation network, this paper directly subtracted two segmentation graphs. Multi-class change detection methods are shown in Figure 5.

Figure 5. Multiclass change detection method.

First, we use a predictive network to obtain two semantic segmentation results. Then the two segmentation result figures are directly subtracted to obtain a segmentation difference figure. The difference between the segmentation difference map and the original segmentation result map is that pixel 0 in the segmentation difference map means that the ground has not changed, while pixel, not 0 means that the ground has changed. Therefore, the specific change types of the two images can be obtained by using the segmentation difference map to choose the two segmentation maps.

4. Experiment

The experiment part firstly introduces the data set and the evaluation index of the detection effect and then describes the model parameter setting. Next, to verify the effectiveness of the class rebalancing strategy, three comparative experiments are set up in this paper. In the first experiment, only the change detection framework was used without the class balancing strategy. In the second experiment, a simple class balancing strategy was set up. In the third experiment, the class rebalancing strategy was used to obtain a distribution-based enhanced data set, which was used to train the baseline SSL model for multi-class change detection.

4.1. Datasets and evaluation metrics

4.1.1. Dataset description

The data set applied in this paper is mainly divided into two parts, the first part is the pre-training of semantic segmentation SSL model, and the second part is the data set used for change detection. The data set used for training the SSL model of semantic segmentation is mainly semantic segmentation data set. In order to train the SSL model of semantic segmentation, two similar classes are introduced in this paper. The Gaofen Image Dataset (GID) is a large Dataset for land use and land cover (LULC) classification (Tong et al. 2018). It contains 150 high-quality GF-2 images from more than 60 different cities in China, covering a geographical area of more than 50,000 km². GID images have high intraclass diversity and low interclass separability. This data set includes panchromatic images with a spatial resolution of 1m and multispectral images with a spatial resolution of 4 m. In this paper, 1m panchromatic images were selected for training with an image size of 6908×7300. The DeepGlobe Land Cover Classification Challenge (DLCC) is a public dataset that provides high-resolution sub-mi satellite imagery, with a focus on rural areas (Demir et al. 2018). The dataset contains 10,146 satellite images with a size of 20448×20448. This paper selects 803 images from the training set Table 1.

Table 1. Datasets description.

Dataset		Region	Resolution	Size	Number of images
Semantic segmentation	GID	China	1m	6908×7300	15
	DLCC	–	1m	20448×20448	803
Multiclass CD	SECOND	China	3m	512×512	4662

4.1.2. Evaluation metrics

The essence of multi-class change detection of remote sensing images is to recognize and mark the ground change type of remote sensing image in the image. Therefore, this paper uses the conventional image segmentation evaluation index as the measurement standard of the method proposed in this paper. True Positive, False Positive, True Negative and False Negative are common indicators used to measure the effect of image detection or recognition in image tasks. Since the essence of the multi-class change detection task is to detect and segment ground targets, IoU and mIoU are used in this paper to evaluate the effectiveness of the proposed method. (2) $\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$(2) (3) $\mathrm{m}\mathrm{I}\mathrm{o}\mathrm{U}={\displaystyle \frac{1}{\mathrm{N}}\sum _{\mathrm{i}=1}^{\mathrm{N}}\mathrm{I}\mathrm{o}{\mathrm{U}}_{\mathrm{i}}}$(3) Formula 2 and Formula 3 are the calculation formulas of IoU and mIoU respectively, where TP refers to True Positive, that is, Positive cases that are correctly predicted, FP is Positive cases that are incorrectly predicted, FN is negative cases that are incorrectly predicted, and N is the number of classes in the data. The mIoU is the average of the IoU of all classes in the dataset.

4.2. Experimental setup

4.2.1. Backbone

In this paper, the semantic segmentation method is used to detect the multi-class change in remote sensing images, so the quality of semantic segmentation determines the accuracy of multi-class change detection results. However, the accuracy of semantic segmentation has always been different due to the class imbalance of remote sensing images. In order to test the effect of the distribution-based class rebalancing algorithm, the semantic segmentation framework in this paper is ResNet, VGG, or HRNet with weight sharing. This paper focuses on distribution-based class rebalancing algorithms and minimizes the interpretation of feature extraction models.

4.2.2. Original data and simple data balancing

To verify the superiority of the class rebalancing algorithm, this paper first sets up multi-class change detection based on the original data. This group of experiments does not do any processing on the original data but directly puts the data into the semantic segmentation model completed by pre-training to obtain the results of multi-class change detection.

Then, the experiment set is a selective class rebalance of classes. In this paper, the three classes in the tail are enhanced according to the distribution of data, that is, the three classes of ‘sports field’, ‘trees’, and ‘water’ in the data are randomly rotated and flipped respectively. Since it is image data, clipping requires certain rules. After many experiments and data statistics on the data set, the clipping area with pixel number less than 50×50 is not suitable for data rebalancing. At the same time, this paper sets the labels of other classes in the clipping area to zero, that is, the pixels of non-target classes are ignored when calculating the gradient. Therefore, the subsequent expansion of data sets in distribution-based class rebalancing algorithms also identified this rule as a standard.

Formula 4 is the method to expand the data in this paper, where refers to the number of pixels of class i in the result predicted by the pre-trained SSL model, refers to the proportion of class i in the result predicted by the pre-trained SLL model. The formula means that the result obtained by multiplying $\mathrm{N}\mathrm{u}{\mathrm{m}}_{\mathrm{i}}$ by$|\ln ({\mathrm{P}}_{\mathrm{i}}\left)\right|$ is the number of pixels $\mathrm{N}\mathrm{u}{\mathrm{m}}_{\mathrm{i}}^{\mathrm{n}\mathrm{e}\mathrm{w}}$of class i in the expanded dataset. (4) $\mathrm{N}\mathrm{u}{\mathrm{m}}_{\mathrm{i}}^{\mathrm{n}\mathrm{e}\mathrm{w}}=\mathrm{N}\mathrm{u}{\mathrm{m}}_{\mathrm{i}}\ast |\ln ({\mathrm{P}}_{\mathrm{i}}\left)\right|$(4)

4.2.3. Distribution-based rebalancing

In the experiment part, the classification rebalancing algorithm is mainly set as follows. After various considerations and experiments, this paper uses ResNet, VGG, or HRNet as feature extraction module and FCN as semantic segmentation module.

The iteration number of baseline model training is 70, and the model has already converged. In the experiment of class rebalancing, the clipping rule in the process of data expansion is still as follows: only when the number of pixels in the target area is greater than 50×50 can be used for data expansion. Meanwhile, the labels of other classes are set to zero to ensure that only the target class can be identified in the whole process.

4.3. Multi-class detection based on original data

In this section, a multi-class change detection experiment based on raw data is carried out. In order to compare the effect after the original data and class rebalancing, this section does not process any data and directly inputs the data into the network for pre-training.

Then the pre-trained neural network is used for semantic segmentation, and finally, the semantic segmentation graph is used for change detection. The finally obtained multi-class change detection results are shown in Figure 6. This section compares the effect of VGG16, VGG19, ResNet18, ResNet34, ResNet50, and HRNet as a backbone.

Figure 6. Multi-class change detection based on original data. (a) Two-phase input data, (b) Ground Truth, (c) VGG16, (d) VGG19, (e) ResNet18, (f) ResNet34, (g) ResNet50, (h) HRNet.

In this section, we select the basic network for semantic feature extraction through experiments. As this paper uses the semantic segmentation method to solve the problem of change detection, we need to choose a backbone with stronger feature extraction ability in many basic networks of semantic segmentation, so we mainly choose VGG, ResNet, HRNet, and other deep neural networks as the backbone of feature extraction based on summarizing previous studies. In this experiment, we did not perform any data enhancement on the original data, but just input the original data for training and then test. There are two main reasons for doing this: one is to select a backbone whose features are extracted; the other is to compare with the following two groups of experiments to verify the effectiveness of this method Table 2.

Table 2. Multi-class change detection in raw data.

Backbone	water	ground	low vegetation	trees	building	playground	mIOU
VGG16	0.2564	0.5015	0.4855	0.2744	0.5051	0.3584	0.3969
VGG19	0.2591	0.5088	0.4783	0.2775	0.5127	0.3588	0.3992
ResNet18	0.2878	0.5182	0.4975	0.2833	0.5352	0.3803	0.4171
ResNet34	0.2922	0.5203	0.5043	0.3012	0.5474	0.3895	0.4258
ResNet50	0.3176	0.5191	0.4910	0.3059	0.5437	0.3851	0.4271
HRNet	0.3103	0.5316	0.4853	0.3191	0.5309	0.3872	0.4269

Experimental results show that most classes have great advantages in network training and testing in raw data. DNN is more inclined to predict the uncertain class as the majority class, while the minority class is likely to be replaced by the majority class. Figure 6c and d show the detection effect of VGG16 and VGG19 as backbone respectively. It can be seen from Figure 6e, f, g and h that ResNet and HRNet have better effect as a backbone, but considering the high complexity of HRNet network, the training process is difficult to converge. In addition, under the same training conditions,ResNet network structure is simpler and the training process is shorter. Therefore, ResNet is used as backbone in the following experiments.

In various applications of change detection, the change of minority classes often determines the degree of completion of the task, so the phenomenon of the majority classes replacing minority classes is intolerable for the change detection task. In order to balance the difference between the majority classes and minority classes, the following two experiments are conducted in this paper for the problem of data class imbalance.

4.4. Selective class rebalancing

In this section, we carried out the control experiment of a selective class rebalance of data classes. Because the change detection effect of data class balance was relatively poor without it at all, and there were many missed and wrong checks, we carried out a selective class rebalance of data in the change detection. First, we screened out a few classes in the dataset and then clipped the areas where a few classes were located, followed by rotation, flipping, and other enhancement operations, and finally filled them into the original dataset.

Figure 7 shows the effect of selective class rebalancing of data classes. The results show that after selective class rebalancing, the effect of multi-class change detection has been greatly improved, and the semantic details of detection are more accurate. However, the improvement of detection accuracy for a few classes is not particularly obvious. At the same time, simple class balance is more likely to produce noise, affecting the final accuracy Table 3.

Figure 7. Data classes are selective class rebalanced. (a) Two-phase input data, (b) Ground Truth, (c) ResNet18, (d) ResNet34, (e) ResNet50.

Table 3. Data classes are selective class rebalanced.

Backbone	water	ground	low vegetation	trees	building	playground	mIOU
ResNet18	0.3012	0.5099	0.4965	0.3001	0.5277	0.4019	0.4229
ResNet34	0.3109	0.5183	0.5081	0.3184	0.5387	0.4033	0.4330
ResNet50	0.3155	0.5192	0.4993	0.3198	0.5390	0.4057	0.4331

4.5. Class rebalancing based on distribute

In the distribution-based class rebalancing experiment stage, the SSL model completed by pre-training are firstly used to predict the target data set, and then class statistics is performed on the predicted results. According to the statistical results, remote sensing images are pruned and enhanced, and finally added to the original data set for network training.

The class rebalancing effect based on data distribution is shown in Figure 8. It can be seen clearly that the effects of the two-time phases before and after are more accurate than the experimental effects of the previous two sections. The details of the image can be detected more completely.

Figure 8. Class rebalancing based on distribution. (a) Two-phase input data, (b) Ground Truth, (c) ResNet18, (d) ResNet34, (e) ResNet50.

If the conventional deep learning algorithm needs to improve the segmentation accuracy, it must increase the depth of the network itself. The distribution-based class rebalancing algorithm is an algorithm that does not need to rely on powerful computing power and a deeper network structure. This algorithm uses DNN itself to enhance the characteristics of data, partially enhancing the classes that DNN cannot easily identify, and does not further process the classes that DNN can easily identify. Table 4 shows the results of the distribution-based class rebalancing algorithm.

Table 4. Class rebalancing algorithm based on distribution.

Backbone	water	ground	low vegetation	trees	building	playground	mIOU
ResNet18	0.3698	0.5083	0.5185	0.3872	0.5217	0.4286	0.4557
ResNet34	0.3751	0.5197	0.5174	0.3908	0.5301	0.4321	0.4609
ResNet50	0.3716	0.5170	0.5167	0.3885	0.5458	0.4291	0.4615

4.6. Lateral correlation experiment

The above three experiments step by step verify the role of the class rebalancing algorithm in the multi-class change detection task, and the conclusion is that the class rebalancing algorithm is better than the original data and simple class balance experiment, and can more accurately identify ground changes. In order to better demonstrate the effectiveness of the proposed method, this section compares it with some related multi-class change detection methods. In order to better demonstrate the effectiveness of category rebalancing in the multi-class change detection task, this paper selected some methods to solve the imbalance of class distribution, such as Bi-SRNet (Ding et al. 2021), ASN (Yang et al. 2022), GCF-Net (Tang 2021) Figure 9.

Figure 9. Lateral Correlation experiment. (a) Ground Image, (b) Ground Truth, (c) Bi-SRNet, (d) GCF-Net, (e) ASN, (f) proposed method (ResNet50).

These methods considering the class balance problem have good overall performance accuracy, among which Bi-SRNet can effectively detect changes in most classes, but the accuracy is slightly poor in a few classes. GCF-Net has higher overall accuracy than Bi-SRNet, but the accuracy of most categories is lower than other methods. ASN and the proposed method have better performance and can effectively detect multiple types of changes occurring on the ground. It can be seen from the analysis that the recognition accuracy of these methods on water, trees, and playground is low, which proves that although the method of class balancing is used in this paper, the recognition accuracy of other categories cannot be achieved by class rebalancing algorithm because the proportion of these classes is too small. Although algorithms can solve some problems derived from data, a better way is to obtain data in a more comprehensive way Table 5.

Table 5. Lateral Correlation Experiment.

Method	water	ground	low vegetation	trees	building	playground	mIOU
Bi-SRNet	0.3413	0.5117	0.4927	0.4011	0.5105	0.3948	0.4420
GCF-Net	0.3991	0.4888	0.4824	0.3916	0.5128	0.4474	0.4537
ASN	0.3859	0.4654	0.5297	0.4034	0.5406	0.4354	0.4601
our	0.3716	0.5170	0.5167	0.3885	0.5458	0.4291	0.4615

5. Discussion

5.1. Semantic segmentation and change detection

Traditional change detection mainly uses a neural network to process the pixels of two-phase image pair by pair. Such a processing method has great limitations, for example, ground changes are often related to surrounding pixels. However, the pixel-by-pixel processing method pays less attention to the whole world, mainly focusing on the relationship between the two pixels before and after. Such change detection methods pay more attention to the correlation between pixels. There is not a strong correlation between pixels that have changed, but there is a strong correlation between pixels that have not changed.

Due to the rapid development of deep learning, change detection algorithms mainly rely on deep neural networks, and there are many change detection methods. However, there are little research on change detection from the perspective of semantic segmentation. If the change detection task is considered from this point of view, the precision of semantic segmentation determines the effect of change detection. It is easy to make the change detection task become a simple semantic segmentation task. From the perspective of semantic segmentation, change detection tasks are different. The image obtained after semantic segmentation is a two-phase semantic map. How to explain ground changes from a semantic map? In this paper, the specific ground changes in the changing area of the image of the two-time phases are used to represent the characteristics of the multi-analog changes, and the specific ground classes in the change detection task are accurately represented.

5.2. Effects of class imbalance

One of the common problems of current deep neural networks is that deep neural networks tend to have certain preferences on training data and show such preferences on test data. We often need a relatively balanced deep neural network, and we hope that it will not ignore a few targets in the test because there are few targets of a certain class in the training set. Generally speaking, the classes of training data often appear unbalanced distribution, and some data sets even appear long-tail distribution. This is tricky for many deep neural networks today.

For the long-tail distributed data set, we can choose two solutions: one is to resample the data directly before network training. In this way, the classification distribution of data is directly calculated, regardless of the difficulty of network feature extraction. One problem with this is that some minority class features are so obvious that the neural network only needs one or two training sessions to accurately identify it as a minority class. In other words, there may be a few classes of features that are extremely easy for neural networks to judge, but the data resampling method before network training does not take this special case into account.

The other is the algorithm proposed in this paper. Firstly, we use the pre-trained network to carry out forward propagation, observe the results of pre-training, judge which minority classes the network is more sensitive to, and then form a distribution-based class rebalancing data set to continue training the network. This algorithm can not only take into account the original data extraction ability of the network but also enhance the classes that are easy to misjudge in the training stage, making the DNN used for feature extraction more balanced for each class.

The results of the data balancing method used in the experiment in this paper are shown in Figure 10, and figure 10a represents the class proportion of the original data. It can be seen that the difference between the classes of the original data has reached more than 50 times, which will definitely produce a large preference for the SSL model. In the selective class rebalancing stage, this paper directly multiplies the data of a few classes in the original label, and the results after expansion are shown in Figure 10b. Figure 10c shows the class distribution of the results obtained by the pre-trained SSL model. It can be seen that the SSL model obviously shows a class preference, which is more inclined to judge the uncertain classes as the majority classes, while the judgment ability of minority classes is poor. Figure 10d shows the class proportion after the data is balanced by the distribution-based class rebalancing algorithm. Although the degree of class balance of data may be similar, this paper makes use of the preference of the pre-trained SSL model for other data set classes, which makes the detection effect better.

Figure 10. The proportion of data classes in the three experiments. (a) The class proportion of the original data, (b) The class proportion after selective class rebalancing, (c) The class proportion of the segmentation result using a pre-trained neural network, (d) The class proportion after balancing the data using a distribution-based class rebalancing algorithm.

5.3. Weight sharing

In the current twin structure, training two SSL models can consume time and computing power exponentially, and training one SSL model can save a significant amount of computing power and time. In contrast, in two-time change detection tasks, it is a good idea to train an SSL model and then copy it to use on another branch. However, there are some problems, that is, after training on one data set, the adaptive data style is often limited to that data set, even in change detection tasks: SSL models trained on the previous data set cannot adapt to the data on the latter one.

The essence of weight sharing is to reduce the training complexity of the network and save computing power and time. If the weight is not shared, it means that two SSL models need to be trained, which not only directly doubles the number of parameters but also doubles or even exponentially increases the computational complexity. Weight sharing is one of the important means to accelerate the multi-class change detection model in this paper.

6. Conclusion

In order to solve the problem of unbalanced data class distribution in the current multi-analog change detection algorithm, a multi-class change detection method based on class rebalancing is proposed in this paper. The proposed method uses a distribution-based class rebalancing algorithm to balance the classes in the data set so that the model can learn all the features better. This method is divided into two stages: the first stage is a distribution-based class rebalancing algorithm, and the second stage is multi-class change detection. In the stage of distribution-based class rebalancing algorithm, we train the semantic segmentation model iteratively and rebalance the classes of data at the same time. We train the semantic segmentation network several times with the balanced dataset and then obtain a network with low-class sensitivity. In the phase of change detection, this paper firstly uses a semantic segmentation network to perform semantic segmentation for two-phase images and then subtracts the segmentation graph to obtain a binary graph. Finally, the binary graph is used to choose the semantic segmentation results and obtain the multi-analog change detection results.

The effectiveness and accuracy of the proposed method are compared vertically in this paper because the research on multi-class change detection is in its infancy and there are few comparative studies. The validity and accuracy of the proposed method are verified in the experiments of multi-analog change detection based on raw data, selective class rebalance experiments based on data classes, and class rebalancing experiments based on distribution.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The authors confirm that all datasets used in this study are open to the public, and the source of the data can be found in the introduction section of datasets.

Additional information

Funding

This work was supported by Guizhou Science and Technology Cooperation Program: [Grant Number QKH[2016]5103].