Deep learning for change detection in remote sensing: a review

Figures & data

Figure 1. Publication numbers per year of DLCD.

Table 1. The first proposed reference and DLCD reference for each DLNN.

DLNN	Deep learning Reference	DLCD reference
CNN	LeCun et al. (1998)	Iino et al. (2018)
AE	Hinton and Salakhutdinov (2006)	Chen, Shi, and Gong (2016)
DBN	Hinton, Osindero, and Teh (2006)	Argyridis and Argialas (2016)
RNN	Bengio, Simard, and Frasconi (1994)	Lyu, Lu, and Mou (2016)
GANs	Goodfellow et al. (2014)	Gong et al. (2017a)

Figure 2. The usage of spectral information in DLCD.

Figure 3. An example of the usage of spatial information in DLCD.

Figure 4. The usage of temporal information in DLCD.

Figure 5. An example of the usage of multi-sensor information in DLCD.

Figure 6. The structure of DLCD methods. Yellow boxes represent DLNNs while blue boxes denote the interaction of multi-date information. (a) Post-classification change method (PCCM). (b) Differencing method (DM). (c) Direct classification method (DCM). (d) Differencing neural network method (DNNM). (e) Mapping transformation method (MTM). (f) Recurrent method (RM). (g) Adversarial method (AM).

Table 2. The taxonomy, difference, definition, advantages, limitations, and applications of DLCD methods.

Category	Difference	Sub-category	Definition	Advantages	Limitations	Applications
3.1.Separate DLNNs (No interaction between DLNNs)	Adapted from conventional change detection methods, using DLNNs to refine features. They share the same principles with their corresponding conventional methods.	Post-Classification Change Method (PCCM)	Compares multi-date classification maps to identify changes.	abb3”>Easy to implement. Applicable to multi-sensor images. Provides complete from-to changes.	Errors from individual classification maps accumulated in the final change map.	Urban land use (Nemoto et al. 2017)
		Differencing Method (DM)	Compares the difference of image radiance or deep features to identify changes.	Easy to implement. Easy to interpret change detection results.	Difficult to provide complete from-to changes.	Urban land use (Xu et al. 2013), water and hazard (Zhao et al. 2014), and vegetation (Li et al. 2017)
		Direct Classification Method (DCM)	Stacks multi-date images or deep features to conduct classification directly.	Easy to implement.	Difficult to construct training samples. Difficult to provide complete from-to changes.	Urban land use (Zhang, Zhang, and Du 2016), water (Gao et al. 2019a)
3.2.Coupled DLNNs (DLNNs interact)	Novel strategies that modify deep learning models. The processes cannot be directly compared to conventional change detection methods.	Differencing Neural Network Method (DNNM)	Builds a cost function between two DLNNs to highlight the difference of deep features.	Accurate deep learning network parameters obtained.	Complex to implement. Difficult to construct training samples. Difficult to provide complete from-to changes.	Urban land use (Chu, Cao, and Hayat 2016), water and hazard (Chen, Shi, and Gong 2016), and vegetation (Geng et al. 2017)
		Mapping Transformation Method (MTM)	Constructs a transformation function layer between inconsistent deep features.	Applicable to multi-source or multi-spatial- resolution images.	Complex to implement. Difficult to construct training samples. Difficult to provide complete from-to changes.	Urban land use (Zhan et al. 2018), water and hazard (Zhang, Zhang, and Du 2016), and vegetation (Su et al. 2017)
		Recurrent Method (RM)	Uses recurrent connections between multi-date images to learn deep features.	Makes full use of temporal features.	Complex to implement. Difficult to construct training samples. Difficult to provide complete from-to changes.	Urban land use (Lyu, Lu, and Mou 2016) and vegetation (Song et al. 2018)
		Adversarial Method (AM)	Makes generator and discriminator neural networks play against each other.	Generates high-quality information.	Complex to implement. Difficult to provide complete from-to changes.	Urban land use and water (Gong et al. 2017a)

Figure 7. Distribution of overall accuracies for DLCD methods (differencing method (DM), direct classification method (DCM), differencing neural network method (DNNM), mapping transformation method (MTM), recurrent method (RM), and adversarial method (AM)).

Figure 8. Distribution of overall accuracies of binary changes for urban land use, water, hazard, and vegetation applications using the conventional change detection (CD) and DLCD methods.

Figure 9. Distribution of overall accuracies of multi-class changes for urban land use, water, hazard, and vegetation applications using the conventional change detection (CD) and DLCD methods.

Figure 10. Distribution of overall accuracies of from-to changes for urban land use, water, hazard, and vegetation applications using the conventional change detection (CD) and DLCD methods.

Table 3. The definition, advantages, limitations, and examples of solutions to the training sample dilemma in DLCD.

Category	Sub-category	Definition	Advantages	Limitations	Examples
5.1.1.Generating large training samples	Data Augmentation Method (DAM)	Utilizes affine transformation or data generation method to extend training samples.	Easy to implement.	Initial training samples needed. Accuracy of the extended training samples directly depends on initial training samples.	Zhan et al. (2017) Wang et al. (2018a)
	Supervised Change Detection Method (SCD)	Uses traditional supervised change detection methods to extend training samples.	Accurate training samples extended.	Initial training samples needed. Complex to implement.	Zhang et al. (2016b)
	Unsupervised Change Detection Method (USCD)	Uses traditional unsupervised change detection methods to generate training samples.	No initial training samples are needed.	Inaccurate training samples generated.	Zhao et al. (2014) Gong et al. (2015)
5.1.2.Adapting to small training samples	Semi-Supervised Method (SSM)	Uses the combination of labeled and unlabeled training samples for change detection.	Only a few training samples are needed.	Not transferred to other multi-date images.	Connors and Vatsavai (2017) Gong et al. (2019b)
	Transfer Learning Method (TLM)	Uses the pre-trained model from other training data to detect changes in the current multi-date images.	Only a few or no training samples are needed. Transferable to other images.	Transferability depends on the spectral similarity between the training data and the target image.	Lyu, Lu, and Mou (2016) Khan et al. (2017)
	Unsupervised DLCD Method (UDLCDM)	Combines unsupervised DLNNs with unsupervised change detection for change detection.	Fully automatic. No training samples are needed.	The application scenarios are restrained by unsupervised DLNNs and unsupervised change detection.	Su et al. (2016) Liu et al. (2016a)

Figure 11. The relationship between DLCD, remote sensing, change detection, and deep learning.

Table A1. Nomenclature.

Abbreviations	Meaning
DLCD	Deep Learning Change Detection
PCCMs	Post-Classification Change Methods
DCMs	Direct Classification Methods
CVA	Change Vector Analysis
DI	Difference Image
DLNNs	Deep Learning Neural Networks
CNNs	Convolutional Neural Networks
AEs	Auto-Encoders
DBNs	Deep Belief Networks
RNNs	Recurrent Neural Networks
GANs	Generative Adversarial Networks
DM	Differencing Methods
DNNM	Differencing Neural Network Method
MTM	Mapping Transformation Method
RM	Recurrent Method
AM	Adversarial Method
PCA	Principal Component Analysis
SDAE	Stacked Denoised Auto-Encoder
IQR	Inter-Quartile Range
CD	Change Detection
DAM	Data Augmentation Method
SCDM	Supervised Change Detection Method
USCDM	Unsupervised Change Detection Method
SSMs	Semi-Supervised Methods
TLM	Transfer Learning Method
UDLCDMs	Unsupervised DLCD Methods
GCN	Graph Convolutional Network

LeCun, Y., L. Bottou, Y. Bengio, and P. Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86 (11): 2278–2324. doi:10.1109/5.726791.

 Web of Science ®Google Scholar

Iino, S., R. Ito, K. Doi, T. Imaizumi, and S. Hikosaka. 2018. “CNN-Based Generation of High-Accuracy Urban Distribution Maps Utilising SAR Satellite Imagery for Short-Term Change Monitoring.” International Journal of Image and Data Fusion 9 (4): 302–318. doi:10.1080/19479832.2018.1491897.

 Web of Science ®Google Scholar

Hinton, G.E., and R.R. Salakhutdinov. 2006. “Reducing the Dimensionality of Data with Neural Networks.” Science 313 (5786): 504–507. doi:10.1126/science.1127647.

 PubMed Web of Science ®Google Scholar

Chen, F., J. Shi, and M. Gong. 2016. “Differencing Neural Network for Change Detection in Synthetic Aperture Radar Images.” International Conference on Bio-Inspired Computing: Theories and Applications 431–437. doi:10.1007/978-981-10-3611-8_38.Chen.

 Google Scholar

Hinton, G.E., S. Osindero, and Y.W. Teh. 2006. “A Fast Learning Algorithm for Deep Belief Nets.” Neural Computation 18 (7): 1527–1554. doi:10.1162/neco.2006.18.7.1527.

 PubMed Web of Science ®Google Scholar

Argyridis, A., and D.P. Argialas. 2016. “Building Change Detection Through Multi-Scale GEOBIA Approach by Integrating Deep Belief Networks with Fuzzy Ontologies.” International Journal of Image and Data Fusion 7 (2): 148–171. doi:10.1080/19479832.2016.1158211.

 Web of Science ®Google Scholar

Bengio, Y., P. Simard, and P. Frasconi. 1994. “Learning Long-Term Dependencies with Gradient Descent is Difficult.” IEEE Transactions on Neural Networks 5 (2): 157–166. doi:10.1109/72.279181.

 PubMed Web of Science ®Google Scholar

Lyu, H., H. Lu, and L. Mou. 2016. “Learning a Transferable Change Rule from a Recurrent Neural Network for Land Cover Change Detection.” Remote Sensing 8 (6): 506. doi:10.3390/rs8060506.

 Web of Science ®Google Scholar

Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. “Generative Adversarial Nets.” Advances in Neural Information Processing Systems 27. doi:10.5555/2969033.2969125.

 Google Scholar

Gong, M., X. Niu, P. Zhang, and Z. Li. 2017a. “Generative Adversarial Networks for Change Detection in Multispectral Imagery.” IEEE Geoscience and Remote Sensing Letters 14 (12): 2310–2314. doi:10.1109/Lgrs.2017.2762694.

 Web of Science ®Google Scholar

Nemoto, K., R. Hamaguchi, M. Sato, A. Fujita, T. Imaizumi, and S. Hikosaka. 2017. “Building Change Detection via a Combination of Cnns Using Only RGB Aerial Imageries.” Remote Sensing Technologies and Applications in Urban Environments II 10431: 104310J. doi:10.1117/12.2277912.

 Google Scholar

Xu, Y., S. Xiang, C. Huo, and C. Pan. 2013. “Change Detection Based on Auto-Encoder Model for VHR Images.” Mippr 2013: Pattern Recognition and Computer Vision 8919: 891902. doi:10.1117/12.2031104.

 Google Scholar

Zhao, J., M. Gong, J. Liu, and L. Jiao. 2014. “Deep Learning to Classify Difference Image for Image Change Detection.” In 2014 International Joint Conference on Neural Networks (IJCNN), 411–417. doi:10.1109/IJCNN.2014.6889510.

 Google Scholar

Li, Y., L. Zhou, G. Lu, B. Hou, and L. Jiao. 2017. “Change Detection in Synthetic Aperture Radar Images Based on Log-Mean Operator and Stacked Auto-Encoder.” In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 3090–3306. doi:10.1109/IGARSS.2017.8127652.

 Google Scholar

Zhang, L., L. Zhang, and B. Du. 2016. “Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art.” IEEE Geoscience and Remote Sensing Magazine 4 (2): 22–40. doi:10.1109/MGRS.2016.2540798.

 Web of Science ®Google Scholar

Gao, F., X. Wang, Y. Gao, J. Dong, and S. Wang. 2019a. “Sea Ice Change Detection in SAR Images Based on Convolutional-Wavelet Neural Networks.” IEEE Geoscience and Remote Sensing Letters 16 (8): 1240–1244. doi:10.1109/LGRS.2019.2895656.

 Web of Science ®Google Scholar

Chu, Y., G. Cao, and H. Hayat. 2016. “Change Detection of Remote Sensing Image Based on Deep Neural Networks.” Proceedings of the 2016 2nd International Conference on Artificial Intelligence and Industrial Engineering 133 (1): 262–267. doi:10.2991/aiie-16.2016.61.

 Google Scholar

Geng, J., H. Wang, J. Fan, and X. Ma. 2017. “Change Detection of SAR Images Based on Supervised Contractive Autoencoders and Fuzzy Clustering.” In 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP), 1–3. doi:10.1109/RSIP.2017.7958819.

 Google Scholar

Zhan, T., M. Gong, J. Liu, and P. Zhang. 2018. “Iterative Feature Mapping Network for Detecting Multiple Changes in Multi-Source Remote Sensing Images.” ISPRS Journal of Photogrammetry and Remote Sensing 146: 38–51. doi:10.1016/j.isprsjprs.2018.09.002.

 Web of Science ®Google Scholar

Su, L., M. Gong, P. Zhang, M. Zhang, J. Liu, and H. Yang. 2017. “Deep Learning and Mapping Based Ternary Change Detection for Information Unbalanced Images.” Pattern Recognition 66: 213–228. doi:10.1016/j.patcog.2017.01.002.

 Web of Science ®Google Scholar

Song, A., J. Choi, Y. Han, and Y. Kim. 2018. “Change Detection in Hyperspectral Images Using Recurrent 3D Fully Convolutional Networks.” Remote Sensing 10 (11): 1827. doi:10.3390/rs10111827.

 Web of Science ®Google Scholar

Zhan, Y., K. Fu, M. Yan, X. Sun, H. Wang, and X. Qiu. 2017. “Change Detection Based on Deep Siamese Convolutional Network for Optical Aerial Images.” IEEE Geoscience and Remote Sensing Letters 14 (10): 1845–1849. doi:10.1109/LGRS.2017.2738149.

 Web of Science ®Google Scholar

Wang, Q., X. Zhang, G. Chen, F. Dai, Y. Gong, and K. Zhu. 2018a. “Change Detection Based on Faster R-CNN for High-Resolution Remote Sensing Images.” Remote Sensing Letters 9 (10): 923–932. doi:10.1080/2150704x.2018.1492172.

 Web of Science ®Google Scholar

Zhang, P., M. Gong, L. Su, J. Liu, and Z. Li. 2016b. “Change Detection Based on Deep Feature Representation and Mapping Transformation for Multi-Spatial-Resolution Remote Sensing Images.” ISPRS Journal of Photogrammetry and Remote Sensing 116: 24–41. doi:10.1016/j.isprsjprs.2016.02.013.

 Web of Science ®Google Scholar

Gong, M., J. Zhao, J. Liu, Q. Miao, and L. Jiao. 2015. “Change Detection in Synthetic Aperture Radar Images Based on Deep Neural Networks.” IEEE Transactions on Neural Networks and Learning Systems 27 (1): 125–138. doi:10.1109/TNNLS.2015.2435783.

 PubMed Web of Science ®Google Scholar

Connors, C., and R.R. Vatsavai. 2017. “Semi-Supervised Deep Generative Models for Change Detection in Very High Resolution Imagery.” In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 1063–1066. doi:10.1109/IGARSS.2017.8127139.

 Google Scholar

Gong, M., Y. Yang, T. Zhan, X. Niu, and S. Li. 2019b. “A Generative Discriminatory Classified Network for Change Detection in Multispectral Imagery.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (1): 321–333. doi:10.1109/JSTARS.2018.2887108.

 Web of Science ®Google Scholar

Khan, S.H., X. He, F. Porikli, and M. Bennamoun. 2017. “Forest Change Detection in Incomplete Satellite Images with Deep Neural Networks.” IEEE Transactions on Geoscience and Remote Sensing 55 (9): 5407–5423. doi:10.1109/TGRS.2017.2707528.

 Web of Science ®Google Scholar

Su, L., J. Shi, P. Zhang, Z. Wang, and M. Gong. 2016. “Detecting Multiple Changes from Multi-Temporal Images by Using Stacked Denosing Autoencoder Based Change Vector Analysis.” In 2016 International Joint Conference on Neural Networks (IJCNN), 1269–1276. doi:10.1109/IJCNN.2016.7727343.

 Google Scholar

Liu, J., M. Gong, K. Qin, and P. Zhang. 2016a. “A Deep Convolutional Coupling Network for Change Detection Based on Heterogeneous Optical and Radar Images.” IEEE Transactions on Neural Networks and Learning Systems 29 (3): 545–559. doi:10.1109/TNNLS.2016.2636227.

 PubMed Web of Science ®Google Scholar

Data availability statement (DAS)

The data that support the findings of this study are available from the corresponding author [L. Wang], upon reasonable request.