Mapping Rice Paddies in Complex Landscapes with Convolutional Neural Networks and Phenological Metrics

ABSTRACT

Rice mapping with remote sensing imagery provides an alternative means for estimating crop-yield and performing land management due to the large geographical coverage and low cost of remotely sensed data. Rice mapping in Southern China, however, is very difficult as rice paddies are patchy and fragmented, reflecting the undulating and varied topography. In addition, abandoned lands widely exist in Southern China due to rapid urbanization. Abandoned lands are easily confused with paddy fields, thereby degrading the classification accuracy of rice paddies in such complex landscape regions. To address this problem, the present study proposes an innovative method for rice mapping through combining a convolutional neural network (CNN) model and a decision tree (DT) method with phenological metrics. First, a pre-trained LeNet-5 Model using the UC Merced Dataset was developed to classify the cropland class from other land cover types, i.e. built-up, rivers, forests. Then, paddy rice field was separated from abandoned land in the cropland class using a DT model with phenological metrics derived from the time-series data of the normalized difference vegetation index (NDVI). The accuracy of the proposed classification methods was compared with three other classification techniques, namely, back propagation neural network (BPNN), original CNN, pre-trained CNN applied to HJ-1 A/B charge-coupled device (CCD) images of Zhuzhou City, Hunan Province, China. Results suggest that the proposed method achieved an overall accuracy of 93.56%, much higher than those of other methods. This indicates that the proposed method can efficiently accommodate the challenges of rice mapping in regions with complex landscapes.

KEYWORDS:

• Rice mapping
• Image classification
• deep learning
• CNNs
• phenological metrics

1. Introduction

Rice is a staple crop that feeds the majority of the world’s population (Xiao et al. 2006). Mapping rice paddies with a timely and efficient means is essential for maintaining agricultural and environmental sustainability, as well as food and water security. Remote sensing technology provides a viable approach for large-scale rice paddy mapping. Many studies using different kinds of optical remote sensing data (eg.MODIS,AVHRR,SPOT VEGETATION) have demonstrated advantages in rice monitoring at regional and global scales (Onojeghuo et al. 2018; Zhang et al. 2017; McCloy et al. 1987). Although with some success, rice mapping remains challenging in fragmented landscapes, such as the rice-planting areas in southern China (Xie et al. 2008; Wang et al. 2015; Du et al. 2019). The rice-planting areas in this region are relatively small, irregular, and fragmented because of the undulating terrain, and generally mixed with other vegetated land. The main challenges are the undulating terrain restrictions and vegetation coverage in areas with complex landscapes, particularly when fragmented paddy fields are mixed with other vegetation types (Hansen et al. 2012; Phiri and Morgenroth 2017; Xie, Sha, and Yu 2008). The difficulty of paddy rice mapping is compounded by the similar spectral characteristics of paddy rice and other land vegetated covers. (Feng et al. 2015; Qiu et al. 2015; Chen et al. 2016; Son et al. 2013). Furthermore, the accelerated urbanization in China over the past decades has left a large number of deserted farmlands, which have turned into abandoned lands. The existence of these land types leads to the difficulties of rice mapping by remote sensing data due to their spatial proximity and spectral similarity to those of rice paddies. It is our interest to explore a suitable extraction process for precious rice mapping in complex landscapes, where the paddy rice area is easily confused with other vegetated covers as mentioned above. However, with the spatial, spectral or temporal features alone, the above methods generally preclude an accurate classification of rice patches in complex landscapes. The features of paddy fields are often confused with those of other vegetated covers.

Existing land cover classification technologies have exploited different classes of features, such as temporal, spectral, and spatial features extracted from remote sensing imagery (Gudex-Cross, Pontius, and Adams 2017; Qiu et al. 2015, 2017 Gumma et al. 2011; Kumhalova and Matejkova 2017; Qin et al. 2015). Phenological metrics, which can be derived from temporal vegetation indices, namely, the NDVI, the enhanced vegetation index (EVI), and the land surface water index (LSWI), have also been employed for vegetation classification. With these methods, land types are distinguished by setting a threshold for each index (Pena-Barragan et al. 2011; Kim et al. 2017; Gumma et al. 2018). Further, a number of machine learning classification methods, such as the artificial neural networks (ANNs), back propagation neural networks (BPNNs) decision trees (DT), support vector machine (SVM), and convolutional neural networks (CNNs), have been proposed to take advantage of the spectral and spatial features of land covers (Li, Yang, and Wang (2016); XU et al. (2005); Liu et al. (2013)). ANN is a multi-layer perceptron system inspired by human neurons, and it enables the distributed information storage and parallel co-processing. The DT approach reaches a decision by following the rules at each node. For rice mapping, DT has been adopted for its simplicity, interpretability and fast computation (Kontgis, Schneider, and Ozdogan 2015). Comparatively, SVM can separate different features by hyperplanes constructed in a high- or infinite-dimensional space. However, shallow neural network methods (such as ANNs, and BPNNs) and statistical methods (such as DTs, and SVMs) cannot fully extract the spectral and spatial features (Kontgis, Schneider, and Ozdogan 2015; Wang et al. 2016). ANNs, are with shallow neural networks that only extract low-level features representing the highly specific components of the image. BPNN is a multi-layer neural network, which can be trained with a training dataset in which output is compared with desired output and error is propagated back to input until the minimal MSE is achieved. (Rumelhart, Hinton, and Williams 1986).In contrast, the deeper layers of deep learning methods fuse various low-level features into higher-level features to represent more complicated spectral and spatial components. Therefore, deep learning improves the classification accuracy by acquiring high-level features. (Dong et al. 2016; Yang et al. 2017; Zhang et al. 2015). As one of the most widely used deep neural networks in image classification, CNN performs well on simple feature datasets such as the MNIST (Modified National Institute of Standards and Technology) database, and also on complicated feature datasets such as ImageNet (a visual database designed for research in visual object recognition software), ISPRS Benchmarks (a dataset consisting of state-of-the-art sensor data and covering different relevant tasks), and WHU-RS19(a dataset consisting of 19 scene classes of high-resolution remote sensing imagery)(Leutheuser, Schuldhaus, and Eskofier 2013; Liao and Carneiro 2017; Nogueira, Penatti, and Santos 2016). Most CNN techniques are performed on very high resolution remote sensing images to extract spatial features. However, spatial features alone cannot effectively solve the problem of rice mapping in complex landscapes, and the high resolution image cannot provide effective phenological information (Su 2017; Liao and Carneiro 2017; Nogueira, Penatti, and Santos 2016; Yue et al. 2015; Zhang et al. 2017; Boschetti 2013; Kim and Yeom 2015; Rottensteiner et al. 2012; Xia et al. 2017; Yang and Newsam 2010). Further, phenology-based-only methods struggle to identify reliable and unique annual reference profiles for the features of different land coves, especially in complex regions with variable topography, climate, landscape composition and configuration (Bargiel 2017; Chockalingam and Mondal et al. 2017). In such regions with non-uniform phenology, an appropriate model of rice mapping remains a major challenge.

The method proposed in this paper combines deep learning with phenological metrics for rice mapping in regions with complex landscapes. The combined deep learning and phenological metrics were proposed to extract the spectral/spatial and temporal features of the paddy fields, respectively. Classification results of the proposed method are compared against those of three standard methods (BPNN, original CNN and pre-trained CNN). Using the proposed method, researchers and agricultural managers are expected to achieve accurate rice mapping in complex landscape regions, and agronomic planners can facilitate their crop management.

2. Study area and datasets

2.1 Study area

The study area is located in Zhuzhou city, Hunan province, China (Figure 1). The hilly terrain of Zhuzhou City typifies a complex landscape. The cropland is mainly distributed along both sides of the Xiangjiang River, which flows across the city from south to north. Zhuzhou is characterized by high-volume food production, primarily from cultivated paddies. According to field survey data, the paddy rice in Zhuzhou is grown annually from early June to mid-September, a period of approximately 110 days. The distribution of rice planting in Zhuzhou typifies the patchy, fragmented crop distribution of hilly regions in South China. The four major land cover categories are built-up, water, forest and cropland, where the cropland includes abandoned land and paddy rice. Because of the topography restriction and mixed landscapes, these land cover categories are difficult to distinguish.

Figure 1. Location of the study area(112°17′-114°07′E, 26°03′-28°01′N).

2.2 Data

2.2.1 Experimental data

The remote sensing data for classification were HJ-1 A/B charge-coupled device (CCD) images acquired by the Chinese Environment and Disaster Reduction Small Satellites. These data were downloaded from the China Center for Resources Satellite Data and Application (http://www.cresda.com/CN/). With their mid–high spatial resolution (30 m) and high temporal resolution (two days), CCD images are advantageous for the present study. Each CCD image consists of four channels: band 1: 0.43–0.52 μm (blue), band 2: 0.52–0.60 μm (green), band 3: 0.63–0.69 μm (red) and band 4: 0.76–0.90 μm (near-infrared). Sixteen CCD images with satisfactory data quality and within the crop growing period in 2013 were selected for classification. The acquisition times of these images are shown in Table 1. All images were pre-processed sequentially by radiometric calibration, atmospheric correction and geometric correction.

Table 1. Acquisition times of the CCD images.

Number	Date	DOY1	Number	Date	DOY1
1 2 3 4 5 6 7 8	13/Jun 17/Jun 30/Jun 08/Jul 12/Jul 25/Jul 29/Jul 07/Aug	163 167 180 188 192 205 209 218	9 10 11 12 13 14 15 16	09/Aug 11/Aug 13/Aug 25/Aug 09/Sep 15/Sep 19/Sep 21/Sep	220 222 224 236 251 261 265 267

¹Note: DOY, day of year

2.2.2 Validation data

The validation data were obtained from GF-1, high-resolution Google Earth images and field survey. The GF-1 image is the first image of the Chinese High Resolution Earth Observation System. It has a high temporal resolution (four days) and a high spatial resolution (two 2-m panchromatic/8-m multispectral cameras and four 16-m wide-field imagers). The GF-1 image consists of four channels: band 1: 0.45–0.52 μm (blue), band 2: 0.52–0.59 μm (green), band 3: 0.63–0.69 μm (red) and band 4: 0.77–0.89 μm (near-infrared). Moreover, multi-resolution (up to 0.5 m) satellite images were provided by Google Earth. Fifty validation sites (each covering several pixels and are obtained by field survey) with good visual interpretation on GF-1, Google Earth images and the field survey were selected for validation. The reference sites are indicated with yellow triangles in Figure 1.

3. Methods

The proposed classification method includes two major steps: (1) preliminary classification of cropland, forest, buildup and water by the pre-trained CNN method; (2) further classification of paddy rice and abandoned land by the DT method (see Figure 2). The classification results of proposed method were evaluated and compared with BPNN, original CNN, and pre-trained CNN methods.

Figure 2. Flowchart of the proposed methodology.

3.1 Preliminary classification by CNN

3.1.1 Classical CNN architecture

A CNN consists of several layers with different functions: an input layer, a convolutional layer, a pool layer, a full connection layer and an output layer (Lecun et al. 1998; Tajbakhsh and Suzuki 2016). The training data are imported using the input layer, and their features are extracted with the convolutional layer. More convolution layers will add increasingly deeper features. The pool layer reduces the complexity of the whole network, and extracts the key features. The full connection layer connects all the features of the pool layer, and sends the output values for further handling. The output layer classifies the data into different categories.

Many available neural network structures include LeNet-5, AlexNet and Vgg-16 Net (Bui et al. 2011). The proper CNN architecture, rather than the CNN depth, is related to the classified image. Considering the factors of image resolution and classification object size, the classical network, LeNet-5, is selected by cross-validation. LeNet-5, a classical CNN proposed by LeCun in (1998), successfully recognizes isolated objects, and its classification is more abstract and robust than those of other common methods (Lecun et al. 1998; Tong, Li, and Zhu 2017). Considering the spatial resolution of the classified imagery and the sizes of the classified objects, we chose a LeNet-5 model for the current classification problems. The details of the LeNet-5 model are covered in Lecun et al. (1998).

3.1.2 Pre-training the LeNet-5 model using the UC merced dataset

Deep learning is advantageous because it automatically learns the efficient knowledge of the classified objects, and builds a complicated model in more difficult classification situations. However, complex models require a large amount of labeled data for training, because the depth of the structure is positively correlated with the amount of training data. If the labeled data are insufficient to support the whole training process, the deep learning model returns an unsatisfactory result. Therefore, the correct amount of training data guarantees the rationality and reliability of the training model. Traditionally, the training data are obtained through visual interpretation, which is limited and time-consuming. When the number of labeled samples is limited, the trained parameters of the CNN cannot obtain the optimal values. The data shortage problems can be solved by the so-called transfer learning method, which also reduces the training cost and avoids the overfitting problem.

Two primary transfer learning strategies are presently available for pattern recognition (including the classification of remote sensing images) (Pan and Yang 2010; Nogueira, Penatti, and Santos 2016; Pena-Barragan et al. 2011; Qian et al. 2017). The first strategy is the pre-trained model, which extracts features from small-size datasets containing high-similarity data. The CNN model trains a large dataset, and saves the optimal weight parameters for fine-tuning on the small dataset. The second strategy is the fine-tuned model, which has the same architecture as the pre-trained model. Starting from randomly initialized weight parameters, this model is retrained on the new dataset. This model is suitable for small-sized datasets containing low-similarity data (Nogueira, Penatti, and Santos 2016). The present study must analyze a small dataset of remote sensing imagery with high similarity to the pre-trained dataset. Thus, the first transfer learning strategy, based on the pre-trained model, is suitable for our purpose.

To obtain suitable weight parameters in the pre-trained dataset, a large dataset is usually required. Several remote sensing datasets, such as the UC Merced land use dataset and the NWPU-RESISC45 dataset, are available. The UC Merced dataset, downloadable from UC Merced http://vision.ucmerced.edu/, is widely used for model pre-training. The images in the UC Merced dataset have been manually extracted from large images in the USGS National Map Urban Area Imagery Collection, which maps various urban areas around the country. As the UC Merced data are similar to the classification images in the present study, they were applied as the pre-trained dataset (Avramović and Risojević 2016). The pixel resolution of this public domain imagery is one foot (0.3048 m). The dataset contains 21 classes each with 100 images. Examples of the four land classes adopted in the present study are shown in the left panel of Figure 3. The original images (RGB bands consisting of (256$\times$ 256) pixels) were resampled to (28$\times$28) pixels. The data pre-processing and pre-trained model structure is illustrated in the center and right panels of Figure 3, respectively.

Figure 3. Original images, data pre-processing and feature extraction by LeNet-5.

To circumvent the large amount of training data required for deep learning, the transfer learning mechanism sets the model parameters using a large dataset with high similarity to the analyzed data, then fine-tunes the parameters on the specific dataset, which was established manually by selecting the images from the image to be classified. Given the high correlativity between the model parameters and classification accuracy, the model parameters must be carefully selected to ensure high-quality final results. However, there are no widely-applied weight-parameter selection methods for the various classification situations. Referring to previous studies, we selected four model parameters for testing and discussion: kernel size, learning rate, batch size and activation function.

3.2 Further classification coupled with phenological metrics

3.2.1 Smoothing of the NDVI time-series data

The NDVI is correlated with the photosynthetic capacity of the vegetation, and is calculated as follows (Tucker 1979).

(1) $\mathrm{N}\mathrm{D}\mathrm{V}\mathrm{I}=\frac{\left({\rho }_{\mathrm{N}\mathrm{I}\mathrm{R}}-{\rho }_{\mathrm{R}\mathrm{E}\mathrm{D}}\right)}{\left({\rho }_{\mathrm{N}\mathrm{I}\mathrm{R}}+{\rho }_{\mathrm{R}\mathrm{E}\mathrm{D}}\right)}$(1)

where ${\rho }_{\mathrm{N}\mathrm{I}\mathrm{R}}$ and ${\rho }_{\mathrm{R}\mathrm{E}\mathrm{D}}$ are the spectral reflectance of the near-infrared and red bands, respectively. Due to cloud and atmospheric contamination, NDVI time series are generally discontinuous and noisy. Therefore, prior to the phenology extraction, the noise must be removed from the time series by a filter method (Bradley et al. 2007).

Here, the NDVI time series was smoothed by the S–G filter, named after its proposers (Chen 2004; Savitzky et al. 1964). The S–G filter, which is widely used in time series reconstruction, smooths the time series by least-squares-fitting convolution and calculates groups of adjacent values. Figure 4(a) shows an S–G filtered NDVI time-series extracted from the raw NDVI data, along with the phenological metrics.

Figure 4. Acquiring the phenological metrics of a paddy rice field.

3.2.2 Rice classification using the phenological metrics

As the phenological metrics, we selected the Start of Growing Season (SOS), End of Growing Season (EOS) and Length of Growing Season (LOS; see Figure 4). The SOS and EOS were determined as 20% of the vegetation growth and decay amplitudes, respectively, between the maximum and minimum NDVI. They were measured from the left and right minimum levels, respectively. Meanwhile, the LOS defines the time between the start and end of the growing season. These three phenological metrics can distinguish paddy rice from abandoned lands, as shown in Figure 4. They can also extract the paddy field from cropland; the latter is then classified as abandoned land. To meet this need, the classification results of the CNN method are further classified using the DT method programmed in the classification algorithms.

The DT in this paper is structured as shown in (Figure 4(b-c)). The decision nodes were determined using field survey (Liu et al. 2017). As traditional agriculture operations are inconsistent, the phenological metrics were obtained as ranges. The dates of the SOS, EOS and LOS were determined as after DOY(day of the year)$140,$ before DOY 270 and DOY 90–125, respectively.

4. Results and analysis

4.1 Determination of optimal pre-trained CNN model parameters

Figure 5(a) plots the accuracy of the models with different kernel sizes (3, 4 and 5). The accuracy was almost independent of kernel size, although the accuracy slightly improved with largest kernel size ranges between the 20th and $200\mathrm{t}\mathrm{h}$ epoch. After Epoch 200, the result of the (4 × 4) kernel gradually converged to that of the (5 × 5) kernel. To ensure high accuracy and the inclusion of the fine features during training, and considering the experimental data size, we selected the (4$\times$4) kernel for the CNN.

Figure 5. Optimizing the model parameters. Plotted are the model accuracy versus time characteristics of the CNN for different (a) kernel sizes, (b) learning rates, (c) batch sizes and (d) activity functions.

The next parameter is the learning rate (LR), which defines the learning efficiency of the CNN model. An inappropriately fast LR causes fluctuation decline in the training curve, and an mis-specified slow LR can trap the algorithm in local optima. The accuracy at different LRs are plotted in Figure 5(b) (the inset is a zoom-in of the region framed by the red dotted rectangle in the main image). After a given number of epochs, the accuracy differences among the LRs were quite small. With a sufficiently large value of epochs, the differences gradually diminished, and eventually vanished. Thus, to ensure an adequate training speed while avoiding local optima, we selected a moderate learning rate (LR = 0.2).

The batch size refers to the number of samples selected at one training time. When selecting the batch size, one usually considers the size of the dataset and the required training accuracy. As shown in Figure 5(c), reducing the batch size increases the debugging speed. Considering the size of the dataset, we set the batch size to 50 rather than 100, although other studies have shown that a large batch size reduces the vibration over a certain range of the training curve, and optimizes the convergence accuracy (Shin et al. 2016; Kovács et al. 2017).

Finally, the activation function of a node defines the output of that node for a given input or set of inputs to the computational network. An appropriately chosen activation function is critical for obtaining good training results. Owing to the simple network structure in this study, the rectified linear unit (ReLU) activation function was not remarkably superior to the sigmoid activation function (see Figure 5(d)). However, considering the short operation time, high efficiency and maintenance of the gradient characteristics, we selected the ReLU activation function. The effects of the different parameter settings on the final results are more intuitively shown in Table 2.

Table 2. Accuracy assessment of the CNN model parameters.

	Kernel Size			Learning Rate				Batch Size		Activation Function
Parameter	3	4	5	0.05	0.1	0.2	0.5	50	100	ReLU	Sig^1
Accuracy	96%	98%	97%	98%	99%	99%	99%	98%	96%	99%	98%

¹Note: sig, sig is short for sigmoid

4.2 Performance of the preliminary classification with CNN

To improve the analysis, the classification results of the original untrained and the pre-trained CNN models were compared with those of the classical BPNN method, a classical shallow ANN that has long been used in remote-sensing image classification.

BPNN is composed of three layers: input layer, output layer and hidden layer.The classical BP network model was adopted, and the parameter setting was selected according to the experimental results. In order to reflect the difference in application accuracy of the method more objectively, the training accuracy of the three models is the same 99% before the test sample verification.The classification results of the four land cover types by the three methods are shown in (Figure 6(a-c)). Additionally, the classification accuracy and their confidence intervals are presented in Table 3. These results were calculated using the accuracy assessment method proposed by Olofsson et al, (2013, 2014). This method summarized the accuracy assessment by reporting the estimated error matrix in terms of proportion of area and estimates of accuracy (Olofsson et al. 2013, 2014).

Table 3. Classification accuracy of (a) BPNN, (b) Original CNN, and (c) the pre-trained CNN.

	BPNN			Original CNN			Pre-trained CNN
Class	UA^1	PA^1	$\pm$2S^1	UA^1	PA^1	$\pm$2S^1	UA^1	PA^1	$\pm$2S^1
Cropland	0.53	0.11	0.01	0.43	0.33	0.02	0.82	0.62	0.01
Forest	0.50	0.88	0.04	0.58	0.77	0.04	0.89	0.97	0.03
Built-up	0.49	0.86	0.01	0.47	0.11	0.00	1.00	1.00	0.00
Water	0.96	0.83	0.00	0.86	0.96	0.00	1.00	0.89	0.00
OA	52.90%		0.02	55.15%		0.02	89.86%		0.02

¹Note: UA is short for user’s accuracy; PA is short for producer’s accuracy; OA is overall accuracy; $\pm$2S is the 95% confidence interval for accuracy; S is the estimated standard error of the estimated area proportion.

Figure 6. Classification results in Area A: (a) BPNN, (b) original CNN, (c) the pre-trained CNN and (d) the proposed coupling method. Classification results in Area B: (e) the pre-trained CNN and (f) the proposed coupling method. PRF: paddy rice field, AL: abandoned land.

As confirmed in Figure 6(a-c), the proposed method achieved much higher classification accuracy than the two existing methods, especially in the cropland and built-up categories. As the BPNN is a shallow network, the number of network parameters in this method is relatively small, so the detailed characteristics are weakly evaluated. Meanwhile, the small training dataset in the original CNN model cannot fully train the model. Consequently, the weight parameters are incompletely trained and the accuracy requirements are not met. Owing to their poor performances, the BPNN and original CNN methods are not further considered in this manuscript. Further classification will be carried out on the results of fine-tuned CNN model and accuracy evaluation will be achieved.

4.3 Performance of further classification method coupled with phenological metrics

Figure 6(d) shows the classification results after coupling the pre-trained CNN model with the phenological metrics. The overall accuracy of the proposed coupling method was 93.56%. The five categories – paddy fields, forest, built-up, water and abandoned land – are clearly exhibited in Table 4, and the abandoned land is separated from the cropland as a new category. Forest, built-up, and water were classified more accurately than paddy fields and abandoned land. The abandoned land mainly distributes in the west of the Xiangjiang River, and the patches are highly fragmented. However, in the east of the Xiangjiang River, abandoned lands are sparse and scattered, and paddy-field patches are large and concentrated.

Table 4. Classification accuracy of the proposed coupling method.

Class	Water	Built-up	Forest	PRF	AL	Map^1	$W_i$^1	UA	PA	$\pm$2S^1
Water	1383	0	0	0	0	12,270	0.05	1.00	0.72	0.00
Built-up	0	737	0	0	0	30,407	0.13	1.00	1.00	0.00
Forest	12	0	550	22	0	161,455	0.68	0.94	0.99	0.02
PRF	28	0	24	438	59	21,300	0.09	0.80	0.64	0.00
AL	11	0	0	98	301	13,758	0.06	0.73	0.82	0.00
OA	93.56%

¹Note: PRF is short for paddy rice field, AL is short for abandoned land, map is the estimated area of each class, $W_i$ is the portion of the area mapped as category i,UA is short for user’s accuracy; PA is short for producer’s accuracy; OA is overall accuracy; $\pm$2S is the 95% confidence interval for accuracy; S is the estimated standard error of the estimated area proportion.

To further verify the effectiveness of the proposed method, we investigated a large research area. The classification results of the pre-trained CNN and the proposed method are shown in panels (e) and (f) of Figure 6, respectively. The conclusion of this supplementary experiment is consistent with that of the previous experiment. The overall accuracy of the proposed coupling method was 93.56%. Abandoned lands are mainly distributed in the west of the Xiangjiang River, as observed in area A of the previous classification, but the patches are lowly fragmented and most of the abandoned lands are located near built-up areas. In the east of the Xiangjiang River, abandoned lands are scattered and sparse, whereas the paddy-field patches are large and concentrated. Most importantly, unlike the other tested methods (the BPNN, original CNN and pre-trained CNN), the proposed method effectively separated the abandoned land from the cropland, and realized accurate rice mapping.

5. Discussion

This study proposed an efficient method for rice mapping in complex landscape regions. Deep learning and phenological metrics were combined to extract the spectral/spatial features and the temporal features of paddy fields, respectively. Coupling the deep learning and phenological metrics can effectively improve the classification accuracy of remote sensing images, especially in rice mapping and abandoned land identification.

Mapping crop area using remote sensing imagery is essential for agricultural production. However Rice is usually mapped by extracting the time-series features from MODIS data. The spatial resolution of MODIS is suitable only for analyzing large rice-growing areas, and is inadequate for analyzing rice-growing areas in complex landscapes. The present study was completed on mid-range spatial resolution data, which captured the finer features that are omitted in low-resolution data, but at lower image-acquisition cost and over a larger coverage. The mid-spatial resolution classification did not adversely compromise the precision (which exceeded 93.56%). In addition, as the rice paddies were obtained from the cropland using the time features, we also indirectly obtained the distribution of abandoned land using this method. Increasing the amount of abandoned lands may reduce the total grain production, thereby exacerbating the food crisis. By understanding the distribution and evolution of the abandoned lands, agricultural policy makers can effectively formulate their solutions.

Our method resulted in a precise paddy rice map, but several limitations remain. First, the selected CNN structure depends on the situation of the study area and the resolution of the remote sensing images. The present CNN model is optimized by adjusting the activation function and parameter settings. The method is suitable for rice mapping in complex landscape regions such as Zhuzhou, where the rice distribution is fragmented. Therefore, the CNN structure and optimal parameters require adjustment to the situation. Secondly, the accuracy of phenological metrics depends on the weather conditions. The traditional agriculture mechanism desynchronizes the phenological phases of the crops, and judgment based on empirical knowledge is inherently uncertain. Thirdly, Because of the image size difference between the pre-trained dataset (UC Merced Land Use Dataset) and the fine-tuned dataset, the pre-trained data are compressed from a dimension of 256 by 256 to 28 by 28, resulting in a certain loss of information. Due to the limitation of training data size, we pay more attention to the advantages of CNN in mining the features of image topology, and texture, the loss of color information in compression was ignored. Finally, the classification process cannot extract and use the deep characteristics of the time-series data images. To excavate the deeper information, the method must simultaneously learn the features of the time-series characteristics and the spectral/spatial characteristics.

In further work, the spectral, spatial and temporal feature extractions should be integrated into a single model. Moreover, we should also employ images with finer spectral resolution, and increase both the depth of the convolutional network and the identification precision of the mixed pixels.

6. Conclusions

We proposed an innovative method to accurately map paddy fields in complex landscapes with time-series remote sensing imagery. The method integrates a deep learning method with phenological metrics. In a series of experiments, the proposed method outperformed other state-of-the-art classification methods, achieving an overall mapping accuracy of 93.56%$\pm$ 0.01%. The proposed method demonstrated its effectiveness in paddy-mapping under complicated landscape conditions, and correctly classified abandoned lands. The method is expected to assist researchers and agricultural managers of rice mapping in complex landscape regions, and it can also provide useful information (e.g. crop yield estimates, land management decisions, land uses and cover changes) for agronomic planners.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the National Natural Science Foundation of China Youth Science Foundation Project [Grant No. 41701387].