LuoJiaAI: A cloud-based artificial intelligence platform for remote sensing image interpretation

ABSTRACT

The rapid processing, analysis, and mining of remote-sensing big data based on intelligent interpretation technology using remote-sensing cloud computing platforms (RS-CCPs) have recently become a new trend. The existing RS-CCPs mainly focus on developing and optimizing high-performance data storage and intelligent computing for common visual representation, which ignores remote sensing data characteristics such as large image size, large-scale change, multiple data channels, and geographic knowledge embedding, thus impairing computational efficiency and accuracy. We construct a LuoJiaAI platform composed of a standard large-scale sample database (LuoJiaSET) and a dedicated deep learning framework (LuoJiaNET) to achieve state-of-the-art performance on five typical remote sensing interpretation tasks, including scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction. The details of the LuoJiaAI application experiment can be found at the white paper for LuoJiaAI industrial application. In addition, LuoJiaAI is an open-source RS-CCP that supports the latest Open Geospatial Consortium (OGC) standards for better developing and sharing Earth Artificial Intelligence (AI) algorithms and products on benchmark datasets. LuoJiaAI narrows the gap between the sample database and deep learning frameworks through a user-friendly data-framework collaboration mechanism, showing great potential in high-precision remote sensing mapping applications.

KEYWORDS:

• Artificial intelligence
• cloud computing platform
• remote-sensing intelligent interpretation
• sample database
• deep learning framework

1. Introduction

The rapid development of remote sensing technology has obtained a large volume of multi-source (multi-type, multi-temporal, and multi-scale) Earth observation data (Deren et al. 2017; Gong 2018). With the continuous enrichment of data resources, traditional desktop-based remote sensing platforms can no longer satisfy the higher requirements for dense storage, rapid processing, and analysis of remote sensing big data (Yang et al. 2017; Kumar and Mutanga 2018; Mutanga and Kumar 2019; Wang et al. 2020). In recent years, large-scale cloud computing systems have provided unprecedented opportunities for geospatial information mining. A wide variety of remote sensing cloud computing platforms (RS-CCPs) have been developed to facilitate massive data processing, including Google Earth Engine (Gorelick et al. 2017), NASA Earth Exchange (Nemani 2011), Descartes Labs (https://descarteslabs.com/), Geoscience Data Cube (Lewis et al. 2017), Planetary Computer (https://planetarycomputer.microsoft.com/), the CASEarth EarthDataMiner (Liu, Wang, and Zhong 2020), and PIE-Engine (https://www.piesat.cn/en/PIE-Engine.html). An increasing number of commercial companies, including Amazon (https://aws.amazon.com/earth/), Alibaba (https://engine-aiearth.aliyun.com/), and SenseTime (https://rs.sensetime.com/), are actively constructing RS-CCP to facilitate large-scale geospatial data processing. These cloud-based platforms provide effective cloud computing services for large-scale geospatial analysis that have been widely employed in water management, crop production forecasting, climate monitoring, and environmental protection (Hansen et al. 2013; Pekel et al. 2016; Badgley, Field, and Berry 2017; Kontgis et al. 2017; Xiong et al. 2017; MacDonald and Mordecai 2019; Ceccherini et al. 2020; Gao et al. 2020; Nemani et al. 2015). Figure 1 shows the timeline of existing popular RS-CCPs.

Figure 1. Developmental line of existing popular RS-CCPs.

With the development of technologies, including big data, cloud computing, and Artificial Intelligence (AI), Deep Learning (DL) has made remarkable progress in numerous fields, including image recognition, drug discovery, natural language processing, and self-driving cars, since the ImageNet Challenge in 2012 (LeCun, Bengio, and Hinton 2015; Russakovsky et al. 2015; Rao and Frtunikj 2018; Chen et al. 2018; Young et al. 2018). In the remote sensing field, researchers integrate DL with remote sensing interpretation technology and apply them in large-scale Earth observation data, which has achieved remarkable results in various image interpretation tasks, such as scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction (Zhu et al. 2017; Sun et al. 2022; Yuan et al. 2020; Li, Huang, and Gong 2019; Ma et al. 2019). Current DL frameworks, such as PyTorch (Paszke et al. 2019), TensorFlow (Abadi et al. 2016), PaddlePaddle (Ma et al. 2019), and MindSpore (Huawei Technologies Co. 2022), are employed in VHR image interpretation applications for AI models designing, training, and deploying through a high-level programming interface. Some RS-CCPs, such as Google Earth Engine and PIE-Engine, have involved these DL frameworks in increasing the intelligence level for processing and analyzing remote sensing big data.

The key components of RS-CCPs are available databases, Application Programming Interfaces (APIs) provided by computational frameworks, and computational infrastructures. Although many academic institutions and corporations have developed RS-CCPs and promoted numerous critical achievements in Earth AI Sciences, there is a lack of a standard large-scale sample database for image intelligent interpretation tasks that allows users to efficiently retrieve, access, label, and share data. Besides, the computational frameworks used in existing RS-CCPs mainly focus on developing and optimizing high-performance data storage and computing in the perspective of common visual representation, which ignores remote sensing data characteristics such as large image size, large-scale change, multiple data channels, and geographic knowledge embedding, thus impairing the computational efficiency and accuracy.

To solve the above limitations in existing RS-CCPs, we construct a LuoJiaAI platform that brings Huawei’s massive data storage and computational capabilities for intelligent interpretation and analysis of remote sensing data to achieve state-of-the-art performance on various image interpretation tasks. In summary, we make the following contributions:

As the first contribution of LuoJiaAI, LuoJiaSET builds a large-scale sample database with a unified and scalable classification scheme, an Internet-based crowdsourcing sample collection tool with human–machine interaction, and a sample platform (Homepage, http://58.48.42.237/luojiaSet) for better data preparation and sharing in remote sensing intelligent interpretation.

The second contribution of LuoJiaAI is designing and implementing a dedicated DL framework called LuoJiaNET with remote sensing data characteristics embedded into the underlying system to improve computational efficiency and performance accuracy (Source code, https://github.com/WHULuoJiaTeam/luojianet). LuoJiaAI also develops a front-end application platform (Homepage, http://58.48.42.237/luojiaNet) with a visual modeling tool that provides online creating, training, and deploying models for improving users’ productivity. In addition, there is a growing trend for Earth AI Sciences to create libraries of standardized and benchmark datasets. These benchmark datasets can facilitate users to evaluate developed models on a common and standardized dataset efficiently. LuoJiaAI supports the latest Open Geospatial Consortium (OGC) standards in geographic information and service interfaces (https://www.ogc.org/projects/groups/trainingdmlswg) for better data sharing and model developing (Yue et al. 2022).

LuoJiaAI accelerates the integration of AI algorithms and RS-CCP, and promotes the sharing and development of standardized data, algorithms, and products, showing great potential in rapid intelligent processing and analysis of remote sensing big data.

The rest of this paper is organized as follows. Section 2 reviews related works on RS-CCPs, sample databases, and deep learning models and frameworks. Section 3 illustrates the platform overview briefly. Section 4 first demonstrates how to organize and build a standard large-scale sample database, Internet-based sample collection, and sample sharing platform. Section 5 introduces the construction method of a dedicated framework with remote sensing data characteristics and a visual modeling tool based on graphical user interface. Section 6 briefly reviews some application experiments in typical image intelligent interpretation tasks. Section 7 provides the conclusions and further research directions for improvement.

2. Background and related work

2.1. Remote sensing cloud computing platforms

RS-CCPs break through the bottleneck of traditional remote sensing technology and promote the technological innovation of data processing and analysis by combining huge data resources with high-performance cloud servers (Amani et al. 2020). Users can only focus on developing the fundamental theories and application models of remote sensing without having difficulties obtaining data and computing resources, thus significantly improving the efficiency of massive data processing and information mining. The main components of existing RS-CCPs include:

1. Available database. Most RS-CCPs provide a publicly available database that includes remote-sensing imagery from low to medium spatial resolution (e.g. Landsat, Sentinel, and MODIS), elevation datasets, land use and land cover datasets, and simulation datasets (Toth and Jóźków 2016). The CASEarth EarthDataMiner also provides public multidisciplinary datasets, like ecological, atmospheric, oceanic, primary geographic, and ground-measure datasets. Except for publicly available datasets, some RS-CCPs like Descartes Labs also provide business datasets, such as the SPOT satellite dataset and ship trajectory AIS (Automatic Identification System) dataset.

2. Application Programming Interfaces (APIs). Users can rapidly access, compute, and analyze data from the available database of RS-CCP by utilizing APIs provided by the computational frameworks (Bisong 2019).

3. Computational infrastructures. The large-scale Earth datasets and high-complexity DL models require massive computational resources (Tamiminia et al. 2020). Most of RS-CCPs are built based on large-scale cloud-based computing platforms (e.g. Google Cloud Platform, Amazon Cloud Platform, Huawei Cloud Platform, and Alibaba Cloud Platform) for quick processing of enormous geospatial datasets without suffering the IT pains of parallel computing and networking.

2.2. Sample databases for remote sensing

DL models are data-driven methods that learn complex patterns from Earth observation data (Cheng et al. 2018). The available datasets that include observations and associated labels are thus essential in remote-sensing intelligent interpretation. A number of sample datasets for different image interpretation tasks are publicly available, with new sample datasets being constantly released, including scene classification sample datasets, e.g. WHU-RS19 (Dai and Yang 2010), NWPU-RESISC45 (Cheng, Han, and Lu 2017), and AID (Xia et al. 2017), object detection sample datasets, e.g. VEDAI (Razakarivony and Jurie 2016) and DOTA (Xia et al. 2018), land-use classification sample datasets, e.g. GID (Tong et al. 2020) and INRIA (Maggiori et al. 2017), change detection sample datasets, e.g. HRSCD (Daudt et al. 2019) and WHU-BCD (Ji, Wei, and Lu 2018), and multi-view 3D reconstruction sample datasets, e.g. WHU-MVS (Liu and Ji 2020) and US3D (Marc et al. 2019). Although many RS-CCPs have included the publicly released datasets in their sample databases, it remains several problems in remote-sensing intelligent interpretation:

1. The existing classification schemes of current available sample databases are not in a standardized and unified manner (Tenopir et al. 2020). For example, there are numerous classification schemes for land-use classification tasks. It is often difficult to harmonize these classification schemes due to the heterogeneity of data attributes such as names, resolution, projection, observation time, data format, and semantics of classes.

2. A large-scale sample database with different data distributions is the key factor for improving the performance and generalization of DL models in remote sensing intelligent interpretation (Li and Hsu 2020). However, the limited number and variety of data in existing RS-CCPs make it hard to obtain a good generalization approach to make DL models fit beyond the training dataset.

3. The majority of an image interpretation task is typically spent on data preparation (Reichstein et al. 2019). Acquiring a large-scale sample database is very costly as labeling is usually manually done by in-house labor. A common solution is to crowdsource hand labeling tasks using services like Amazon Mechanical Turk. However, this leaves nonprofessional users with the time-consuming and difficult task of locating, labeling, and integrating disparate datasets. In the remote sensing field, a lack of an open data labeling and organization platform for large-scale and diverse sample database is an obstacle to realizing efficient image interpretation in geoscience applications.

2.3. Deep learning models and frameworks for remote sensing

Convolutional Neural Networks (CNNs) have been demonstrated to be strong and achieved state-of-art performance in many vision tasks (e.g. image classification, object detection, and semantic segmentation) (Alzubaidi et al. 2021). Examples include AlexNet (Krizhevsky, Sutskever, and Hinton 2017), VGG-Net (Simonyan and Zisserman 2014), GoogleNet (Szegedy et al. 2015), ResNet (He et al. 2016), DeepLab (Chen et al. 2017), and HRNet (Sun et al. 2019). These common vision models have numerous variants and form a wide variety of advanced CNN-based networks in different remote sensing image interpretation tasks, such as NEX-Net (Anwer et al. 2018), S2A-Net (Han et al. 2021), FreeNet (Zheng et al. 2020), DSIFN (Zhang et al. 2020), and MVS-Net (Yao et al. 2018). On the other hand, neural architecture search has become a new trend in model architecture design, including reinforcement learning strategy, evolutionary algorithm, and gradient-based method (Ren et al. 2021). DL frameworks accelerate the development of artificial intelligence techniques in vision applications by providing convenient high-level interfaces for users to design DL models on different computing devices (Chen et al. 2018). Some RS-CCPs have directly integrated these DL frameworks (like TensorFlow and PyTorch) into traditional modeling workflows to increase the intelligence level. However, the current DL models and frameworks used in geoscience AI research mainly focus on processing natural images (e.g. indoor-outdoor small-size images), which ignore the characteristics of remote sensing data such as large image size, large-scale change, multiple data channels, and geographic knowledge embedding, thus impairing the computational efficiency and accuracy.

3. Platform overview

Figure 2 shows a platform overview of LuoJiaAI. LuoJiaAI consists of a standard large-scale sample database (LuoJiaSET) and a dedicated DL framework (LuoJiaNET) for five typical remote sensing interpretation applications with a high-performance cloud storage and computation service. The platform is accessed and controlled through an Internet-accessible API and an associated user-friendly interactive development environment that enables rapid data processing and visualization of results.

Figure 2. A simplified platform diagram of LuoJiaAI.

The LuoJiaSET houses an extensive publicly available sample database for image interpretation, including remote-sensing sample datasets from various satellite and aerial imaging systems in optical and non-optical data types. All of these sample datasets are well organized in LuoJiaSET by a standard large-scale database that allows efficient retrieval, access, label, and share, removing the barriers associated with data management and computing. Users can access and analyze data using a library of operators provided by the data service interface and web console with OGC standard from the LuoJiaSET application layer.

The LuoJiaNET develops a dedicated framework and optimization method for exploiting remote sensing data characteristics (e.g. large image size, large-scale change, multiple data channels, and geographic knowledge embedding) to increase computational efficiency and accuracy in five typical image interpretation tasks. LuoJiaNET also develops a front-end application platform with a visual modeling tool to provide online model services (e.g. model creation, model training, and model deployment) to improve user productivity. Users can sign up for the LuoJiaNET homepage to access the user’s guide, tutorials, model examples, function references, and educational curricula.

4. LuoJiaSET

In remote sensing image intelligent interpretation, superior performance comes from large amounts of standard and high-quality labeled remote sensing data (Li et al. 2020; Van Etten, Lindenbaum, and Bacastow 2018; Sumbul et al. 2019). LuoJiaSET constructs a standard large-scale sample database with a unified and scalable classification scheme for multi-type sensors, multi-temporal and multi-scale remote sensing data (Section 3.1), and develops an Internet-based crowdsourcing sample collection tool and a sample sharing platform that provide strong data support for high-precision remote sensing mapping applications (Section 3.2).

4.1. Standard large-scale sample database

This section highlights the building steps of a standard large-scale sample database that supports multi-dimensional global retrieval and data analysis.

4.1.1. Sample model design

LuoJiaSET defines the corresponding sample types for typical remote sensing interpretation tasks, such as scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction. Figure 3 demonstrates the logical model of all sample types. In this model, the ground object samples are organized by planar images (e.g. single-phase and change detection samples) and multi-view stereo images. Different samples can have the same ground object, and samples from different application tasks can be related in accordance with their geographical coordinate. Each application task can include remote sensing data from different sensor types.

Figure 3. Logical model of sample type.

4.1.2. Unified and scalable classification scheme

LuoJiaSET builds a unified classification scheme that supports flexible and extended new categories for the sample database. The sample classification framework is shown in Figure 4. The main idea of this construction strategy is as follows: The whole samples in the database are divided into two concept hierarchies based on the national geographic information classification scheme (GB/T 30322). The ground cover category is defined by the dichotomy in the first classification level. For the second classification level, geographic attributes are added to existing categories for defining new classifications. The concepts of artificial-related ground objects in a two-layer classification scheme are extended by the definition of moving and fixed objects in the object detection task, and a new scalable classification scheme is formed for the whole sample datasets.

Figure 4. Sample classification framework.

4.1.3. Sample data integration

Figure 5 demonstrates the technical process of multi-source sample data integration. The original classification scheme in existing open-source datasets should be reformed and mapped to a new unified and scalable classification scheme adopted in LuoJiaSET. All sample data are transformed under the unified encoding rules and relational database structure designed by the LuoJiaSET sample database, and global queries are created for multi-dimensional retrieval and data analysis. The information of all sample datasets, including metadata, attribute table, classification scheme, and copyright, is stored in their corresponding metadata table to facilitate traceability, quality control, and ownership identification.

Figure 5. Flowchart of multi-source sample data integration.

4.2. Internet-based sample collection and sharing service

This section highlights the essential components of the sample sharing platform, including the crowdsourcing sample collection platform with human-machine interaction, and the primary infrastructure layers and high-level application Programming Interfaces (APIs).

4.2.1. Crowdsourcing sample collection with human-machine interaction

LuoJiaSET develops an Internet-based sample collection platform that supports the functions of crowdsourcing collection, online verification, and dynamic sample expansion. In the remote sensing field, label sampling is costly, laborious, and has a high demand for specialized geographic knowledge for sample annotators. Therefore, we propose an automatic/semi-automatic combined labeling tool to improve the efficiency and accuracy of label sampling. The main purpose of this tool is to allow participants to interact with automatic interpretation results obtained from trained network models and attempt to reduce recognition errors. The samples that meet the quality requirement after being verified online by professional staff will be accepted by the sample database. Figure 6 shows the whole technical flow diagram of this strategy.

Figure 6. Technical flow diagram of automatic/semi-automatic combined labeling.

4.2.2. Sample sharing platform

LuoJiaSET develops an open-source sample sharing platform for remote sensing interpretation, which offers the function of global queries for multi-dimensional retrieval, statistical analysis of data, and data sharing services. Figure 7 shows the overall framework of this platform. The infrastructure layer is constructed by distributed storage systems, DL Graphical Processing Unit (GPU) clusters, file systems, database systems, and a modern network environment from Object Storage Service of Huawei’s AI Computing Center in Wuhan. The abilities of storage, expansion, maintenance, and copyright protection of multi-source remote sensing data are supplied in the data storage layer. The service layer provides the APIs for data access and operation and supports multi-dimensional semantic queries and data delivery service. The high application layer provides users with more advanced online functions in data access and analysis, such as spatiotemporal data mining, statistical analysis of sample attributes, and knowledge discovery in the whole database.

Figure 7. Overall framework of the sample sharing service.

On the basis of the proposed classification scheme and workflow, the LuoJiaSET sample sharing platform has processed 83 open-source datasets and built a database containing about 5 million samples that cover five typical remote sensing image interpretation tasks, such as scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction. The statistical analysis of the whole datasets is shown in Figure 8, and the front-end applications for sample query and display are shown in Figures 9 and 10, respectively. The LuoJiaSET platform provides high-level service APIs in the OGC Training Data Markup Language for AI standard for better data sharing and model developing (Figure 11).

Figure 8. Dataset statistics in LuoJiaSET platform.

Figure 9. Sample query interface in LuoJiaSET platform.

Figure 10. Dataset information displayed in LuoJiaSET platform.

Figure 11. High-level service APIs in LuoJiaSET platform.

5. LuoJiaNET

The DL framework is the crucial infrastructure to connect the hardware, software, and AI applications (Liermann 2021; Chen 2021). LuoJiaNET designs the dedicated DL framework with a Graph-Model-Spectral collaboration core layer for embedding remote-sensing data characteristics (e.g. large image size, large-scale change, multiple data channels, and geographic knowledge embedding) into the underlying system. LuoJiaNET’s simplified system architecture is shown in Figure 12. The layer above the core of LuoJiaNET is front-end applications and user interfaces, and the hardware device and data access layer are under the core of LuoJiaNET. The core layer of LuoJiaNET includes Intermediate Representations (IRs), computational graph optimization, execution engine, memory management, visual optimization, security protection, remote sensing data characteristics, and data-framework collaboration. Section 4.1 illustrates the graph-based core framework modules of LuoJiaNET. Section 4.2 demonstrates the Model-Spectral driven data processing module to satisfy the demands for remote sensing characteristics embedding. Section 4.3 introduces the overall architecture design of LuoJiaNET’s front-end layer.

Figure 12. Simplified system architecture diagram for LuoJiaNET.

5.1. Graph based core framework modules

The core modules of LuoJiaNET are employed to build the Directed Acyclic Graphs (DAGs) for data processing. The neural networks can handle input remote sensing data from multiple layers, integrate remote sensing features into data flow and control flow, and implement data-framework collaboration through DAGs representation. Specifically, the core modules of the dedicated DL framework include the construction of network operators and device management, IRs and their compilation optimization, and automatic differentiation, and modules with remote-sensing data characteristics, such as memory scalability for large-scale images and adaptive optimization methods for optimizing multi-scale and multi-channel image features.

In the LuoJiaNET framework, the hardware module provides a unified interface and shields the underlying hardware differences (e.g. CPU, GPU, and NPU). The data organization and management module is applied to achieve high-efficiency storage and read/write for the sample database. The data organization and management module takes charge of data-framework collaboration tasks, including automatic data augmentation and refinement in LuoJiaSET’s database. LuoJiaNET has compiled and tested the Geospatial Data Abstraction Library (GDAL) for reading and writing various raster and vector data formats under different operating systems. Without introducing any closed-source projects, the GDAL enables LuoJiaNET to have a controllable and efficient Input-Output (I/O) module to process multi-source remote sensing data.

5.2. Model-spectral driven data processing module

This section highlights the strategies of remote-sensing data characteristics embedded in the LuoJiaNET framework, including a memory scalability module and a parallel computing strategy for large-scale image processing, the adaptive optimization method for adaptive learning multi-scale and multi-channel image spectral features, and an auto-parallel strategy for embedding geographic knowledge. The experiments demonstrate the effectiveness of the proposed data processing modules.

5.2.1. Large-scale image processing

5.2.1.1. Indirect buffer group mapping

LuoJiaNET proposes a memory scalability module to construct the correlation buffer areas of local features and global spatial context for supporting large-scale image loading, mapping, and learning. Figure 13 demonstrates the brief procedure of this Indirect Buffer Group Mapping (IBGM) method. The large-scale image is divided into different global correlation buffers based on labeled data. The pointers reserved in global correlation buffer areas are called global indirect buffer (GCP) unit. In the GCP Unit, the technique of quadtree spatial indexing is constructed to obtain context features in each local context processing (LCP) unit. The GCP unit and LCP unit features are concatenated in the memory scalability module and weighted by General Matrix Multiplication unit in the network model to achieve the global-local context representation and learning.

Figure 13. Process of IBGM method in memory scalability module for large-scale image processing.

Table 1 reports the mean Intersection over Union (MIoU) results for comparison of the patch-based processing and IBGM method on Gaofen Image Dataset (GID) using the HRNetV2 model (Sun et al. 2019). As shown in Table 1, the network with IBGM method outperforms the traditional patch-based processing method by 1.25% in overall MIoU. Figure 14 visualizes the prediction results on the validation set, which proves that global-local context information is essential. The proposed IBGM method supports large-scale image loading and training, and outperforms the traditional patch-based image processing method.

Figure 14. Visualization of different large-scale image processing methods on GID.

Table 1. Land-use classification results on GID with HRNetV2 model.

Methods	MIoU (%)
	Background	Build-up	Farmland	Forest	Meadow	Water	Overall
HRNetV2	61.03	62.86	63.36	28.02	53.17	78.65	57.85
+ IBGM	63.46	63.93	64.56	29.13	53.25	80.29	59.10

5.2.1.2. Distributed operator decomposition

LuoJiaNET proposes a distributed parallel computing strategy based on the idea of operator equivalent decomposition to handle large-scale image calculation in neural networks (Figure 15). In this strategy, the large-scale input images need to be cut and distributed to multiple training cards. The networks, such as the fully convolutional network (FCN) model (Long, Shelhamer, and Darrell 2015), need to be reformed through IRs and compiler optimization to have the ability to calculate large-scale images in the LuoJiaNET framework directly. The core of this method is to decompose the original large-scale input feature into several small pieces of features for network computing. These small features are combined and recovered to the original size by using image stitching technology, making the computational results equivalent to the initial large-scale input size calculation. Hence, one training card only needs to take charge of a small piece of the large-scale input image, thereby vastly reducing the requirements for device memory.

Figure 15. Workflow of distributed parallel computing strategy.

Table 2 reports the overall MIoU for comparison of the patch-based processing and distributed parallel computing strategy on GID using the FCN8s model. As shown in Table 2, large-scale image input with distributed parallel computing strategy outperforms the traditional patch-based processing method by 5.05% in overall MIoU. Figure 16 visualizes the prediction results on the validation set. This result proves that the proposed distributed parallel computing strategy enables the network to directly process large-scale images and improve the extraction accuracy by learning larger spatial contexts of images.

Figure 16. Visualization of different large-scale image processing methods on GID.

Table 2. Land-use classification results on GID with FCN8s model.

Methods (Backbone=FCN8s)	Overall
	MIoU (%)
Patch-based Processing	57.85
Distributed Parallel Strategy	62.90

5.2.2. Adaptive optimization

5.2.2.1. Adaptive scale memory pool

LuoJiaNET constructs a scale memory pool for adaptive storing, learning, and optimizing multi-scale stacked image features (Figure 17). This method’s key technologies include 1) Multi-scale image pyramid construction. Similar to the Laplacian scale space construction, the high-order pyramid is introduced to reduce the loss of scale variation in building an image pyramid. 2) Scale memory pool construction. For each LCP unit image feature (Section 4.2.1), its current-scale, previous-scale, and next-scale images are all stored in the scale memory pool. In each network training iteration, the scale memory pool stacks multiple-scale features and sends them to a DL network for adaptive learning.

Figure 17. Process of adaptive optimization method for optimizing multi-scale and multi-channel image features.

5.2.2.2. Dynamic feature selection filter

LuoJiaNET proposes an adaptive optimization method based on a dynamic feature selection filter to learn and optimize data channels adaptively. As shown in Figure 17, this optimization method uses a fixed memory array size to dynamically store and update the best features in end-to-end network training. The similarity between predicted value and label is the criterion for introducing the high-quality features for each ground object in the dynamic feature selection filter. These high-quality features will be applied as additional weights to guide network training.

Table 3 reports the overall OA and Kappa for validating the adaptive optimization method on the WHU-Hi dataset (Zhong et al. 2020) using the FreeNet model. As shown in Table 3, the network with adaptive optimization is improved by 1.54% in OA and 1.88% in Kappa. Figure 18 visualizes the prediction results on the validation set. This result proves that the proposed adaptive optimization method can bring performance improvements for network models.

Figure 18. Visualization of the adaptive optimization method on WHU-Hi.

Table 3. Land-use classification results on WHU-Hi with FreeNet model.

Methods	Overall
	OA (%)	Kappa (%)
FreeNet	94.47	94.02
+ Adaptive Optimization	96.01	95.90

5.2.3. Parallel geographic knowledge embedding

LuoJiaNET proposes an auto-parallel computing strategy embedding geographic knowledge for processing different types of remote sensing images (e.g. multi-spectral, hyperspectral, and SAR). This prior knowledge can be integrated into the cost function of the LuoJiaNET framework in multi-dimensional mixed auto-parallel computing by developing the specific operators for prior geographic knowledge extraction (e.g. image texture, boundary, and physics-based feature extraction), thereby achieving the optimal computational assignment of DAGs on multiple hardware devices. Figure 19 demonstrates this auto-parallel computing strategy. The auto-parallel cost function contains 1) Delay and bandwidth communication cost of transmission operators between devices. 2) Operation cost on different devices based on auto-parallel strategy. 3) Geographic knowledge-based cost in parallel training.

Figure 19. Auto-parallel computing strategy embedding prior geographic knowledge.

In the experiment, we choose the gray-level co-occurrence matrices (GLCMs) (Ojala, Pietikäinen, and Harwood 1996) of remote sensing images as the prior geographic knowledge. Table 4 reports the top-1 accuracy and top-5 accuracy for comparison of nonparallel training, auto-parallel training with GLCM image features, and auto-parallel training with GLCMs embedded in the cost function on the WHU-RS19 dataset (Xia et al. 2010) using the ResNet-50 model. As shown in Table 4, auto-parallel training with GLCM image features outperforms the nonparallel training strategy by 3% in top1-accuracy. The top-5 and top-1 accuracies of auto-parallel training with GLCMs embedded in the cost function are increased to 99% and 87%, respectively. This result proves that the proposed parallel optimization method can significantly improve network accuracy.

Table 4. Scene classification results on WHU-RS19 with different parallel computing strategies.

Methods	Top1-acc (%)	Top5-acc (%)
ResNet-50	86.5	98.5
+ Auto-parallel (+ GLCM Input)	89.5	98.0
+ Auto-parallel (+ GLCM in Cost Function)	87.0	99.0

5.3. Framework sharing platform

This section highlights the technology stack of LuoJiaNET’s front-end application architecture. The human-machine interactive modeling tool on the LuoJiaNET platform is introduced to improve users’ productivity.

5.3.1. Front-end application architecture

LuoJiaNET’s front-end application architecture has three browser/server architecture-based layers, including the interface layer, application layer, and data layer (Figure 20). Supported by Ascend APU DL server of Huawei’s AI Computing Center in Wuhan, the LuoJiaNET platform deploys training services in the cloud to provide operator-friendly applications and efficient background computing service for users. The LuoJiaNET platform links the Cesium web map (https://cesium.com/) with the model prediction results through their geographical coordinates, thereby enabling users to view interpretation results in the front-end map window directly.

Figure 20. Overall architecture of the LuoJiaNET’s front-end application.

As shown in Figure 21, the LuoJiaNET sharing service platform’s main page is composed of the parameter setting bar and map visualization window. Users can switch the map online and freely view it at any map’s zoom level. Users can draw a region on the satellite map and then employ the network models from LuoJiaNET’s model database (Figure 22) to accomplish five typical remote sensing intelligent interpretation tasks, such as scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction. The final model prediction results can be directly displayed on the map for quick check and output.

Figure 21. Main page of the front-end application, with object detection taken as an example.

Figure 22. Model database of LuoJiaNET.

5.3.2. Visual modeling tool

LuoJiaNET develops a visual modeling tool based on GUI to provide data-framework collaboration. The overall architecture of this visual modeling tool is shown in Figure 23. This tool transfers the complex neural network structure into a form of comprehensible DAGs diagram representation to improve users’ productivity in building models. The input data are successively transmitted, calculated, and flow into the training or storage terminal of LuoJiaNET through every graph node in DAGs. This tool sets up a technological workflow of online data acquisition, network creating, training, and deployment, which successfully hides the complex programming and computing work in the background services to intensely facilitate nonprofessional users.

Figure 23. Overall architecture of the LuoJiaNET’s visual modeling tool.

The main page of LuoJiaNET’s visual modeling tool is shown in Figure 24. The left side is the operator database of various network models. Users can directly drag the operators into the main modeling window to generate the corresponding DAGs node of models. Users can perform actions, such as copying and undoing nodes, by using the toolbar at the top of the main window. The right side of the main page is the model parameter editor and monitoring tool for the training process. Users can access the script editor on the bottom side of the main window to import APIs from LuoJiaSET’s sample database to obtain remote sensing data and write Python code for model training and inference.

Figure 24. Main view of visual modeling tool.

6. Applications

LuoJiaAI is in used across different remote-sensing interpretation tasks, such as scene classification (Figure 25), object detection (Figure 26), land-use classification (Figure 27), change detection (Figure 28), and multi-view 3D reconstruction (Figure 29). The results of these validation experiments prove the effectiveness of LuoJiaAI and provide huge support for further optimizing and improving the LuoJiaAI’s overall architecture. The quantitative and qualitative experiments can be referred to the https://github.com/WHULuoJiaTeam/luojianet/blob/master/whitepaper.pdf, and the codes for these AI models can be found at https://github.com/WHULuoJiaTeam/luojianet/tree/master/model_zoo.

Figure 25. Visualization on AID using ResNet-50 in scene classification task.

Figure 26. Visualization on DOTAV2.0 (Ding et al. 2021) using Mask-RCNN (He et al. 2017) in object detection task.

Figure 27. Visualization on ISPRS-Vaihingen (Rottensteiner et al. 2012) using U-Net (Ronneberger, Fischer, and Brox 2015) and DeepLabV3+ (Chen et al. 2018) in land-use classification task.

Figure 28. Visualization on WHU-BCD using DTCDSCN (Liu et al. 2020) in change detection task.

Figure 29. Visualization on WHU-Stereo (Li et al. 2022) using GC-Net (Ni et al. 2020) in multi-view 3D reconstruction task.

7. Conclusion and future work

In this paper, we illustrate the design considerations and methods for constructing the LuoJiaAI platform, which is composed of a large-scale sample database with a unified and scalable classification scheme (LuoJiaSET) and a dedicated DL framework with remote sensing data characteristics (LuoJiaNET). It is a cloud-based AI platform with OGC standards for intelligent interpretation and analysis of remote sensing data that brings Huawei’s massive data storage and computational capabilities to achieve state-of-the-art performance on five typical remote sensing interpretation tasks, including scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction.

The RS-CCP is bound to be the long-term topic in remote-sensing interpretation with continuous data and computing power growth. The research and development of LuoJiaAI are still in the infancy stage. For LuoJiaSET, we will continue expanding the sample database’s size and improving sharing service platform functions, such as multi-dimensional retrieval and data analysis. For LuoJiaNET, we will further study the strategies for large-scale image processing, optimization methods for IRs with remote sensing data characteristics, data-framework collaboration mechanisms for automatically obtaining high-quality labeled data, and more visual tools and APIs in front-end web service for better human-machine interaction. LuoJiaAI shows a great potential to break the separation of remote sensing data, intelligent interpretation algorithms, and cloud-based computing resources in high-precision remote sensing mapping tasks.

Acknowledgments

We appreciate the effort led by WHU-LuoJiaAI Group and Huawei Artificial Intelligence Group. Great thanks to all the colleagues who have provided their time and advice while writing this paper. Special thanks to Dr. Liangcun Jiang, Dr. Yue Xu, Mr. Haowei Jia, Mr. Yuanxin Zhao, Ms. Siqi Liu, Mr. Bingnan Yang, Mr. Zhenzhang Yang, Mr. Teng Su, Mr. Guowen Zhang, Mr. Laiping Ding, Mr. Shenkai Zhoung, Mr. Jingjun Wang, Mr. Kunyang Tian, Pro. Peng Yue, Pro. Shunping Ji, Pro. Xin Huang, Pro. Haigang Sui, Pro. Guisong Xia, Pro. Yansheng Li, Pro. Kaimin Sun, Pro. Juhua Liu, Pro. Mengting Yuan, Dr. Jiayi Li, and Dr. Xinyu Wang for all the kind support. Thanks to the rest of the WHU-LuoJiaAI Group: Mr. Bin Wang, Ms. Jin Liu, Mr. Haotian Teng, Mr. Jianxun Wang, Mr. Wangbin Li, Mr. Lilin Tu, Mr. Ning Zhou, Mr. Hengwei Zhao, Mr. Yang Pan, Mr. Wei Chen. Finally, we would like to thank the providers of the thousands of public datasets in LuoJiaAI.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

The data that support the findings of this study are available on GitHub at https://github.com/WHULuoJiaTeam.

Additional information

Funding

This work is supported by the Chinese National Natural Science Foundation Projects [grant number 41901265], a Major Program of the National Natural Science Foundation of China [grant number 92038301], and was supported in part by the Special Fund of Hubei Luojia Laboratory [grant number 220100028].

Notes on contributors

Zhan Zhang

Zhan Zhang is a PhD student at the State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan, China. His research interests mainly include machine learning system, remote sensing intelligent interpretation, and machine vision. He contributed to the LuoJiaAI platform design and integration, LuoJiaNET’s remote sensing characteristics module implementation, and manuscript writing of this paper.

Mi Zhang

Mi Zhang is an associate researcher at the School of Remote Sensing and Information Engineering, Wuhan University. He serves as the chief artificial intelligence scientist in Handleray Corporation and technical director in WHU-LuoJiaAI Group. His research interests mainly include computer vision, machine learning, with particular interest in semantic object segmentation and the construction of DL framework. He contributed to the overall architecture design and implementation of the LuoJiaNET, and manuscript writing of this paper.

Jianya Gong

Jianya Gong is the Academician of Chinese Academy of Sciences, Professor and Dean of School of Remote Sensing and Information Engineering, Wuhan University. His research interests mainly include remote sensing image processing, spatial data infrastructure, geospatial data interoperability, and artificial intelligence. He contributed to the LuoJiaAI project initiation, and implementation of this paper.

Xiangyun Hu

Xiangyun Hu is a professor and head of the department of photogrammetry with the School of Remote Sensing and Information Engineering, Wuhan University. His research interests mainly include artificial intelligence and pattern recognition, remote sensing software development, and remote sensing intelligent interpretation. He led the WHU-LuoJiaAI Group and contributed to the methodology and implementation of this paper.

Hanjiang Xiong

Hanjiang Xiong is a professor in 3-D geographic information system at Wuhan University. His research interests mainly include geospatial data management, 3-D visualization, augmented reality, and indoor and outdoor geographic information system. He contributed to the implementation of this paper.

Huan Zhou

Huan Zhou is a master’s student at the Department of Land-Surveying and Geo-Informatics, Hong Kong Polytechnic University. His research interest is in web server development and cartographic visualization. He contributed to the overall architecture design and implementation of the LuoJiaNET’s front-end layer, and use cases of this paper.

Zhipeng Cao

Zhipeng Cao is a PhD student at the School of Remote Sensing and Information Engineering, Wuhan University. His research interest is in big spatiotemporal data management. He contributed to the implementation of this paper.