With the development of new technologies such as 5G/6G, cloud computing, the Internet of Things, and artificial intelligence, humanity has entered the era of the interconnection of everything. The traditional remote sensing satellite data acquisition, processing, and application models have been unable to meet the popular and real-time application needs. In order to meet the service needs of the geospatial information industry in the era of the Internet of Things, it has become an inevitable trend to move from isolated remote sensing satellites to space-based information networks.
On 15 January 2023, the first Internet intelligent remote sensing () scientific experimental satellite-“Luojia3–01”, was successfully launched in Taiyuan, China. Facing the major needs of national aerospace technology. Luojia3–01 is an important achievement in exploring new theories, technologies, and equipment for aerospace information intelligence services. It has established an open mode experimental verification platform for remote sensing and communication integration, which can demonstrate the entire process of information acquisition, intelligent processing, sparse representation, compression transmission, and real-time distribution of terminals worldwide.
Figure 1. Internet intelligent remote sensing satellite real time service.
Reflecting the goal of “fast, accurate, and intelligent” remote sensing data application in the spatial information network environment, LuoJia3–01 provides an open on-orbit experimental verification platform for relevant scientific research institutions (users can obtain interested remote sensing data, design and develop relevant APP software to carry out on-orbit experiments on the LuoJia3–01 platform), and will also prepare the technology to promote the next step of real-time intelligent services for space-based information under the Internet.
1. Design of a new generation remote sensing satellite with multimode open intelligent interconnection
Luojia3–01 is equipped with a light and small optical remote sensing camera for multi-mode imaging, which has an on-orbit high-performance processing platform, an open software platform, and supports flexible loading and installation of intelligent processing APP on orbit. It has the following four core features.
1.1. Multi-mode
The satellites have a platform with strong maneuverability, high pointing accuracy, and stability, on which agile high-resolution video gaze imaging can be performed. The optical axis of the camera always points toward a fixed ground target point. During the imaging time period, continuous and fast tracking of the target point is achieved by adjusting the satellite attitude in real-time, meeting the requirement of long-term continuous observation of hot spots for up to 90 seconds. On the other hand, satellites can perform three-axis attitude maneuvers within a deviation of 35° from the sub satellite point, dynamically tracking targets from different angles, and achieving multi angle stereo imaging. When push frame imaging on a planar array, the optical axis is controlled by attitude control to point along the satellite’s flight direction and scan at a uniform speed on the ground. The relative ground speed is less than the satellite’s flight speed. By controlling the camera, multiple single images can be formed, with a point of overlap between each image. The multiple overlapping images form a strip. During push scan imaging, the satellite maintains a fixed attitude and can perform dual CMOS combination imaging to obtain continuous large area images. The multi-mode characteristics such as high-resolution video, multi angle stereo, and continuous area imaging can meet the observation needs of different application scenarios.
1.2. Intelligence
A breakthrough as “task planning, sensor correcting, target detection, intelligent compression” has been achieved on-orbit real-time processing technology, to solve the problem of delayed processing of massive remote sensing data. In response to the problem of untimely processing of massive remote sensing data, a creative “task planning, sensor calibration, target detection, and intelligent compression” on orbit real-time intelligent processing technology system has been proposed to solve the problem of untimely processing of massive remote sensing data. Aiming at the difficulty to process data in real time under the limited conditions on board, a task driven satellite ground collaborative on orbit streaming processing architecture is proposed, and a satellite ground collaborative on orbit service mechanism for data migration and computing migration is constructed. The data in the area of interest is precisely locked, and the heterogeneous multi-core computing platforms of DSP, FPGA, and GPU are adapted to realize the task of “acquiring while processing while transmitting”. In order to achieve real-time intelligent information extraction on the satellite, a “ground training onboard detection feedback update” satellite ground linkage framework was designed through the intelligent processing unit onboard, to solve the problem of deep learning models being unable to migrate to new target detection. Transforming traditional offline processing and analysis tasks to onboard for the first time can effectively enhance the intelligence and effectiveness of satellite services. For the difficult transmission and poor reconstruction quality of massive remote sensing data, sparse representation of satellite remote sensing images and on orbit high magnification compression technology are proposed to break through the compression limit of single source data and effectively remove long-range redundant information from temporal remote sensing image data.
1.3. Interconnection
Interconnection is to realize the mutual communication and transmission of “satellite-receiving station – mobile phone”. It mainly uses the ground station to carry out mission registration and data downtransmission, and completes the distribution of on-orbit processing data to the mobile user terminal. Firstly, according to the observation requirements of users in the area of interest or target, the client is verified through integrated demonstration, and the observation task requirements are submitted. The ground measurement and control data transmission station will inject the task requirements into the intelligent remote sensing satellite. The autonomous task planning of intelligent remote sensing satellite generates and executes the task instructions, conducts imaging when the satellite passes through the task area, and obtains the required images; The on-board intelligent processing unit carries out real-time data positioning, geometric correction, cloud detection, target detection, change detection, moving target extraction and tracking, and performs intelligent and efficient compression, and transmits the data to the ground receiving station in real time through the satellite-ground transmission link. Finally, real-time distribution is carried out through the ground network, and on-orbit processing results are fed back to the user’s mobile terminal by means of 5G mobile phone base stations or wifi hotspots, supporting the provision of real-time intelligent services of data acquisition end-to-end remote sensing information to the general public. This mode innovatively integrates the satellite-ground link with 5G mobile communication, opening up the two-way link between satellite and mobile phone, and realizing the minute-level intelligent service from global remote sensing data to mobile phone.
1.4. Open
LuoJia3–01 provides open data source and satellite on orbit algorithm platform services. High-performance hardware processing platform and open software platform (including basic operating system environment, user API function interface, image processing basic library, deep learning software framework, etc.) are designed to support the flexible implementation of diversified intelligent applications on orbit. In addition to ordering interested data on demand, users can also independently design and develop relevant APP software, flexibly fill, update and uninstall intelligent APP algorithms on orbit, and carry out on-orbit scientific experiments according to different mission requirements. At present, nine APPs including target detection, change detection and image compression are pre-installed on the satellite before launched.
2. Multimode imaging for diverse user’s needs
2.1. High resolution continuous area imaging
Luojia3–01 is equipped with a small and lightweight sub meter resolution Bayer color area array camera, which can obtain images with a resolution of better than 0.72 m at subsatellite points on a 500 km orbit. At the same time, it can perform push frame or push broom continuous area imaging, with maximum imaging time of 10 minutes and width of more than 10 kilometers.
2.2. High resolution video real-time imaging
Satellite video staring imaging and transmission is a key issue in the multimode and intelligent technology of Luojia3–01. The effectiveness of video staring requires that the camera accurately point to the target area during high-speed motion of the satellite. At the same time, the large amount of video staring imaging data and the time-consuming satellite to ground transmission pose challenges for intelligent transmission. Luojia3–01 uses on-orbit high magnification compression and transmission technology independently developed by Wuhan University, with a maximum transmission speed of 2Gb/min, effectively solving the transmission delay problem caused by bandwidth constraints. Video staring imaging data can serve target monitoring tasks with high time sensitivity requirements, such as port ships, highway vehicle flow, and airborne moving target tracking and monitoring tasks.
2.3. Multi-angle 3D imaging
LuoJia3–01 has the ability of high agility, high stability and high precision image acquisition. Under the imaging mode of video staring, it can realize multiple multi-angle observation of an area with the same orbit and obtain high precision three-dimensional image. After processing by real 3D reconstruction software, different 3D geographical scenes can be constructed to meet the requirements of terrain level, city level and component level, which will play an important role in urban planning and management, natural disaster assessment and other fields.
Real-scene 3D modeling mainly includes aerial triangulation, dense matching, point cloud network and texture mapping. SIFT is used in aerial triangulation to extract image feature points and match images. On the basis of matched connection points, geometric distortion correction is carried out by integrating alternating approach method and RFM model transformation, and adjustment processing of uncontrolled area network is completed for large-scale image, and sparse point cloud of scene is generated. Point cloud matching makes full use of the redundant condition of “multi-view”, which can be matched by pairwise stereo image pairs to carry out result weighting and verification. It can also be iterated according to different conditions of base height ratio and side swing Angle combined with terrain characteristics to get the dense point cloud relatively closer to the true value. Here, SGM algorithm is used for pixel by pixel matching. By looking for areas with image consistency, the image is generated against dense point clouds. Certain interpolation and filtering algorithms are used to get the digital surface model DSM. To determine the depth of pixels relative to the camera plane (i.e. three-dimensional point coordinates). Based on DSM products, Delaunay triangulation network was constructed, and high-quality 3D model was generated through triangulation segmentation and texture mapping.
Selected 50 frames of multi-view image data set in Xigaze area of China, and the high-resolution airborne lidar (). Lidar data of 20 km2 subset in this area were used as ground truth values to evaluate the accuracy and integrity indexes. After measurement and verification, the absolute coordinate error of the model was within 1.0 m without control points. The computing efficiency is that a single node can reconstruct an area of about 600 km2 in a day. The computer environment is CPU of Intel I7-9700K, the graphics card of NVIDIA GeForce GTX 2070, and the memory of 64GB.
Figure 2. Real-scene 3D reconstruction data.
3. Mission driven high precision on-orbit positioning and intelligent processing technology
In order to meet the requirements for efficient data compression of remote sensing images in an on-orbit environment, Luojia3–01 has adopted a task oriented on-orbit intelligent compression method, which achieves near real-time high magnification compression of regions of interest through adaptive bit rate allocation for different task requirements. The tests have shown that still-image compression ratio is better than 30 times, and dynamic-video compression ratio is better than 200 times.
Besides, Luojia3–01 has achieved a breakthrough in actively discovering and identifying interested targets, which can perform on-orbit cloud detection, target detection, moving target tracking (), scene classification, change detection, etc. Compared to the traditional mode of “transmission before analysis” for existing remote sensing satellites, satellite on-orbit target detection accuracy and moving target tracking accuracy can meet most real-time intelligent analysis application requirements. At the same time, the intelligent processing unit carried by Luojia3–01 also provides an important computing platform for deep learning model reasoning. Combining deep learning algorithms, it makes the satellite more intelligent, providing an important guarantee for real-time intelligent services of remote sensing information.
Figure 3. Moving target tracking results.
4. A new B2C mode of internet intelligent remote sensing satellite real time service
The on-orbit processing and real-time transmission characteristics of the Luojia3–01 have broken the traditional satellite remote-sensing data service model, creating a new B2C (Business-to-Customer) model for the Internet intelligent remote sensing satellite on-orbit processing and real-time transmission service for ground mobile terminals, shortening the time for satellite data acquisition, processing, transmission, and distribution, and meeting the “fast, accurate, and intelligent” real-time information service requirements for remote sensing information. It can effectively improve the dynamic emergency response efficiency of emergencies and provide strong support for the popularization of remote-sensing data mobile terminal services.
On-orbit scientific experiments and tests have shown that it takes only about 8 minutes from acquiring data from the satellite terminal to receiving and displaying satellite video data on the user’s mobile terminal, achieving a minute level real-time intelligent service for remote sensing information.
5. Summary and prospect
Focusing on the significant demand for “fast, accurate, and intelligent” remote sensing information services, Luojia3–01 has built a new generation of remote sensing satellite platforms with multimode open intelligent interconnection, breaking through mission driven on orbit high-precision positioning and intelligent processing technology, and creating a new B2C model for real-time services for intelligent remote sensing satellites on the Internet. In the future, it will provide a real-time verification platform for on-orbit functional software related to scientific research, and provide intelligent remote sensing services based on mobile phone apps to the public, promoting the leapfrog development of satellite remote sensing technology from traditional professional data services to popular, intelligent, and real-time remote sensing information services.