mcRPL: a general purpose parallel raster processing library on distributed heterogeneous architectures

Abstract

Parallel computing on distributed heterogeneous architectures (e.g. computing clusters with multiple CPUs and GPUs) can significantly improve the computational efficiency and scalability of complicated algorithms, but it is theoretically and technically complex. Parallel raster processing libraries reduce the development complexity of parallel raster algorithms by hiding parallel computing details; however, no existing library sufficiently utilizes distributed heterogeneous computing resources. A general-purpose raster processing library (mcRPL) combining multi-process parallelism and multi-thread parallelism is proposed to enable parallel raster processing on distributed heterogeneous architectures with multiple CPUs and GPUs. Additionally, an adaptive hardware assignment strategy is proposed to fully utilize available processors in various hardware environments. A series of task-processing strategies are adopted to aim toward maximizing the utilization of the computing capacity of involved processors. Experiments revealed that two raster algorithms parallelized using mcRPL for spatiotemporal data fusion and land-use change simulation were 170.7- and 143.2-fold faster than original serial algorithms using 8 and 16 GPUs, respectively. While hiding the details of mixed parallelism and reducing the development complexity, mcRPL provides user-friendly interfaces for the development of parallel raster algorithms to enhance computational performance and enable large-scale raster computing tasks with extensive data volumes.

Keywords:

• Parallel computing
• raster processing
• heterogeneous computing
• distributed memory
• shared memory

Authors’ contributions

Huan Gao, Xuantong Peng, and Qingfeng Guan proposed the idea of this research. Qingfeng Guan designed the framework of the library. Huan Gao and Xuantong Peng developed the library and debugged the code. Jingyi Wang, Ziqi Liu, and Xue Yang contributed to code debugging. Huan Gao and Jingyi Wang conducted experiments and drafted the manuscript. Qingfeng Guan and Wen Zeng amended the manuscript.

Disclosure statement

No potential competing interest was reported by the author(s).

Data and codes availability statement

Source codes of mcRPL and test data are available at the following link: https://github.com/HPSCIL/mcRPL

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [Grant No. 42171466].

Notes on contributors

Huan Gao

Huan Gao received a B.E. degree from the China University of Geosciences, Wuhan, China, in 2018, where he is pursuing a Ph.D. degree in surveying and mapping. His research focuses on high-performance geospatial computing.

Xuantong Peng

Xuantong Peng received an M.S. degree in Cartography and Geographical information engineering from China University of Geosciences, Wuhan, China, in 2019. His research focus on high-performance geospatial computing.

Qingfeng Guan

Qingfeng Guan is a professor in the School of Geography and Information Engineering at China University of Geosciences (Wuhan). His research focuses on high-performance spatial intelligent computing and urban modeling.

Jingyi Wang

Jingyi Wang received the M.S. degree in Cartography and Geographical information engineering from China University of Geosciences, Wuhan, China, in 2022. Her research focuses on data mining and spatiotemporal modeling.

Ziqi Liu

Ziqi Liu is a master candidate in the School of Geography and Information Engineering at China University of Geosciences (Wuhan). His research focuses on high-performance geospatial computing.

Xue Yang received a B.E. degree from the China University of Geosciences, Wuhan, China, in 2018, where she is pursuing a Ph.D. degree in surveying and mapping. Her research focuses on high-performance geospatial computing.

Wen Zeng

Wen Zeng is a Professor in the School of Geography and Information Engineering at China University of Geosciences (Wuhan). His research focuses on data acquisition and modeling for urban geographic information systems, geographic networks optimization, geographic information systems for transportation, and municipal infrastructure management.