A deep-learning model for semantic segmentation of meshes from UAV oblique images

ABSTRACT

Though mesh generated by UAV-based oblique photogrammetry has revolutionized the large urban scene data representation, its semantic segmentation still challenging due to structural irregularities. In this paper, a deep learning model is proposed for the mesh semantic segmentation. To efficiently capture local geometry and neighbourhood context within the mesh, graph concept and transformer blocks are employed for mesh representation and feature learning. Based on these components, a hierarchical network architecture is presented. Experimental results demonstrate that the proposed network performs well on the self-made Wuhan dataset. The ablation study shows the improvement brought by each component in the network. By comparison, both mIoU and mF1 of our network are much higher than the prior work on the public SUM dataset, achieving 83.7% and 73.7%.

Highlights

• A deep learning framework considering the topology and texture is provided for mesh semantic segmentation from UAV oblique images.

• Transformer blocks are employed for feature learning to efficiently capture neighbourhood context.

• Graphs indicating the topology are generated from mesh to facilitate feeding unstructured data to the deep learning framework.

• Results on SUM-Helsinki benchmark datasets demonstrate the effectiveness of the proposed method in comparison with other state-of-the-art methods.

Acknowledgement

The SUM dataset is provided by the Amsterdam Institute for Advanced Metropolitan Solutions, Delft University of Technology: https://3d.bk.tudelft.nl/projects/meshannotation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Author statement

Yetao Yang designed experiments; Rongkui Tang completed the coding and carried out experiments; Rongkui Tang and Chen Zhang analysed experimental results; Rongkui Tang, Mengjiao Xia and Yetao Yang wrote the manuscript.

Data availability statement

The Wuhan dataset supporting the findings of this study are available in figshare.com with the identifier(s) at the link https://doi.org/10.1080/01431161.2022.2111665.

Additional information

Funding

The research is funded by the National Natural Science Foundation of China (grant numbers 41201430) and Hubei Technology Innovation Major Project (grant number 2018AAA029)