Instance segmentation of point cloud captured by RGB-D sensor based on deep learning

ABSTRACT

RGB-D sensors are gradually introduced into the robotic system to help the machine understand its surroundings. Among the point cloud processing methods, instance segmentation of point cloud is extremely important since the quality of segmentation will affect the performance of subsequent algorithms. In this paper, the 3D reconstruction process of RGB-D sensor is analyzed, and a framework PointSeg is proposed to handle the instance segmentation of point cloud captured by RGB-D sensor. The PointSeg realizes point cloud instance segmentation by applying the deep learning method YOLACT++ to instance segment the color image first and then matching the instance information with the point cloud. In addition, an experimental platform that is equipped with a Kinect v2 is built, and a dataset is set up and then the PointSeg is tested on the dataset. The result shows that the PointSeg achieves point cloud instance segmentation according to the instance information extracted from color images, which not only has good real-time performance but also has better instance segmentation accuracy compared with the method of conducting instance segmentation directly on point cloud data, and the introduction of data augmentation in the training phase can achieve good training effect even on a small training dataset.

KEYWORDS:

• Point cloud instance segmentation
• RGB-D sensor
• deep learning

Acknowledgments

This work is financially supported by the National Natural Science Foundation of China (Grant No. 81827804), Science Fund for Creative Research Groups of National Natural Science Foundation of China (No. 51821093), and National Natural Science Foundation of China (Grant No. 51805477). We would also like to thank the Training Platform of Robots and Intelligent Manufacturing of Zhejiang University for providing our experimental equipment.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 81827804), Science Fund for Creative Research Groups of National Natural Science Foundation of China (No. 51821093), and National Natural Science Foundation of China (Grant No. 51805477).