Advanced search
Publication Cover
International Journal of Computer Integrated Manufacturing Volume 34, 2021 - Issue 9
Submit an article Journal homepage
856
Views
3
CrossRef citations to date
0
Altmetric
Research Article

Instance segmentation of point cloud captured by RGB-D sensor based on deep learning

Zhengtuo Wanga The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;b Key Laboratory of 3D Printing Process and Equipment of Zhejiang Province, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, ChinaView further author information
,
Yuetong Xua The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;b Key Laboratory of 3D Printing Process and Equipment of Zhejiang Province, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, ChinaCorrespondence[email protected]
View further author information
,
Jiongyan Yua The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;b Key Laboratory of 3D Printing Process and Equipment of Zhejiang Province, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;c Suzhou Zijingang Intelligent Manufacturing Equipment Co., Ltd., Suzhou, ChinaView further author information
,
Guanhua Xua The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;b Key Laboratory of 3D Printing Process and Equipment of Zhejiang Province, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, ChinaView further author information
,
Jianzhong Fua The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, China;b Key Laboratory of 3D Printing Process and Equipment of Zhejiang Province, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, ChinaView further author information
&
Tianyi Gua The State Key Laboratory of Fluid Power and Mechatronic Systems, Institute of Manufacturing Technology and Equipment Automation, College of Mechanical Engineering, Zhejiang University, Hangzhou, ChinaView further author information
Pages 950-963 | Received 24 Jul 2020, Accepted 19 May 2021, Published online: 02 Jul 2021

References

  • Andrianakos, G., N. Dimitropoulos, G. Michalos, and S. Makris. 2019. “An Approach for Monitoring the Execution of Human Based Assembly Operations Using Machine Learning.” Procedia CIRP 86: 198–203. doi:https://doi.org/10.1016/j.procir.2020.01.040.
  • Anh-Vu, V., L. Truong-Hong, D. F. Laefer, and M. Bertolotto. 2015. “Octree-Based Region Growing for Point Cloud Segmentation.” ISPRS Journal of Photogrammetry and Remote Sensing 104: 88–100. doi:https://doi.org/10.1016/j.isprsjprs.2015.01.011.
  • Barnard, S. T., and M. A. Fischler. 1982. “Computational Stereo.” Computational Stereo 14 (4): 553–572. doi:https://doi.org/10.1145/356893.356896.
  • Bolya, D. 2020. “YOLACT++: Better Real-Time Instance Segmentation.” IEEE Transactions on Pattern Analysis and Machine Intelligence. doi:https://doi.org/10.1109/TPAMI.2020.3014297.
  • Bolya, D., C. Zhou, F. Xiao, and Y. J. Lee. 2019. “YOLACT: Real-Time Instance Segmentation.” In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 9156–9165, Seoul, SOUTH KOREA. doi:10.1109/ICCV.2019.00925.
  • Caggiano, A., T. Segreto, and R. Teti. 2020. “Cloud Manufacturing Architecture for Part Quality Assessment.” Cogent Engineering 7 (1): 1715524. doi:https://doi.org/10.1080/23311916.2020.1715524. Edited by Duc Pham.
  • Chen, K., J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, and W. Feng. 2019. “Hybrid Task Cascade for Instance Segmentation.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4969–4978, Long Beach, CA. doi:10.1109/CVPR.2019.00511.
  • Chen, S. Y., Y. F. Li, and J. Zhang. 2008. “Vision Processing for Realtime 3-D Data Acquisition Based on Coded Structured Light.” IEEE Transactions on Image Processing 17 (2): 167–176. doi:https://doi.org/10.1109/TIP.2007.914755.
  • Cheng-Yang, F., M. Shvets, and A. C. Berg. 2019. “RetinaMask: Learning to Predict Masks Improves State-of-the-Art Single-Shot Detection for Free.” ArXiv Preprint ArXiv:1901.03353.
  • El-Sayed, E., R. F. Abdel-Kader, H. Nashaat, and M. Marei. 2018. “Plane Detection in 3D Point Cloud Using Octree-Balanced Density down-Sampling and Iterative Adaptive Plane Extraction.” IET Image Processing 12 (9): 1595–1605. doi:https://doi.org/10.1049/iet-ipr.2017.1076.
  • Feng, X., Y. Gui, S. Rong, and L. Ming. 2017. “An Adaptive Directional Filter for Photon Counting Lidar Point Cloud Data.” Journal of Infrared and Millimeter Waves 36 (1): 107–113. doi:https://doi.org/10.11972/j.1001-9014.2017.01.019.
  • Geetha, M., and R. Rakendu. 2014. “An Improved Method for Segmentation of Point Cloud Using Minimum Spanning Tree.” In 2014 International Conference on Communication and Signal Processing, 833–837, Melmaruvathur, INDIA. doi:10.1109/ICCSP.2014.6949960.
  • Gesto Diaz, M., F. Tombari, P. Rodriguez-Gonzalvez, and D. Gonzalez-Aguilera. 2015. “Analysis and Evaluation between the First and the Second Generation of RGB-D Sensors.” IEEE Sensors Journal 15 (11): 6507–6516. doi:https://doi.org/10.1109/JSEN.2015.2459139.
  • He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-Cnn.” In 2017 IEEE International Conference on Computer Vision (ICCV), 2980–2988, Venice, ITALY. doi:10.1109/ICCV.2017.322.
  • He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep Residual Learning for Image Recognition.” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778, Seattle, WA. doi:10.1109/CVPR.2016.90.
  • Huang, Z., L. Huang, Y. Gong, C. Huang, and X. Wang. 2019. “Mask Scoring R-Cnn.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6402–6411, Long Beach, CA. doi:10.1109/CVPR.2019.00657.
  • Kirsten, E., L. C. Inocencio, M. R. Veronez, L. G. Da Silveira, F. Bordin, and F. P. Marson. 2018. “3D Data Acquisition Using Stereo Camera.” In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, 9214–9217, Valencia, SPAIN. doi:10.1109/IGARSS.2018.8519568.
  • Kuan, Y. W., N. O. Ee, and L. S. Wei. 2019. “Comparative Study of Intel R200, Kinect V2, and Primesense RGB-D Sensors Performance Outdoors.” IEEE Sensors Journal 19 (19): 8741–8750. doi:https://doi.org/10.1109/JSEN.2019.2920976.
  • Lari, Z., and A. Habib. 2014. “An Adaptive Approach for the Segmentation and Extraction of Planar and Linear/Cylindrical Features from Laser Scanning Data.” ISPRS Journal of Photogrammetry and Remote Sensing 93: 192–212. doi:https://doi.org/10.1016/j.isprsjprs.2013.12.001.
  • Li, L., Y. Ke, and K. Jiang. 2009. “Surface Reconstruction Based on Computer Stereo Vision Using Structured Light Projection.” In 2009 International Conference on Intelligent Human-Machine Systems and Cybernetics, 2:451–454, Hangzhou, PEOPLES R CHINA. doi:10.1109/IHMSC.2009.235.
  • Li, Y., G. F. Tong, J. C. Yang, L. Q. Zhang, H. Peng, and H. S. Gao. 2019a. “3D Point Cloud Scene Data Acquisition and Its Key Technologies for Scene Understanding.” Laser & Optoelectronics Progress 56 (4): 21–34. doi:https://doi.org/10.3788/LOP56.040002.
  • Li, Y., Q. Haozhi, J. Dai, J. Xiangyang, and Y. Wei. 2017. “Fully Convolutional Instance-Aware Semantic Segmentation.” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4438–4446, Honolulu, HI. doi:10.1109/CVPR.2017.472.
  • Li, Y., Y. Wang, and Y. Xie. 2019. “Using Consecutive Point Clouds for Pose and Motion Estimation of Tumbling Non-Cooperative Target.” Advances in Space Research 63 (5): 1576–1587. doi:https://doi.org/10.1016/j.asr.2018.11.024.
  • Li, Y., G. Tong, D. Xiance, X. Yang, J. Zhang, and L. Yang. 2019b. “A Single Point-Based Multilevel Features Fusion and Pyramid Neighborhood Optimization Method for ALS Point Cloud Classification.” Applied Sciences 9 (5): 951. doi:https://doi.org/10.3390/app9050951.
  • Liang, Z., M. Yang, and C. Wang.  2020. “3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation.„ IEEE Robotics and Automation Letters, 5(3), 4915-4922. doi:https://doi.org/10.1109/LRA.2020.3004802.
  • Lim, K. B., and W. L. Kee. 2012. “Geometrical-Analysis-Based Algorithm for Stereo Matching of Single-Lens Binocular and Multi-Ocular Stereovision System.” Journal of Electronic Science and Technology 10 (02): 107–112. doi:https://doi.org/10.3969/j.1674-862X.2012.02.003.
  • Lin, T.-Y., P. Dollár, R. Girshick, H. Kaiming, B. Hariharan, and S. Belongie. 2017. “Feature Pyramid Networks for Object Detection.” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 936–944, Honolulu, HI. doi:10.1109/CVPR.2017.106.
  • Lin, Y., C. Wang, D. Zhai, L. Wei, and L. Jonathan. 2018. “Toward Better Boundary Preserved Supervoxel Segmentation for 3D Point Clouds.” ISPRS Journal of Photogrammetry and Remote Sensing 143: 39–47. doi:https://doi.org/10.1016/j.isprsjprs.2018.05.004.
  • Lin, Y.-J., R. R. Benziger, and A. Habib. 2016. “Planar-Based Adaptive down-Sampling of Point Clouds.” Photogrammetric Engineering and Remote Sensing 82 (5): 955–966. doi:https://doi.org/10.14358/PERS.82.12.955.
  • Liu, C., and Y. Furukawa. 2019. “Masc: Multi-Scale Affinity with Sparse Convolution for 3d Instance Segmentation.” ArXiv Preprint ArXiv:1902.04478.
  • Liu, L., G. Zhao, and B. Yuming. 2016. “Point Cloud Based Relative Pose Estimation of a Satellite in Close Range.” Sensors 16 (6): 824. doi:https://doi.org/10.3390/s16060824.
  • Liu, S., Q. Lu, H. Qin, J. Shi, and J. Jia. 2018a. “Path Aggregation Network for Instance Segmentation.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8759–8768, Salt Lake City, UT. doi:10.1109/CVPR.2018.00913.
  • Liu, S., D. Gao, P. Wang, X. Guo, X. Jing, and D.-X. Liu. 2018b. “A Depth-Based Weighted Point Cloud Registration for Indoor Scene.” Sensors 18 (11): 3608. doi:https://doi.org/10.3390/s18113608.
  • Liu, W., J. Sun, L. Wanyi, H. Ting, and P. Wang. 2019. “Deep Learning on Point Clouds and Its Application: A Survey.” Sensors 19 (19): 4188. doi:https://doi.org/10.3390/s19194188.
  • Narita, G., T. Seno, T. Ishikawa, and Y. Kaji. 2019. “PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things.” In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 4205–4212, Macau, PEOPLES R CHINA. doi:10.1109/IROS40897.2019.8967890.
  • Ozbay, E., and A. Cinar. 2019. “A Comparative Study of Object Classification Methods Using 3D Zernike Moment on 3D Point Clouds.” Traitement Du Signal 36 (6): 549–555. doi:https://doi.org/10.18280/ts.360610.
  • Segreto, T., A. Bottillo, R. Teti, L. M. Galantucci, F. Lavecchia, and M. B. Galantucci. 2017. “Non-Contact Reverse Engineering Modeling for Additive Manufacturing of down Scaled Cultural Artefacts.” Procedia CIRP 62: 481–486. doi:https://doi.org/10.1016/j.procir.2017.03.042.
  • Segreto, T., A. Caggiano, and M. D. Doriana. 2013. “Assessment of Laser-Based Reverse Engineering Systems for Tangible Cultural Heritage Conservation.” International Journal of Computer Integrated Manufacturing 26 (9): 857–865. doi:https://doi.org/10.1080/0951192X.2013.799781. Taylor & Francis.
  • Shijun, J., R. Yongcong, J. Zhao, L. Xiaolong, and G. Hong. 2017. “An Improved Method for Registration of Point Cloud.” Optik 140: 451–458. doi:https://doi.org/10.1016/j.ijleo.2017.01.041.
  • Stein, S. C., M. Schoeler, J. Papon, and F. Worgotter. 2014. “Object Partitioning Using Local Convexity.” In 2014 IEEE Conference on Computer Vision and Pattern Recognition, 304–311, Columbus, OH. doi:10.1109/CVPR.2014.46.
  • Tae, L. W., and C. E. Oestreich. 2019. “Model-Free Pose Estimation Using Point Cloud Data.” Acta Astronautica 165: 298–311. doi:https://doi.org/10.1016/j.actaastro.2019.09.007.
  • Wang, W., Y. Ronald, Q. Huang, and U. Neumann. 2018. “SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2569–2578, Salt Lake City, UT. doi:10.1109/CVPR.2018.00272.
  • Wang, X., T. Kong, C. Shen, Y. Jiang, and L. Lei. 2020. “SOLO: Segmenting Objects by Locations.”
  • Wang, X., S. Liu, X. Shen, C. Shen, and J. Jia. 2019. “Associatively Segmenting Instances and Semantics in Point Clouds.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4091–4100, Long Beach, CA. doi:10.1109/CVPR.2019.00422.
  • Wiedemeyer, T. 2015. “IAI Kinect2.” Institute for Artificial Intelligence, University Bremen. https://github.com/code-iai/iai_kinect2.
  • Won, J., T. Czerniawski, and F. Leite. 2020. “Semantic Segmentation of Point Clouds of Building Interiors with Deep Learning: Augmenting Training Datasets with Synthetic BIM-Based Point Clouds.” Automation in Construction 113: 103144. doi:https://doi.org/10.1016/j.autcon.2020.103144.
  • Xu, Y., Y. Zhen, R. Huang, L. Hoegner, and U. Stilla. 2020. “Robust Segmentation and Localization of Structural Planes from Photogrammetric Point Clouds in Construction Sites.” Automation in Construction 117: 103206. doi:https://doi.org/10.1016/j.autcon.2020.103206.
  • Ya-nan, W., W. Ting-feng, T. Yu-zhen, and S. Tao. 2017. “Improved Local Convexity Algorithm of Segmentation for 3D Point Cloud.” Chinese Optics 10 (3): 348–354. doi:https://doi.org/10.3788/CO.20171003.0348.
  • Yang, B., J. Wang, R. Clark, H. Qingyong, S. Wang, A. Markham, and N. Trigoni. 2019. “Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds.”
  • Yang, J., Z. Gan, L. Kun, and C. Hou. 2015. “Graph-Based Segmentation for RGB-D Data Using 3-D Geometry Enhanced Superpixels.” IEEE Transactions on Cybernetics 45 (5): 927–940. doi:https://doi.org/10.1109/TCYB.2014.2340032.
  • Yi, L., W. Zhao, H. Wang, M. Sung, and L. Guibas. 2019. “Gspn: Generative Shape Proposal Network for 3d Instance Segmentation in Point Cloud.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3942–3951, Long Beach, CA. doi:10.1109/CVPR.2019.00407.
  • Yun, D., S. Kim, H. Heo, and K. H. Ko. 2015. “Automated Registration of Multi-View Point Clouds Using Sphere Targets.” Advanced Engineering Informatics 29 (4): 930–939. doi:https://doi.org/10.1016/j.aei.2015.09.008.
  • Zhang, J., X. Lin, and X. Ning. 2013. “SVM-Based Classification of Segmented Airborne LiDAR Point Clouds in Urban Areas.” Remote Sensing 5 (8): 3749–3775. doi:https://doi.org/10.3390/rs5083749.
  • Zhong, F., R. Kumar, and C. Quan. 2019. “A Cost-Effective Single-Shot Structured Light System for 3D Shape Measurement.” IEEE Sensors Journal 19 (17): 7335–7346. doi:https://doi.org/10.1109/JSEN.2019.2915986.
  • Zhou, Q.-Y., J. Park, and V. Koltun. 2018. “Open3D: A Modern Library for 3D Data Processing.” ArXiv Preprint ArXiv:1801.09847.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Order Reprints Request Corporate Permissions

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

Request Academic Permissions

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.