A fast scene geometric modeling approach for digital twins combining neural rendering and model retrieval

References

• Agapaki, E., and I. Brilakis. 2022. “Geometric Digital Twinning of Industrial Facilities: Retrieval of Industrial Shapes.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2202.04834.
   Google Scholar
• Agrawal, A., M. Fischer, and V. Singh. 2022. “Digital Twin: From Concept to Practice.” Journal of Management in Engineering 38 (3): 06022001. https://doi.org/10.1061/(ASCE)ME.1943-5479.0001034.
   Web of Science ®Google Scholar
• Al-Ali, A. R., R. Gupta, T. Zaman Batool, T. Landolsi, F. Aloul, and A. Al Nabulsi. 2020. “Digital Twin Conceptual Model within the Context of Internet of Things.” Future Internet 12 (10): 163. https://doi.org/10.3390/fi12100163.
   Web of Science ®Google Scholar
• Basamakis, F. P., A. Christos Bavelos, D. Dimosthenopoulos, A. Papavasileiou, and S. Makris. 2022. “Deep Object Detection Framework for Automated Quality Inspection in Assembly Operations.” Procedia CIRP 115:166–171. https://doi.org/10.1016/j.procir.2022.10.068.
   Google Scholar
• Botín-Sanabria, D. M., A.-S. Mihaita, R. E. Peimbert-García, M. A. Ramírez-Moreno, R. A. Ramírez-Mendoza, and J. de Lozoya-Santos. 2022. “Digital Twin Technology Challenges and Applications: A Comprehensive Review.” Remote Sensing 14 (6): 1335. https://doi.org/10.3390/rs14061335.
   Web of Science ®Google Scholar
• Chen, T., S. Kornblith, M. Norouzi, and G. Hinton. 2020. “A Simple Framework for Contrastive Learning of Visual Representations.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2002.05709.
   Google Scholar
• Dong, Z., B. Yang, Y. Liu, F. Liang, B. Li, and Y. Zang. 2017. “A Novel Binary Shape Context for 3D Local Surface Description.” ISPRS Journal of Photogrammetry and Remote Sensing 130:431–452. https://doi.org/10.1016/j.isprsjprs.2017.06.012.
   Web of Science ®Google Scholar
• Gao, K., Y. Gao, H. He, D. Lu, L. Xu, and J. Li. 2022. “NeRF: Neural Radiance Field in 3D Vision, a Comprehensive Review.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2210.00379.
   Google Scholar
• Gupta, S., S. Modgil, and A. Gunasekaran. 2020. “Big Data in Lean Six Sigma: A Review and Further Research Directions.” International Journal of Production Research 58 (3): 947–969. https://doi.org/10.1080/00207543.2019.1598599.
   Web of Science ®Google Scholar
• Hadsell, R., S. Chopra, and Y. LeCun. 2006. “Dimensionality Reduction by Learning an Invariant Mapping.” 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA, 1735–1742. https://doi.org/10.1109/CVPR.2006.100.
   Google Scholar
• Hu, J., F. Xiao, Q. Jin, G. Zhao, and P. Lou. 2023. “Synthetic Data Generation Based on RDB-CycleGAN for Industrial Object Detection.” Mathematics 11 (22): 4588. https://doi.org/10.3390/math11224588.
   Web of Science ®Google Scholar
• Ivanov, S., K. Nikolskaya, G. Radchenko, L. Sokolinsky, and M. Zymbler. 2020. “Digital Twin of City: Concept Overview.” 2020 Global Smart Industry Conference (GloSIC), Chelyabinsk, Russia, 178–186. https://doi.org/10.1109/GloSIC50886.2020.9267879.
   Google Scholar
• Järvelin, K., and J. Kekäläinen. 2002. “Cumulated Gain-Based Evaluation of IR Techniques.” ACM Transactions on Information Systems 20 (4): 422–446. https://doi.org/10.1145/582415.582418.
   Web of Science ®Google Scholar
• Jin, J., H. Xu, and B. Leng. 2022. “Adaptive Points Sampling for Implicit Field Reconstruction of Industrial Digital Twin.” Sensors 22 (17): 6630. https://doi.org/10.3390/s22176630.
   PubMed Web of Science ®Google Scholar
• Kazhdan, M., T. Funkhouser, and S. Rusinkiewicz. 2003. “Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors.” Proceedings of the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, SGP ’03, 156–164. Goslar, DEU. Eurographics Association.
   Google Scholar
• Kosiorek, A. R., H. Strathmann, D. Zoran, P. Moreno, R. Schneider, S. Mokrá, and D. J. Rezende. 2021. “NeRF-VAE: A Geometry Aware 3D Scene Generative Model.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2104.00587.
   Google Scholar
• Lee, J., H. Son, C. Kim, and C. Kim. 2013. “Skeleton-Based 3D Reconstruction of As-Built Pipelines from Laser-Scan Data.” Automation in Construction 35:199–207. https://doi.org/10.1016/j.autcon.2013.05.009.
   Web of Science ®Google Scholar
• Li, G., M. Müller, A. Thabet, and B. Ghanem. 2019. “DeepGcns: Can GCNs Go As Deep As CNNs?” ArXiv Preprint. https://doi.org/10.48550/arXiv.1904.03751.
   Google Scholar
• Liu, S., Y. Lu, J. Li, D. Song, X. Sun, and J. Bao. 2021. “Multi-Scale Evolution Mechanism and Knowledge Construction of a Digital Twin Mimic Model.” Robotics and Computer-Integrated Manufacturing 71:102123. https://doi.org/10.1016/j.rcim.2021.102123.
   Web of Science ®Google Scholar
• Long, L., Y. Xia, M. Yang, B. Wang, and Y. Pan. 2022. “Retrieval of a 3D CAD Model of a Transformer Substation Based on Point Cloud Data.” Automation 3 (4): 563–578. https://doi.org/10.3390/automation3040028.
   Google Scholar
• Lu, Q., L. Chen, S. Li, and M. Pitt. 2020. “Semi-Automatic Geometric Digital Twinning for Existing Buildings Based on Images and CAD Drawings.” Automation in Construction 115:103183. https://doi.org/10.1016/j.autcon.2020.103183.
   Web of Science ®Google Scholar
• Manettas, C., N. Nikolakis, and K. Alexopoulos. 2021. “Synthetic Datasets for Deep Learning in Computer-Vision Assisted Tasks in Manufacturing.” Procedia CIRP 103:237–242. https://doi.org/10.1016/j.procir.2021.10.038.
   Google Scholar
• Meng, X., W. Chen, and B. Yang. 2023. “NeAT: Learning Neural Implicit Surfaces with Arbitrary Topologies from Multi-View Images.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2303.12012.
   Google Scholar
• Mirzaei, A., T. Aumentado-Armstrong, K. G. Derpanis, J. Kelly, M. A. Brubaker, I. Gilitschenski, and A. Levinshtein. 2022. “SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2211.12254.
   Google Scholar
• Nie, W.-Z., M.-J. Ren, A.-A. Liu, Z. Mao, and J. Nie. 2021. “M-GCN: Multi-Branch Graph Convolution Network for 2D Image-Based on 3D Model Retrieval.” IEEE Transactions on Multimedia 23:1962–1976. https://doi.org/10.1109/TMM.2020.3006371.
   Web of Science ®Google Scholar
• Oechsle, M., S. Peng, and A. Geiger. 2021. “UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2104.10078.
   Google Scholar
• Pomerleau, F., F. Colas, and R. Siegwart. 2015. “A Review of Point Cloud Registration Algorithms for Mobile Robotics.” Foundations and Trends in Robotics 4 (1): 1–104. https://doi.org/10.1561/2300000035.
   Google Scholar
• Qin, F., S. Qiu, S. Gao, and J. Bai. 2022. “3D CAD Model Retrieval Based on Sketch and Unsupervised Variational Autoencoder.” Advanced Engineering Informatics 51:101427. https://doi.org/10.1016/j.aei.2021.101427.
   Web of Science ®Google Scholar
• Qu, W., J. Li, R. Zhang, S. Liu, and J. Bao. 2023. “Adaptive Planning of Human– Robot Collaborative Disassembly for End-Of-Life Lithium-Ion Batteries Based on Digital Twin.” Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02081-9.
   Web of Science ®Google Scholar
• Raj, A., G. Dwivedi, A. Sharma, A. B. Lopes de Sousa Jabbour, and S. Rajak. 2020. “Barriers to the Adoption of Industry 4.0 Technologies in the Manufacturing Sector: An Inter-Country Comparative Perspective.” International Journal of Production Economics 224:107546. https://doi.org/10.1016/j.ijpe.2019.107546.
   Web of Science ®Google Scholar
• Riveiro, B., H. González-Jorge, J. Martínez-Sánchez, L. Díaz-Vilariño, and P. Arias. 2015. “Automatic Detection of Zebra Crossings from Mobile LiDAR Data.” Optics & Laser Technology 70:63–70. https://doi.org/10.1016/j.optlastec.2015.01.011.
   Web of Science ®Google Scholar
• Ruan, Y., H. H. Lee, Y. Zhang, K. Zhang, & A. X. Chang. 2024. TriCoLo: Trimodal Contrastive Loss for Text To Shape Retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Hawaii, USA, 5815–5825.
   Google Scholar
• Salihu, D., A. Misik, M. Hofbauer, and E. Steinbach. 2022. “S2CMAF: Multi-Method Assessment Fusion for Scan-To-CAD Methods.” 2022 IEEE International Symposium on Multimedia (ISM), Italy, 129–136. https://doi.org/10.1109/ISM55400.2022.00026.
   Google Scholar
• Schroff, F., D. Kalenichenko, and J. Philbin. 2015. “FaceNet: A Unified Embedding for Face Recognition and Clustering.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 815–823. https://doi.org/10.1109/CVPR.2015.7298682.
   Google Scholar
• Semeraro, C., M. Lezoche, H. Panetto, and M. Dassisti. 2021. “Digital Twin Paradigm: A Systematic Literature Review.” Computers in Industry 130:103469. https://doi.org/10.1016/j.compind.2021.103469.
   Web of Science ®Google Scholar
• Sjarov, M., T. Lechler, J. Fuchs, M. Brossog, A. Selmaier, F. Faltus, T. Donhauser, and J. Franke. 2020. “The Digital Twin Concept in Industry – a Review and Systematization.” 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vienna, Austria, 1789–1796. https://doi.org/10.1109/ETFA46521.2020.9212089.
   Google Scholar
• Sommer, M., S. Stobrawa, and M. von Soden. 2019. “Automatic Generation of Digital Twin Based on Scanning and Object Recognition.” Transdisciplinary Engineering for Complex Socio-Technical Systems: 645–654. https://doi.org/10.3233/ATDE190174.
   Google Scholar
• Song, J., S. Liu, T. Ma, Y. Sun, F. Tao, and J. Bao. 2023. “Resilient Digital Twin Modeling: A Transferable Approach.” Advanced Engineering Informatics 58:102148. https://doi.org/10.1016/j.aei.2023.102148.
   Web of Science ®Google Scholar
• Su, H., S. Maji, E. Kalogerakis, and E. Learned-Miller. 2015. “Multi-View Convolutional Neural Networks for 3D Shape Recognition.” ArXiv Preprint. https://doi.org/10.48550/arXiv.1505.00880.
   Google Scholar
• Tao, F., and Q. Qi. 2019. “Make More Digital Twins.” Nature 573 (7775): 490–491. https://doi.org/10.1038/d41586-019-02849-1.
   PubMed Web of Science ®Google Scholar
• Tao, F., X. Sun, J. Cheng, Y. Zhu, W. Liu, Y. Wang, H. Xu, T. Hu, X. Liu, T. Liu, Z. Sun, J. Xu, J. Bao, F. Xiang, and X. Jin. 2023. “makeTwin: A Reference Architecture for Digital Twin Software Platform.” Chinese Journal of Aeronautics 37 (1): 1–18. https://doi.org/10.1016/j.cja.2023.05.002.
   Web of Science ®Google Scholar
• Tao, F., B. Xiao, Q. Qi, J. Cheng, and P. Ji. 2022. “Digital Twin Modeling.” Journal of Manufacturing Systems 64:372–389. https://doi.org/10.1016/j.jmsy.2022.06.015.
   Web of Science ®Google Scholar
• VanDerhorn, E., and S. Mahadevan. 2021. “Digital Twin: Generalization, Characterization and Implementation.” Decision Support Systems 145:113524. https://doi.org/10.1016/j.dss.2021.113524.
   Web of Science ®Google Scholar
• Wang, C., M. Cheng, F. Sohel, M. Bennamoun, and J. Li. 2019. “NormalNet: A Voxel-Based CNN for 3D Object Classification and Retrieval.” Neurocomputing 323:139–147. https://doi.org/10.1016/j.neucom.2018.09.075.
   Web of Science ®Google Scholar
• Wang, P., L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. 2021. “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-View Reconstruction.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2106.10689.
   Google Scholar
• Wei, J., L. Hu, C. Wang, and L. Kneip. 2022. “Accurate Instance-Level CAD Model Retrieval in a Large-Scale Database.” 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 9879–9885. https://doi.org/10.1109/IROS47612.2022.9981296.
   Google Scholar
• Xie, C., C. Xia, M. Ma, Z. Zhao, X. Chen, and J. Li. 2022. “Pyramid Grafting Network for One-Stage High Resolution Saliency Detection.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2204.05041.
   Google Scholar
• Yuhao, Z., H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. 2022. “Contrastive Learning of Medical Visual Representations from Paired Images and Text.” ArXiv Preprint. https://doi.org/10.48550/arXiv.2010.00747.
   Google Scholar
• Zhang, C., and Y. Chen. 2020. “A Review of Research Relevant to the Emerging Industry Trends: Industry 4.0, IoT, Blockchain, and Business Analytics.” Journal of Industrial Integration and Management 5 (1): 165–180. https://doi.org/10.1142/S2424862219500192.
   Google Scholar
• Zheng, H., T. Liu, J. Liu, and J. Bao. 2023. “Visual Analytics for Digital Twins: A Conceptual Framework and Case Study.” Journal of Intelligent Manufacturing 35 (4): 1671–1686. https://doi.org/10.1007/s10845-023-02135-y.
   Web of Science ®Google Scholar
• Zhuang, C., J. Liu, and H. Xiong. 2018. “Digital Twin-Based Smart Production Management and Control Framework for the Complex Product Assembly Shop-Floor.” The International Journal of Advanced Manufacturing Technology 96 (1): 1149–1163. https://doi.org/10.1007/s00170-018-1617-6.
   Web of Science ®Google Scholar
• Zou, G., J. Hua, Z. Lai, X. Gu, and M. Dong. 2009. “Intrinsic Geometric Scale Space by Shape Diffusion.” IEEE Transactions on Visualization and Computer Graphics 15 (6): 1193–1200. https://doi.org/10.1109/TVCG.2009.159.
   PubMed Web of Science ®Google Scholar