Deep learning-supported machine vision-based hybrid system combining inhomogeneous 2D and 3D data for the identification of surface defects

References

• Ali, M. A., & Lun, A. K. (2019). A cascading fuzzy logic with image processing algorithm–based defect detection for automatic visual inspection of industrial cylindrical object’s surface. International Journal of Advanced Manufacturing Technology, 102(1–4), 81–28. https://doi.org/10.1007/s00170-018-3171-7
   Web of Science ®Google Scholar
• Arun, S., Huang, T. S., Blostein, S. D. (1987). Least-square fitting of two 3-D point sets. IEEE Transactions on Pattern Analysis & Machine Intelligence, 9(5), 698–700. https://doi.org/10.1109/TPAMI.1987.4767965
   PubMed Web of Science ®Google Scholar
• Badmos, O., Kopp, A., Bernthaler, T., & Schneider, G. (2020). Image-based defect detection in lithium-ion battery electrode using convolutional neural networks. Journal of Intelligent Manufacturing, 31(4), 885–897. https://doi.org/10.1007/s10845-019-01484-x
   Web of Science ®Google Scholar
• Bak, C., Roy, A. G., & Son, H. (2021). Quality prediction for aluminum diecasting process based on shallow neural network and data feature selection technique. CIRP Journal of Manufacturing Science and Technology, 33, 327–338. https://doi.org/10.1016/j.cirpj.2021.04.001
   Web of Science ®Google Scholar
• Ballard, D. H. (1981). Generalizing the Hough transform to detect arbitrary shapes. Pattern Recognition, 13(2), 111–122. https://doi.org/10.1016/0031-3203(81)90009-1
   Web of Science ®Google Scholar
• Bharambe, C., Jaybhaye, M. D., Dalmiya, A., Daund, C., & Shinde, D. (2023). Analyzing casting defects in high-pressure die casting industrial case study. Materials Today: Proceedings, 72, 1079–1083. https://doi.org/10.1016/j.matpr.2022.09.166
   Google Scholar
• Bulnes, F. G., Usamentiaga, R., Garcia, D. F., & Molleda, J. (2016). An efficient method for defect detection during the manufacturing of web materials. Journal of Intelligent Manufacturing, 27(2), 431–445. https://doi.org/10.1007/s10845-014-0876-9
   Web of Science ®Google Scholar
• Cavaliere, G., Borgianni, Y., & Rampone, E. (2022). Development of a system for the analysis of surface defects in die-cast components using machine vision. In D, T. Matt, R. Vidoni, E. Rauch, & P. Dallasega (Eds.), Managing and Implementing the Digital Transformation. ISIEA 2022 (Vol. 525, pp. 74–86). Springer, Cham: Lecture Notes in Networks and Systems. https://doi.org/10.1007/978-3-031-14317-5_7
   Google Scholar
• Cavaliere, G., Borgianni, Y., & Schäfer, C. (2021). Study on an in-line automated system for surface defect analysis of aluminium die-cast components using artificial intelligence. Acta Technica NapocenActa Technica Napocensis Series Applied Mathematics, Mechanics and Engineeringsis, 64(3), 475–486.
   Web of Science ®Google Scholar
• Chow, J. K., Su, Z., Wu, J., Tan, P. S., Mao, X., & Wang, Y. H. (2020). Anomaly detection of defects on concrete structures with the convolutional autoencoder. Advanced Engineering Informatics, 45, 101105. https://doi.org/10.1016/j.aei.2020.101105
   Web of Science ®Google Scholar
• DiLeo, G., Liguori, C., Pietrosanto, A., & Sommella, P. (2017). A vision system for the online quality monitoring of industrial manufacturing. Optics and Lasers in Engineering, 89, 162–168. https://doi.org/10.1016/j.optlaseng.2016.05.007
   Web of Science ®Google Scholar
• Dong, Z., Sun, X., Liu, W., & Yang, H. (2018). Measurement of free-form curved surfaces using laser triangulation. Sensors, 18(10), 3527. https://doi.org/10.3390/s18103527
   PubMed Web of Science ®Google Scholar
• Duan, L., Yang, K., & Ruan, L. (2020). Research on automatic recognition of casting defects based on deep learning. Institute of Electrical and Electronics Engineers Access, 9, 12209–12216. https://doi.org/10.1109/ACCESS.2020.3048432
   Google Scholar
• Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, 27.
   Google Scholar
• Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., & Chen, T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354–377. https://doi.org/10.1016/j.patcog.2017.10.013
   Web of Science ®Google Scholar
• Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., & Bennamoun, M. (2020). Deep learning for 3d point clouds: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43(12), 4338–4364. https://doi.org/10.1109/TPAMI.2020.3005434
   Web of Science ®Google Scholar
• Hatab, M., Malekmohamadi, H., & Amira, A. (2021). Surface defect detection using YOLO network. Intelligent Systems and Applications: Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 1, 3-4 September 2020, London, UK (pp. 505–515). Springer International Publishing.
   Google Scholar
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 20-25 June 2009, Miami, FL (pp. 770–778).
   Google Scholar
• Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. https://doi.org/10.1126/science.1127647
   PubMed Web of Science ®Google Scholar
• Jia, H., & Liu, W. (2023). Anomaly detection in images with shared autoencoders. Frontiers in Neurorobotics, 16, 1046867. https://doi.org/10.3389/fnbot.2022.1046867
   PubMed Web of Science ®Google Scholar
• Jones, C. W., & O’Connor, D. (2018). A hybrid 2D/3D inspection concept with smart routing optimisation for high throughput, high dynamic range and traceable critical dimension metrology. Measurement Science and Technology, 29(7), 074004. https://doi.org/10.1088/1361-6501/aababd
   Web of Science ®Google Scholar
• Jovančević, I., Pham, H. H., Orteu, J. J., Gilblas, R., Harvent, J., Maurice, X., & Brèthes, L. (2017). 3D point cloud analysis for detection and characterization of defects on airplane exterior surface. Journal of Nondestructive Evaluation, 36(4), 1–17. https://doi.org/10.1007/s10921-017-0453-1
   Web of Science ®Google Scholar
• Kaminsky, R. S., Snavely, N., Seitz, S. M., & Szeliski, R. (2009, June). Alignment of 3D point clouds to overhead images. 2009 IEEE computer society conference on computer vision and pattern recognition workshops, 20-25 June 2009, Miami, FL (pp. 63–70). IEEE.
   Google Scholar
• Kim, D. H., Kim, T. J., Wang, X., Kim, M., Quan, Y. J., Oh, J. W., Min, S. H., Kim, H., Bhandari, B., Yang, I., & Ahn, S. H. (2018). Smart machining process using machine learning: A review and perspective on machining industry. International Journal of Precision Engineering and Manufacturing-Green Technology, 5(4), 555–568. https://doi.org/10.1007/s40684-018-0057-y
   Web of Science ®Google Scholar
• Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., & Pietikäinen, M. (2020). Deep learning for generic object detection: A survey. International Journal of Computer Vision, 128(2), 261–318. https://doi.org/10.1007/s11263-019-01247-4
   Web of Science ®Google Scholar
• Liu, X., Zhou, Q., Zhao, J., Shen, H., & Xiong, X. (2019). Fault diagnosis of rotating machinery under noisy environment conditions based on a 1-D convolutional autoencoder and 1-D convolutional neural network. Sensors, 19(4), 972. https://doi.org/10.3390/s19040972
   PubMed Web of Science ®Google Scholar
• Malamas, E. N., Petrakis, E. G., Zervakis, M., Petit, L., & Legat, J.-D. (2003). A survey on industrial vision systems, applications and tools. Image and Vision Computing, 21(2), 171–188. https://doi.org/10.1016/S0262-8856(02)00152-X
   Web of Science ®Google Scholar
• Mery, D. (2020). Aluminum casting inspection using deep learning: A method based on convolutional neural networks. Journal of Nondestructive Evaluation, 39(1), 12. https://doi.org/10.1007/s10921-020-0655-9
   Web of Science ®Google Scholar
• Mullany, B., Savio, E., Haitjema, H., & Leach, R. (2022). The implication and evaluation of geometrical imperfections on manufactured surfaces. CIRP Annals, 71(2), 717–739. https://doi.org/10.1016/j.cirp.2022.05.004
   Web of Science ®Google Scholar
• Nikolić, F., Štajduhar, I., & Čanađija, M. (2023). Casting defects detection in aluminum alloys using deep learning: A classification approach. International Journal of Metalcasting, 17(1), 386–398. https://doi.org/10.1007/s40962-022-00777-x
   Web of Science ®Google Scholar
• Niu, T., Li, B., Li, W., Qiu, Y., & Niu, S. (2021). Positive-sample-based surface defect detection using memory-augmented adversarial autoencoders. IEEE/ASME Transactions on Mechatronics, 27(1), 46–57. https://doi.org/10.1109/TMECH.2021.3058147
   Web of Science ®Google Scholar
• Ördek, B., Borgianni, Y., & Coatanea, E. (2024). Machine learning-supported manufacturing: A review and directions for future research. Production and Manufacturing Research, 12(1), 2326526. https://doi.org/10.1080/21693277.2024.2326526
   Google Scholar
• Papananias, M., McLeay, T. E., Obajemu, O., Mahfouf, M., & Kadirkamanathan, V. (2020). Inspection by exception: A new machine learning-based approach for multistage manufacturing. Applied Soft Computing, 97, 106787. https://doi.org/10.1016/j.asoc.2020.106787
   Web of Science ®Google Scholar
• Parlak, İ. E., & Emel, E. (2023). Deep learning-based detection of aluminum casting defects and their types. Engineering Applications of Artificial Intelligence, 118, 105636. https://doi.org/10.1016/j.engappai.2022.105636
   Web of Science ®Google Scholar
• Penumuru, D. P., Muthuswamy, S., & Karumbu, P. (2019). Identification and classification of materials using machine vision and machine learning in the context of industry 4.0. Journal of Intelligent Manufacturing, 31(5), 1229–1241. https://doi.org/10.1007/s10845-019-01508-6
   Web of Science ®Google Scholar
• Pérez, L., Rodríguez, Í., Rodríguez, N., Usamentiaga, R., & García, D. F. (2016). Robot guidance using machine vision techniques in industrial environments: A comparative review. Sensors, 16(3), 335. https://doi.org/10.3390/s16030335
   PubMed Web of Science ®Google Scholar
• Prunella, M., Scardigno, R. M., Buongiorno, D., Brunetti, A., Longo, N., Carli, R., Dotoli, M., & Bevilacqua, V. (2023). Deep learning for automatic vision-based recognition of industrial surface defects: A survey. Institute of Electrical and Electronics Engineers Access, 11, 43370–43423. https://doi.org/10.1109/ACCESS.2023.3271748
   Google Scholar
• Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2017). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 21-26 July 2017, Honolulu, HI, USA (pp. 652–660).
   Google Scholar
• Qi, S., Yang, J., & Zhong, Z. (2020). A review on industrial surface defect detection based on deep learning technology. Proceedings of the 2020 3rd International Conference on Machine Learning and Machine Intelligence, 18-20 September 2020, Hangzhou, China (pp. 24–30).
   Google Scholar
• Rahimi, A., Anvaripour, M., & Hayat, K. (2021, June). Object detection using deep learning in a manufacturing plant to improve manual inspection. 2021 IEEE International Conference on Prognostics and Health Management (ICPHM), 7-9 June 2021, Detroit, MI (pp. 1–7). IEEE.
   Google Scholar
• Ren, Z., Fang, F., Yan, N., & Wu, Y. (2022). State of the art in defect detection based on machine vision. International Journal of Precision Engineering and Manufacturing-Green Technology, 9(2), 661–691. https://doi.org/10.1007/s40684-021-00343-6
   Web of Science ®Google Scholar
• Rong, M., & Shen, S. (2023). 3D Semantic segmentation of aerial photogrammetry models based on orthographic projection. IEEE Transactions on Circuits and Systems for Video Technology, 33(12), 7425–7437. https://doi.org/10.1109/TCSVT.2023.3273224
   Web of Science ®Google Scholar
• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
   Web of Science ®Google Scholar
• Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Müller, K. R. (2016). Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11), 2660–2673. https://doi.org/10.1109/TNNLS.2016.2599820
   Web of Science ®Google Scholar
• Savio, E., De Chiffre, L., Carmignato, S., & Meinertz, J. (2016). Economic benefits of metrology in manufacturing. CIRP Annals, 65(1), 495–498. https://doi.org/10.1016/j.cirp.2016.04.020
   Web of Science ®Google Scholar
• Schlegl, T., Seeböck, P., Waldstein, S. M., Langs, G., & Schmidt-Erfurth, U. (2019). f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54, 30–44. https://doi.org/10.1016/j.media.2019.01.010
   PubMed Web of Science ®Google Scholar
• Schöch, A., Salvadori, A., Germann, I., Balemi, S., Bach, C., Ghiotti, A., Carmignato, S., Maurizio, A. L., & Savio, E. (2015). High-speed measurement of complex shaped parts at elevated temperature by laser triangulation. International Journal of Automation Technology, 9(5), 558–566. https://doi.org/10.20965/ijat.2015.p0558
   Google Scholar
• Scime, L., & Beuth, J. (2018). A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process. Additive Manufacturing, 24, 273–286. https://doi.org/10.1016/j.addma.2018.09.034
   Web of Science ®Google Scholar
• Sharifzadeh, S., Biro, I., Lohse, N., & Kinnell, P. (2018). Abnormality detection strategies for surface inspection using robot mounted laser scanners. Mechatronics, 51, 59–74. https://doi.org/10.1016/j.mechatronics.2018.03.001
   Web of Science ®Google Scholar
• Silva, R. L., Rudek, M., Szejka, A. L., & Junior, O. C. (2018, July 2–4). Machine vision systems for industrial quality control inspections. Product lifecycle management to support Industry 4.0: 15th IFIP WG 5.1 International Conference, PLM 2018, Proceedings 15, July 2–4, 2018, Turin, Italy (pp. 631–641). Springer International Publishing.
   Google Scholar
• Song, K., & Yan, Y. (2013). A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Applied Surface Science, 285, 858–864. https://doi.org/10.1016/j.apsusc.2013.09.002
   Web of Science ®Google Scholar
• Steger, C., Ulrich, M., & Wiedemann, C. (2018). Machine vision algorithms and applications. John Wiley & Sons.
   Google Scholar
• Tan, X., & Triggs, B. (2007). Fusing Gabor and LBP feature sets for kernel based face recognition under difficult lighting conditions. Analysis and Modeling of faces and gestures. Lecture Notes in Computer Science, 4778, 168–182. https://doi.org/10.1007/978-3-540-75690-3_18
   Google Scholar
• Tang, B., Chen, L., Sun, W., & Lin, Z. K. (2023). Review of surface defect detection of steel products based on machine vision. IET Image Processing, 17(2), 303–322. https://doi.org/10.1049/ipr2.12647
   Web of Science ®Google Scholar
• Targ, S., Almeida, D., & Lyman, K. (2016). Resnet in resnet: Generalizing residual architectures. arXiv Preprint arXiv: 160308029.
   Google Scholar
• Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008, July). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine learning, 5-9 July 2008, Helsinki, Finland (pp. 1096–1103.
   Google Scholar
• Wang, J., Fu, P., & Gao, R. X. (2019). Machine vision intelligence for product defect inspection based on deep learning and Hough transform. Journal of Manufacturing Systems, 51, 52–60. https://doi.org/10.1016/j.jmsy.2019.03.002
   Web of Science ®Google Scholar
• Wang, T., Chen, Y., Qiao, M., & Snoussi, H. (2018). A fast and robust convolutional neural network-based defect detection model in product quality control. International Journal of Advanced Manufacturing Technology, 94(9–12), 3465–3471. https://doi.org/10.1007/s00170-017-0882-0
   Web of Science ®Google Scholar
• Wu, Y., Qin, Y., Wang, Z., & Jia, L. (2018). A UAV-based visual inspection method for rail surface defects. Applied Sciences, 8(7), 1028. https://doi.org/10.3390/app8071028
   Google Scholar
• Xu, H., & Huang, Z. (2021). Annotation-free defect detection for glasses based on convolutional auto-encoder with skip connections. Materials Letters, 299, 130065. https://doi.org/10.1016/j.matlet.2021.130065
   Web of Science ®Google Scholar
• Yan, Z., Shi, B., Sun, L., & Xiao, J. (2020). Surface defect detection of aluminum alloy welds with 3D depth image and 2D gray image. International Journal of Advanced Manufacturing Technology, 110(3–4), 741–752. https://doi.org/10.1007/s00170-020-05882-x
   Web of Science ®Google Scholar
• Ye, J., Stewart, E., & Roberts, C. (2019). Use of a 3D model to improve the performance of laser-based railway track inspection. Proceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail & Rapid Transit, 233(3), 337–355. https://doi.org/10.1177/0954409718795714
   Web of Science ®Google Scholar
• Yue, Y., Finley, T., Radlinski, F., & Joachims, T. (2007). A support vector method for optimizing average precision. Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 23-27 July 2007, Amsterdam, The Netherlands (pp. 271–278.
   Google Scholar
• Zheng, X., Zheng, S., Kong, Y., & Chen, J. (2021). Recent advances in surface defect inspection of industrial products using deep learning techniques. International Journal of Advanced Manufacturing Technology, 113(1–2), 35–58. https://doi.org/10.1007/s00170-021-06592-8
   Web of Science ®Google Scholar