Application of machine learning algorithm in the sheet metal industry: an exploratory case study

ABSTRACT

This study solved a practical problem in a case in the sheet metal industry using machine learning and deep learning algorithms. The problem in the case company was related to detecting the minimum gaps between components, which were produced after the punching operation of a metal sheet. Due to the narrow gaps between the components, an automated sheer machine could not grip the rest of the sheet skeleton properly after the punching operation. This resulted in some of the scraped sheet on the worktable being left behind, which needed a human operator to intervene. This caused an extra trigger to the production line that resulted in a break in production. To solve this critical problem, the relevant images of the components and the gaps between them were analyzed using machine learning and deep learning techniques. The outcome of this study contributed to eliminating the production bottleneck by optimizing the gaps between the punched components. This optimization process facilitated the easy and safe movement of the gripper machine and contributed to minimizing the sheet waste.

KEYWORDS:

• Machine learning
• deep learning
• detection of gaps
• sheet metal industry
• case study

1. Introduction

The advance of technological trends has driven the rapid diffusion of machine learning across industries (Sarkar et al. 2016; O’Donovan et al. 2019; Gottsegan, 2020; Cakir, Guvenc, and Mistikoglu 2021). Today’s growth of technological, societal, and competitive pressures is pushing companies to transform and innovate. Due to the rapid progress of machine-learning technologies with their added benefits, organizational managers at all levels need to familiarize themselves with these techniques. Such organizational changes contribute to ensure creating maximum value for companies. However, there is a common challenge to the usability of machine-learning methods in companies, where a trade-off between interpretability and accuracy is needed (Srdoc et al., 2007; Sarkar et al. 2016). It is important to understand the decisions made by machine-learning-based techniques, which should also be transparent and interpretable.

Machine learning is a technique that is concerned with enabling computer programs automatically with the objective of improving the performance of some tasks through experience. The application of machine learning in the area of manufacturing can be very fruitful (Sung 2018; Sharp, Ak, and Hedberg 2018; Dutta, Kaliannan, and Subramaniam 2021). In today’s business, the application of machine learning is considered to be one of the most disruptive innovations and can be a strong enabler of competitive advantage (Ren 2021; Cakir, Guvenc, and Mistikoglu 2021). The application of machine learning has been around for more than 60 years; however, only in recent time has it showed significant potential for enhancing economies and societies (Lee and Shin 2020).

There are various applications of machine learning methods that have been applied in different industrial sectors in recent years. These methods have been applied in various industrial segments such as healthcare, banking, manufacturing, transportation, etc (Li, Wang, and Wang 2019; Xie 2020; Gupta et al. 2020; Danton et al. 2020). Studies have shown that the adoption of machine learning algorithms can contribute to reducing costs by between 20%–25% across banking, IT operations, infrastructure, and maintenance, improve customer retention and acquisition, etc. (Accenture, 2018). By transforming business processes into intelligent and automated processes, companies can more effectively utilize their resources for higher-value activities and be able to offer better products and services to their customers.

This study is focused on maintaining the punching operation of a sheet metal company in Finland. This company operates on a global scale to deliver customized components produced from sheets. It currently has problems in sheet material handling after the necessary punching operations. The company has noticed that frequently after punching operations waste sheets are stuck over the worktable because the automated grippers fail to grip the sheets completely and human operators are needed to remove the remaining parts of the sheets and put them into trash bins. This often creates a halt in production and thus a loss in production for the company. This problem occurs mostly due to too narrow gaps being produced between the components following the punching operation. To overcome this problem, machine learning and deep learning algorithms were used to study the gaps between components with the objective of optimizing the gaps.

The rest of the article is organized as follows: Section 2 comprises a literature review on machine learning, and the methodology of the study is presented in Section 3. A description of the case company follows in in Section 4, and the overall study results are analyzed and presented in Section 5. Various practical outcomes are elaborated in terms of managerial implications at Section 6. The study concludes in Section 7 by presenting its contributions and limitations its and suggestions for future research.

2. Literature review

2.1 Digitalization of industries and machine learning

Digitalization means the incorporation of digital technology into all aspects of everyday life of industrial communities. Today’s digital era emphasizes the importance of integrating digital transformation in industries in order to improve the overall efficiency of the business processes. The digital transformation of business processes provides an unprecedented flow of high-quality information to industries by using automated real-time data analytics and process analysis. Because of the digitalization of manufacturing industries, it is essential nowadays to introduce digital tools and technologies into this segment. This digitalization process results in an enormous amount of valuable data, which needs to be analyzed in order to improve production processes and to detect the causes of problems. In this environment, machine learning and deep learning can be useful in saving energy, time, and resources, and avoiding waste (Tsai and Chang 2018; Weichert et al. 2019). Machine learning, which is a branch of artificial intelligence, is used to progressively enhance the performance of tasks based on big data collected in the digitalized world (Bianconi et al. 2014; Li, Wang, and Wang 2019; Carbo-Valverde, Cuadros-Solas, and Rodríguez-Fernández 2020).

It is a new go-to strategy to add value in business processes through the digitalization of workflows using machine learning and advanced analytics (Li, Wang, and Wang 2019; Hajizadeh 2019; Kang, Catal, and Tekinerdogan 2020). Companies are striving to embrace the new digital technologies, but find it challenging to put their models into production in order to achieve tangible outputs and gain favorable benefits. Due to the rapid growth in digitalization, machine-learning algorithms offer high potential for the business process optimization domain. Nowadays, machine-learning algorithms are widely implemented in the production environment (Zhang, Wang, and Gao 2019). The application of machine learning is encouraged by industrial communities due to its additional capabilities to save on resources, machining time and energy, and increase yield in areas where traditional methods such as six sigma strategies have reached their limits (Köksal, Batmaz, and Testik 2011; Golkarnareji et al., 2019; Ren 2021; Xu et al., 2021).

The emergence of digitalization in manufacturing processes has attracted industries and evolved over recent decades. The traditional concept of computer-integrated manufacturing is transferred to digital manufacturing, where the technologies and tools facilitate the integration of product and process design before starting actual production and support the ramp-up phases . In digital manufacturing, particular emphasis is given to the optimization of networked production facilities, where real time data is necessary for the decision-making process. Machine learning, which is a diverse field of artificial intelligence, has the ability to automatically learn from data and make predictions based on data. This technique has been widely used in digital manufacturing for various purposes, such as predictive maintenance (Joseph et al. 2014; Dalzochio et al., 2020), demand forecasting (Huber and Stuckenschmidt 2020; Jayant, Agarwal, and Gupta 2021) to process monitoring (Cakir, Guvenc, and Mistikoglu 2021), and optimization (Weichert et al. 2019; Min et al. 2019).

The term machine learning describes algorithms used to identify and extract valuable data patterns to improve quality. The implementation of machine learning is not a new field. However, with the recent growth of computational power, many industries have been considering employing artificial intelligence solutions to improve production processes (Weichert et al. 2019; Ademujimi, Brundage, and Prabhu 2017). The development of machine learning has been very successful at tackling complex tasks (Gottsegan, 2020). It is also noticeable that although machine learning and many deep-learning methods are known to create highly accurate outcomes, they often lack interpretability owing to their black-box nature (Lee and Shin 2020). It is therefore critical, and often challenging, to identify the most appropriate machine-learning method for specific applications. There are other challenges also for organizational managers when deploying machine learning, such as ethical issues, a shortage of machine-learning engineers, and issues of data-quality cost-benefit (Lee and Shin 2020).

Although machine learning provides new opportunities in terms of supporting a data-rich digital manufacturing environment, it often gives rise to additional challenges. The machine learning algorithm requires compliance with defined standards, which is required for data science and optimization techniques (Parkes, Özcan, and Karapetyan 2015). In addition, there is also a challenge in finding expertise related to data science and optimization areas, which requires an inherently different skill set (Baumers and Özcan 2016). Moreover, there are challenges to implementing machine learning such as the selection of the right algorithm, the right set of data, data processing, data labelling, etc. Furthermore, there are several other challenges during the implementation process such as managing the model version, redeveloping the models, managing data versions, etc. (Gupta 2017). Problems associated with machine learning methodology need to be addressed at regular intervals.

2.2 Application of machine learning in various industrial segments

Machine learning methodology is applied in various fields of research work. Yeh and Deng (Yeh and Deng 2012) applied machine-learning methodology in cost estimation of the product life cycle. They used two machine-learning methods, namely back-propagation neural networks (BPNs) and least squares support vector machines (LS-SVMs) in order to solve product life cycle cost estimation problems. Carbonneau, Laframboise, and Vahidov (2008) used advanced machine learning to investigate the applicability of forecasting distorted demand at the end of a supply chain (bullwhip effect). The authors compared this method with other available methods, such as naïve forecasting, trend, moving average, and linear regression, and showed that the machine learning method produced a better performance than the regression model. Machine learning-based algorithms have also been applied to manage the human resource systems of intelligent manufacturing industry. The application of machine learning in human resource management in enterprises has been investigated by matching, screening and filtering the user’s job characteristics and the employee’s requirements with the objective of improving the effective pursuit of the most suitable goals for both users and enterprises (Xie 2020; Garg et al. 2021).

The application of machine learning algorithms can significantly improve the operation and development of manufacturing industries (Sung 2018). The automotive industry is a leading user of machine learning to assess and minimize risk. Such risk assessments are conducted by identifying the risks first, which is successfully performed by adopting machine-learning algorithms (Hegde and Rokseth 2020). Financial businesses such as banks adopt machine learning in order to analyze various transaction scenarios, to enable the processing of large data sets faster, and to mak accurate forecasts for various automated trading functionalities (Lee 2017: Rizzi, Wang, and Zielinski 2020). Weichert et al. (2019) reviewed the application of machine learning in industries to optimize methods on the shop floor in order to improve production processes in the manufacturing industry. Additionally, they also discussed how the machine-learning approach used in industries saves energy, time, and resources, and avoids waste.

In the case of service industry, machine-learning algorithms have also been successfully applied. applied machine learning in the healthcare industry to identify ethical considerations, while Gupta and Sedamkar (2020) deployed machine learning to correctly identify the percentage of sick versus healthy people. For the tourism and hospitality industry, machine learning has shown significant potential. Sun et al. (2019) used a machine learning algorithm to forecast the number of tourists arriving in a certain period. They proposed a forecasting framework that applies a combination of machine learning and internet search, with the aim of improving the overall forecasting performance with respect to both forecasting accuracy and robustness analysis. Sanchez-Medina and C-Sanchez (2020) used machine learning to successfully forecast hotel booking cancellations. Other areas of machine learning applications can be seen in transportation (Huang and Zhu 2021), flood prediction (Motta et al., 2021), telecommunications (de Andres, 2020), smart city (Zekić-Sušac, Mitrović, and Has 2021), amongst others.

Machine learning algorithms are used in industries for various purposes such as clustering, classification, forecasting, etc. In the case of clustering, objects are grouped in order to reveal patterns, such as feedback from groups of customers to improve customer satisfaction. Unsupervised machine learning is used in clustering due to the class labels of some objects not being known beforehand (Najafabadi, 2017). For classification processes, all the observations are already categorized or classed for purposes of training and testing. Machine learning is helpful in classifying the business operations. Chandran (2018) studied a brewing company using a machine-learning classifier to determine better routes for its drivers. Musani (2018) deployed a machine learning based freshness algorithm to prioritize the flow of perishable goods worldwide. In addition to clustering and classification, machine learning is also deployed to identify data patterns that are used to predict future events. For instance, machine learning can be used to scrutinize data to detect market signals that will affect future market performance (Chandran 2018). The JP Morgan trading team has developed a proprietary machine-learning model in order to find the best execution strategy for trading orders (McDowell 2018). Table 1 highlights the application of machine learning in some important industrial sectors.

Table 1. Application of machine learning algorithms in various industrial sectors

Serial number	Field of application	Specific contribution	Reference
1	Architectural design	Machine learning algorithm is used to assess two cases of architectural practices with performance-based design, fabrication and learning.	Tamke et al. (2018)
2	Steel industry	Artificial intelligence and machine learning are applied to develop plant coordination, control, raw materials, energy optimization and quality management.	Backman, Kyllönen, and Helaakoski (2019)
3	Process industry	Review of future research on data mining and analytics conducted for the benefit of process industry.	Ge et al. (2017)
4	Food industry	Machine learning was implemented to predict deviations in food production, reducing uncertainties and minimizing the amount of waste of food. It also works as a tool to help to identify possible production anomalies.	Garre, Ruiz, and Hontoria (2020)
5	Construction industry	Machine learning algorithm was applied to execute a successful contract between stakeholders. It also revealed the factors influencing the possibility of contractual default and tried to define corrective actions for a customer.	Valpeters et al. (2018)
6	Industrial pumps	Anomalies among industrial pumps are detected by machine learning algorithm that supports condition monitoring and prevents pump failure.	Dutta, Kaliannan, and Subramaniam (2021)
7	Machine industry	The possibility of utilizing machine learning in machining processes is reviewed in order to improve product quality levels and productivity rates and to optimize design and process parameters.	Kim et al. (2018)
8	Petrochemical industry	A framework based on digital twin and machine-learning approach is presented to support petrochemical and other process manufacturing industries to dynamically adapt to the changing environment and to respond in a timely manner to improve economic benefits.	Min et al. (2019)
9	Mineral processing	A thorough review on the applicability of machine learning in mineral processing with respect to success level, area of application and major problems is presented with a view to equipping researchers and industrial practitioners with structured knowledge.	McCoy and Auret (2019)
10	Construction industry	This research study applies machine-learning technique to analyze 16 critical factors with the aim of assessing the impact of the identified factors in predicting the severity of construction accidents.	Zhu et al. (2020)
11	Oil and gas industry	A machine-learning algorithm was applied to cluster oilfields which are very similar, which contributed to significantly reducing the engineering effort and operator involvement in developing the models for each well.	Patel and Patwardhan (2018)
12	Textile industry	In addition to machine learning algorithms, data mining studies, including classification and clustering techniques, are implemented in the textile industry to deal with different problems where traditional methods are not useful.	Yildirim, Birant, and Alpyildiz (2018)
13	Power industry	Machine learning methods are applied to select the most effective management actions (preventive measures) and to prevent a critical situation from developing into an emergency in a power system.	Massel et al. (2019)
14	Healthcare industry	In order to reduce healthcare cost and to improve patient safety and healthcare quality, machine learning can be used as an important tool or technique.	Nithya and Ilango (2017)
15	Pottery industry	A number of unsupervised and supervised machine learning methods are applied to test the applicability of defining geochemical fingerprints to track or attest the provenance of samples.	Anglisano et al. (2020)

2.3 Application of machine learning in sheet metal industries

There are only a few applications of machine learning in sheet metal industries. Machine learning and deep learning algorithms are successfully applied to sheet metal industries to select a suitable manufacturing process and to achieve the final geometry of a metal part that is unstructured and heavily reliant on human expertise (Hamouche and Loukaides 2018; Chiu, Tsai, and T-l 2020). Stoerke et al. (2016) proposed a new methodology to increase the geometric accuracy forming of sheet metal work pieces. They proposed the use of a machine learning model, which applies reinforcement learning as a flexible and promising solution to increase the geometric accuracy in incremental sheet forming processes.

Machine learning is also used to predict the defects of sheet metal forming processes (Deb, Ribeiro, and Prates 2018; Tsai and Chang 2018). Kwitek (2016) demonstrated how sheet metal fabrication machines can use machine learning in order to gain competitive advantage. He used machine learning to focus on predictive maintenance, which contributed to improving operations management. Hamouche and Loukaides (2018) applied machine learning to sheet metal forming, which is considered a critical component of modern manufacturing. In their study, a machine learning approach was used for the first time with the objective of identifying the manufacturing process, which formed a part solely from the final geometry. By implementing a mapping of the mean and Gaussian curvatures through machine learning, a high accuracy rate was established that automated the operational processes from the initial stage as design to manufacture, thus eliminating the requirement for human experts in matching each product with a suitable forming method.

Kashid and Kumar 2012) performed a review of the applications of artificial neural networks to sheet metal work. Zwierzycki, Nicholas, and Thomsen (2018) applied pre-process supervised machine learning to predict and improve sheet-forming tolerance and generate corrected fabrication models. Lin and Chang 1995) proposed a model using machine learning from neural networks in an expert system of sheet metal bending tooling. In this study, the authors developed a learning model with conditional attributes to develop an expert system of sheet metal bending tooling selection. Wu et al. (1999) deployed a combination of artificial neural network and machine learning methods for investigating surface defects in sheet metal forming.

2.4 Related challenges to implementing machine-learning methods in industries

There are several challenges in deploying a machine-learning approach in industries, such as data privacy, data accessibility, and data sharing. In addition, there are other related challenges such as the quality of data, ethical issues, shortage of experts, cost benefit analysis etc. In terms of ethical challenges, the application of machine learning is concerned with data privacy and protection rules (e.g. personal medical record data). During algorithm development for machine learning, training data is needed, which may exhibit an inevitable bias if there are any latent biases in that data (e.g. bias accompanied with gender, age, race) (Sharp, Ak, and Hedberg 2018). In order to develop an ethical machine learning algorithm, cooperation between researchers, developers and policy makers is needed. Turilli 2020)recommended that machine-learning algorithms should reflect the same ethical principles as human workers to ensure consistency with an organization’s ethical standards.

In addition to the ethical challenge, there is a shortage of machine learning engineers. The demand for artificial intelligence (AI) talents has increased fast in recent years, resulting in a high number of AI-related vacancies (Culbertson, 2018). In the American job market, machine-learning engineers are listed as one of the top emerging professions (LinkedIn, 2017). It will take years to adequately meet the demand for machine learning engineers in industry (Woolf and McIntyre 2018).

The challenge related to data quality refers to the fitness of data for specific applications of machine learning. It is important to have high-quality data for the successful application of machine learning to solve problems. The performance of machine learning or deep learning highly depends on the data quality (Greenspan, van Ginneken, and Summers 2016). If the study data are collected from various sources in an unstructured format, the quality of the data deteriorates quickly. Machine learning algorithms demand structured data, although the majority of data are unstructured (e.g. voice, text, image data, etc.), which are often difficult to process (Lee 2017). To get fruitful outcomes from machine learning algorithms, companies need to establish a data-quality control process to develop quality metrics, collect new data, evaluate data quality, remove inaccurate data from the training data set, and assess the trade-off between quality-assurance costs and gains (Lee and Shin 2020).

The final challenge of applying machine learning is to analyze the relevant costs and benefits. Although the relevant benefits for using machine learning are huge, it can still be difficult to proving the value of the investment to stakeholders owing to the delay between investment and reward (Deloitte Technology, 2018). Organizational managers need to keep in mind that machine learning cannot solve all the business problems; therefore, it is necessary to investigate the investment analysis with all stakeholders, including top management and users, before its deployment. In the case of a risky and irreversible project, traditional investment-evaluation techniques may not be suitable to capture the value of machine learning projects. It is therefore advisable to adopt a real-option approach to investment justification, which offers managers the opportunity to optimize the planning of machine learning projects (Lee and Lee 2015).

3. Study methodology

This study was conducted in a case company, engaged in sheet metal work, where the problem was to identify the optimal gaps between components in order to more easily handle the waste sheet after the punching operation. To identify and visualize the gaps, both machine learning and deep learning algorithms were adopted to analyze images of the sheets provided by the company. These algorithms were deployed as part of the image processing technique to visualize the gaps clearly. The provided images were converted to suitable ones, which could easily be analyzed by using machine and deep learning algorithms.

In order to study the gaps, all images of the components over the sheet were analyzed. The study of these images helps to identify which parts of the images are actual components, and which parts are waste materials to trash. This gap optimization process also helps the case company to maintain its operational schedule between the day and night production lines by employing the most critical, and non-critical components based on the gaps. This study analyzes the gaps between components by converting the components’ images from vector to pixel images through a machine learning algorithm. This image conversion process facilitates the study and contributes to the company by analyzing the design of the components and the gaps necessary for a stable and optimal production process.

The input of the machine-learning algorithm was an image of a metal sheet, which contains the components’ drawings and their corresponding thresholds. This image is used to identify the acceptable distance between any two components. In addition, the given image is a .svg file and a gray scale image. The output of the algorithm is the same image as the input with the location of a too narrow gap (less than the given threshold) between components, and it is marked by a circle. The output image should show all the possible narrow areas between components.

This study uses Python 3.7.0 to program the necessary stages of the machine-learning algorithm. During this project, there were several library files used, which are listed and briefly described in Table 2. These library files were used to achieve better images of the gaps between components. In order to solve the problems related to the gaps, it is essential to obtain better visibility of the images.

Table 2. List of library files used during the study

Serial no.	Library file	Description of the library file and its functionality
1	requirements.txt:	Contains all the needed libraries. In addition, this file is also used for installing these libraries.
2	setup.sh:	Contain the commands for setting up a virtual environment as well as installing the needed libraries.
3	run.sh:	Contains the commands for activating the virtual environment and running the application.
4	Config.py:	Contains all the default parameter values; you can modify it directly in this file.
5	Preprocessing.py:	This file is responsible for pre-processing the input images so that we can pass these as input to our application. Preprocessing.py contains some functions; each function does the specific task.
6	Component.py:	We are working with the image input. In particular, we measure the distance of any 2 of the components on this image, so it is better for us if we get the components from the image and treat them as objects. In this class Component, we define some needed attributes and methods.
7	Image.py:	This file represents the Image object. In this class, we also define some needed attributes and methods for Image object.
8	Processing.py:	This file performs the main purpose of this application. It plots the circles in every single pixel in the components, and after that these circles are responsible for measuring the distance.

The most direct way to measure the distances of components in an image is by measuring the coordinates of these in a pixel image. Since the input is not a pixel image (it is an svg image), it should be converted into pixel. By doing that, the input can now be handled as a 2D matrix $W$ with $m$ rows and $n$ columns (m, n are known as the size of the converted image, which we can adjust in the converting process). Each entry in row $i^{th}$ ($1\le i\le m$1) and column $j^{th}$($1\le j\le n$) of this matrix, $W_{ij}$, is proportional to the brightness of the image at that point. Because the input is a gray scale image, each entry $W_{ij}$takes a value between 0 to 255 (0 is taken to be black, 255 is taken to be white).

3.1 Justification for using machine-learning algorithm in this study

Sheet metal companies demand a high level of knowledge and expertise from the competent designers. To achieve such expertise in this industrial sector, various artificial intelligence techniques are being deployed with the objective of reducing complexity, minimizing the human work force as well as improving operational excellence. The machine learning algorithm as part of artificial intelligence is one of the most powerful techniques for solving engineering problems and reducing complexity and minimizing the use of human expertise and time taken for production processes.

There are several successful applications of ML in manufacturing industries. The major application of ML is monitoring (Chinnam 2002), especially in the areas of quality monitoring, machine condition monitoring, fault diagnosis, tool wear, optimization, etc.(Wuest eal.2016).In addition, ML is also used in manufacturing in image recognition, where the images are used to identify damaged products. These applications of ML in different manufacturing and optimization problems demonstrate the vast adaptability and applicability of the ML algorithm.

Component quality is an essential parameter for the sheet metal industry, where it is a tedious job to manually inspect for component defects caused by the punching operation. In order to avoid this, it is useful to use an image-based component investigation strategy. This vision-based investigation system is considered a powerful and lasting solution (Ghatnekar 2018). Using computer vision and machine learning can provide a robust and effective approach to overcome the challenge of detecting and classifying component defects. Image recognition is considered an important strategy, where an image is used to detect which type of defect the components or parts have. In real-time applications, the image recognition technique is widely used in industries (Ghatnekar 2018). The machine-learning algorithm also helps to predict the occurrence of defects in sheet metal forming processes (Dib, Oliveira, and Marques et al. 2020), and in the identificationof parts (Sheu et al. 2020). These unique characteristics of the ML algorithm influenced and justified the authors in deploy this methodology in the study.

Other than machine learning algorithm, other algorithms are used in sheet metal industries for various purposes. For instance, linear programming can be used to minimize the cost of sheet metal punching when batching orders (Herrmann and Delalio 2001). Kakandikar, Darade, and Nandedkar (2009) deployed genetic algorithm to optimize the geometry parameters (e.g. die design, hammering sequence, blank holder pressure, etc.) in the sheet metal industry. Ashokkumar et al. (2020) proposed the use of ranking algorithm to optimize the quality of parts produced in sheet metal forming. Jiao and Xing (2018) used heuristic algorithm in analyzing the assembly deformation of parts, clamps and supporting locators in the sheet metal industry.

3.2 Description of the machine learning algorithm

• Step 1: Convert input image from .svg file (vector image) to .jpg or .png file (pixel image)

• Step 2: From the pixel image input, obtain the coordinates of the components. Figure 1 displays the necessary program for conversion of vector image to pixel image. Figure 2 visualizes the converted image from vector to pixel image.

Figure 1. Function to obtain the coordinates of components.

Figure 2. Transformation of vector image to pixel image.

• Step 3: Get rid of the body of the component.

The main purpose of this algorithm is computing the distance between components. In particular, the distance between the boundaries of those components needs to be measured. Therefore, there is no need to take care of the body. Furthermore, getting rid of the body also helps to reduce the running time of the algorithm.

• Step 4: Perform the detection

Let us assume that the threshold (minimum acceptable distance) is d (pixels). On every pixel p in the boundary, we plot a circle center p and radius d/2. Therefore, if any 2 circles which come from different components are overlapping each other, then this is the location we need to mark (because at this location, the distance of the 2 components is less than d). Figure 3 shows various circles over the components needed to find the gaps between them.

Figure 3. Various boundary circles over a single component.(a) The sample input image with (b) The corresponding graph white components.

3.3 Coordinates of components

As mentioned earlier, the most direct way to measure the distance d_ab between point a and point b in 2D dimension is from its coordinates: (x_a, y_a) and (x_b, y_b) respectively, using the formula:

(1) $d_{ab}=\sqrt{{(x_a-x_b)}^2+{(y_a-y_b)}^2}$(1)

However, how do we obtain the coordinates of the components in an image? The image basically contains two parts: components and background. The color of the background is completely different from the components. In our case, the background is black, while the component is not, which means in the matrix that if any entry takes the value of 0, it must belong to the background. Based on that, we have two approaches to solve this problem.

3.4 Algorithm used for the detection of narrow gaps

To detect the narrow gaps (less than a given threshold d), the authors performed the following steps:

Segment and identify the components in images.

Separate the components from each other. For each component, detect whether the gap between it and the others is larger than d or not.

Mark the area where the gap is less than d.

3.4.1 Segmentation and separation of the components

An input image has two parts: components (white) and background (black). It can be represented as an m × n matrix W, whose entries range from 0 (black) to 255 (white). Since the background is black, the components can be extracted by removing all the ‘0ʹ entries in the matrix. Furthermore, it is easy to see that the entries, which represent a component, can be formed in a set of vertices in an undirected graph G as shown in Figure 4. Any vertex (also an entry of matrix W) ‘u’ in G has at most 8 adjacent nodes, which are entries around ‘u’. To detect a component in the image, ^{firstly iterating over entries in W, if there is an entry W}ij 6 = ^{0, consider it as a root, then reach other nodes. If} it is reaching a node such that at least one of its adjacent nodes has non-zero value, then this node would be a part of the components. The process terminates if we reach the background i.e. entry that has ‘0ʹ value. Algorithm 1 below shows the pseudocode of the implementation of the above algorithm in order to obtain the coordinates of the components as well as their boundary.

Figure 4. The sample input image and its corresponding graph.

3.4.2 Detection of narrow gap

The information in Figure 5 is used to explain the detection of narrow gaps between two components in a worksheet. The gaps between components P and Q, as shown in Figure 5, need to be identified. In order to identify the gaps, let us assume two arbitrary points, A and B, on components P and Q respectively. The distance between point A and point B is assumed as ‘d’. In order to detect the gaps between components P and Q, let us consider two circles, OA and OB, on components P and Q with centers at points A and B respectively with equal radius r = d/2. Based on this idea, the authors suggest an algorithm to identify the narrow gaps between components P and Q. According to the proposed algorithm, the narrow gaps between ^{two circles O}A and ^OB are not overlapping if the distance between points A and B is satisfied as:

$d_{AB}\le r_A+r_B=\frac{d}{2}+\frac{d}{2}=d$

Figure 5. Two overlapping circles of different components.

Algorithm:

Function Main (W):

for each entry u in W do

if u has value/= 0 and u has not been visited, then

Get – components (W, u)

Function Get- components (W, u):

Declare B as a set of boundary points

Declare P as a set of points of a component, including the boundary points. Initialize an empty queue Q

Insert u to B and Q

Mark u as visited entry while Q is not empty do

v = Q.back()

for each entry a around v do

if a has value = 0 and a has not been visited, then

Mark a as visited point

Insert a to Q

If a is a boundary point, then

Insert a to B

The pseudocode of this algorithm is described as below.

Function Detection(W):

for all components P in image do

for all points A in boundary of P do

Draw a circle OA, center in A, radius d

if there exists a point B in another component with P, in which the circle OB overlaps with OA

then

Output A and B as error points

3.4.3 Polygonization

To generate a polygon to visualize the error ”zones”, the authors applied Grahm’s Scan algorithm to find the convex hull of finite set of error points in the plane. Given the error points, this algorithm allows for the construction of a convex hull in O(n log n) time using only comparison, addition and multiplication operations. The details of Grahm’s Scan algorithm can be found in (Kong, Everett, and Toussaint 1990).

4. Study of a sheet metal company: an exploratory case example

4.1 Description of the case company

The case company operates in the sheet metal forming industry and is a manufacturer of sophisticated sheet metal forming machines. The company has a line of machines/solutions with various levels of automation available. One of the most sophisticated ones is fully automated and requires little human interaction. It has several product lines such as TheBEND – sheet metal bending, TheCOMBI – multifunctional systems, e.g. punching and laser cutting, TheLASER – sheet metal cutting with a laser with some products providing also welding and drilling capabilities, ThePUNCH – sheet metal punching, TheSYSTEM – a versatile range of solutions, which combines functionalities of the case company’s machines in one automated production line, TheSOFTWARE – a number of additional software solutions, which further optimize machine operations. In addition, With the Tulus® software family the company is capable of parts order and inventory handling, work scheduling and machine capacity monitoring, control and monitoring of machines’ tasks, control of material storage, production reports, integration with ERP (enterprise resource planning), and acting as MES (manufacturing execution system). The case company’s products are used in many industries: aerospace, agricultural, automotive, domestic appliances, elevators, HVAC, hospital and lab equipment, to list just a few. The case company is an innovative company that always searches for and is open to new ideas and close cooperation at different levels.

There are many facilities of the company in Italy, Finland, the USA and China, from which the company delivers machines and systems all over the world. The company’s sales and service network is active in over 80 countries, with direct presence or through a network of specialized dealers. The company’s product family offers highly advanced servo-electric solutions for punching, bending and integrated processes and is the widest in the world. All the products from the company are developed according to the ‘green means’ concept, combining sustainability and productivity.

Nesting applications are very advanced in the sheet metal industry. The initial target for the nesting algorithms is to reduce waste and optimize the utilization of the metal sheet process in the case company. However, once the components to be nested are organized too close to each other there is a risk that parts or the metal sheet frame that finally will be waste material will drop to the table. In the case of fully automated nesting like Libellula for SI Engineering (https://www.libellula.eu/media/sicam-tutorial/?lang=en), there is a risk that the nesting is too tight and the scrap that is supposed to stay on the frame of the metal sheet scrap will drop out. In case of manual operations, this is not a big issue – the operator simply removes the extra scrap from the worktable between loading a new metal sheet to the machine. However, in the case of fully automated work, like running the machine station unmanned during the night shift, the scrap material will cause a much bigger problem.

Once the scrap component drops on the worktable, the whole production process stops and waits for the morning for the operator to fix the error triggering situation. The simplest solution for this kind of problem is to organize the work queue so that the nesting that historically has not caused this kind of problem will be entered to a separate job queue to be manufactured during the unmanned working shift. In addition, a human operator can pre-check the nesting and estimate whether the nest will cause extra scrap. The most optimal solution would be a machine learning solution to automatically find the weak points from the nest and organize the job queue in order to optimize the throughput during the unmanned working shift.

4.2 Case definition and assumptions

This research focused on the problem of estimating the gaps between components for the sheet metal industry. We suppose all structural parts of the sheets are made with the same kinds of materials and fabricated by conventional machining processes. After analyzing the component punching problem, the gaps between the components are decided. These include the image of the components, weight of the sheets from which components were produced, punching rate, number of sheets punched, and cutting tool changes in the punching operation. To simplify the punching operation and increase its production rate in general, we considered the speed of punching rate, number of sheets in the process, and the number of changes of cutting tools in the process. In this case, we divided the punching complexity to four different levels (simple, medium, complex, and very complex). The study suggests that punched components can be categorized into two different levels for every geometric complexity of the structural components. The new punching problem contains two input variables (gap and complexity of the sheets) and one output (production rates). We try to find the relationship between inputs and outputs through machine learning methods.

This research study was conducted using machine and deep learning techniques to identify the weak points from the metal sheet. The process was as lean as possible and the images of the nesting results were used as input. The result ended with two job queues: one that needs operator presence and one that can be run unmanned. Besides the input images, the operator should be able to fine tune the settings of the validation process of the nests by specifying how wide the gap can be with various materials to keep the metal sheet scrap frame in one piece. The source images were in vector graphic format, which were converted to pixel graphics. After being able to operate the images in pixel graphic format, the machine learning algorithm was developed in order to identify the boundaries of the components on the sheet. After identifying the components, the actual gaps between the parts were measured.

5. Analysis of results

After applying input image as shown in Figure 6 (a) with the parameter threshold of 1.5 pixels, the output image is obtained as in Figure 6 (b). From the output, it can be seen that the algorithm works quiet well. It detects exactly the narrow locations of less than 1.5 pixels (the circles in Figure 6(b)). In addition, it also returns the boundary exactly and removes the body of components.

Figure 6. Input and output images of the worksheet.(a) Input Image (b) Output Image.

5.1 Sensitivity to noise

The proposed algorithm goes through every single pixel in the input; therefore, it is very sensitive to noise, which could be the variation of brightness or color information in the image input. This problem can be solved by preprocessing the input image.

5.2 Running time

As mentioned earlier, this algorithm takes O (m ✕ n). In general, this is quite good for an application. For more complicated input (more noise, more components), this algorithm takes a little more time time to run. This problem can be solved by vectorizing the operations in the algorithm. This can produce a better running time by forgetting to use for-loop and using vectorized arithmetic instead, as shown in Figure 7.

Figure 7. Comparative speeds of execution for conditional loop-based code block.

5.3 Gap detection

The gaps between each of the components are critical in order to punch the components after punching on the metal sheet. If the gaps between the components are too narrow, this creates a problem over the worktable by leaving some of the sections of the sheet on it. The rest of the parts stick on the worktable and trigger a stop of the automated grippers responsible for moving away the waste sheet to the trash bin. This also causes a complete shutdown of the punching operation and consequent production loss. The target of the case company was therefore to optimize the gaps between the components so that waste metal sheet does not stay on the worktable.

In order to optimize the gaps between the punched components, some pictures attached with the design of the components over the sheet metal were used to analyze the gaps. These pictures were in vector image format, and were converted to pixel images. This conversion makes the pictures more visible in order to optimize the gaps between the components. In addition, the authors emphasize here that the input image shows the components in white color and the background in black.

Figure 8(a) visualizes the gaps between several components which are more than 5 pixels. They satisfy the design requirement, and as can be seen, no error is noticed. On the other hand, Figure 8(b) displays the gaps between components of more than 5 pixels which are not acceptable in terms of the design requirement and create problems on the worktable. Due to such inconsistencies with the design requirements, several errors are noticed after analysis, as are also shown in Figure 8(b).

Figure 8. Image and its pixel matrix.(a) Acceptable Gaps (b) Unacceptable Gaps.

5.4 Analysis of the gaps between the components

In order to optimize the gaps between punched components, some pictures attached with the design of the components over the sheet metal were taken into consideration to analyze the gaps. These pictures were in vector image format, which were converted to pixel image by machine learning algorithm. This conversion makes the pictures more visible to optimize the gaps between the components.

Figure 9 (a) visualizes the gaps between two adjacent components; it is 1.5 pixels and satisfies the design requirement with no error being noticed, as displayed in Figure 9 (b).

Figure 9. Visualization of acceptable gap between components and associated no-errors.(a) Acceptable gaps between components (b) No-errors due to acceptable gaps between components.

On the other hand, Figure 10(a) displays the gaps between components of more than 1.5 pixels: it is not acceptable in terms of design requirement and creates a problem on the worktable. Due to such inconsistencies with the design requirements, several errors are noticed after analysis and visualized in Figure 10(b).

Figure 10. Visualization of non-tolerable gap between components and associated errors.(a) Non-tolerable gap between components (b) Errors due to non-tolerable gap between components.

Such errors continue to increase as long as the optimum gaps between the components are not achieved. This optimization is done by computer vision algorithm, which is essentially a part of machine learning, as explained earlier. This optimization process facilitates the waste sheet management process after the necessary operations. Figure 11 visualizes the unacceptable gaps between the components as red polygons.

Figure 11. Red polygons show the too narrow gaps between components.

6. Managerial implications

Operational excellence is a key criterion for organizations to be profitable with higher customer satisfaction. Organizations are striving to achieve operational excellence within their limited resources. Any complexities or bottlenecks represent a heavy cost to organizational success and negatively affect customer goodwill. Organizational managers therefore always need to be careful to avoid such negative impacts on their organizations. Managers need to be capable of developing their situational awareness, which might evolve from their operational processes.

In this research, the overall objective was to bring about operational excellence by overcoming problems existing in an organization. The studied case company was suffering an operational problem, which was solved by applying an up-to-date technique known as computer vision. This computer vision, which is a branch of machine learning, helped to solve the existing problem in the company. This technique involved image processing which is deployed to detect the gaps between punched components. This image processing can be applied to other operational processes within organizations to solve other related problems.

The study results highlighted in this research provide a guideline for organizational managers to deploy state-of the-art technologies and tools for mutual benefit. In deploying such technologies and tools, managers also need to trade off their benefits with the associated costs involved. In using the deployed image processing technique within machine learning algorithm, managers would be able to avoid limitations to their operational processes and optimize the processes for competitive advantage.

7. Discussion and conclusions

During the study, a component analysis technique was used to create and narrow gap detection between components from images. It was observed from this study that the proposed algorithm works well, except for the fact that there are some problems that still need to be solved to obtain better results such as sensitivity to noise and speeding up the running time. The component analysis technique was adopted through image processing, where the computer vision principle was used. This image processing technique, which is an integral part of machine learning, is used to make the gaps between components clearly visible. It is critical to maintain the gaps between components during the drawing stage that supports the punching operation.

This study clearly identifies and contributes to improving the operation limitations of a case sheet metal company in Finland. The application of the component analysis technique, which was formulated through machine learning, helps the company’s operational flexibility and minimizes the production time. It ensures the maintaining of the shifts at the company during both day and night. For instance, in the case of complex components, where the gaps between them was critical, they were separated and allocated to the day shift when human operators can remove the waste sheet after the punching operation if blocked by the automated gripper. On the other hand, less complicated components, where there are sufficient gaps between the components, are allocated to the night shift, where automated grippers comfortably remove the waste sheets after operation. In this way, this gap optimization technique through machine learning application contributes to maintaining production stability and quality at the same time.

This study includes some limitations that can be improved in a future study. For instance, the runtime was not fast enough to execute the system within a reasonable time. The image processing can also be made noise free to produce faster image conversion, which is necessary to detect the gaps between components. Moreover, this study experienced many coordinate points during the error recognition process that can be minimized through optimization of the gaps between components as much as possible. In addition, a future study might continue to investigate and compare the study findings with other available algorithms and techniques such as genetic algorithm, heuristic algorithm, computer vision, simulation, etc. with the objective of solving the operational bottlenecks of the case company. Moreover, further study also can be initiated to investigate the component gaps through partial least square regression and popular feature extraction method.

Disclosure statement

No potential conflict of interest was reported by the author(s).