Selecting advanced analytics in manufacturing: a decision support model

Abstract

Advanced analytics offers new means by which to increase efficiency. However, real-world applications of advanced analytics in manufacturing are scarce. One reason is that the management task of selecting advanced analytics technologies (AATs) for application areas in manufacturing is not well understood. In practice, choosing AATs is difficult because a myriad of potential techniques (e.g. diagnostic, predictive, and prescriptive) are suitable for different areas in the value chain (e.g. planning, scheduling, or quality assurance). It is thus challenging for managers to identify AATs that yield economic benefit. We propose a multi-criteria decision model that managers can use to select efficient AATs tailored to company-specific needs. Based on a data envelopment analysis, our model evaluates the efficiency of each AAT with respect to cost drivers and performance across common application areas in manufacturing. The effectiveness of our decision model is demonstrated by applying it to two manufacturing companies. For each company, a customized portfolio of efficient AATs is derived for a sample of use cases. Thereby, we aid management decision-making concerning the efficient allocation of corporate resources. Our decision model not only facilitates optimal financial allocation for operations in the short-term but also guides long-term strategic investments in AATs.

Keywords:

• Advanced analytics
• data envelopment analysis
• decision model
• manufacturing

Introduction

Advanced analytics offers companies new possibilities to generate knowledge and business value from large datasets (Chehbi-Gamoura et al. 2019; Choi, Wallace, and Wang 2018; De-Arteaga, Feuerriegel, and Saar-Tsechansky 2022; Dogan and Birant 2021; Feuerriegel et al. 2022; Misic and Perakis 2019). It enables evidence-based decisions and actions informed by data. In this way, advanced analytics can become a critical differentiator for companies seeking to outcompete competitors (LaValle et al. 2011; Sivarajah et al. 2017).

There is a large untapped potential for process improvement and innovation using advanced analytics in manufacturing (Feng and Shanthikumar 2018; Kusiak 2017; Lee 2018; Lorenz et al. 2020; Senoner, Netland, and Feuerriegel 2022; Sony and Naik 2019; Wolf et al. 2019). Despite the promising prospects, manufacturers are not exploiting the full potential of advanced analytics (Stentoft et al. 2021). McKinsey estimated that <30% of the efficiency potential has been tapped so far (Henke et al. 2016). A core problem is that decision-makers in manufacturing face the complex task of selecting a suitable set (i.e. portfolio) of advanced analytics technologies (AATs). Concrete examples of AATs are reinforcement, semi-supervised, or deep learning techniques.¹ Each AAT requires different resources during implementation and, furthermore, AATs vary in performance with respect to different application areas. Accordingly, decision-makers have to decide which selection of AATs entails the greatest economic benefit based on their company-specific needs. However, this choice is non-trivial for two main reasons. First, decision-makers lack knowledge concerning valuable application areas for advanced analytics (Fleming et al. 2018; LaValle et al. 2011). Second, they require decision models to aid them in the selection of suitable AATs from a plethora of different techniques (Wolf et al. 2019; Zangiacomi et al. 2019).

To aid managers in the manufacturing sector, this paper proposes a multi-criteria decision model that identifies a set of efficient AATs across different applications. Based on a data envelopment analysis (DEA), our decision model evaluates each AAT with respect to multiple cost drivers and effectiveness across common application areas. The DEA is particularly well suited for this problem as it can consider multiple input and output variables of the different applications. This is essential as the portfolio decision is complex, requiring to take various options and resources into account. To the best of our knowledge, this is the first use of a DEA model to allocate resources (e.g. financial, human) in conjunction with advanced analytics. This contribution can foster the further adoption of advanced analytics in manufacturing.

The applicability of our decision model is demonstrated using examples from two manufacturing companies. For each, we conducted expert interviews with associated project managers to identify both company-specific requirements and cost drivers. We evaluate efficiency across different AATs and, based on the results, recommend which advanced analytics techniques the manufacturing companies should use. Thereby, we guide resource allocation to AATs.

Our work has several implications for managers. Using our approach, decision-makers in manufacturing can support their selection of effective AATs. This is often challenging in practice since not merely a single AAT but rather a set of multiple AATs is required to improve efficiency in a manufacturing system. Given the recommended set of AATs, managers can improve human resource allocation, for example by hiring personnel with particular skills and steering short- and long-term financial investments in the AATs. Our decision model is flexible: it can be customized along different dimensions, namely the available AATs, cost drivers, and performance measures. Hence, our approach has wide applicability in diverse manufacturing settings.

The remainder of this paper is structured as follows. Section Related work provides an introduction to DEA and highlights the existing research gap. Section Model development develops our decision model to aid manufacturing management in selecting a set of efficient AATs. Section Empirical findings empirically demonstrates the effectiveness of our decision model in two manufacturing companies. Section Discussion discusses managerial implications, while Section Conclusion concludes.

Related work

In the following, we review existing approaches of selecting a set of efficient AATs for manufacturing. We then continue to provide a general introduction to DEA as a potential solution to overcome the limitations of the existing approaches.

Decision support for AAT selection

Research has identified many prominent applications of advanced analytics in manufacturing (Cheng et al. 2018; Choudhary, Harding, and Tiwari 2009; Köksal, Batmaz, and Testik 2011; Lorenz et al. 2021; Schuh et al. 2019; Sharp, Ak, and Hedberg 2018; Wang et al. 2018; Wuest et al. 2016). However, little research has been devoted to decision models that equip management with tools to select a suitable set of efficient AATs. Specifically, prior literature has suggested decision models for identifying and configuring single AATs for problem-specific applications, but not for choosing among multiple AATs. In this regard, this paper offers a novel and specific solution to a general problem that has been insufficiently addressed in the past.

Examples for selecting and configuring a single AAT are as follows. Villanueva Zacarias, Reimann, and Mitschang (2018) suggest a framework that automatically recommends suitable analytics techniques with respect to a domain-specific problem at hand. Lechevalier et al. (2018) present a framework for the semi-automatic generation and configuration of neural networks. However, this research stream is limited to single AATs rather than choosing a strategic selection of promising AATs. A promising way to overcome these limitations of the existing methods is offered by the DEA.

Data envelopment analysis

In management science, the evaluation of efficient entities is usually based on the DEA (Braglia and Petroni 1999; Braglia, Zanoni, and Zavanella 2003; Emrouznejad and Yang 2018; Jagoda and Thangarajah 2014; Liu, Lu, and Lu 2016). DEA is a method to assess the relative efficiency of homogeneous decision-making units (DMUs) that convert multiple inputs into multiple outputs (Charnes, Cooper, and Rhodes 1978). ‘Inputs’ are resources used by the DMUs, while ‘outputs’ represent generated benefits (Golany and Roll 1989). DMUs are generic entities; they can be organizations, divisions, industries, projects, or individuals.

The efficiency of each individual DMU_j, j = 1,…, n, is evaluated based on m non-negative inputs x_i = (x_1,j,…, x_m,j) and s non-negative outputs y_r = (y_1,j,…, y_s,j). The DEA is based on a common measure to evaluate the efficiency of each DMU_j, j = 1,…, n, i.e. (1) $$\frac{{\sum }_{r\hspace{0.17em}=\hspace{0.17em}1}^s{y}_r}{{\sum }_{i\hspace{0.17em}=\hspace{0.17em}1}^m{x}_i}.$$(1)

The strengths of the DEA are as follows. First, it is an empirical (‘data-oriented’) approach (Bogetoft and Otto 2011; Jagoda and Thangarajah 2014). Second, the DEA is non-parametric and thus does not require explicit characterizations of relations like linearity, which are customarily used in statistical regressions and related approaches (Cooper, Seiford, and Tone 2006). Third, the DEA can simultaneously handle multiple input and output variables that would otherwise not be comparable to one another (Cooper, Seiford, and Zhu 2011). This property of processing arbitrary factors ensures the DEA’s widespread applicability as a tool for benchmarking in many domains (Braglia, Zanoni, and Zavanella 2003; Chen, Cook, and Lim 2019; Liu, Lu, and Lu 2016).

The two basic DEA models are the Charnes, Cooper, and Rhodes (CCR) model (Charnes, Cooper, and Rhodes 1978) and the Banker, Charnes, Cooper (BCC) model (Banker, Charnes, and Cooper 1984). Both the CCR and the BCC draw upon a technology space T, defined as any possible input-output combinations, that are limited by the efficiency frontier. The technology space T is subject to two production economic regularities, namely free disposability and convexity (Cooper, Seiford, and Tone 2006). Free disposability assumes that one can produce at least the same number of outputs with more input. Convexity assumes that any weighted average of feasible input-output combinations is also a feasible production plan.

Both the CCR and the BCC model draw upon these two assumptions, yet they differ with respect to how input and output can be scaled, i.e. the returns-to-scale (RTS) assumption. Changing RTS assumptions affects the shape of the efficiency frontier and thus the technology space T (see Figure 1). The CCR model builds on the assumption of constant returns-to-scale (CRS) of activities. In other words, one can assume that we can arbitrarily scale inputs x_i and outputs y_r in a linear fashion (Cooper, Seiford, and Tone 2006). In combination with free disposability and convexity, the technology space for possible input-output pairs can thus be expressed as (2) $$T_{\mathit{crs}}=\left\{\left(x_i,y_r\right)\in {\mathbb{R}}_{+}^m\times {\mathbb{R}}_{+}^s|x_i\ge \sum _{j=1}^n{\lambda }_j{x}_{ij},\mathrm{\hspace{0.33em}}{y}_r\le \sum _{j=1}^n{\lambda }_j{y}_{rj},\mathrm{\hspace{0.33em}}{\lambda }_j\ge 0\forall j\right\},$$(2) where λ_j, j = 1,…, n, are the weights of the efficiency reference set (ERS), that is, a linear convex combination of efficient DMUs located on the frontier. The BCC model differs from the CCR model by additionally enforcing the condition ${\sum }_{j=1}^n{\lambda }_j=1.$ This implies a variable returns-to-scale (VRS) assumption and thus yields (3) $$T_{\mathit{vrs}}=\left\{T_{\mathit{crs}}|\sum _{j=1}^n{\lambda }_j=1\right\}.$$(3)

Figure 1. Illustrative efficiency frontier and technology space T under CRS (solid line) and VRS (dashed line) with s = 1 and m = 1. DMUs are shown as points.

Because of this additional restriction, the technology frontier of the BCC model is less optimistic regarding potential input-output combinations than the CCR model. As a consequence, a BCC model with VRS always yields a greater or equal number of efficient DMUs relative to a CCR model with CRS, as shown in Figure 1. Generally, the choice of the RTS assumption depends on the particular use case (Doyle and Green 1993). As we shall see later, our research setting requires a model that builds upon a VRS assumption.

Previously, the DEA has been widely considered a suitable decision model for strategic performance evaluation at the managerial level. For instance, researchers have applied the DEA in contexts, such as computer printers (Doyle 1991), robotics manufacturing systems (Cook, Kress, and Seiford 1993), software projects (Mahmood, Pettingell, and Shaskevich 1996), industrial robots (Baker and Talluri 1997) and automotive dealerships (Almohri, Chinnam, and Colosimo 2019).

However, to the best of our knowledge, the DEA has not yet been considered for selecting a set of efficient AATs in manufacturing. Since the DEA allows to select an efficient set consisting of multiple technologies, AATs can easily be added or removed to represent the latest development of AATs over time. In this regard, the DEA accounts for dynamic developments.

Model development

In the following, we first state our problem generally. We then present our proposed DEA framework. The resulting decision model requires suitable input and output variables, which we introduce subsequently. In the end, we formulate the optimization problem mathematically.

Problem statement

Our goal is to develop a multi-criteria decision model that identifies a set of efficient AATs in manufacturing. We model this decision problem via a DEA. The DEA is suitable because we (i) want to consider multiple input and output variables, (ii) have to handle incommensurate variables with different units, and (iii) require a scalar measure that allows for a ranking of examined AATs.

Proposed DEA framework

For our decision model, we propose an input-oriented BCC model in its envelopment form. Relevant inputs are given by financial resources that are required to implement an AAT, namely data preparation, modelling, and estimation costs. Relevant outputs are given by the generated effectiveness of an AAT across different application areas. Each output represents one aggregated effectiveness indicator based on three common measures, namely prediction performance, interpretability, and online learning ability (described in detail in the next chapter). Figure 2 shows our DEA framework. Each AAT is described by three cost drivers and in terms of its effectiveness in six application areas.

Figure 2. Proposed DEA framework in order to assess the efficiency of AATs.

We opt for the input-oriented DEA variant because cost drivers can be considered flexible while the performance of each AAT in an application area is predetermined by the underlying technology. A manager, for instance, can hire better data scientists to improve modelling or data preparation (e.g. data collection mechanisms can be optimized to reduce data pre-processing costs). Estimation costs can be reduced, for instance, if appropriate computing infrastructure is acquired. However, the effectiveness of an AAT is exogenously determined by the underlying technology itself.

Furthermore, we opt for the BCC model as it allows for VRS. In our problem setting, it can be assumed that an increase in cost drivers will not result in an unlimited proportional change of effectiveness (Hastie, Tibshirani, and Friedman 2009; Bishop 2006). Another convenient advantage of the BCC model is that its VRS assumption additionally mitigates the shortcoming of AATs that are not fully homogeneous. Consequently, our previously mentioned drawback is alleviated by the fact that our BCC approach better accommodates the scale effects between different AATs in the efficiency analysis (Baker and Talluri 1997).

The reasons for solving the model using the envelopment form are 3-fold. First, the computational effort is considerably smaller compared to the multiplier model (number of constraints m + s ≪ n). Second, the envelopment form allows us to find an optimal, i.e. zero-slack, solution. Third, the interpretation of results is more straightforward because the solutions are directly related to inputs and outputs that correspond to original data compared to virtual multipliers (Cooper, Seiford, and Tone 2006).

Model parametrisation

In the following, we describe (i) cost drivers, (ii) application areas, and (iii) effectiveness measures of AATs in the respective application areas in detail.

Cost drivers

Relevant cost drivers for AATs are (i) data preparation, (ii) modelling, and (iii) estimation costs (Cheng et al. 2018; Villanueva Zacarias, Reimann, and Mitschang 2018; Hazen et al. 2014; Hastie, Tibshirani, and Friedman 2009; Kotsiantis, Zaharakis, and Pintelas 2007; Wuest et al. 2016). Each cost driver is detailed in the following.

Data preparation costs quantify the effort needed for pre-processing. Pre-processing costs for different AATs can vary substantially with respect to missing values, (ii) outliers, (iii) feature engineering (including transformation), and (iv) feature selection. For instance, some AATs, such as deep learning, learn feature representations implicitly and thus largely circumvent the need for manual pre-processing (Goodfellow, Bengio, and Courville 2017; LeCun, Bengio, and Hinton 2015). The LASSO comes with an implicit capacity for feature selection and thus automatically encourages parsimonious models (i.e. models with fewer parameters). In manufacturing, data pre-processing is usually labour intensive due to known issues in data quality (Hazen et al. 2014; Wang et al. 2018; Wuest et al. 2016).

Modelling costs arise from the manual effort that is needed to develop a model specification. This comprises the selection of both a model architecture and the best hyper-parameter configuration. Modelling costs can vary strongly between different AATs (Hastie, Tibshirani, and Friedman 2009; Kotsiantis, Zaharakis, and Pintelas 2007). Some models (e.g. regression models) leave only a few degrees of freedom. Other AATs require extensive manual effort to optimize a large set of (hyper-) parameters. This particularly applies to deep learning, where the choice of a suitable architecture is almost as important as hyper-parameter tuning (Kraus, Feuerriegel, and Oztekin 2020).

Estimation costs are characterized by computational efficiency. This includes demand for customized hardware (e.g. graphics processing units), required memory, and run-time. These costs vary considerably among different AATs: for instance, some can better handle large-scale data sets or large numbers of predictors (Kraus, Feuerriegel, and Oztekin 2020). This is especially relevant in manufacturing, since data-sets are often very large, both in terms of observations and the number of variables (Cheng et al. 2018; Villanueva Zacarias, Reimann, and Mitschang 2018; Wuest et al. 2016).

Application areas of advanced analytics in manufacturing

Multiple application scenarios of advanced analytics in manufacturing have previously been identified (Cheng et al. 2018; Choudhary, Harding, and Tiwari 2009; Köksal, Batmaz, and Testik 2011; Sharp, Ak, and Hedberg 2018; Wuest et al. 2016; Wolf et al. 2019). However, a clear classification of application areas is lacking. We thus conducted a systematic literature review (see Supplementary Appendix A for details) of analytics applications in manufacturing and derived the following classifications that we utilize throughout this paper: (i) production planning, (ii) production scheduling, (iii) quality inspection, (iv) quality driver identification, (v) process control, and (vi) maintenance planning.

Production planning requires accurate forecasts of demand (Chui et al. 2018; Esmaeilian, Behdad, and Wang 2016; Tao et al. 2018). However, this is challenging given that manufacturers are located at the upstream end of a long supply chain with multiple distributors, wholesalers, and retailers between the customer and themselves. This introduces multiple sources of variability and thus impedes straightforward forecasts of demand (Carbonneau, Laframboise, and Vahidov 2008). To aid better planning, advanced analytics for demand forecasting can be implemented. This commonly builds upon historical time series models (Ferreira, Lee, and Simchi-Levi 2016), which are often expanded by external user data (Boone et al. 2018; Chong et al. 2017; Cui et al. 2018; Lau, Zhang, and Xu 2018).

Production scheduling assigns incoming orders to machines based on a predetermined decision objective, e.g. the earliest due dates or the shortest average completion time. Job scheduling can be loosely categorized further into (i) job-shop scheduling and (ii) flow-shop scheduling. In the former, job-shop scheduling, n independent jobs have to be processed by a particular pre-determined subset of m machines, but without explicit orders (Zhang, Song, and Wu 2013). In contrast, flow-shop scheduling is characterized by n jobs that have to be processed on the same number of m machines in a sequence (Noroozi, Mokhtari, and Kamal 2013).

Quality inspection uses sensor data to analyze the condition of a product, e.g. scrap or good parts (Weimer, Scholz-Reiter, and Shpitalni 2016). Quality inspection has become a prevalent automation tool to reduce labour force and increase yield rates in manufacturing (Köksal, Batmaz, and Testik 2011; Wu and Zhang 2010). For instance, quality inspection based on visual sensing has the great advantage of contactless inspection and is, therefore, not influenced by the type, condition, electromagnetic field, or temperature of the target (Park et al. 2016). The inferred quality level is then used to determine whether and in which market a product is sold (Li, Xia, and Yue 2019).

Quality driver identification determines root causes that affect the quality (Chen, Tseng, and Wang 2005). Examples of such factors are the quality of materials or environmental conditions (e.g. temperature). These factors interact with each other and consequently make it difficult to systematically infer which factor is responsible for poor quality output (Hsu and Chien 2007; Kraus and Feuerriegel 2019; Tao et al. 2018). Once quality drivers have been inferred, actions can afterward be implemented with the objective of process improvement (Zantek, Wright, and Plante 2002).

Process control dynamically determines machine parameters so that a target measure is achieved, such as, for instance, maintaining a certain output quality or optimizing energy efficiency (Cheng et al. 2018; Elwany, Gebraeel, and Maillart 2011). Process control based on manual decision-making is often inefficient due to the abundant number of free parameters in modern manufacturing equipment. In contrast, advanced analytics promises the prospect of choosing parameters that result in improved productivity and product quality, as well as reduced costs (Tao et al. 2018).²

Maintenance planning introduces condition-based services, so that maintenance events are scheduled proactively or when a breakdown is identified to reduce the downtime of manufacturing machinery and tool equipment (Olde Keizer et al. 2018; Sharp, Ak, and Hedberg 2018). Maintenance planning can be further subdivided depending on whether actions are triggered reactively or proactively. Reactive maintenance infers the (latent) condition of a system in real-time from observable sensor data. Based on this, it triggers maintenance when the condition deteriorates below a certain safety threshold, such as the growth of a fault frequency. A major drawback of reactive maintenance is that it cannot anticipate a breakdown ahead of time. In contrast, proactive maintenance (often termed ‘predictive maintenance’) forecasts the remaining useful life of a system and, on this basis, triggers maintenance events in advance (Mazhar, Kara, and Kaebernick 2007; Sharp, Ak, and Hedberg 2018). To this end, it looks at the current and past states of the system, as well as its expected future operational load. This has multiple benefits in practice: it prevents breakdowns, prolongs the lifespan, and ensures the availability of spare parts in due course (Tobon-Mejia et al. 2012).

Effectiveness measures of AATs in application areas

The AATs considered are as follows. We evaluate 31 AATs for our analysis. These AATs were selected based on an extensive review of the existing literature on advanced analytics (see Supplementary Appendix A for details). We subdivide all AATs into three main categories: (i) diagnostic, (ii) predictive, and (iii) prescriptive techniques.

Relevant effectiveness measures for AATs are (i) prediction performance, (ii) interpretability, and (iii) online learning (i.e. the ability of an AAT to be regularly updated with new data). The dimensions of prediction performance and interpretability are commonly used to gauge the suitability of a model. However, these measures are negatively correlated with each other and thus entail a trade-off that must be made for each use case (Hastie, Tibshirani, and Friedman 2009). Models with higher prediction performance are often black-box models and thus lack interpretability. Online learning is regarded as a crucial ability in manufacturing (Bang et al. 2019; Cheng et al. 2018; Sharp, Ak, and Hedberg 2018; Wuest et al. 2016). For a detailed overview of the evaluation, we refer to Supplementary Appendix B.

We use a ten-point scale to assess each AAT with respect to costs and effectiveness. That is, we followed Hastie, Tibshirani, and Friedman (2009) and Kotsiantis, Zaharakis, and Pintelas (2007) as they provide comprehensive rankings of AATs with respect to data preparation, modelling, and estimation costs, as well as prediction performance, interpretability, and online learning ability. To include domain-specific expertise, we additionally took findings from Wolf et al. (2019) into account. Scoring each AAT according to input and output variables is a crucial task within the presented approach as this includes qualitative and subjective judgement. Therefore, we see this data set as a starting point that can be deliberately adjusted.

Figure 3 shows the resulting parametrization of each AAT. For instance, image-based models usually have high estimation and modelling costs due to a high number of input variables (pixel features) and sophisticated model structures (e.g. a deep convolutional neural network). These AATs often perform best on visual quality inspection tasks, but also for maintenance planning where they can identify potentially deteriorating parts.

Figure 3. Costs and effectiveness intensities across different AATs.

Mathematical formulation

When applying DEA to a group of decision-making units, an optimization problem must be solved for each DMU. In its basic form, a DEA model is a quotient programming problem. This is because the efficiency value of a decision unit is a quotient with the sum of the weighted outputs in the numerator and the sum of the weighted inputs in the denominator.

Let AAT_j, j = 1,…, n, refer to the DMUs to be evaluated. Furthermore, each AAT_j is associated with m non-negative inputs x_i = (x_1,j,…, x_m,j) and s non-negative outputs y_r = (y_1,j, …, y_s,j). These are weighted using virtual multipliers v_i ≥ 0 for inputs and u_r ≥ 0 for outputs (Cooper, Seiford, and Tone 2006). Given the input and output data, we measure the relative efficiency θ of each AAT_j once and hence need n optimizations. Let the AAT_j to be evaluated on any trail be designated as AAT_o, o = 1,…, n. For a given AAT_o, the objective function is thus given by (4) $$\underset{u_r,v_i}{\max }\hspace{1em}\theta =\frac{{\sum }_{r=1}^s{u}_r{y}_{ro}}{{\sum }_{i=1}^m{v}_i{x}_{io}},$$(4) (5) $$\mathrm{s}.\mathrm{t}.\hspace{1em}\frac{{\sum }_{r=1}^s{u}_r{y}_{rj}}{{\sum }_{i=1}^m{v}_i{x}_{ij}}\le 1,\hspace{1em}j=1,\dots ,n,$$(5) (6) $$u_r,v_i\ge 0\hspace{1em}\forall r,i.$$(6)

Put simply, the DEA tries to adapt the individual weights v_i and u_r to maximize the relative efficiency scalar θ for the AAT_o under study. Equation (5)(5) $$\mathrm{s}.\mathrm{t}.\hspace{1em}\frac{{\sum }_{r=1}^s{u}_r{y}_{rj}}{{\sum }_{i=1}^m{v}_i{x}_{ij}}\le 1,\hspace{1em}j=1,\dots ,n,$$(5) ensures that the efficiency scalar is equal to or below 1 for all AAT_j, j = 1, …, n. AATs with an efficiency rating of 1 creates the so-called efficiency frontier.

The solution to a quotient programming problem is not easy because the objective function is not linear. Therefore, the problem is transformed into a linear programming problem using the so-called Charnes-Cooper transformation (Cooper, Seiford, and Tone 2006). (7) $$\underset{u_r,v_i}{\max }\hspace{1em}\theta =\sum _{r=1}^s{u}_r{y}_{ro},$$(7) (8) $$\mathrm{s}.\mathrm{t}.\hspace{1em}\sum _{i=1}^m{v}_i{x}_{io}=1,$$(8) (9) $$\sum _{r=1}^s{u}_r{y}_{rj}-\sum _{i=1}^m{v}_i{x}_{ij}\le 0,\hspace{1em}j=1,\dots ,n,$$(9) (10) $$u_r,v_i\ge 0\hspace{1em}\forall r,j.$$(10)

For each AAT, we compute confidence intervals for the bias-corrected efficiencies using a bootstrapping approach (Simar and Wilson 1998). The confidence intervals estimate the range within which a true efficiency occurs. This allows us to answer the question of AATs from the same group differ significantly in their efficiency. We provide further mathematical details in Supplementary Appendix C.

Empirical findings

In the following, we apply the proposed framework in two specific industrial settings.

Setting

The decision model supports manufacturing companies to decide in which AATs to invest. This question is especially relevant for larger companies that are undergoing a digital transformation. To demonstrate our decision model, we cooperate with two manufacturing companies, namely Geberit Produktions AG and ABB Turbo Systems AG. Geberit Produktions AG manufactures sanitary products for the end-consumer market. ABB Turbo Systems AG constructs turbochargers for vehicles and power facilities. During two separate expert interviews with associated project managers, we identify (i) which application areas are of relevance due to different business needs and (ii) which cost drivers are critical factors with regard to the decision-making process. We use semi-structured interviews (Yin 2011) to evaluate the two most important cost drivers and the two most important application areas for each company (see Figure 4). In the first part of the interview, the interviewee described the environment of the company. In the second part, the interviewee reveals the current challenges and objectives of their manufacturing division. The third part of the interview asks to rank the different cost and application areas of the model by importance for the setting of the company. These qualitative results from these expert interviews were used to tailor our decision model to the different needs of both firms. The presented cases shall therefore reveal the applicability and illustrate the use of the model.

Figure 4. Exemplary selection of chosen cost drivers and application areas.

Empirical application 1: Geberit Produktions AG

Description

Geberit Produktions AG specializes in manufacturing and supplying sanitary parts and related systems. Geberit is headquartered in Switzerland, it is a leader in the European market and has a global presence through its subsidiaries.

Prospectively, the most relevant application areas for Geberit Produktions AG are production planning, quality driver identification, and process control. Production planning is a key enabler to reduce high inventory levels. Geberit Produktions AG requires production planning to allow for short delivery times to its customers. Quality driver identification is another promising application area for advanced analytics and can help maintain high productivity within production processes. Process control is important since Geberit Produktions AG uses expensive, automated equipment. According to our informants, data preparation and modelling are especially relevant cost drivers. Data preparation costs are critical because Geberit Produktions AG uses decentralized databases for each production plant, which hampers data fusion (Fink, Netland, and Feuerriegel 2022). Modelling costs are relevant because of limited expertise in the field of advanced analytics. Estimation costs are regarded as negligible because of existing hardware capacities.

Based on our findings from the expert interview, our model only includes data preparation and modelling costs, as well as the effectiveness of AATs in production planning, quality driver identification, and process control.

Results

Table 1 shows the results from our DEA. Eight out of 31 AATs show an efficiency rating of θ^∗ = 1. However, there are only seven BCC-efficient AATs that satisfy both θ^∗ = 1 and ($s^{-*}=0, s^{+*}=0$), namely (2.) visual, (4.) fully-supervised, (7.) supervised deep learning, (16.) time-series models, (21.) mathematical optimization, (23.) Markov decision process and (24.) hidden Markov models. Consequently, these AATs also represent the ERS for the remaining 24 inefficient AATs (λ_j-weights can be found in Supplementary Appendix D). Even though the AAT (17.) unsupervised learning reaches θ^∗ = 1, it cannot be considered BCC-efficient because it still shows output shortfalls (5 units for production planning, 9.4 units for quality driver identification, and 0.5 units for process control). All remaining AATs must radially reduce all inputs by their θ^∗-value while additionally decreasing present input excesses and increasing present output shortfalls.³

Table 1. DEA results for Geberit.

AAT	Efficiency	Confidence interval*	Selection	Input slacks		Output slacks
				DPC	MC	PP	QDI	PC
Diagnostic
1. Descriptive	0.909	[0.825 0.908]		1.749	0	0	2.604	0.004
2. Visual	1	[0.880 0.998]	X	0	0	0	0	0
3. Clustering	0.919	[0.855 0.917]		0	0	1.534	1.418	0.153
Predictive
4. Fully-supervised	1	[0.877 0.999]	X	0	0	0	0	0
5. Weakly-supervised	0.898	[0.833 0.896]		0	0.097	0	2.628	0.544
6. Semi-supervised	0.888	[0.827 0.887]		0	0	0.401	2.141	0.450
7. Supervised deep learning	1	[0.876 0.998]	X	0	0	0	0	0
8. Inexact-supervised	0.963	[0.879 0.959]		0	0	0	1.472	0.783
9. Distant-supervised	0.867	[0.818 0.866]		0	0	2.976	1.469	0.328
10. Generative	0.908	[0.851 0.906]		0	1.088	0.500	3.100	0.500
11. Instance-based	0.868	[0.809 0.866]		0.434	0	0	0.472	0.072
12. Rule-based	0.950	[0.883 0.949]		0.041	0	0	0.616	0.066
13. Knowledge-based	0.953	[0.892 0.953]		0	0	0	0.700	0
14. Image-based models	0.894	[0.859 0.893]		0	1.056	5	8.900	0.500
15. Audio-based models	0.922	[0.886 0.922]		0	1.397	5	9.600	0.500
16. Time-series models	1	[0.870 0.999]	X	0	0	0	0	0
17. Unsupervised	1	[0.957 0.999]		0	0	5	9.400	0.500
Prescriptive
18. Reinforcement learning	0.988	[0.926 0.987]		0	0	0.508	1.140	0
19. Inverse reinforcement learning	0.975	[0.896 0.973]		0	0.304	0.400	2.189	0
20. Apprenticeship learning	0.990	[0.910 0.989]		0	0.351	0.400	2.289	0
21. Mathematical optimisation	1	[0.877 0.997]	X	0	0	0	0	0
22. Evolutionary	0.945	[0.849 0.942]		0	0	0.182	0.757	0
23. Markov decision process	1	[0.875 0.999]	X	0	0	0	0	0
24. Hidden Markov models	1	[0.876 0.998]	X	0	0	0	0	0
25. Open loop simulation	0.931	[0.856 0.930]		0	0	0	4.038	0
26. Closed loop simulation	0.900	[0.870 0.899]		0	0	2.161	5.292	0
27. Dynamic programming	0.997	[0.944 0.994]		0.883	0	0	1.324	0
28. Inductive logic programming	0.925	[0.879 0.922]		0	0	0.653	7.107	0
29. Fuzzy modelling	0.873	[0.815 0.872]		0	0	0.433	0	0
30. Stochastic processes	0.935	[0.865 0.932]		0	0.471	0	0.288	0
31. Queuing models	0.862	[0.809 0.861]		0	0	1.046	5.271	0.105

AATs with an x are efficient and part of the selection.

DPC: data preparation costs; MC: modelling costs; PP: production planning; QDI: quality driver identification; PC: process control.

*The confidence intervals are for the bias corrected efficiencies at the 95% level.

There are multiple interpretations of our findings. In line with our expectations, the DEA evaluates mathematical optimization as efficient because these techniques are suitable for process control tasks, for instance, the identification of ideal process parameters. Hidden Markov models are best suited to quality driver identification. These techniques can be used to monitor quality in sequential settings where individual process stages are unobservable. For time-series data, time-series models and Markov decision processes are needed to forecast expected demand from historical production volumes and then schedule the machine processes accordingly. Visual is evaluated to be efficient because these techniques entail relatively low modelling costs and allow for the easy communication of results to the management. Finally, fully-supervised and supervised deep learning are required in all of the above tasks and are therefore evaluated as BCC-efficient.

Empirical application 2: ABB Turbo Systems AG

In the following, we first provide background information on ABB Turbo Systems AG and report on our findings from the expert interview. Then we present the numerical results from our DEA.

Description

ABB Turbo Systems AG is part of ABB AG, a Swiss-Swedish multinational corporation operating mainly in the areas of robotics, power, heavy electrical equipment, and automation technology. ABB Turbo Systems AG produces turbochargers for vessels, diesel trains, power plant facilities, and large construction vehicles.

The most relevant application areas for the manufacturing branch of ABB Turbo Systems AG are quality inspection and maintenance planning. Quality inspection is especially important to identify potential fine fissures after the friction welding process. Maintenance planning is crucial to reduce production downtime because redundant assembly capacities do not exist. According to our interviews, data preparation and modelling are relevant cost drivers of analytics projects. Data preparation costs are critical because of multiple proprietary interfaces. Modelling costs are relevant because of limited expertise in the field of advanced analytics. Estimation costs are regarded as fairly negligible because of existing hardware capacities.

Based on our findings from the expert interview, our model only includes data preparation and modelling costs, as well as the effectiveness of AATs in quality inspection and maintenance planning.

Results

Table 2 shows the results from our DEA. In this case, six out of 31 AATs show an efficiency rating θ^∗ = 1. However, there are only five BCC-efficient AATs that satisfy both θ^∗ = 1 and ($s^{-*}=0, s^{+*}=0$), namely (2.) visual, (7.) supervised deep learning, (14.) image-based models, (16.) time-series models, and (23.) Markov decision process (λ_j-weights can be found in Supplementary Appendix E). Compared to the case of Geberit Produktions AG, the DEA yields a different selection of efficient AATs, which is tailored to the application requirements of ABB Turbo Systems AG.

Table 2. DEA results for ABB.

AAT	Efficiency	Confidence interval*	Selection	Input slacks		Output slacks
				DPC	MC	QI	MP
Diagnostic
1. Descriptive	0.900	[0.794 0.899]		1.700	0	3.400	1.200
2. Visual	1	[0.856 0.998]	X	0	0	0	0
3. Clustering	0.919	[0.862 0.916]		0	0	1.496	3.077
Predictive
4. Fully-supervised	0.881	[0.813 0.880]		0	1.906	1.100	4.400
5. Weakly-supervised	0.894	[0.823 0.894]		0	0.073	4.700	2
6. Semi-supervised	0.888	[0.819 0.886]		0	0	5.611	1.661
7. Supervised deep learning	1	[0.835 0.999]	X	0	0	0	0
8. Inexact-supervised	0.985	[0.869 0.980]		0	0	0	2.896
9. Distant-supervised	0.993	[0.906 0.988]		0	0	0	2.800
10. Generative	0.908	[0.836 0.907]		0	1.088	2	2.400
11. Instance-based	0.935	[0.845 0.929]		0.727	0	0	1.696
12. Rule-based	0.908	[0.807 0.904]		0	0	0	0.055
13. Knowledge-based	0.862	[0.796 0.859]		0	0	0.712	3.056
14. Image-based models	1	[0.819 0.994]	X	0	0	0	0
15. Audio-based models	0.922	[0.850 0.921]		0	1.397	5.400	1.400
16. Time-series models	1	[0.815 0.995]	X	0	0	0	0
17. Unsupervised	1	[0.951 0.999]		0	0	7.400	10
Prescriptive
18. Reinforcement learning	0.853	[0.809 0.851]		0	0	3.258	7.833
19. Inverse reinforcement learning	0.945	[0.854 0.942]		0	1.224	0	4.300
20. Apprenticeship learning	0.959	[0.868 0.957]		0	1.269	0	4.200
21. Mathematical optimisation	0.812	[0.775 0.810]		0	0	5.291	5.457
22. Evolutionary	0.770	[0.740 0.769]		0	0	5.952	6.880
23. Markov decision process	1	[0.856 0.997]	X	0	0	0	0
24. Hidden Markov models	0.920	[0.827 0.918]		0	0	1.960	0
25. Open loop simulation	0.919	[0.830 0.916]		0	0	2.454	0.178
26. Closed loop simulation	0.880	[0.846 0.878]		0	0	4.655	5.403
27. Dynamic programming	0.815	[0.739 0.812]		0	0	2.664	1.585
28. Inductive logic programming	0.925	[0.871 0.923]		0	0	1.506	6.368
29. Fuzzy modelling	0.790	[0.718 0.788]		0	0	0.349	1.475
30. Stochastic processes	0.881	[0.786 0.880]		0	0.497	5.400	0.500
31. Queuing models	0.862	[0.808 0.860]		0	0	4.316	3.357

AATs with an x are efficient and part of the selection.

DPC: data preparation costs; MC: modelling costs; QI: quality inspection; MP: maintenance planning.

*The confidence intervals are for the bias corrected efficiencies at the 95% level.

The selection can be interpreted as follows. Our decision model evaluates supervised deep learning and image-based models as efficient because these techniques are best suited to visual quality inspection tasks. Supervised deep learning techniques can automatically learn features from high-dimensional data and hence are especially suited to learning patterns from unstructured image or audio data (Wang et al. 2018; Wuest et al. 2016). Time-series models and Markov decision processes are suitable solutions for predictive maintenance. Finally, visual AATs are also evaluated to be efficient because these techniques allow one to easily communicate results to the management of a traditional manufacturing company.

Summary of findings

The numerical results of Geberit Produktions AG and ABB Turbo Systems AG demonstrate the ability of our decision model to provide customized recommendations regarding the set of efficient AATs to which resources should be allocated. Each DEA evaluates a selection of fairly different AATs. Both selections include visual, supervised deep learning, times-series models, and a Markov decision process. However, for Geberit Produktions AG, our decision model additionally incorporates fully-supervised, mathematical optimization and hidden Markov models into the selection of efficient AATs. This is due to their need for process control and quality driver identification in multi-stage process lines. For ABB Turbo Systems AG, image-based models are further determined to be efficient due to the firm’s requirement for inspection services.

Discussion

Contributions to research

For the production management literature, the DEA represents a powerful tool for multi-criteria decision-making. In the past, the DEA has been widely applied for (i) intra-organizational comparisons (e. g. projects, different sub-units), (ii) inter-organizational comparisons (e.g. companies operating in the same domain), and (iii) longitudinal, panel, or dynamic comparisons of a single unit (Liu, Lu, and Lu 2016). Our study contributes to category (i) by evaluating the relative efficiency of different AATs within a manufacturing company.

Our research extends the literature in at least two directions. First, we study common application areas in manufacturing where advanced analytics can add value. Second, we propose relevant cost drivers and effectiveness measures that can be used to assess the efficiency of different AATs. Ultimately, we address an important management decision problem: in which selection of AATs should a manufacturing company invest? To the best of our knowledge, we are the first to use a DEA model for this purpose.

Generalisability and extensibility

The proposed method builds upon different literature streams on advanced analytics in manufacturing and decision support systems. Hence, we see no theoretical reason, why the proposed model should not be generalizable beyond the cases presented in this paper. The application of the method in the two industrial cases illustrated that the method could be applied and yield concrete insights. Both cases showed how applying the model to the specific environments can support the decision-making on investments in AATs. Following the different modes of generalization as described by Lee and Baskerville (2003), the goal of this study is to test and validate the utility of the presented approach in an actual business setting. Hence, even though the method was tested in only two cases, we posit that the approach is transferable to further use cases not only in manufacturing: Its flexible design can help to solve similar allocation problems for use cases outside of manufacturing management and thus ensures widespread applicability.

The generalisability is shown based on two existing cases. The industrial cases were chosen from the manufacturing sector; specifically from companies implementing digital technologies (i.e. first experience with it). For other companies in this setting, the method will demonstrate similar usability and applicability as in the presented cases.

Further, our approach has similar limitations to previous DEA applications (cf. Dyson et al. 2001). One key assumption for applying the DEA is that the DMUs under evaluation are homogeneous, meaning that they (i) undertake the same activity with similar objectives, (ii) have similar ranges of resources, and (iii) operate in similar environments (Cooper, Seiford, and Tone 2006). However, in our research, AATs are not expected to be fully homogeneous since they are based on different underlying technologies. As a remedy, Cook et al. (2012) and Cook et al. (2013) proposed to break the set of AATs into different groups with those that show the same outputs being in the same group. Within these different groups, separate DEA can be computed. However, the greater the number of required splits, the more difficult it is to estimate meaningful efficiency (Cook et al. 2012). Additionally, in many situations, inputs and outputs are volatile and complex so that they are difficult to measure in an accurate way. One way to include this volatility into the computation of the technology frontier is through stochastic frontier analysis (SFA) (Aigner, Lovell, and Schmidt 1977). SFA assumes stochastic relationships between the inputs used and the outputs produced. Specifically, it allows to assume that deviations from the frontier may reflect not only inefficiencies but also noise in the data. Although this approach seems intriguing, it comes with the drawback of making strong a priori assumptions about the function translating inputs to outputs and about the data generation process. These assumptions are specifically critical in cases with few DMUs, such as ours. It is expected that, with further advances in artificial intelligence, the range of suitable areas is likely to grow. Hence, users should carefully consider adaptations to their company-specific needs. We recommend leaving this task to domain experts (e.g., process engineers) with a background in analytics. They possess the relevant knowledge to assess the suitability of AATs for domain-specific problems.

The strength of our proposed decision model lies in its extensibility. We successfully demonstrate its flexibility by applying it to two manufacturing companies with different requirements. The capacity of our model to be customized according to companies’ needs is a key advantage. This allows management to include different application areas or cost drivers that better represent the decision-making process. Accordingly, the use of our decision model is not only limited to manufacturing management but could be extended to other business units in a straightforward manner.

Managerial implications

This work addresses an important management decision problem: in which advanced analytics techniques should a manufacturing company invest? By selecting a set of efficient AATs, our research addresses the complex task of allocating resources (e.g. financial and human resources) in the domain of advanced analytics.

Our research proposes an approach that can lead to valuable managerial guidance and actions for managers in manufacturing. The recommended selection of AATs can be used along various dimensions. First, managers can improve human resource allocation by hiring data scientists with the necessary qualifications (i.e. those that are part of the selection). Second, decision-makers can steer short-term financial investments into efficient AATs that solve present problems with respect to relevant application areas. Third, managers can later start to invest in AATs that were not found to be BCC-efficient but still show high-efficiency scores (e.g. >0.95). Accordingly, our approach not only benefits financial allocation in the short-term but also provides guidance for long-term strategic development to harness the full potential of advanced analytics in manufacturing.

In practice, choosing the right set of AATs in manufacturing is a complex task. We agree that it requires multiple approaches to do a final selection of AATs (e.g. Villanueva Zacarias, Reimann, and Mitschang 2018; Lechevalier et al. 2018). The proposed DEA model marks a starting point to support the decision process by suggesting efficient solutions. This is an important step as it reduces the complexity of the decision under uncertainty by suggesting a pre-selection out of the long list of existing AATs. However, as it remains as a decision support model, it cannot provide a final solution. Hence, from the pre-selected list of efficient AATs, practitioners need to further investigate with which they want to continue. We further recommend to embed this process in the cross-industry standard process for data mining, a well-established open standard process model for data mining in the industry.

Conclusion

Advanced analytics offers new ways to improve and innovate manufacturing processes. However, the practical implementation of advanced analytics is still in its early stages. Hence, decision-makers are left with the complex task of allocating resources (e.g. financial and human) to a set of efficient AATs that entails the highest company-specific economic benefit.

This paper proposes a multi-criteria decision model to identify a set of efficient AATs to address relevant, company-specific needs. Our research is a pivotal, yet heretofore missing, element to foster the use of advanced analytics in manufacturing. The effectiveness of our decision model was demonstrated in two manufacturing companies, Geberit Produktions AG and ABB Turbo Systems AG. Our numerical results identify a tailored set of efficient AATs for each company. Thereby, we offer a decision support tool that can effectively facilitate resource allocation in the domain of advanced analytics.

Acknowledgements

The authors would like to thank Geberit Produktions AG in Jona and ABB Turbo Systems AG in Baden for their valuable input on this topic, which greatly assisted the research.

Additional information

Notes on contributors

Rafael Lorenz

Rafael Lorenz holds a Ph.D. from ETH Zurich where he graduated from the Chair of Production and Operations Management (POM). His doctoral thesis focused on the effective use of digital technologies to enhance lean production systems. His research was published in leading journals, including the International Journal of Operations & Production Management and the International Journal of Production Research.

Mathias Kraus

Mathias Kraus holds the Assistant Professorship for Data Analytics at the Institute of Information Systems at FAU Erlangen-Nürnberg. Before that, he worked as a research assistant at ETH Zurich and University of Freiburg. The main objective of his research is to develop innovative methodology from the fields of statistics, machine learning, and big data to contribute to further research developments in data-driven decision support across various tasks.

Hergen Wolf

Hergen Wolf studied Industrial Engineering at Technical University Dresden. For his master’s thesis, he joined the Chair of Management Information Systems (MIS) at ETH Zurich. His research focussed on applications of advanced analytics techniques in manufacturing and was published at the APMS. After his studies, Hergen joined Sunfire, a German clean-tech company that develops industrial electrolyzers for the production of renewable hydrogen. As Director Product Management, he is today responsible for Sunfire’s strategic product and service development.

Stefan Feuerriegel

Stefan Feuerriegel is a full professor at LMU Munich School of Management at LMU Munich, where he heads the Institute of AI in Management. Previously, he was an assistant professor at ETH Zurich. In his research, Stefan develops, implements, and evaluates Artificial Intelligence technologies that improve management decision-making.

Torbjørn H. Netland

Torbjørn H. Netland is the Head of Chair of Production and Operations Management (www.pom.ethz.ch) at ETH Zurich, Switzerland, and Co-founder of EthonAI. He is a Member of the World Economic Forum’s Global Future Council on Advanced Manufacturing and Value Chains and a Fellow of the European Academy for Industrial Management. His research on manufacturing excellence appears in leading scientific journals, such as Management Science, Journal of Operations Management, Production and Operations Management, MIT Sloan Management Review, International Journal of Operations & Production Management, and Production Planning and Control. He is co-author of the textbook Introduction to Manufacturing: An Industrial Engineering and Management Perspective (Routledge 2022). He is also the recipient of two Shingo Research Awards and numerous teaching awards.

Notes

1 In this paper, AATs subsume all algorithms from the wider realm of artificial intelligence and machine learning that aid data-driven decision-making.

2 In theory, process control represents a natural application for (safe) reinforcement learning techniques (Sutton and Barto 2018); however, research in this field is still in its infancy and, instead, problem-specific algorithms are usually developed.

3 For instance, in order to be considered BCC-efficient, the AAT (18.) reinforcement learning must decrease its inputs radially by 0.988 and additionally increase its output for production planning by 0.508 units and for quality driver identification by 1.14 units.