Combining the analytical hierarchy process, fuzzy expert systems, and the exponential risk priority number for the holistic evaluation of innovation projects in manufacturing

ABSTRACT

Manufacturing companies operate in a complex environment and need efficient manufacturing processes to remain competitive. Therefore, evaluation methods are essential for decision makers when selecting manufacturing innovation projects (MIPs). However, most approaches are not suitable for strategic use and do not consider all relevant evaluation dimensions. To address this issue, this work presents an approach to evaluate and select MIPs holistically, considering potential, effort, and risk. The approach enables the analysis of the strategic impact of an MIP using a fuzzy expert system, and further evaluates the implementation effort and risk using a combination of the Analytical Hierarchy Process and the Exponential Risk Priority Number. The approach was developed using the results of a systematic literature review and expert-based methods. Finally, the approach was validated in an industrial case study and enabled a transparent evaluation of the strategic potential, effort, and risk of two MIPs, leading to informed project selection.

KEYWORDS:

• Innovation
• project evaluation
• manufacturing
• fuzzy expert systems
• analytical hierarchy process

1. Introduction

Rapid technological developments, global markets, and short product life cycles force manufacturing companies to constantly develop new products and optimise manufacturing costs to defend or gain market share (Kaivo-Oja & Lauraeus, 2018). While the development of functional and high-quality products is crucial for the economic performance of manufacturing companies (Rajapathirana & Hui, 2018), producing these products at the correct cost, with the right quality, and with short lead time is the essential prerequisite. Consequently, delivering high-quality products requires a flexible and efficient manufacturing system with optimised technological, organisational, and human elements (Mack et al., 2016).

To ensure this flexibility and efficiency in the long term, manufacturing systems must be subject to continuous improvement processes and regularly integrate innovation (Dombrowski & Henningsen, 2019). Recent developments in artificial intelligence (Chui et al., 2023) or additive manufacturing (Despeisse et al., 2017) demonstrate how radical these changes can affect manufacturing processes. By selecting appropriate manufacturing innovation projects (MIPs), operational potential regarding cost, time, quality, flexibility, and strategic advantages regarding process knowledge or process transparency can be realised (Gärtner et al., 2023). Therefore, the evaluation and selection of suitable MIPs, such as the use of a new technology or a new equipment layout, is of particular importance to the economic sustainability of a manufacturing company (Chu & Zhao, 2021).

However, the number of MIPs constantly increases, and implementation is usually carried out with limited resources and under time pressure (Bauer et al., 2020). In addition, implementing an MIP is often associated with risks, as the development and implementation phases are characterised by uncertainties (Spur & Eßer, 2010). Hence, it is crucial for managers to select strategically relevant MIPs, given limited financial and temporal resources.

Therefore, evaluation methods are applied to make the selection process transparent and to avoid subjective influences, especially in the case of strategic decisions (Schuh et al., 2021). Theoretical studies aim to provide decision makers with suitable approaches to enable a holistic and strategic evaluation and selection of MIPs to optimise the use of resources and innovation potential.

Consequently, the scientific objective of this study is to provide a combined approach for the manufacturing-specific, holistic, and strategic evaluation of early-stage MIPs. Achieving this scientific objective leads to a theoretical and practical contribution. Firstly, the paper aims to present a new evaluation approach and discusses the applicability of different methods for evaluating potential, effort, and risk to provide a basis for future theoretical work. Secondly, the approach is intended to significantly improve the strategic selection of MIPs in manufacturing, enabling companies to make targeted use of their resources.

The paper is structured as follows. Section 2 provides a general overview of the basic concepts of IM. In Section 3, the state of the research is identified and analysed based on a literature review, and a research gap is systematically derived. Section 4 then presents the research methodology to develop the evaluation approach. In Section 5, the approach and its application in the industrial case study are presented. Section 6 discusses the results, identifies limitations, and presents an outlook on future research.

2. Fundamentals of innovation evaluation in manufacturing

The term innovation was first characterised by Schumpeter (1961), who defined innovation as the combination of input factors for developing new products and processes. In addition, scientific publications agree that a distinction between invention and innovation is essential to define the concept of innovation (Gärtner et al., 2022; Kaschny et al., 2015). An invention is an idea for a new or improved product or process (Kaschny et al., 2015). The subsequent implementation or commercialisation of this product or process is called innovation. In summary, a manufacturing innovation is the successful implementation of a technological or process-related idea into the manufacturing system that improves the performance indicators of the system or the quality of the manufactured products.

Innovation management (IM) combines technological and managerial activities into an overarching corporate function (Hofer, Brandl, et al., 2020). Therefore, IM aims to manage innovation activities efficiently to achieve long-term competitive advantages by delivering innovations at the right time, in the right quality, and at low cost (Vahs et al., 2023). Furthermore, IM in manufacturing can be divided into four phases: preparation, development, implementation, and experience (Spur et al., 2013). Thereby, innovation preparation includes the identification of innovation needs, innovation ideas, and the definition of innovation projects (Spur et al., 2013). In the context of limited resources, MIPs must be evaluated to select the most suitable ones from those defined.

The research fields of technology management (TM) and change management (CM) are components of IM and differ in their focus and scope. TM focuses exclusively on technologies for products or manufacturing processes and primarily considers the process phases from identification to the completion of technology development (Hofer, Brandl, et al., 2020). CM focuses on the transformation of a system or organisation and is concerned with the process phases from innovation development to implementation (Koch et al., 2016).

The selection of project alternatives can become complex due to considering multiple evaluation criteria, dependencies between criteria, and individual preferences (Tzeng & Huang, 2011). Hence, multi-attribute decision making (MADM) methods are applied to evaluate a limited number of alternatives, considering multiple criteria and preferences (Tzeng & Huang, 2011). Among the most common MADM methods are the Analytical Hierarchy Process (AHP) (Saaty, 1980), the Analytical Network Process (ANP) (Saaty, 2006), the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) (Beckmann et al., 1981), and the Multi-criteria Optimization and Compromise Solution (VIKOR) method (Opricovic & Tzeng, 2004).

The AHP was introduced by Saaty (1977) and aims to structure and solve decision problems through a hierarchical structure focused on an overall objective. Generally, the AHP is carried out in five consecutive steps (Saaty, 1980). In the first step, the problem hierarchy is determined by the user. Subsequently, the weighting of the evaluation criteria and the evaluation of the alternatives is carried out. For this purpose, $\mathit{n}\times \mathit{n}$ matrices are used (see Equation 1) (Saaty, 1980).

(1) $\mathit{n}=\left[\begin{array}{cccc}1& a_{12}& \dots & a_{1\mathit{n}}\\ a_{21}& 1& \dots & a_{2\mathit{n}}\\ ⋮& ⋮& \ddots & \dots \\ a_{\mathit{n}1}& a_{\mathit{n}2}& \dots & 1\end{array}\right]$(1)

To determine the priorities of the elements, $a_{\mathit{ij}}$ is expressed according to a fixed scale to represent the importance or evaluation of an alternative in row $\mathit{i}$ in relation to column $\mathit{j}$. The matrix is reciprocal, and $a_{\mathit{ij}}=\frac{1}{{\mathit{a}}_{\mathit{ji}}}$ for each $1\le \mathit{i},\mathit{j}\le \mathit{n}$ applies.

The next step is to perform a consistency check. For this purpose, the method developed by Saaty (1977) can be used, where a consistency index and a consistency ratio are calculated, which should be less than 10% to indicate acceptable consistency. A detailed description of the procedure can be found in the publications by Saaty (1977, 1980).

Subsequently, each decision alternative’s weights or priority values are calculated using the eigenvalue method. Here, the priority vector $\mathit{p}$ is determined by solving Equation (2)(2) $\mathit{Ap}=\mathit{np}$(2) . Thereby, $\mathit{A}$ is the $\mathit{n}\times \mathit{n}$ matrix, $\mathit{p}$ is the eigenvector, and $\mathit{n}$ is the dimension of $\mathit{A}$. The values of the eigenvector $\mathit{p}$ are then the weights of the criteria or the priorities of the alternatives.

(2) $\mathit{Ap}=\mathit{np}$(2)

In the final step, the overall priority of a decision alternative is calculated by an additive aggregation of the weights and priority values of the respective alternatives for the criteria under consideration. This additive aggregation is expressed by Equation (3)(3) $P_i=\sum _j{w}_j\cdot p_{\mathit{ij}}$(3) , where $P_i$ is the global priority of alternative $\mathit{i}$, $p_{\mathit{ij}}$ is the local priority for criterion $\mathit{j}$ and $w_j$ is the weight of criterion $\mathit{j}$ (Saaty, 1980).

(3) $P_i=\sum _j{w}_j\cdot p_{\mathit{ij}}$(3)

In addition to the AHP, fuzzy set theory (FST) is often used to handle uncertainties in decision making processes, as demonstrated by A. K. Sahu et al. (2024) and N. K. A. K. Sahu et al. (2018). Furthermore, it can be applied to model trade-offs between different objectives (Zadeh, 2015). FST provides a way of modelling human thought processes by combining the knowledge of various experts into a rule base called a fuzzy expert system (FES). Within this rule base, expert knowledge is stored using linguistic ‘if … then …’ statements to evaluate fuzzified input parameters and draw conclusions using output parameters.

Therefore, linguistic variables are linked by membership functions ${\mathit{\mu }}_a$ that can map values between zero, meaning no membership, and one, meaning full membership. Membership functions are mostly modelled by triangle functions that represent the assignment of the input value ${\mathit{\mu }}_a\left(\mathit{x}\right)$ to the output value $\mathit{u}\left(\mathit{x}\right)$. The value range of the membership functions is normalised, where $\mathit{u}\left(\mathit{x}\right):\mathit{x}\to \left[0,1\right]$ applies (Bothe, 1993). Figure 1 exemplarily visualises a triangular function. These membership functions are based on rules formulated by experts and represent a linguistic range of values on a numerical scale. For example, an expert might describe a correlation between a potential and a strategic goal in terms of low, medium, and high. At the same time, the membership functions are defined as visualised in Figure 1. A sharp input of 2.5 would then use the terms low and medium.

Figure 1. Fuzzy system with sharp input and output values.

The main linguistic operators for linking fuzzy sets are NOT, AND, and OR, which must be expressed as mathematical operators in the fuzzy system. In practical applications, minimum operators for the AND links and maximum operators for the OR links have proven reliable due to their low computational complexity (Traeger, 1994). Lastly, the defuzzification is considered. To this end, the most common defuzzification method is the centroid method, which outputs the x-coordinate of the area’s centre of gravity (Bothe, 1993).

To identify the uncertainties inherent in a project or product, an evaluation of risk is carried out. A widely used method in the manufacturing industry is the Failure Mode and Effects Analysis (FMEA) approach (Bujna & Kielbasa, 2019). FMEA allows to categorise failure modes by analysing their severity (S), probability of occurrence (O), and probability of detection (D). Therefore, the S, O, and D factors for the defined failure modes are evaluated to calculate a risk priority number for each failure mode. Chang et al. (2014) have extended the FMEA method by introducing the Exponential Risk Priority Number (ERPN) to eliminate fundamental problems such as duplication of results.

3. Literature review

A systematic literature review (SLR) was conducted to identify the state of the research, following the methodology outlined by Rowley and Keegan (2020). The literature review was performed by a research associate with several years of experience in TM and IM under the methodological guidance of a professor of industrial management and assembly technology.

In preparation for the SLR, keywords and search terms (Table 1) were derived from the scope of research outlined in Section 2. Subsequently, the search terms were combined logically using Boolean operators to formulate search strings. The search engine of the selected databases filtered the articles based on the alignment of title, abstract, and keywords with the constructed search strings. Most articles and conference proceedings were collected from Scopus and were the initial source for generating preliminary findings. In addition, literature searches were conducted using the databases provided by Web of Science, ScienceDirect, Google Scholar, TUM University Library, and Fraunhofer Publica.

Table 1. Inclusion criteria for the systematic literature review.

Inclusion criteria	Description
Search terms	MADM, MCDM, Assessment, Evaluation, Decision making, Approach, Innovation, Technology, Change, Project, Manufacturing, Production, Process, Shop floor, Factory
Subject areas	Engineering, Decision Science, Economics, Business, Management, Computer Science
Year	1986–2023
Languages	English, German

The SLR methodology was carried out sequentially to identify the most relevant publications (Figure 2). Using the inclusion criteria listed in Table 1, more than 10,000 publications were identified initially. In the first step, publications were excluded by reviewing their titles, abstracts, and tables of contents. In this way, the number of relevant publications was reduced significantly. The remaining publications were collected and included in the final full-text analysis. A publication was not included in the second step if the full-text analysis indicated it was irrelevant. In the subsequent second step, a backward and forward snowballing approach was used to identify additional relevant contributions by examining references for further essential contributions and reviewing subsequent citations (Watson & Webster, 2020). Following this structured approach, 39 relevant approaches from the research areas of IM, TM, and CM were extracted and included in the final analysis of the state of the research.

Figure 2. Overview of the systematic literature review approach and the results.

Initially, four evaluation criteria were defined to analyse the approaches identified systematically:

(C1) Manufacturing focus: Is the approach tailored to manufacturing by manufacturing-specific evaluation criteria and dimensions?

(C2) MADM method: What MADM method is applied, and does the approach use a single MADM method, or are different evaluation methods combined?

(C3) Holistic evaluation: Does the approach include the evaluation dimensions of potential, effort, and risk?

(C4) Strategic applicability: Do the results of the approach provide strategically relevant information, and does the approach use qualitative expert judgment or quantitative data as input?

Based on the defined criteria, four findings could be drawn from analysing the identified approaches. Firstly, it became clear that most of the approaches analysed focus on manufacturing and use manufacturing-related evaluation criteria. However, in the field of IM, most approaches focus on evaluating products or product ideas. The approaches from Mitchell et al. (2022) and Chen et al. (2020) can be presented as examples.

Secondly, it was found that more than half of the approaches use a single MADM method to evaluate projects or technologies. The AHP emerged as the most widely used method due to its transparency and ease of application, as demonstrated by Bayazit (2005) and Yurdakul (2004). In addition, FST is often used to incorporate linguistic parameters or store expert knowledge, as presented by Wang et al. (2017), Ordoobadi (2008), and Brönner (2020). Nevertheless, combining multiple methods is also used in about one-third of the approaches analysed and is particularly common in IM and CM.

Thirdly, it was observed that most contributions from IM focus on evaluating project potential and risk. The approaches from Horoz (2017) and Liang and Li (2008) can be stated as examples. In contrast, fewer approaches include effort-related criteria in the evaluation, as presented by Mitchell et al. (2022) and Gärtner et al. (2023). A similar picture emerged from the analysis of TM approaches. Most contributions consider potential related to products or processes in their evaluation. The approaches of Schuh et al. (2021), Hofer et al. (2021), and Schuh et al. (2020) can be considered representative. By contrast, only a few approaches include criteria related to effort or risk in their evaluation, as presented by Larsson et al. (2018) and Farooq and O’Brien (2010). Looking at CM approaches, it became noticeable that the evaluation of operational and strategic potential is emphasised, as presented in the approaches from Singh et al. (2023), Luthra et al. (2017), and Plehn et al. (2016). However, the approaches from Yong (2006), Mousavi et al. (2013), Bauer et al. (2017), Yurdakul (2004), and Brönner (2020) still consider cost- and risk-related evaluation criteria.

Fourthly, it became apparent that about half of the approaches analysed already integrate strategic evaluation factors. In the field of IM, the approaches from Stefanovic et al. (2013), Wang et al. (2017), Arabshahi and Fazlollahtabar (2019), Chen et al. (2020), Mitchell et al. (2022), and Romano et al. (2023) include product-relevant strategic aspects such as market growth or scalability. In TM and CM, strategic impacts of projects on manufacturing and products, such as flexibility and quality, are considered, as presented by Rebentisch et al. (2016), Schuh and Scholz (2019), and Brönner (2020). However, it was found that none of the approaches analysed establishes a link between the innovation project and the strategic goals of a manufacturing company.

Furthermore, the analysis of the input data used in the evaluation approaches showed that most approaches use qualitative expert judgment, converted into quantitative data with the help of scales. The approaches from Hofer et al. (2021) and Schuh et al. (2023) can be presented as an example of this. However, some approaches also integrate quantitative methods into the decision making process, such as simulation modelling or profitability calculations, as presented by Chan and Abhary (1996), Horoz (2017), and Lukic et al. (2017).

After concluding the analysis, the approaches of Romano et al. (2023), Schuh et al. (2023), and Singh et al. (2023) were identified as representative and recent approaches of the research areas IM, TM, and CM. For this reason, these approaches will be presented in more detail.

Romano et al. (2023) present a method for evaluating and prioritising product-specific innovation projects based on the theoretical foundations of quality function development and FST. The approach consists of the main steps of data collection, criteria weighting, and project evaluation. The authors provide 40 evaluation criteria, divided into four main groups, and a scale for evaluating the maturity of each criterion. The user weighs and evaluates the desired criteria using linguistic variables. Fuzzy logic is applied to reduce uncertainty and subjectivity in the weighting and evaluation to provide a more robust basis for decision making. However, the structured approach does not allow a holistic assessment of MIPs because it uses only product-specific criteria. Furthermore, it is not possible to include quantitative data in the evaluation, which limits its application flexibility.

Schuh et al. (2023) present a method for potential-oriented evaluation of technologies to support technology selection decisions in early phases. For this purpose, potential dimensions are systematised, and nine potential classes and their interdependencies are derived. The individual potential classes are then evaluated by analysing technology-specific criteria within each class and by analysing quantitative data. However, the further use of these results within the methodological selection process remains to be detailed. Furthermore, the approach focuses mainly on potential-related criteria, thus conceiving a one-sided evaluation. In addition, transferability and applicability are severely limited as the evaluation steps are not presented in detail, and a high level of data availability is a prerequisite.

Singh et al. (2023) present a hybrid MADM methodology that combines FST, ANP, TOPSIS, and the VIKOR method. ANP and FST model and evaluate dependencies between evaluation criteria and alternatives considering uncertainty and ambiguity. Subsequently, the TOPSIS and VIKOR methods rank the alternatives considering positive and negative influencing factors. However, the approach does not enable a holistic and strategic evaluation, as the criteria and their strategic relevance are unclear. In addition, the possibility of integrating quantitative data is not demonstrated. Finally, the industrial applicability is limited by the complexity of the combination of methods and the need for initialisation by several experts in each application.

The analysis of the identified approaches based on the defined criteria C1 to C4 is summarised in Table 2.

Table 2. Overview of the analysed approaches from the state of the research.

	Author	C1	C2		C3			C4
		Manufacturing focus	Single method	Method combination	Potential	Effort	Risk	Strategic point of view	Integration of quantitative data
Innovation management	Liang and Li (2008)	✓	✓		✓	✓	✓
	Stefanovic et al. (2013)		✓		✓	✓	✓	✓
	Chang et al. (2014)		✓				✓
	Lukic et al. (2017)	✓	✓		✓				✓
	Wang et al. (2017)	✓		✓				✓
	Horoz (2017)			✓	✓		✓		✓
	Arabshahi and Fazlollahtabar (2019)		✓				✓	✓
	Chen et al. (2020)			✓	✓			✓
	Mitchell et al. (2022)		✓		✓	✓		✓	✓
	Romano et al. (2023)			✓		✓	✓	✓
	Gärtner et al. (2023)	✓		✓	✓	✓	✓
Technology management	Chan and Abhary (1996)	✓		✓	✓				✓
	Bayazit (2005)	✓	✓		✓		✓	✓
	Chan et al. (2006)	✓	✓		✓
	Ordoobadi (2008)		✓			✓	✓		✓
	Farooq and O’Brien (2010)	✓		✓	✓		✓
	Becattini et al. (2015)	✓	✓		✓			✓
	Aliakbari Nouri et al. (2015)	✓		✓	✓		✓
	Rebentisch et al. (2016)		✓		✓			✓	✓
	Larsson et al. (2018)	✓	✓		✓	✓		✓
	Schuh and Scholz (2019)		✓		✓		✓	✓
	Essakly et al. (2019)	✓	✓		✓
	Ghaleb et al. (2020)	✓	✓		✓
	Schuh et al. (2020)	✓	✓		✓			✓
	Hofer et al. (2021)	✓	✓		✓			✓
	Schuh et al. (2021)	✓	✓		✓			✓
	Schuh et al. (2023)		✓		✓		✓	✓
Change management	Yurdakul (2004)	✓	✓		✓			✓
	Yong (2006)	✓		✓	✓	✓
	Aydin and Kahraman (2010)	✓		✓	✓				✓
	Vinodh et al. (2011)	✓	✓		✓
	Yang and Deuse (2012)	✓		✓	✓				✓
	Mousavi et al. (2013)	✓		✓		✓			✓
	Plehn et al. (2016)	✓	✓		✓			✓
	Bauer et al. (2017)	✓	✓			✓		✓	✓
	Luthra et al. (2017)	✓		✓	✓
	Bujna and Kielbasa (2019)	✓	✓				✓
	Brönner (2020)	✓	✓		✓			✓
	Singh et al. (2023)	✓		✓	✓

After reviewing the publications identified and interpreting the results, four shortcomings (SCs) in the current state of the research have been identified.

Firstly, most of the analysed approaches use a single MADM approach. In conclusion, combining several evaluation methods in a well-structured and understandable way to take advantage of different methodologies is not yet sufficiently adopted (SC1).

Secondly, most approaches fail to provide a holistic evaluation of MIPs considering all relevant dimensions. It became apparent that the dimensions of effort and risk are only addressed by a few contributions (SC2).

Thirdly, it was found that the minority of the contributions use quantitative data to evaluate MIP alternatives in terms of manufacturing performance. Furthermore, the combination of qualitative expert judgment and quantitative data is not addressed sufficiently (SC3).

Fourthly, manufacturing-specific evaluation approaches carry out potential analyses focusing on the manufacturing unit or the products manufactured and do not draw any conclusions about the strategic potential of the MIP from the company’s point of view (SC4).

In summary, it was found that none of the approaches analysed was able to address all the shortcomings identified. Therefore, research is needed to develop a structured approach to MIP evaluation tailored to the manufacturing context, combining several evaluation methods and considering all three dimensions: potential, effort, and risk. In addition, the evaluation approach should be applicable in the early phase of the innovation process, incorporating qualitative expert judgment and quantitative data and generating strategically valuable information. The formulated research need should form the basis for the subsequent development of the evaluation approach and for differentiating it from existing approaches.

4. Methodical approach

The development of the approach followed a four-phase procedure according to the Design Research Methodology (DRM) presented by Blessing and Chakrabarti (2009) (Figure 3). The approach was developed as part of a doctoral thesis, so the phases were carried out based on a type 3 design research project, as suggested by Blessing and Chakrabarti (2009).

Figure 3. Scientific methodology for the development of the evaluation approach.

The research objective is derived in the Research Clarification (RC) phase. Therefore, an SLR was used as a scientific method (Section 3). The second phase, the Descriptive Study I (DS I), aims to evaluate the prerequisites for a manufacturing-specific project evaluation approach. Therefore, it focuses on identifying, selecting, and weighing evaluation criteria relevant to manufacturing. Hence, three scientific methods were applied. Firstly, an SLR was used to identify an initial list of criteria. Secondly, semi-structured expert interviews were conducted to select and refine the relevant evaluation criteria. Thirdly, the list of criteria was validated in an expert workshop. To conclude the DS I, a suitable weighting method was selected from the methods identified in the RC phase.

The Prescriptive Study (PS) phase aims to provide modules for the holistic evaluation of MIPs and an approach for project selection. Therefore, three scientific methods were applied to derive the evaluation modules and the selection approach. Firstly, an SLR was conducted to identify existing MADM approaches and methods for visualising data. Secondly, the relevant requirements for the evaluation modules were formulated based on the SCs of the state of research and the results of DS I. Subsequently, the identified MADM and visualisation approaches were analysed regarding their applicability, and suitable methods were selected. Thirdly, an FES was developed to approximate the relationships between the manufacturing potential of an MIP and the strategic goals of a manufacturing company. For this purpose, a two-stage Delphi study was conducted with industrial experts.

In the fourth phase, the Descriptive Study II (DS II), the approach for the holistic evaluation and selection of MIPs was validated in an industrial case study.

5. Approach for the holistic evaluation of manufacturing innovation projects

5.1. General concept of the approach

The approach consists of three main steps: evaluation preparation (EP), project evaluation (PE), and project selection (PS). Figure 4 shows a schematic representation of the approach based on the Business Process Model Notation.

Figure 4. Overview of the processual concept of the evaluation approach.

In the EP step, the relevant evaluation criteria of the MIP are selected, and the weights are calculated. The EP step is performed only once to ensure that all MIPs are evaluated using the same criteria and weights.

In the PE step, a holistic evaluation is performed by determining each MIP’s potential, effort, and risk based on the previously defined criteria and their weights. Both qualitative expert judgment and quantitative data can be used to evaluate potential and effort. Project risk is evaluated by defining failure modes and using qualitative expert judgment. The PE step is repeated for each MIP to be evaluated. As a result, the strategic potential, the required effort, and the associated risk scores for each MIP are available to the decision maker.

The PS step presents the results using appropriate graphs to enable transparent decision making. Therefore, the decision maker is provided with a portfolio overview and detailed insights into the evaluation results of each MIP.

5.2. Evaluation preparation

The EP step aims to provide a criteria catalogue for the holistic evaluation of MIPs. Furthermore, a weighting method must be provided to ensure the incorporation of preferences from companies or individuals in the evaluation approach.

The scientific approach and the catalogue of evaluation criteria have already been published by Gärtner et al. (2023). Therefore, the scientific procedure and results are summarised in the following.

Firstly, a SLR was carried out, which identified 121 relevant approaches. Secondly, eight semi-structured expert interviews were conducted with representatives from the manufacturing industry to specify the criteria identified. Thirdly, the criteria catalogue was verified in a workshop with eight researchers from a German research institute for production technology. As a result of this structured approach, the main criteria of potential, effort, and risk were systematised into 11 categories with 34 sub-criteria and a total of 123 detailed criteria (Figure 5). The first main criterion, potential, describes the changes to the manufacturing system that can be expected after the implementation of an MIP. The second main criterion, effort, describes the financial and time-related effort required to develop and implement the MIP. The third main criterion, risk, represents the uncertainties associated with developing and utilising the MIP. The full criteria catalogue is presented in the publication by Gärtner et al. (2023).

Figure 5. Structure of the criteria catalogue (Gärtner et al., 2023).

In addition, the EP step aims to allow individual decision makers to express preferences. Consequently, a subjective weighting method must be included in the approach (Zardari et al., 2015). Therefore, the pairwise comparison method was chosen because of its transparency and comprehensible procedure.

In summary, the following steps are carried out in the EP phase:

• Selection of the relevant detailed criteria for the evaluation dimensions of potential, effort, and risk from the criteria catalogue provided by Gärtner et al. (2023).

• Prioritisation of the selected detailed criteria, sub-criteria and criteria according to the method of Saaty (1980).

• Prioritisation of the FMEA factors S, O, and D according to the method of Saaty (1980).

• Carrying out the consistency check, according to Saaty (1980).

• Calculate the weights for the selected detailed criteria, sub-criteria and criteria following the method of Saaty (1980).

5.3. Project evaluation

The PE step aims to enable a holistic evaluation of MIPs. Therefore, appropriate MADM methods must be identified, selected, and combined.

Firstly, the results of the SLR presented in Section 3 were used to identify the most relevant MADM methods. Secondly, the identified methods were evaluated according to general and specific requirements to select the most suitable ones. The general requirements are based on the publication of Lindemann (2003) and include the method’s comprehensibility, adaptability, and ease of application. Furthermore, specific requirements are based on the identified SCs and include the appropriate combination of suitable methods, a holistic evaluation, the integration of various data, and a strategic perspective.

Subsequently, the evaluation modules were designed in three steps. In the first step, it was defined that the evaluation approach should consist of three individual modules, considering potential, effort, and risk, to fulfil the specific requirement for a holistic evaluation.

In the second step, appropriate MADM methods were selected to carry out the evaluation within each module. The AHP was found to be suitable for all evaluation modules for three reasons. Firstly, unlike the ANP, the AHP allows the hierarchical structure of the evaluation criteria catalogue to be represented, as already demonstrated by A. K. A. K. Sahu et al. (2018). Secondly, as emphasised in the approach from N. K. Sahu et al. (2015), the AHP is characterised by its comprehensible and flexible applicability compared to other MADM methods to ensure the broadest possible industrial applicability. Thirdly, the AHP provides an excellent capability to integrate qualitative expert judgment and quantitative data, as stated by N. K. Sahu et al. (2015).

In the third step, FES, feature scaling, and FMEA were selected as suitable methods to combine with AHP. An FES was selected, as it allows trade-offs between manufacturing objectives and strategic objectives of the manufacturing company to be correlated. Furthermore, an FES can weigh and adapt these trade-offs according to the individual company, thus allowing a broad industrial application. Additionally, an FES enables the storage and application of expert knowledge on strategic implications, as the database is usually uncertain in the early stages of an MIP. Therefore, the FES was identified as an appropriate method to determine the strategic relevance of an MIP, thus satisfying the specific requirement of strategic applicability. Consequently, the potential of an MIP is defined as the input parameter, and the strategic goals of manufacturing companies are defined as output parameters for an FES. Additionally, feature scaling enables using quantitative data in the MIP evaluation to meet specific requirements.

Furthermore, FMEA was chosen for two reasons. Firstly, FMEA is the most widely used method for risk assessment in manufacturing, which facilitates the overall industrial applicability of the approach. Secondly, FMEA offers the possibility of considering risks in a differentiated and company-specific manner through three factors and weightings. Consequently, it was found that the FMEA meets the specific requirements of the evaluation approach.

In summary, all three evaluation modules are based on AHP but are extended by integrating additional methods. Thus, the potential evaluation module combines the AHP with an FES and the concept of feature scaling. The effort evaluation module is based on the AHP in combination with the concept of feature scaling. Finally, the risk evaluation module combines the AHP and FMEA. In conclusion, all three modules satisfy the specific requirement for the combination of suitable methods.

5.3.1. Module 1: potential evaluation

The potential evaluation module consists of five elements that are carried out sequentially. The first element, the initialisation, focuses on selecting the relevant detailed criteria for the MIP evaluation and calculating the respective weights. Therefore, the elaborated criteria catalogue is used to select detailed criteria, and a pairwise comparison is performed to obtain the respective weights (see Section 5.2).

The second element involves the definition of transformation rules to enable the use of quantitative data. For this purpose, feature scaling is applied, and mathematical rules are defined to allow transformation from unit-bound to dimensionless values. For each evaluation criterion, where unit-bound quantitative data is available, a best case, worst case, and median case of expected change is defined. Subsequently, equally distributed intervals are formed between these three reference points, allowing unit-bound data to be transformed into an equivalent on a dimensionless scale.

The third element is the evaluation, where the AHP is used to evaluate the potential of an MIP in manufacturing. Therefore, a Likert scale from one to nine is used, as proposed by Saaty (1977). Thus, a score of one represents the lowest level of performance, while nine represents the highest level. Where quantitative data is available, the transformation rules are applied to derive the equivalents on the Likert scale. Otherwise, expert judgment is used, and the scores of several experts may be aggregated using the median to increase reliability. To include the status of a manufacturing system, its current performance indicators are defined as the reference scenario and included in the comparison of the MIPs under consideration.

With the selected detailed criteria, weights, and evaluation values, the AHP allows the calculation of scores for the manufacturing potential of each MIP regarding cost, quality, time, changeability, social, ecology, and strategy (Figure 6).

Figure 6. Exemplary application of the AHP in the potential evaluation module.

In summary, the following steps are carried out to compute the manufacturing potential of each MIP:

• Prioritisation of the respective MIPs concerning the selected detailed criteria according to the method of Saaty (1980).

• Perform the consistency check according to Saaty (1980).

• Calculation of the priority scores of the respective MIPs regarding the selected detailed criteria according to the method of Saaty (1980).

• Normalisation of the priority scores of the respective MIPs to the initially used range.

• Calculation of the priority scores of the respective MIPs for the sub-criteria and criteria by additive aggregation of the weights and priority scores of the detailed criteria according to the method of Saaty (1980).

The evaluation of the strategic potential of the MIP is the fourth element, and it aims to provide a conclusion on the extent to which an MIP meets the overall strategic goals of a manufacturing company. Therefore, an FES, which consists of three components, is used.

Firstly, the input parameters resulting from the AHP are transformed into fuzzy sets using linguistic variables. Hence, the fuzzification process converts the input parameters into a value on the interval between zero and one by applying a set of membership functions. Consequently, five membership functions are considered to convert the input into fuzzy numbers, which are expressed in the form of five linguistic terms: significant decrease, decrease, no change, increase, and significant increase. These linguistic terms correspond to a set of five triangular fuzzy numbers on a scale of zero to ten, where each membership function has an interval of length four.

Secondly, a set of fuzzy rules is applied between input and output parameters based on expert knowledge. The definition of these fuzzy rules is based on the generation of ‘if … then …’ statements, consisting of a predecessor representing the input and a successor representing the output. In this approach, the input is represented by the AHP results, and the output is based on the 14 general strategic goals of manufacturing companies as proposed by Schuh et al. (2021). As suggested by Lin et al. (2012), domain mapping matrices (DMM) with the size $\mathit{n}\times \mathit{m}$ for $\mathit{n}$ input and $\mathit{m}$ output parameters in combination with a two-stage Delphi study (Mahajan et al., 1976) were used to derive the fuzzy rules.

The Delphi study was conducted in two stages with 15 experts from different roles, companies, and industries to avoid bias (Table 3). The stated input and output parameters formed the DMM used in the study.

Table 3. Overview of the participating experts of the two-stage Delphi study.

Expert	Job	Company	Industry
Expert 1	Project Manager	A	Aviation
Expert 2	Project Manager	B	Consulting
Expert 3	Founder	C	Consulting
Expert 4	Industrial Engineer	A	Aviation
Expert 5	Project Manager	D	Engineering
Expert 6	Manager	E	Consulting
Expert 7	Production foreman	F	Automotive
Expert 8	Senior Consultant	G	Consulting
Expert 9	PhD Associate	H	Research
Expert 10	Line Manager	I	Healthcare
Expert 11	PhD Associate	J	Research
Expert 12	Senior Consultant	G	Consulting
Expert 13	Professor	K	Research
Expert 14	Line Manager	A	Aviation
Expert 15	Director	L	Engineering

In the first stage of the Delphi study, each expert independently identified the correlation between manufacturing potential and the strategic goals of a manufacturing company. For this purpose, an 11-point Likert scale ranging from zero to ten was used in the DMM. A score of zero indicates no correlation, and ten indicates a strong correlation.

After identifying the cause-effect relationships in the DMM, each expert determined the associated rules individually. Therefore, the input and output parameters were compared using a domain structure matrix (DSM), and the parameter characteristics were described using a membership function. Thus, the relationship between two parameters is represented by an ‘if … then …’ statement (Figure 7).

Figure 7. Example of a rule definition for the FES.

At the end of the first stage of the Delphi study, the experts’ input was summarised by calculating the median and standard deviation for each value in the DMM and the individual DSMs. The summary was then sent to the experts involved. In the second stage of the Delphi study, the experts were interviewed a second time using the same procedure as in the first stage. In conclusion, all participants confirmed that the summarised results accurately reflected the relationships between MIP potential and the strategic goals of manufacturing companies.

However, several experts noted that the number of dependencies was too high. Consequently, correlations with a median value of less than or equal to five should not be further considered to obtain a more transparent set of fuzzy rules. Furthermore, it was suggested that the values of the remaining correlations should be redistributed across the initial range of zero to ten to achieve a better gradation between individual rules. The DMM presented in Figure 8 shows the results of the Delphi study, outlining 52 relationships between MIP potential and the strategic goals of manufacturing companies. The numerical values indicate the strength of the relationship between the elements, while the colours indicate the type of DSM. A dark green box indicates a DSM with a linear relationship between the potential factor and the strategic goal (Figure 9a). A light green field represents a DSM with an S-shaped relationship between the potential factor and the strategic goal (Figure 9b).

Figure 8. Results of the two-staged Delphi study: interdependencies between manufacturing potential and strategic goals of manufacturing companies.

Figure 9. a) Characteristics of the linear input-output relationship, b) Characteristics of the S-shaped input-output relationship.

After conducting the two-stage Delphi study, the results were used to define an FES containing 260 unique fuzzy rules. Within the inference process, the FES is used to transform the fuzzy inputs into fuzzy outputs. Therefore, the FES applies fuzzy logic operators to the input parameters and rules to derive the degree of membership of each rule in the output. In this approach, the logical operator AND is the minimum operator, and the logical operator OR is the maximum operator. When a particular rule of the system is triggered, an implication relation is used to calculate a fuzzy number. This calculation is based on the degree of the membership function of the input and output parameters. The maximum operator is applied to aggregate the outputs of each triggered rule. Subsequently, the fuzzy system applies defuzzification to convert the output of the FES into crisp values. Therefore, the centroid method is used.

Finally, in the fifth element, the fuzzy system returns 14 strategic potential scores to the user, determining the degree of fulfilment of each strategic goal on the Likert scale from one to nine. The closer the score is to nine, the more the implementation of the MIP enables that strategic goal to be achieved.

5.3.2. Module 2: effort evaluation

The second module enables an evaluation of the financial and temporal one-off effort to develop and implement an MIP and consists of four elements. The first element, the initialisation, focuses on selecting the relevant detailed criteria from the elaborated criteria catalogue and calculating the respective weights (see Section 5.2).

The second element involves transforming the unit-bound input data into dimensionless values. Therefore, feature scaling is applied by defining mathematical transformation rules, as described in Section 5.3.1. In contrast to the feature scaling of the potential evaluation module, only a maximum value is defined for the financial or temporal effort, which is assigned to the value of nine on the Likert scale.

In the third element, the evaluation of the temporal and financial effort required for each MIP is carried out. Where quantitative data is available, the transformation rules are applied to derive the equivalents on the Likert scale. Otherwise, expert judgment is used, and the scores of several experts may be aggregated using the median to increase reliability. A score of one indicates that no effort is required, while a score of nine indicates that the MIP requires maximum effort. Finally, by applying the AHP, a result for the financial and time-related effort can be obtained (Figure 10). A detailed description of the AHP computation steps is given in Section 5.3.1.

Figure 10. Example of the AHP calculation steps in the effort evaluation module.

The resulting effort score forms the fourth element and determines the financial and temporal effort required to develop and implement an MIP. The closer the score is to nine, the more effort is required.

5.3.3. Module 3: risk evaluation

The risk evaluation module allows the evaluation of the uncertainties inherent in MIPs during development and use due to internal and external factors. It consists of four elements.

The first element, the initialisation, focuses on selecting relevant detailed risk criteria from the elaborated criteria catalogue. Furthermore, the respective weights for the criteria and the risk factors S, O, and D are calculated. The methodological approach is described in Section 5.2.

In the subsequent second element, failure modes are defined for each selected detailed criterion using expert-based methods such as expert workshops. In the third element, the AHP is applied in combination with the extended FMEA proposed by Chang et al. (2014). Therefore, each failure mode and the risk factors S, O, and D are evaluated for each MIP with the help of experts using a Likert scale ranging from one to ten. For S, a score of one indicates no risk, while a score of ten indicates that the failure mode would be hazardous to the overall project. For O, a score of one indicates that the occurrence of the failure mode is impossible, while a score of ten indicates that the risk of occurrence is extremely high. For D, a score of one indicates that timely detection is inevitable, while a score of ten indicates that detection is uncertain. Subsequently, the ERPN is calculated for each failure mode using the formula proposed by Chang et al. (2014) (see Equation 4).

(4) $\mathit{ERPN}=3^{w_s\ast \mathit{S}}+3^{w_o\ast \mathit{O}}+3^{w_D\ast \mathit{D}}$(4)

If more than one failure mode is defined for a detailed risk criterion, the failure mode with the highest ERPN is considered further. Subsequently, the AHP is applied using the ERPN for each failure mode and the respective weights to obtain a final score for development, utilisation, and external risks (Figure 11). The fourth element of the risk evaluation module is the return of the ERPN to the user, describing the risk of each MIP.

Figure 11. Example of the AHP calculation steps in the risk evaluation module.

In summary, the following steps are carried out to compute the manufacturing risk of each MIP:

• Definition of failure modes for the selected detailed risk criteria.

• Prioritisation of the respective MIPs concerning the risk factors S, O, and D and the defined failure modes according to the method of Saaty (1980).

• Perform the consistency check according to Saaty (1980).

• Normalisation of the risk factors S, O, and D for the defined failure modes of the respective MIPs to the initially used range.

• Calculate the ERPN of the respective MIPs regarding the defined failure modes according to the method of Chang et al. (2014).

• Calculate the ERPN of the respective MIPs for the sub-criteria and criteria by additive aggregation of the weights and ERPN of the detailed criteria according to the method of Saaty (1980).

5.4. Project selection

The PS step aims to visualise the results of the PE step to enable a differentiated project selection, considering the potential, effort, and risk. Therefore, different visualisation options were identified, and an appropriate visualisation was derived (van der Aalst et al., 2012).

A hybrid visualisation approach was chosen. On the one hand, a portfolio overview is generated, which compares the MIPs’ potential, effort, and risk using a bubble chart. Thereby, the vertical axis represents the expected strategic potential of the MIP, while the horizontal axis depicts the effort required to implement the MIP. The bubble size includes a third dimension and represents the level of risk inherent to the MIP. On the other hand, an overview of the detailed evaluation results from the PE step is included. Therefore, the most relevant evaluation results regarding potential, effort, and risk are presented in detail to provide insight for decision makers. Figure 12 shows an exemplary dashboard for the hybrid visualisation approach of the evaluation results.

Figure 12. Exemplary visualisation of the portfolio overview and the detailed evaluation results.

Finally, the approach enables decision makers to select MIPs that best meet the company’s strategic goals, require an acceptable level of effort, and reflect the company’s risk tolerance.

5.5. Case study

The industrial case study was carried out in collaboration with a semiconductor company, and two MIPs were evaluated using the developed approach. The need for innovation stems from the high cost of silicon carbide (SiC) based products compared to their silicon (Si) based equivalents. The availability of SiC wafers is limited to a specific diameter, and changing the diameter of the wafers presents many technical challenges. However, manufacturing systems are currently designed and optimised to produce Si wafers. Therefore, semiconductor manufacturers must match the available size of SiC wafers with the existing equipment. To address this issue, the company has identified two potential MIPs: Project A and Project B. Project A involves embedding the SiC wafer onto another carrier wafer. In contrast, Project B involves retrofitting the line when a certain number of wafers of the current unproduced diameter is collected at the front end. The proposed evaluation approach was applied in three phases (Figure 13).

Figure 13. Phases of the industrial case study.

The evaluation preparation was carried out in three steps. Firstly, the criteria catalogue was used, and four semi-structured expert interviews were conducted with experts from the company to derive the relevant detailed criteria. Secondly, the selected evaluation criteria were verified with the experts in a workshop. Subsequently, the company’s preferences were incorporated by weighing the criteria using pairwise comparisons. The criteria considered and their respective weights are shown in Table 4.

Table 4. Criteria and weights used in the industrial case study.

Main criteria	Weight (local)	Subcriteria	Weight (local)	Detailed criteria	Weight (local)
Cost	0.2	Organisation	0.5	Production planning	1
		Production	0.5	Processing	0.5
				Supply	0.5
Quality	0.2	Process	1	Yield	0.5
				Failure rate in the field	0.5
Time	0.2	Waiting & layover	0.5	Waiting & layover	1
		Processing	0.5	Processing	1
		Planned downtime	0	Cleaning	1
Changeability	0.2	Flexibility	1	Volume flexibility	1
Strategy	0.2	Process competency	1	Strategic & implicit knowledge	1
Financial	0.5	Technical development	0.33	Prototyping	0.33
				Testing & validation	0.33
				Integration & ramp-up	0.33
		Structures	0.33	Utilities & tools	0.5
				Conversion & installation work	0.5
		Qualification & consulting	0.33	Process re-qualification	1
		Opportunity	0	Production downtime	1
Temporal	0.5	Technical development	0.5	Prototyping	0.33
				Testing & validation	0.33
				Integration & ramp-up	0.33
		Structures	0.5	Conversion & installation work	1
		Qualification & consulting	0	Process re-qualification	1
Innovation project (realisation)	0.33	Time	0.5	Duration of use	1
		Organisation	0.5	Communication & knowledge management	1
Production (utilisation)	0.33	Technological	1	Technological fit	0.5
				TRL achievement	0.5
External risk	0.33	Market	1	SiC wafers for large-scale production	1

Subsequently, the project evaluation was carried out for Project A and Project B. To this end, a discrete event simulation model was created using AnyLogic to derive quantitative data for the potential evaluation.

The model simulates the flow of SiC wafers with different dimensions through the internal manufacturing processes. A pre-determined number of wafers are collected in each manufacturing batch and then run through the entire process. The characteristics of each batch vary according to wafer size, including processing time, number of chips per wafer, and cost of raw material. In setting up the simulation model, assumptions were made with the experts about material costs, conversion costs (e.g. labour and energy), and processes (e.g. production planning, process times, and process availability). As a result, the simulation model provided quantitative data on the performance of the projects and the reference scenario in terms of processing costs, delivery costs, waiting time, retrofitting time, and processing time. The derived data was then transformed into dimensionless values using transformation rules verified by the internal experts. The remaining detailed criteria were evaluated through semi-structured interviews with four experts.

Additionally, a software application was developed using MATLAB to perform the weight calculation, the AHP, and calculate the output using the FES. The user is guided through the evaluation process within the software application, and the necessary values are collected and stored in a database (Figure 14).

Figure 14. Overview of the software application for the industrial case study.

In the third phase, the evaluation results were visualised to provide insight into the potential, effort, and risk of the MIPs. Therefore, Microsoft Power BI was used to create a dashboard containing the project portfolio and a detailed insight into the results (Figure 15). Several conclusions could be drawn from the evaluation results. It was found that Project A has a very positive impact on the performance indicators of the manufacturing system and the achievement of strategic goals compared to the reference scenario. By applying the FES, the strategic implications of Project A could be specified very well, and it became clear that Project A particularly addresses strategic goals (e.g. business expansion and first-mover advantages) relevant to an innovative company that wants to gain market share. In contrast, Project B has a neutral potential compared to the current situation. The strategic impact analysis showed that Project B particularly supports the goals of a company that wants to act defensively to protect its market share (e.g. quality improvement). However, due to the high technological complexity of Project A, it is associated with higher financial and time-related effort and implementation risk. The holistic evaluation approach provided insights that although the implementation of Project A would involve a very high technical effort and risk in terms of development, integration, and employee training, it could be implemented approximately as quickly as Project B in terms of development time.

Figure 15. Portfolio overview and detailed MIP evaluation results of the industrial case study.

6. Discussion

6.1. Summary

Manufacturing companies constantly strive to achieve greater efficiency and transparency in their manufacturing processes to produce at lower cost, with shorter lead time, or with better quality. Technological or processual innovation must be regularly integrated into an existing manufacturing system to enable this continuous improvement process. However, as manufacturing companies have limited resources for implementing MIPs, decision makers are regularly forced to evaluate MIPs and select the most suitable ones. To structure this process, MADM methods are used to evaluate a limited number of projects in the presence of multiple criteria. However, there is no manufacturing-specific evaluation approach that includes different types of inputs and considers the three evaluation dimensions of potential, effort, and risk. Furthermore, the existing approaches do not enable any conclusions to be drawn about the strategic impact of an MIP from the companies’ perspective. Therefore, this contribution aims to provide an evaluation approach that enables a holistic evaluation and selection of MIPs, filling the identified research gap.

To develop the approach, the DRM was applied based on a type 3 research project. To this end, an SLR was conducted to identify the state of research, derive SCs, and define a research objective. In the subsequent DS I, a criteria catalogue for evaluating MIPs was elaborated using literature- and expert-based methods. Furthermore, an appropriate weighting method was selected. In the subsequent PS step, suitable evaluation methods were identified and combined to develop three modules for evaluating the potential, effort, and risk of an MIP. In addition, a visualisation of the evaluation results was proposed to enable transparent project selection. Finally, the developed approach was implemented in a software tool and applied in an industrial case study (DS II). In this way, the approach was validated, and limitations and starting points for future research were derived. In summary, the core result is a manufacturing-specific and strategically applicable approach for the holistic evaluation and selection of MIPs, which combines the AHP, an FES, and FMEA and allows the use of qualitative expert judgment and quantitative data.

6.2. Contributions

From a theoretical point of view, the approach represents an extension of the current state of research, as it addresses the identified SCs.

The analysis of the approaches identified has shown that combining different evaluation methods is helpful, as it allows the advantages of the individual methods to be combined (SC1). However, care must be taken to ensure industrial applicability and adaptability. This work addresses this gap by combining the AHP with an FES and FMEA into an evaluation method that sensibly combines the advantages of different methods. On the one hand, the AHP is easy to apply, considers the evaluation criteria’s hierarchical structure, and allows for incorporating individual preferences. On the other hand, an FES enables the description of strategic potential, and FMEA allows a manufacturing-specific risk evaluation. Based on this contribution, the developed approach does not contradict existing scientific approaches but rather represents a further application of the fundamental methods presented by Saaty (1977), Zadeh (2015), and Chang et al. (2014). In addition, the results of the approach are in line with the approaches of Wang et al. (2017), Aydin and Kahraman (2010), and Brönner (2020), who also found the combination of AHP and FES to be both theoretically and practically relevant. Furthermore, the elaborated approach is consistent with the general trend of combining the AHP with alternative MADM approaches to take advantage of different strengths, as presented by Luthra et al. (2017), Mousavi et al. (2013), and Yang and Deuse (2012).

Moreover, it became clear that existing approaches don’t enable a manufacturing-specific and holistic evaluation of innovation projects (SC2). Therefore, two theoretical contributions were provided in this approach. Firstly, a manufacturing-specific catalogue of criteria was integrated into the evaluation approach. Secondly, three evaluation modules were developed to evaluate the potential, effort, and risk of MIPs. The developed approach, therefore, represents a further use and recombination of existing scientific approaches, as it integrates the approach of Gärtner et al. (2023) and combines the evaluation dimensions presented by Schuh et al. (2021), Romano et al. (2023), and Arabshahi and Fazlollahtabar (2019).

In addition, the analysis of the existing literature showed that few approaches allow the use of quantitative data and almost exclusively use qualitative expert judgment as input (SC3). To address this gap, this approach indicates that the integration of transformation rules into the evaluation approach is promising to allow quantitative data to be used as input in combination with expert judgment. As a result, the approach represents an extension of the state of the art. It helps to combine simulation-based approaches, as presented by Chan and Abhary (1996), and expert-based approaches, as presented by Hofer et al. (2021). Furthermore, the results show that the application of AHP is suitable for the integration of qualitative and quantitative data and is therefore consistent with the approaches of Chan and Abhary (1996), Aydin and Kahraman (2010), and Lukic et al. (2017).

Furthermore, the literature analysis showed that many of the identified evaluation approaches are not able to draw conclusions about the strategic potential of an MIP from the company’s point of view (SC4). The approach addresses this gap and shows that using an FES is suitable to estimate the strategic effects of an MIP based on stored expert knowledge and empirical values. In addition, an FES is suitable for fast adaptation to a specific industry or company. Thus, the approach is in line with the general trend to use FST within MADM to model trade-offs between strategic goals, store expert knowledge, deal with limited data and uncertainty, or incorporate qualitative judgment as presented by Ordoobadi (2008), Brönner (2020), and Singh et al. (2023).

In practice, decision makers are often challenged to use resources effectively. The aim is to achieve the greatest possible operational and strategic efficiencies in manufacturing, which in turn lead to competitive advantages for the company. To achieve this, evaluation methods must be able to evaluate the potential, effort, and risk of the respective MIPs in a comprehensive and comparable way. However, existing methods usually focus on only one of these three evaluation dimensions, require specific data and application knowledge, or do not provide strategically useful information.

To address this practical gap, the method enables companies to perform a holistic evaluation and selection of MIPs. It also presents an FES to draw conclusions on the strategic impact of an MIP. The case study showed that the holistic evaluation provides significant added value as decision makers became aware that higher potential is associated with higher effort and risk during implementation. Furthermore, MIPs can be selected more specifically according to strategic aspects, as the approach depicts the impact of MIPs on the company’s strategic goals. The case study highlighted the importance of aligning the strategic potential of the MIP with the strategic goals of the company, as this allows projects to be selected in a much more targeted manner. Therefore, the result of this work enables practitioners to evaluate MIPs holistically and make an informed decision on resource allocation.

Consequently, two critical conclusions for managers could be drawn from the results. Firstly, project selection should only be carried out by taking a holistic view of the relevant evaluation criteria to identify dependencies between these for each decision alternative. Secondly, the impact of the decision alternatives on the company’s strategic objectives should always be considered when analysing the potential to decide in line with the optimal overall strategy.

In summary, the elaborated approach for the holistic evaluation and selection of MIPs contributes to theoretical and practical purposes. The results suggest that the evaluation and selection of MIPs should take a holistic and strategic perspective and allow the integration of quantitative data to enable informed decisions at the beginning of the innovation process.

6.3. Limitations

In the third step of the discussion, four limiting aspects of the approach are highlighted. Firstly, literature and expert-based methods such as the SLR and the Delphi study were used to develop the approach. Using alternative search terms or including alternative experts may lead to a different selection of methods or an alternative FES. However, the multi-stage research process, the careful selection of search terms, and the representative composition of the expert panel ensure that the research results are still valid.

Secondly, the developed FES represents only an initial approach that needs to be adapted for industry- and company-specific applications. This adaptation of the FES may require the identification of implicit expert knowledge and the acquisition of specific knowledge on FST, which may limit the industrial application.

The third limitation is that the risk evaluation can only be conducted with the help of qualitative expert judgment. The conducted review could not identify a method that enables the integration of quantitative data into the risk evaluation.

Fourthly, the developed approach has been implemented as a prototype and applied only in one industrial case study. In the context of the chosen research methodology, this constitutes sufficient validation. However, applying the method in multiple case studies could lead to further insights and starting points for refinement and future research.

6.4. Future research

Concluding, three starting points for future research could be identified. Firstly, the development of new methods for the strategic evaluation of MIP effort and risk could be of interest. Future contributions could therefore focus on the applicability of MADM methods for the strategic evaluation of these dimensions and extend the approach developed. Secondly, the integration of quantitative data into strategic MADM could be further researched. Hence, future research could focus on integrating simulation models and data farming approaches into the holistic evaluation of early-stage MIPs. Thirdly, integrating sensitivity analysis into the developed approach could be a further development. This way, the decision maker could understand how the evaluation results might change with different weights and input data. Consequently, future research could focus on developing and integrating a sensitivity analysis and further improving the robustness of the evaluation approach.

Author contributions

Conceptualisation, Q.G.; methodology, Q.G.; software, A.B. and H.M.; validation, Q.G., A.B. and H.M.; formal analysis, A.B. and H.M.; investigation, A.B. and H.M.; resources, Q.G. and A.B.; data curation, Q.G., A.B. and H.M.; writing – original draft preparation, Q.G.; writing – review and editing, Q.G., A.B. and H.M.; visualisation, Q.G.; supervision, G.R.; project administration, Q.G. and G.R.; funding acquisition, G.R. All authors have read and agreed to the published version of the manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy (StMWi) under Grant ROB-2-3410.20-04-10-21.