Physics-based and data-driven hybrid modeling in manufacturing: a review

ABSTRACT

Manufacturing, an industry set in the physical world, is undergoing its digital transformation, also known as the fourth industrial revolution. Sensors, connectivity, and platforms provide unprecedented access to quantify the quality and diversity of manufacturing data. Progress in data-driven modeling is exponential across all industries. This leads to the question of how physics-based and data-driven modeling can be utilized in a hybrid modeling approach to advance our understanding of processes, materials, and systems in manufacturing. In this review, we focus on discrete manufacturing based on the understanding that hybrid modeling is more mature in process manufacturing. This paper aims to provide an overview of projects where hybrid modeling was used in manufacturing and introduce various ways of composing hybrid models. We provide examples highlighting the implementation of models, structure and expand on metrics to test and validate hybrid models, discuss challenges, and future research directions of hybrid modeling in manufacturing.

KEYWORDS:

• AI
• machine learning
• hybrid modeling
• smart manufacturing
• hybrid analytics

1. Introduction

Manufacturing as an industry has embraced data and machine learning applications for a long time Harding et al. (2006). Manufacturing operations and their dynamics are diverse and complex across the supply chain. Thus, various machine learning (ML) approaches should be adapted for different processes and problems to enable smart manufacturing systems. ML models rely on available data to produce reliable, high-quality, value-adding results. These data-driven models have proven especially useful and powerful across industries and are incredibly transformative in data-rich (high quantity) and knowledge-sparse areas.

However, manufacturing is centered around the physical transformation of materials at its core. There is a long history of dedicated work to understand the physics behind manufacturing operations.

For example, we can understand the significance of parameters influencing grinding processes using detailed first-principle equations to determine specific grinding energy Sutherland et al. (2018). Physics-based models make certain assumptions, thus making them challenging to work with (e.g. insignificance of parameters, idealized constants), while data-driven methods have issues with data volume, variety, velocity, and veracity. Data-driven models also require a sufficient amount of data from all classes for training. This can be problematic in manufacturing as operators work to eliminate defects, so defect data may be hard to come by. Many machine learning algorithms perform better with balanced datasets (e.g. 50% good, 50% bad parts) Wuest et al. (2016). Yet, this is not a realistic approach in all situations, as a manufacturing process resulting in 50% scrap is far from competitive.

Physics-informed hybrid machine learning approach or hybrid modeling, which is defined as the combined use of physics-based and data-driven models to achieve more accurate and physically consistent predictions by leveraging the advantages of both physics-based and data-driven methods, Karpatne et al. (2017); Kapusuzoglu and Mahadevan (2020) could help address these issues by leveraging the strengths of both approaches. Hybrid models often outperform purely data-driven models when the physics components do not contain oversimplifications, incomplete modeling, insufficient parameters, or incorrect assumptions.

While prior reviews have surveyed standalone physics-based Cubillo et al. (2016); Bayat et al. (2021) or data-driven models Wang et al. (2022); Cerquitelli et al. (2021) in manufacturing, this paper specifically focuses on reviewing applications of the hybrid of physics and data-driven modeling to provide manufacturing researchers and practitioners with a comprehensive overview of how hybrid modeling can be leveraged as an emerging tool. This analysis focuses specifically on combinations of physics-based and data-driven models, not combinations of multiple physics or data-driven models.

The primary aim of this paper is to outline the current practices in manufacturing optimization and introduce readers to why a physics-informed hybrid machine learning approach might outperform traditional ML optimization procedures. It offers a general perspective on the integration of physics and data-driven models, followed by a more detailed exploration of specific objectives and how the reviewed papers align with these objectives. Additionally, it discusses quantifiable measures for evaluating the efficiency of hybrid models and presents challenges encountered in manufacturing, along with suggested solutions. This article aims to inform readers about the possibilities when using these physics and data-driven hybrid combinations in manufacturing optimization. Some papers discuss hybrid modeling in general and in other industries. However, our review concentrates on applications within the manufacturing domain, focusing on the specific objectives the researchers accomplished and how they fit into the broader context of physics and data-driven hybrid combinations. We categorize and discuss these papers under their respective sections, although several papers align with multiple integration objectives.

The remainder of this paper is organized as follows: Section Two provides a brief background on the topic and explains the methodology of the review. Section Three presents the main results of the review and provides different hybrid modeling approaches and applications in manufacturing systems. In Section Four, challenges and future research directions are discussed. Finally, we present conclusions drawn from the previous sections in Section Five. This paper aims to cohesively synthesize learning and best practices to guide further research and the real-world deployment of hybrid manufacturing models.

2. Background and review methodology

Understanding the underlying physics of manufacturing processes is the key to enabling optimization. Several traditional tools exist that can be used for manufacturing process optimization. Heuristics Chauvet et al. (2003), simulation Aksarayli and Yildiz (2011), lower-dimensional modeling Wattamwar et al. (2008), Six Sigma Naeem et al. (2016), lean manufacturing Sabadka et al. (2017), and the design of experiments Beg et al. (2019) are a few examples of such methods. These methods aim to optimize manufacturing processes, supply chains, and pipelines. With improved hardware and software, data accessibility across processes has increased and continues to improve. However, supply chain optimization often focuses on minimizing variation for maximum stability. This limits data variety, as experiments that introduce variability are avoided due to the costs of time, resources, and effort. Limited data variety makes modeling difficult.

In this article, we define ‘physics’ as any model, heuristic, or algorithm that is explainable, avoiding the black-box nature of machine learning models. Thus, the papers presented here do not consider hybrid black box, data-driven, or physics-based models in isolation. Instead, we consider a combination of an explainable physics model and a black-box data-driven model. With this focus, we also demonstrate how the integration of physics and data-driven models can work together to achieve various objectives. The authors of this paper have worked on hybrid modeling for energy-efficient CNC grinding and have effectively applied some of these techniques. With minimal access to the machine elsewhere, we achieved the objective of at least a 15% energy reduction in a CNC grinding machine using these methods. Additionally, for a particular part, we achieved a 41% reduction in time and a 37% reduction in energy consumption.

Scopus and Web of Science (WoS) serve as curated sources of high-quality academic articles, and for this reason, we selected these two databases for our literature study. To capture all relevant instances of published research featuring combinations of physics and data-driven hybrid modeling, we began by including synonyms of the involved terms, as illustrated in Figure 1. Our search specifically targeted hybrid modeling scenarios based on the combination of physics and data-driven approaches. Thus, we considered terms that could signify the same concept. Our approach was informed by a previously conducted bibliometric analysis Kasilingam et al. (2021) that focused on hybrid modeling in manufacturing, related terminology, and the most commonly used methods and processes. This analysis helped define the review methodology, constraints, time range, and search string used in this article.

Figure 1. Overview of the published hybrid models in the manufacturing domain.

The defined search term returned 251 unique articles from WoS and Scopus. Following a comprehensive manual review, we checked if these articles indeed featured a combination of physics and data-driven models, as required by our inclusion criteria. This process reduced the final number of articles to 81. Any articles lacking sufficient information on the physics portion or how the integration occurred were eliminated. Furthermore, generic digital twin articles that did not provide details on implementing manufacturing processes were also excluded. An overview of the article review methodology is presented in Figure 2.

Figure 2. Methodology of review of hybrid modeling in manufacturing articles.

3. Hybrid modeling approaches and applications

Integrating physics knowledge into data-driven models has numerous applications in the manufacturing domain, and there are multiple methods to achieve this integration. In this section, we will explore various approaches found in the literature, which can be employed to infuse physics knowledge into data-driven models and create hybrid models. We will also highlight different studies that have utilized these methods within the manufacturing domain. All references in this section, whether cited in the text or presented in table format, constitute the main results extracted from our literature review of 81 papers.

In our reviewed paper, we have identified the top five manufacturing categories with the highest frequencies where hybrid models were applied, as illustrated in Table 1. It comes as no surprise that machine monitoring is the field with the most applications published. It should be pointed out that additive manufacturing and welding have some overlapping similarities when it comes to the models used in metal additive manufacturing, another indicator that the availability and quality of physics models is a crucial factor for the successful application of hybrid models. Below is a high-level overview of how the physics and data-driven models can be integrated.

Table 1. Top five manufacturing categories across reviewed papers.

Category	Count	Reference
Machine Monitoring	17	Yucesan and Viana (2020); Destro et al. (2020); Sun et al. (2020) Qi and Park (2020); Elkhashap et al. (2020) Nannapaneni and Mahadevan (2016); Tidriri et al. (2016) Destro et al. (2020); Yucesan and Viana (2020); Sadoughi and Chao (2019) Yumei et al. (2017); Tian et al. (2015); Pillai et al. (2016); Agana et al. (2018) AlThobiani et al. (2014); Junbo et al. (2015); Z. Chen et al. (2017)
Additive Manufacturing	8	Moges et al. (2021); Ren et al. (2019) Haghighi and Lin (2020); Spiegel et al. (2019) Kapusuzoglu and Mahadevan (2020); Yazdi et al. (2020) Gaikwad et al. (2020); Yang et al. (2017)
Tool Wear	8	Lotfi et al. (2016); Zhang et al. (2021) Mei et al. (2019); Hanachi et al. (2019); Sun et al. (2018) Śpiewak et al. (1999); Luo et al. (2020); Hong et al. (2019)
Welding	8	Pasandideh et al. (2014); Tabar et al. (2020) Olleak and Zhimin (2020); Mondal et al. (2020); Sim (2018) Reinhart et al. (2017); Zhang et al. (2016); Pittner et al. (2008)
Chemical Processes	4	Bhutani et al. (2006); Glassey and Von Stosch (2018) Xiong and Jutan (2002); Slatteke and Astrom (2005)

3.1. Integrating physics knowledge into machine learning models

Manufacturing, as an industry that is fundamentally centered around the manipulation of physical objects, has a long history of dedicated work to understand the physics of its processes and operations. The performance of hybrid models, when compared to purely data-driven models, is often better when the physics model component does not contain oversimplifications, partial modeling, insufficient parameters/representation, or incorrect assumptions. Mechanistic models are typically derived from conservation laws (material, momentum, and energy), as well as thermodynamic, kinetic, and/or transport laws forming the foundation of many physics-based models in a manufacturing context. Below are examples of how physics and data-driven models can be integrated.

Physics-Guided Loss Function is the first approach that can be employed. This can be done by incorporating physics knowledge into data-driven models by integrating physical knowledge into the ML loss functions. This helps ML models capture dynamic patterns that align with established physical laws. The addition of physics-based loss aims to ensure consistency with physical laws. It is weighted by a hyperparameter, $\mathit{\gamma }$, which is determined alongside other ML hyperparameters through validation data or a nested cross-validation setup. Regularization by physical constraints allows the model to learn even with unlabeled data, as the computation of the physics-based loss does not require observational data.

Physics-guided initialization is another approach. Neural network weights are often initialized according to a random distribution before training. Poor initialization can cause models to become stuck in local minima, especially for deep neural networks. However, if physical or other contextual knowledge is used to inform weight initialization, model training can be accelerated, and fewer training samples may be required. One possibility is to use simulated data from the physics-based model to pre-train the ML model. Physics-guided initialization can also be accomplished using a self-supervised learning method. One approach involves using a physics-based model to simulate intermediate physical variables, which can then be used to pre-train ML models by adding supervision to hidden layers.

Finally, we can employ a physics-guided design of ML architecture. New ML architectures can be designed to leverage the specific characteristics of the problem at hand. The modular and flexible nature of neural networks makes them ideal candidates for architecture modification. For example, domain knowledge can be used to specify node connections that capture physics-based dependencies among variables. One way to incorporate physical principles into neural network design is to assign physical meaning to specific neurons in the neural network. It is also possible to explicitly declare physically relevant variables. Another related approach is to fix one or more weights within the neural network to physically meaningful values or parameters and make them non-modifiable during training. Recurrent neural networks (RNNs) encode temporal invariance, and convolutional neural networks (CNNs) can implicitly encode spatial translation, rotation, and scale invariance. Similarly, scientific modeling tasks may require other invariances based on physical laws. Architecture modifications incorporating symmetry are also commonly seen in dynamic systems research involving differential equations. Raissi and Em Karniadakis (2018) demonstrated that the covariance function can explicitly encode the underlying physical laws expressed by differential equations to solve Partial Differential Equations (PDE) and learn with smaller datasets.

3.2. Hybrid models applications in manufacturing

Here we will discuss the utilization of available fundamental knowledge to account for the impact of certain factors and provide structure to a comprehensive model. This knowledge enables a clear distinction between established, validated relationships and new variables essential for data-driven or statistical models, in addition to physics-based models. We will elaborate on the integration of physics and data-driven models, offering real-world examples from manufacturing. Hybrid models find applications in various scenarios, including refining existing models, enhancing output resolution, replacing interconnected component models, optimizing computational efficiency, solving partial differential equations, uncovering static characteristics, generating synthetic data, and estimating output uncertainty based on input. This categorization highlights the diverse applications of physics models in the papers under consideration. Interested readers are referred to the seminal book about hybrid modeling in process industries Glassey and Von Stosch (2018). Below, we will go over each application and provide examples of each category in our literature review.

3.2.1. Improving or replacing physics model

While physics models are grounded in established physical laws, they often serve as approximations of reality due to our incomplete understanding of certain processes, resulting in inherent biases. Furthermore, these models frequently rely on a multitude of parameters, the values of which must be estimated using limited observational data, further compromising their performance, particularly when dealing with the heterogeneity of processes across different spatial and temporal scales. The limitations of physics-based models transcend disciplinary boundaries and are widely acknowledged in the scientific community. In contrast, machine learning models, such as neural networks, can discern intricate structures and patterns even in complex scenarios where explicit programming of a system’s exact physical characteristics is unfeasible, provided they are supplied with a sufficient volume of data. Given this ability to automatically extract complex relationships from data, ML models appear promising for scientific problems with physical processes that are not fully understood but have data of adequate quality and quantity available. However, the black-box application of ML has met with limited success in scientific domains for several reasons. Integrated physics-ML models are expected to better capture the dynamics of scientific systems and advance the understanding of underlying physical processes Nouri et al. (2022); Hermann et al. (2022); Zhou et al. (2021); Guo et al. (2021); Huang et al. (2022); Kim et al. (2022). Table 2. Shows more examples of improving physics-based models using data-driven techniques.

Table 2. Examples where hybrid models used to improve or replace physics models.

Authors	Methodology	Application	Improvement
Papacharalampopoulos et al. (2021)	Improved physics model using ML + Buckingham-Pi	Laser Welding	Improved physics model
Yucesan and Viana (2020)	NN + Physics	Bearings	modeled the L10 fatigue life
Tabar et al. (2020)	Surrogate Model	Spot Welding	Improved optimization process
Rajesh et al. (2017)	Physics-Embedded ML	Electrical Discharge Machining	Improved metal removal rate
Lotfi et al. (2016)	Geometrical Model + NN	Machining	Accurate chip type prediction
Aivaliotis et al. (2021)	Empirical equations + geometrical Modeling	Predictive Maintenance	Optimized oval-round pass design
Zhang et al. (2021)	High-fidelity simulation + SSAE-PHMM	Tool Wear Prediction	Effective real-time tool monitoring
Yazdi et al. (2020)	Wavelet Transform +DL	Additive Manufacturing	Real-time defect detection
Huang et al. (2020)	Math model + LSTM network	Production Systems	Real-time product completion time prediction
Vorkov et al. (2019)	Analytical model + ML regression	Air Bending	Improved bending allowance prediction
Sun et al. (2018)	Wiener Process + Data-Driven Model	Cutting Tool RUL Prediction	Enhanced accuracy in RUL prediction
ZYang et al. (2017)	Gray-Box Models	Various Applications	Reduced predictive errors
Zhang et al. (2016)	fault tree + Bayesian network	Automatic Welding	Enhanced fault diagnosis accuracy
Nannapaneni and Mahadevan (2016)	Hierarchical Bayesian Network	Manufacturing Process	Improved process performance estimation
Ambrogio et al. (2008)	Neural Networks + FEA simulation	Steel Hard Machining	Modeled Residual stress
Slatteke and Astrom (2005)	IPZ Model + Mass-Energy Balances	Paper Manufacturing	Improved pressure dynamics description
Cho et al. (1997)	Neural Network + Mathematical Model	Rolling Mill	Reduced prediction error
Rahimi et al. (2021)	Kalman filters-based physics model + CNN	Milling Chatter Detection	Improved detection rate

3.2.2. Downscaling

Complex mechanistic models can more precisely capture physical reality than simpler models. This enhanced precision is often achieved by incorporating a wider array of components that account for a greater number of processes at finer spatial or temporal resolutions. However, due to the computational costs and modeling complexities involved, many models are operated at coarser resolutions than what is necessary for accurately representing underlying physical processes. To address this, downscaling techniques have been widely employed to capture physical variables that require modeling at a finer resolution. These downscaling techniques can be categorized into two groups: statistical downscaling and dynamical downscaling.

Statistical downscaling involves using empirical models to predict finer-resolution variables based on coarser-resolution data. This mapping between different resolutions can encompass complex non-linear relationships that traditional empirical models may struggle to precisely represent. Given the significant time required to run these complex models, there is a growing interest in utilizing ML models as surrogate models – models that approximate simulation-driven input-output data – to predict target variables at a higher resolution. While state-of-the-art ML methods can be applied in both statistical and dynamical downscaling, ensuring that the learned ML component remains consistent with established physical laws and enhances overall simulation performance remains a challenge. Table 3. Summarize the studies we found in the literature that used downscaling techniques.

Table 3. Examples where downscaling used in hybrid models.

Authors	Methodology	Application	Improvement
Xiong and Jutan (2002)	Neural network + grey-box-based model	Chemical processes	Reduced predictive errors
Song et al. (2021)	Newtonian kinetics andFEA + DL	Mechanical system vibrations monitoring	Proposed framework and hybrid model
Śpiewak et al. (1999)	EMA + Lumped Parameter System model	Structural analysis	Improved accuracy and reliability

3.2.3. Parameterization

In parameterization, specific complex dynamical processes are simplified into static parameters using physical approximations. One common approach to estimating these parameters involves conducting a grid search across various combinations of parameter values to find the best match with observations, a process known as parameter calibration. Failing to achieve correct parameterization can reduce the model’s robustness, and errors stemming from imperfect parameterization can adversely impact other components of the entire physics-based model, ultimately affecting the modeling of critical physical processes.

An increasingly considered alternative is to replace overly complex processes that cannot be physically represented in the model with simplified dynamic or statistical/ML processes. This allows for the direct learning of new parameterizations from observations and/or high-resolution model simulations using ML methods. This approach has found applications in diverse domains, including additive manufacturing Aljarrah et al. (2022); Liu et al. (2022); Liu and Wang (2022), biopharmaceutical manufacturing Gerogiorgis and Castro-Rodriguez (2021), monitoring Yang et al. (2022), robotics Aivaliotis et al. (2021), smelting Liang et al. (2021), machining Zotov and Kadirkamanathan (2021), and chemical processes Cai et al. (2021). Table 4. shows more examples of parametrization in hybrid models.

Table 4. Examples where parameterization used in hybrid models.

Authors	Methodology	Application	Improvement
Zhang et al. (2022)	Physics-based model for non-GD&T features	Tool wear estimation	Improved tool wear estimation
Moges et al. (2021)	High-fidelity CFD model+ polynomial regression + Kriging methods	Additive manufacturing	Improved melt pool prediction
Hu et al. (2019)	Simulation + SVM	Iron ore sintering	Real-time prediction
Haghighi and Lin (2020)	Time-delayed linear time-invariant model	Electrohydrodynamic Jet Printing	Improved process performance and reliability
Sun et al. (2020)	Nominal + deviation terms	Complex industrial processes	Comprehensive state space modeling
Sim et al. (2018)	Theoretical model + spline interpolation	Manufacturing	Improved nugget size estimation

3.2.4. Reduced-order models

Reduced-Order Models (ROMs) are computationally efficient representations of complex models. Typically, constructing ROMs involves dimensionality reduction to capture the essential dynamical characteristics of often large, high-fidelity simulations and physical systems models. However, this reduction to a lower-dimensional subspace naturally results in a loss of accuracy.

One recent focus in the realm of ML-based ROMs is approximating the dominant modes of the Koopman (or composition) operator as a dimensionality reduction technique. The Koopman operator is an infinite-dimensional linear operator that encodes the temporal evolution of the system state through nonlinear dynamics. While dynamic mode decomposition is the most common technique for approximating the Koopman operator, recent efforts have been made to approximate Koopman operator embeddings using deep learning models, which have shown superior performance compared to existing methods. Incorporating physics-based knowledge into the learning of the Koopman operator can enhance generalizability and interpretability, aspects that current ML methods in this field often lack. Consequently, integrating principles from physics-based models has the potential to reduce the search space, making ROM training more robust and allowing the model to be trained with less data in various scenarios Roland et al. (2021); Farlessyost and Singh (2022). More examples of using ROMs in hybrid modeling are shown in Table 5.

Table 5. Examples where reduced-order models used in hybrid models.

Authors	Methodology	Application	Improvement
Hürkamp et al. (2021)	FEM + Proper Orthogonal Decomposition	Thermoplastic composites	Optimization of bond strength
Kharwar et al. (2022)	Multiobjective physicsmodel + PCA	Nanocomposite polymer manufacturing	Enhancements in quality and productivity
Zhou et al. (2019)	Self-adaptive population genetic algorithm	Iron making	Optimizing conflicting objectives in ironmaking
Qi and Park (2020)	Kinematic model + digital twin model	Grinding machine	Development of a reduced-order model

3.2.5. Forward solving partial differential equations

Finite Element Method (FEM) and Finite Difference Method (FDM) based models are often prohibitively expensive. In such cases, traditional methods may not be ideal or, in some instances, even feasible. A common approach is to employ an ML model as a surrogate for the solution to reduce computation time. Neural networks can significantly reduce the computational demands of traditional numerical methods to a single forward pass of a neural network. Notably, solutions obtained through neural networks are naturally differentiable. They possess a closed analytic form that can be easily applied to subsequent calculations, a feature not typically found in more traditional solving methods.

Taking it a step further, deep neural network models have demonstrated success in approximating solutions for high-dimensional physics-based Partial Differential Equations (PDEs) that were previously considered unsuitable for approximation by ML, as observed in the works of Chen and Ahmadi (2022); Teren et al. (2022); Yang and Özel (2021); Ren et al. (2022). More examples of using forward solving PDEs in hybrid modeling are shown in Table 6.

Table 6. Examples where hybrid models used to solve PDEs.

Authors	Methodology	Application	Improvement
Ahmad et al. (2014)	Gray-box model with Random forests	Metallurgy	Achieving superior performance for molten steel temperature prediction
Haghighi and Lin (2020)	Thermal profilemodel + ANN	Additive Manufacturing	Achieving high accuracy
Pittner et al. (2008)	ANN + phenomenological modeling	Welding	modeling uncertainties of the physics model

3.2.6. Inverse modeling

The forward modeling of a physical system involves utilizing the system’s physical parameters, such as mass, temperature, charge, physical dimensions, or structure, to predict the system’s next state or its output effects. In contrast, inverse modeling employs a system’s potentially noisy output to deduce the underlying physical parameters or inputs. Inverse problems often hold great significance in physics-based modeling communities because they can reveal valuable information that cannot be directly observed.

The solution to an inverse problem can often be computationally expensive due to the potential need for millions of forward model evaluations for estimator assessment or the characterization of posterior distributions of physical parameters. ML-based surrogate models, among other methods like reduced-order models, are emerging as viable options because they can represent high-dimensional phenomena with abundant data and operate considerably faster than most physical simulations. Applying physics-based constraints and stopping conditions based on material properties can guide the optimization process, enhancing data efficiency and the ability to address ill-posed inverse problems. Examples of using inverse modeling in hybrid modeling are shown in Table 7.

Table 7. Examples where inverse modeling used in hybrid models.

Authors	Methodology	Application	Improvement
Destro et al. (2020)	State estimators based on first principles	Fault detection and diagnosis	Superior fault detection and diagnosis performance
Hwang et al. (2020)	Inverse calculation + DNN + GBM + ANN	Coil thickness control	Precise control of coil thickness in hot rolling
Reinhart et al. (2017)	Hybrid modeling withLR + ELM + RBF	Robotics	Effective modeling for control applications
Umbrello et al. (2008)	ANN + FEM	Machining (Hard turning of AISI 52,100 bearing steel)	Low prediction errors
Islam et al. (2022)	Hybrid ML approach with ANN + VSG	Thermoplastic composite manufacturing	Optimization of manufacturing

3.2.7. Discovering governing equations

When the governing equations of a dynamical system are known explicitly, they allow for more robust forecasting, control, and the opportunity for analysis of system stability and bifurcations through increased interpretability. If a mathematical model accurately describes the processes governing the observed data, it can generalize to data outside of the training domain. Advances in machine learning for discovering these governing equations have become an active research area with rich potential to integrate applied mathematics and physics principles with modern machine learning methods Mekarthy (2019). Early works on the data-driven discovery of physical laws relied on heuristics and expert guidance. They were focused on rediscovering known, non-differential laws in different scientific disciplines from artificial data. Recently, work has used sparse regression built on a dictionary of functions and partial derivatives to construct governing equations. These sparse identification techniques are based on the principle of Occam’s Razor, where the goal is to use only a few equation terms to describe any given nonlinear system. Table 8. shows examples of this application in the literature.

Table 8. Examples of discovering governing equations usage in hybrid models.

Authors	Methodology	Application	Improvement
Carrascal and Alberdi (2010)	Hybrid evolutionary approach	Abrasive Water Jet Machining	Proposed a novel approach
Nami et al. (1997)	Neural networks + analytical model	Metal-organic Chemical Vapor Deposition	Improved analytical expression for deposition rate

3.2.8. Data generation

Data generation approaches are valuable for creating virtual simulations of scientific data under specific conditions. Traditional physics-based approaches for data generation often depend on running physical simulations or conducting physical experiments, which can be exceedingly time-consuming. Moreover, these approaches are constrained by what can be produced by physics-based models. Therefore, there is a growing interest in generative ML approaches that learn data distributions in unsupervised settings, offering the potential to generate novel data beyond the capabilities of traditional methods Sarishvili et al. (2021); Relan et al. (2021).

The concept behind generative models is to capture the underlying probabilistic distribution to generate similar data. Recent advances in deep learning have led to the development of new generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). These models have demonstrated significantly improved performance in learning non-linear relationships and extracting representative latent embeddings from observational data. An emerging area of research is the engineering of GANs that can incorporate prior knowledge of physics, including physical laws and invariance properties. The physics-based constraints ensure that the generated artificial samples exhibit the same morphology distribution as authentic data, thus greatly reducing the vast design space required for data generation. Examples of data generation applications in hybrid models are illustrated in Table 9.

Table 9. Examples of using data generation in hybrid models.

Authors	Methodology	Application	Improvement
Cardillo et al. (2021)	intracellular modeling + model-based design of experiments	Data generation for vaccines	Cost-effective data generation
Kapusuzoglu and Mahadevan (2020)	deep neural networks	Additive Manufacturing	predicting bond quality and porosity of FFF parts
Olleak and Zhimin (2020)	Utilize FEM to augment ML model	Additive Manufacturing	highly efficient and accurate melt pool prediction
Gaikwad et al. (2020)	Simulation predictions + in-situ sensor data	3D printed part quality inspection	Improved detection rate

3.2.9. Uncertainty quantification

Uncertainty Quantification (UQ) necessitates the accurate characterization of the entire distribution rather than merely making a point prediction. This approach enables various analyses, such as assessing the proximity of predictions to acceptable values and conducting sensitivity analyses of input features. Within the ML community, Gaussian processes have often served as the primary technique for quantifying uncertainty in simulating physical processes. In contrast, traditional methods like Monte Carlo (MC) are typically unfeasible due to the substantial number of forward model evaluations required to obtain statistically meaningful results when applying uncertainty quantification to physics models.

The integration of physics knowledge into ML for UQ holds the potential to provide a more robust characterization of uncertainty Schoeneberger et al. (2021); Wang et al. (2022); McGowan et al. (2022). For instance, ML surrogate models run the risk of producing physically inconsistent predictions, and the incorporation of physics principles could mitigate this issue. Additionally, it’s worth noting that the reduced data requirements of ML, driven by the necessity to adhere to known physical laws, may alleviate some of the high computational costs associated with bayesian neural networks for UQ. Examples of uncertainty quantification applications in hybrid models are illustrated in Table 10.

Table 10. Examples of using uncertainty quantification in hybrid models.

Authors	Methodology	Application	Improvement
Ren et al. (2019)	Simulation + Gaussian process-based analytical computations	Additive manufacturing	High prediction accuracy
Mondal et al. (2020)	3D analytical model + Bayesian networks	Metal additive manufacturing	Excellent accuracy with uncertainty quantification
Mei et al. (2019)	Multi-domain system analytical model	CNC machine tools	Improved compensation control accuracy
Hanachi et al. (2019)	Adaptive neuro-fuzzy inference system	Not specified	Enhanced accuracy
Xie et al. (2022)	Physics-data hybrid method + heat transfer law in the loss function	Temperature field modeling in energy deposition	Accurate temperature prediction

3.3. Evaluation metrics for hybrid modeling in manufacturing

Evaluation metrics are used to qualify the efficiency and prediction accuracy of the statistical or machine learning model. This part provides a summary of commonly used performance metrics that are appropriate for hybrid models in the manufacturing domain. The summary provided here will enhance the general comprehension of metrics for hybrid modeling and make it easier to choose them for use in various industrial applications. This section will cover the existing evaluation metrics for these classification and regression tasks individually. We will also provide examples of using the mentioned metrics in our review.

3.3.1. Metrics for classification problems

Based on our review, the classification problem in the manufacturing domain includes chatter detection Yang and Rai (2019); Rahimi et al. (2021), bearing fault detection Tian et al. (2015); Sun et al. (2017), turbine blades crack detection Pillai et al. (2016). Model classification performance can be assessed using various evaluation criteria, including accuracy, precision, recall, and F1 score.

Accuracy is the most commonly used metric for classification problems, quantifying the ratio of correctly predicted samples to the total number of prediction samples. The performance of hybrid models has been assessed in various manufacturing applications using accuracy measures Yumei et al. (2017); Sadoughi and Chao (2019); Sun et al. (2017). However, accuracy is most effective when the dataset has an equal distribution among classes. In the manufacturing process, collected datasets typically suffer from imbalanced class issues. Specifically, fault data in manufacturing industries is often overwhelmed by instances of faultless data Fathy et al. (2020). To address these accuracy shortcomings and enhance assessment robustness in the presence of an imbalanced dataset, precision and recall metrics are proposed. Precision calculates the percentage of true positive samples within all predicted positive samples, while recall measures the machine learning model’s ability to correctly identify true positives within all actual positive samples. These two metrics are better suited to handle imbalanced datasets and offer a more objective description of hybrid model classification performance compared to current state-of-the-art methods. In a study by Yazdi et al. (2020), these metrics were used to confirm the superiority of the hybrid model over conventional VGG and SVM models for additive manufacturing process monitoring tasks. However, evaluating a model’s performance can be challenging when precision is high and recall is low, or vice versa. The F1 score addresses this by measuring both precision and recall simultaneously, providing a more comprehensive assessment of the predictive hybrid model’s performance Yazdi et al. (2020).

3.3.2. Metrics for regression problems

The regression modeling establishes a mapping between input variables and continuous hybrid model output variables where both input and output variables can be multivariate. Regression problems encompass a range of applications, including molten steel temperature prediction Ahmad et al. (2014); Okura et al. (2013), additive manufacturing melt pool size prediction Olleak and Zhimin (2020); Mondal et al. (2020), remaining useful life estimation Aivaliotis et al. (2021); H. Sun et al. (2018), and more.

Table 11 provides a detailed summary of evaluation metrics for regression problems. Among these metrics, mean squared error (MSE) and mean absolute error (MAE) are the most commonly used to assess hybrid model performance. MSE represents the mean of the sum of squares of the differences between actual and predicted values and is utilized in studies such as Z. Liu et al. (2019), Reinhart et al. (2017), and Ambrogio et al. (2008).On the other hand, MAE quantifies the absolute error between predictions and ground truth values, as demonstrated by Cho et al. (1997), Swischuk et al. (2019), and Haghighi and Lin (2020). In addition to these widely used metrics, there are unique metrics tailored to specific problems. In Hong et al. (2019), a new metric called ‘score’ is defined to assess remaining useful life prediction results for gas turbine engines, which may be more tolerant and less critical than RUL prediction for turbofan engines.

Table 11. Regression evaluation metrics for hybrid models.

Name	Definition	Application	Reference
Mean squared error	$\frac{{\sum }_{\mathit{i}=1}^n{({\hat{y}}_{\mathit{i}}-y_i)}^2}{\mathit{n}}$	bearing fatigue prognosis	Yucesan and Viana (2020)
Mean absolute error	$\frac{{\sum }_{\mathit{i}=1}^n|{\hat{y}}_{\mathit{i}}-y_i|}{\mathit{n}}$	molten steel temperature control	Ahmad et al. (2014)
Average relative error magnitude	$\frac{1}{n}(\frac{{\sum }_{\mathit{i}=1}^n|{\hat{y}}_{\mathit{i}}-y_i|}{y_i})$	melt-pool width prediction	Moges et al. (2021)
Root mean square error	$\sqrt{\frac{{\sum }_{\mathit{i}=1}^n|{\hat{y}}_{\mathit{i}}^2-y_i^2|}{\mathit{n}}}$	melt-pool volumes and pre-deposition temperatures prediction	Ren et al. (2019)
$R^2$correlation coefficient	$\frac{\mathit{SSR}}{\mathit{SST}}$ or$1-\frac{\mathit{SSE}}{\mathit{SST}}$
		molten steel temperature	Okura et al. (2013)
Normalized RMSE	$\frac{\mathit{RMSE}}{{\mathit{y}}_{\mathit{max}}-y_{\mathit{min}}}$	prediction	Hu et al. (2019)
Maximum relative error	$\underset{1\le \mathit{i}\le \mathit{n}}{max}(\frac{|{\hat{y}}_{\mathit{i}}-y_i|}{y_i})\times 100\mathrm{\%}$
Proportional error	$\frac{{\sum }_{\mathit{i}=1}^n{({\hat{y}}_{\mathit{i}}-y_i)}^2}{{\sum }_{\mathit{i}=1}^n{(y_i)}^2}$
Maximum error	$\mathit{max}|{\hat{y}}_{\mathit{i}}-y_i|$	comprehensive coke
		ratio prediction	Hu et al. (2019)
Mean absolute percentage error	$\frac{{\sum }_{\mathit{i}=1}^n|\frac{|y_i-{\hat{y}}_{\mathit{i}}|}{\mathit{y}}|}{\mathit{n}}$	detect flaws in AM parts	Gaikwad et al. (2020)
Realistic prediction ratio	$\frac{{\sum }_{\mathit{i}=1}^n[{\hat{y}}_{\mathit{i}}\ge {\bar{y}}_{\mathit{i}}]}{\mathit{n}}\times 100\mathrm{\%}$	product completion time prediction	Huang et al. (2020)
U-pooling(area metric)	${\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}|F_{y_i}(\mathit{y})-F_{{\hat{y}}_{\mathit{i}}}(\mathit{y})|\mathit{dy}$	melt-pool size prediction	Olleak and Zhimin (2020)
Normalized MAE	$\frac{\mathit{MAE}}{{\mathit{y}}_{\mathit{max}}-y_{\mathit{min}}}$	tool wear prediction	Hanachi et al. (2019)
Relative success ratio	$\frac{\mathit{NS}}{\mathit{NS}+\mathit{NF}}$	nugget diameter prediction	Sim et al. (2018)
Akaike information criteria	$2\mathit{k}-2\mathit{ln}(\mathit{L}(\mathit{x},\hat{\mu }))$		Nannapaneni and Mahadevan (2016)
Bayes information criteria	−2*lnL+kln(n)	injection molding energy consumption prediction	Nannapaneni and Mahadevan (2016)
Score	$\mathit{S}={\sum }_{\mathit{i}=1}^n{s}_k$	engine remaining useful
		life prediction	Hong et al. (2019)

4. Challenges and future research directions

Despite recent advancements in the field of hybrid modeling techniques within the manufacturing industry, as presented in this paper, several challenges and knowledge gaps persist. Further research is necessary to address these current issues and to expand the use and functionality of hybrid models. In this section, we will discuss potential future research directions related to hybrid modeling in manufacturing and the manufacturing industry.

4.1. Small datasets and data limitations

Sufficient and accurate (high-quality) data are required to characterize, develop, validate, and assess hybrid models in the manufacturing domain. Unfortunately, the complexity of manufacturing data collection and processing often limits access to sufficiently large and high-quality datasets. Setting up the required connectivity and pre-processing is an expensive and time-consuming process. The raw manufacturing data may come from different distributed sources in various formats, such as RGB and depth images from the camera Yazdi et al. (2020); Wang et al. (2018), acoustic signals from microphones Yang and Rai (2019); Rahimi et al. (2021), and numerical values from vibration sensors Sadoughi and Chao (2019). These raw datasets generally have variable sampling rates, necessitating additional processing with image/signal preprocessing and data fusion methods before being fed into the hybrid model.

Furthermore, the data labeling work for most manufacturing problems typically requires domain knowledge. For example, detecting minor flaws or bearing issues always necessitates the involvement of skilled personnel or experts. Otherwise, the training of the hybrid model may be complicated by inconsistent and erroneous labels. The amount of qualifying production data for hybrid model training is constrained by the aforementioned problems. However, most machine learning models, especially deep learning models, are data-hungry algorithms, as they perform better with a large amount of data. Deep learning models trained with small datasets need better generalization. Developing data-efficient and robust hybrid models with fewer labels is imperative, not limited to labeled data constraints, and can expand their applications.

Semi-supervised learning, already utilized in semiconductor production Kang et al. (2016) and laser powder-bed fusion fault detection Okaro et al. (2019), is one option that requires fewer labeled data and a vast amount of unlabeled data. Unlabeled data, containing only input variables, can be collected through various sensors with less effort than labeled data. The reduction in the demand for labeled data can significantly enhance its feasibility in the manufacturing sector. However, semi-supervised learning-based hybrid models have yet to be deployed in the industrial sector and should be a focus of future development.

Moreover, knowledge sharing accumulates and maintains knowledge learned from previous tasks and seamlessly applies it to learning new tasks and solving new problems with minimal data and effort Adadi (2021). Transfer Learning is a representative knowledge-sharing technique, that has been applied to machine fault diagnostics Wang and Gao (2020) and the tool’s remaining useful life prediction Sun et al. (2018). Merging physics models with transfer learning architectures provides a promising solution to the data limitation and warrants more research in the area.

Apart from hybrid models, data augmentation techniques can also create additional data by modifying the original data without changing its distribution Perez and Wang (2017). In the manufacturing domain, data augmentation methods include variational autoencoders Yun et al. (2020), Gaussian Noise Li et al. (2020), and synthetic data augmentation through generative adversarial networks Jain et al. (2020).While good prediction results can be obtained with each augmentation technique independently, a combined augmentation technique can be a future research direction, expected to leverage the advantages of different approaches and further enhance the hybrid model’s accuracy and efficiency.

4.2. Imbalanced data in manufacturing applications

Imbalanced data is a term used to describe the uneven representation of different classes in a classification task. In manufacturing, imbalanced data is more likely to occur in defect/fault detection situations Saqlain et al. (2020); Lee et al. (2020). Compared with the standard sample, the defects sample generally occupies a small portion of the overall dataset, which can lead to biased learning and misclassification issues. This issue is challenging to overcome, given the manufacturing objective of producing high-quality parts and continually improving processes. For example, in the study by Saqlain et al. (2020); Lee et al. (2020) the eight wafer faults account for only 14.8% of all labeled datasets. This data imbalance and the resulting misclassifications may lead to issues from the analytic point.

While we cannot change the nature of the data due to the manufacturing objective, we can address the problem in other ways. Three strategies show promise for addressing the issue of data imbalance in the manufacturing sector: data-driven methods, algorithmic-based strategies, and hybrid approaches Fathy et al. (2020). Since the selection of a solution depends heavily on the dataset’s characteristics, there is no one-size-fits-all solution and no clear guidance for solution selection. Developing well-defined guidance is critical to improving the performance of hybrid models when dealing with imbalanced data.

Furthermore, most solutions for imbalanced data issues have traditionally focused on image data, such as image rotation, flipping, and noise addition. Research addressing imbalanced data issues in non-image manufacturing data is limited and should be expanded upon Johnson and Khoshgoftaar (2019).

4.3. Benchmark problems for hybrid modeling in manufacturing

Benchmark problems enable the performance evaluation of different machine learning algorithms and serve as a common platform for addressing challenging domain problems, which have been shared by many research groups over an extended time frame Dueben et al. (2022). In the field of computer vision, benchmark problems are well-established and include datasets like COCO Lin et al. (2014), CIFAR-10 Krizhevsky and Hinton (2009), and KITTI Geiger et al. (2012). However, the manufacturing domain has a limited number of benchmark problems Farahani, McCormick, Gianinny, et al. (2023). The Case Western Reserve University (CWRU) bearing dataset Smith and Randall (2015), has been the most extensively utilized benchmark problem in the diagnostics category for bearing defect identification. Consequently, most diagnostics applications in the selected articles primarily focus on defect detection. In contrast, there are relatively few articles that address chatter and crack detection due to a lack of suitable benchmark problems. Some publicly available manufacturing datasets from NASA and others that can be useful in this regard were introduced in Farahani, McCormick, Harik, et al. (2023).

Manufacturing encompasses various categories beyond diagnostics, including manufacturing process control and human-machine interaction, among others. Ideally, each type of application should have a distinct benchmark dataset to stimulate competitive progress. In general, the growth of state-of-the-art techniques is directly associated with the complexity of the benchmark problems Rai and Sahu (2020). Future work could involve formulating benchmark problems for hybrid models across various manufacturing processes, significantly accelerating research in hybrid models, and expanding their practical applications.

4.4. Hybrid model selection

Proper model selection is a crucial precondition for the successful implementation of a model. The manufacturing industry has achieved remarkable success with data-driven models, particularly deep learning, owing to the intuitive model selection guidelines that are already in place. For instance, CNN has been widely used for analyzing visual images Imoto et al. (2018), the LSTM model has demonstrated its ability to learn from sequential data Lindemann et al. (2020), and Autoencoder architecture is capable of feature extraction without label information J. Wang et al. (2018).

In manufacturing, hybrid models can be classified into three main categories based on their interaction methods: combination Tian et al. (2015), constraint Kapusuzoglu and Mahadevan (2020), and embedding approaches Slatteke and Astrom (2005). However, the criteria for selecting relatively novel hybrid models are less standardized compared to predominantly data-driven approaches. This selection process is primarily influenced by the domain expert’s opinion at the current stage. For example, the combination approach is commonly used in most bearing defect detection problems Sun et al. (2017); Sadoughi and Chao (2019); Tian et al. (2015); Junbo et al. (2015); Chen et al. (2017). However, none of these papers address the rationale for selecting the combination technique or demonstrate its superiority over the other two approaches. Therefore, it is essential to determine how to choose the appropriate components from both physics and data-driven methods, as well as the interface style, based on the various challenges at hand. Ineffective implementation, resulting from a lack of clear direction, negatively impacts the model’s performance. It is imperative to develop a universal hybrid model selection approach to address manufacturing problems, as this can significantly enhance performance and broaden the model’s applicability.

5. Conclusion and Limiations

This paper provides an overview of hybrid modeling in manufacturing. Our literature analysis focuses on the clear combination of physics-based and data-driven modeling in a manufacturing context. We have observed that the reviewed literature covers only 10 use cases related to traditional manufacturing processes, such as machining, milling, rolling, smelting, drilling, forming, and sintering.

In this review, we have gathered and presented information on how new hybrid models can be developed and designed. We also offer an overview of alternative architectures for hybrid models and have collected common testing and evaluation metrics for such models. Furthermore, we discuss key challenges associated with hybrid models in a manufacturing context, which are crucial for those interested in developing and implementing hybrid models on the manufacturing shop floor and beyond. Additive manufacturing, machine monitoring, tool wear, and welding are the predominant areas where hybrid models have been implemented in the reviewed literature to date. When reflecting on these dominant areas, we notice that readily available access to data, compared to other areas, exists for both additive manufacturing and monitoring. Additionally, the availability of sophisticated physics models, either as simulations or through experimentation, stands out. Thus, for future work, it is essential to investigate how to make data and physics models for traditional manufacturing processes more readily available, enabling the implementation of hybrid models and the ensuing positive results.

It is worth noting that manufacturing challenges go beyond optimization problems. This article explores the various objectives that can be pursued in this context, including modeling, data generation, and uncertainty quantification. Manufacturing researchers and practitioners must consider whether new projects can be initiated to create innovative models that enhance our understanding of processes and interactions in unprecedented ways.

The limitations of this work should be acknowledged. The literature review, while following a strict methodology, may still exhibit certain biases. For example, the term ‘production’ is often used internationally to refer to manufacturing activities, and this usage may have been missed by our defined search terms. Similarly, the variant spelling of ‘optimization’ was not considered, potentially affecting the results and leading to more use cases that could have been presented. Thermal sciences are well-documented in physics and are relatively common among hybrid modeling use cases, whether in additive manufacturing or welding. Moreover, a substantial portion of the data and analyses presented in this paper are inherently subjective and influenced by our perceptions. Recognizing that the complete elimination of subjectivity is impossible and acknowledging that such an endeavor may not be entirely desirable, our commitment is to be transparent about the process and methodology employed. We aim to ensure that our audience is well-informed about our biases, intent, understanding, and the consequential impact on the content of this paper. Finally, some additional papers may get indexed after our initial database search so they may not be considered here, and the result may differ under different assumptions.

Acknowledgments

The authors thank the anonymous reviewers for their constructive comments that improved the paper substantially.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This material is based upon the study supported by (i) the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy (EERE) under the Advanced Manufacturing Office Award Number DE-EE0007613 (Disclaimer: This report was prepared as an account of the work sponsored by an agency of the United States Government. Neither the United States government nor any agency thereof, nor any of its employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness.); and (ii) the National Science Foundation under Grant No. 2119654. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.