Physics-Informed neural network solver for numerical analysis in geoengineering

ABSTRACT

Engineering-scale problems generally can be described by partial differential equations (PDEs) or ordinary differential equations (ODEs). Analytical, semi-analytical and numerical analysis are commonly used for deriving the solutions of such PDEs/ODEs. Recently, a novel physics-informed neural network (PINN) solver has emerged as a promising alternative to solve PDEs/ODEs. PINN resembles a mesh-free method which leverages the strong non-linear ability of the deep learning algorithms (e.g. neural networks) to automatically search for the correct spatial-temporal responses constrained by embedded PDEs/ODEs. This study comprehensively reviews the current state of PINN including its principles for the forward and inverse problems, baseline algorithms for PINN, enhanced PINN variants combined with special sampling strategies and loss functions. PINN shows an easier modelling process and superior feasibility for inverse problems compared to conventional numerical methods. Meanwhile, the limitations and challenges of applications of current PINN solvers to constitutive modelling and multi-scale/phase problems are also discussed in terms of convergence ability and computational costs. PINN has exhibited its huge potential in geoengineering and brings a revolutionary way for numerous domain problems.

KEYWORDS:

• Neural networks
• physics-informed
• machine learning
• soils
• constitutive modelling
• partial differential equations

1. Introduction

In engineering practice, the spatial-temporal responses of systems are often characterized by mathematical formulations such as partial differential equations (PDEs) or ordinary differential equations (ODEs). In the geotechnical domain, PDEs have also been extensively used to model a series of canonical problems such as consolidation (Terzaghi 1925), cavity expansion (Yu 2000) and landslides (Vardoulakis 2002).

To solve PDEs/ODEs, the use of analytical and semi-analytical methods has been fully investigated by many researchers (Zhou et al. 2018; Feng, Ni, and Mei 2019; Ho and Fatahi 2016). Although analytical and semi-analytical solutions are accurate and convenient, they are limited to simple cases such as low-dimensional problems and simple constitutive relations (Zhang, Yin, and Sheil 2023b; Asaoka 1978). Therefore, much attention has been given to numerical techniques to solve PDEs/ODEs, including finite element methods (FEM) (Feyel 2003; Zhu, Yin, and Graham 2001; Earij et al. 2017; Cividini and Gioda 2004), finite difference methods (Zhou, Deng, and Wang 2013; Ma et al. 2020) and meshless numerical methods such as the smoothed particle hydrodynamics (Monaghan 1992) and the material point method (MPM) (Yerro, Rohe, and Soga 2017; Sulsky, Chen, and Schreyer 1994). However, mesh-based numerical methods still dominate the numerical simulation for engineering practice considering the limitations of mesh-free methods. For instance, the stability of MPM is associated with the time integration procedure of dynamic equations, thus the time increment should be significantly small compared with the FEM (Więckowski 2004).

Despite the great success that numerical methods have achieved, their further progress has been restricted by many challenges and limitations, which can be generally summarised in two aspects: (1) the computational costs of numerical methods are high, especially in multiscale and high-dimensional engineering problems, which hinder high-resolution simulations in engineering practice (Geers et al. 2017; Liu et al. 2021); (2) while inverse problems are quite common in the engineering domain, solving inverse problems is prohibitively expensive where abundant computational trials are required to be carefully implemented.

Recently, due to the fast progress of data science and machine learning (ML), physics-informed neural networks (PINNs) have gained a new round of attention and their history can be dated back to the end of the last century (Lee and Kang 1990). Psichogios and Ungar (1992) proposed a hybrid learning method that incorporated physics laws, e.g. mass balance or energy balance, into the network structure, and the results demonstrated that this hybrid physics-informed method can achieve better generalization and extrapolation ability compared to standard “black-box” ML. Lagaris, Likas, and Fotiadis (1998) successfully solved different PDEs/ODEs considering initial/boundary conditions. In this work, the structure of the trial solution was designed to satisfy hard initial/boundary conditions and the other half of the trial solution was obtained from the output of neural networks to satisfy PDEs/ODEs. More recently, Raissi, Perdikaris, and Karniadakis (2019) applied PINNs to directly solve PDEs, in which the loss values on the initial/boundary conditions and the residuals on the PDEs are added to the loss function of neural works. The core of PINNs is incorporating prior information such as known physics laws and PDEs/ODEs into the neural networks. In this way, the outputs generated by the network can be constrained by physics laws. Compared with traditional data-driven approaches, PINNs can successfully alleviate overfitting issues and can still maintain robustness with a small amount of data or even with imperfect data (Zhang, Liu, and Sun 2020). Its potential on both forward and inverse problems has been successfully proved, while PINNs have achieved competitive performance compared with exact values. To date, this novel approach has been extensively used in numerous engineering and scientific problems, such as fluid mechanics (Cai et al. 2022; Sun et al. 2020; Mao, Jagtap, and Karniadakis 2020; Jin et al. 2021), heat transfer (Cai et al. 2021; Cai et al. 2020), solid mechanics (Abueidda et al. 2022; Diao et al. 2023) and material science (Liu and Wang 2019).

The objective of this review is to provide a comprehensive analysis and discussion of the characterizations of current PINNs. First, the basic modelling framework of a PINN-based PDE solver is introduced. Then, different baseline neural networks, variants, sampling methods and loss functions used in PINNs are introduced. Finally, we demonstrate the limitations and challenges of applying PINNs for engineering problems.

2. Framework of PINNs

A general PDE subjected to initial/boundary conditions in engineering problems can be described by (Rao, Sun, and Liu 2021): (1) $\begin{array}{rl}& u_t(t,\mathit{x})+\mathcal{D}[u(t,\mathit{x})]=0,\mathit{x}\in \mathrm{\Omega },t\in [0,T]\\ & u(t,\mathit{x})={\mathcal{B}}_D(t),x\in \mathrm{\partial }{\mathrm{\Omega }}_D,t\in [0,T]\end{array}$(1) (2) $\mathbf{n}\cdot \mathrm{\nabla }u(t,\mathit{x})={\mathcal{B}}_N(t),\mathit{x}\in \mathrm{\partial }{\mathrm{\Omega }}_N,t\in [0,T]$(2) (3) $\begin{array}{rl}& u(0,\mathit{x})={\mathcal{I}}_0(\mathit{x}),\mathit{x}\in \mathrm{\Omega }\\ & u_t(0,\mathit{x})={\mathcal{I}}_1(\mathit{x}),\mathit{x}\in \mathrm{\Omega }\end{array}$(3) where t and x are the temporal and spatial coordinates, u_t(t, x) is the differential of u(t, x) to t, u(t, x) is the hidden solution, $\mathcal{D}$[·] is a nonlinear differentiation operator, $\mathrm{\Omega }$ is the space of input variables, $\mathrm{\partial }{\mathrm{\Omega }}_D$ and $\mathrm{\partial }{\mathrm{\Omega }}_N$ are the Dirichlet and Neumann boundaries, respectively, ${\mathcal{B}}_D(t)$ and ${\mathcal{B}}_N(t)$ are the prescribed functions in Dirichlet and Neumann boundaries, respectively, T is the maximum computation time, n denotes the unit outer normal vector of boundaries, $\mathrm{\nabla }$ is the gradient operator, ${\mathcal{I}}_0(x)$ and ${\mathcal{I}}_1(x)$ are the initial states of the system. A detailed illustration of training PINNs to solve Equation (1) is shown in Figure 1, where x and t are the inputs of the network and u is the output. During the training process, the PINN iteratively adjusts the weights W and biases b of the network, until the loss value is smaller than the tolerance value $\epsilon$. This process is achieved by solving the following optimization problem: (4) ${\theta }^{\ast }=\arg \underset{\theta }{min}\mathcal{L}(\theta t,\mathit{x})$(4) where $\theta$ = [W, b], which are the hyperparameters of the network, $\mathcal{L}$ is the loss function that needs to be minimized.

Figure 1. Framework of a generic PINN (Karniadakis et al. 2021).

In the majority of existing research on PINNs, the weights and biases are fine-tuned predominantly via gradient-based optimization methods. These methods minimize the discrepancy between the predicted and expected outputs by iteratively adjusting the parameters in the direction of the steepest descent of the gradient, which can be generally described by using the following equations (Ruder 2016): (5) ${\theta }^{(n+1)}={\theta }^{(n)}\text{–}\eta \cdot {\mathrm{\nabla }}_{\theta }\mathcal{L}({\theta }^{(n)}t,\mathit{x})$(5) where ${\theta }^{(n)}$ denotes the hyperparameters in the n^th iteration step, $\eta$ is the learning rate. However, a growing body of evidence has demonstrated that gradient descent exhibits pathological behaviour when training PINNs to solve complex PDEs (Wang, Yu, and Perdikaris 2022). Limited research in the field of PINNs has explored the feasibility of employing non-gradient-based optimization methods, such as using a particle swarm optimization approach, rather than gradient descent, to train PINNs (Davi and Braga-Neto 2022). Once the model is well trained, the solution of the embedded PDEs can be solved.

In most literature, the loss function is defined as the mean square error (MSE) of the predictions for the sake of its differentiability (Daolun et al. 2021; Shen et al. 2022; Lu and Mei 2022; Bishop and Nasrabadi 2006), which allows to efficiently implement error backpropagation and gradient descent (Goodfellow, Bengio, and Courville 2016).

The loss function of PINNs generally includes the residual value of governing PDEs/ODEs, initial/boundary conditions and measured data (if available), which has been shown using the following equations: (6) $\mathcal{L}=w_1{\mathcal{L}}_{\mathrm{PDE}}+w_2{\mathcal{L}}_{\mathrm{BC}}+w_3{\mathcal{L}}_{\mathrm{IC}}+w_4{\mathcal{L}}_{\mathrm{data}}$(6) (7) ${\mathcal{L}}_{\mathrm{PDE}}=\Vert u_t(t,\mathit{x})+\mathcal{D}[u(t,\mathit{x})]{\Vert }_{\mathrm{\Omega }\times [0,T]}^2$(7) (8) $\begin{array}{rl}{\mathcal{L}}_{\mathrm{BC}}=& \Vert {u(t,\mathit{x})\text{–}{\mathcal{B}}_D(t)\Vert }_{\mathrm{\partial }{\mathrm{\Omega }}_D\times [0,T]}^2\\ & +\Vert \mathbf{n}\cdot \mathrm{\nabla }u(t,\mathit{x})\text{–}{\mathcal{B}}_N(t){\Vert }_{\mathrm{\partial }{\mathrm{\Omega }}_N\times [0,T]}^2\end{array}$(8) (9) ${\mathcal{L}}_{\mathrm{IC}}=\Vert u(0,\mathit{x})\text{–}{\mathcal{I}}_0(\mathit{x}){\Vert }_{\mathrm{\Omega }\times [t=0]}^2+\Vert u_t(0,\mathit{x})\text{–}{\mathcal{I}}_1(\mathit{x}){\Vert }_{\mathrm{\Omega }\times [t=0]}^2$(9) (10) ${\mathcal{L}}_{\mathrm{data}}=\Vert u(t,\mathit{x})\text{–}{u}^{\ast }(t,\mathit{x}){\Vert }_{\mathrm{\Omega }\times [0,T]}^2$(10) where $\mathcal{L}$_PDE, $\mathcal{L}$_BC and $\mathcal{L}$_IC are the residual values of governing PDEs, boundary and initial conditions, respectively, which are regarded as the incorporated physics constraints in the networks, $\mathcal{L}$_data is the residual value from measured data, which can be regarded as the pure data-driven part in the networks, u is the prediction from the PINN and $u^{\ast }$ is the corresponding exact value, $\Vert \cdot {\Vert }^2$ is the mean square error (MSE) operator, w₁, w₂, w₃, w₄ are weights of each loss term which can be used to eliminate the imbalance between different loss terms. As the imbalance among multiple loss terms may lead to gradient pathologies and even the failure of training PINNs, these weights of different loss terms should be finely defined by the user or tuned automatically, and they are crucial for the performance improvement of PINNs (Karniadakis et al. 2021; Wang, Teng, and Perdikaris 2021). While empirically tunning these parameters in a trial-and-error fashion can be tedious and time-consuming, many researchers have proposed approaches to tune the weights of different loss terms. These approaches can be generally categorized into two types: (1) fixed weights for each loss term (Wight and Zhao 2020; Elhamod et al. 2022; van der Meer, Oosterlee, and Borovykh 2022), and (2) adaptive weight balancing approaches that continuously adjust the weights of each loss term during the training process of PINNs (Kim et al. 2021; Wang, Teng, and Perdikaris 2021; Wang, Yu, and Perdikaris 2022; Shin, Darbon, and Karniadakis 2020; Liu and Wang 2021; McClenny and Braga-Neto 2020).

By adding the residual of PDEs/ODEs to the loss function, PDEs/ODEs can be cast into the loss function of PINNs and the final output can be constrained by the prior information. To calculate the residual of PDEs/ODEs in PINNs, it is necessary to compute the derivatives of network outputs with respect to the network inputs. This process can be achieved by using automatic differentiation (AD), which facilitates the development of PINN (Baydin, Pearlmutter, and Radul 2018). Unlike traditional numerical methods, AD can handle differential operators in a meshless way and does not differentiate data (Lu et al. 2021). To be specific, in the implementation of PINNs, AD can automatically calculate the derivatives of the neural network outputs with respect to the input variables, and this procedure is theoretically based on the chain rule. A simple feedforward neural network has been illustrated in Figure 2 to show how AD and the chain rule work. As shown in Figure 2, the input variables of the network are x₁ and x₂, and the output variable is y; v₁, v₂, v₃ and v₄ are the outputs of each neuron for each layer. The derivatives of y with respect to x₁ and x₂ can be calculated through the chain rule, as depicted in Figure 2. Some popular deep learning frameworks such as Tensorflow (Abadi et al. 2016) and Pytorch (Paszke et al. 2019) have effectively incorporated AD.

Figure 2. Illustration of implementing AD based on the chain rule (Rao, Sun, and Liu 2021).

3. Methodology and applications

3.1. Principles for solving forward and inverse problems

For PDEs with deterministic parameters, the aim of training a PINN is to obtain an accurate solution, which is regarded as the forward problem. Many research works have demonstrated the feasibility of using PINNs to obtain the solution of PDEs in forward problems (Raissi, Perdikaris, and Karniadakis 2017; Haghighat, Amini, and Juanes 2022; Niu et al. 2023). Abueidda, Lu, and Koric (2021) used PINNs to solve various boundary value problems (BVPs) containing different constitutive laws of solid materials. In this research, PINN was implemented as a PDE solver in a fairly straightforward manner. Different from conventional ML-based methods that rely on a large amount of data, this novel PINN-based paradigm can extract solutions from fully-understand physical laws with given initial and boundary conditions. Thus, when using PINNs to solve forward BVPs, accurate solutions can be obtained without resorting to data generation through numerical modelling software or laboratory tests (Rao, Sun, and Liu 2021; Abueidda, Lu, and Koric 2021; Sun et al. 2020; Zhu et al. 2019). The framework of using PINNs as a forward PDE solver is depicted in Figure 3(a). In this framework, there is no extra labelled data can be obtained except for initial and boundary conditions, and therefore the loss function only involves the loss value from initial conditions, boundary conditions and residuals of PDEs. If more data can be obtained when solving forward PDEs, the residual on data can also be incorporated into the loss function.

For most complex physics systems, the physics laws are partially known (Tartakovsky et al. 2018), and therefore the unknown parameters in equations should be identified first according to the observed data. PINN characterizes this kind of inverse problem (Cuomo et al. 2022; Lu et al. 2021; Raissi, Perdikaris, and Karniadakis 2019). One of the most outstanding advantages of PINNs is that they can solve inverse problems as easily as solving forward problems (Karniadakis et al. 2021; Lu et al. 2021; He et al. 2020; Raissi, Perdikaris, and Karniadakis 2019; Tartakovsky et al. 2020). Given the distinguished ability of PINNs to recover PDEs and solve PDEs, scholars have applied PINNs to various inverse problems in different domains, such as solid materials (Haghighat et al. 2021), supersonic flows (Jagtap et al. 2022), unknown diffusion coefficient estimation (Tartakovsky et al. 2018).

The framework for using PINNs to solve inverse problems is shown in Figure 3(b). Similarly, a general form of a PDE with unknown parameters can be described using the following equation (Raissi, Perdikaris, and Karniadakis 2019): (11) $u_t(t,\mathit{x})+\mathcal{D}[u(t,\mathit{x})\lambda ]=0,\mathit{x}\in \mathrm{\Omega },t\in [0,T]$(11) where $\lambda$ is the model parameter that needs to be identified. In this context, the unknown parameter is seen as a variable that would be updated during the training process, as shown in Figure 3(b). Similarly, the unknown parameter $\lambda$ and the variable u can be approximated by minimizing the MSE loss, which also contains the residual of data and the residual of PDEs: (12) $\begin{array}{rl}\mathcal{L}=& w_1\cdot \Vert u_t+\mathcal{D}[u\lambda ]{\Vert }_{\mathrm{\Omega }\times [0,T]}^2+w_2\cdot \\ & (\Vert u\text{–}{\mathcal{B}}_D(t\lambda ){\Vert }_{\mathrm{\partial }{\mathrm{\Omega }}_D\times [0,T]}^2+\Vert \mathbf{n}\cdot \mathrm{\nabla u}\text{–}{\mathcal{B}}_N(t\lambda ){\Vert }_{\mathrm{\partial }{\mathrm{\Omega }}_N\times [0,T]}^2)\\ & +w_3\cdot (\Vert u\text{–}{\mathcal{I}}_0(\mathit{x}\lambda ){\Vert }_{\mathrm{\Omega }\times [t=0]}^2+\Vert u_t\text{–}{\mathcal{I}}_1(\mathit{x}\lambda ){\Vert }_{\mathrm{\Omega }\times [t=0]}^2)\\ & +w_4\cdot \Vert u\text{–}{u}^{\ast }{\Vert }_{\mathrm{\Omega }\times [0,T]}^2\end{array}$(12) It should be noted that labelled data is indispensable for addressing inverse problems since there are unknown parameters that need to be identified.

3.2. Baseline neural networks

Before training, the underlying algorithm of PINNs is required to be determined based on the essence of the studied problem. Herein, three commonly used algorithms, fully-connected neural networks (FCNN), long short-term memory (LSTM) neural networks and convolutional neural networks (CNN) for building PINNs are introduced.

3.2.1. Fully-connected neural networks

The FCNN-based PINN models have prevailed over all PINN implementations (Cai, Wang, Wang, and Perdikaris 2021; Kharazmi, Zhang, and Karniadakis 2021; Jagtap et al. 2022). This FCNN-based PINN has been employed in various engineering domains, such as the multi-phase flow in porous media (Haghighat, Amini, and Juanes 2022), solid mechanics (Haghighat et al. 2021), surrogate modelling for fluid flows (Sun et al. 2020), geotechnical constitutive modelling (Zhang, Yin, and Sheil 2023a), and geotechnical hydromechanical modelling (Zhang et al. 2022). Generally, the prior information is embedded in the loss function of FCNN in the form of PDE/ODE residuals. The FCNN consists of the input, hidden and output layers. The hidden layers receive information from the input layer and pass the transformed information to the output layer. Supposing that the total number of hidden layers is L, this process can be described by using the following equations (Zhang, Yin, and Jin 2021): (13) $y_l={\sigma }_l({\mathbf{W}}_l{\mathit{y}}_{l-1}+{\mathit{b}}_l),l\in [1,L]$(13) (14) ${\mathit{y}}_{L+1}={\mathbf{W}}_{L+1}{\mathit{y}}_L+{\mathit{b}}_{L+1}$(14) where y_l is the output of the l^th layer and the input of the (l + 1)^th layer, W_l and b_l are the weight matrix and bias vector of the l^th layer, respectively, y₀ (l = 1) and y_L + 1 is the input and output of the FCNN, respectively, ${\sigma }_l$ is the activation function of each hidden layer.

During the training procedure, the parameters of the FCNN, W and b, are optimized gradually to minimize the discrepancy between the predictions and measurements. Finally, the output value y_L + 1 is generated as a prediction of the real value. The hyperparameters of the FCNN, such as the number of hidden layers and the number of neurons in each layer, should be tuned finely, which may have a significant impact on the convergence process and prediction accuracy (Cuomo et al. 2022). Especially when the data is sparse, which is a common case in PINN models, the network size affects the accuracy of the predictions significantly (He et al. 2020). To date, PINNs with FCNN architecture commonly determine the hyperparameters in a trial-and-error way (Raissi, Perdikaris, and Karniadakis 2019; Tartakovsky et al. 2020; He et al. 2020).

3.2.2. Long short-term memory neural network

LSTM neural networks (Hochreiter and Schmidhuber 1997) are also used to develop PINN-based models. LSTM is characterised by handling sequential data and dealing with long-time-lag tasks (Yang et al. 2019; Ma et al. 2022; Chen et al. 2023). As sequence-dependent problems are typical in the engineering domain, some attempts have been made to integrate physics into LSTM. For instance, Zhang, Liu, and Sun (2020) proposed a physics-informed multi-LSTM network. In this hybrid framework, two physics-informed multi-LSTM networks, e.g. phyLSTM² and phyLSTM³, are developed for modelling state space variables, restoring force, and hysteretic parameters. The physical laws are not only encoded in the network architecture but also embedded in the loss function. The proposed physics-informed models outperform purely data-driven models in terms of robustness, interpretability and generalizability. Zhang et al. (2022) proposed a physics-constrained LSTM to capture soil behaviour, in which the PDE-based constitutive laws are incorporated into the loss function to constrain the solution. The results demonstrated that this physics-constrained LSTM neural network surpassed traditional LSTM neural networks. Lahariya et al. (2022) also proposed physics-informed LSTM networks (PhyLSTMs) to model the evaporate cooling system, in which the residual of system dynamics is incorporated into the loss function. The authors also compare the performance of typical PINNs and PhyLSTMs, and the results demonstrated that PhyLSTMs can reach lower estimation errors and converge faster. To date, less research work focuses on the combination of PINNs with LSTM.

Overall, further research is expected to enhance the integration of conventional PINN frameworks with LSTM networks, with the aim of addressing a wider range of engineering problems with better performance and more flexibility.

The basic building block of the LSTM is the memory cell, which has been illustrated in Figure 4. The information can be saved or forgotten by the memory cell, which is conducted by the input gate, forget gate and output gate, as shown (Hochreiter and Schmidhuber 1997): (15) $i_t=\sigma ({\mathbf{W}}_i\ast [{\mathit{h}}_{t-1},{\mathit{x}}_t]+{\mathit{b}}_i)$(15) (16) $f_t=\sigma ({\mathbf{W}}_f\ast [{\mathit{h}}_{t-1},{\mathit{x}}_t]+{\mathit{b}}_f)$(16) (17) $o_t=\sigma ({\mathbf{W}}_o\ast [{\mathit{h}}_{t-1},{\mathit{x}}_t]+{\mathit{b}}_o)$(17) where i_t, f_t, and o_t are the values of the input gate, forget gate and the output gate at the timestep t, respectively, $\sigma$ is the sigmoid function, W_i, W_f and W_o are the weight matrices of the input gate, forget gate and the output gate, respectively, h_t−1 is the hidden state at the timestep $t-1$, x_t is the input at the timestep t, b_i, b_f and b_o are the bias vectors of the input gate, forget gate and the output gate, respectively. Thereafter, the current cell state and the hidden state can be updated by using (Hochreiter and Schmidhuber 1997): (18) ${\tilde{c}}_t=\tanh ({\mathbf{W}}_c[{\mathit{h}}_{t-1},{\mathit{x}}_t]+{\mathit{b}}_c)$(18) (19) $c_t=i_t\odot {\tilde{c}}_t+f_t\odot c_{t-1}$(19) (20) ${\mathit{h}}_t=o_t\odot \tanh (c_t)$(20) where ${\tilde{c}}_t$ is the temporary cell state, tanh is the activation function, W_c and b_c are the weight matrix and the bias vector of the memory cell, respectively, c_t and c_t−1 are the updated cell state and the previous cell state, respectively, h_t is the updated hidden state, $\odot$ denotes for the element-wise multiplication. $i_t\odot {\tilde{c}}_t$ represents the selected information in the current timestep while $f_t\odot c_{t-1}$ represents the forgotten information from the previous timestep.

Figure 3. Framework of PINNs for (a) forward problems; and (b) inverse problems.

3.2.3. Convolutional neural network

CNN recently has also been used to construct a PINN-based model. CNN characterizes solving problems with large-scale spatio-temporal domains. Recently, scientists have made some attempts to incorporate physical constraints into CNNs, either in hard or soft manners. To embed hard constraints into the networks, the physical constraints should be encoded into the network architecture and design. In this way, the satisfaction of prior knowledge can be completely guaranteed. For instance, Kim et al. (2019) proposed a physics-informed CNN to conduct fluid simulations with divergence-free conditions to be hard-coded into the network. Similarly, Mohan et al. (2020) simulated three-dimensional turbulence with a physics-informed CNN, where the conservation laws and boundary conditions are strictly enforced all the time. Until now, many existing research works focus on using physical constraints in a soft manner in CNNs, by treating the physical constraints as penalty terms that are incorporated into loss functions in the form of PDE/ODE residuals (Gao, Sun, and Wang 2021; Sharma et al. 2018; Subramaniam et al. 2020; Wu et al. 2020). A standard CNN is limited to tackling regular geometries and uniform grids, while most scientific and engineering problems tend to have complex domains. Gao, Sun, and Wang (2021) used an elliptic coordinate mapping technique to transform an irregular solution domain into a regular domain, by which PDEs with irregular domains can be solved by CNNs.

The framework of CNN is shown in Figure 5. Generally, CNN consists of several convolutional, pooling and fully-connected layers. The size of the input image is denoted by h^r (height) × w^r (width) × c^r (channel, 1 for greyscale image and 3 for colour image). The convolutional layer is made of a series of filters or kernels, each having the same size of f × f. These filters or kernels slide on the input image from left to right and from top to bottom with a fixed stride s to extract specific features. This process is achieved by performing the element-wise multiplication between the filter matrix and the corresponding region of the input image. This region, termed the receptive field, matches the size of the filter. After convolved by n filters, the size of the input image is rescaled to h^c × w^c × c^c × n^c, and feature maps are generated for each filter subsequently. The size of the feature map can be determined by using the following equation (Zhang and Yin 2021): (21) $S_i^c=\lceil {\displaystyle \frac{S_i^r}{s}\rceil ,i=h,w}$(21) where $S_i^c$ and $S_i^r$ are the size of images along the i^th dimension in the convolutional and input layers, respectively, $\lceil \rceil$ is the ceiling function which maps x to the least integer larger than or equal to x.

Figure 4. Memory cell of an LSTM-based neural network (Yu et al. 2019).

The pooling layers are subsequently used to further reduce the number of parameters and dimensionality, and thus improve the training efficiency and prevent overfitting. The size of a h^c × w^c × c^c × n^c feature map is rescaled to h^p × w^p × c^p × n^c through the pooling layer. This process is achieved by replacing a specific area of the input image with a designated value, commonly the maximum value (Maximum Pooling) or the average value (Average Pooling). Finally, the outputs from the pooling layer are flattened and pass through a fully connected network to implement classification or regression tasks.

Overall, three different baseline neural networks are introduced in this section. The physics-informed FCNN can be used to model almost all scientific problems. This is because an FCNN with multiple hidden layers and proper training can approximate complicated functions or even any universal functions (Sun, Sengupta, and Juniper 2023). When solving sequence-based problems, using LSTM neural networks as an underlying architecture for PINNs is recommended. Additionally, when solving a problem with gridded two-dimensional or even high-dimensional datasets, physics-informed CNNs can be employed to reduce computational costs and improve training efficiency (Zhu et al. 2019).

4. Enhanced PINNs and applications

4.1. PINN variants

Based on the traditional forms of PINNs, numerous researchers focus on inventing various PINN variants to overcome their limitations and improve their performance. For instance, various domain segmentation methods combined with PINNs were proposed to solve complex problems with large spatial or time domains, as shown in Table 1. Meng et al. (2020) proposed a parareal physics-informed neural network (PPINN) for the long-time integration of time-dependent PDEs. In this framework, the long-time domain is decomposed into small sub-domains with the same interval. As the computational cost of conventional PINNs dealing with large data sets is prohibitively expensive, this approach can achieve significant speed-up and high computational efficiency by parallelly computing on small data sets. Similarly, Jagtap, Kharazmi, and Karniadakis (2020) proposed a conservative physics-informed neural network (cPINN) on discrete sub-domains. After obtaining every single solution for each separate sub-domain by using different PINN architectures, the final complete solution will be generated by patching every single solution together. This strategy takes advantage of network parallelization and therefore improves the training efficiency and breaks the curse of dimensionality. Furthermore, the domain segmentation method was further explored by Jagtap and Karniadakis (2021) and the extended physics-informed neural network (XPINN) was proposed to solve any type of PDEs. In this framework, the temporal and spatial domains can be decomposed in any arbitrary way. The error values on the interface are added to the loss function to ensure the continuity between subdomains. However, this framework somehow still fails to fulfill the continuity conditions on the interface due to the high-order derivatives in PDEs. The XPINN was further improved by Hu et al. (2022) as they extended it to the augmented physics-informed neural network (APINN). APINN employs a soft domain decomposition method. To be specific, the subdomains in APINN are characterized by continuous divisions rather than rigid boundaries. This is achieved by using gating networks, which are pre-trained and assign weights for subdomains. In this way, APINN does not incorporate intricate interface losses to preserve the continuity between subdomains.

Table 1. Various domain decomposition methods for PINNs.

PINN variants	Features	References
PPINN	Discrete the long-time domain	Meng et al. (2020)
cPINN	Decomposite the whole time-space domain; For nonlinear conservation laws	Jagtap, Kharazmi, and Karniadakis (2020)
XPINN	Discrete the domain arbitrarily; Applicable to any PDEs	Jagtap and Karniadakis (2021)
APINN	A soft-mannered decomposition method can be discovered through optimization	Hu et al. (2022)
hp-VPINN	The loss is presented in a variational form	Kharazmi, Zhang, and Karniadakis (2021); Kharazmi, Zhang, and Karniadakis (2019)

Another type of sub-domain method involves hp-variational forms of PINNs (hp-VPINNs) (Kharazmi, Zhang, and Karniadakis 2021, 2019). Each sub-domain is projected onto the space of high-order polynomials. By applying variational principles to Equation 1, the loss ${\mathcal{L}}^{({\mathrm{e}}_{\mathrm{x}}{\mathrm{e}}_{\mathrm{t}})}$ of each sub-domain ${\mathrm{\Omega }}_{{\mathrm{e}}_{\mathrm{x}}{\mathrm{e}}_{\mathrm{t}}}$ is presented in the form of integrals (Kharazmi, Zhang, and Karniadakis 2021): (22) ${\mathcal{R}}_{k_1{k}_2}^{(e_x{e}_t)}=\int \int (u_t(t,\mathit{x})+\mathcal{D}[u(t,\mathit{x})]){\varphi }_{k_1}^{(e_x)}{\varphi }_{k_2}^{(e_t)}\mathrm{d}{\mathrm{\Omega }}_{e_x{e}_t}$(22) (23) ${\mathcal{L}}^{(e_x{e}_t)}={\displaystyle \frac{1}{K_1{K}_2}{\sum }_{k=1}^{K_1{K}_2}{\Vert {\mathcal{R}}_{k_1{k}_2}^{(e_x{e}_t)}\Vert }^2}$(23) where ${\mathcal{R}}_{k_1{k}_2}^{(e_x{e}_t)}$ is the variational residual, ${\varphi }_{{\mathrm{k}}_1}^{({\mathrm{e}}_{\mathrm{x}})}$ and ${\varphi }_{{\mathrm{k}}_2}^{({\mathrm{e}}_{\mathrm{t}})}$ are the test functions of the size K₁ and K₂ in two dimensions, respectively. The integrals can be computed through numerical integration techniques such as Gauss quadrature rules. The training results demonstrate the superior performance of hp-VPINNs compared to classical PINNs when solving equations with non-smooth solutions characterized by singularities, steep solutions and sharp changes.

Apart from addressing general PDEs and ODEs, the traditional framework of PINNs is modified to solve multiple types of equations such as fractional equations (Pang, Lu, and Karniadakis 2019), nonlinear integro-differential equations (Yuan et al. 2022; Lu et al. 2021), interger-order PDEs (Raissi, Perdikaris, and Karniadakis 2019) and stochastic PDEs (Zhang, Guo, and Karniadakis 2020). Furthermore, integrating the Fourier feature into PINNs allows the network to approximate high-frequency functions in a low-dimensional domain (Tancik et al. 2020). Besides, to quantify the uncertainty of the generated results, the Bayesian framework can be combined with traditional PINNs, which are so-called BPINNs (Yang, Xuhui, and Karniadakis 2021). Overall, various PINN variants have been proposed to bypass the limitations of traditional PINNs and to improve their performance in solving complex problems involving different types of equations and uncertainty quantification.

4.2. Sampling methods

To ensure the effectiveness and accuracy of PINNs before the iterative training procedure, the training data should be sampled within the solution domain and at the edge of boundaries, using various sampling methods. An effective sampling strategy plays a fundamental role in ensuring the convergence of PINNs (Daw et al. 2023). Typically, current sampling can be classified into two categories: a fixed sampling method and an adaptive sampling method.

The fixed sampling strategy collects data points at the outset of training and keeps them the same throughout the training process. These selected data points can be uniformly or randomly distributed (Sun et al. 2020; Zhu et al. 2019). Regarding the randomly distributed data, various sampling methods including normal sampling, Latin hypercube sampling (Rao, Sun, and Liu 2021), Sobol' sequence (Pang, Lu, and Karniadakis 2019; Guo et al. 2022), Halton sequence and Hammersley sequence (Wu et al. 2023) have been used. Figure 6 shows their distributions.

Figure 5. Framework of convolutional neural network (Zhang and Yin 2021).

Figure 6. Fixed sampling methods: (a) uniform; (b) normal sampling; (c) Latin hypercube sampling; (d) Sobol' sequence; (e) Halton sequence; (f) Hammersley sequence.

Regarding the adaptive sampling method, a well-known strategy for optimizing the distribution of collocation points in each training epoch was presented by Lu et al. (2021), which is the so-called residual-based adaptive refinement (RAR) method. The RAR method works by continuously adding collocation points in regions where the PDE residual is relatively large until the mean residual is smaller than a threshold, which improves the training efficiency. More adaptive sampling methods have been proposed to enhance the training efficiency of PINNs (Sun, Sengupta, and Juniper 2023; Wu et al. 2023; Nabian, Gladstone, and Meidani 2021). The core of these strategies is to increase the density of training points in regions with poor predictions so that the trained model can better capture the underlying physics of problems. These adaptive sampling strategies were further improved by Retain-Resample-Release (R3) sampling method (Daw et al. 2023). The R3 method works by retaining training points in high residual regions, releasing training points that have been resolved, and maintaining a non-zero representation of points resampled from a uniform distribution over the entire domain. When the solution domain is very complex, adaptive sampling methods are recommended to improve the performance of PINNs.

4.3. Loss functions

Another crucial factor in obtaining an effective and accurate PINN model is the formulation of loss functions, which is still an open question in the study of PINNs. There are two commonly used types of loss functions in the field of computational mechanics: collocation point-based loss functions and energy-based loss functions. Most PINN models available in the literature enforce collocation point-based loss functions (Equations (6)–(10)), which are based on the strong form of PDEs. The training process aims to reduce the residuals of PDEs and initial/boundary conditions on each collocation point. Adam optimizer (Kingma and Ba 2014) and L-BFGS optimizer (Liu and Nocedal 1989) are generally applied to minimize the loss value. Apart from the widely adopted MSE for residual measurement, alternatives such as mean absolute error (MAE) have also been adopted. For instance, Rezaei et al. (2022) incorporate both MSE and MAE in the loss function when training PINNs, in order to keep the values of different loss terms in the same order. Despite the extensive applications of the collocation point-based loss function due to its simple and straightforward manner, it still has inherent limitations. These limitations can be summarized as: (1) The gradient pathologies are widely observed while the loss function has multiple residual terms and each term has different scales, thus resulting in a local minima solution rather than the global minima (Wang, Teng, and Perdikaris 2021; Elhamod et al. 2022); (2) The trained PINN model based on the collocation point residuals may lack generalization. While PINNs can generate accurate values on sampled collocation points, the error in the rest domain may be relatively large (Alzubaidi et al. 2021).

An alternative form of loss functions, known as energy-based loss functions is also employed in the training of PINNs. Unlike the conventional strong-form collocation point-based loss function, this approach is theoretically based on the weak form of PDEs, wherein the equilibrium equations and Neumann boundary conditions are implicitly incorporated into the loss function. One of the representative works of applying energy-based loss function in PINNs has been done by Samaniego et al. (2020). In this research, a so-called deep energy method (DEM) was proposed based on the principle of minimum potential energy. Specifically, the overall potential energy of the system is minimized when it reaches an equilibrium state, and this minimization process can be seen as an optimization problem which can be easily implemented by neural networks (Lanczos 2012). Therefore, the potential energy of the system can be used as the formulation of loss functions, which can be described by using the following equation (Samaniego et al. 2020): (24) $\mathcal{L}(u{)}_{\mathrm{energy}}=\mathrm{\Pi }(u)=U(u)\text{–}W(u)$(24) where $\mathit{\Pi }$ is the total system potential energy, $U$ indicates the internal energy and W indicates the external work. Samaniego et al. (2020) have successfully applied the proposed DEM method for displacement prediction of structures with linear elasticity, elastodynamics, and hyperelasticity constitutive models. He et al. (2023) later extended the DEM method to elastoplasticity problems for displacement prediction. The results demonstrate that by using the DEM instead of collocation point-based functions, the solutions are more accurate and gradient calculation is more efficient. Furthermore, an enhanced DEM method was proposed by Nguyen-Thanh et al. (2021), wherein the isoparametric element was integrated with DEM to transfer the irregular physical domain to a regular square domain. More implementations of formulating energy-based loss functions based on the principle of minimum potential energy can be found in Li, Bazant, and Zhu (2021) and Zhuang et al. (2021).

The energy-based loss function can also be formulated by using the variational energy of the system. Take the general form of a PDE in Equation 1 as an example, its energy function $\mathcal{E}(u)$ can be described by using (Goswami et al. 2020; Goswami et al. 2022): (25) $\mathcal{E}(u)=\int (\mathcal{g}[u_t(t,\mathit{x})+\mathcal{D}[u(t,\mathit{x})]])\mathrm{d}\mathrm{\Omega }$(25) where $\mathcal{g}$ is a differentiable function. It should be emphasized that all derivative terms in the $\mathcal{E}(u)$ can be efficiently and automatically calculated by AD.

In contrast to the computation of collocation point-based loss functions, the implementation of integrals is essential when computing energy-based loss functions. However, as it is almost impossible to analytically compute the complex integrals, the value of the integral should be approximated via various numerical integration methods, such as the Gauss quadrature rules (Kharazmi, Zhang, and Karniadakis 2019), the quasi-Monte Carlo integration (Morokoff and Caflisch 1995) and sparse grid quadratures (Smolyak 1963). Compared with conventional collocation point-based loss functions, energy-based loss functions have a lower order of derivatives, and therefore contribute to faster convergence (Goswami et al. 2020). Besides, the Neumann boundary condition and traction-free boundary condition can be automatically satisfied in the formulation of energy-based loss functions. However, the Dirichlet boundary conditions can only be exactly satisfied by modifying the output of the neural network (Goswami et al. 2020; Samaniego et al. 2020; Rezaei et al. 2022). The modification process could be challenging and intricate, particularly when dealing with complex or discontinuous boundary conditions. Li, Bazant, and Zhu (2021) also conducted a comparative analysis of collocation point-based and energy-based loss functions, evaluating the respective advantages and disadvantages of each. To date, more formulations of loss functions have been proposed, aiming at overcoming different difficulties when training PINNs. For instance, Fuhg and Bouklas (2022) proposed a mix-form of loss functions that combine the collocation point-based loss function with the energy-based loss function together. In this research, the collocation point-based loss function consists of residuals of governing equations and boundary conditions, and it was summed up with energy-based loss terms. By using this novel form of loss functions, the stress and displacement concentration features can be greatly resolved. Bai et al. (2023) also modified the loss function by using the least squares weighted residual method, and applied the modified framework for solving two-dimensional and three-dimensional solid mechanics problems. The training results showed that this modified loss function can predict both the displacement and stress field with accuracy and robustness.

4.4. Applications

In the field of geoengineering, extensive research has been done to investigate the feasibility of utilizing PINNs for geotechnical simulations. For instance, Bekele (2021) employed PINNs to successfully solve forward and inverse Terzaghi's one-dimensional consolidation problems. The application of PINNs for one-dimensional consolidation problems was further promoted by Zhang, Yin, and Sheil (2023b). This research systematically incorporated surcharge and vacuum loading conditions as boundary conditions. Furthermore, a Monte Carlo-based dropout uncertainty analysis was integrated into PINNs to effectively capture the inherent uncertainty characteristics of geotechnical problems. Another enhanced PINN model, the so-called affine physics–informed neural network (AfPINN) has also been proposed by Mandl et al. (2023) to solve consolidation problems. In this research, each output of the network should be rescaled by using scaling and offset factors, which can be understood as an affine transformation. In this case, the distribution of output variables can be tailored to be closer to the desired distribution. Besides, by scaling the gradient, the optimization process can be accelerated. When applying this AfPINN to the one-dimensional consolidation case, it generated more accurate results than conventional PINNs. The application of PINNs for simple consolidation problems was further expanded to more complex scenarios such as two-dimensionality (Lu and Mei 2022) or nonlinearity (Lan et al. 2023). PINNs have also been applied to modelling unsaturated flow in porous media. For instance, Depina et al. (2022) successfully employed PINNs to solve inverse groundwater flow problems modelled with the Richards partial differential equation and the vanGenuchten constitutive model. Haghighat, Amini, and Juanes (2022) also conducted a study on modelling unsaturated laminar flow in porous media, and they demonstrated that formulating the problem in the dimensionless form, along with a sequential training strategy and adaptive weighting methods significantly improved the stability of the PINN approach. Yang and Mei (2022) utilized PINNs to simulate the soil-water infiltration process, and the generated predictions agree with the numerical results. The applications of PINNs to other geotechnical problems have also been investigated, such as tunnelling (Zhang et al. 2023; Xu et al. 2023) and pile-soil interaction modelling (Vahab et al. 2023).

Overall, despite the great potential that PINNs have showcased, the applications of PINNs in the geotechnical domain are still sparse. Geoengineering problems are characterized by inherent complexity, and it is still challenging to accurately simulate complicated geoengineering problems through PINNs. More attention deserves to be paid to the geotechnical community.

5. Limitations and challenges

Despite the remarkable success that PINNs have achieved, there still exist limitations and challenges. One of the main challenges is modelling multiscale and multiphysics systems. Despite the outstanding success that PINNs have achieved in modelling complex systems (Lin et al. 2021; Liu and Wang 2021; Shin, Darbon, and Karniadakis 2020), it still requires tremendous investigations and development to improve the applicability of PINNs in engineering-scale problems. When modelling complex multiscale systems, the outputs of traditional PINNs may exhibit inaccuracies. Specifically, the networks excel in capturing low-frequency features whereas show weak performance in capturing high-frequency features (Rahaman et al. 2019; Leung, Lin, and Zhang 2022). When modelling multiphysics problems, the computational costs could be expensive to learn multiple physics parameters simultaneously (Karniadakis et al. 2021).

To better capture the high-frequency features in multiscale problems, some novel methods are proposed to address this issue. One of the most commonly used techniques is domain decomposition (Meng et al. 2020; Jagtap, Kharazmi, and Karniadakis 2020; Jagtap and Karniadakis 2021; Hu et al. 2022), which targets discretizing the time or space domain and calculating each sub-domain separately. Another efficient technique is passing the inputs through a Fourier feature mapping, which enables networks to learn high-frequency functions with fewer iterations (Tancik et al. 2020; Wang, Wang, and Perdikaris 2021). To better solve multiphysics problems with lower computational costs, using DeepM&Mnet is an effective way to construct complex models that can generate coupled solutions after learning each physics separately (Mao et al. 2021; Cai et al. 2021). The building block of this DeepM&Mnet framework is the DeepOnets, which is proposed by Lu et al. (2021) and is able to learn nonlinear functions accurately and effectively. Despite these advancements, we are still in the very early stages of tackling multiscale and multiphysics problems in engineering practice. Therefore, more novel techniques remain to be explored to improve the performance of PINNs in multiscale and multiphysics modelling.

Another challenge for PINNs is their performance in dealing with complex constitutive models, particularly in the geotechnical domain. Soils are characterized by various complex behaviours including multiphysics coupling (Yang et al. 2019), high dimensionality (Bi et al. 2021), anisotropy (Yin et al. 2010), non-coaxiality (Tian and Yao 2017), stress dilatancy (He et al. 2022), time-dependence (Wang et al. 2018), stress-path dependence (Chang and Zhang 2013). An attempt to apply PINNs to elastoplastic models has been made by Haghighat et al. (2021), and the authors pointed out that standard physics-constrained multi-layer neural networks may fail to capture localised deformation in elastoplastic problems. Haghighat, Abouali, and Vaziri (2023) also proposed an elastoplastic PDE solver to deal with constitutive models. However, this PDE solver was only validated in uniaxial and biaxial loading conditions. While the real loading condition in nature is much more complex, the effectiveness of the proposed methodology may not be guaranteed under more complex conditions. Moreover, simulating constitutive relations always requires a large amount of strain-stress data, especially when the complexity of constitutive relations increases. To better modelling constitutive relations without resorting to deeper and wider PINNs, as well as prevent the need for a large dataset, the multi-fidelity residual framework has been integrated with PINNs (Zhang, Yin, and Sheil 2023a; Zhang et al. 2022). Besides, some successful trials of utilizing PINNs for modelling soil behaviours have been done, including path-dependent behaviour (Zhang et al. 2022), through which the potential of PINNs to capture complex soil behaviours is demonstrated. However, this topic of research still has a limited number of available studies.

Actually, the fundamental obstacle underlying the aforementioned problems is the convergence issue. Given that most physics-informed learning involves minimizing complicated loss functions that have multiple loss terms, there would be non-convex optimization difficulties and therefore leading to the instability of the training process and convergence to local minima or saddle points (Karniadakis et al. 2021; Lee et al. 2016; Blum and Rivest 1992). For example, Wang, Yu, and Perdikaris (2022) employed a PINN to solve a nonlinear hyperbolic transport PDE with a non-convex flux function. In this work, the neural network completely fails to identify the exact solution. Researchers have been endeavouring to improve convergence by optimizing the architecture of neural networks or using adaptive activation functions (Jagtap, Kawaguchi, and Karniadakis 2020; Jagtap, Kawaguchi, and Karniadakis 2020; Wang, Teng, and Perdikaris 2021). Additionally, to mitigate the imbalance of different loss terms to the total loss, a self-adaptive loss balancing strategy was tailored to achieve higher accuracy and better convergence (Xiang et al. 2022; Anagnostopoulos, Toscano, and Stergiopulos 2023). Another example is to use curriculum regularization or pose the problem as a sequence-to-sequence learning task to alleviate the optimization difficulties (Wang, Yu, and Perdikaris 2022). While significant progress has been made in developing convergence acceleration strategies for PINNs, there is a growing need to investigate more flexible optimization methods to improve the stability of PINNs. A key issue of this endeavour invloves digging out more theoretical relationships between optimization or acceleration algorithms and the performance of PINNs.

6. Conclusions

This study provided a comprehensive review of the current state of physics-informed neural networks (PINNs), which provides an alternative solver for deriving solutions of partial differential equations (PDEs). Firstly, the framework for employing PINNs in solving forward and inverse PDEs was introduced, followed by an elaboration of three baseline neural networks including fully-connected neural networks (FCNN), long short-term memory (LSTM) neural networks and convolutional neural networks (CNN) used for developing PINNs. Then, some novel strategies that focus on improving the performance of PINNs were introduced from three aspects: PINN variants, sampling methods including fixed and adaptive sampling strategy, and loss functions involving collocation point-based and energy-based formulations. Finally, the limitations and challenges of current PINNs were summarized. The main conclusions are made as follows:

1. FCNN is still the predominant choice for training PINNs and it has been widely used in engineering practice. The LSTM neural network characterizes handling sequence-based problems and CNN exhibits great potential to solve problems with large-scale spatial-temporal domains or high-dimensionality. Further research is anticipated to explore the integration of PINNs with various neural networks, enabling PINNs to tackle a broader spectrum of engineering problems.

2. Various domain decomposition methods have been proposed to improve the training efficiency of PINNs and bypass the curse of dimensionality when dealing with large and complex domains. Two representative models are the extended physics-informed neural network (XPINN) and the hp-variational physics-informed neural network (hp-VPINN). XPINN can be used to discrete domain arbitrarily and therefore has been used to solve any type of PDEs. hp-VPINN presents the loss in each subdomain in a variational form and showcases their accuracy and effectiveness when tackling non-smooth functions.

3. Two sampling methods, fixed sampling and adaptive sampling are compared. Fixed sampling methods, characterized by keeping training datasets unchanged throughout the training procedure, dominate the application of sampling in PINNs due to their simplicity. However, in the context of solving complex problems in which convergence problems may occur, adaptive sampling methods are recommended.

4. The framework of two types of loss functions is introduced and compared in this study. The collocation point-based loss functions, which incorporate residuals of PDEs/ODEs, measured data, and initial/boundary conditions in sampled points, are widely adopted when training PINNs. On the other hand, the energy-based loss functions, characterized by their relatively lower order of derivatives, have exhibited superior performance to traditional collocation point-based loss functions in terms of faster convergence, but energy-based loss functions have been rarely used. Therefore, further research efforts are expected to investigate the applicability and effectiveness of using energy-based loss functions or even more formulations of loss functions in PINNs.

5. Current PINNs still encounter challenges when addressing complex systems, including multiscale or multiphysics problems, as well as modelling complex constitutive relations. Although some trials have been made to overcome limitations, most of these trials are still limited to simple nonlinear functions (e.g. low dimensional and single phase) and their applicability to engineering-scale problems may not be guaranteed. These limitations primarily stem from poor convergence. In order to prompt the application of PINNs and prove their significance in solving practical problems, there is a need for the development of additional learning algorithms that target enhancing convergence and accelerating the optimization process.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

All data that support the findings of this study are available from the corresponding author upon reasonable request.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by Hong Kong Polytechnic University; Research Grants Council (RGC) of the Hong Kong Special Administrative Region Government (HKSARG) of China: [grant number 15220221; 15229223]; Royal Society under the Newton International Fellowship.