Continuous analogue to iterative optimization for PDE-constrained inverse problems

ABSTRACT

The parameters of many physical processes are unknown and have to be inferred from experimental data. The corresponding parameter estimation problem is often solved using iterative methods such as steepest descent methods combined with trust regions. For a few problem classes also continuous analogues of iterative methods are available. In this work, we expand the application of continuous analogues to function spaces and consider PDE (partial differential equation)-constrained optimization problems. We derive a class of continuous analogues, here coupled ODE (ordinary differential equation)–PDE models, and prove their convergence to the optimum under mild assumptions. We establish sufficient bounds for local stability and convergence for the tuning parameter of this class of continuous analogues, the retraction parameter. To evaluate the continuous analogues, we study the parameter estimation for a model of gradient formation in biological tissues. We observe good convergence properties, indicating that the continuous analogues are an interesting alternative to state-of-the-art iterative optimization methods.

KEYWORDS:

• Partial differential equations
• optimization
• continuous analogues
• mathematical biology
• steady state

MSC SUBJECT CLASSIFICATIONS:

• 93D20
• 49N45
• 35K57
• 37N40

1. Introduction

Partial differential equations (PDEs) are used in various application areas to describe physical, chemical, biological, economic, or social processes. The parameters of these processes are often unknown and need to be estimated from experimental data [1,2]. The maximum likelihood and maximum a posteriori parameter estimates are given by the solutions of PDE-constrained optimization problems. Since PDE-constrained optimization problems are in general nonlinear, non-convex as well as computationally challenging, efficient and reliable optimization methods are required.

Over the last decades, a large number of numerical methods for PDE-constrained optimization problems have been proposed (see [3–8] and references therein). Beyond methodological contributions, there is numerous literature on parameter estimation in special applications available [9]. Most of the theoretical and applied work focuses on iterative methods, which generate a discrete sequence of points along which the objective function decreases. In this manuscript, we pursue an alternative route and consider continuous analogues (see [10–14] and references therein).

Continuous analogues are formulations of iterative optimization methods in terms of differential equations and have been derived for a series of optimizers for real-valued optimization problems, including the Levenberg–Marquardt and the Newton–Raphson method [11,15]. For many constrained and unconstrained optimization problems, continuous analogues exhibit larger regions of attraction and more robust convergence than discrete iterative methods [15]. In a recent study, the concept of continuous analogues has been employed to develop a tailored parameter optimization method for ordinary differential equation (ODE) models with steady-state constraints [16]. This method outperformed iterative constrained optimization methods with respect to convergence and computational efficiency [16], and complemented methods based on computer algebra [17]. Although continuous analogues are generally promising, we are not aware of continuous analogues for solving PDE-constrained optimization problems.

In this manuscript, we formulate a continuous analogue for PDE-constrained optimization problems based on the structure of the PDE and the local geometry of its solution space. The continuous analogue is a coupled system of ODEs and PDEs, which has the optima of the PDE-constrained optimization problem as equilibrium points. We establish local stability and convergence under mild assumptions. The continuous analogue enables us to use adaptive numerical methods for solving optimization problems with PDE constraints. Beyond the generalization of previous results for ODEs (i.e. [16]) to function spaces, we provide rationales and constraints for the choice of tuning parameters, in particular of the retraction factor. The methods are applied to a model of gradient formation in biological tissues.

The manuscript is structured as follows . In Sections 2 and 3, we introduce the considered class of models and PDE-constrained optimization problems. For these classes, we propose a continuous analogue to descent methods in Section 4. In Section 5, we prove convergence of the continuous analogue to local optima. The assumptions under which stability and convergence to local optima are achieved are discussed in Section 6. The properties of the continuous analogues are studied in Section 7 for a model of gradient formation in biological tissues.

2. Mathematical model

We consider parameter estimation for models with elliptic and parabolic PDE constraints. In the parabolic case, the initial condition $u_0$ of the parabolic PDE is defined as the solution of an elliptic PDE: (1) $\begin{array}{rl}{u}_t\left(t\right)& =C(\theta ,t,u(t\left)\right)t\in ]0,T[\\ u\left(0\right)& =u_0,\end{array}$(1) with (2) $0=C_0(\theta ,u_0),$(2) in which $u\left(t\right)$ is the time-dependent solution of the elliptic PDE, and C and $C_0$ are operators. The considered combination of parabolic and elliptic PDEs is flexible and allows for the analysis of many practically relevant scenarios. Here, $u_0$ denotes a stable steady state of the unperturbed system, while u denotes the transient solution of the perturbed system starting in the steady state of the unperturbed system $u_0$. The observables of the models are defined via observation operators $B(\theta ,t,u(t\left)\right)$ and $B_0(\theta ,u_0)$ (3) $y\left(t\right)=B(\theta ,t,u(t\left)\right),y_0=B_0(\theta ,u_0).$(3) We work in the following functional analytic setting. Let V be a separable Banach space with dual space $V^{\ast }$ and let V be continuously and densely embedded into a Hilbert space H, such that $V\subseteq H\cong H^{\ast }\subseteq V^{\ast }$ forms a Gelfand triple. Furthermore, we consider the initial value $u_0\in V$ and the transient state $u\in W(0,T)=L^2(0,T;V)\cap H^1(0,T;V^{\ast })$. The parameters θ are assumed to be finite dimensional and real-valued, $\theta \in {\mathbb{R}}^{n_{\theta }}$. For each t and θ, the observation operators B, $B_0$ are mappings, $B:{\mathbb{R}}^{n_{\theta }}\times ]0,T[\times V\to Z$, $B_0:{\mathbb{R}}^{n_{\theta }}\times V\to Z_0$, where the observation spaces $Z,Z_0$ are Hilbert spaces. The operators C and $C_0$ are mappings, $C:{\mathbb{R}}^{n_{\theta }}\times ]0,T[\times V\to V^{\ast }$ and $C_0:{\mathbb{R}}^{n_{\theta }}\times V\to V^{\ast }$. Existence of a weak solution holds, e.g. under the following assumptions on the differential operator C ([18] p.770 ff.):

Assumption 2.1

There exists ${\epsilon }_{\theta }>0$ such that for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)=\{\theta \in {\mathbb{R}}^{n_{\theta }}|\parallel \theta -{\theta }^{\ast }\parallel <{\epsilon }_{\theta }\}\subseteq {\mathbb{R}}^{n_{\theta }},$ the following holds.

• The operator $-C(\theta ,t,\cdot )$ is monotone and hemicontinuous for each $t\in ]0,T[$.

• The operator $-C(\theta ,t,\cdot )$ is coercive for each $t\in ]0,T[,$ i.e. there exist constants $c_1>0$ and $c_2\ge 0$ such that $-{⟨C(\theta ,t,v),v⟩}_{V^{\ast },V}\ge c_1\parallel v{\parallel }_V^2-c_2\text{for all}v\in V,t\in ]0,T[.$

• The operator $C(\theta ,\cdot ,\cdot )$ satisfies a growth condition, i.e. there exist a non-negative function $c_3\in L^2(0,T)$ and a constant $c_4>0$ such that $\parallel C(\theta ,t,v){\parallel }_{V^{\ast }}\le c_3\left(t\right)+c_4\parallel v{\parallel }_V\text{for all}v\in V,t\in ]0,T[.$

• The function $t\mapsto ⟨C(\theta ,\cdot ,w),v{⟩}_{V^{\ast },V}$ is measurable on $]0,T[$ for all $v,w\in V$.

For the differential operator $C_0$, the first two items in Assumption 2.1 are assumed to hold, cf. Assumption 5.2 below.

In mathematical biology, the differential operator C is often semilinear and describes a reaction–diffusion–advection equation, $C(\theta ,t,u)=f(k,u)-{\mathrm{\nabla }}_x\cdot (vu-D{\mathrm{\nabla }}_{x}u),$ in which $u\left(t\right)\in V$ is a concentration vector, $x\in \mathrm{\Omega }\subseteq {\mathbb{R}}^n$ is the spatial location and $f:{\mathbb{R}}^{n_k}\times V\to V^{\ast }$ is the reaction term. The parameters $\theta =(k,v,D)$ are the velocity vector $v\in {\mathbb{R}}^n$, the diffusion matrix $D\in {\mathbb{R}}^{n\times n}$ and the kinetic parameters $k\in {\mathbb{R}}^{n_k}$. If D is positive definite and f fulfils a certain growth condition, then this operator satisfies Assumption 2.1.

3. Parameter estimation problem

We consider the estimation of the unknown model parameters θ from noise-corrupted measurements of the observables y. To obtain estimates for the unknown parameters, an objective function is minimized, e.g. the negative log-likelihood function or the sum-of-squared-residuals. In the following, we distinguish two cases.

3.1. Elliptic and parabolic PDE constraints

In the general case, observations are available for the initial state and the transient phase. The objective function $\tilde{J}$ depends on the parameters and the parameter-dependent solutions of the parabolic and the elliptic PDE, $\tilde{J}:{\mathbb{R}}^{n_{\theta }}\times V\times W(0,T)\to \mathbb{R}$, e.g. $\tilde{J}(\theta ,u_0,u)=\frac{1}{2}\parallel y_0-B_0(\theta ,u_0){\parallel }_{Z_0}^2+\frac{1}{2}{\int }_0^T\parallel y\left(t\right)-B(\theta ,t,u){\parallel }_Z^2\mathrm{d}t$. The optimization problem is given by (4) $\begin{array}{rl}\underset{\theta ,u_0,u}{min}& \tilde{J}(\theta ,u_0,u)\\ \text{s.t.}& u_t=C(\theta ,t,u)u\left(0\right)=u_0\\ & 0=C_0(\theta ,u_0).\end{array}$(4)

3.2. Elliptic PDE constraint

In many applications, only experimental data for the steady state of a process are available. Possible reasons include fast equilibration of the process and limitations of experimental devices. In this case, the problem is simplified and the objective function J depends only on the parameters and parameter-dependent solutions of the elliptic PDE, $J:{\mathbb{R}}^{n_{\theta }}\times V\to \mathbb{R}$, e.g. $J(\theta ,u_0)=\frac{1}{2}\parallel y_0-B_0(\theta ,u_0){\parallel }_{Z_0}^2$. The optimization problem is given by (5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) The reduced formulation of the optimization problem (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ) is given by (6) $\underset{\theta }{min}j\left(\theta \right):=J(\theta ,{\varphi }_0(\theta \left)\right),$(6) in which ${\varphi }_0\left(\theta \right)$ denotes the parameter-dependent solution of $C_0(\theta ,u_0)=0$, and $j:{\mathbb{R}}^{n_{\theta }}\to \mathbb{R}$ denotes the reduced objective function.

4. Continuous analogue of descent methods for PDE-constrained problems

Optimization problems of types (4(4) $\begin{array}{rl}\underset{\theta ,u_0,u}{min}& \tilde{J}(\theta ,u_0,u)\\ \text{s.t.}& u_t=C(\theta ,t,u)u\left(0\right)=u_0\\ & 0=C_0(\theta ,u_0).\end{array}$(4) ) and (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ) are currently often solved using iterative descent methods, combined with trust regions. In the following, we develop a continuous analogue of an iterative descent method for PDEs. For simplicity, we first consider the case of an elliptic PDE constraint  (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ) and afterwards generalize the results to the case of mixed parabolic and elliptic PDE constraints (4(4) $\begin{array}{rl}\underset{\theta ,u_0,u}{min}& \tilde{J}(\theta ,u_0,u)\\ \text{s.t.}& u_t=C(\theta ,t,u)u\left(0\right)=u_0\\ & 0=C_0(\theta ,u_0).\end{array}$(4) ).

4.1. Elliptic PDE Constraint

To solve optimization problem (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ), we derive a coupled ODE–PDE system. The trajectory of this continuous analogue evolves in parameter and state space on the manifold defined by $C_0(\theta ,u_0)=0$ towards a local minimum. To evolve on the manifold, the continuous analogue exploits the first-order geometry of the manifold, i.e. its tangent space.

Mathematically, the first-order geometry of $C_0(\theta ,{\varphi }_0(\theta \left)\right)=0$ is defined by the sensitivity equations (7) $\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,{\varphi }_0(\theta \left)\right)\frac{\mathrm{\partial }{\varphi }_0}{\mathrm{\partial }{\theta }_i}\left(\theta \right)+\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,{\varphi }_0(\theta \left)\right)=0,i\in \{1,\dots ,n_{\theta }\}.$(7) The sensitivity equations can be reformulated to $\frac{\mathrm{\partial }{\varphi }_0}{\mathrm{\partial }{\theta }_i}\left(\theta \right)=-{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,{\varphi }_0(\theta \left)\right)\right)}^{-1}\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,{\varphi }_0(\theta \left)\right),i\in \{1,\dots ,n_{\theta }\},$ provided the inverse of $\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)(\theta ,{\varphi }_0(\theta \left)\right)$ exists. We extend ${\mathrm{\nabla }}_{\theta }{\varphi }_0$ to points $(\theta ,u_0)$ not necessarily lying on the solution manifold of $C_0=0$ by defining the operator $S_0:{\mathbb{R}}^{n_{\theta }}\times V\to V^{n_{\theta }}$ that provides the solution of the sensitivity equations (8) $\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0){S_0}_i(\theta ,u_0)+\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)=0,i\in \{1,\dots ,n_{\theta }\},$(8) for given θ and $u_0$. With (7(7) $\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,{\varphi }_0(\theta \left)\right)\frac{\mathrm{\partial }{\varphi }_0}{\mathrm{\partial }{\theta }_i}\left(\theta \right)+\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,{\varphi }_0(\theta \left)\right)=0,i\in \{1,\dots ,n_{\theta }\}.$(7) ), it holds that (9) $S_0(\theta ,{\varphi }_0(\theta \left)\right)={\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right),$(9) provided the solutions to (7(7) $\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,{\varphi }_0(\theta \left)\right)\frac{\mathrm{\partial }{\varphi }_0}{\mathrm{\partial }{\theta }_i}\left(\theta \right)+\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,{\varphi }_0(\theta \left)\right)=0,i\in \{1,\dots ,n_{\theta }\}.$(7) ) and (8(8) $\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0){S_0}_i(\theta ,u_0)+\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)=0,i\in \{1,\dots ,n_{\theta }\},$(8) ) are unique.

In order to couple changes in θ with appropriate changes in $u_0$, we now use the fact that the function ${\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)$ provides the first-order term of the Taylor series expansion of the steady state with respect to the parameter vector θ, (10) ${\varphi }_0(\theta +r\mathrm{\Delta }\theta )={\varphi }_0\left(\theta \right)+{\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)r\mathrm{\Delta }\theta +o\left(r\right)\text{as}r\to 0,r\in \mathbb{R}.$(10) Defining ${\hat{\varphi }}_0\left(r\right):={\varphi }_0(\theta +r\mathrm{\Delta }\theta )$ for some $\mathrm{\Delta }\theta \in {\mathbb{R}}^{n_{\theta }}$ and differentiating (10(10) ${\varphi }_0(\theta +r\mathrm{\Delta }\theta )={\varphi }_0\left(\theta \right)+{\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)r\mathrm{\Delta }\theta +o\left(r\right)\text{as}r\to 0,r\in \mathbb{R}.$(10) ) with respect to r yields (11) $\frac{\mathrm{d}{\hat{\varphi }}_0}{\mathrm{d}r}\left(r\right)={\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)\mathrm{\Delta }\theta +o\left(1\right)=S_0(\theta +r\mathrm{\Delta }\theta ,{\hat{\varphi }}_0(r\left)\right)\mathrm{\Delta }\theta +o\left(1\right)\text{as}r\to 0.$(11) This relation motivates the formulation of the coupled ODE–PDE model (12) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)=S_0(\theta ,u_0)g(\theta ,u_0),u_0\left(0\right)=u_{0,0},\end{array}$(12) using the artificial time parameter r. For a change in the parameters $\mathrm{d}\theta /\mathrm{d}r$, the update in $u_0$ is chosen according to (11(11) $\frac{\mathrm{d}{\hat{\varphi }}_0}{\mathrm{d}r}\left(r\right)={\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)\mathrm{\Delta }\theta +o\left(1\right)=S_0(\theta +r\mathrm{\Delta }\theta ,{\hat{\varphi }}_0(r\left)\right)\mathrm{\Delta }\theta +o\left(1\right)\text{as}r\to 0.$(11) ). Solutions of this dynamical system evolve on the manifold $C_0\left(\theta \right(r),u_0(r\left)\right)=0$ for arbitrary parameter update directions $g:{\mathbb{R}}^{n_{\theta }}\times V\to {\mathbb{R}}^{n_{\theta }}$, provided that the initial state is on the manifold $C_0({\theta }_0,u_{0,0})=0$. The state variables of this coupled ODE–PDE system are θ and $u_0$, and the path variable is r. To solve optimization problem (6(6) $\underset{\theta }{min}j\left(\theta \right):=J(\theta ,{\varphi }_0(\theta \left)\right),$(6) ), g is chosen as an arbitrary descent direction satisfying $\mathrm{\nabla }j\left(\theta {)}^{\mathrm{T}}g\right(\theta ,u_0)<0,$ more precisely satisfying Assumption 5.5 below. For example, g can be chosen as a steepest descent direction (13) $g=\underset{\parallel v{\parallel }_{\ast }\le 1}{\mathrm{argmin}}\mathrm{\nabla }j(\theta {)}^{\mathrm{T}}v$(13) for some norm $\parallel \cdot {\parallel }_{\ast }$. For the Euclidian norm, we obtain the gradient descent direction, (14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) in which we substituted ${\varphi }_0\left(\theta \right)$ by $u_0$ extending the definition also to states $u_0$ that are not on the steady-state manifold. Likewise, defining $\parallel v{\parallel }_{\ast }^2:=v^{\mathrm{T}}H(\theta ,u_0)v$ with some positive definite matrix $H(\theta ,u_0)$, so that (15) $g(\theta ,u_0)=H(\theta ,u_0{)}^{-1}d(\theta ,u_0)$(15) leads to a descent direction. Using, e.g. the Hessian of j, (16) $\begin{array}{rl}{H}_{ij}(\theta ,u_0)& =\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{\theta }_i\mathrm{\partial }{\theta }_j}(\theta ,u_0)+{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_j}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}\\ & +{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_i}(\theta ,u_0),{S_0}_j(\theta ,u_0)⟩}_{V^{\ast },V}+\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0^2}(\theta ,u_0)\left({S_0}_i(\theta ,u_0),{S_0}_j(\theta ,u_0)\right)\\ & +{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{T_0}_{i,j}(\theta ,u_0)⟩}_{V^{\ast },V},i,j\in \{1,\dots ,n_{\theta }\},\end{array}$(16) with the second-order sensitivities defined by $\begin{array}{rl}& \frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0){T_0}_{i,j}(\theta ,u_0)+\frac{{\mathrm{\partial }}^2{C}_0}{\mathrm{\partial }{u}_0^2}(\theta ,u_0)\left({S_0}_i(\theta ,u_0),{S_0}_j(\theta ,u_0)\right)+\frac{{\mathrm{\partial }}^2{C}_0}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_j}(\theta ,u_0){S_0}_i(\theta ,u_0)\\ & +\frac{{\mathrm{\partial }}^2{C}_0}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_i}(\theta ,u_0){S_0}_j(\theta ,u_0)+\frac{{\mathrm{\partial }}^2{C}_0}{\mathrm{\partial }{\theta }_i\mathrm{\partial }{\theta }_j}(\theta ,u_0)=0,i,j\in \{1,\dots ,n_{\theta }\},\end{array}$ leads to Newton, Gauss-Newton (upon skipping the ${T_0}_{i,j}$ terms) or quasi-Newton methods (where approximations to the Hessian (16(16) $\begin{array}{rl}{H}_{ij}(\theta ,u_0)& =\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{\theta }_i\mathrm{\partial }{\theta }_j}(\theta ,u_0)+{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_j}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}\\ & +{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_i}(\theta ,u_0),{S_0}_j(\theta ,u_0)⟩}_{V^{\ast },V}+\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0^2}(\theta ,u_0)\left({S_0}_i(\theta ,u_0),{S_0}_j(\theta ,u_0)\right)\\ & +{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{T_0}_{i,j}(\theta ,u_0)⟩}_{V^{\ast },V},i,j\in \{1,\dots ,n_{\theta }\},\end{array}$(16) ) are computed via low rank updates). As the Hessian is not guaranteed to be positive definite, regularization with a scaled identity matrix, $H_{\mu }(\theta ,u_0)=H(\theta ,u_0)+\mu I$ with $\mu >0$, might be useful. However, how to choose μ and possible continuous update rules are out of the scope of this work. The coupled ODE–PDE systems (12(12) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)=S_0(\theta ,u_0)g(\theta ,u_0),u_0\left(0\right)=u_{0,0},\end{array}$(12) ) can be solved using numerical time-stepping methods. These numerical methods might, however, accumulate errors resulting in the divergence of the state $\left(\theta \right(r),u_0(r\left)\right)$ from the steady-state manifold. Additionally, the initial state, $u_{0,0}$, might not be on the steady-state manifold. To account for this, we include the retraction term $\lambda C_0(\theta ,u_0)$ in the evolution equation of $u_0$, with retraction factor $\lambda >0$. This yields the following continuous analogue of a descent method for optimization problems with elliptic PDE constraints, (17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) As, for fixed θ, the equation $C_0(\theta ,u_0)=0$ defines a stable steady state of the PDE (2(2) $0=C_0(\theta ,u_0),$(2) ), the retraction term stabilizes the manifold. For $\lambda \gg 1$, the system should first converge to the steady state ${\varphi }_0\left({\theta }_0\right)$ for the initial parameter ${\theta }_0$ and then move along the manifold to a local optimum ${\theta }^{\ast }$ as illustrated in Figure 1.

Figure 1. The state of the system is illustrated along the trajectory of (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ). In the first phase, the equilibration phase, the system converges to the manifold. The solution is not feasible during this phase as the equality constraint, $C_0(\theta ,u_0)=0$, is violated. In the course of the equilibration, the objective function value might increase. In the second phase, the minimization phase, the objective function is minimized along the steady-state manifold.

4.2. Elliptic and parabolic PDE constraints

The continuous analogue for descent with elliptic PDE constraints can be generalized to problems with parabolic and elliptic PDE constraints. One possibility for doing so is to consider the partially reduced problem (18) $\begin{array}{rl}\underset{\theta ,u_0}{min}& \stackrel{\mathcal{~}}{\mathcal{J}}(\theta ,u_0):=\tilde{J}(\theta ,u_0,\varphi (\theta ,u_0\left)\right)\\ \text{s.t.}& 0=C_0(\theta ,u_0),\end{array}$(18) in which $u=\varphi (\theta ,u_0)$ denotes the solution to $u_t=C(\theta ,t,u)$ with $u\left(0\right)=u_0$. Given this formulation, we can use continuous analogue (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) with (19) ${\tilde{g}}_i(\theta ,u_0)=-\frac{\mathrm{\partial }\tilde{\mathcal{J}}}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-\frac{\mathrm{\partial }\tilde{\mathcal{J}}}{\mathrm{\partial }{u}_0}(\theta ,u_0){S_0}_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\}.$(19) To avoid the need for the solution operator $\varphi :{\mathbb{R}}^{n_{\theta }}\times V\to W(0,T)$, alternatively, a continuous analogue of the full problem can also be formulated. This is beyond the scope of this study, though, and will be subject of future research.

5. Local stability and convergence to a local optimum

The behaviour of the coupled ODE–PDE system (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) introduced in the previous section depends on the properties of the objective function and the PDE model, as well as the retraction factor λ. To prove that a solution of (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) with an appropriate retraction factor λ is well defined and converges to the local minimizer $({\theta }^{\ast },u_0^{\ast })=({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)$ of (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ), we impose the following assumptions.

Assumption 5.1

The descent direction vanishes at the minimizer ${\theta }^{\ast }$ of the optimization problem $\underset{\theta }{min}j\left(\theta \right),$ $g({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)=0.$

Assumption 5.2

There exists ${\epsilon }_{\theta }>0$ such that for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)=\{\theta \in {\mathbb{R}}^{n_{\theta }}|\parallel \theta -{\theta }^{\ast }\parallel <{\epsilon }_{\theta }\}\subseteq {\mathbb{R}}^{n_{\theta }},$ the following holds.

• The operator $-C_0(\theta ,\cdot )$ is monotone and hemicontinuous.

• The operator $-C_0(\theta ,\cdot )$ is coercive.

Assumption 5.3

The function $C_0(\theta ,u_0)$ is locally uniformly monotonically decreasing, i.e. there exist ${\gamma }_c>0$ and ${\epsilon }_{u_0}>0$ such that ${⟨C_0(\theta ,u_0^1)-C_0(\theta ,u_0^2),u_0^1-u_0^2⟩}_{V^{\ast },V}\le -{\gamma }_c\parallel u_0^1-u_0^2{\parallel }_V^2$ for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$ and $u_0^1,u_0^2\in B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right):=\{u_0\in V|\parallel u_0-{\varphi }_0({\theta }^{\ast }){\parallel }_V<{\epsilon }_{u_0}\}$.

Assumption 5.4

The sensitivity $S_0(\theta ,u_0)$ is locally Lipschitz continuous with respect to $u_0,$ i.e. there exists $L_{S_0}\ge 0$ such that $\parallel S_0(\theta ,u_0^1)-S_0(\theta ,u_0^2){\parallel }_{V^{n_{\theta }}}\le L_{S_0}\parallel u_0^1-u_0^2{\parallel }_V$ for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$ and $u_0^1,u_0^2\in B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$.

Assumption 5.5

The mapping $\theta \mapsto g(\theta ,{\varphi }_0(\theta \left)\right)$ is uniformly monotonically decreasing on $B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right),$ i.e. there exists ${\gamma }_g>0$ such that $\left(g\right(\theta ,{\varphi }_0\left(\theta \right))-g({\theta }^{\ast },{\varphi }_0\left({\theta }^{\ast }\right)\left){)}^{\mathrm{T}}\right(\theta -{\theta }^{\ast })\le -{\gamma }_g\parallel \theta -{\theta }^{\ast }{\parallel }^2$ for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$.

Assumption 5.6

The descent direction g is locally Lipschitz continuous with respect to $u_0,$ i.e. there exists $L_g\ge 0$ such that $\parallel g(\theta ,u_0^1)-g(\theta ,u_0^2)\parallel \le L_g\parallel u_0^1-u_0^2{\parallel }_V$ for all $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$ and $u_0^1,u_0^2\in B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)\subseteq V$.

Moreover, g is uniformly bounded on $B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)\times B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right),$ i.e there exists $K_g\ge 0$ such that $\parallel g(\theta ,u_0)\parallel \le K_g$ for all $(\theta ,u_0)\in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)\times B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$.

5.1. Elliptic PDE constraints

Using Assumptions 5.1–5.6 and the existence of a weak solution (Assumption 2.1), we can prove the following theorem on stability and convergence for the continuous analogue of the descent method for elliptic PDE constraints.

Theorem 5.1

Let Assumptions 5.1–5.6 be satisfied. Then there exists a ${\lambda }^{\ast }\ge 0$ such that for all $\lambda >{\lambda }^{\ast }$ solutions to (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) are well defined for all r>0 and the local minimizer $({\theta }^{\ast },u_0^{\ast })$ of the optimization problem (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ) is a locally exponentially stable steady state of the system (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ).

Proof.

Define $\tilde{\theta }:=\theta -{\theta }^{\ast }$ and ${\tilde{u}}_0=u_0-{\varphi }_0\left(\theta \right)$, with $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$ and $u_0\in B_{{\epsilon }_{u_0}}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$, where ${\varphi }_0\left({\theta }^{\ast }\right)=u_0^{\ast }$ exists, because of Assumption 5.2. We further define a Lyapunov function $\mathcal{V}\left(r\right)=\frac{1}{2}\parallel \tilde{\theta }\left(r\right){\parallel }^2+\frac{1}{2}\parallel {\tilde{u}}_0\left(r\right){\parallel }_H^2$. To prove Theorem 5.1, we will show that the Lyapunov function decreases exponentially. The derivative along the trajectories is given by $\frac{\mathrm{d}}{\mathrm{d}r}\mathcal{V}\left(r\right)=\frac{\mathrm{d}}{\mathrm{d}r}\left(\frac{1}{2}\parallel \tilde{\theta }\left(r\right){\parallel }^2\right)+\frac{\mathrm{d}}{\mathrm{d}r}\left(\frac{1}{2}\parallel {\tilde{u}}_0\left(r\right){\parallel }_H^2\right).$ First, we bound the first summand from above, using (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ), and Assumptions 5.1, 5.5 and 5.6, $\begin{array}{rl}\frac{\mathrm{d}}{\mathrm{d}r}\frac{1}{2}\parallel \tilde{\theta }{\parallel }^2& ={\left(\frac{\mathrm{d}}{\mathrm{d}r}\tilde{\theta }\right)}^{\mathrm{T}}\tilde{\theta }\\ & =\left(g\right(\theta ,u_0)-g(\theta ,{\varphi }_0\left(\theta \right)){)}^{\mathrm{T}}\tilde{\theta }+(g(\theta ,{\varphi }_0(\theta \left)\right)-g({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right){)}^{\mathrm{T}}\tilde{\theta }\\ & \le \parallel g(\theta ,u_0)-g(\theta ,{\varphi }_0(\theta \left)\right)\parallel \parallel \tilde{\theta }\parallel -{\gamma }_g\parallel \theta -{\theta }^{\ast }{\parallel }^2\\ & \le L_g\parallel {\tilde{u}}_0\parallel \parallel \tilde{\theta }\parallel -{\gamma }_g\parallel \tilde{\theta }{\parallel }^2.\end{array}$ Second, we bound the second summand from above, using (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) and $C_0(\theta ,{\varphi }_0(\theta \left)\right)=0$, as well as the fact that by Assumption 5.3 we have (9(9) $S_0(\theta ,{\varphi }_0(\theta \left)\right)={\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right),$(9) ), $\begin{array}{rl}\frac{\mathrm{d}}{\mathrm{d}r}\frac{1}{2}\parallel {\tilde{u}}_0{\parallel }_H^2& ={\left(\frac{\mathrm{d}{\tilde{u}}_0}{\mathrm{d}r},{\tilde{u}}_0\right)}_{V^{\ast },V}={\left(\frac{\mathrm{d}{u}_0}{\mathrm{d}r}-{\mathrm{\nabla }}_{\theta }{\varphi }_0\left(\theta \right)\frac{\mathrm{d}\theta }{\mathrm{d}r},{\tilde{u}}_0\right)}_{V^{\ast },V}\\ & ={\left(\left(S_0\right(\theta ,u_0)-S_0(\theta ,{\varphi }_0\left(\theta \right)\left)\right)g(\theta ,u_0),{\tilde{u}}_0\right)}_{V^{\ast },V}\\ & +\lambda {\left(C_0(\theta ,u_0)-C_0(\theta ,{\varphi }_0(\theta \left)\right),{\tilde{u}}_0\right)}_{V^{\ast },V}\\ & \le \parallel \left(S_0(\theta ,u_0)-S_0(\theta ,{\varphi }_0(\theta \left)\right)\right){\parallel }_{V^{n_{\theta }}}\parallel g(\theta ,u_0)\parallel \parallel {\tilde{u}}_0{\parallel }_V\\ & +\lambda {\left(C_0(\theta ,u_0)-C_0(\theta ,{\varphi }_0(\theta \left)\right),{\tilde{u}}_0\right)}_{V^{\ast },V}.\end{array}$ With Assumptions 5.3, 5.4 and 5.6, we get $\frac{\mathrm{d}}{\mathrm{d}r}\frac{1}{2}\parallel {\tilde{u}}_0{\parallel }_H^2\le (L_{S_0}{K}_g-\lambda {\gamma }_c)\parallel {\tilde{u}}_0{\parallel }_V^2.$ Hence, we can estimate the derivative of the Lyapunov function, $\frac{\mathrm{d}}{\mathrm{d}r}\mathcal{V}\left(r\right)\le -(-L_{S_0}{K}_g+\lambda {\gamma }_c)\parallel {\tilde{u}}_0{\parallel }_V^2+L_g\parallel {\tilde{u}}_0{\parallel }_V\parallel \tilde{\theta }\parallel -{\gamma }_g\parallel \tilde{\theta }{\parallel }^2.$ To show that $\mathcal{V}$ decays exponentially, we have to show that $\frac{\mathrm{d}}{\mathrm{d}r}\mathcal{V}\left(r\right)\le -a\mathcal{V}\left(r\right)$ for some a>0. Based on our estimates, proving Theorem 5.1 reduces to finding a>0 with (20) $0\le \left(-L_{S_0}{K}_g+\lambda {\gamma }_c-\frac{a}{2}\right)\parallel {\tilde{u}}_0{\parallel }_V^2-L_g\parallel {\tilde{u}}_0{\parallel }_V\parallel \tilde{\theta }\parallel +\left({\gamma }_g-\frac{a}{2}\right)\parallel \tilde{\theta }{\parallel }^2.$(20) We want this inequality to be valid without restrictions on $\parallel \tilde{\theta }\parallel$ or $\parallel {\tilde{u}}_0{\parallel }_V$. Due to the last term, we can therefore only consider values of a that are smaller than $2{\gamma }_g$. Hence, (20(20) $0\le \left(-L_{S_0}{K}_g+\lambda {\gamma }_c-\frac{a}{2}\right)\parallel {\tilde{u}}_0{\parallel }_V^2-L_g\parallel {\tilde{u}}_0{\parallel }_V\parallel \tilde{\theta }\parallel +\left({\gamma }_g-\frac{a}{2}\right)\parallel \tilde{\theta }{\parallel }^2.$(20) ) is equivalent to $0\le {\left(\sqrt{{\gamma }_g-\frac{a}{2}}\parallel \tilde{\theta }\parallel -\frac{L_g}{2\sqrt{{\gamma }_g-\frac{a}{2}}}\parallel {\tilde{u}}_0{\parallel }_V\right)}^2+\left(-L_{S_0}{K}_g+\lambda {\gamma }_c-\frac{a}{2}-\frac{L_g^2}{4({\gamma }_g-\frac{a}{2})}\right)\parallel {\tilde{u}}_0{\parallel }_V^2.$ Since the first term in the inequality is greater or equal to 0, we have to find a>0 such that $\lambda {\gamma }_c-\frac{a}{2}-L_{S_0}{K}_g-\frac{L_g^2}{4({\gamma }_g-\frac{a}{2})}\ge 0.$ Multiplying with $4{\gamma }_g-2a$, we obtain a quadratic inequality for a $a^2+2(-{\gamma }_g-\lambda {\gamma }_c+L_{S_0}{K}_g)a+(4\lambda {\gamma }_c{\gamma }_g-4L_{S_0}{K}_g{\gamma }_g-L_g^2)\ge 0.$ The roots of the quadratic polynomial are given by $a_{1,2}={\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g\pm \sqrt{d},$ with discriminant $d=({\gamma }_g-\lambda {\gamma }_c+L_{S_0}{K}_g{)}^2+L_g^2.$ The discriminant is always positive, therefore, $a_1={\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g-\sqrt{d}<a_2={\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g+\sqrt{d}$ are real roots. In the following, we will assume that $a_1>0$, which can be achieved by choosing λ such that $\lambda >{\lambda }^{\ast }=L_{S_0}{K}_g/{\gamma }_c+L_g^2/\left(4{\gamma }_g{\gamma }_c\right)\ge 0$. This choice is justified as follows. As the square root, $\sqrt{d}$, is always positive, ${\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g>0$, i.e. $\lambda >(L_{S_0}{K}_g-{\gamma }_g)/{\gamma }_c$ needs to hold to ensure $a_1>0$. Squaring both sides of the inequality (21) ${\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g>\sqrt{d}$(21) yields $\begin{array}{rl}({\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g{)}^2& >({\gamma }_g-\lambda {\gamma }_c+L_{S_0}{K}_g{)}^2+L_g^2\\ \iff \lambda & >\frac{L_g^2}{4{\gamma }_g{\gamma }_c}+\frac{L_{S_0}{K}_g}{{\gamma }_c}.\end{array}$ Taking $\lambda >{\lambda }^{\ast }:=max\left\{\frac{L_{S_0}{K}_g-{\gamma }_g}{{\gamma }_c},\frac{L_g^2}{4{\gamma }_g{\gamma }_c}+\frac{L_{S_0}{K}_g}{{\gamma }_c}\right\}=\frac{L_g^2}{4{\gamma }_g{\gamma }_c}+\frac{L_{S_0}{K}_g}{{\gamma }_c}$ ensures $a_1>0$.

Therefore, a either fulfils $0<a<a_1$ with $a<2{\gamma }_g$ or $a_2<a<2{\gamma }_g$, provided $\lambda >{\lambda }^{\ast }$. Hence, we distinguish the following three cases:

1. $2{\gamma }_g<a_1<a_2$,

2. $a_1<a_2\le 2{\gamma }_g$,

3. $a_1\le 2{\gamma }_g\le a_2$,

for the relation of $2{\gamma }_g$, $a_1$ and $a_2$ as illustrated in Figure 2.

Figure 2. The function $f\left(a\right)=a^2+(-2{\gamma }_g-2\lambda {\gamma }_c+2L_{S_0}{K}_g)a+(4\lambda {\gamma }_c{\gamma }_g-4L_{S_0}{K}_g{\gamma }_g-L_g^2)$ is illustrated with the two roots $a_1$ and $a_2$ and the three different positions of $2{\gamma }_g$, as well as possible positions of a.

Case (1): $2{\gamma }_g<a_1$ is equivalent to $\lambda {\gamma }_c-L_{S_0}{K}_g-{\gamma }_g>\sqrt{d}.$ If the term $\lambda {\gamma }_c-L_{S_0}{K}_g-{\gamma }_g$ is negative, the inequality cannot be valid. The term $\lambda {\gamma }_c-L_{S_0}{K}_g-{\gamma }_g$ is non-negative if $\lambda \ge ({\gamma }_g+L_{S_0}{K}_g)/{\gamma }_c$. In this case, we can square the inequality and get a contradiction ($0>L_g^2$).

Case (2): $a_2\le 2{\gamma }_g$ is equivalent to $\sqrt{d}\le {\gamma }_g+L_{S_0}{K}_g-\lambda {\gamma }_c.$ This leads to a contradiction with the same arguments as in case (1).

Case (3): $a_1\le 2{\gamma }_g$ is equivalent to $-{\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g\le \sqrt{d}.$ The left-hand side $-{\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g$ is non-negative for all $\lambda \ge ({\gamma }_g+L_{S_0}{K}_g)/{\gamma }_c$. With squaring, we get $0\le L_g^2$. On the other hand, if the term $-{\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g$ is negative, that is $\lambda <({\gamma }_g+L_{S_0}{K}_g)/{\gamma }_c$, we have $\sqrt{d}>0$. This is true for all λ, because d>0. In total, we find $a_1\le 2{\gamma }_g$ for all $\lambda >0$. Analogously we get for $a_2$ that $a_2\ge 2{\gamma }_g$ is fulfilled for all $\lambda >0$. Hence, we know that $a_1\le 2{\gamma }_g\le a_2$ holds for all $\lambda >0$ and only case (3) is valid.

Altogether, we find that a lies in the interval $[0,a_1]$ provided $\lambda >{\lambda }^{\ast }$. In this case, it also holds that $\frac{\mathrm{d}}{\mathrm{d}r}\mathcal{V}\left(r\right)\le -\frac{a}{2}\left(\parallel {\tilde{u}}_0\left(r\right){\parallel }_V^2+\parallel \tilde{\theta }\left(r\right){\parallel }^2\right)\le -\frac{a}{2K_{V\to H}^2}\parallel {\tilde{u}}_0\left(r\right){\parallel }_H^2-\frac{a}{2}\parallel \tilde{\theta }\left(r\right){\parallel }^2\le -\tilde{a}\mathcal{V}\left(r\right),$ with $\tilde{a}=-\left(a/2\right)min\{1/K_{V\to H}^2,1\}$, where $K_{V\to H}$ is the embedding constant.

Remark 5.1

To tune the choice of the retraction factor λ, we now consider the fact that the value of a determines the speed at which $\mathcal{V}\left(r\right)$ decreases, thus a convenient choice of the retraction factor $\lambda >{\lambda }^{\ast }$ maximizes a to yield the fastest exponential decay. In our case, this means maximizing $a\left(\lambda \right)=a_1={\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g-\sqrt{d\left(\lambda \right)}$ with respect to λ. An elementary computation yields $\frac{\mathrm{d}a}{\mathrm{d}\lambda }\left(\lambda \right)={\gamma }_c+\frac{{\gamma }_g{\gamma }_c-\lambda {\gamma }_c^2+L_{S_0}{K}_g{\gamma }_c}{\sqrt{d\left(\lambda \right)}}\ge 0$ with equality iff $L_g=0$, thus $\lambda \mapsto a\left(\lambda \right)$ is monotonically increasing (strictly, if $L_g>0$) and therefore $\begin{array}{rl}\underset{\lambda \in ({\lambda }^{\ast },\mathrm{\infty })}{sup}a\left(\lambda \right)& =\underset{\lambda \to \mathrm{\infty }}{lim}a\left(\lambda \right)=\underset{\lambda \to \mathrm{\infty }}{lim}{\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g-\sqrt{({\gamma }_g-\lambda {\gamma }_c+L_{S_0}{K}_g{)}^2+L_g^2}\\ & =2{\gamma }_g.\end{array}$ In case $L_g=0$, we have $a\left(\lambda \right)={\gamma }_g+\lambda {\gamma }_c-L_{S_0}{K}_g-|{\gamma }_g-\lambda {\gamma }_c+L_{S_0}{K}_g|$. Distinguishing the two cases for the absolute value yields the maximal value $a\left(\lambda \right)=2{\gamma }_g$, attained at all $\lambda \ge ({\gamma }_g-L_{S_0}{K}_g)/{\gamma }_c$.

This shows that (unless $L_g=0$) the exponential decay is maximized by choosing $\lambda >{\lambda }^{\ast }$ as large as possible. Nevertheless, in practice, λ should not be chosen too large in order to avoid stiffness of system (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ).

Remark 5.2

The proof provides a lower bound for the retraction factor λ, namely $\lambda >{\lambda }^{\ast }=L_g^2/\left(4{\gamma }_g{\gamma }_c\right)+L_{S_0}{K}_g/{\gamma }_c$. In specific applications, it might not always be possible to explicitly compute all involved constants. If this is the case, an alternative Lyapunov function can be used to derive a lower bound for λ. A possible candidate for this Lyapunov function is (22) $\mathcal{V}\left(r\right)=j\left(\theta \right(r\left)\right)-j\left({\theta }^{\ast }\right)+\frac{1}{2}\parallel u_0-{\varphi }_0\left(\theta \right(r\left)\right){\parallel }_H^2.$(22) With this choice and analogous computations as above, different lower bounds involving different constants can be derived. The lower bound for the retraction factor can be estimated as $\lambda >{\hat{\lambda }}^{\ast }=\hat{L}/{\gamma }_c+{\hat{L}}_g^2/\left(4{\gamma }_c\right)$ with (23) $\hat{L}=\left\{\begin{array}{ll}\frac{\left(\right(S_0(\theta ,u_0)-S_0(\theta ,{\varphi }_0(\theta \left)\right)\left)g\right(\theta ,u_0),u_0-{\varphi }_0(\theta ){)}_{V^{\ast },V}}{\parallel u_0-{\varphi }_0\left(\theta \right){\parallel }_V^2},& \mathrm{if}{u}_0\ne {\varphi }_0\left(\theta \right),\\ 0,& \mathrm{else},\end{array}\right.$(23) and (24) ${\hat{L}}_g=\left\{\begin{array}{ll}\frac{g(\theta ,{\varphi }_0(\theta \left){)}^{\mathrm{T}}\right(g(\theta ,{\varphi }_0(\theta \left)\right)-g(\theta ,u_0))}{\parallel u_0-{\varphi }_0\left(\theta \right){\parallel }_V\parallel g(\theta ,{\varphi }_0(\theta \left)\right)\parallel },& \mathrm{if}{u}_0\ne {\varphi }_0\left(\theta \right)\text{ and }g(\theta ,{\varphi }_0(\theta \left)\right)\ne 0,\\ 0,& \mathrm{else}.\end{array}\right.$(24) This bound depends on θ, the current parameter estimates during computation, and therefore requires a posteriori adaptation of the retraction factor. A practical implementation of such a retraction factor choice involves the evaluation of functionals of the residuals $g(\theta ,{\varphi }_0(\theta \left)\right)-g(\theta ,u_0)$, $u_0-{\varphi }_0\left(\theta \right)$ as well as sensitivities $\left(S_0\right(\theta ,u_0)-S_0(\theta ,{\varphi }_0\left(\theta \right)\left)\right)g(\theta ,u_0)$ (which can be done approximately, on a coarser computational mesh, and using adjoint techniques) is subject of future work.

Remark 5.3

If the cost function is locally convex but not locally uniformly convex, the parameters are identifiable but not strictly identifiable [19]. In this case, $g(\theta ,{\varphi }_0)$ defined by (14(14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) ) is not uniformly monotone, but just monotone, i.e. formally ${\gamma }_g=0$. In this case, using a projection P on the orthogonal complement of the null space of ${\mathrm{\nabla }}^{2}j\left({\theta }^{\ast }\right)$ might facilitate the proof of convergence on this subspace. A possible Lyapunov function in this case is given by $\mathcal{V}\left(r\right)=\frac{1}{2}\parallel Pg(\theta ,{\varphi }_0(\theta \left)\right){\parallel }^2+\frac{1}{2}\parallel u_0-{\varphi }_0\left(\theta \right){\parallel }_X^2$. Denoting the smallest positive eigenvalue of $\left(\mathrm{d}g/\mathrm{d}\theta \right)\left(\theta {\varphi }_0\right(\theta \left)\right)$ with $\underset{\_}{\mu }$, we require that ${\xi }^{\mathrm{T}}P\left(\mathrm{d}g/\mathrm{d}\theta \right)(\theta ,{\varphi }_0(\theta \left)\right)\xi \le -\underset{\_}{\mu }\parallel P\xi {\parallel }^2$ for all $\xi \in {\mathbb{R}}^{n_{\theta }}$. Then a retraction factor λ should be chosen $\lambda >{\lambda }^{\ast }=\hat{L}/{\gamma }_c+{\hat{L}}_g^2/\left(4{\gamma }_c\underset{\_}{\mu }\right)$, with (25) $\hat{L}=\left\{\begin{array}{ll}\frac{\left(\right(S_0(\theta ,u_0)-S_0(\theta ,{\varphi }_0(\theta \left)\right)\left)g\right(\theta ,u_0),u_0-{\varphi }_0(\theta ){)}_{V^{\ast },V}}{\parallel u_0-{\varphi }_0\left(\theta \right){\parallel }_V^2},& \mathrm{if}{u}_0\ne {\varphi }_0\left(\theta \right),\\ 0,& \mathrm{else},\end{array}\right.$(25) and (26) ${\hat{L}}_g=\left\{\begin{array}{ll}\frac{g(\theta ,{\varphi }_0(\theta \left){)}^{\mathrm{T}}\right(g(\theta ,{\varphi }_0(\theta \left)\right)-g(\theta ,u_0))}{\parallel u_0-{\varphi }_0\left(\theta \right){\parallel }_V\parallel g(\theta ,{\varphi }_0(\theta \left)\right)\parallel },& \mathrm{if}{u}_0\ne {\varphi }_0\left(\theta \right)\text{ and }g(\theta ,{\varphi }_0(\theta \left)\right)\ne 0,\\ 0,& \mathrm{else}.\end{array}\right.$(26) However, the null space of ${\mathrm{\nabla }}^{2}j\left({\theta }^{\ast }\right)$ depends on the unknown optimal parameter ${\theta }^{\ast }$ and can in general not be assessed a priori, thus leaving this approach for further investigation.

Remark 5.4

Locality in Theorem 5.1 is only imposed in terms of the size of ${\epsilon }_{\theta },{\epsilon }_{u_0}$ in Assumptions 5.2–5.6. As a matter of fact, Assumptions 5.2, 5.3, 5.4, 5.6 will typically hold for $(\theta ,u_0)$ values in a larger neighbourhood of the solution, possibly after imposing certain constraints on the parameter values such as nonnegativity, in order to ensure that the model defined by $C_0$ is valid. Smallness of ${\epsilon }_{\theta },{\epsilon }_{u_0}$ will mainly be required in Assumption 5.5 to guarantee, together with Assumption 5.1, that the descent direction defined by g really points towards $({\theta }_{\ast },{\varphi }_0({\theta }_{\ast }\left)\right)$.

To enlarge the convergence radius, globalization strategies for the solution of nonlinear equations or optimization problems can be employed. In particular, as the analysis in [20,21] indicates, the strategy of using a particular time-stepping algorithm together with an appropriate step size control in place of the continuous flow defined by (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ), might enable to get rid of the locality assumption in Theorem 5.1. Indeed, with $F(\theta ,u_0)=\left(\begin{array}{c}-d(\theta ,u_0)\\ -\lambda C_0(\theta ,u_0)\end{array}\right),M(\theta ,u_0)={\left(\begin{array}{cc}H(\theta ,u_0)& 0\\ -S_0(\theta ,u_0)H(\theta ,u_0)& I\end{array}\right)}^{-1},$ for the choice (15(15) $g(\theta ,u_0)=H(\theta ,u_0{)}^{-1}d(\theta ,u_0)$(15) ), method (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) can at least formally be cast into the framework of [20, Equation (5.1)], see also the introduction of [21] for the infinite dimensional setting relevant here. However, the detailed conditions from [21] would still have to be verified for our special setting, which will be subject of future research.

5.2. Elliptic and parabolic PDE constraints

As we consider the partially reduced form of the optimization problem with elliptic and parabolic PDE constraints (18(18) $\begin{array}{rl}\underset{\theta ,u_0}{min}& \stackrel{\mathcal{~}}{\mathcal{J}}(\theta ,u_0):=\tilde{J}(\theta ,u_0,\varphi (\theta ,u_0\left)\right)\\ \text{s.t.}& 0=C_0(\theta ,u_0),\end{array}$(18) ), the results established for the elliptic problem can be easily transferred given the existence of a solution operator for the parabolic problem.

Theorem 5.2

Let Assumptions 2.1, 5.1–5.6 be satisfied with g replaced by $\tilde{g}$ according to (19(19) ${\tilde{g}}_i(\theta ,u_0)=-\frac{\mathrm{\partial }\tilde{\mathcal{J}}}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-\frac{\mathrm{\partial }\tilde{\mathcal{J}}}{\mathrm{\partial }{u}_0}(\theta ,u_0){S_0}_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\}.$(19) ). Then there exists a ${\lambda }^{\ast }>0$ such that for all $\lambda >{\lambda }^{\ast }$ solutions to (27) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =\tilde{g}(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)\tilde{g}(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(r\right)=u_{0,0},\end{array}$(27) are well defined for all r>0 and the local minimizer $({\theta }^{\ast },u_0^{\ast })$ of the optimization problem (4(4) $\begin{array}{rl}\underset{\theta ,u_0,u}{min}& \tilde{J}(\theta ,u_0,u)\\ \text{s.t.}& u_t=C(\theta ,t,u)u\left(0\right)=u_0\\ & 0=C_0(\theta ,u_0).\end{array}$(4) ) is a locally exponentially stable steady state of the system (27(27) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =\tilde{g}(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)\tilde{g}(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(r\right)=u_{0,0},\end{array}$(27) ).

With the setting introduced in the last paragraph of Section  4, the result directly follows from Theorem 5.1.

6. Discussion of the assumptions

In this section, we discuss Assumptions 5.1–5.6 in more detail. In particular, we provide sufficient conditions for these assumptions which show that the assumptions are rather weak and fulfilled by many application problems.

Remark 6.1

Assumption 5.1

For the choice (14(14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) ) of g, this condition is satisfied due to the identity $0={\mathrm{\nabla }}_{\theta }j\left({\theta }^{\ast }\right)=-{\mathrm{\nabla }}_{\theta }J(\theta ,\varphi ({\theta }^{\ast }\left)\right)-\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,\varphi ({\theta }^{\ast }\left)\right)S_0(\theta ,\varphi ({\theta }^{\ast }\left)\right).$ This remains valid more generally for descent directions defined by (15(15) $g(\theta ,u_0)=H(\theta ,u_0{)}^{-1}d(\theta ,u_0)$(15) ), such as Newton-type methods.

Remark 6.2

Assumptions 5.2 and 5.3

Dynamical systems in engineering, physics and life sciences will typically exhibit locally exponentially stable steady-state solutions. This implies that these systems are locally uniformly monotone. This is especially true for the optimal parameters ${\theta }^{\ast }$ and thus (for smooth $C_0(\theta ,u_0)$) also in a neighbourhood of $({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)$. Hence, around the true parameters, Assumption 5.3 and thus also the monotonicity part of Assumption 5.2 is locally fulfilled for most real-world systems.

An additional consequence of uniform monotonicity, hemicontinuity and coercivity of $C_0$ according to Assumption 5.2 is uniform monotonicity, hemicontinuity and coercivity of its linearization and thus, by the Theorem of Browder and Minty, existence and boundedness of the inverse $\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right({\theta }^{\ast },u_0^{\ast }){)}^{-1}:V^{\ast }\to V$, is assured, as will be used below.

Remark 6.3

Assumption 5.4

The Lipschitz continuity of S is satisfied for example if

• $\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0:{\mathbb{R}}^{n_{\theta }}\times V\to \mathcal{L}(V,V^{\ast })$ is continuous,

• ${\mathrm{\nabla }}_{\theta }{C}_0$ and $\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0$ are locally Lipschitz continuous with respect to $u_0$ in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$, with Lipschitz constants $L_{C_0,\theta }$ and $L_{C_0,u_0}$, respectively,

• ${\mathrm{\nabla }}_{\theta }{C}_0$ is uniformly bounded on this neighbourhood by $K_{C_0,\theta }$,

• $\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)({\theta }^{\ast },u_0^{\ast }{)}^{-1}$ is bounded by $K_{C_0,u_0}$, as discussed in Remark 6.2,

provided that $C_0$ is continuously (Frèchet) differentiable. This can be seen as follows.

We want to show that $\parallel S_{0_i}(\theta ,u_0^1)-S_{0_i}(\theta ,u_0^2){\parallel }_V\le L_{S_i}\parallel u_0^1-u_0^2{\parallel }_V$ for all $i\in \{1,\dots ,n_{\theta }\}$, $\theta \in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)$ and $u_0^1,u_0^2\in B_{{\epsilon }_u}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$, where $S_{0_i}$ is the ith component of $S_0$.

Using $S_{0_i}(\theta ,u_0)=\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right(\theta ,u_0\left){)}^{-1}\right(\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{\theta }_i\right)(\theta ,u_0))$ for $i\in \{1,\dots ,n_{\theta }\}$ yields $\begin{array}{rl}& {∥S_{0_i}(\theta ,u_0^1)-S_{0_i}(\theta ,u_0^2)∥}_V\\ & \le {∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)\right)}^{-1}∥}_{V^{\ast }\to V}{∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0^1)-\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0^2)∥}_{V^{\ast }}\\ & +{∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)\right)}^{-1}-{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^2)\right)}^{-1}∥}_{V^{\ast }\to V}{∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0^2)∥}_{V^{\ast }}.\end{array}$ With $A^{-1}-B^{-1}=A^{-1}(B-A)B^{-1}$, we can further estimate $\begin{array}{rl}& {∥S_{0_i}(\theta ,u_0^1)-S_{0_i}(\theta ,u_0^2)∥}_V\\ & \le {∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)\right)}^{-1}∥}_{V^{\ast }\to V}\left({∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0^1)-\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(q,u_0^2)∥}_{V^{\ast }}\right.\\ & +{∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^2)-\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)∥}_{V\to V^{\ast }}{∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)\right)}^{-1}∥}_{V^{\ast }\to V}\left.{∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{\theta }_i}(\theta ,u_0^2)∥}_{V^{\ast }}\right).\end{array}$ The fact that the inverses $\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right(\theta ,u_0^1){)}^{-1}$, $\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right(\theta ,u_0^2){)}^{-1}$ exist and are bounded follows from the regularity of $\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)$ and a perturbation argument. Taking $L=\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)$ and $M_i=\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)(\theta ,u_0^i))$, i=1,2 and using that by continuity of $\left(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\right)(\theta ,u_0)$ in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$ and by possibly decreasing ${\epsilon }_{\theta }$, ${\epsilon }_{u_0}$, we get $\parallel M_i-L\parallel ={∥\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^i)-\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}({\theta }^{\ast },{\varphi }_0({\theta }^{\ast }\left)\right)∥}_{V\to V^{\ast }}<\frac{1}{\parallel L^{-1}{\parallel }_V}$ for any $(\theta ,u_0^i)\in B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)\times B_{{\epsilon }_u}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$. Therewith the operator $M_i^{-1}=\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right(\theta ,u_0){)}^{-1}$ exists and is bounded by ${∥M_i^{-1}∥}_V={∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_i)\right)}^{-1}∥}_V\le \frac{\parallel L^{-1}\parallel }{1-\parallel L^{-1}\parallel \parallel L-M_i\parallel }\le K_{C_0,u_0}.$ Using this bound and local Lipschitz continuity with respect to $u_0$ of the derivatives $\mathrm{\partial }{C}_0/\mathrm{\partial }{\theta }_i$ and $\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0$ results in ${∥S_{0_i}(\theta ,u_0^1)-S_{0_i}(\theta ,u_0^2)∥}_V\le K_{C_0,u_0}\left(L_{C_0,{\theta }_i}+K_{C_0,u_0}{K}_{C_0,{\theta }_i}{L}_{C_0,u_0}\right){∥u_0^1-u_0^2∥}_V$ and hence ${∥S_0(\theta ,u_0^1)-S_0(\theta ,u_0^2)∥}_V\le L_{S_0}{∥u_0^1-u_0^2∥}_V$ with $L_{S_0}=\sum _{i=1}^{n_{\theta }}\left\{K_{C_0,u_0}\right(L_{C_0,{\theta }_i}+K_{C_0,u_0}{K}_{C_0,{\theta }_i}{L}_{C_0,u_0}\left)\right\}$, where $K_{C_0,{\theta }_i}$ denotes the bound of the derivative w.r.t. ${\theta }_i$ of the steady-state residual.

Remark 6.4

Assumption 5.5

If the descent direction g is defined by (14(14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) ), then, according to [19], Assumption 5.5 is equivalent to practical identifiability, which implies local structural identifiability of the parameter vector θ. The same holds true for more general choices (15(15) $g(\theta ,u_0)=H(\theta ,u_0{)}^{-1}d(\theta ,u_0)$(15) ), such as Newton-type methods, under a uniform positivity condition on $H(\theta ,u_0)$.

If local structural identifiability cannot be guaranteed, one can still use regularization [22], e.g. by adding a term $\alpha (\theta -{\theta }_p{)}^{\mathrm{T}}{\mathrm{\Gamma }}^{-1}(\theta -{\theta }_p)$ with positive definite Γ and positive α to the cost function $\mathcal{J}(\theta ,u_0)$ in (5(5) $\begin{array}{rl}\underset{\theta ,u_0}{min}& J(\theta ,u_0)\\ \text{s.t.}& 0=C_0(\theta ,u_0).\end{array}$(5) ) for which (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) yields a minimizer $\left({\theta }^{\ast }\right(\alpha ),u_0^{\ast }(\alpha \left)\right)$. Regularization theory, e.g. [22], provides convergence of ${\theta }^{\ast }\left(\alpha \right)$ to a parameter $\overline{\theta }$ that is consistent with the observations as $\alpha \to 0$.

Remark 6.5

Assumption 5.6

As an example, we check Assumption 5.6 for the case of a gradient descent-based update of θ, i.e. g according to (14(14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) ). We find $\begin{array}{rl}& ∥g(\theta ,u_0^1)-g(\theta ,u_0^2)∥\\ & =∥-{\mathrm{\nabla }}_{\theta }J(\theta ,u_0^1)-\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)S_0(\theta ,u_0^1)+{\mathrm{\nabla }}_{\theta }J(\theta ,u_0^2)+\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0^2)S_0(\theta ,u_0^2)∥\\ & \le ∥{\mathrm{\nabla }}_{\theta }J(\theta ,u_0^1)-{\mathrm{\nabla }}_{\theta }J(\theta ,u_0^2)∥+{∥\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)-\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0^2)∥}_{V\to \mathbb{R}}{∥S_0(\theta ,u_0^1)∥}_{V^{n_{\theta }}}\\ & +{∥\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0^2)∥}_{V\to \mathbb{R}}{∥S_0(\theta ,u_0^1)-S_0(\theta ,u_0^2)∥}_{V^{n_{\theta }}}.\end{array}$ Here, $S_0$ is Lipschitz continuous according to Remark 6.3. Also boundedness of $S_0$ is fulfilled under the assumptions made in Remark 6.3, namely that $\left(\right(\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0\left)\right(\theta ,u_0){)}^{-1}$ and ${\mathrm{\nabla }}_{\theta }{C}_0$ are locally uniformly bounded. (28) $\begin{array}{rl}{∥S_0(\theta ,u_0^1)∥}_{V^{n_{\theta }}}& \le {∥{\left(\frac{\mathrm{\partial }{C}_0}{\mathrm{\partial }{u}_0}(\theta ,u_0^1)\right)}^{-1}∥}_{V^{\ast }\to V}{∥{\mathrm{\nabla }}_{\theta }{C}_0(\theta ,u_0^1)∥}_{V^{{\ast }^{n_{\theta }}}}\\ & \le K_{C_0,u_0}{K}_{C_0,\theta }=:K_{S_0}.\end{array}$(28) If J is differentiable and the derivatives ${\mathrm{\nabla }}_{\theta }J$ and $\mathrm{\partial }J/\mathrm{\partial }{u}_0$ are locally Lipschitz continuous with respect to $u_0$ with Lipschitz constants $L_{J,\theta ;u_0}$ and $L_{J,u_0;u_0}$, respectively, and $\mathrm{\partial }j/\mathrm{\partial }{u}_0$ is uniformly bounded on $B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)\times B_{{\epsilon }_u}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$, we can conclude $∥g(\theta ,u_0^1)-g(\theta ,u_0^2)∥\le L_g\parallel u_0^1-u_0^2{\parallel }_V,$ with $L_g=L_{J,\theta ;u_0}+K_{J,u_0}{L}_{S_0}+L_{J,u_0;u_0}{K}_{S_0}$.

The more general choice (15(15) $g(\theta ,u_0)=H(\theta ,u_0{)}^{-1}d(\theta ,u_0)$(15) ) requires additionally Lipschitz continuity of $H(\theta ,u_0)$, which in case of Newton-type methods with (16(16) $\begin{array}{rl}{H}_{ij}(\theta ,u_0)& =\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{\theta }_i\mathrm{\partial }{\theta }_j}(\theta ,u_0)+{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_j}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}\\ & +{⟨\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0\mathrm{\partial }{\theta }_i}(\theta ,u_0),{S_0}_j(\theta ,u_0)⟩}_{V^{\ast },V}+\frac{{\mathrm{\partial }}^{2}J}{\mathrm{\partial }{u}_0^2}(\theta ,u_0)\left({S_0}_i(\theta ,u_0),{S_0}_j(\theta ,u_0)\right)\\ & +{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{T_0}_{i,j}(\theta ,u_0)⟩}_{V^{\ast },V},i,j\in \{1,\dots ,n_{\theta }\},\end{array}$(16) ) amounts to higher smoothness of J and $C_0$.

Thus we have shown the following proposition.

Proposition 6.1

Let Assumption 6.1 be satisfied. Then there exist ${\epsilon }_{\theta },{\epsilon }_{u_0}>0$ such that on $B_{{\epsilon }_{\theta }}\left({\theta }^{\ast }\right)\times B_{{\epsilon }_u}\left({\varphi }_0\right({\theta }^{\ast }\left)\right)$, Assumptions 5.1–5.6 are fulfilled.

Assumption 6.1

1. 1. $C_0(\theta ,\cdot )$ is locally uniformly monotonically decreasing.

   2. $C_0$ is continuous and continuously $($Fréchet$)$ differentiable in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$.

   3. ${\mathrm{\nabla }}_{\theta }{C}_0$ and $\mathrm{\partial }{C}_0/\mathrm{\partial }{u}_0$ are locally Lipschitz continuous w.r.t. $u_0$ and ${\mathrm{\nabla }}_{\theta }{C}_0$ is bounded in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$.

2. $g(\theta ,u_0)$ is defined by (14(14) $g_i(\theta ,u_0):=-\frac{\mathrm{\partial }J}{\mathrm{\partial }{\theta }_i}(\theta ,u_0)-{⟨\frac{\mathrm{\partial }J}{\mathrm{\partial }{u}_0}(\theta ,u_0),{S_0}_i(\theta ,u_0)⟩}_{V^{\ast },V}=:d_i(\theta ,u_0),i\in \{1,\dots ,n_{\theta }\},$(14) ) and θ is locally structurally identifiable.

3. 1. J is continuously $($Fréchet$)$ differentiable in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$.

   2. $\mathrm{\partial }J/\mathrm{\partial }{u}_0$ and ${\mathrm{\nabla }}_{\theta }J$ are locally Lipschitz continuous with respect to $u_0$.

   3. $\mathrm{\partial }J/\mathrm{\partial }{u}_0$ is bounded in a neighbourhood of $({\theta }^{\ast },u_0^{\ast })$.

7. Application

To illustrate the continuous analogue of the descent method, we use it to study CCL21 gradient formation in biological tissues. This process is highly relevant in immune response [23,24] and described by a reaction–diffusion equation [25]. In the following, we outline the model, estimate its parameters using the approach proposed in this paper and analyse the results.

7.1. Model formulation

CCL21 gradients are necessary for the guidance of dendritic cells towards lymphatic vessels [26]. They are formed by the combination of several biological processes. The chemokine CCL21 is produced in the lymphatic vessels, which cover a subset domain ${\mathrm{\Omega }}_L$ of the domain Ω of interest, ${\mathrm{\Omega }}_L\subset \mathrm{\Omega }$. The source term is defined via the function $Q\left(x\right)=\left\{\begin{array}{ll}1,& \mathrm{for}x\in {\mathrm{\Omega }}_L,\\ 0,& \mathrm{otherwise}.\end{array}\right.$ The concentration of free CCL21 is denoted by u. Free CCL21 binds to a sugar whose concentration is denoted by s. The binding yields immobilized CCL21 whose concentration is denoted by c. The parameters $k_1,k_{-1},D,\gamma ,\alpha$ denote the binding and unbinding rates, the diffusion coefficient, the degradation rate and the production rate of CCL21 from the lymphatic vessels, respectively. A PDE model for the process has been developed in [25] and is given by (29) $\begin{array}{rl}& u_t-D\mathrm{\Delta }u=\alpha Q-k_{1}us+k_{-1}c-\gamma u,\\ & \dot{s}=-k_{1}us+k_{-1}c,\\ & \dot{c}=k_{1}us-k_{-1}c,\end{array}$(29) for $t\in ]0,T[$ and $x\in \mathrm{\Omega }$, with initial conditions $u(0,x)=c(0,x)=0$, $s(0,x)=s_0$ and no-flux boundary conditions $\left(\mathrm{\partial }/\mathrm{\partial }\nu \right)u=0$ where ν is the outer normal on Ω. The parameter $s_0$ denotes the initial sugar concentration.

As the formation of the gradient is apparently fast, we consider the steady state of (29(29) $\begin{array}{rl}& u_t-D\mathrm{\Delta }u=\alpha Q-k_{1}us+k_{-1}c-\gamma u,\\ & \dot{s}=-k_{1}us+k_{-1}c,\\ & \dot{c}=k_{1}us-k_{-1}c,\end{array}$(29) ). By considering the PDE for the time evolution of s and c, we find that $c=s_0{u}_s/(1+u_s)$ and $s=s_0/(u_s+1)$ with the scaled CCL21 concentration $u_s:=k_{1}u/k_{-1}$. Using the additional reformulation $\tilde{D}=D/\gamma$, $\tilde{\alpha }=\alpha k_1/\left(\gamma k_{-1}\right)$, the scaled steady-state concentration of CCL21, $u_s$, has to fulfil $0=\tilde{D}\mathrm{\Delta }\tilde{\mathrm{\Delta }}{u}_s+\tilde{\alpha }Q-u_s$ and the boundary conditions $\left(\mathrm{\partial }/\mathrm{\partial }\nu \right)u_s=0$.

For the considered process, imaging data have been collected [27]. These images provide information about the localization of the lymphatic vessels (encoded in Q) and the concentration of immobilized CCL21. As the measured intensity values are corrupted by background fluorescence and as the data are not normalized, we model the readout following [25] as $y_i=s_l\left(b+{\int }_{A_i}c(t,x)\mathrm{d}x\right),$ where b denotes the intensity of the background fluorescence, $s_l$ is a scaling constant and $A_i\subset \mathrm{\Omega }$ is the domain of the pixel k. As the parameters are structurally non-identifiable, we reformulate the models in terms of $\tilde{b}=s_{l}b$ and ${\tilde{s}}_0=s_l{s}_0$ in the parameter estimation to one parameter and just consider $\tilde{b}$ and ${\tilde{s}}_0$.

The optimization problem is then given by (30) $\begin{array}{rl}\underset{\theta ,u_s}{min}& J(\theta ,u_s)=\frac{1}{2}\left\{\sum _{i=1}^M\log \left(2\pi {\sigma }_i^2{\overline{y}}_i\right)+{\left(\frac{\log \left({\overline{y}}_i\right)-\log \left(y_i\right)}{{\sigma }_i}\right)}^2\right\}\\ \mathrm{s}.\mathrm{t}.& -\tilde{D}\mathrm{\Delta }{u}_s+u_s=\tilde{\alpha }Q,x\in \mathrm{\Omega }\\ & \frac{\mathrm{\partial }}{\mathrm{\partial }\nu }{u}_s=0,x\in \mathrm{\partial \Omega }\\ & y_i=\tilde{b}+{\int }_{A_i}\frac{{\tilde{s}}_0{u}_s\left(x\right)}{u_s\left(x\right)+1}\mathrm{d}x=\tilde{b}+{\tilde{s}}_0{h}_i\left(u_s\right),\end{array}$(30) where $\mathrm{\Omega }\subseteq {\mathbb{R}}^2$, ${\sigma }_i$ is the scale parameter of the log-normally distributed measurement error and $h_i\left(u_s\right)={\int }_{A_i}\left(u_s\left(x\right)/\left(u_s\right(x)+1)\right)\mathrm{d}x$, $i=1,\dots ,M$. The parameter vector θ is given by $\theta =(\tilde{D},\tilde{\alpha },{\tilde{s}}_0,\tilde{b},\sigma )\in {\mathbb{R}}^{n_{\theta }}$, with $n_{\theta }=5$.

All parameters are assumed to be non-negative due to their biological meaning. The spaces V and $V^{\ast }$ for which we examine the problem are $V=H^1\left(\mathrm{\Omega }\right)\text{and}{V}^{\ast }=H^1(\mathrm{\Omega }{)}^{\ast }$. The operator $C_0$ is given by $C_0(\theta ,u_s)=\tilde{D}\mathrm{\Delta }{u}_s-u_s+\tilde{\alpha }Q$. For these spaces and operators, it can easily be checked that all assumptions for applying the method (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ) are satisfied.

7.2. Numerical implementation

For the numerical simulation of the biological process, we employed a finite element discretization of the PDE model. The discretization was obtained using the MATLAB PDE toolbox and accounts for the stationary topology of the model (Figure 3A). The mesh consists of 2170 elements and the concentrations in these elements are the state variables of the discretization. For parameter optimization using the coupled ODE–PDE model (17(17) $\begin{array}{rl}\frac{\mathrm{d}\theta }{\mathrm{d}r}\left(r\right)& =g(\theta ,u_0),\theta \left(0\right)={\theta }_0,\\ \frac{\mathrm{d}{u}_0}{\mathrm{d}r}\left(r\right)& =S_0(\theta ,u_0)g(\theta ,u_0)+\lambda C_0(\theta ,u_0),u_0\left(0\right)=u_{0,0}.\end{array}$(17) ), the same mesh was employed and the states of the discretized PDE were coupled with the ODE for the parameters. This yields a model with 2170+5 equations. The simulation-based method for parameter estimation was implemented in MATLAB extending the routine published in [16]. The numerical simulation was performed using the MATLAB ODE solver ode15s, an implicit scheme applicable to stiff problems. To accelerate the calculations, we implemented the Jacobian of the coupled ODE–PDE model. The simulation of the continuous analogue was terminated, if the gradient of the right-hand side became small, i.e. $\parallel C_0(\theta ,u_0){\parallel }_{V^{\ast }}/\parallel u_0{\parallel }_V\le {10}^{-6}$. Furthermore, simulations were interrupted whenever the objective function value became complex, which can happen due to the log-transformation of the output.

Figure 3. (A) Geometry of a lymphoid vessel obtained from biological imaging data [27]. (B) Simulated data of the CCL21 gradient generated by simulating model (29(29) $\begin{array}{rl}& u_t-D\mathrm{\Delta }u=\alpha Q-k_{1}us+k_{-1}c-\gamma u,\\ & \dot{s}=-k_{1}us+k_{-1}c,\\ & \dot{c}=k_{1}us-k_{-1}c,\end{array}$(29) ).

7.3. Simulated data

To evaluate the convergence properties of the proposed algorithm for the models, we considered published simulated data for the ground truth (similar to [28]). The geometry of lymphatic tissue was extracted from the available imaging data [27] using the MATLAB PDE toolbox. On this geometry, the discretized PDE was simulated using biologically plausible parameter values (Table 1). The simulated data for CCL21 gradient formation process were corrupted by noise to obtain a plausible scenario (Figure 3B).

Table 1. True parameters, estimated parameters and parameter ranges for the latin hypercube sampling for the CCL21 model.

Name	True value	Estimates	Lower bound sampling	Upper bound sampling
D	8.50 ${10}^1$	8.51 ${10}^1$	1.50 ${10}^{-2}$	2.50 ${10}^4$
α	2.40 ${10}^{-1}$	2.36 ${10}^{-1}$	4.50 ${10}^{-5}$	2.00 ${10}^1$
$s_0$	2.10 ${10}^{-1}$	2.13 ${10}^{-1}$	2.50 ${10}^{-3}$	3.00 ${10}^0$
b	1.30 ${10}^{-2}$	1.30 ${10}^{-2}$	2.50 ${10}^{-3}$	1.00 ${10}^0$
σ	5.00 ${10}^{-2}$	4.99 ${10}^{-2}$	2.50 ${10}^{-3}$	3.00 ${10}^0$

7.4. Optimization

The objective function for most parameter estimation problems is non-convex and can be multimodal. For this reason, we employed multi-start local optimization using the continuous analogue for which we have established local convergence in this paper. The starting points for the local optimizations were sampled using a latin hypercube approach with lower and upper bounds provided in Table 1. We used a linear parametrization for the states and a log-parametrization $\xi =\log \left(\theta \right)$ for the parameters following previous evaluations for biochemical systems [29]. We did not implement any bounds for values of parameter or states. The implementation of the multi-start local optimization is based upon the MATLAB toolbox PESTO [30]. The implementation of the objective function and finite element schemes was adapted from [25]. For the local optimization with the continuous analogue, we chose the negative gradient as descent direction.

As a reference, we performed also multi-start local optimization using a discrete iterative optimization method. We used the state-of-the-art optimizer fmincon.m with the starting points sampled for the continuous analogue and the interior point algorithm implemented in the MATLAB Optimization Toolbox. This interior point algorithm employs either a Newton step, where the Hessian is approximated by the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm, or a conjugate gradient step using a trust region [31–33]. The optimizer was provided with the objective function, the nonlinear constraint, as well as the corresponding derivatives. We used the same parametrization as for the continuous analogue and additionally constrained parameter values for $\xi =\log \left(\theta \right)$ in the optimization by the same upper and lower bounds used for the sampling (Table 1). The value of $u_0$ at the nodes of the mesh for the finite element discretization was constrained using upper and lower bounds for the optimization to lie in $[-1,3]$. A total of 2000 iterations and 4000 function evaluations was allowed.

7.5. Comparison of continuous analogue and discrete iterative procedure

We performed 100 local optimization runs with the continuous analogue for a retraction factor $\lambda ={10}^7$ and a discrete iterative method (Figure 4A). Both methods found the same best parameter value (Table 1) and achieved a good fit to the data. The assessment of the results revealed a good convergence of the continuous analogue. Almost $90\mathrm{\%}$ of the runs achieved an objective function value which was comparable with the best objective function value found across all runs (relative difference $<0.001\mathrm{\%}$). Overall, 96% of the runs finished successfully, meaning that either the optimization was stopped because the stopping criterion was fulfilled or the maximum number of iterations was reached, while 4% of the runs stopped prematurely.

Figure 4. Results of parameter estimation for CCL21 model. (A) Sorted objective function values for the multi-start optimization with continuous analogue ($\lambda ={10}^7$) and discrete iterative procedure. Converged runs are indicated in blue. (B) CPU time needed per optimizer run for the optimization using the continuous analogue and the discrete iterative procedure (lighter grey colour indicates runs which stopped because the maximal number of iterations was reached). The box covers the range between the 25th and the 75th percentile of the distribution. The median CPU time is indicated by a line. (C) Histogram of values for ${\hat{\lambda }}^{\ast }$ (Remark 5.2) obtained for 1000 points sampled in parameter-state space. (D) Percentage of completed runs (top), converged runs (middle)and median as well as 25th and 75th percentile of the runtime of completed runs (bottom) for different values of λ. For each value of λ, 100 local optimization runs were performed.

The discrete iterative optimization converged for $66\mathrm{\%}$ of the runs to the optimal value (Figure 4A). Accordingly, the success rate was substantially lower than for the proposed continuous analogue. Of the runs which did not converge to the global optimum 25 runs were stopped because the maximal number of iterations was reached.

For the considered problem, the continuous analogue outperformed the discrete iterative method regarding the CPU time (Figure 4B). We found a median CPU time of 15 minutes for the continuous solver and 174 minutes for the discrete iterative procedure. In light of the fact that the discrete iterative method uses second-order information, it is interesting to observe that a continuous analogue using the negative gradient is more efficient. One possible explanation is that the efficiency of the continuous analogue is a result of the application of sophisticated numerical solvers. The adaptive, implicit solver ode15s, which is provided with the analytical Jacobian of the ODE–PDE model, might facilitate large step-sizes and fast convergence. Indeed, the Jacobian also provides second-order information.

7.6. Evaluation of retraction factor influence

As an analytical calculation of the bound for the retraction factor was not possible, we sampled 1000 points in parameter-state space and evaluated the estimate for the lower bound ${\hat{\lambda }}^{\ast }$ (Remark 5.2). The histogram of the resulting values for ${\hat{\lambda }}^{\ast }$ is presented in Figure 4C. The values for ${\hat{\lambda }}^{\ast }$ span many orders of magnitude, and the distribution peaks at ${10}^4$. This result indicated that for different starting points very different retraction factors might be ideal.

To investigate the convergence properties for the different values of the retraction factor λ, we performed 100 local optimization runs for a range of different retraction factors. For each retraction factor, we assessed the number of completed runs and the number of converged runs (Figure 4D). Interestingly, as λ was increased the percentage of completed runs decreased. Yet, for large retraction factors many of the completed runs also converged, while for small retraction factors no runs converged as the maximum number of iterations becomes too large. The median CPU time for the optimization of one run decreased for increasing values of λ (Figure 4D). Notably, for the small values, the median CPU time was nearly six to seven times higher than the smallest one. The quantiles indicate that also the variability was higher for small values of λ. These results indicated that the retraction factor should be chosen large enough but not too large.

In summary, the analysis of the model of CCL21 gradient formation revealed that the retraction factor λ has a substantial influence on the convergence properties as well as the run time. For low values of λ starts did not converge while for large values of λ increasing stiffness of the problem could be observed. In an intermediate regime, which could here also be found by random sampling, we found the best convergence properties.

8. Conclusion and outlook

Parameter estimation is an important problem in a wide range of applications. Robustness and performance of the available iterative methods is, however, often limited. In this study, we introduced continuous analogues of descent methods for optimization with PDE constraints. For these continuous analogues, we proved local convergence of their solutions to the optima. The necessary assumptions are fulfilled for a wide range of application problems, rendering the results interesting for several research fields.

We demonstrated the applicability of continuous analogues for a model of gradient formation in biological tissues and compared them with an iterative discrete procedure. The results highlight the potential of the continuous analogues, e.g. a high convergence rate and lower computation times than the discrete iterative procedure. For the comparison, we used the MATLAB optimization routine fmincon.m, a state-of-the-art discrete iterative procedure. Alternatives would be IPOPT or KNITRO. As fmincon is a generic interior point method, there might apparently be approaches which are efficient for the considered PDE-constrained problems (see also the no free lunch theorem [34]). The evaluation of the influence of the retraction factor revealed the importance of an appropriate choice of the retraction factor as well as the issue of premature stopping. In this study, we provide a lower bound for λ which ensures local convergence. As this bound might, however, be conservative and can only be assessed pointwise, the use of adaptive methods might be interesting. To address the issue of premature stopping, bounds for parameters and state variables have to be implemented, e.g. by including log-barrier functions [35] in the objective function or through projection into the feasible space.

In the application problem, we only considered elliptic PDE constraints as for the proposed continuous analogues parabolic constraints can be encapsulated in the objective function. This changes the objective function landscape and indirectly influences the convergence. Conceptually, it should also be possible to formulate continuous analogues which do not require a solution operator for the parabolic PDE but also have the solution of the parabolic PDE as a state variable. This mathematically more elegant approach is left for future research.

In conclusion, this study presented continuous analogues for a new problem class. Similar to other problem classes for which continuous analogues have been established [15,16], we expect an improvement of convergence and computation time. The continuous analogues for optimization also complement recent work on simulation-based methods for uncertainty analysis [36]. The efficient implementation of these methods in easily accessible software packages should be a focus of future research as it would render the methods available to a broad community.

The method and its analysis apply as they are to the case of infinite dimensional parameters θ. However, in that situation, the inverse problem of identifying θ is often ill-posed, so the assumption of practical identifiability (cf. Assumption 5.5 and Remark 6.4) might not be satisfied. To restore stability, regularization can be employed, as pointed out in Remark 6.4.

Future research in this context will be concerned with globalization strategies, such as those proposed in [20,21], cf. Remark 5.4.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

B.K. acknowledges financial support by the Austrian Science Fund FWF under grants I2271 and P30054. A.F. and J.H. acknowledge financial support by the German Research Foundation (DFG) under grant HA7376/1-1.