Infinite horizon impulse control of stochastic functional differential equations driven by Lévy processes

Abstract

We consider impulse control of stochastic functional differential equations (SFDEs) driven by Lévy processes under an additional $L^p$-Lipschitz condition on the coefficients. Our results, which are first derived for a general stochastic optimization problem over infinite horizon impulse controls and then applied to the case of a controlled SFDE, apply to the infinite horizon as well as the random horizon settings. The methodology employed to show existence of optimal controls is a probabilistic one based on the concept of Snell envelopes.

Keywords:

• Impulse control
• infinite horizon
• Lévy processes
• snell envelope

1. Introduction

The standard stochastic impulse control problem is an optimal control problem that arises when an operator controls a dynamical system by intervening on the system at a discrete set of stopping times. Generally, an intervention can be represented by an element in the control set U which we assume to be a compact subset of ${\mathbb{R}}^m$.

In impulse control the control-law, thus, takes the form $u=({\tau }_1,\dots ,{\tau }_N;{\text{β}}_1,\dots ,{\text{β}}_N)$, where ${\tau }_1\le {\tau }_2\le \cdots \le {\tau }_N$ is a sequence of times when the operator intervenes on the system and ${\text{β}}_j$ is the impulse that the operator affects the system with at time ${\tau }_j$. The standard impulse control problem in infinite horizon can be formulated as finding a control that maximizes (1) $\begin{array}{r}\mathbb{E}\left[{\int }_0^{\mathrm{\infty }}{e}^{-{\rho }_{0}s}\varphi \right(s,X_s^u)\mathrm{d}s-\sum _{j=1}^N{e}^{-{\rho }_0{\tau }_j}\ell ({\tau }_j,X_{{\tau }_j}^u,{\text{β}}_j\left)\right],\end{array}$(1) where ${\rho }_0>0$ is a constant referred to as the discount factor, $X^u$ is an ${\mathbb{R}}^m$-valued controlled stochastic process that jumps at interventions (e.g.  by setting $X_{{\tau }_j}^u=\mathrm{\Gamma }({\tau }_j,X_{{\tau }_j-}^u,{\text{β}}_j)$ for some deterministic function Γ) and the deterministic functions¹ $\varphi :{\mathbb{R}}_{+}\times {\mathbb{R}}^m\to \mathbb{R}$ and $\ell :{\mathbb{R}}_{+}\times {\mathbb{R}}^m\times U\to \mathbb{R}$ give the running reward and the intervention costs, respectively. The quantity $\ell (t,x,b)$, thus, represents the cost incurred by applying the impulse $b\in U$ at time $t\in {\mathbb{R}}_{+}$ when the state is x.

As impulse control problems appear in a vast number of real-world applications (see e.g. [19,23] for applications in finance and [2,5] for applications in energy) a lot of attention has been given to various types of problems where the control is of impulse type. In the standard Markovian setting, where $X^u$ solves a stochastic differential equation (SDE) driven by a Lévy process on $[{\tau }_j,{\tau }_{j+1})$, the relation to quasi-variational inequalities has frequently been exploited to find optimal controls (see the seminal work in [3] or turn to [21] for a more recent textbook). In the non-Markovian framework an impulse control problem in finite horizon ($T<\mathrm{\infty }$) was solved in [8] by utilizing the link between optimal stopping and reflected BSDEs (originally discovered in [13]) while considering the reward functional $\begin{array}{r}\mathbb{E}\left[{\int }_0^T\varphi \right(s,\omega ,L_s^u)\mathrm{d}s-\sum _{j=1}^{N}c({\text{β}}_j\left)\right],\end{array}$where $\varphi :[0,T]\times \mathrm{\Omega }\times {\mathbb{R}}^n\to \mathbb{R}$ is now a random (and not necessarily Markovian) field and the controlled process $L^u$ takes the particular form $L_t^u:=L_t+\sum _{j=1}^N{\mathbb{1}}_{[{\tau }_j\le t]}{\text{β}}_j$, with L an (exogenous) non-controlled process and assuming that U is a finite set. Relevant is also the treatment of multi-modes optimal switching problems in a non-Markovian setting in [11].

In [16] the original work of [8] was extended to incorporate delivery lag by setting $L_t^u:=L_t+\sum _{j=1}^N{\mathbb{1}}_{[{\tau }_j+\mathrm{\Delta }\le t]}{\text{β}}_j$ for a fixed $\mathrm{\Delta }>0$. As in [22] the work in [16] is based on the assumption that ${\tau }_{j+1}\ge {\tau }_j+\mathrm{\Delta }$. This work was later extended by considering the infinite horizon setting in [10]. Notable is also the recent work on finite horizon impulse control of SFDEs driven by a Brownian motion in [17].

In the present article we take a different approach to all the above mentioned works by considering the abstract reward functional (2) $\begin{array}{r}J\left(u\right):=\mathbb{E}\left[\phi \right({\tau }_1,\dots ,{\tau }_N;{\text{β}}_1,\dots ,{\text{β}}_N)-\sum _{j=1}^{N}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j\left)\right],\end{array}$(2) where the terminal reward φ maps controls to values of the real line and is measurable with respect to $\mathcal{G}\otimes \mathcal{B}\left(\mathcal{D}\right)$, where $\mathcal{B}\left(\mathcal{D}\right)$ is the Borel σ-field of $\mathcal{D}:={\cup }_{i=0}^{\mathrm{\infty }}{D}^i$, with $D^0:=\mathrm{\varnothing }$, $D^i:=\left\{\right(t_1,\dots ,t_i;b_1,\dots ,b_i):0\le t_1\le \cdots \le t_i,b_j\in U,\mathrm{f}\mathrm{o}\mathrm{r}j=1,\dots ,i\}$ for $i\ge 1$ and $\mathcal{G}$ is the σ-field of a complete probability space $(\mathrm{\Omega },\mathcal{G},\mathbb{P})$. The intervention cost c is also assumed to be a $\mathcal{G}\otimes \mathcal{B}\left(\mathcal{D}\right)$-measurable map in addition to being bounded from below by a deterministic positive function. We consider the partial information setting and assume that we observe the system through a filtration $\mathbb{F}:=\{{\mathcal{F}}_t{\}}_{t\ge 0}$ of sub-σ-fields of $\mathcal{G}$ and thus restrict our attention to $\mathbb{F}$-adapted controls.

To indicate the applicability of the results we consider the special case when (3) $\begin{array}{r}\phi \left(u\right)={\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^u)\mathrm{d}s,\end{array}$(3) and (4) $\begin{array}{r}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j)=e^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{({\tau }_1,\dots ,{\tau }_{j-1};{\text{β}}_1,\dots ,{\text{β}}_{j-1})},{\text{β}}_j),\end{array}$(4) where $X^u$ solves an impulsively controlled stochastic functional differential equation (SFDE) driven by a Lévy process under an additional $L^p$-type Lipschitz condition on the coefficients of the SFDE. Furthermore, we will see that the results easily extend to problems with a random horizon which allows us to model aspects such as default in financial applications. We thus extend the result in [17] on the one hand by considering a more general driving noise but also by considering both the infinite and the random horizon settings. Our treatment of the random horizon problem also motivates the exploration of partial information as optimal controls may be fundamentally different in the partial information setting.

The main contributions of the present work are twofold. First, we show that the problem of maximizing J has a solution under certain assumptions on φ and c, summarized in the definition of an admissible reward pair, by finding an optimal control in terms of a family of interconnected value processes. We refer to this family of processes as a verification family. Furthermore, we give a set of conditions under which the reward pair defined by (3(3) $\begin{array}{r}\phi \left(u\right)={\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^u)\mathrm{d}s,\end{array}$(3) )–(4(4) $\begin{array}{r}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j)=e^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{({\tau }_1,\dots ,{\tau }_{j-1};{\text{β}}_1,\dots ,{\text{β}}_{j-1})},{\text{β}}_j),\end{array}$(4) ) is admissible.

The remainder of the article is organized as follows. In the next section we state the problem, set the notations used throughout the article and detail the set of assumptions that are made. In particular we introduce the notion of an admissible reward pair. Then, in Section 3 a verification theorem is derived. This verification theorem is an extension of the verification theorem for the multi-modes optimal switching problem with memory developed in [24] and presumes the existence of a verification family. In Section 4 we show that, under the assumptions made, there exists a verification family whenever $(\phi ,c)$ is an admissible reward pair, thus proving existence of an optimal control for the impulse control problem with the cost functional J defined in (2(2) $\begin{array}{r}J\left(u\right):=\mathbb{E}\left[\phi \right({\tau }_1,\dots ,{\tau }_N;{\text{β}}_1,\dots ,{\text{β}}_N)-\sum _{j=1}^{N}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j\left)\right],\end{array}$(2) ). Then, in Section 5 we show that a type of impulse control problems for controlled SFDEs satisfies the conditions on φ and c as prescribed in the definition of an admissible reward pair, both in the infinite and random horizon settings. Finally in the appendix, we recall some results, such as the Snell envelope, that are useful when showing existence of optimal controls.

2. Preliminaries

Let $(\mathrm{\Omega },\mathcal{G},\mathbb{P})$ be a complete probability space, and $\mathbb{F}:=({\mathcal{F}}_t{)}_{t\ge 0}$ a filtration of sub-σ-fields of $\mathcal{G}$ satisfying the usual conditions in addition to being quasi-left continuous. We assume that ${\mathcal{F}}_0$ is the trivial σ-field and define $\mathcal{F}:={\mathcal{F}}_{\mathrm{\infty }}$.

Throughout, we will use the following notations. Let:

• ${\mathcal{P}}_{\mathbb{F}}$ be the σ-algebra on ${\mathbb{R}}_{+}\times \mathrm{\Omega }$ of $\mathbb{F}$-progressively measurable subsets.

• For $p\ge 1$, ${\mathcal{S}}^p$ be the set of all finite, $\mathbb{R}$-valued, ${\mathcal{P}}_{\mathbb{F}}$-measurable, càdlàg  processes $(Z_t:t\ge 0)$ such that $\mathbb{E}\left[\underset{t\in {\mathbb{R}}_{+}}{sup}|Z_t{|}^p\right]<\mathrm{\infty }$ and let ${\mathcal{S}}_{\mathit{q}}^p$ be the subset of processes that are quasi-left upper semi-continuous (see Appendix 1 for a definition of quasi-left continuity).

• $\mathcal{T}$ be the set of all $\mathbb{F}$-stopping times and for each $\eta \in \mathcal{T}$ we let ${\mathcal{T}}_{\eta }$ be the corresponding subsets of stopping times τ such that $\tau \ge \eta$, $\mathbb{P}$-a.s. Furthermore, we let ${\mathcal{T}}^f$ (resp. ${\mathcal{T}}_{\eta }^f$) be the subset of $\mathcal{T}$ (resp. ${\mathcal{T}}_{\eta }$) with all stopping times τ for which $\mathbb{P}[\tau <\mathrm{\infty }]=1$.

• For each $\tau \in \mathcal{T}$, $\mathcal{A}\left(\tau \right)$ be the set of all ${\mathcal{F}}_{\tau }$-measurable random variables taking values in U.

• $\mathcal{U}$ be the set of all controls $u=({\tau }_1,\dots ,{\tau }_N;{\text{β}}_1,\dots ,{\text{β}}_N)$, where $({\tau }_j{)}_{j=1}^{\mathrm{\infty }}$ (the intervention times) is a non-decreasing sequence of $\mathbb{F}$-stopping times, ${\text{β}}_j\in \mathcal{A}\left({\tau }_j\right)$ (the interventions) and $N:=sup\{j\ge 1:{\tau }_j<\mathrm{\infty }\}\vee 0$ is the (random, $\mathcal{F}$-measurable) number of interventions.

• ${\mathcal{U}}^f$ denote the subset of $u\in \mathcal{U}$ for which $N\left(t\right):=sup\{j:{\tau }_j\le t\}$ is $\mathbb{P}$-a.s. finite on compacts (i.e.  ${\mathcal{U}}^f:=\{u\in \mathcal{U}:\mathbb{P}[\{\omega \in \mathrm{\Omega }:N(t)>k,\mathrm{\forall }k>0\}]=0,\mathrm{\forall }t\in {\mathbb{R}}_{+}\}$) and for all $k\ge 0$ we let ${\mathcal{U}}^k$ be the set of all controls $({\tau }_1,\dots ,{\tau }_{N\wedge k};{\text{β}}_1,\dots ,{\text{β}}_{N\wedge k})$ truncated at k interventions and set ${\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}:={\cup }_{k\ge 0}{\mathcal{U}}^k$.

• For a random interval A (i.e.  a set of the type $[{\eta }_1,{\eta }_2]$, $[{\eta }_1,{\eta }_2)$, $({\eta }_1,{\eta }_2]$ or $({\eta }_1,{\eta }_2)$ for some ${\eta }_1,{\eta }_2\in \mathcal{T}$) ${\mathcal{U}}_A$ (and ${\mathcal{U}}_A^f$ resp. ${\mathcal{U}}_A^k$) be the subset of $\mathcal{U}$ (and ${\mathcal{U}}^f$ resp. ${\mathcal{U}}^k$) with ${\tau }_j\in A$, $\mathbb{P}$-a.s. for $j=1,\dots ,N$. When the interval is $A=[\eta ,\mathrm{\infty }]$ for some $\eta \in \mathcal{T}$ we use the shorthand ${\mathcal{U}}_{\eta }$ (and ${\mathcal{U}}_{\eta }^f$ resp. ${\mathcal{U}}_{\eta }^k$).

• ${\mathcal{D}}^f$ be the subset of $\mathcal{D}$ with all finite sequences and for $k\ge 0$ we let ${\mathcal{D}}^k:={\cup }_{i=0}^k{D}^i$.

• $v=(t,b)$, with $t:=(t_1,\dots ,t_n)$ and $b:=(b_1,\dots ,b_n)$, where n is possibly infinite, denote a generic element of $\mathcal{D}$.

• For $v=(t,b)\in {\mathcal{D}}^f$ and $v^{\prime }=(t^{\prime },b^{\prime })\in \mathcal{D}$, we introduce the composition, denoted by °, defined as² $v\circ v^{\prime }:=(t_1,\dots ,t_n,t_1^{\prime }\vee t_n,\dots ,t_{n^{\prime }}^{\prime }\vee t_n;b_1,\dots ,b_n,b_1^{\prime },\dots ,b_{n^{\prime }}^{\prime })$. For $v\in \mathcal{D}$, we define the truncation to $k\ge 0$ interventions as $[v{]}_k:=(t_1,\dots ,t_{k\wedge n};b_1,\dots ,b_{k\wedge n})$.

• The composition operator ° be extended to controls by setting $u\circ \tilde{u}:=({\tau }_1,\dots ,{\tau }_{N\wedge k},{\tilde{\tau }}_1\vee {\tau }_k,\dots ,{\tilde{\tau }}_{\hat{N}}\vee {\tau }_k;{\text{β}}_1,\dots ,{\text{β}}_{N\wedge k},{\widetilde{\text{β}}}_1,\dots ,{\widetilde{\text{β}}}_{\hat{N}})$, where $\hat{N}:=sup\{j\ge 1:{\tilde{\tau }}_j\vee {\tau }_k<\mathrm{\infty }\}\vee 0$, whenever $u\in {\mathcal{U}}^k$ and $\tilde{u}:=({\tilde{\tau }}_1,\dots ,{\tilde{\tau }}_{\tilde{N}};{\widetilde{\text{β}}}_1,\dots ,{\widetilde{\text{β}}}_{\tilde{N}})\in \mathcal{U}$.

• ${\mathrm{\Pi }}_l:=\{0,1/2^l,2/2^l,\dots \}$ for $l\ge 0$ and set $\mathrm{\Pi }:={\cup }_{l=1}^{\mathrm{\infty }}{\mathrm{\Pi }}_l$.

Furthermore, we introduce the following set:

Definition 2.1

We let ${\mathcal{H}}_{\mathbb{F}}^{\prime }$ be the set of all ${\mathcal{P}}_{\mathbb{F}}\otimes \mathcal{B}\left(U\right)$-measurable maps³ $h:{\mathbb{R}}_{+}\times \mathrm{\Omega }\times U\to \mathbb{R}$ such that the collection $\left\{h\right(\tau ,\text{β}):\tau \in {\mathcal{T}}^f,\text{β}\in \mathcal{A}(\tau \left)\right\}$ is uniformly integrable and (outside of a $\mathbb{P}$-null set) we have for all $(t,b)\in {\mathbb{R}}_{+}\times U$:

1. The limit $\underset{(t^{\prime },b^{\prime })\to (t,b)}{lim}h(t^{\prime },b^{\prime })$ exists;

2. $\underset{t^{\prime }\searrow t}{lim}\underset{b^{\prime }\in U}{sup}|h(t^{\prime },b^{\prime })-h(t,b^{\prime })|=0$;

3. $\underset{b^{\prime }\to b}{lim}h(t,b^{\prime })\le h(t,b)$.

Furthermore, let ${\mathcal{H}}_{\mathbb{F}}$ be the set of all $h\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$ such that for any predictable stopping time $\theta \in \mathcal{T}$ and any announcing sequence ${\theta }_j\nearrow \theta$ with ${\theta }_j\in {\mathcal{T}}^f$ we have $\underset{j\to \mathrm{\infty }}{lim sup}\underset{b\in U}{sup}\left\{h\right({\theta }_j,b)-h(\theta ,b\left)\right\}\le 0$, $\mathbb{P}$-a.s.

The sets ${\mathcal{H}}_{\mathbb{F}}$ and ${\mathcal{H}}_{\mathbb{F}}^{\prime }$ will play an important role in the characterization of optimal controls.

2.1. Problem formulation

With the notations above, the problem we deal with is characterized by the following objects:

• $(\mathrm{\Omega },\mathcal{G},\mathbb{F},\mathbb{P})$ a complete probability space.

• A $\mathcal{G}\otimes \mathcal{B}\left(\mathcal{D}\right)$-measurable map $\phi :\mathcal{D}\to \mathbb{R}$.

• A $\mathcal{G}\otimes \mathcal{B}\left(\mathcal{D}\right)$-measurable map $c:\mathcal{D}\to {\mathbb{R}}_{+}$.

To obtain existence of optimal controls we need to make some assumptions on the involved objects. The assumptions that we will use are summarized in the definition of what we refer to as an admissible reward pair:

Definition 2.2

We call the pair $(\phi ,c)$ an admissible reward pair if, for some⁴ p>2:

1. The terminal reward φ and the intervention cost c are both right-continuous in the intervention times (uniformly in the interventions) and satisfy the following bounds:

   1. $\underset{u\in \mathcal{U}}{sup}\mathbb{E}\left[|\phi \right(u\left){|}^p\right]<\mathrm{\infty }$.

   2. $\underset{u\in \mathcal{U}}{sup}\mathbb{E}\left[|c\right(u\left){|}^2\right]<\mathrm{\infty }$ and $c\left(v\right)\ge \delta \left(t_n\right)$ for all $v\in {\mathcal{D}}^f$, $\mathbb{P}$-a.s., where $\delta :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ is a deterministic, continuous, non-increasing and positive function, i.e.  $\delta \left(s\right)\ge \delta \left(t\right)>0$, whenever $0\le s\le t<\mathrm{\infty }$.

2. For every $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ and every $k\ge 0$ and T>0, there are maps ${\chi }^T\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$, $\chi \in {\mathcal{H}}_{\mathbb{F}}$ and $(t,b)\mapsto -c_{t,b}^v\in {\mathcal{H}}_{\mathbb{F}}$ such that for all $\tau \in \mathcal{T}$ and $b\in U$ we have $\begin{array}{rl}\chi (\tau ,b)& =\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,b)\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,b)\circ [u{]}_j)|{\mathcal{F}}_{\tau }],\\ {\chi }^T(\tau ,b)& =\underset{u\in {\mathcal{U}}_{[\tau ,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,b)\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,b)\circ [u{]}_j)|{\mathcal{F}}_{\tau }],\end{array}$and $\begin{array}{r}{c}_{\tau ,b}^v=\mathbb{E}[c(v\circ (\tau ,b\left)\right)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. (with a $\mathbb{P}$-null exception set that can be chosen independently of b).

3. We have $\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}},v\in {\mathcal{U}}_T^f}{sup}\mathbb{E}\left[|\phi \right(u\circ v)-\phi (u\left){|}^p\right]\to 0$ as $T\to \mathrm{\infty }$

The conditions in the above definition are mainly standard assumptions for infinite horizon stochastic impulse control problems translated to our setting (see e.g.  [10]). Condition (i.a) together with positivity of the intervention cost, c, in (i.b) implies that the expected maximal reward is finite. Condition (iii) implies that the future has diminishing impact on the total reward and can be seen as a generalization of the deterministic discounting applied in (1(1) $\begin{array}{r}\mathbb{E}\left[{\int }_0^{\mathrm{\infty }}{e}^{-{\rho }_{0}s}\varphi \right(s,X_s^u)\mathrm{d}s-\sum _{j=1}^N{e}^{-{\rho }_0{\tau }_j}\ell ({\tau }_j,X_{{\tau }_j}^u,{\text{β}}_j\left)\right],\end{array}$(1) ). We show below that the boundedness of the intervention costs from below by a positive function together with (i. (a) imply that, with probability one, the optimal control (whenever it exists) can only make a finite number of interventions within any compact time interval.

Remark 2.1

Note that we may hide part of the intervention cost within the function φ which implies that, similar to the setting in [20], we can handle problems with negative intervention costs as long as a type of martingale condition is satisfied.

Recall the reward functional given by (2(2) $\begin{array}{r}J\left(u\right):=\mathbb{E}\left[\phi \right({\tau }_1,\dots ,{\tau }_N;{\text{β}}_1,\dots ,{\text{β}}_N)-\sum _{j=1}^{N}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j\left)\right],\end{array}$(2) ). The problem we deal with might be formulated as:

Problem 2.1

Find $u^{\ast }\in \mathcal{U}$, such that (5) $\begin{array}{r}J\left(u^{\ast }\right)=\underset{u\in \mathcal{U}}{sup}J\left(u\right),\end{array}$(5) when $(\phi ,c)$ is an admissible reward pair.

Throughout Sections 3–4 we will thus assume that $(\phi ,c)$ is an admissible reward pair, before we state a set of conditions under which we are able to show that a particular $(\phi ,c)$ of the form (3(3) $\begin{array}{r}\phi \left(u\right)={\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^u)\mathrm{d}s,\end{array}$(3) )–(4(4) $\begin{array}{r}c({\tau }_1,\dots ,{\tau }_j;{\text{β}}_1,\dots ,{\text{β}}_j)=e^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{({\tau }_1,\dots ,{\tau }_{j-1};{\text{β}}_1,\dots ,{\text{β}}_{j-1})},{\text{β}}_j),\end{array}$(4) ) is an admissible reward pair.

As a step in solving Problem 1 we need the following proposition which is a reduction result for impulse control problems.

Proposition 2.3

Suppose there is a $u^{\ast }\in \mathcal{U}$ such that $J\left(u^{\ast }\right)\ge J\left(u\right)$ for all $u\in {\mathcal{U}}^f$. Then $u^{\ast }$ is an optimal control for (5(5) $\begin{array}{r}J\left(u^{\ast }\right)=\underset{u\in \mathcal{U}}{sup}J\left(u\right),\end{array}$(5) ), i.e. $J\left(u^{\ast }\right)\ge J\left(u\right)$ for all $u\in \mathcal{U}$.

Proof.

Pick $\hat{u}:=({\hat{\tau }}_1,\dots ;{\widehat{\text{β}}}_1,\dots )\in \mathcal{U}\setminus {\mathcal{U}}^f$. Then there is a $t\in {\mathbb{R}}_{+}$ such that $\mathbb{P}\left[B\right]>0$ with $B:=\{\omega \in \mathrm{\Omega }:\hat{N}(t)>k,\mathrm{\forall }k>0\}$, where $\hat{N}\left(t\right):=max\{j:{\hat{\tau }}_j\le t\}\vee 0$. Furthermore, by positivity of the intervention costs⁵ $\begin{array}{r}J\left(\hat{u}\right)\le \underset{u\in \mathcal{U}}{sup}\mathbb{E}[|\phi \left(u\right)|]-k\delta \left(t\right)\mathbb{P}\left[B\right]\le C-k\delta \left(t\right)\mathbb{P}\left[B\right],\end{array}$for all $k\ge 0$, by Definition 2.2.(i). However, again by Definition 2.2.(i.a) we have $J\left(\mathrm{\varnothing }\right)\ge -C$. Hence, $\hat{u}$ is dominated by the strategy of doing nothing and the assertion follows.

2.2. Relevant properties of ${\mathcal{H}}_{\mathbb{F}}$

We note the following properties:

Lemma 2.4

1. If $h,h^{\prime }\in {\mathcal{H}}_{\mathbb{F}}$ (resp. $h,h^{\prime }\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$), then $h+h^{\prime }$ and $h\vee h^{\prime }$ are also in ${\mathcal{H}}_{\mathbb{F}}$ (resp. ${\mathcal{H}}_{\mathbb{F}}^{\prime }$).

2. If $h\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$ then there is a ${\mathcal{P}}_{\mathbb{F}}$-measurable càdlàg  process, $h^{\ast }$, of class [D], such that $h_{\tau }^{\ast }=\underset{b\in U}{sup}h(\tau ,b)$, $\mathbb{P}$-a.s. for each $\tau \in {\mathcal{T}}^f$. If $h\in {\mathcal{H}}_{\mathbb{F}}$ then $h^{\ast }$ is quasi-left upper semi-continuous.

3. If $h,h^{\prime }\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$, then $\left(\underset{b\in U}{sup}|h^{\prime }\right(t,b)-h(t,b)|:t\ge 0)$ is ${\mathcal{P}}_{\mathbb{F}}$-measurable and càdlàg. 

4. If $(h^k{)}_{k\ge 0}$ is a sequence in ${\mathcal{H}}_{\mathbb{F}}$ that converges uniformly to some h (outside of a $\mathbb{P}$-null set) then $h\in {\mathcal{H}}_{\mathbb{F}}$.

Proof.

Moving on to property b), we let $s_j^l:=j{2}^{-l}$ and note that for $j\ge 0$, there is a ${\text{β}}_{j+1}^l\in \mathcal{A}\left(s_j^l\right)$ such that $\begin{array}{r}\underset{b\in U}{sup}\mathbb{E}[h(s_{j+1}^l,b)|{\mathcal{F}}_{s_j^l}]=\mathbb{E}[h(s_{j+1}^l,{\text{β}}_{j+1}^l)|{\mathcal{F}}_{s_j^l}]\end{array}$$\mathbb{P}$-a.s., by Corollary A.5 in Appendix 3. By Theorem A.6 we can define the sequence of càdlàg  processes $({\hat{h}}^l{)}_{l\ge 0}$ as $\begin{array}{r}{\hat{h}}_t^l:=\sum _{j=0}^{\mathrm{\infty }}{\mathbb{1}}_{[s_j^l,s_{j+1}^l)}\left(t\right)\mathbb{E}[h(t,{\text{β}}_{j+1}^l)|{\mathcal{F}}_t].\end{array}$Now, we let ${\underset{\_}{h}}^0:={\hat{h}}^0$ and then recursively define ${\underset{\_}{h}}^l:={\underset{\_}{h}}^{l-1}\vee {\hat{h}}^l$ for $l\ge 1$. Then, $({\underset{\_}{h}}^l{)}_{l\ge 0}$ is a non-decreasing sequence of càdlàg  processes and $\begin{array}{r}\underset{b\in U}{sup}h(t,b)\ge {\underset{\_}{h}}_t^l,\end{array}$$\mathbb{P}$-a.s., for all $t\in [0,\mathrm{\infty })$ and all $l\ge 0$. Furthermore, for $t\ge 0$ we let ${\iota }_l:=max\{j:j{2}^{-l}\le t\}$ and get that $\begin{array}{rl}\underset{b\in U}{sup}h(t,b)-{\underset{\_}{h}}_t^l& =\underset{b\in U}{sup}h(t,b)-\mathbb{E}[h(s_{{\iota }_l+1}^l,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_{s_{{\iota }_l}^l}]+\mathbb{E}[h(s_{{\iota }_l+1}^l,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_{s_{{\iota }_l}^l}]\\ & -\mathbb{E}[h(t,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_t]\\ & \le \underset{b\in U}{sup}\left\{h\right(t,b)-\mathbb{E}[h(s_{{\iota }_l+1}^l,b\left)|{\mathcal{F}}_{s_{{\iota }_l}^l}]\right\}+\mathbb{E}[h(s_{{\iota }_l+1}^l,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_{s_{{\iota }_l}^l}]\\ & -\mathbb{E}[h(t,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_t].\end{array}$Since $s_{{\iota }_l}^l\nearrow t$ we have, by quasi-left continuity of the filtration and uniform integrability that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim}\mathbb{E}[h(s_{{\iota }_l+1}^l,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_{s_{{\iota }_l}^l}]=\underset{l\to \mathrm{\infty }}{lim}\mathbb{E}[h(s_{{\iota }_l+1}^l,{\text{β}}_{{\iota }_l+1}^l)|{\mathcal{F}}_t].\end{array}$Now, as $s_{{\iota }_l+1}^l\searrow t$, it follows by Definition 2.1.(ii) that ${\underset{\_}{h}}_t^l\nearrow \underset{b\in U}{sup}h(t,b)$, $\mathbb{P}$-a.s., as $l\to \mathrm{\infty }$. We note that ${\underset{\_}{h}}^l$ is an increasing, uniformly bounded, sequence of ${\mathcal{P}}_{\mathbb{F}}$-measurable càdlàg  processes. The sequence, thus, converges to a ${\mathcal{P}}_{\mathbb{F}}$-measurable process, $h^{\ast }$. It remains to show that $h^{\ast }$ is càdlàg, quasi-left upper semi-continuous and that it agrees with $\underset{b\in U}{sup}h(t,b)$ on stopping times.

To show that the limit is càdlàg  we let $\begin{array}{r}{\overline{h}}_t^l:=\sum _{j=0}^{\mathrm{\infty }}{\mathbb{1}}_{[s_j^l,s_{j+1}^l)}\left(t\right)\underset{(r,b)\in [t,s_{j+1}^l)\times U}{sup}h(r,b).\end{array}$We note that ${\overline{h}}^l$ has left and right limits and that, furthermore, $\underset{t^{\prime }\searrow t}{lim}{\overline{h}}_{t^{\prime }}^l\le {\overline{h}}_t^l$. Now, if $\underset{t^{\prime }\searrow t}{lim}{\overline{h}}_{t^{\prime }}^l<{\overline{h}}_t^l$ then $\begin{array}{r}{\overline{h}}_t^l=\underset{b\in U}{sup}h(t,b)=h(t,{\text{β}}_t)\end{array}$for some ${\mathcal{F}}_t$-measurable ${\text{β}}_t$, but then we would have $\begin{array}{r}h(t,{\text{β}}_t)>\underset{t^{\prime }\searrow t}{lim}{\overline{h}}_{t^{\prime }}^l\ge \underset{t^{\prime }\searrow t}{lim}h(t^{\prime },{\text{β}}_t)\end{array}$contradicting the fact that $\left(h\right(s,{\text{β}}_t):s\ge t)$ is right continuous. We conclude that $({\overline{h}}^l{)}_{l\ge 0}$ is a non-increasing sequence with ${\overline{h}}^l$ a $\{{\mathcal{F}}_{t+2^{-l}}{\}}_{t\ge 2^{-l}}$-adapted càdlàg  process and $\begin{array}{r}\underset{b\in U}{sup}h(t,b)\le {\overline{h}}_t^l.\end{array}$For $t\ge 0$ we know that there is a non-increasing sequence $({\tilde{\tau }}_l{)}_{l\ge 0}$, with ${\tilde{\tau }}_l$ a ${\mathcal{F}}_{t+2^{-l}}$-measurable r.v. taking values in $[t,t+2^{-l})$ such that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim}{\overline{h}}_t^l=\underset{l\to \mathrm{\infty }}{lim}\underset{b\in U}{sup}h({\tilde{\tau }}_l,b).\end{array}$Now, as $h\in {\mathcal{H}}_{\mathbb{F}}$ we have $\underset{t^{\prime }\searrow t}{lim}\underset{b\in U}{sup}|h(t^{\prime },b)-h(t,b)|=0$. It follows that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim}{\overline{h}}_t^l-\underset{b\in U}{sup}h(t,b)=\underset{l\to \mathrm{\infty }}{lim}\underset{b\in U}{sup}h({\tilde{\tau }}_l,b)-\underset{b\in U}{sup}h(t,b)\le \underset{l\to \mathrm{\infty }}{lim}\underset{b\in U}{sup}\left\{h\right({\tilde{\tau }}_l,b)-h(t,b\left)\right\}=0,\end{array}$$\mathbb{P}$-a.s., and in particular we find that ${\overline{h}}_t^l\searrow h_t^{\ast }$, $\mathbb{P}$-a.s. as $l\to \mathrm{\infty }$. This gives that for any sequence $t_j\searrow t$ and $l\ge 0$, $\begin{array}{r}\underset{t_j\searrow t}{lim inf}{h}_{t_j}^{\ast }\ge \underset{t_j\searrow t}{lim inf}{\underset{\_}{h}}_{t_j}^l={\underset{\_}{h}}_t^l\end{array}$and $\begin{array}{r}\underset{t_j\searrow t}{lim sup}{h}_{t_j}^{\ast }\le \underset{t_j\searrow t}{lim sup}{\overline{h}}_{t_j}^l={\overline{h}}_t^l.\end{array}$Letting l tend to infinity we find that $\begin{array}{r}\underset{t_j\searrow t}{lim inf}{h}_{t_j}^{\ast }=\underset{t_j\searrow t}{lim sup}{h}_{t_j}^{\ast }=h_t^{\ast }.\end{array}$Similarly we get existence of left-limits and by the uniform integrability property imposed on members of ${\mathcal{H}}_{\mathbb{F}}$ we conclude that $h^{\ast }$ is a càdlàg, ${\mathcal{P}}_{\mathbb{F}}$-measurable process of class [D].

For $\tau \in \mathcal{T}$, let $({\tau }_l{)}_{l\ge 0}$ be a non-increasing sequence of stopping times in ${\mathcal{T}}^{\mathrm{\Pi }}$ (the subset of $\mathcal{T}$ with all stopping times taking values in the countable set $\mathrm{\Pi }$) such that ${\tau }_l\searrow \tau$. We may, for example, set ${\tau }_l:=inf\{s\in {\mathrm{\Pi }}_l:s\ge \tau \}$. Since $\mathrm{\Pi }$ is countable we have (6) $\begin{array}{r}{h}_{{\tau }_l}^{\ast }=\underset{b\in U}{sup}h({\tau }_l,b),\end{array}$(6) $\mathbb{P}$-a.s. Now, by right-continuity we get that $\underset{l\to \mathrm{\infty }}{lim}{h}_{{\tau }_l}^{\ast }=h_{\tau }^{\ast }$. Letting $({\text{β}}_l{)}_{l\ge 0}$ be a sequence of maximizers for the right-hand side of (6(6) $\begin{array}{r}{h}_{{\tau }_l}^{\ast }=\underset{b\in U}{sup}h({\tau }_l,b),\end{array}$(6) ) at times $({\tau }_l{)}_{l\ge 0}$ we get $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim sup}\underset{b\in U}{sup}h({\tau }_l,b)=\underset{l\to \mathrm{\infty }}{lim sup}h({\tau }_l,{\text{β}}_l),\end{array}$$\mathbb{P}$-a.s. Moreover, for each $\omega \in \mathrm{\Omega }$ there is a subsequence $\left({\iota }_j\right(\omega ){)}_{j\ge 1}$ such that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim sup}h({\tau }_l,{\text{β}}_l)=\underset{j\to \mathrm{\infty }}{lim}h({\tau }_{{\iota }_j},{\text{β}}_{{\iota }_j}).\end{array}$Since U is compact, there is a subsequence $\left({\iota }_j^{\prime }\right(\omega ){)}_{j\ge 1}\subset ({\iota }_j\left(\omega \right){)}_{j\ge 1}$ such that $\left({\text{β}}_{{\iota }_j^{\prime }}\right(\omega ){)}_{j\ge 0}$ converges to some $\widetilde{\text{β}}\left(\omega \right)\in U$ and so we have $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim sup}\underset{b\in U}{sup}h({\tau }_l,b)=\underset{j\to \mathrm{\infty }}{lim}\left(h\right({\tau }_{{\iota }_j^{\prime }},{\text{β}}_{{\iota }_j^{\prime }})-h(\tau ,{\text{β}}_{{\iota }_j^{\prime }}\left)\right)+\underset{j\to \mathrm{\infty }}{lim}\left(h\right(\tau ,{\text{β}}_{{\iota }_j^{\prime }})-h(\tau ,\widetilde{\text{β}}\left)\right)+h(\tau ,\widetilde{\text{β}}).\end{array}$Now, $\begin{array}{r}\underset{j\to \mathrm{\infty }}{lim}\left(h\right({\tau }_{{\iota }_j^{\prime }},{\text{β}}_{{\iota }_j^{\prime }})-h(\tau ,{\text{β}}_{{\iota }_j^{\prime }}\left)\right)=0,\end{array}$$\mathbb{P}$-a.s., by Definition 2.1.(ii) and $\begin{array}{r}\underset{j\to \mathrm{\infty }}{lim}\left(h\right(\tau ,{\text{β}}_{{\iota }_j^{\prime }})-h(\tau ,\widetilde{\text{β}}\left)\right)\le 0,\end{array}$$\mathbb{P}$-a.s., by the upper semi-continuity declared in Definition 2.1.(iii). Further, as $h(\tau ,\widetilde{\text{β}})\le \underset{b\in U}{sup}h(\tau ,b)$ we conclude that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim sup}\underset{b\in U}{sup}h({\tau }_l,b)\le \underset{b\in U}{sup}h(\tau ,b),\end{array}$$\mathbb{P}$-a.s. On the other hand, there is a $\text{β}\in \mathcal{A}\left(\tau \right)$ such that $\underset{b\in U}{sup}h(\tau ,b)=h(\tau ,\text{β})$, $\mathbb{P}$-a.s., and we have $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim inf}\underset{b\in U}{sup}h({\tau }_l,b)=\underset{l\to \mathrm{\infty }}{lim inf}h({\tau }_l,{\text{β}}_l)\ge \underset{l\to \mathrm{\infty }}{lim}h({\tau }_l,\text{β})=h(\tau ,\text{β}),\end{array}$$\mathbb{P}$-a.s. This implies that the limit exists with $\underset{l\to \mathrm{\infty }}{lim}\underset{b\in U}{sup}h({\tau }_l,b)=\underset{b\in U}{sup}h(\tau ,b)$, $\mathbb{P}$-a.s., establishing that $h_{\tau }^{\ast }=\underset{b\in U}{sup}h(\tau ,b)$, $\mathbb{P}$-a.s. Finally, if $h\in {\mathcal{H}}_{\mathbb{F}}$ then quasi-left upper semi-continuity of $h^{\ast }$ is immediate from Definition 2.1 and (b) follows.

Property (c) follows similarly by noting that for any ${ϵ}_l>0$ we can chose ${\text{β}}_{j+1}^l\in \mathcal{A}\left(s_j^l\right)$ such that $\begin{array}{r}\underset{b\in U}{sup}\mathbb{E}[|h^{\prime }(s_{j+1}^l,b)-h(s_{j+1}^l,b)||{\mathcal{F}}_{s_j^l}]=\mathbb{E}[|h^{\prime }(s_{j+1}^l,{\text{β}}_{j+1}^l)-h(s_{j+1}^l,{\text{β}}_{j+1}^l)||{\mathcal{F}}_{s_j^l}]+{ϵ}_l\end{array}$and we can choose the sequence $({ϵ}_l{)}_{l\ge 0}$ such that ${ϵ}_l\searrow 0$.

Concerning the last property we note that for each $ϵ>0$ we can, by uniform convergence, choose a $\mathbb{P}$-a.s. finite $k\left(\omega \right)\ge 0$ such that $|h(t,b)-h^k(t,b)|\le ϵ$ for all $(t,b)\in {\mathbb{R}}_{+}\times U$. Then, $\begin{array}{rl}& \underset{(t^{\prime },b^{\prime })\to (t,b)}{lim sup}h(t^{\prime },b^{\prime })-\underset{(t^{\prime },b^{\prime })\to (t,b)}{lim inf}h(t^{\prime },b^{\prime })\\ & \le \underset{(t^{\prime },b^{\prime })\to (t,b)}{lim sup}{h}^k(t^{\prime },b^{\prime })-\underset{(t^{\prime },b^{\prime })\to (t,b)}{lim inf}{h}^k(t^{\prime },b^{\prime })+2ϵ=2ϵ\end{array}$and property (i) in Definition 2.1 follows as $ϵ>0$ was arbitrary. The remaining properties follow similarly.

3. A verification theorem

Our approach to finding a solution to Problem 1 is based on deriving an optimal control under the assumption that a specific family of processes exists, and then showing that the family does indeed exist. We will refer to any such family of processes as a verification family. Before making precise the concept of a verification family we introduce the notion of consistency:

Definition 3.1

We refer to a family of processes $\left(\right(X_t^v{)}_{t\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ as being consistent if for each $u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, the map $h:{\mathbb{R}}_{+}\times \mathrm{\Omega }\times U\to \mathbb{R}$ given by $h(t,b)=X_t^{u\circ (t,b)}$ is ${\mathcal{P}}_{\mathbb{F}}\otimes \mathcal{B}\left(U\right)$-measurable and for each $\tau \in \mathcal{T}$ and each $\text{β}\in \mathcal{A}\left(\tau \right)$ we have $X_{\tau }^{u\circ (\tau ,\text{β})}=h(\tau ,\text{β})$, $\mathbb{P}$-a.s.

We are now ready to state the definition of a verification family:

Definition 3.2

We define a verification family to be a consistent family of càdlàg  supermartingales $\left(\right(Y_s^v{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ such that for each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$:

1. The family satisfies the recursion (7) $\begin{array}{r}{Y}_s^v=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b)}\}|{\mathcal{F}}_s].\end{array}$(7)

2. The family is uniformly bounded in the sense that $\begin{array}{r}\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}\left[\underset{s\in [0,\mathrm{\infty }]}{sup}|Y_s^u{|}^2\right]<\mathrm{\infty }.\end{array}$

3. The map $(t,b)\mapsto Y_t^{v\circ (t,b)}$ belongs to ${\mathcal{H}}_{\mathbb{F}}$.

4. $\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{s\in [T,\mathrm{\infty }]}{sup}|Y_s^u-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_s\left]|\right]\to 0$, as $T\to \mathrm{\infty }$.

The purpose of the present section is to reduce the solution of Problem 1 to showing existence of a verification family. This is accomplished by the following verification theorem (the proof of which follows along the lines of the proof of Theorem 2 in [9]):

Theorem 3.3

Assume that there exists a verification family $\left(\right(Y_s^v{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ and let:

• the sequence $({\tau }_j^{\ast }{)}_{j=1}^{\mathrm{\infty }}$ be given by (8) $\begin{array}{r}{\tau }_j^{\ast }:=inf\{s\ge {\tau }_{j-1}^{\ast }:Y_s^{[u^{\ast }{]}_{j-1}}=\underset{b\in U}{sup}\{-c_{s,b}^{[u^{\ast }{]}_{j-1}}+Y_s^{[u^{\ast }{]}_{j-1}\circ (s,b)}\}\},\end{array}$(8) using the convention that $inf\mathrm{\varnothing }=\mathrm{\infty }$, with ${\tau }_0^{\ast }=0$ and set $N^{\ast }=sup\{j\ge 0:{\tau }_j^{\ast }<\mathrm{\infty }\}$;

• the sequence $({\text{β}}_j^{\ast }{)}_{j=1}^{\mathrm{\infty }}$ be defined recursively as a measurable selection of (9) $\begin{array}{r}{\text{β}}_j^{\ast }\in \underset{b\in U}{\arg max}\{-c_{{\tau }_j^{\ast },b}^{[u^{\ast }{]}_{j-1}}+Y_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_{j-1}\circ ({\tau }_j^{\ast },b)}\}.\end{array}$(9)

Then $u^{\ast }=({\tau }_1^{\ast },\dots ,{\tau }_{N^{\ast }}^{\ast };{\text{β}}_1^{\ast },\dots ,{\text{β}}_{N^{\ast }}^{\ast })\in {\mathcal{U}}^f$ is an optimal control for (5(5) $\begin{array}{r}J\left(u^{\ast }\right)=\underset{u\in \mathcal{U}}{sup}J\left(u\right),\end{array}$(5) ) in the sense that $J\left(u^{\ast }\right)=\underset{u\in \mathcal{U}}{sup}J\left(u\right)$. Moreover, the family is unique (i.e.  there is at most one verification family, up to indistinguishability of the maps $t\mapsto Y_t^v$ and $(t,b)\mapsto Y_t^{v\circ (t,b)}$) and $Y_0=\underset{u\in \mathcal{U}}{sup}J\left(u\right)$ (where $Y:=Y^{\mathrm{\varnothing }}$).

Proof.

The proof is divided into three steps where we first, in Step 1, show that for any $0\le j\le N^{\ast }$ we have (10) $\begin{array}{r}{Y}_s^{[u^{\ast }{]}_j}=\mathbb{E}[{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }=\mathrm{\infty }]}\phi \left(u^{\ast }\right)+{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }<\mathrm{\infty }]}\{-c(\left[u^{\ast }{]}_{j+1}\right)+Y_{{\tau }_{j+1}^{\ast }}^{[u^{\ast }{]}_{j+1}}\}|{\mathcal{F}}_s],\end{array}$(10) $\mathbb{P}$-a.s. for $s\in [{\tau }_j^{\ast },{\tau }_{j+1}^{\ast }]$. Following this, in Step 2, we show that $Y_0=J\left(u^{\ast }\right)$. Then in Step 3 we show that $u^{\ast }$ is the optimal control, establishing (i) and (ii). A straightforward generalization to arbitrary initial conditions $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ then gives that (11) $\begin{array}{r}{Y}_s^v=\underset{u\in {\mathcal{U}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)-\sum _{j=1}^{N}c(v\circ [u{]}_j)|{\mathcal{F}}_s],\end{array}$(11) by which uniqueness follows. Below we refer to the properties of a verification family in Definition 3.2 simply as properties a), b), a) and d).

Step 1 We start by showing that for each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ the recursion (7(7) $\begin{array}{r}{Y}_s^v=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b)}\}|{\mathcal{F}}_s].\end{array}$(7) ) can be written in terms of a $\mathbb{F}$-stopping time and that the inner supremum is attained, $\mathbb{P}$-a.s. In particular, this will imply the existence of a maximizer in (9(9) $\begin{array}{r}{\text{β}}_j^{\ast }\in \underset{b\in U}{\arg max}\{-c_{{\tau }_j^{\ast },b}^{[u^{\ast }{]}_{j-1}}+Y_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_{j-1}\circ ({\tau }_j^{\ast },b)}\}.\end{array}$(9) ). From (7(7) $\begin{array}{r}{Y}_s^v=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b)}\}|{\mathcal{F}}_s].\end{array}$(7) ) and consistency we note that $Y^v$ is the smallest supermartingale that dominates (12) $\begin{array}{r}{R}^v:=({\mathbb{1}}_{[s=\mathrm{\infty }]}\mathbb{E}[\phi \left(v\right)|\mathcal{F}]+{\mathbb{1}}_{[s<\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{s,b}^v+Y_s^{v\circ (s,b)}\}:0\le s\le \mathrm{\infty }).\end{array}$(12) By Property (c) and Definition 2.2.(ii) we have that the map $(t,b)\mapsto -c_{t,b}^v+Y_t^{v\circ (t,b)}$ belongs to ${\mathcal{H}}_{\mathbb{F}}$. It thus follows from Lemma 2.4.(b) that $R^v$ is a càdlàg  process of class [D] that is quasi-left upper semi-continuous on $[0,\mathrm{\infty })$. Furthermore, by Property (d) and positivity of the intervention costs we note that $\underset{t\to \mathrm{\infty }}{lim sup}{R}_t^v\le R_{\mathrm{\infty }}^v$, $\mathbb{P}$-a.s. By Theorem A.1.(iii) in Appendix 2 and consistency we conclude that for any $\theta \in {\mathcal{T}}^f$, the stopping time ${\tau }^{\theta }\in {\mathcal{T}}_{\theta }$ given by $\begin{array}{r}{\tau }^{\theta }:=inf\{s\ge \theta :Y_s^v=\underset{b\in U}{sup}\{-c_{s,b}^v+Y_s^{v\circ (s,b)}\}\}\end{array}$is such that: $\begin{array}{r}{Y}_{\theta }^v=\mathbb{E}[{\mathbb{1}}_{[{\tau }^{\theta }=\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[{\tau }^{\theta }<\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{{\tau }^{\theta },b}^v+Y_{{\tau }^{\theta }}^{v\circ ({\tau }^{\theta },b)}\}|{\mathcal{F}}_{\theta }].\end{array}$Now, since $(t,b)\mapsto -c_{t,b}^v+Y_t^{v\circ (t,b)}\in {\mathcal{H}}_{\mathbb{F}}$, the map $b\mapsto -c_{{\tau }^{\theta },b}^v+Y_{{\tau }^{\theta }}^{v\circ ({\tau }^{\theta },b)}$ is ${\mathcal{F}}_{{\tau }^{\theta }}\otimes \mathcal{B}\left(U\right)$-measurable and u.s.c. on $\{{\tau }^{\theta }<\mathrm{\infty }\}\setminus \mathcal{N}$ for some $\mathbb{P}$-null set $\mathcal{N}$. Corollary A.5 of Appendix 3 and consistency then implies that there is a ${\text{β}}^{\theta }\in \mathcal{A}\left({\tau }^{\theta }\right)$ such that $\begin{array}{r}{Y}_{\theta }^v=\mathbb{E}[{\mathbb{1}}_{[{\tau }^{\theta }=\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[{\tau }^{\theta }<\mathrm{\infty }]}\{-c(v\circ ({\tau }^{\theta },{\text{β}}^{\theta }))+Y_{{\tau }^{\theta }}^{v\circ ({\tau }^{\theta },{\text{β}}^{\theta })}\}|{\mathcal{F}}_{\theta }],\end{array}$$\mathbb{P}$-a.s., and in particular (10(10) $\begin{array}{r}{Y}_s^{[u^{\ast }{]}_j}=\mathbb{E}[{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }=\mathrm{\infty }]}\phi \left(u^{\ast }\right)+{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }<\mathrm{\infty }]}\{-c(\left[u^{\ast }{]}_{j+1}\right)+Y_{{\tau }_{j+1}^{\ast }}^{[u^{\ast }{]}_{j+1}}\}|{\mathcal{F}}_s],\end{array}$(10) ) holds. As mentioned above, this also implies the existence of a ${\mathcal{F}}_{{\tau }_j^{\ast }}$-measurable ${\text{β}}_j^{\ast }$ satisfying (9(9) $\begin{array}{r}{\text{β}}_j^{\ast }\in \underset{b\in U}{\arg max}\{-c_{{\tau }_j^{\ast },b}^{[u^{\ast }{]}_{j-1}}+Y_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_{j-1}\circ ({\tau }_j^{\ast },b)}\}.\end{array}$(9) ).

Step 2 We now show that $Y_0=J\left(u^{\ast }\right)$. We start by noting that Y is the Snell envelope of $\begin{array}{r}({\mathbb{1}}_{[s=\mathrm{\infty }]}\mathbb{E}[\phi \left(\mathrm{\varnothing }\right)|\mathcal{F}]+{\mathbb{1}}_{[s<\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{s,b}+Y_s^{s,b}\}:0\le s\le \mathrm{\infty })\end{array}$and by Step 1 we thus have (since ${\mathcal{F}}_0$ is trivial) that $\begin{array}{r}{Y}_0=\mathbb{E}[{\mathbb{1}}_{[{\tau }_1^{\ast }=\mathrm{\infty }]}\phi \left(\mathrm{\varnothing }\right)+{\mathbb{1}}_{[{\tau }_1^{\ast }<\mathrm{\infty }]}\{-c({\tau }_1^{\ast },{\text{β}}_1^{\ast })+Y_{{\tau }_1^{\ast }}^{{\tau }_1^{\ast },{\text{β}}_1^{\ast }}\}].\end{array}$Moving on we pick $j\in \{1,\dots ,N^{\ast }\}$ and note that $[u^{\ast }{]}_j\in {\mathcal{U}}^j$. But then, by Step 1, we have that $\begin{array}{r}{Y}_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_j}=\mathbb{E}[{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }=\mathrm{\infty }]}\phi \left(u^{\ast }\right)+{\mathbb{1}}_{[{\tau }_{j+1}^{\ast }<\mathrm{\infty }]}\{-c(\left[u^{\ast }{]}_{j+1}\right)+Y_{{\tau }_{j+1}^{\ast }}^{[u^{\ast }{]}_{j+1}}\}|{\mathcal{F}}_{{\tau }_j^{\ast }}].\end{array}$By induction we get that for each $K\ge 0$ we have, (13) $\begin{array}{r}{Y}_0=\mathbb{E}[{\mathbb{1}}_{[N^{\ast }\le K]}\phi \left(u^{\ast }\right)-\sum _{j=1}^{N^{\ast }\wedge K}c\left(\right[u^{\ast }{]}_j)+{\mathbb{1}}_{[N^{\ast }>K]}\{-c\left(\right[u^{\ast }{]}_{K+1})+Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}\}].\end{array}$(13) Now, arguing as in the proof of Proposition 2.3 and using Property (b) we find that $u^{\ast }\in {\mathcal{U}}^f$. To show that the right-hand side of (13(13) $\begin{array}{r}{Y}_0=\mathbb{E}[{\mathbb{1}}_{[N^{\ast }\le K]}\phi \left(u^{\ast }\right)-\sum _{j=1}^{N^{\ast }\wedge K}c\left(\right[u^{\ast }{]}_j)+{\mathbb{1}}_{[N^{\ast }>K]}\{-c\left(\right[u^{\ast }{]}_{K+1})+Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}\}].\end{array}$(13) ) equals $J\left(u^{\ast }\right)$ we note that (13(13) $\begin{array}{r}{Y}_0=\mathbb{E}[{\mathbb{1}}_{[N^{\ast }\le K]}\phi \left(u^{\ast }\right)-\sum _{j=1}^{N^{\ast }\wedge K}c\left(\right[u^{\ast }{]}_j)+{\mathbb{1}}_{[N^{\ast }>K]}\{-c\left(\right[u^{\ast }{]}_{K+1})+Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}\}].\end{array}$(13) ) can be rewritten as $\begin{array}{r}{Y}_0=\mathbb{E}[\phi \left(\right[u^{\ast }{]}_{K+1})-\sum _{j=1}^{N^{\ast }\wedge K+1}c(\left[u^{\ast }{]}_j\right)+{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]\end{array}$which gives (14) $\begin{array}{rl}|Y_0-J\left(u^{\ast }\right)|& \le \mathbb{E}[|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]+\mathbb{E}[\sum _{j=K+2}^{N^{\ast }}c(\left[u^{\ast }{]}_j\right)]\\ & +|\mathbb{E}[{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]|\end{array}$(14) for all $K\ge 0$. For the first term on the right-hand side we note that for any T>0 we have $\begin{array}{rl}& \mathbb{E}[|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]\\ & \le \mathbb{E}[{\mathbb{1}}_{[{\tau }_{K+2}^{\ast }<T]}|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]+\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}},v\in {\mathcal{U}}_T}{sup}\mathbb{E}[|\phi (u\circ v)-\phi \left(u\right)|]\\ & \le 2\mathbb{P}[{\tau }_{K+2}^{\ast }<T{]}^{1/2}\underset{u\in \mathcal{U}}{sup}\mathbb{E}[|\phi \left(u\right){|}^2{]}^{1/2}+\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}},v\in {\mathcal{U}}_T}{sup}\mathbb{E}\left[|\phi \right(u\circ v)-\phi (u\left)|\right].\end{array}$where we have used Hölder's inequality to arrive at the last inequality. As $(\phi ,c)$ is an admissible reward pair, Definition 2.2.(iii) gives that the second term can be made arbitrarily small by choosing T sufficiently large and Definition 2.2.(i) implies that the first term tends to zero as $K\to \mathrm{\infty }$ for all finite T, since $u^{\ast }\in {\mathcal{U}}^f$. We, thus, conclude that the first term on the right-hand side in (14(14) $\begin{array}{rl}|Y_0-J\left(u^{\ast }\right)|& \le \mathbb{E}[|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]+\mathbb{E}[\sum _{j=K+2}^{N^{\ast }}c(\left[u^{\ast }{]}_j\right)]\\ & +|\mathbb{E}[{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]|\end{array}$(14) ) tends to zero as $K\to \mathrm{\infty }$.

For the second term we note that letting $K\to \mathrm{\infty }$ in (13(13) $\begin{array}{r}{Y}_0=\mathbb{E}[{\mathbb{1}}_{[N^{\ast }\le K]}\phi \left(u^{\ast }\right)-\sum _{j=1}^{N^{\ast }\wedge K}c\left(\right[u^{\ast }{]}_j)+{\mathbb{1}}_{[N^{\ast }>K]}\{-c\left(\right[u^{\ast }{]}_{K+1})+Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}\}].\end{array}$(13) ) and using Property (b) and Definition 2.2.(i.a) we find that $\sum _{j=1}^{N^{\ast }\wedge l}c\left(\right[u^{\ast }{]}_j)$ converges increasingly to a limit in $L^2(\mathrm{\Omega },\mathbb{P})$ as $l\to \mathrm{\infty }$. Hence, the second term on the right-hand side of (14(14) $\begin{array}{rl}|Y_0-J\left(u^{\ast }\right)|& \le \mathbb{E}[|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]+\mathbb{E}[\sum _{j=K+2}^{N^{\ast }}c(\left[u^{\ast }{]}_j\right)]\\ & +|\mathbb{E}[{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]|\end{array}$(14) ) also tends to zero as $K\to \mathrm{\infty }$.

Conditioning on ${\mathcal{F}}_{{\tau }_{K+1}^{\ast }}$ in the third term of the right-hand side of (14(14) $\begin{array}{rl}|Y_0-J\left(u^{\ast }\right)|& \le \mathbb{E}[|\phi \left(\right[u^{\ast }{]}_{K+1})-\phi (u^{\ast })|]+\mathbb{E}[\sum _{j=K+2}^{N^{\ast }}c(\left[u^{\ast }{]}_j\right)]\\ & +|\mathbb{E}[{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]|\end{array}$(14) ) and noting that $\{N^{\ast }>K\}$ is ${\mathcal{F}}_{{\tau }_{K+1}^{\ast }}$-measurable⁶ we find that, similar to the above case, we have for any $T\ge 0$ that $\begin{array}{rl}|\mathbb{E}[{\mathbb{1}}_{[N^{\ast }>K]}\{Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\phi (\left[u^{\ast }{]}_{K+1}\right)\}]|& \le \mathbb{E}[|Y_{{\tau }_{K+1}^{\ast }}^{[u^{\ast }{]}_{K+1}}-\mathbb{E}\left[\phi \right(\left[u^{\ast }{]}_{K+1}\right)|{\mathcal{F}}_{{\tau }_{K+1}^{\ast }}]|]\\ & \le C\mathbb{P}[{\tau }_{K+1}^{\ast }<T{]}^{1/2}\\ & +\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{s\in [T,\mathrm{\infty }]}{sup}|Y_s^u-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_s\left]|\right],\end{array}$where the second term can be made arbitrarily small by Property  (d) and we conclude that $Y_0=J\left(u^{\ast }\right)$.

Step 3 It remains to show that the strategy $u^{\ast }$ is optimal. To do this we pick any other strategy $\hat{u}:=({\hat{\tau }}_1,\dots ,{\hat{\tau }}_{\hat{N}};{\widehat{\text{β}}}_1,\dots ,{\widehat{\text{β}}}_{\hat{N}})\in {\mathcal{U}}^f$. By Step 2 and the definition of $Y_0$ in (7(7) $\begin{array}{r}{Y}_s^v=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b)}\}|{\mathcal{F}}_s].\end{array}$(7) ) we have $\begin{array}{rl}J\left(u^{\ast }\right)=Y_0& \ge \mathbb{E}[{\mathbb{1}}_{[\hat{N}=0]}\phi \left(\mathrm{\varnothing }\right)+{\mathbb{1}}_{[\hat{N}>0]}\underset{b\in U}{sup}\{-c_{{\hat{\tau }}_1,b}+Y_{{\hat{\tau }}_1}^{{\hat{\tau }}_1;b}\}]\\ & \ge \mathbb{E}[{\mathbb{1}}_{[\hat{N}=0]}\phi \left(\mathrm{\varnothing }\right)+{\mathbb{1}}_{[\hat{N}>0]}\{-c({\hat{\tau }}_1;{\widehat{\text{β}}}_1)+Y_{{\hat{\tau }}_1}^{{\hat{\tau }}_1;{\widehat{\text{β}}}_1}\}]\end{array}$but in the same way $\begin{array}{r}{Y}_{{\hat{\tau }}_1}^{{\hat{\tau }}_1,{\widehat{\text{β}}}_1}\ge \mathbb{E}[{\mathbb{1}}_{[\hat{N}=1]}\phi ({\hat{\tau }}_1,{\widehat{\text{β}}}_1)+{\mathbb{1}}_{[\hat{N}>1]}\{-c({\hat{\tau }}_1,{\hat{\tau }}_2;{\widehat{\text{β}}}_1,{\widehat{\text{β}}}_2)+Y_{{\hat{\tau }}_2}^{{\hat{\tau }}_1,{\hat{\tau }}_2;{\widehat{\text{β}}}_1,{\widehat{\text{β}}}_2}\}|{\mathcal{F}}_{{\hat{\tau }}_1}],\end{array}$$\mathbb{P}$ -a.s. Repeating this procedure K times gives $\begin{array}{r}J\left(u^{\ast }\right)\ge {\hat{Y}}_K:=\mathbb{E}[{\mathbb{1}}_{[\hat{N}\le K]}\phi \left(\hat{u}\right)-\sum _{j=1}^{\hat{N}\wedge K}c\left(\right[\hat{u}{]}_j)+{\mathbb{1}}_{[\hat{N}>K]}\{-c\left(\right[\hat{u}{]}_{K+1})+Y_{{\hat{\tau }}_{K+1}}^{[\hat{u}{]}_{K+1}}\}].\end{array}$Now, we have $\begin{array}{rl}J\left(\hat{u}\right)-{\hat{Y}}_K& \le \mathbb{E}[{\mathbb{1}}_{[\hat{N}>K]}\left\{\phi \right(\hat{u})-Y_{{\hat{\tau }}_{K+1}}^{[\hat{u}{]}_{K+1}}\}]\\ & =\mathbb{E}[{\mathbb{1}}_{[\hat{N}>K]}\left\{\phi \right(\left[\hat{u}{]}_{K+1}\right)-Y_{{\hat{\tau }}_{K+1}}^{[\hat{u}{]}_{K+1}}\}]+\mathbb{E}[{\mathbb{1}}_{[\hat{N}>K]}\{\phi \left(\hat{u}\right)-\phi \left(\right[\hat{u}{]}_{K+1}\left)\right\}]\\ & \le \mathbb{E}[|\mathbb{E}[\phi \left(\right[\hat{u}{]}_{K+1})|{\mathcal{F}}_{{\hat{\tau }}_{K+1}}]-Y_{{\hat{\tau }}_{K+1}}^{[\hat{u}{]}_{K+1}}|]+\mathbb{E}[|\phi (\hat{u})-\phi (\left[\hat{u}{]}_{K+1}\right)|]\end{array}$where the right-hand side tends to zero as $K\to \mathrm{\infty }$ by repeating the argument in Step 2, which is possible since $\hat{u}\in {\mathcal{U}}^f$. We conclude that $J\left(u^{\ast }\right)\ge J\left(\hat{u}\right)$ for all $\hat{u}\in {\mathcal{U}}^f$ and it follows by Proposition 2.3 that $u^{\ast }$ is an optimal control for Problem 1.

4. Existence of the verification family

Theorem 3.3 presumes existence of the verification family $\left(\right(Y_s^v{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$. To obtain a satisfactory solution to Problem 1, we thus need to establish that a verification family exists. This is the topic of the present section. We will follow the standard existence proof which goes by applying a Picard iteration (see [5,11,15]). We first show that there exists a sequence of consistent families of processes $\left(\right(Y_s^{v,k}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{k\ge 0}$ that satisfy the recursion (15) $\begin{array}{r}{Y}_s^{v,0}:=\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_s]\end{array}$(15) and (16) $\begin{array}{r}{Y}_s^{v,k}:=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k-1}\}|{\mathcal{F}}_s]\end{array}$(16) for $k\ge 1$. Then, we show that the limit family obtained by letting $k\to \mathrm{\infty }$ is a verification family.

Proposition 4.1

There is a sequence of consistent families of càdlàg  supermartingales $\left(\right(Y_s^{v,k}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{k\ge 0}$ such that for each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$:

1. The sequence satisfies the recursion (15(15) $\begin{array}{r}{Y}_s^{v,0}:=\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_s]\end{array}$(15) )–(16(16) $\begin{array}{r}{Y}_s^{v,k}:=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k-1}\}|{\mathcal{F}}_s]\end{array}$(16) ).

2. There is a K>0 (that does not depend on k) such that, $\begin{array}{r}\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{s\in [0,\mathrm{\infty }]}{sup}|Y_s^{u,k+1}{|}^2]\le K.\end{array}$

3. For each $k\ge 0$, the map $(t,b)\mapsto Y_t^{v\circ (t,b),k}$ belongs to ${\mathcal{H}}_{\mathbb{F}}$.

4. $\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{t\in [T,\mathrm{\infty }]}{sup}|Y_t^{u,k}-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_t\left]|\right]\to 0$, as $T\to \mathrm{\infty }$, uniformly in k.

The proof of Proposition 4.1 will be based on two lemmas and the following induction hypothesis:

Hypthesis (VF.k). There is a sequence of consistent families of càdlàg  supermartingales $\left(\right(Y_s^{v,k^{\prime }}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{0\le k^{\prime }\le k}$ such that for $k^{\prime }=0,\dots ,k$ and $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$:

1. The relation (15(15) $\begin{array}{r}{Y}_s^{v,0}:=\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_s]\end{array}$(15) ) holds for $k^{\prime }=0$ and (16(16) $\begin{array}{r}{Y}_s^{v,k}:=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k-1}\}|{\mathcal{F}}_s]\end{array}$(16) ) holds for $k^{\prime }>0$.

2. The map $(t,b)\mapsto Y_t^{v\circ (t,b),k^{\prime }}$ belongs to ${\mathcal{H}}_{\mathbb{F}}$.

We note that Hypothesis VF.k lacks properties (b) and (d) of Proposition 4.1. In the following two propositions we show that these are implicit.

Lemma 4.2

Assume that Hypothesis VF.k holds for some $k\ge 0$. Then, the sequence of families of processes $\left(\right(Y_s^{v,k^{\prime }}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{0\le k^{\prime }\le k}$ is well defined and uniformly bounded in the sense that there is a K>0 (that does not depend on k) such that, $\begin{array}{r}\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{s\in [0,\mathrm{\infty }]}{sup}|Y_s^{u,k+1}{|}^2]\le K.\end{array}$Furthermore, for each $v\in {\mathcal{D}}^f$ (whenever it is well defined) the collection $\left\{\underset{b\in U}{sup}\right\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k}\}:\tau \in {\mathcal{T}}^f,k\ge 0\}$ of random variables is uniformly integrable.

Proof.

We note that under Hypothesis VF.k, the sequence of families $\left(\right(Y_s^{v,k^{\prime }}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{0\le k^{\prime }\le k+1}$ exists and is uniquely defined up to indistinguishability for each $Y^{v,k^{\prime }}$ by repeated application of Theorem A.1 in Appendix 2. By the definition of $Y^{v,k+1}$ and positivity of the intervention costs we have that for any $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, $\begin{array}{r}\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_s]\le Y_s^{v,k+1}\le \underset{u\in {\mathcal{U}}_s^{k+1}}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)|{\mathcal{F}}_s].\end{array}$For $k\ge 0$, we define the càdlàg  supermartingale $Z_s^k:={\mathrm{e}\mathrm{s}\mathrm{s}sup}_{u\in {\mathcal{U}}_s^k}\mathbb{E}\left[\phi \right(v\circ u\left)|{\mathcal{F}}_s\right]$ and the stopping times ${\tau }^{z,k}:=inf\{s\ge 0:|Z_s^k{|}^p\ge z\}$ for all $z\ge 0$. Then, by Definition 2.2.(i.a) we have $\begin{array}{rl}\mathbb{E}[|Z_{{\tau }^{z,k}}^k{|}^p]& \le \mathbb{E}[\underset{u\in {\mathcal{U}}_{{\tau }^{z,k}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ u){|}^p|{\mathcal{F}}_{{\tau }^{z,k}}]]\\ & \le \underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[|\phi \left(u\right){|}^p]\le C.\end{array}$In particular, by right-continuity this implies that $\begin{array}{r}\mathbb{P}[\underset{s\in [0,\mathrm{\infty })}{sup}|Z_s^k{|}^p\ge z]\le \frac{C}{z}\wedge 1\end{array}$or $\begin{array}{r}\mathbb{P}[\underset{s\in [0,\mathrm{\infty })}{sup}|Z_s^k{|}^2\ge z]\le \frac{C}{z^{p/2}}\wedge 1,\end{array}$where C does not depend on v and k. The first assertion now follows as $\begin{array}{r}\mathbb{E}[\underset{s\in [0,\mathrm{\infty })}{sup}|Z_s^k{|}^2]\le {\int }_0^{\mathrm{\infty }}(\frac{C}{z^{p/2}}\wedge 1)\mathrm{d}z<\mathrm{\infty }.\end{array}$Concerning the second claim, note that for each $\tau \in {\mathcal{T}}^f$ and each $ϵ>0$, repeating the proof of Corollary A.5 in Appendix 3 with $g^{ϵ}:=g-ϵ$ instead of g we find that there is a ${\text{β}}^{ϵ}\in \mathcal{A}\left(\tau \right)$ such that $\begin{array}{r}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k}\}\le -c_{\tau ,{\text{β}}^{ϵ}}^v+Y_{\tau }^{v\circ (\tau ,{\text{β}}^{ϵ}),k}+ϵ.\end{array}$Now, $\begin{array}{rl}\mathbb{E}[\underset{b\in U}{sup}|-c_{\tau ,b}^v+Y_{\tau }^{v\circ (\tau ,b),k}{|}^2]& \le 2\mathbb{E}[|-c_{\tau ,{\text{β}}^{ϵ}}^v+Y_{\tau }^{v\circ (\tau ,{\text{β}}^{ϵ}),k}{|}^2]+2{ϵ}^2\\ & \le 4\mathbb{E}[|c(v\circ (\tau ,{\text{β}}^{ϵ}\left)\right){|}^2]+4\mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}|Y_t^{v\circ (\tau ,{\text{β}}^{ϵ}),k}{|}^2]+2{ϵ}^2\end{array}$where the right-hand side is bounded, uniformly in $(\tau ,{\text{β}}^{ϵ})$ and $k\ge 0$, by the above in combination with Definition 2.2.(i.b) and Doob's maximal inequality.

We also have the following diminishing future impact property:

Lemma 4.3

Assume again that Hypothesis VF.k holds for some $k\ge 0$. Then, $\begin{array}{r}\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{t\in [T,\mathrm{\infty }]}{sup}|Y_t^{u,k+1}-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_t]|]\to 0,\end{array}$as $T\to \mathrm{\infty }$, uniformly in k.

Proof.

By the properties of the essential supremum and positivity of the intervention costs we have for every $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, $\begin{array}{r}\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_t]\le Y_t^{v,k+1}\le \underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)|{\mathcal{F}}_t].\end{array}$This implies that $\begin{array}{r}|Y_t^{v,k+1}-\mathbb{E}[\phi \left(v\right)|{\mathcal{F}}_t]|\le \underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ u)-\phi \left(v\right)||{\mathcal{F}}_t].\end{array}$The desired result now follows by a similar argument to the one used in the proof of Lemma 4.2 and Definition 2.2.(iii).

Proof

Proof of Proposition 4.1

First, note that by Definition 2.2.(ii) there is a $h\in {\mathcal{H}}_{\mathbb{F}}$ such that for $\tau \in {\mathcal{T}}^f$ we have (17) $\begin{array}{r}h(\tau ,\text{β})=\mathbb{E}[-c(v\circ (\tau ,\text{β}\left)\right)+\phi (v\circ (\tau ,\text{β}\left)\right)|{\mathcal{F}}_{\tau }],\end{array}$(17) for all $\text{β}\in \mathcal{A}\left(\tau \right)$, $\mathbb{P}$-a.s. The statement, thus, holds for k=0.

Moving on we assume that VF.k holds for some $k\ge 0$. But then, by Lemmas 4.2 and 4.3 we can applying a reasoning similar to that in the proof of Theorem 3.3 to find that $Y^{v,k+1}$ is a càdlàg  supermartingale with (18) $\begin{array}{r}{Y}_t^{v,k+1}=\underset{u\in {\mathcal{U}}_t^{k+1}}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)-\sum _{j=1}^{N}c(v\circ [u{]}_j)|{\mathcal{F}}_t].\end{array}$(18) By Definition 2.2.(ii) it follows that there is a consistent family satisfying (18(18) $\begin{array}{r}{Y}_t^{v,k+1}=\underset{u\in {\mathcal{U}}_t^{k+1}}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)-\sum _{j=1}^{N}c(v\circ [u{]}_j)|{\mathcal{F}}_t].\end{array}$(18) ) such that $(t,b)\mapsto Y_t^{v\circ (t,b),k+1}\in {\mathcal{H}}_{\mathbb{F}}$ and we conclude that VF.k+1 holds as well. By induction this extends to all $k\ge 0$.

The objective in the remainder of this section is to show that the limit family that we get when letting $k\to \mathrm{\infty }$ in $\left(\right(Y_s^{v,k}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ is a verification family.

Proposition 4.4

For each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, the limit ${\overline{Y}}^v:=\underset{k\to \mathrm{\infty }}{lim}{Y}^{v,k}$, exists as an increasing pointwise limit, $\mathbb{P}$-a.s.

Proof.

Since ${\mathcal{U}}_t^k\subset {\mathcal{U}}_t^{k+1}$ we have that $Y_t^{v,k}\le Y_t^{v,k+1}$, $\mathbb{P}$-a.s. Moreover, by Proposition 4.1 the sequence is bounded $\mathbb{P}$-a.s., thus, it converges $\mathbb{P}$-a.s. for all $t\in [0,\mathrm{\infty }]$.

To assess the type of convergence that we have for the sequence $Y^{v,k}$, we introduce a sequence of families of processes corresponding to a truncation of the time interval. For each T>0 and $k\ge 0$, we define the consistent family $({(}^T{Y}_t^{v,k}{)}_{t\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ of càdlàg  supermartingales as $\begin{array}{r}{}^T{Y}_t^{v,k}=\underset{u\in {\mathcal{U}}_{[t,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ u)-\sum _{j=1}^{N}c(v\circ [u{]}_j)|{\mathcal{F}}_t]\end{array}$for all $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ with $(t,b)\mapsto {}^T{Y}_t^{v\circ (t,b),k}\in {\mathcal{H}}_{\mathbb{F}}^{\prime }$. Then,

Lemma 4.5

The sequence $({(}^T{Y}_s^{v,k}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}{)}_{k\ge 0}$ satisfies:

1. $Y^{v,0}\le {}^T{Y}^{v,k}\le Y^{v,k}$.

2. For each $ϵ>0$ there is a $T\ge 0$ such that $\mathbb{P}\left[{\cup }_{k=0}^{\mathrm{\infty }}{B}_k^{T,ϵ}\right]<ϵ$, with $\begin{array}{r}{B}_k^{T,ϵ}:=\{\omega \in \mathrm{\Omega }:\underset{s\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|Y_s^{v\circ (s,b),k}-{}^T{Y}_s^{v\circ (s,b),k}|>ϵ\}.\end{array}$

3. There is a $\mathbb{P}$-a.s. finite $\mathcal{F}$-measurable random variable ξ and a constant q>0 such that $\begin{array}{r}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}{|}^T{Y}_t^{v\circ (t,b),k}-{}^T{Y}_t^{v\circ (t,b),k^{\prime }}|\le \xi /(k^{\prime }{)}^q\end{array}$$\mathbb{P}$-a.s. for each $0<k^{\prime }\le k$.

Proof.

The inequality in (i) follows from noting that $\mathrm{\varnothing }\in {\mathcal{U}}_{[t,T)}^k\subset {\mathcal{U}}_t^k$.

For the second statement we note that by Lemma 2.4.(c), the process $(\underset{b\in U}{sup}|Y_s^{v\circ (s,b),k}-{}^T{Y}_s^{v\circ (s,b),k}|:s\ge 0)$ is ${\mathcal{P}}_{\mathbb{F}}$-measurable and càdlàg.  Now, each $u\in {\mathcal{U}}_{\tau }^k$ can be decomposed as $u=u_1\circ u_2$ with $u_1\in {\mathcal{U}}_{[\tau ,T)}^k$ and $u_2\in {\mathcal{U}}_T^k$, which implies that

(19) $\begin{array}{rl}{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}& -{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}=\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & -\underset{u\in {\mathcal{U}}_{[\tau ,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & \le \underset{u_1\in {\mathcal{U}}_{[\tau ,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u_1\circ u_2)-\sum _{j=1}^{N_1+N_2}c(v\circ (\tau ,\text{β})\circ [u_1\circ u_2{]}_j)|{\mathcal{F}}_{\tau }]\\ & -\underset{u\in {\mathcal{U}}_{[\tau ,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & \le \underset{u_1\in {\mathcal{U}}_{[\tau ,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (\tau ,\text{β})\circ u_1\circ u_2)-\phi (v\circ (\tau ,\text{β})\circ u_1)||{\mathcal{F}}_{\tau }],\end{array}$(19) where $N_1$ and $N_2$ are the number of interventions in $u_1$ and $u_2$, respectively. We thus define the sets $\begin{array}{rl}{\tilde{B}}_k^{T,ϵ}& :=\{\omega \in \mathrm{\Omega }:\underset{s\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}\underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (s,b)\circ u_1\circ u_2)\\ & -\phi (v\circ (s,b)\circ u_1)||{\mathcal{F}}_s]\ge ϵ\}\end{array}$and have by (19(19) $\begin{array}{rl}{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}& -{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}=\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & -\underset{u\in {\mathcal{U}}_{[\tau ,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & \le \underset{u_1\in {\mathcal{U}}_{[\tau ,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u_1\circ u_2)-\sum _{j=1}^{N_1+N_2}c(v\circ (\tau ,\text{β})\circ [u_1\circ u_2{]}_j)|{\mathcal{F}}_{\tau }]\\ & -\underset{u\in {\mathcal{U}}_{[\tau ,T)}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u)-\sum _{j=1}^{N}c(v\circ (\tau ,\text{β})\circ [u{]}_j)|{\mathcal{F}}_{\tau }]\\ & \le \underset{u_1\in {\mathcal{U}}_{[\tau ,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (\tau ,\text{β})\circ u_1\circ u_2)-\phi (v\circ (\tau ,\text{β})\circ u_1)||{\mathcal{F}}_{\tau }],\end{array}$(19) ) that $B_k^{T,ϵ}\subset {\tilde{B}}_k^{T,ϵ}$ for $k\ge 0$. Furthermore, as ${\mathcal{U}}_{[\tau ,T)}^k\subset {\mathcal{U}}_{[\tau ,T)}^{k+1}$ and ${\mathcal{U}}_T^k\subset {\mathcal{U}}_T^{k+1}$ we find that ${\tilde{B}}_k^{T,ϵ}\subset {\tilde{B}}_{k+1}^{T,ϵ}$ for all $k\ge 0$.

Now, let $\begin{array}{rl}{\tau }_k^{ϵ}& :=inf\{s\ge 0:\underset{b\in U}{sup}\underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (s,b)\circ u_1\circ u_2)\\ & -\phi (v\circ (s,b)\circ u_1)||{\mathcal{F}}_s]\ge ϵ\}\end{array}$(recalling our convention that $inf\mathrm{\varnothing }=\mathrm{\infty })$ and pick ${\text{β}}_k^{ϵ}$ such that $\begin{array}{rl}& \underset{b\in U}{sup}\underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ ({\tau }_k^{ϵ},b)\circ u_1\circ u_2)-\phi (v\circ ({\tau }_k^{ϵ},b)\circ u_1)||{\mathcal{F}}_{{\tau }_k^{ϵ}}]\\ & \le \underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1\circ u_2)-\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1)||{\mathcal{F}}_{{\tau }_k^{ϵ}}]+ϵ/2.\end{array}$Then, by right continuity we have ${\tilde{B}}_k^{T,ϵ}\subset {\hat{B}}_k^{T,ϵ}$ where $\begin{array}{rl}{\hat{B}}_k^{T,ϵ}& :=\{\omega \in \mathrm{\Omega }:\underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1\circ u_2)\\ & -\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1)||{\mathcal{F}}_{{\tau }_k^{ϵ}}]\ge ϵ/2\}.\end{array}$We thus only need to show that there is a T>0 such that $\mathbb{P}\left[{\hat{B}}_k^{T,ϵ}\right]<ϵ$ for all $k\ge 0$. We have, $\begin{array}{rl}& \mathbb{E}[\underset{u_1\in {\mathcal{U}}_{[0,T)}^k,u_2\in {\mathcal{U}}_T^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1\circ u_2)-\phi (v\circ ({\tau }_k^{ϵ},{\text{β}}_k^{ϵ})\circ u_1)||{\mathcal{F}}_{{\tau }_k^{ϵ}}]]\\ & \le \underset{u_1\in {\mathcal{U}}_{[0,T)}^{\mathrm{l}\mathrm{i}\mathrm{m}},u_2\in {\mathcal{U}}_T^f}{sup}\mathbb{E}[|\phi (u_1\circ u_2)-\phi \left(u_1\right)||],\end{array}$where right-hand side is independent of k and tends to 0 as $T\to \mathrm{\infty }$ by Definition 2.2.(iii). We thus conclude that there is a $T=T\left(ϵ\right)$ such that $\mathbb{P}\left[{\hat{B}}_k^{T,ϵ}\right]<ϵ$ for all $k\ge 0$.

Concerning the third statement, we note that for $r\in (1,2)$, we have for each $\tau \in \mathcal{T}$ and all $\text{β}\in \mathcal{A}\left(\tau \right)$, that $\begin{array}{rl}{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}& \le \underset{s\in [0,\mathrm{\infty }]}{sup}{Y}_s^{v\circ (\tau ,\text{β}),k}\\ & \le \underset{s\in [0,\mathrm{\infty }]}{sup}\underset{u\in {\mathcal{U}}_s^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}\left[|\phi \right(v\circ (\tau ,\text{β})\circ u\left)||{\mathcal{F}}_s\right]\\ & \le 1+\underset{s\in [0,\mathrm{\infty }]}{sup}\underset{u\in {\mathcal{U}}_s^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}\left[|\phi \right(v\circ (\tau ,\text{β})\circ u\left){|}^r|{\mathcal{F}}_s\right]=:K\left(\omega \right)\end{array}$and similarly $\begin{array}{r}{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}\ge -K\left(\omega \right)\end{array}$for all $k\ge 0$ (where the inequalities hold $\mathbb{P}$-a.s.). Now, arguing as in the proof of Proposition 4.2 we have $\begin{array}{r}\mathbb{E}[\underset{s\in [0,\mathrm{\infty }]}{sup}\underset{u\in {\mathcal{U}}_s^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}\left[|\phi \right(v\circ (\tau ,\text{β})\circ u){|}^r|{\mathcal{F}}_s{]}^{2/r}]<\mathrm{\infty }\end{array}$and we conclude that there is a $\mathbb{P}$-null set $\mathcal{N}$ such that for each $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$ we have $K\left(\omega \right)<\mathrm{\infty }$.

For $ϵ>0$, let $u^{k,ϵ}:=({\tau }_1^{k,ϵ},\dots ,{\tau }_{N^{k,ϵ}}^{k,ϵ};{\text{β}}_1^{k,ϵ},\dots ,{\text{β}}_{N^{k,ϵ}}^{k,ϵ})\in {\mathcal{U}}_{[\tau ,T)}^k$ be such that $\begin{array}{r}{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}\le \mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u^{k,ϵ})-\sum _{j=1}^{N^{k,ϵ}}c(v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_j)|{\mathcal{F}}_{\tau }]+ϵ.\end{array}$We note that on $[\tau \ge T]$ we have $N^{k,ϵ}=0$ and get that for $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$ (in the remainder of the proof $\mathcal{N}$ denotes a generic $\mathbb{P}$-null set), we have $\begin{array}{rl}-K-ϵ& \le \mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ u^{k,ϵ})-\sum _{j=1}^{N^{k,ϵ}}c(v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_j)|{\mathcal{F}}_{\tau }]\\ & \le K-\delta \left(T\right)\mathbb{E}[N^{k,ϵ}|{\mathcal{F}}_{\tau }]\end{array}$and we conclude that $\mathbb{E}\left[{\mathbb{1}}_{[N^{k,ϵ}>k^{\prime }]}|{\mathcal{F}}_{\tau }\right]<(2K+ϵ)/\left(\delta \right(T\left)k^{\prime }\right)$ for all $k^{\prime }\ge 0$.

Now, for all $0\le k^{\prime }\le k$ we have, $\begin{array}{rl}{}^T{\breve{Y}}_{\tau }^{v\circ (\tau ,\text{β}),k,k^{\prime }}& :=\mathbb{E}[\phi (v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_{k^{\prime }})-\sum _{j=1}^{N^{k,ϵ}\wedge k^{\prime }}c(v\circ (\tau ,\text{β})\circ \left[u^{k,ϵ}{]}_j\right)|{\mathcal{F}}_{\tau }]\\ & \le {}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k^{\prime }}\le {}^T{Y}_{\tau }^{v\circ (s,b),k},\end{array}$where we have introduced ${}^T{\breve{Y}}_{\tau }^{v\circ (\tau ,\text{β}),k,k^{\prime }}$ corresponding to the truncation $[u^{k,ϵ}{]}_{k^{\prime }}:=$

$({\tau }_1^{k,ϵ},\dots ,{\tau }_{N^{k,ϵ}\wedge k^{\prime }}^{k,ϵ};{\text{β}}_1^{k,ϵ},\dots ,{\text{β}}_{N^{k,ϵ}\wedge k^{\prime }}^{k,ϵ})$ of $u^{k,ϵ}$. As the truncation only affects the performance of the controller when $N^{k,ϵ}>k^{\prime }$ we have $\begin{array}{rl}{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}-{}^T{\breve{Y}}_{\tau }^{v\circ (\tau ,\text{β}),k,k^{\prime }}& \le \mathbb{E}[{\mathbb{1}}_{[N^{k,ϵ}>k^{\prime }]}(\phi (v\circ u^{k,ϵ})-\sum _{j=k^{\prime }+1}^{N^{k,ϵ}}c(v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_j)\\ & -\phi (v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_{k^{\prime }}))|{\mathcal{F}}_{\tau }]+ϵ\\ & \le \mathbb{E}[{\mathbb{1}}_{[N^{k,ϵ}>k^{\prime }]}(\phi (v\circ (\tau ,\text{β})\circ u^{k,ϵ})\\ & -\phi (v\circ (\tau ,\text{β})\circ [u^{k,ϵ}{]}_{k^{\prime }}))|{\mathcal{F}}_{\tau }]+ϵ.\end{array}$Applying Hölder's inequality we get that for $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$, $\begin{array}{rl}{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}-{}^T{\breve{Y}}_{\tau }^{v\circ (\tau ,\text{β}),k,k^{\prime }}& \le 2\mathbb{E}[{\mathbb{1}}_{[N^{k,ϵ}>k^{\prime }]}|{\mathcal{F}}_{\tau }{]}^{1/q}\underset{u\in {\mathcal{U}}_{\tau }^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (\tau ,\text{β})\circ u){|}^r|{\mathcal{F}}_{\tau }{]}^{1/r}+ϵ\\ & \le 2^{1+\frac{1}{q}}\frac{K\left(\omega \right)+ϵ}{\left(\delta \right(T)k^{\prime }{)}^{1/q}}+ϵ,\end{array}$with $\frac{1}{r}+\frac{1}{q}=1$. Since $\delta \left(T\right)>0$, there is thus a $\mathbb{P}$-a.s. finite $\mathcal{F}$-measurable r.v. $\xi =\xi \left(\omega \right)$ such that (for all τ and β) we have $\begin{array}{r}|{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}-{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k^{\prime }}|\le \frac{\xi }{(k^{\prime }{)}^{1/q}}+Cϵ.\end{array}$Since, $\text{β}\in \mathcal{A}\left(\tau \right)$ was arbitrary we can choose β such that $\begin{array}{r}\underset{b\in U}{sup}|{}^T{Y}_{\tau }^{v\circ (\tau ,b),k}-{}^T{Y}_{\tau }^{v\circ (\tau ,b),k^{\prime }}|\le |{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k}-{}^T{Y}_{\tau }^{v\circ (\tau ,\text{β}),k^{\prime }}|+ϵ\le \frac{\xi }{(k^{\prime }{)}^{1/q}}+Cϵ,\end{array}$$\mathbb{P}$-a.s. and by right-continuity the last statement follows as $ϵ>0$ was arbitrary.

Proposition 4.6

For each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, we have $\begin{array}{r}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|Y_t^{v\circ (t,b),k}-{\overline{Y}}_t^{v\circ (t,b)}|\to 0\end{array}$as $k\to \mathrm{\infty }$, outside of a $\mathbb{P}$-null set.

Proof.

By Lemma 4.5.(ii) there exist for each $ϵ>0$, a $T\ge 0$ and a measurable set $B\subset \mathrm{\Omega }$ with $\mathbb{P}\left[B\right]\ge 1-ϵ$ such that $\begin{array}{r}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|Y_t^{v\circ (t,b),k}-Y_t^{v\circ (t,b),k^{\prime }}|\le \underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|{}^T{Y}_t^{v\circ (t,b),k}-{}^T{Y}_t^{v\circ (t,b),k^{\prime }}|+2ϵ\end{array}$for all $0\le k\le k^{\prime }$ and $\omega \in B$. Furthermore, by Lemma 4.5.(iii) there is a $\mathbb{P}$-a.s. finite r.v., ξ, such that $\begin{array}{r}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|{}^T{Y}_t^{v\circ (t,b),k}-{}^T{Y}_t^{v\circ (t,b),k^{\prime }}|\le \frac{\xi }{(k{)}^q}.\end{array}$Combining these and taking the limit as $k,k^{\prime }\to \mathrm{\infty }$ we find that $\begin{array}{r}\underset{k\to \mathrm{\infty }}{lim}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|{\overline{Y}}_t^{v\circ (t,b)}-Y_t^{v\circ (t,b),k}|\le \underset{k\to \mathrm{\infty }}{lim}\underset{k^{\prime }\to \mathrm{\infty }}{lim}\underset{t\in [0,\mathrm{\infty }]}{sup}\underset{b\in U}{sup}|Y_t^{v\circ (t,b),k^{\prime }}-Y_t^{v\circ (t,b),k}|\le 2ϵ\end{array}$on $B\setminus \mathcal{N}$ for some $\mathbb{P}$-null set $\mathcal{N}$. Now, as $ϵ>0$ was arbitrary the statement follows.

We are now ready to show that a verification family exists, establishing the existence of optimal controls for Problem 1.

Proposition 4.7

A verification family exists.

Proof.

Letting $h^k(t,b):=Y_t^{v\circ (t,b),k}$ we have by Proposition 4.6 that $h^k(t,b)$ converges uniformly in $(t,b)$ to $\overline{h}(t,b):={\overline{Y}}_t^{v\circ (t,b)}$ as $k\to \mathrm{\infty }$ (outside of a $\mathbb{P}$-null set). Since $h^k\in {\mathcal{H}}_{\mathbb{F}}$ by Proposition 4.1.c), we have by Lemma 2.4.(d) that $\overline{h}\in {\mathcal{H}}_{\mathbb{F}}$. In particular, we conclude that property c) in the definition of a verification family holds for the limit family $\left(\right({\overline{Y}}_s^v{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$.

Moreover, for each $\tau \in \mathcal{T}$ and $\text{β}\in \mathcal{A}\left(\tau \right)$ we have that $Y_{\tau }^{v\circ (\tau ,\text{β}),k}\nearrow {\overline{Y}}_{\tau }^{v\circ (\tau ,\text{β})}$, $\mathbb{P}$-a.s., as $k\to \mathrm{\infty }$ and we conclude by consistency of $\left(\right({\overline{Y}}_s^{v,k}{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$ for each $k\ge 0$ that ${\overline{Y}}_{\tau }^{v\circ (\tau ,\text{β})}=\overline{h}(\tau ,\text{β})$, $\mathbb{P}$-a.s., implying consistency of $\left(\right({\overline{Y}}_s^v{)}_{s\ge 0}:v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}})$.

We treat each of the remaining properties separately:

(a) By the above and (b) of Lemma 2.4 we have that $\underset{b\in U}{sup}\{-c_{s,b}^v+{\overline{Y}}_s^{v\circ (s,b)}\}$ is a càdlàg, quasi-left upper semi-continuous process of class [D]. In particular we note that $\begin{array}{r}{\mathbb{1}}_{[s=\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[s<\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{s,b}^v+{\overline{Y}}_s^{v\circ (s,b)}\}\end{array}$is càdlàg.  Applying (iv) of Theorem A.1 in Appendix 2 then gives $\begin{array}{r}{\overline{Y}}_s^v=\underset{\tau \in {\mathcal{T}}_s}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\tau =\mathrm{\infty }]}\phi \left(v\right)+{\mathbb{1}}_{[\tau <\mathrm{\infty }]}\underset{b\in U}{sup}\{-c_{\tau ,b}^v+{\overline{Y}}_{\tau }^{v\circ (\tau ,b)}\}|{\mathcal{F}}_s].\end{array}$(b) By Proposition 4.1 we have that $\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}\left[\underset{t\in [0,\mathrm{\infty }]}{sup}|Y_t^{u,k}{|}^2\right]$ is uniformly bounded in k. From this it follows immediately that $\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}\left[\underset{t\in [0,\mathrm{\infty }]}{sup}|{\overline{Y}}_t^u{|}^2\right]<\mathrm{\infty }$.

(d) We have that $\begin{array}{rl}\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{t\in [T,\mathrm{\infty }]}{sup}|{\overline{Y}}_t^u-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_t]|]& \le \underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{t\in [0,\mathrm{\infty }]}{sup}|{\overline{Y}}_t^u-Y_t^{u,k}|]\\ & +\underset{u\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}}{sup}\mathbb{E}[\underset{t\in [T,\mathrm{\infty }]}{sup}|Y_t^{u,k}-\mathbb{E}[\phi \left(u\right)|{\mathcal{F}}_t]|]\end{array}$where the first term on the right-hand side can be made arbitrarily small by choosing k sufficiently large and the last term tends to 0 as $T\to \mathrm{\infty }$ for all $k\ge 0$.

5. Application to impulse control of SFDEs

In [24] a finite horizon impulse control problem with a discrete set U was solved when the underlying process followed a stochastic delay differential equation (SDDE) under a loop condition on the impulses. This problem was motivated by hydropower operation where the flow-times between different power plants induce delays in the dynamics of the controlled system.

In this section we extend the results from [24] by considering a discounted infinite horizon setting, allowing an uncountable control set U and also by taking the dynamics of the underlying process to follow a stochastic functional differential equation. Furthermore, our prior treatment of the problem with abstract reward, φ, and intervention cost, c, allows us to consider a less restrictive set of assumptions on the coefficients in the problem formulation. In particular, we are able to remove the loop condition.

Our treatment of non-Markovian impulse control problems in infinite horizon should also be compared to [10] where an infinite horizon impulse control problem in a non-Markovian framework with a fixed discrete delay is considered. The work presented in this section goes in a different direction by having an underlying dynamics driven by a Lévy process that is affected by the impulses in the control, resulting in a more complex relation between the control and the output of the performance functional. Furthermore, we investigate the important extension to random horizon which turns out to be a trivial modification of our initial problem.

Throughout this section, we will only consider controls for which ${\tau }_j\to \mathrm{\infty }$, $\mathbb{P}$-a.s., and restrict our attention to the setting when the underlying uncertainty stems from a process $X^u$, with $u\in {\mathcal{U}}^f$, defined as $X^u:=\underset{k\to \mathrm{\infty }}{lim}{X}^{u,k}$ where (20) $\begin{array}{rl}{X}_t^{u,0}& =x\left(t\right)\mathrm{f}\mathrm{o}\mathrm{r}t\in (-\mathrm{\infty },0)\end{array}$(20) (21) $\begin{array}{rl}{X}_t^{u,0}& =x\left(0\right)+{\int }_0^{t}a(s,(X_r^{u,0}{)}_{r\le s})\mathrm{d}s+{\int }_0^t\sigma (s,\left(X_r^{u,0}{)}_{r\le s}\right)d{B}_s\\ & +{\int }_0^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,0}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge 0\end{array}$(21) for some $x\in \mathbb{D}$, the set of all (deterministic) uniformly bounded, càdlàg  functions $x:\mathbb{R}\to {\mathbb{R}}^d$, and (22) $\begin{array}{rl}{X}_t^{u,k}& =X_t^{u,k-1}\mathrm{f}\mathrm{o}\mathrm{r}t\in [0,{\tau }_k)\end{array}$(22) (23) $\begin{array}{rl}{X}_t^{u,k}& =\mathrm{\Gamma }({\tau }_k,(X_s^{u,k-1}{)}_{s\le {\tau }_k},{\text{β}}_k)+{\int }_{{\tau }_k}^{t}a(s,\left(X_r^{u,k}{)}_{r\le s}\right)\mathrm{d}s+{\int }_{{\tau }_k}^t\sigma (s,(X_r^{u,k}{)}_{r\le s})d{B}_s\\ & +{\int }_{{\tau }_k+}^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,k}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge {\tau }_k.\end{array}$(23) The dynamics of $X^u$ are driven by a d-dimensional Brownian motion B and a Poisson random measure P with intensity measure $\varrho (\mathrm{d}s;\mathrm{d}z)=\mathrm{d}s\times \mu \left(\mathrm{d}z\right)$, where $\mu \left(\mathrm{d}z\right)$ is the Lévy measure on ${\mathbb{R}}^d$ of P and $\tilde{P}(\mathrm{d}s;\mathrm{d}z):=(P-\varrho )(\mathrm{d}s;\mathrm{d}z)$ is called the compensated jump martingale random measure of P. We assume that $\mathbb{G}:=\{{\mathcal{G}}_t{\}}_{t\ge 0}$ is the natural filtration generated by B and P, with $\mathcal{G}={\mathcal{G}}_{\mathrm{\infty }}:=\underset{t\to \mathrm{\infty }}{lim}{\mathcal{G}}_t$.

As mentioned above, we assume that all uncertainty comes from the process $X^u$ and consider the discounted setting with a continuous discount factor $\rho :{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$. The reward functional is then (24) $\begin{array}{r}J\left(u\right)=\mathbb{E}[{\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t-\sum _{j=1}^N{e}^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{u,j-1},{\text{β}}_j)].\end{array}$(24)

5.1. Assumptions

We assume that the involved coefficients satisfy the following constraints:

Assumption 5.1

For any $t,t^{\prime }\ge 0$, $b,b^{\prime }\in U$, $x,x^{\prime }\in {\mathbb{R}}^d$ and $y,y^{\prime }\in \mathbb{D}$ and for some $q\ge 2$ and p>2 we have:

1. The function $\mathrm{\Gamma }:{\mathbb{R}}_{+}\times \mathbb{D}\times U\to {\mathbb{R}}^d$ satisfies the Lipschitz condition $\begin{array}{rl}|\mathrm{\Gamma }(t,(y_s{)}_{s\le t},b)-\mathrm{\Gamma }(t^{\prime },(y_s^{\prime }{)}_{s\le t^{\prime }},b^{\prime })|& \le C({\int }_{-\mathrm{\infty }}^{t\wedge t^{\prime }}|y_s^{\prime }-y_s|\mathrm{d}s+|y_{t^{\prime }}^{\prime }-y_t|\\ & +(|t^{\prime }-t|+|b^{\prime }-b|)(1+\underset{s\le t}{sup}|y_s|+\underset{s\le t^{\prime }}{sup}|y_s^{\prime }|))\end{array}$and the growth condition $\begin{array}{r}|\mathrm{\Gamma }(t,(y_s{)}_{s\le t},b)|\le K_{\mathrm{\Gamma }}\vee |y_t|.\end{array}$for some constant $K_{\mathrm{\Gamma }}>0$.

2. The coefficients $a:{\mathbb{R}}_{+}\times \mathbb{D}\to {\mathbb{R}}^d$ and $\sigma :{\mathbb{R}}_{+}\times \mathbb{D}\to {\mathbb{R}}^{d\times d}$ are continuous in t and satisfy the growth condition $\begin{array}{r}|a(t,(y_s{)}_{s\le t})|+|\sigma (t,\left(y_s{)}_{s\le t}\right)|\le C(1+\underset{s\le t}{sup}|y_s|)\end{array}$and the Lipschitz continuity $\begin{array}{rl}|a(t,(y_s{)}_{s\le t})-a(t,\left(y_s^{\prime }{)}_{s\le t}\right)|+|\sigma (t,(y_s{)}_{s\le t})-\sigma (t,\left(y_s^{\prime }{)}_{s\le t}\right)|& \le C\underset{s\le t}{sup}|y_s^{\prime }-y_s|,\\ {\int }_0^t|a(s,(y_r{)}_{r\le s})-a(s,\left(y_r^{\prime }{)}_{r\le s}\right)|\mathrm{d}s& \le C{\int }_{-\mathrm{\infty }}^t|y_s^{\prime }-y_s|\mathrm{d}s\\ {\int }_0^t|\sigma (s,(y_r{)}_{r\le s})-\sigma (s,\left(y_r^{\prime }{)}_{r\le s}\right){|}^2\mathrm{d}s& \le C{\int }_{-\mathrm{\infty }}^t|y_s^{\prime }-y_s{|}^2\mathrm{d}s.\end{array}$

3. There is a $\overline{\gamma }:{\mathbb{R}}^d\to {\mathbb{R}}_{+}$, with $\int {\overline{\gamma }}^{pq}\left(z\right)\mu \left(\mathrm{d}z\right)<\mathrm{\infty }$ such that $\gamma :{\mathbb{R}}_{+}\times \mathbb{D}\times {\mathbb{R}}^d\to {\mathbb{R}}^d$ satisfies $\begin{array}{rl}|\gamma (t,(y_s{)}_{s<t},z)|& \le \overline{\gamma }\left(z\right)(1+\underset{s<t}{sup}|y_s|),\\ |\gamma (t,(y_s{)}_{s<t},z)-\gamma (t,\left(y_s^{\prime }{)}_{s<t}\right),z)|& \le \overline{\gamma }\left(z\right)\underset{s<t}{sup}|y_s-y_s^{\prime }|,\\ {\int }_0^t|\gamma (s,(y_r{)}_{r<s},z)-\gamma (s,(y_r^{\prime }{)}_{r<s},z){|}^{2(m+2)}\mathrm{d}s& \le {\overline{\gamma }}^{2(m+2)}\left(z\right){\int }_{-\mathrm{\infty }}^t|y_s^{\prime }-y_s{|}^{2(m+2)}\mathrm{d}s.\end{array}$

4. The running reward $\varphi :{\mathbb{R}}_{+}\times {\mathbb{R}}^d\to \mathbb{R}$ is $\mathcal{B}({\mathbb{R}}_{+}\times {\mathbb{R}}^d)$-measurable and satisfies the growth condition $\begin{array}{r}|\varphi (t,x)|\le C(1+|x{|}^q).\end{array}$Moreover, there is a non-decreasing function $C_{\varphi }:{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ such that for all $L\ge 0$, $\begin{array}{r}|\varphi (t,x)-\varphi (t,x^{\prime })|\le C_{\varphi }\left(L\right)|x-x^{\prime }|,\end{array}$whenever $|x|\vee |x^{\prime }|\le L$.

5. There is a finite collection of closed connected subsets $(U_i{)}_{i=1}^M$ of U and corresponding maps ${\ell }_i:{\mathbb{R}}_{+}\times {\mathbb{R}}^d\times U_i\to \mathbb{R}$ that are jointly continuous in $(t,x,b)$, bounded from below, i.e.  $\begin{array}{r}{\ell }_i(t,x,b)\ge \delta >0,\end{array}$of polynomial growth, $\begin{array}{r}|{\ell }_i(t,x,b)|\le C(1+|x{|}^q),\end{array}$and locally Lipschitz in x, i.e.  there is a non-decreasing function $C_{\ell }:{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ such that for all $L\ge 0$, $\begin{array}{r}|{\ell }_i(t,x,b)-{\ell }_i(t,x^{\prime },b)|\le C_{\ell }\left(L\right)|x-x^{\prime }|,\end{array}$whenever $|x|\vee |x^{\prime }|\le L$, and we have $\ell (t,x,b)=\underset{i:b\in U_i}{min}{\ell }_i(t,x,b)$.

Note that the growth condition on Γ in (i) implies that interventions can only increase the magnitude of the state $X_t$ as long as $|X_t|<K_{\mathrm{\Gamma }}$. In particular, this avoids the problem of explosions in a finite time due to impulses.

Remark 5.1

To see that the above SFDE is a generalization of discrete delay SDDEs with Lipschitz coefficients note that if $\chi :{\mathbb{R}}_{+}\times ({\mathbb{R}}^d{)}^{k+1}\to \mathbb{R}$ satisfies $\begin{array}{r}|\chi (t,x_1,\dots ,x_{k+1})-\chi (t,x_1,\dots ,x_{k+1})|\le C(|x_1-x_1^{\prime }|+\cdots +|x_{k+1}-x_{k+1}^{\prime }|)\end{array}$for each $(x_1,\dots ,x_{k+1}),(x_1^{\prime },\dots ,x_{k+1}^{\prime })\in ({\mathbb{R}}^d{)}^{k+1}$, then for $l\ge 0$ we have $\begin{array}{rl}& {\int }_0^t|\chi (s,y_s,y_{s-{\delta }_1},\dots ,y_{s-{\delta }_k})-\chi (s,y_s^{\prime },y_{s-{\delta }_1}^{\prime },\dots ,y_{s-{\delta }_k}^{\prime }){|}^l\mathrm{d}s\\ & \le C{\int }_0^t(|y_s-y_s^{\prime }|+|y_{s-{\delta }_1}-y_{s-{\delta }_1}^{\prime }|\cdots +|y_{s-{\delta }_k}-y_{s-{\delta }_k}^{\prime }|{)}^l\mathrm{d}s\\ & \le C{\int }_{-\mathrm{\infty }}^t|y_s-y_s^{\prime }{|}^l\mathrm{d}s.\end{array}$

Remark 5.2

In the above assumptions the involved coefficients are all deterministic. We remark that a trivial extension is to allow these to depend on ω as well in which case the coefficients in the Lipschitz conditions can be taken to be non-decreasing, $\mathbb{P}$-a.s. finite, ${\mathcal{P}}_{\mathbb{G}}$-measurable càdlàg  processes.

The motivation for allowing intervention costs that are discontinuous in b is the important application of production systems, where increasing the production beyond a certain threshold may necessitate a costly startup of additional production units.

5.2. Existence of optimal controls

In this section we show that the problem of maximizing the reward functional (24(24) $\begin{array}{r}J\left(u\right)=\mathbb{E}[{\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t-\sum _{j=1}^N{e}^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{u,j-1},{\text{β}}_j)].\end{array}$(24) ) has a solution. Throughout we will, for notational simplicity, only consider the one-dimensional case (d=1), but we note that all results extend trivially to higher dimensions. We start with the following moment estimate:

Proposition 5.2

Under Assumption 5.1, the SFDE (20(20) $\begin{array}{rl}{X}_t^{u,0}& =x\left(t\right)\mathrm{f}\mathrm{o}\mathrm{r}t\in (-\mathrm{\infty },0)\end{array}$(20) )–(23(23) $\begin{array}{rl}{X}_t^{u,k}& =\mathrm{\Gamma }({\tau }_k,(X_s^{u,k-1}{)}_{s\le {\tau }_k},{\text{β}}_k)+{\int }_{{\tau }_k}^{t}a(s,\left(X_r^{u,k}{)}_{r\le s}\right)\mathrm{d}s+{\int }_{{\tau }_k}^t\sigma (s,(X_r^{u,k}{)}_{r\le s})d{B}_s\\ & +{\int }_{{\tau }_k+}^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,k}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge {\tau }_k.\end{array}$(23) ) admits a unique solution for each $u\in {\mathcal{U}}^f$. Furthermore, the solution has moments of order pq on compacts, in particular we have for T>0, that (25) $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{t\in [0,T]}{sup}|X_t^u{|}^{pq}]\le C,\end{array}$(25) where $C=C(T,pq)$ and for each $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, we have (26) $\begin{array}{r}\mathbb{E}[\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}{|}^q|{\mathcal{F}}_t]{|}^2]\le C\end{array}$(26) where $C=C(T,q)$.

Proof.

By repeated use of Theorem 3.2 in [1] existence and uniqueness of solutions to (20(20) $\begin{array}{rl}{X}_t^{u,0}& =x\left(t\right)\mathrm{f}\mathrm{o}\mathrm{r}t\in (-\mathrm{\infty },0)\end{array}$(20) )–(23(23) $\begin{array}{rl}{X}_t^{u,k}& =\mathrm{\Gamma }({\tau }_k,(X_s^{u,k-1}{)}_{s\le {\tau }_k},{\text{β}}_k)+{\int }_{{\tau }_k}^{t}a(s,\left(X_r^{u,k}{)}_{r\le s}\right)\mathrm{d}s+{\int }_{{\tau }_k}^t\sigma (s,(X_r^{u,k}{)}_{r\le s})d{B}_s\\ & +{\int }_{{\tau }_k+}^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,k}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge {\tau }_k.\end{array}$(23) ) follows since ${\tau }_j\to \mathrm{\infty }$, $\mathbb{P}$-a.s. By Assumption 5.1.(i) we get, for $t\in [{\tau }_j,T]$, using integration by parts, that $\begin{array}{rl}|X_t^{u,j}{|}^2& =|X_{{\tau }_j}^{u,j}{|}^2+2{\int }_{{\tau }_j+}^t{X}_{s-}^{u,j}d{X}_s^{u,j}+{\int }_{{\tau }_j+}^{t}d[X^{u,j},X^{u,j}{]}_s\\ & \le K_{\mathrm{\Gamma }}^2\vee |X_{{\tau }_j}^{u,j-1}{|}^2+2{\int }_{{\tau }_j+}^t{X}_{s-}^{u,j}d{X}_s^{u,j}+{\int }_{{\tau }_j+}^{t}d[X^{u,j},X^{u,j}{]}_s.\end{array}$We note that if $|X_t^{u,j}|>K_{\mathrm{\Gamma }}$ and $|X_s^{u,j}|\le K_{\mathrm{\Gamma }}$ for some $s\in [0,t)$ then there is a largest time $\zeta <t$ such that $|X_{\zeta -}^{u,j}|\le K_{\mathrm{\Gamma }}$. This means that during the interval $(\zeta ,t]$ interventions will not increase the magnitude $|X^{u,j}|$. By induction, since $|x_0|$ is finite, we find that $\begin{array}{r}|X_t^{u,j}{|}^2\le C+\sum _{i=0}^j\{2{\int }_{\zeta \vee ({\tilde{\tau }}_i+)}^{t\wedge {\tilde{\tau }}_{i+1}}{X}_{s-}^{u,i}d{X}_s^{u,i}+{\int }_{\zeta \vee ({\tilde{\tau }}_i+)}^{t\wedge {\tilde{\tau }}_{i+1}}d[X^{u,i},X^{u,i}{]}_s\}\end{array}$for all $t\in [0,T]$, where $\zeta =sup\{s\ge 0:|X_s^u|\le K_{\mathrm{\Gamma }}\}\vee 0$, ${\tilde{\tau }}_0+=0$, ${\tilde{\tau }}_i={\tau }_i$ for $i=1,\dots ,j$ and ${\tilde{\tau }}_{j+1}=\mathrm{\infty }$. Letting $\begin{array}{r}{R}_t:=\sum _{i=0}^j\{2{\int }_{{\tilde{\tau }}_i+}^{t\wedge {\tilde{\tau }}_{i+1}}{X}_{s-}^{u,i}d{X}_s^{u,i}+{\int }_{{\tilde{\tau }}_i+}^{t\wedge {\tilde{\tau }}_{i+1}}d[X^{u,i},X^{u,j}{]}_s\}\end{array}$we thus find that for $p\ge 2$, $\begin{array}{r}\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{u,j}{|}^p|{\mathcal{F}}_t]\le C(1+\mathbb{E}[|X_t^{u,j}{|}^p+\underset{s\in [t,T]}{sup}|R_s-R_t{|}^{p/2}|{\mathcal{F}}_t]).\end{array}$Now, since $X^{u,i}$ and $X^{u,j}$ coincide on $[0,{\tau }_{i+1\wedge j+1})$ we have $\begin{array}{rl}\sum _{i=0}^j{\int }_{{\tilde{\tau }}_i+}^{t\wedge {\tilde{\tau }}_{i+1}}{X}_{s-}^{u,i}d{X}_s^{u,i}& ={\int }_0^t{X}_s^{u,j}a(s,(X_r^{u,j}{)}_{r\le s})\mathrm{d}s+{\int }_0^t{X}_s^{u,j}\sigma (s,\left(X_r^{u,j}{)}_{r\le s}\right)d{W}_s\\ & +{\int }_0^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}{X}_{s-}^{u,j}\gamma (s,(X_r^{u,j}{)}_{r<s},z\left)\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\end{array}$and $\begin{array}{rl}\sum _{i=0}^j{\int }_{{\tilde{\tau }}_i+}^{t\wedge {\tilde{\tau }}_{i+1}}d[X^{u,i},X^{u,j}{]}_s& ={\int }_0^t{\sigma }^2(s,(X_r^{u,j}{)}_{r\le s})\mathrm{d}s\\ & +{\int }_0^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}{\gamma }^2(s,(X_r^{u,j}{)}_{r<s},z\left)P\right(\mathrm{d}s,\mathrm{d}z).\end{array}$From Assumption 5.1.(ii)-(iii) and the Burkholder-Davis-Gundy inequality we get that $\begin{array}{rl}\mathbb{E}[\underset{t\le s\le T}{sup}|R_s-R_t{|}^{p/2}|{\mathcal{F}}_t]& \le C\mathbb{E}[|{\int }_t^T(1+\underset{r\le s}{sup}|X_r^{u,j}{|}^4)\mathrm{d}s{|}^{p/4}\\ & +|{\int }_t^T(1+\underset{r\le s}{sup}|X_r^{u,j}{|}^2)\mathrm{d}s{|}^{p/2}|{\mathcal{F}}_t]\\ & \le C(1+T^{p/2-1})\mathbb{E}[{\int }_t^T(1+\underset{r\le s}{sup}|X_r^{u,j}{|}^p)\mathrm{d}s|{\mathcal{F}}_t]\end{array}$and Grönwall's lemma gives that (27) $\begin{array}{r}\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{u,j}{|}^p|{\mathcal{F}}_t]\le C(1+\mathbb{E}[\underset{s\in [0,t]}{sup}|X_s^{u,j}{|}^p|{\mathcal{F}}_t]),\end{array}$(27) $\mathbb{P}$-a.s., where the constant $C=C(T,q)$ does not depend on u or j and (25(25) $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{t\in [0,T]}{sup}|X_t^u{|}^{pq}]\le C,\end{array}$(25) ) follows by letting t=0. We now give a more straightforward way of showing (26(26) $\begin{array}{r}\mathbb{E}[\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}{|}^q|{\mathcal{F}}_t]{|}^2]\le C\end{array}$(26) ) than the method used in the proof of Lemma 4.2. Applying (27(27) $\begin{array}{r}\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{u,j}{|}^p|{\mathcal{F}}_t]\le C(1+\mathbb{E}[\underset{s\in [0,t]}{sup}|X_s^{u,j}{|}^p|{\mathcal{F}}_t]),\end{array}$(27) ) to the left-hand side of (26(26) $\begin{array}{r}\mathbb{E}[\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}{|}^q|{\mathcal{F}}_t]{|}^2]\le C\end{array}$(26) ) we get $\begin{array}{rl}\mathbb{E}[\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}{|}^q|{\mathcal{F}}_t]{|}^2]& \le C(1+\mathbb{E}[\underset{t\in [0,T]}{sup}|\mathbb{E}[\underset{s\in [0,t]}{sup}|X_s^v{|}^q|{\mathcal{F}}_t]{|}^2\\ & +\underset{(t,b)\in [0,T]\times U}{sup}|\mathbb{E}[|\mathrm{\Gamma }(t,X_t^v,b){|}^q|{\mathcal{F}}_t]{|}^2])\\ & \le C(1+\mathbb{E}[\underset{t\in [0,T]}{sup}|\mathbb{E}[\underset{s\in [0,t]}{sup}|X_s^v{|}^q|{\mathcal{F}}_t]{|}^2])\\ & \le C(1+\mathbb{E}[\underset{t\in [0,T]}{sup}|X_t^v{|}^{2q}])\end{array}$and the desired result follows from (25(25) $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{t\in [0,T]}{sup}|X_t^u{|}^{pq}]\le C,\end{array}$(25) ).

Lemma 5.3

For each $k\ge 0$, there is a $\mathbb{P}$-null set $\mathcal{N}$ such that for all $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$ and all $(\mathbf{t},\mathbf{b})\in {\mathcal{D}}^k$ the limit $\underset{(t^{\prime },b^{\prime })\to (t,b)}{lim}{X}^{t^{\prime },b^{\prime }}$ exists in the topology of uniform convergence on compact subsets of ${\mathbb{R}}_{+}\setminus \{t_1,\dots ,t_k\}$. Furthermore, for all $(t,b)\in {\mathbb{R}}_{+}\times U$, we have $\begin{array}{r}\underset{t^{\prime }\searrow t}{lim}\underset{b^{\prime }\to b}{lim}\underset{s\in [t^{\prime },T]}{sup}|X_s^{v\circ (t^{\prime },b^{\prime })\circ u}-X_s^{v\circ (t,b)\circ u}|=0,\end{array}$$\mathbb{P}$-a.s., for any $T\ge 0$, $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ and $u\in {\mathcal{U}}^k$ (with an exception set that is independent of $(t,b)$).

Proof.

Our proof will rely on a pre-localization argument and we introduce the following non-decreasing sequence of stopping times $\begin{array}{r}{\kappa }_K:=inf\{s\ge 0:{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\overline{\gamma }(z\left)P\right(\left\{s\right\},\mathrm{d}z)\ge K\},\end{array}$for $K\ge 0$ and set ${\mathrm{\Lambda }}_K:=[0,{\kappa }_K)$. By, Assumption 5.1.(iii) it then follows that ${\kappa }_K\to \mathrm{\infty }$, $\mathbb{P}$-a.s. as $K\to \mathrm{\infty }$. Furthermore, we note that on ${\mathrm{\Lambda }}_K$ the magnitude of the jumps of $X^u$ due to the Poisson jump integral of (20(20) $\begin{array}{rl}{X}_t^{u,0}& =x\left(t\right)\mathrm{f}\mathrm{o}\mathrm{r}t\in (-\mathrm{\infty },0)\end{array}$(20) )–(23(23) $\begin{array}{rl}{X}_t^{u,k}& =\mathrm{\Gamma }({\tau }_k,(X_s^{u,k-1}{)}_{s\le {\tau }_k},{\text{β}}_k)+{\int }_{{\tau }_k}^{t}a(s,\left(X_r^{u,k}{)}_{r\le s}\right)\mathrm{d}s+{\int }_{{\tau }_k}^t\sigma (s,(X_r^{u,k}{)}_{r\le s})d{B}_s\\ & +{\int }_{{\tau }_k+}^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,k}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge {\tau }_k.\end{array}$(23) ) are bounded by $C+K\underset{s\le t}{sup}|X_s^u|$ and repeating the argument in the proof of Proposition 5.2 gives that (28) $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{s\in [0,T]\cap {\mathrm{\Lambda }}_K}{sup}|X_t^u{|}^l]\le C,\end{array}$(28) for all $l\ge 0$.

For $0\le t\le t^{\prime }\le T$, we let ${}^{t,t^{\prime }}{X}^v$ solve the SFDE (20(20) $\begin{array}{rl}{X}_t^{u,0}& =x\left(t\right)\mathrm{f}\mathrm{o}\mathrm{r}t\in (-\mathrm{\infty },0)\end{array}$(20) )–(23(23) $\begin{array}{rl}{X}_t^{u,k}& =\mathrm{\Gamma }({\tau }_k,(X_s^{u,k-1}{)}_{s\le {\tau }_k},{\text{β}}_k)+{\int }_{{\tau }_k}^{t}a(s,\left(X_r^{u,k}{)}_{r\le s}\right)\mathrm{d}s+{\int }_{{\tau }_k}^t\sigma (s,(X_r^{u,k}{)}_{r\le s})d{B}_s\\ & +{\int }_{{\tau }_k+}^t{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}\gamma (s,(X_r^{u,k}{)}_{r<s},z\left)d\tilde{P}\right(\mathrm{d}s,\mathrm{d}z),\mathrm{f}\mathrm{o}\mathrm{r}t\ge {\tau }_k.\end{array}$(23) ) with integrand $(1-{\mathbb{1}}_{(t,t^{\prime }]}(s\left)\right)\gamma (s,\cdot ,\cdot )$ in the jump part and let $j\ge 0$ be the largest integer such that ${\tau }_j\le t^{\prime }$. Then by Assumption 5.1.(i) we have for $l=1,\dots ,j+1$ (recalling that $[u{]}_l=({\tau }_1,\dots ,{\tau }_{N\wedge l};{\text{β}}_1,\dots ,{\text{β}}_{N\wedge l})$ is the truncation of u limiting the number of interventions to l), $\begin{array}{rl}& |{}^{t,t^{\prime }}{X}_{t^{\prime }}^{v\circ \left[\right(t^{\prime },b^{\prime })\circ u{]}_l}-{}^{t,t^{\prime }}{X}_{t^{\prime }}^{v\circ \left[\right(t,b)\circ u{]}_l}|\le C({\int }_t^{{\tau }_{l-1}}|{}^{t,t^{\prime }}{X}_s^{v\circ \left[\right(t^{\prime },b^{\prime })\circ u{]}_{l-1}}-{}^{t,t^{\prime }}{X}_s^{v\circ \left[\right(t,b)\circ u{]}_{l-1}}|\mathrm{d}s\\ & +|{}^{t,t^{\prime }}{X}_{t^{\prime }}^{v\circ \left[\right(t^{\prime },b^{\prime })\circ u{]}_{l-1}}-{}^{t,t^{\prime }}{X}_{t^{\prime }}^{v\circ \left[\right(t,b)\circ u{]}_{l-1}}|+|{}^{t,t^{\prime }}{X}_{t^{\prime }}^{v\circ \left[\right(t,b)\circ u{]}_{l-1}}-{}^{t,t^{\prime }}{X}_{{\tau }_{l-1}}^{v\circ \left[\right(t,b)\circ u{]}_{l-1}}|\\ & +\left(\right(t^{\prime }-{\tau }_{l-1})+{\mathbb{1}}_{[l=1]}|b^{\prime }-b|)(1+\underset{s\le t^{\prime }}{sup}|{}^{t,t^{\prime }}{X}_s^{v\circ \left[\right(t^{\prime },b^{\prime })\circ u{]}_{l-1}}|+\underset{s\le {\tau }_l}{sup}|{}^{t,t^{\prime }}{X}_s^{v\circ \left[\right(t,b)\circ u{]}_{l-1}}|)),\end{array}$with ${\tau }_0:=t$. We define ${}^1{X}^l:={}^{t,t^{\prime }}{X}^{v\circ \left[\right(t,b)\circ u{]}_l}$, ${}^2{X}^l:={}^{t,t^{\prime }}{X}^{v\circ \left[\right(t^{\prime },b^{\prime })\circ u{]}_l}$ and let $\delta X^l:={}^2{X}^l-{}^1{X}^l$ and set $\delta X:=\delta X^{k+1}$. Then, since the jump part is deactivated during $(t,t^{\prime }]$ and by (28(28) $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{s\in [0,T]\cap {\mathrm{\Lambda }}_K}{sup}|X_t^u{|}^l]\le C,\end{array}$(28) ), we have $\begin{array}{r}\mathbb{E}[|\delta X_{t^{\prime }}{|}^{2(m+2)}]\le C(|t^{\prime }-t{|}^{m+2}+|b^{\prime }-b{|}^{m+2}).\end{array}$For $l=j+2,\dots ,N$, we have by Assumption 5.1.(i) that $\begin{array}{r}|\delta X_{{\tau }_l}^{l+1}|\le C({\int }_0^{{\tau }_l}|\delta X_s^l|\mathrm{d}s+|\delta X_{{\tau }_l}^l|).\end{array}$Now, for $s\ge t^{\prime }$, $\begin{array}{r}\delta X_s=\delta X_{t^{\prime }}+\sum _{l=j}^N{\int }_{(t^{\prime }\vee {\tau }_l)+}^{s\wedge {\tau }_{l+1}}d(\delta X^{l+1}{)}_r+\sum _{l=j+1}^N{\mathbb{1}}_{[s\ge {\tau }_l]}(\delta X_{{\tau }_l}^{l+1}-\delta X_{{\tau }_l}^l),\end{array}$with ${\tau }_{N+1}:=\mathrm{\infty }$. Taking the absolute value on both sides we get $\begin{array}{rl}|\delta X_s|& \le |\delta X_{t^{\prime }}|+C\left({\int }_{t^{\prime }}^s|a\right(r,\left({}^2{X}_{\zeta }{)}_{\zeta \le r}\right)-a(r,({}^1{X}_{\zeta }{)}_{\zeta \le r})|\mathrm{d}r\\ & +\sum _{l=j}^N|{\int }_{t^{\prime }\vee {\tau }_l}^{{\tau }_{l+1}\wedge s}\sigma (r,({}^2{X}_{\zeta }{)}_{\zeta \le r})-\sigma (r,\left({}^1{X}_{\zeta }{)}_{\zeta \le r}\right)d{W}_r|\\ & +{\int }_{t^{\prime }+}^s{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}|\gamma (r,({}^2{X}_{\zeta }{)}_{\zeta <r},z)-\gamma (r,({}^1{X}_{\zeta }{)}_{\zeta <s},z)|\tilde{P}(\mathrm{d}r,\mathrm{d}z)+{\int }_t^s|\delta X_r|\mathrm{d}r).\end{array}$The Burkholder-Davis-Gundy inequality now gives $\begin{array}{rl}& \mathbb{E}[\underset{r\in [t^{\prime },s]}{sup}|\delta X_r{|}^{2(m+2)}]\le C\mathbb{E}[|\delta X_{t^{\prime }}{|}^{2(m+2)}\\ & +({\int }_{t^{\prime }}^s|a(r,({}^2{X}_{\zeta }{)}_{\zeta \le r})-a(r,\left({}^1{X}_{\zeta }{)}_{\zeta \le r}\right)|\mathrm{d}r{)}^{2(m+2)}\\ & +({\int }_{t^{\prime }}^s|\sigma (r,({}^2{X}_{\zeta }{)}_{\zeta \le r})-\sigma (r,\left({}^1{X}_{\zeta }{)}_{\zeta \le r}\right){|}^2\mathrm{d}r{)}^{m+2}\\ & +({\int }_{t^{\prime }+}^s{\int }_{{\mathbb{R}}^d\setminus \left\{0\right\}}|\gamma (r,({}^2{X}_{\zeta }{)}_{\zeta <r},z)-\gamma (r,({}^1{X}_{\zeta }{)}_{\zeta <r},z){|}^2\tilde{P}(\mathrm{d}r,\mathrm{d}z){)}^{m+2}\\ & +{\int }_t^s|\delta X_r{|}^{2(m+2)}\mathrm{d}r].\end{array}$Appealing to the boundedness of the jumps and the integral Lipschitz conditions on the coefficients then gives that $\begin{array}{rl}\mathbb{E}[\underset{r\in [t^{\prime },s]\cap {\mathrm{\Lambda }}_K}{sup}|\delta X_r{|}^{2(m+2)}]& \le C({\int }_{t^{\prime }}^s\mathbb{E}[\underset{\zeta \in [t^{\prime },r]\cap {\mathrm{\Lambda }}_K}{sup}|\delta X_{\zeta }{|}^{2(m+2)}]\mathrm{d}r\\ & +|t^{\prime }-t{|}^{m+2}+|b^{\prime }-b{|}^{m+2}),\end{array}$for all $s\in [0,T]$. Now, Grönwall's lemma gives $\begin{array}{r}\mathbb{E}[\underset{s\in [t^{\prime },T]\cap {\mathrm{\Lambda }}_K}{sup}|\delta X_s{|}^{2(m+2)}]\le C(|t^{\prime }-t{|}^{m+2}+|b^{\prime }-b{|}^{m+2}),\end{array}$where C does not depend on $u\in {\mathcal{U}}^k$. Furthermore, for each $t\in [0,T]$ and each $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$ (for some $\mathbb{P}$-null set $\mathcal{N}$) there is a $t^{\prime }>t$ such that $P(\omega ,(t,t^{\prime }],{\mathbb{R}}^d)=0$. Uniform convergence on $[t^{\prime },T]\cap {\mathrm{\Lambda }}_K$, thus, follows by applying a Kolmogorov continuity argument (see e.g.  Theorem 72 in Chapter IV of [25]) and uniform right-continuity follows as ${\kappa }_K\to \mathrm{\infty }$, $\mathbb{P}$-a.s. The existence of limits follows similarly.

Definition 5.4

For all $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ and $u\in {\mathcal{U}}^f$ we define the map ${\mathrm{\Psi }}^{v,u}:{\mathbb{R}}_{+}\times \mathrm{\Omega }\times U$ as $\begin{array}{r}{\mathrm{\Psi }}^{v,u}(t,b):={\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^{v\circ (t,b)\circ u})\mathrm{d}s-\sum _{j=1}^N{e}^{-\rho ({\tau }_j\vee t)}\ell ({\tau }_j\vee t,X_{{\tau }_j\vee t}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j).\end{array}$Moreover, for $T\ge 0$ we define the truncation ${\mathrm{\Psi }}_T^{v,u}:{\mathbb{R}}_{+}\times \mathrm{\Omega }\times U$ of ${\mathrm{\Psi }}^{v,u}$ as $\begin{array}{r}{\mathrm{\Psi }}_T^{v,u}(t,b):={\int }_0^T{e}^{-\rho \left(s\right)}\varphi (s,X_s^{v\circ (t,b)\circ u})\mathrm{d}s-\sum _{j=1}^{N(T-)}{e}^{-\rho ({\tau }_j\vee t)}\ell ({\tau }_j\vee t,X_{{\tau }_j\vee t}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j),\end{array}$where $N(T-):=max\{j:{\tau }_j<T\}$, and for $L\ge 0$ we define the localization ${\mathrm{\Psi }}_{T,L}^{v,u}:{\mathbb{R}}_{+}\times \mathrm{\Omega }\times U\to \mathbb{R}$ of ${\mathrm{\Psi }}_T^{v,u}$ as $\begin{array}{r}{\mathrm{\Psi }}_{T,L}^{v,u}(t,b):={\int }_0^T{e}^{-\rho \left(s\right)}\varphi (s,{}^L{X}_s^{v\circ (t,b)\circ u})\mathrm{d}s-\sum _{j=1}^{N(T-)}{e}^{-\rho ({\tau }_j\vee t)}\ell ({\tau }_j\vee t,{}^L{X}_{{\tau }_j\vee t}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j),\end{array}$where ${}^L{X}_s^u:=\frac{L}{L\vee |X_s^u|}{X}_s^u$.

Corollary 5.5

For each $T,L\ge 0$, $k\ge 0$, $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ and $u\in {\mathcal{U}}^k$ the map $(t,b)\mapsto {\mathrm{\Psi }}_{T,L}^{v,u}(t,b)$ has limits everywhere and is $\mathbb{P}$-a.s. continuous on $[{\eta }_j,{\eta }_{j+1})\times U$, where $({\eta }_j{)}_{j\ge 1}$ are the jump times of P.

Proof.

Let $\begin{array}{r}{\phi }_{T,L}\left(u\right):={\int }_0^T{e}^{-\rho \left(s\right)}\varphi (s,{}^L{X}_s^u)\mathrm{d}s\end{array}$and note that for $0\le t\le t^{\prime }$ we have $\begin{array}{rl}& |{\phi }_{T,L}(v\circ (t^{\prime },b^{\prime })\circ u)-{\phi }_{T,L}(v\circ (t,b)\circ u)|\\ & \le {\int }_t^{t^{\prime }}{e}^{-\rho \left(s\right)}\left(|\varphi \right(s,{}^L{X}_s^{v\circ (t,b)\circ u})|+|\varphi (s,{}^L{X}_s^{v\circ (t^{\prime },b^{\prime })\circ u}\left)|\right)\mathrm{d}s\\ & +{\int }_{t^{\prime }}^T{e}^{-\rho \left(s\right)}|\varphi (s,{}^L{X}_s^{v\circ (t,b)\circ u})-\varphi (s,{}^L{X}_s^{v\circ (t^{\prime },b^{\prime })\circ u})|\mathrm{d}s\\ & \le C(|t^{\prime }-t|+{\int }_{t^{\prime }}^T{e}^{-\rho \left(s\right)}|{}^L{X}_s^{v\circ (t,b)\circ u}-{}^L{X}_s^{v\circ (t^{\prime },b^{\prime })\circ u}|\mathrm{d}s)\\ & \le C(|t^{\prime }-t|+{\int }_{t^{\prime }}^T{e}^{-\rho \left(s\right)}|X_s^{v\circ (t,b)\circ u}-X_s^{v\circ (t^{\prime },b^{\prime })\circ u}|\mathrm{d}s).\end{array}$Now, by Lemma 5.3 it follows immediately that $\begin{array}{r}\underset{t^{\prime }\searrow t}{lim}\underset{b^{\prime }\to b}{lim}|{\phi }_T(v\circ (t^{\prime },b^{\prime })\circ u)-{\phi }_T(v\circ (t,b)\circ u)|=0,\end{array}$and from its proof we have that $\begin{array}{r}\underset{t^{\prime }\nearrow t}{lim}\underset{b^{\prime }\to b}{lim}|{\phi }_T(v\circ (t^{\prime },b^{\prime })\circ u)-{\phi }_T(v\circ (t,b)\circ u)|=0,\end{array}$whenever $t\notin \{{\eta }_1,{\eta }_2,\dots \}$. Concerning the intervention costs we have (29) $\begin{array}{rl}& \sum _{j=1}^{N(T-)}|e^{-\rho ({\tau }_j\vee t^{\prime })}\ell ({\tau }_j\vee t^{\prime },{}^L{X}_{{\tau }_j\vee t^{\prime }}^{v\circ (t^{\prime },b^{\prime })\circ [u{]}_{j-1}},{\text{β}}_j)-e^{-\rho ({\tau }_j\vee t)}\ell ({\tau }_j\vee t,{}^L{X}_{{\tau }_j\vee t}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j)|\\ & \le \sum _{j=1}^{N(T-)}\left\{{\mathbb{1}}_{[0,t^{\prime })}\right({\tau }_j\left)|e^{-\rho \left(t^{\prime }\right)}\ell \right(t^{\prime },{}^L{X}_{t^{\prime }}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j)-e^{-\rho ({\tau }_j\vee t)}\ell ({\tau }_j\vee t,{}^L{X}_{{\tau }_j\vee t}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j)|\\ & +|\ell ({\tau }_j\vee t^{\prime },{}^L{X}_{{\tau }_j\vee t^{\prime }}^{v\circ (t^{\prime },b^{\prime })\circ [u{]}_{j-1}},{\text{β}}_j)-\ell ({\tau }_j\vee t^{\prime },{}^L{X}_{{\tau }_j\vee t^{\prime }}^{v\circ (t,b)\circ [u{]}_{j-1}},{\text{β}}_j)|\}\end{array}$(29) where the first term tends to zero as $t^{\prime }\searrow t$ by joint continuity of ℓ, continuity of ρ and right continuity of X. By continuity of ℓ, the assertion follows by repeating the argument in the proof of Lemma 5.3.

Lemma 5.6

For each T>0 and $k\ge 0$ there is, for every $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$, a $J_T^{\ast }\in {\mathcal{H}}_{\mathbb{F}}$ such that for all $\tau \in \mathcal{T}$ and $b\in U$ we have $\begin{array}{r}{J}_T^{\ast }(\tau ,b)=\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. (with an exception set that is independent of b).

Proof.

For any $K,L\ge 0$ it follows by Corollary 5.5 and Theorem A.6 in Appendix 3 that there is for each $b\in U$ an $\mathbb{F}$-optional càdlàg  process $(Z_t^{b,u}:t\ge 0)$ such that $\begin{array}{r}{Z}_{\tau }^{b,u}=\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. for any $\tau \in \mathcal{T}$. Now, pick a sequence $({ϵ}_l{)}_{l\ge 0}$ of positive real numbers such that ${ϵ}_l\searrow 0$ and for $j,l\ge 0$ define $s_j^l:=j{2}^{-l}$. Then, there is a control $u_j^l\in {\mathcal{U}}_{s_j^l}^k$ such that $\begin{array}{r}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u_j^l}(s_j^l,b)|{\mathcal{F}}_{s_j^l}]\ge \underset{u\in {\mathcal{U}}_{s_j^l}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(s_j^l,b)|{\mathcal{F}}_{s_j^l}]-{ϵ}_l.\end{array}$Define the sequence of càdlàg  processes $({\tilde{Z}}_t^{b,l}:t\ge 0{)}_{l\ge 0}$ as $\begin{array}{r}{\tilde{Z}}_t^{b,l}:=\sum _{j=0}^{\mathrm{\infty }}{\mathbb{1}}_{[s_j^l,s_{j+1}^l)}\left(t\right)Z_t^{b,u_{j+1}^l}\end{array}$and set ${\hat{Z}}_t^{b,l}:=\underset{i\in \{0,\dots ,l\}}{max}{\tilde{Z}}_t^{b,i}$. Then, ${\hat{Z}}^{b,l}$ is an increasing $\mathbb{P}$-a.s. finite sequence of càdlàg  processes and it, thus, converges pointwisely, $\mathbb{P}$-a.s. to a limit $Z^{b,\ast }$ that, moreover, is ${\mathcal{P}}_{\mathbb{F}}$-measurable. We note that for any $l\ge 0$ and $\tau \in {\mathcal{T}}^f$ we have with ${\tau }_l:=inf\{s\ge \tau :s\in {\mathrm{\Pi }}_l\}$ and $u^l:=\sum _{j=1}^{\mathrm{\infty }}{\mathbb{1}}_{[{\tau }_l=s_j^l]}{u}_j^l$, that $\begin{array}{rl}& \underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }]-Z_{\tau }^{b,\ast }\\ & \le \underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }]-\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u^l}(\tau ,b)|{\mathcal{F}}_{\tau }]\\ & =\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }]-\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b)|{\mathcal{F}}_{\tau }]\\ & +\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b)|{\mathcal{F}}_{\tau }]-\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u^l}(\tau ,b)|{\mathcal{F}}_{\tau }]\end{array}$and as $\begin{array}{rl}& |\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b)|{\mathcal{F}}_{\tau }]-\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u^l}(\tau ,b)|{\mathcal{F}}_{\tau }]|\\ & \le \underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b)||{\mathcal{F}}_{\tau }]+{ϵ}_l\end{array}$we get that $\begin{array}{rl}& |Z_{\tau }^{b,\ast }-\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }]|\\ & \le 2\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b)||{\mathcal{F}}_{\tau }]+{ϵ}_l.\end{array}$Jensen's inequality now gives that $\begin{array}{rl}& \mathbb{E}[|Z_{\tau }^{b,\ast }-\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }]{|}^2]\\ & \le 8\underset{u\in {\mathcal{U}}_{\tau }^k}{sup}\mathbb{E}[|{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\tau }_l,b){|}^2]+2{ϵ}_l^2.\end{array}$Letting $l\to \mathrm{\infty }$ and using that the map $t\mapsto {\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(t,b)$ is right-continuous uniformly in u it follows that $\begin{array}{r}{Z}_{\tau }^{b,\ast }=\underset{u\in {\mathcal{U}}_{\tau }^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s., for each stopping time $\tau \in \mathcal{T}$.

We now show that $Z^{b,\ast }$ is a càdlàg  process. First, since $Z^{b,\ast }$ is the limit of an increasing sequence of càdlàg  processes we have that $\underset{t^{\prime }\searrow t}{lim inf}{Z}_{t^{\prime }}^{b,\ast }\ge Z_t^{b,\ast }$. For any $\tau \in {\mathcal{T}}^f$ and $ϵ>0$ let $\begin{array}{r}{\tau }_l:=inf\{s\ge \tau :{\hat{Z}}_s^{b,l}\ge Z_{\tau }^{b,\ast }+ϵ\}.\end{array}$Then as $(Z^{b,l}{)}_{l\ge 0}$ is non-decreasing, the sequence $({\tau }_l{)}_{l\ge 0}$ is non-increasing. Let $B:=\{\omega \in \mathrm{\Omega }:\underset{l\to \mathrm{\infty }}{lim}{\tau }_l=\tau \}$ and note that $B\in {\mathcal{F}}_{\tau }$ by right-continuity of the filtration and $\underset{t^{\prime }\searrow t}{lim sup}{Z}_{t^{\prime }}^{b,\ast }<Z_t^{b,\ast }+ϵ$ on $B^c:=\mathrm{\Omega }\setminus B$. Moreover, with ${\hat{\tau }}_l:={\mathbb{1}}_B{\tau }_l+{\mathbb{1}}_{B^c}\tau$, Fatou's lemma gives $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim inf}\mathbb{E}[Z_{{\hat{\tau }}_l}^{b,\ast }-Z_{\tau }^{b,\ast }]=\underset{l\to \mathrm{\infty }}{lim inf}\mathbb{E}[{\mathbb{1}}_B(Z_{{\hat{\tau }}_l}^{b,\ast }-Z_{\tau }^{b,\ast })]\ge \mathbb{P}\left[B\right]ϵ.\end{array}$On the other hand, we have $\begin{array}{r}\mathbb{E}[Z_{{\hat{\tau }}_l}^{b,\ast }-Z_{\tau }^{b,\ast }]\le \underset{u\in {\mathcal{U}}_{\tau }^k}{sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}({\hat{\tau }}_l,b)|]\to 0\end{array}$as $l\to \mathrm{\infty }$ and we conclude that $\mathbb{P}\left[B\right]=0$ and, since $ϵ>0$ was arbitrary, it follows that $\underset{t^{\prime }\searrow t}{lim sup}{Z}_{t^{\prime }}^{b,\ast }=Z_t^{b,\ast }$.

To prove that $Z^{b,\ast }$ has left limits we define, for $ϵ>0$, the sequence $({\vartheta }_j^{ϵ}{)}_{j\ge 0}$ as ${\vartheta }_0^{ϵ}=0$ and then recursively let $\begin{array}{r}{\vartheta }_j^{ϵ}:=inf\{s\ge {\vartheta }_{j-1}^{ϵ}:\underset{u\in {\mathcal{U}}_s^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|Z_s^{b,u}-Z_{{\vartheta }_{j-1}^{ϵ}}^{b,u}|\ge ϵ\}.\end{array}$We note by the above discussion that ${\vartheta }_j^{ϵ}\in \mathcal{T}$ and furthermore, by right-continuity that ${\vartheta }_j^{ϵ}>{\vartheta }_{j-1}^{ϵ}$ and ${\vartheta }_j^{ϵ}\to \mathrm{\infty }$, $\mathbb{P}$-a.s. If not, we would have ${\vartheta }_j^{ϵ}\nearrow {\vartheta }^{ϵ}\in {\mathcal{T}}^f$ on some set A of positive measure. However, as increments in the jump integral part is $\mathbb{P}$-a.s. zero at predictable times we note by Corollary 5.5 that ${\mathrm{\Psi }}^{v,u}(t,b)$ is continuous in t at ${\vartheta }^{ϵ}$ on $A\setminus \mathcal{N}$ for some $\mathbb{P}$-null set $\mathcal{N}$, uniformly in u. Now, as the filtration is quasi-left continuous this implies that $\begin{array}{r}\underset{j\to \mathrm{\infty }}{lim sup}\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\underset{b\in U}{sup}|Z_{{\vartheta }_j^{ϵ}}^{b,u}-Z_{{\vartheta }_{j-1}^{ϵ}}^{b,u}|=0,\end{array}$on $A\setminus \mathcal{N}$, a contradiction. Letting, $\begin{array}{r}{\breve{Z}}_t^{b,l}:=\sum _{j=0}^{\mathrm{\infty }}{\mathbb{1}}_{[{\vartheta }_j^{1/l},{\vartheta }_{j-1}^{1/l})}\left(t\right)Z_{{\vartheta }_j^{1/l}}^{b,\ast },\end{array}$we find that $({\breve{Z}}^{b,l}{)}_{l\ge 0}$ is a sequence of càdlàg  processes with $\underset{t\in [0,T]}{sup}|Z_t^{b,\ast }-{\breve{Z}}_t^{b,l}|\le 1/l$ and we conclude that $Z^{b,\ast }$ is càdlàg. 

By repeating the argument in the proof of Lemma 5.3 we find that $\begin{array}{r}\underset{u\in {\mathcal{U}}_t^k}{sup}\mathbb{E}[|{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(t,b^{\prime })-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(t,b){|}^{2(m+1)}]\le C|b^{\prime }-b{|}^{m+1},\end{array}$$\mathbb{P}$-a.s. for any $t\in {\mathbb{R}}_{+}$ and $b,b^{\prime }\in U$ and it follows that $\begin{array}{r}\mathbb{E}[\underset{t\in [0,\mathrm{\infty }]}{sup}|Z_t^{b^{\prime },\ast }-Z_t^{b,\ast }{|}^{m+1}]\le C|b^{\prime }-b{|}^{m+1}.\end{array}$Hence, by Kolmogorov's continuity theorem and Corollary 5.5 it follows that there is a unique map $h^{K,L}\in {\mathcal{H}}_{\mathbb{F}}$ such that $\begin{array}{r}{h}^{K,L}(\tau ,b)=\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. for all $b\in U$. By dominated convergence we find that $h^{K,L}$ converges pointwisely to some h as $K,L\to \mathrm{\infty }$. We define the set $\begin{array}{r}{\mathrm{\Xi }}_L:=\{\omega \in \mathrm{\Omega }:\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^f}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}||{\mathcal{F}}_t]>L\}\end{array}$and note that for $r\in (1,2)$, we have $\begin{array}{rl}& \mathbb{E}[\underset{(t,b)\in [0,T]\times U}{sup}|\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(t,b)|{\mathcal{F}}_t]-h^{K,L}(t,b){|}^r]\\ & \le \mathbb{E}[\underset{(t,b)\in [0,T]\times U}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(t,b)-{\mathrm{\Psi }}_{T\wedge {\kappa }_K,L}^{v,u}(t,b)|{\mathcal{F}}_t]{|}^r]\\ & \le C\mathbb{E}[({\mathbb{1}}_{[{\kappa }_K<T]}+{\mathbb{1}}_{\mathrm{\Xi }})\underset{t\in [0,T]}{sup}\underset{u\in {\mathcal{U}}_t^{k+1}}{\mathrm{e}\mathrm{s}\mathrm{s}sup}|\mathbb{E}[\underset{s\in [t,T]}{sup}|X_s^{v\circ u}{|}^q|{\mathcal{F}}_t]{|}^r]\\ & \le C\mathbb{E}[{\mathbb{1}}_{[{\kappa }_K<T]}+{\mathbb{1}}_{{\mathrm{\Xi }}_L}{]}^{1/r^{\prime }}\end{array}$where $\frac{1}{r^{\prime }}+\frac{r}{2}=1$ and the last step follows by Hölder's inequality and Proposition 5.2. Now, the right-hand side of the last inequality goes to zero as $K,L\to \mathrm{\infty }$ by the definition of ${\kappa }_K$ and Proposition 5.2 and by uniform convergence we conclude that there is a $J_T^{\ast }\in {\mathcal{H}}_{\mathbb{F}}$ such that $\begin{array}{r}{J}_T^{\ast }(\tau ,b)=\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(\tau ,b)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. for each $b\in U$.

It remains to show that we can choose the exception set to be independent of b. Let ${\overline{U}}_0\subset {\overline{U}}_1\subset \cdots$ be a sequence of finite subsets of U with $\underset{b\in {\overline{U}}_l}{min}\underset{b^{\prime }\in U}{max}|b^{\prime }-b|\le 2^{-l}$. For $\text{β}\in \mathcal{A}\left(\tau \right)$ define $({\text{β}}_l{)}_{l\ge 0}$ as a measurable selection of ${\text{β}}_l\in {\arg min}_{b\in {\overline{U}}_l}|\text{β}-b|$. Then since ${\text{β}}_l$ takes values in a finite set we have $\begin{array}{r}{J}_T^{\ast }(\tau ,{\text{β}}_l)=\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(\tau ,{\text{β}}_l)|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. By continuity it follows that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim}{J}_T^{\ast }(\tau ,{\text{β}}_l)=J_T^{\ast }(\tau ,\text{β}),\end{array}$$\mathbb{P}$-a.s. Furthermore, by uniform integrability and $\mathbb{P}$-a.s. continuity of ${\mathrm{\Psi }}_T^{v,u}$ uniformly in u we have that $\begin{array}{r}\underset{l\to \mathrm{\infty }}{lim}\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\mathrm{\Psi }}_T^{v,u}(\tau ,\text{β})-{\mathrm{\Psi }}_T^{v,u}(\tau ,{\text{β}}_l)||{\mathcal{F}}_{\tau }]=0\end{array}$and we conclude that $\begin{array}{r}{J}_T^{\ast }(\tau ,\text{β})=\underset{u\in {\mathcal{U}}^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathrm{\Psi }}_T^{v,u}(\tau ,\text{β})|{\mathcal{F}}_{\tau }],\end{array}$$\mathbb{P}$-a.s. From this the statement follows as $\text{β}\in \mathcal{A}\left(\tau \right)$ was arbitrary.

This far we have not made any assumption on the discount factor ρ, other than it being continuous. Clearly, some assumptions on the growth of ρ have to be made in order for the maximization problem to have a finite value. We summarize our assumptions in the following hypothesis:

Hypothesis. [Disc.-A] There an $ϵ>0$ such that $\rho \left(t\right)\ge ϵt$, $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[{\int }_T^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}|\varphi (t,X_t^u){|}^p\mathrm{d}t]\le C{e}^{-ϵT}\end{array}$and $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}{e}^{-p\rho \left(t\right)}|\ell (t,X_t^u,b){|}^p\mathrm{d}t]<\mathrm{\infty }\end{array}$for all $T\ge 0$ and $b\in U$. Furthermore, for each $k\ge 0$ there is an $ϵ>0$ such that for all $T\ge T^{\prime }$ for some $T^{\prime }>0$ we have $\begin{array}{r}\mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\int }_T^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^{v\circ u})\mathrm{d}s||{\mathcal{F}}_t]]\le e^{-ϵT},\end{array}$and $\begin{array}{r}\mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[N\ge 1]}{\mathbb{1}}_{[{\tau }_N\ge T]}{e}^{-\rho \left({\tau }_N\right)}|\ell ({\tau }_N,X_{{\tau }_N}^{v\circ [u{]}_{N-1}},{\text{β}}_N)||{\mathcal{F}}_t]]\le e^{-ϵT}.\end{array}$for all $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$.

Remark 5.3

An important situation where Hypothesis Disc.-A holds with $\rho \left(t\right)={\rho }_{0}t$ for any ${\rho }_0>0$ is when the functions ϕ and ℓ are eventually bounded, i.e.  when there is a $T^{\prime }>0$ such that $|\varphi (t,x)|\le C$ and $|\ell (u\circ (t,x\left)\right)|\le C$ for all $(t,x)\in [T^{\prime },\mathrm{\infty })\times {\mathbb{R}}^d$. Another important case is when $\rho \left(T\right)-\ln \left(C\right(T,pq\left)\right)$ grows linearly in T, where C is the bound in Proposition 5.2.

We are now ready to state the main result of this section, showing that under Assumption 5.1 and Hypothesis Disc.-A an optimal control for the problem of maximizing J exist.

Proposition 5.7

Under Hypothesis Disc.-A there is a $u^{\ast }\in {\mathcal{U}}^f$ such that $J\left(u^{\ast }\right)\ge J\left(u\right)$ for all $u\in {\mathcal{U}}^f$. Furthermore, $u^{\ast }$ is given by the recursion (8(8) $\begin{array}{r}{\tau }_j^{\ast }:=inf\{s\ge {\tau }_{j-1}^{\ast }:Y_s^{[u^{\ast }{]}_{j-1}}=\underset{b\in U}{sup}\{-c_{s,b}^{[u^{\ast }{]}_{j-1}}+Y_s^{[u^{\ast }{]}_{j-1}\circ (s,b)}\}\},\end{array}$(8) )–(9(9) $\begin{array}{r}{\text{β}}_j^{\ast }\in \underset{b\in U}{\arg max}\{-c_{{\tau }_j^{\ast },b}^{[u^{\ast }{]}_{j-1}}+Y_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_{j-1}\circ ({\tau }_j^{\ast },b)}\}.\end{array}$(9) ), with $\begin{array}{r}\phi \left(u\right):={\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t\end{array}$and $\begin{array}{r}c(u\circ (t,b\left)\right):=e^{-\rho (t\vee {\tau }_N)}\ell (t\vee {\tau }_N,X_{t\vee {\tau }_N}^u,b).\end{array}$

Proof.

To show that the assertion is true we need to show that the pair $(\phi ,c)$ is an admissible reward pair. It is clear that the uniform $L^2$-bounds on φ and c in Definition 2.2.(i) hold by Hypothesis Disc.-A. In particular, we note that by Jensen's inequality we get that $\begin{array}{rl}\mathbb{E}[|\phi \left(u\right){|}^p]& =\mathbb{E}[|{\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t{|}^p]\\ & \le C\mathbb{E}[{\int }_0^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}|\varphi (t,X_t^u)\mathrm{d}t{|}^p\mathrm{d}t]\le C.\end{array}$The decreasing importance property stated in Definition 2.2.(iii) follows similarly by noting that for $v\in {\mathcal{U}}_T^f$ with $T\ge T^{\prime }$ we have, by Hypothesis Disc.-A, that $\begin{array}{rl}\mathbb{E}[|\phi (u\circ v)-\phi \left(u\right){|}^p]& =\mathbb{E}[|{\int }_T^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\left(\varphi \right(t,X_t^{u\circ v})-\varphi (t,X_t^u\left)\right)\mathrm{d}t{|}^p]\\ & \le C\mathbb{E}[{\int }_T^{\mathrm{\infty }}{e}^{-\rho \left(t\right)}\left(|\varphi \right(t,X_t^{u\circ v}){|}^p+|\varphi (t,X_t^u\left){|}^p\right)\mathrm{d}t]\\ & \le C{e}^{-ϵT},\end{array}$which tends to 0 as $T\to \mathrm{\infty }$.

Concerning the continuity properties listed in Definition 2.2.(ii) we note that for each $k\ge 0$ and $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$ we have that $\begin{array}{rl}|{\mathrm{\Psi }}^{v,u}(t^{\prime },b^{\prime })-{\mathrm{\Psi }}^{v,u}(t,b)|& \le |{\mathrm{\Psi }}_T^{v,u}(t^{\prime },b^{\prime })-{\mathrm{\Psi }}_T^{v,u}(t,b)|+|{\mathrm{\Psi }}^{v,u}(t^{\prime },b^{\prime })-{\mathrm{\Psi }}_T^{v,u}(t^{\prime },b^{\prime })|\\ & +|{\mathrm{\Psi }}^{v,u}(t,b)-{\mathrm{\Psi }}_T^{v,u}(t,b)|.\end{array}$Now, $\begin{array}{rl}& \mathbb{E}[\underset{(t,b)\in [0,\mathrm{\infty })\times U}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|\phi (v\circ (t,b)\circ u)-{\phi }_T(v\circ (t,b)\circ u)||{\mathcal{F}}_t]]\\ & \le \mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\int }_T^{\mathrm{\infty }}{e}^{-\rho \left(s\right)}\varphi (s,X_s^{v\circ u})\mathrm{d}s||{\mathcal{F}}_t]]\le e^{-ϵT}.\end{array}$and similarly $\begin{array}{rl}& \mathbb{E}[\underset{(t,b)\in [0,\mathrm{\infty })\times U}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|c(v\circ (t,b)\circ u)-{\mathbb{1}}_{[{\tau }_N\vee t<T]}c(v\circ (t,b)\circ u)||{\mathcal{F}}_t]]\\ & \le \mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}\underset{u\in {\mathcal{U}}_t^{k+1}}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[N\ge 1]}{\mathbb{1}}_{[{\tau }_N\ge T]}{e}^{-\rho \left({\tau }_N\right)}|\ell ({\tau }_N,X_{{\tau }_N}^{v\circ [u{]}_{N-1}},{\text{β}}_N)||{\mathcal{F}}_t]]\le e^{-ϵT}.\end{array}$This implies that $\begin{array}{r}\mathbb{P}[\underset{(t,b)\in [0,\mathrm{\infty })\times U}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\mathrm{\Psi }}^{v,u}(t,b)-{\mathrm{\Psi }}_T^{v,u}(t,b)||{\mathcal{F}}_t]\ge e^{-ϵT/2}]\le C{e}^{-ϵT/2}\end{array}$and the Borel-Cantelli lemma gives that $\begin{array}{r}\underset{(t,b)\in {\mathbb{R}}_{+}\times U}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[|{\mathrm{\Psi }}^{v,u}(t,b)-{\mathrm{\Psi }}_T^{v,u}(t,b)||{\mathcal{F}}_t]\to 0,\end{array}$$\mathbb{P}$-a.s., as $T\to \mathrm{\infty }$ for all $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$.

By Lemma 5.6 and uniform convergence it follows from Lemma 2.4.(d) that $J^{\ast }:=\underset{T\to \mathrm{\infty }}{lim}{J}_T^{\ast }\in {\mathcal{H}}_{\mathbb{F}}$. The desired result now follows by Lemma 2.4.(a) while noting that by the construction of ℓ in Assumption 5.1.(v), a simplified version of Lemma 5.6 (without having to consider maximization over u) applied to each of the ${\ell }_i$ gives that there is an $h\in {\mathcal{H}}_{\mathbb{F}}$ such that $h(\tau ,b)=-\mathbb{E}\left[\ell \right(\tau ,X_{\tau }^v,b\left)|{\mathcal{F}}_{\tau }\right]$, $\mathbb{P}$-a.s. (with an exception set that is independent of b).

Remark 5.4

In a perfect information setting, i.e.  when $\mathbb{F}=\mathbb{G}$, we note that $\ell (t,x,b)$ can be taken to be any upper semi-continuous function in b that satisfies the remaining properties of polynomial growth and local Lipschitz continuity.

5.3. The random horizon setting

We turn instead to the reward (30) $\begin{array}{r}{J}^{\eta }\left(u\right)=\mathbb{E}[{\int }_0^{\eta }{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t+e^{-\rho \left(\eta \right)}\psi (\eta ,X_{\eta }^{[u{]}_{N(\eta -)}})-\sum _{j=1}^N{e}^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{\nu ,j-1},{\text{β}}_j)].\end{array}$(30) where η is a $\mathbb{G}$-stopping and $N(\eta -):=sup\{j:{\tau }_j<\eta \}\vee 0$. A notable convention applied in (30(30) $\begin{array}{r}{J}^{\eta }\left(u\right)=\mathbb{E}[{\int }_0^{\eta }{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t+e^{-\rho \left(\eta \right)}\psi (\eta ,X_{\eta }^{[u{]}_{N(\eta -)}})-\sum _{j=1}^N{e}^{-\rho \left({\tau }_j\right)}\ell ({\tau }_j,X_{{\tau }_j}^{\nu ,j-1},{\text{β}}_j)].\end{array}$(30) ) is that the terminal reward disregards interventions made at the horizon. This is natural from an applications perspective as it is generally to late to intervene at a default in a financial setting or at the failure of a unit in an engineering application.

In addition to the requirements listed in Assumption 5.1, we make the following assumptions:

Assumption 5.8

The terminal reward $\psi :{\mathbb{R}}_{+}\times {\mathbb{R}}^d\to \mathbb{R}$ is Borel-measurable, satisfies the growth condition $\begin{array}{r}|\psi (t,x)|\le C(1+|x{|}^q)\end{array}$and there is a non-decreasing continuous function $C_{\psi }:{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ such that for all $L\ge 0$, we have $\begin{array}{r}|\psi (t,x)-\psi (t,x^{\prime })|\le C_{\psi }\left(L\right)|x-x^{\prime }|\end{array}$whenever $|x|\vee |x^{\prime }|\le L$. Moreover, if there is a sequence $({\theta }_j{)}_{j\ge 0}$ in ${\mathcal{T}}^f$ such that ${\theta }_j\nearrow \eta$ on some set $B\in \mathcal{G}$, then there is a $\mathbb{P}$-null set $\mathcal{N}$ such that on $B\setminus \mathcal{N}$ we have for every $(y,b)\in \mathbb{D}\times U$ that (31) $\begin{array}{r}\psi (\eta ,y_{\eta })\ge \psi (\eta ,\mathrm{\Gamma }(\eta ,(y_s{)}_{s\le \eta },b))\end{array}$(31)

We introduce the following hypothesis:

Hypothesis. [Disc.-B] The terminal reward satisfies the bound $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}\mathbb{E}[e^{-p\rho \left(\eta \right)}|\psi (\eta ,X_{\eta }^u){|}^p]<\mathrm{\infty }.\end{array}$Furthermore, for each $k\ge 0$ there is an $ϵ>0$ such that for all $T\ge T^{\prime }$ for some $T^{\prime }>0$ we have $\begin{array}{r}\mathbb{E}[\underset{t\in [0,\mathrm{\infty })}{sup}\underset{u\in {\mathcal{U}}_t^k}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}[{\mathbb{1}}_{[\eta \ge T]}{e}^{-\rho \left(\eta \right)}|\psi (\eta ,X_{\eta }^{v\circ u})||{\mathcal{F}}_t]]\le e^{-ϵT}\end{array}$for all $v\in {\mathcal{U}}^{\mathrm{l}\mathrm{i}\mathrm{m}}$.

We have the following extension of Proposition 5.7.

Proposition 5.9

Under Hypotheses Disc.-A and Disc.-B there is a $u^{\ast }\in {\mathcal{U}}^f$ such that $J^{\eta }\left(u^{\ast }\right)\ge J^{\eta }\left(u\right)$ for all $u\in {\mathcal{U}}^f$. Furthermore, $u^{\ast }$ is given by the recursion (8(8) $\begin{array}{r}{\tau }_j^{\ast }:=inf\{s\ge {\tau }_{j-1}^{\ast }:Y_s^{[u^{\ast }{]}_{j-1}}=\underset{b\in U}{sup}\{-c_{s,b}^{[u^{\ast }{]}_{j-1}}+Y_s^{[u^{\ast }{]}_{j-1}\circ (s,b)}\}\},\end{array}$(8) )–(9(9) $\begin{array}{r}{\text{β}}_j^{\ast }\in \underset{b\in U}{\arg max}\{-c_{{\tau }_j^{\ast },b}^{[u^{\ast }{]}_{j-1}}+Y_{{\tau }_j^{\ast }}^{[u^{\ast }{]}_{j-1}\circ ({\tau }_j^{\ast },b)}\}.\end{array}$(9) ), with $\begin{array}{r}\phi \left(u\right):={\int }_0^{\eta }{e}^{-\rho \left(t\right)}\varphi (t,X_t^u)\mathrm{d}t+e^{-\rho \left(\eta \right)}\psi (\eta ,X_{\eta }^{[u{]}_{N(\eta -)}})\end{array}$and $\begin{array}{r}c(u\circ (t,b\left)\right):=e^{-\rho (t\vee {\tau }_N)}\ell (t\vee {\tau }_N,X_{t\vee {\tau }_N}^u,b).\end{array}$If, in addition η is an $\mathbb{F}$-stopping time, then ${\tau }_j^{\ast }<\eta$ for all $1\le j\le N^{\ast }$.

Proof.

We note that all details in the proof of Proposition 5.7 transfer immediately to this situation except for the quasi-left upper semi-continuity property in the definition of ${\mathcal{H}}_{\mathbb{F}}$ (Definition 2.1). We thus assume that there is a sequence non-decreasing sequence $({\theta }_j{)}_{j\ge 0}$ of stopping times such that ${\theta }_j\nearrow \theta \in \mathcal{T}$. When $\theta <\eta$, $\mathbb{P}$-a.s. left-continuity at θ follows by Lemma 5.3 and the local Lipschitz property of ψ and when $\theta >\eta$, $\mathbb{P}$-a.s. left-continuity at θ is immediate. We thus assume that ${\theta }_j\nearrow \eta$ on some measurable set $B\subset \mathrm{\Omega }$.

Then, we have $\begin{array}{rl}& {\mathbb{1}}_B\left(\phi \right(v\circ ({\theta }_j,b)\circ u)-\phi (v\circ (\theta ,b)\circ u\left)\right)\\ & \le {\int }_{{\theta }_j}^{\theta }{e}^{-\rho \left(t\right)}|\varphi (t,X_t^{v\circ ({\theta }_j,b)\circ u})-\varphi (t,X_t^{v\circ (\theta ,b)\circ u})|\mathrm{d}t\\ & +{\mathbb{1}}_B{e}^{-\rho \left(\eta \right)}\left(\psi \right(\eta ,X_{\eta }^{v\circ ({\theta }_j,b)\circ u})-\psi (\eta ,X_{\eta }^v\left)\right),\end{array}$where the first term on the right-hand side tends to zero, $\mathbb{P}$-a.s. Concerning the second term we have $\begin{array}{rl}& {\mathbb{1}}_B{e}^{-\rho \left(\eta \right)}\left(\psi \right(\eta ,X_{\eta }^{v\circ ({\theta }_j,b)\circ u})-\psi (\eta ,X_{\eta }^v\left)\right)\\ & \le {\mathbb{1}}_B{e}^{-\rho \left(\eta \right)}\left(\psi \right(\eta ,{\mathrm{\Gamma }}_{{\text{β}}_{N(\eta -)}}\circ \cdots \circ {\mathrm{\Gamma }}_{{\text{β}}_1}\circ {\mathrm{\Gamma }}_b(\eta ,X_{\eta }^v)-\psi (\eta ,X_{\eta }^v))\\ & +e^{-\rho \left(\eta \right)}|\psi (\eta ,X_{\eta }^{v\circ ({\theta }_j,b)\circ [u{]}_{N(\eta -)}})-\psi (\eta ,X_{\eta }^{v\circ (\theta ,b)\circ [u{]}_{N(\eta -)}})|,\end{array}$where ${\mathrm{\Gamma }}_b(\cdot ,\cdot ):=\mathrm{\Gamma }(\cdot ,\cdot ,b)$ and ° denotes composition of functions. The first term on the right-hand side is $\mathbb{P}$-a.s. non-positive by Assumption 5.8 and the last term tends to zero, $\mathbb{P}$-a.s., by the local Lipschitz property of ψ and Lemma 5.3 in combination with Proposition 5.2, the polynomial growth condition on ψ and Hypothesis Disc.-B.

The last assertion follows by noting that since c>0 it will never be optimal to intervene at times greater than or equal to η.

We note the following distinction between the finite (deterministic) horizon and the random horizon settings:

Remark 5.5

In the case when $\mathbb{P}(\eta =T)=1$ for some $T\ge 0$ it follows from the proof of Proposition 5.9 that we can relax (31(31) $\begin{array}{r}\psi (\eta ,y_{\eta })\ge \psi (\eta ,\mathrm{\Gamma }(\eta ,(y_s{)}_{s\le \eta },b))\end{array}$(31) ) to $\begin{array}{r}\psi (T,y_T)\ge \psi (T,\mathrm{\Gamma }(T,(y_s{)}_{s\le T},b))-\ell (T,y_T,b).\end{array}$

To see that there is an actual distinction here consider the following example:

Example 5.10

We let $\mathcal{F}$ be the trivial σ-algebra $\{\mathrm{\varnothing },\mathrm{\Omega }\}$ and assume that $\mathbb{P}(\eta =x)=\{\begin{array}{ll}0.5,& x=1\\ 0.5,& x=2\end{array}$. We take $U:=\left\{1\right\}$ and set $X_t={\mathbb{1}}_{[{\tau }_1,\mathrm{\infty })}\left(t\right)$. Then, with the rewards $\varphi \equiv 0$, $\psi (t,x)=x{e}^{|t-1|}$, the intervention cost $\ell (t,x,b)=e^{|t-1|}$ and the discount $\rho \equiv 0$, we get $\begin{array}{r}\underset{u\in {\mathcal{U}}^f}{sup}{J}^{\eta }\left(u\right)=0.5(e^1-1),\end{array}$but there is no control that attains this value.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Swedish Energy Agency through grant number 48405-1

Notes

1 Throughout, we let ${\mathbb{R}}_{+}:=[0,\mathrm{\infty })$

2 We let $\wedge$ (resp. $\vee$) to denote minimum (resp. maximum), so that $x\wedge y:=min(x,y)$.

3 Throughout, we generally suppress dependence on ω and refer to $h\in {\mathcal{H}}_{\mathbb{F}}$ as a map $(t,b)\mapsto h(t,b)$.

4 Requiring that p>2 is for notational convenience only and can easily be loosened to p>1.

5 Throughout, C will denote a generic positive constant that may change value from line to line.

6 By definition $\{N^{\ast }>K\}=\{{\tau }_{K+1}^{\ast }<\mathrm{\infty }\}$, which belongs to ${\mathcal{F}}_{{\tau }_{K+1}^{\ast }}$

Appendices

Appendix 1. Quasi-left continuity

A càdlàg  process $(X_t:t\ge 0)$ is quasi-left continuous if for each predictable stopping time θ and every announcing sequence of stopping times ${\theta }_k\nearrow \theta$ we have $X_{\theta -}:=\underset{k\to \mathrm{\infty }}{lim}{X}_{{\theta }_k}=X_{\theta }$, $\mathbb{P}$-a.s. Similarly, X is quasi-left upper semi-continuous if $X_{\theta -}\le X_{\theta }$, $\mathbb{P}$-a.s. A filtration is quasi-left continuous if ${\mathcal{F}}_{\theta }={\mathcal{F}}_{\theta -}$ for every predictable stopping time θ.

Appendix 2. The Snell envelope

In this section we gather some useful results concerning the Snell envelope. Recall that a progressively measurable process X is of class [D] if the set of random variables $\{X_{\tau }:\tau \in {\mathcal{T}}^f\}$ is uniformly integrable.

Theorem A.1

The Snell envelope

Let $X=(X_t{)}_{t\ge 0}$ be an $\mathbb{F}$-adapted, $\mathbb{R}$-valued, càdlàg  process of class $[$D$]$. Then there exists a unique $($ up to indistinguishability $)$ , $\mathbb{R}$-valued càdlàg  process $Z=(Z_t{)}_{t\ge 0}$ called the Snell envelope of X, such that Z is the smallest supermartingale that dominates X. Moreover, the following holds (with $\mathrm{\Delta }{X}_t:=X_t-X_{t-}$):

1. For any stopping time η, (A1) $\begin{array}{r}{Z}_{\eta }=\underset{\tau \in {\mathcal{T}}_{\eta }}{\mathrm{e}\mathrm{s}\mathrm{s}sup}\mathbb{E}\left[X_{\tau }|{\mathcal{F}}_{\eta }\right].\end{array}$(A1)

2. The Doob-Meyer decomposition of the supermartingale Z implies the existence of a triple $(M,K^c,K^d)$ where $(M_t:t\ge 0)$ is a uniformly integrable right-continuous martingale, $(K_t^c:t\ge 0)$ is a non-decreasing, predictable, continuous process with $K_0^c=0$ and $(K_t^d:t\ge 0)$ is non-decreasing purely discontinuous predictable with $K_0^d=0$, such that (A2) $\begin{array}{r}{Z}_t=M_t-K_t^c-K_t^d.\end{array}$(A2) Furthermore, $\{\mathrm{\Delta }{K}_t^d>0\}\subset \{\mathrm{\Delta }{X}_t<0\}\cap \{Z_{t-}=X_{t-}\}$ for all $t\ge 0$.

3. Let $\eta \in \mathcal{T}$ be given and assume that for any predictable $\theta \in {\mathcal{T}}_{\eta }$ and any increasing sequence $\{{\theta }_j{\}}_{j\ge 0}$ with ${\theta }_j\in {\mathcal{T}}_{\eta }^f$ and $\underset{j\to \mathrm{\infty }}{lim}{\theta }_j=\theta$, $\mathbb{P}$-a.s, we have $\underset{j\to \mathrm{\infty }}{lim sup}{X}_{{\theta }_j}\le X_{\theta }$, $\mathbb{P}$-a.s. Then, the stopping time ${\tau }_{\eta }^{\ast }$ defined by ${\tau }_{\eta }^{\ast }:=inf\{s\ge \eta :Z_s=X_s\}$ (with the convention that $inf\mathrm{\varnothing }=\mathrm{\infty }$) is optimal after η, i.e.  $\begin{array}{r}{Z}_{\eta }=\mathbb{E}\left[X_{{\tau }_{\eta }^{\ast }}|{\mathcal{F}}_{\eta }\right].\end{array}$Furthermore, in this setting the Snell envelope, Z, is quasi-left continuous, i.e.  $K^d\equiv 0$.

4. Let $X^k$ be a sequence of càdlàg  processes converging increasingly and pointwisely to the càdlàg  process X and let $Z^k$ be the Snell envelope of $X^k$. Then the sequence $Z^k$ converges increasingly and pointwisely to a process Z and Z is the Snell envelope of X.

In the above theorem, (i)–(iii) are standard results and proofs can be found in, for example, [12,14]. A finite horizon version of statement (iv), which extends trivially to infinite horizon, was proved in [11].

Appendix 3. The section and projection theorems

In this section we recall two fundamental results from the general theory of stochastic processes, namely the measurable selection and the optional projection theorems.

We consider a complete filtered probability space $(\mathrm{\Omega },\mathcal{F},\mathbb{F},\mathbb{P})$, with $\mathbb{F}:=\left\{{\mathcal{F}}_t\right\}$ a right-continuous filtration. For any space E, we define the projection of a set $A\subset \mathrm{\Omega }\times E$ onto Ω as ${\pi }_{\mathrm{\Omega }}\left(A\right):=\{\omega \in \mathrm{\Omega }:\mathrm{\exists }x\in E,(\omega ,x)\in A\}$.

Theorem A.2

Measurable projection

Let E be a locally compact Polish space. For every $A\in \mathcal{F}\otimes \mathcal{B}\left(E\right)$ the set ${\pi }_{\mathrm{\Omega }}\left(A\right)$ is $\mathcal{F}$-measurable.

A proof can be found in, e.g. [18] (see the proof of Theorem 2.10) or [6] Chapter III. In particular we need the following corollary result:

Corollary A.3

Let $h(\omega ,x)$ be a real valued, measurable function defined on the product space $(\mathrm{\Omega }\times {\mathbb{R}}^m,\mathcal{F}\otimes \mathcal{B}({\mathbb{R}}^m\left)\right)$. Then for all $A\in \mathcal{F}\otimes \mathcal{B}\left({\mathbb{R}}^m\right)$, the function $\begin{array}{r}g\left(\omega \right):=\underset{x\in {\mathbb{R}}^m}{sup}\left\{h\right(\omega ,x):(\omega ,x)\in A\}\end{array}$(with the convention $sup\mathrm{\varnothing }=-\mathrm{\infty }$) is $\mathcal{F}$-measurable.

Proof.

For each $K\in \mathbb{R}$ we have $\left\{g\right(\omega )>K\}={\pi }_{\mathrm{\Omega }}(A\cap h^{-1}((K,\mathrm{\infty }]\left)\right)$. Now, since h is measurable, the set $A\cap h^{-1}\left(\right(K,\mathrm{\infty }\left]\right)$ is in $\mathcal{F}\otimes \mathcal{B}\left({\mathbb{R}}^m\right)$ and the result follows by the measurable projection theorem.

Theorem A.4

Measurable selection

Let $(E,\mathcal{E})$ be a Borel space with $\mathcal{E}:=\mathcal{B}\left(E\right)$. For every $A\in \mathcal{F}\otimes \mathcal{B}\left(E\right)$ there is a $\mathcal{F}$-measurable r.v. β taking values in $\overline{E}:=E\cup \left\{\mathrm{\partial }\right\}$ (with ∂ a cemetery point) such that $\begin{array}{r}\left\{\right(\omega ,\text{β}\left(\omega \right))\in \mathrm{\Omega }\times E\}\subset A\mathrm{a}\mathrm{n}\mathrm{d}\{\omega \in \mathrm{\Omega }:\text{β}(\omega )\in E\}={\pi }_{\mathrm{\Omega }}\left(A\right).\end{array}$

This is a standard result and a proof can be found in [18] (Theorem 2.20) (see also Chapter 7 in [4] where several extensions are given). In particular we need the following well known corollary result:

Corollary A.5

Let $h(\omega ,x)$ be a measurable function defined on the product space $(\mathrm{\Omega }\times {\mathbb{R}}^m,\mathcal{F}\otimes \mathcal{B}({\mathbb{R}}^m\left)\right)$, such that for $\mathbb{P}$-almost every ω the map $x\mapsto h(\omega ,x)$ is upper semi-continuous. Then, with U a compact subset of ${\mathbb{R}}^m$, there exists a $\mathcal{F}$-measurable r.v. β such that $\begin{array}{r}h(\omega ,\text{β}(\omega \left)\right)=\underset{x\in {\mathbb{R}}^n}{sup}\left\{h\right(\omega ,x):(\omega ,x)\in \mathrm{\Omega }\times U\},\end{array}$$\mathbb{P}$-a.s.

Proof.

Since $A:=\mathrm{\Omega }\times U\in \mathcal{F}\otimes \mathcal{B}\left(E\right)$ (where now $E={\mathbb{R}}^m$) the function $g\left(\omega \right)=\underset{x\in E}{sup}\left\{h\right(\omega ,x):(\omega ,x)\in A\}$ is $\mathcal{F}$-measurable. Furthermore, as h is $\mathcal{F}\otimes \mathcal{B}\left(E\right)$-measurable, the set $B:=\left\{\right(\omega ,x)\in \mathrm{\Omega }\times U:h(\omega ,x)=g(\omega \left)\right\}$ is in $\mathcal{F}\otimes \mathcal{B}\left(E\right)$. Now, by Theorem A.4 there is a $\mathcal{F}$-measurable $\overline{E}$-valued r.v. β such that $\left\{\right(\omega ,\text{β}\left(\omega \right))\in \mathrm{\Omega }\times E\}\subset B$ and $\{\omega \in \mathrm{\Omega }:\text{β}(\omega )\in E\}={\pi }_{\mathrm{\Omega }}\left(B\right)$. As U is compact and $b\mapsto h(\omega ,b)$ is u.s.c. on $\mathrm{\Omega }\setminus \mathcal{N}$ with $\mathbb{P}\left(\mathcal{N}\right)=0$, we have that $B^{\omega }:=\{b\in U:(\omega ,b)\in B\}=\{b\in U:h(\omega ,b)=g(\omega \left)\right\}\ne \mathrm{\varnothing }$ for all $\omega \in \mathrm{\Omega }\setminus \mathcal{N}$ and, hence, $\mathbb{P}\left({\pi }_{\mathrm{\Omega }}\right(B\left)\right)=1$.

The last result that we need is the optional projection theorem.

Theorem A.6

Optional projection

Assume that $(X_t:t\ge 0)$ is a measurable process (not necessarily adapted to the filtration $\mathbb{F}$) with $\mathbb{E}\left[|X_{\tau }|\right]<\mathrm{\infty }$ for all stopping times $\tau \in \mathcal{T}$, then there exists a unique optional process ${(}^o{X}_t:t\ge 0)$ such that $\begin{array}{r}{\mathbb{1}}_{[\tau <\mathrm{\infty }]}{}^o{X}_{\tau }=\mathbb{E}[{\mathbb{1}}_{[\tau <\mathrm{\infty }]}{X}_{\tau }|{\mathcal{F}}_{\tau }],\end{array}$for all stopping times $\tau \in \mathcal{T}$. If, furthermore, X is càdlàg  then ${}^{o}X$ is also càdlàg.

A proof of Theorem A.6 can be found in Chapter VI, pp. 103 of [7].