Personalized control system via reinforcement learning: maximizing utility based on user ratings

Abstract

In this paper, we address the design of personalized control systems, which pursue individual objectives defined for each user. To this end, a problem of reinforcement learning is formulated where an individual objective function is estimated based on the user rating on his/her current control system and its corresponding optimal controller is updated. The novelty of the problem setting is in the modelling of the user rating. The rating is modelled by a quantization of the user utility gained from his/her control system, defined by the value of the objective function at his/her control experience. We propose an algorithm of the estimation to update the control law. Through a numerical experiment, we find out that the proposed algorithm realizes the personalized control system.

Keywords:

• Personalization
• user rating
• optimal control
• reinforcement learning
• maximizing utility

1. Introduction

Robust design has been a basic design concept of control systems: common control systems are designed for various system-users such that the systems operate stably without any concern for the difference between user-environment. Beyond the robustness, “personalization” can be an advanced design concept of control systems: an individual control system is designed for each system-user such that the system operates pursuing high control performance and improving user utility [1,2]. In this paper, we address the personalization of control systems, in particular, a design methodology of the control systems that possess a function of adaptation of improving user utility.

As studied in other research fields, a key to personalization is the modelling of system-users [3,4]. For example, in [5,6], data on manual driving is used to model driver intent. In the previous works, the user modelling is addressed based on the measured data on the user actions to the control system. On the other hand, in this paper, the data on the action is not available, and instead, the result of user rating on the control system is available. The user rating is collected for every control operation and is used to estimate the private objective function of the user, and the optimal controller is updated based on the estimated objective function. By repeating the controller update, we aim at maximizing the user utility gained in the control system.

As an application of the presented personalized control system, let us imagine an automatic driving system. The system drives automatically, and the user rates the driving control system based on his/her comfort, for example, once a month. The control system accesses the result of the user rating to estimate his/her preference, modelled by a parameterized objective function. Then, the implemented control law is updated based on the estimated objective function.

In general, a problem of reinforcing system performance by learning techniques is called reinforcement learning (RL), which has been addressed in the literature. See e.g. [7] and its references therein. RL has been applied to the design of feedback control systems [8–11]. In [8,9], continuous control problems are addressed unlike standard RL problems, and the control law is updated directly. Furthermore, in [10,11], RL is combined with model predictive control, and the objective function is tuned to update the optimal control law indirectly. The main difference in this paper and the literature is the assumption on the objective function and/or reward. In the literature, the objective function is designable, while in this paper, the function is not designable and is pre-defined but hidden by a system-user.

The rest of this paper is organized as follows. In Section 2, the models of control systems and system-user are given, and the problem of the control system update is formulated. In Section 3, we propose an algorithm of estimating the user objective function, which plays a central role in personalization. In Section 4, we give a convergence analysis of the proposed algorithm. In Section 5, we present a numerical experiment using the proposed algorithm. In Section 6, the conclusion is given.

2. Problem formulation

2.1. Model of control systems

We consider a control system that is composed of a plant system and a controller, which are denoted by P and K, respectively. The plant system is modelled by the following discrete-time state space equation $P:x(k+1)=h\left(x\right(k),u(k\left)\right),$where $k\in {\mathbb{N}}_{+}$ is the discrete-time, and $x\in {\mathbb{R}}^n$ and $u\in {\mathbb{R}}^m$ are the state and control input, respectively. Symbol $h:{\mathbb{R}}^n\times {\mathbb{R}}^m\mapsto {\mathbb{R}}^n$ represents a function.

The controller is designed based on the estimate of an individual objective function, which is private and is defined for each user. Let J and $\hat{J}$ represent the objective function and its estimate, respectively. Then, the control law is described by the following optimization problem $K:\{\begin{array}{ll}{\displaystyle \underset{\mathit{u}}{min}}& \hat{J}(\mathit{x},\mathit{u})\\ \mathrm{s}.\mathrm{t}.& x(k+1)=h\left(x\right(k),u(k\left)\right),{}^{\mathrm{\forall }}k\\ & \left(x\right(k),u(k\left)\right)\in \mathcal{X}\times \mathcal{U},{}^{\mathrm{\forall }}k,\end{array}$where $\mathit{x}$ and $\mathit{u}$ are stacked vectors composed of the sequences of the state and input, respectively, i.e. $\begin{array}{rl}\mathit{x}& =\left[x\right(k+1{)}^{\mathrm{\top }},x(k+2{)}^{\mathrm{\top }},\dots ,x(k+N{)}^{\mathrm{\top }}{]}^{\mathrm{\top }},\\ \mathit{u}& =\left[u\right(k{)}^{\mathrm{\top }},u(k+1{)}^{\mathrm{\top }},\dots ,u(k+N-1{)}^{\mathrm{\top }}{]}^{\mathrm{\top }},\end{array}$and $\mathcal{X}$ and $\mathcal{U}$ are the state and input constraints, respectively. In the following discussion, $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$ represents the measured data on $\mathit{x}$ and $\mathit{u}$, obtained in a control experiment and called experience for the control system.

In addition to P and K, we should note that a system-user participates in the control system. His/her objective function is modelled by (1) $J(\mathit{x},\mathit{u})=f_0(\mathit{x},\mathit{u})+\sum _{i=1}^{\ell }{q}_i{f}_i(\mathit{x},\mathit{u}),$(1) where $f_i(\mathit{x},\mathit{u}):{\mathbb{R}}^{nN}\times {\mathbb{R}}^{mN}\mapsto {\mathbb{R}}_{+},i\in \{0,1,\dots ,\ell \}$ are non-negative functions defined by a system-designer and $q_i>0,i\in \{1,2,\dots ,\ell \}$ are weighting parameters to be estimated. As (1(1) $J(\mathit{x},\mathit{u})=f_0(\mathit{x},\mathit{u})+\sum _{i=1}^{\ell }{q}_i{f}_i(\mathit{x},\mathit{u}),$(1) ), user's objective is parameterized by $q_i,i\in \{1,2,\dots ,\ell \}$. The system-user has his/her objective function (1(1) $J(\mathit{x},\mathit{u})=f_0(\mathit{x},\mathit{u})+\sum _{i=1}^{\ell }{q}_i{f}_i(\mathit{x},\mathit{u}),$(1) ) in his/her mind, and the user's weighting parameter $q_i,i\in \{1,2,\dots ,\ell \}$ are not accessible directly for control systems update. Instead, the user rating including the information on $q_i,i\in \{1,2,\dots ,\ell \}$ is available for the update.

In this paper, we aim at maximizing the users utility achieved in the control system $(P,K)$ by updating K. To this end, the individual and private objective function J is estimated, and the control law K is updated based on the estimate $\hat{J}$. In other words, such “personalization” of the control system is achieved by accurate estimate of $q_i,i\in \{1,2,\dots ,\ell \}$. The estimation of J, i.e. that of $q_i,i\in \{1,2,\dots ,\ell \}$, is based on user rating without accessing bare data $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$ unlike the conventional works [10,11].

2.2. Model of system-user

In the problem setting, we explicitly take into account the presence of a system-user. The user rates the control system $(P,K)$ based on the utility determined by his/her experience, denoted by $J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$. An example of the user rating is realized by a questionnaire. The user gives m-grade evaluation based on his/her satisfaction with the control system as illustrated in Figure 1.

Figure 1. An example of user rating: five-grade evaluation is given to the control system, such as Excellent/Very Good/Good/Average/Poor.

Recall that in control system $(P,K)$, controller K pursues the performance in the sense of the estimated objective function $\hat{J}$, parameterized by ${\hat{q}}_i,i\in \{1,2,\dots ,\ell \}$, to give experience $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$ to the user. There exists a gap in the estimated utility $\hat{J}({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$ and true utility $J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$. We assume that the user rating depends on the gap: the user rates the control system high (low) if the gap is small (large).

To model the user rating, we define the gap in the utility gained from experience $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$ as (2) $E_J({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})=\frac{\hat{J}({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})-J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})}{J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})}.$(2) Based on (2(2) $E_J({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})=\frac{\hat{J}({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})-J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})}{J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})}.$(2) ), the user rating r is modelled by a piecewise constant function as (3) $r=\{\begin{array}{ll}{M}_1& \mathrm{i}\mathrm{f}|E_J|\in [0,a_1]\\ M_2& \mathrm{i}\mathrm{f}|E_J|\in (a_1,a_2]\\ ⋮& \\ M_{m-1}& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-2},a_{m-1}]\\ M_m& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-1},\mathrm{\infty }),\end{array}$(3)

where $M_i,i\in \{1,2,\dots ,m\}$ are positive constants satisfying $M_1>M_2>\cdots >M_m$, and $a_i,i\in \{1,2,\dots ,m-1\}$ are also positive constants that indicate the range of existence of $|E_J|$. One can assume the rating as $(M_1,M_2,\dots ,M_5,M_6)=(100,80,\dots ,20,0)$ for six-grade evaluation. In this setting, since the value of r is quantized, controller K cannot access the exact value of $E_J({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$.

Remark 2.1

$E_J>-1$ holds since $\hat{J}>0$. This fact is used for the analysis given in Section 4.

We impose a technical assumption on the model of the system-user, denoted by H. In addition to user rating r, system-user H gives the sign of $E_J$ to the control system based on his/her experience $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$. Then, the model of H is described by (4) $\begin{array}{rl}& H:R({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})\\ & =\left\{r\right({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}),\mathrm{sgn}\left(\hat{J}\right({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})-J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\left)\right)\},\end{array}$(4) where $\mathrm{sgn}(\cdot )$ is the sign function. We see that this R is a “reward” in the reinforcement learning framework. Controller K can access reward R, which depends on user-experience $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$, to estimate $J^{\ast }$ and to update its control law.

Remark 2.2

A similar problem of estimating objective functions and/or rewards, which generate the control actions, is known as inverse reinforcement learning (IRL). See e.g. [12,13] for the problem setting and e.g. [14–16] for its applications. In most of the IRL frameworks, the control law is pre-defined and fixed, and its generating data is available for the estimation. On the other hand, in this paper, the control law is not fixed and to be updated, and the rating of a system-user, who is not included in the control loop, is available.

The block diagram of the control system with the user rating is illustrated in Figure 2. In the figure, the blue line connecting the controller and the plant indicates the loop of the control operation, while the red line connecting the user, controller, and plant indicates the loop of the controller update.

Figure 2. Personalized control system updated based on user rating.

2.3. Problem of controller update

Control system $(P,K)$ is updated based on user rating $R({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$. The flow of the update is given as follows.

Flow of controller update

1. The control system at version τ, denoted by $(P,K_{\tau })$, is operated, and the user gains experience $\{{\mathit{x}}_{\tau }^{\left(d\right)},{\mathit{u}}_{\tau }^{\left(d\right)}\}$.

2. The user gives his/her rating $R({\mathit{x}}_{\tau }^{\left(d\right)},{\mathit{u}}_{\tau }^{\left(d\right)})$, i.e. a reward for $(P,K_{\tau })$ is given to the controller.

3. Parameters $q_i,i\in \{1,2,\dots ,\ell \}$ are estimated.

4. Control law K is updated based on the estimated objective function ${\hat{J}}_{\tau }$.

5. The version of the control system is updated as $\tau \leftarrow \tau +1$, and go to Stage 1.

Note that control law K is uniquely determined once the parameter estimation is performed. This implies the essence of the controller update is the parameter estimation, addressed at Stage 3. The estimation problem is given as follows:

Problem 2.1

Given $R({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$, estimate $q_i^{\ast },i\in \{1,2,\dots ,\ell \}$.

3. Parameter estimation algorithm

In this section, we propose an algorithm of estimating parameters that characterize the user objective function given in (1(1) $J(\mathit{x},\mathit{u})=f_0(\mathit{x},\mathit{u})+\sum _{i=1}^{\ell }{q}_i{f}_i(\mathit{x},\mathit{u}),$(1) ). To simplify the discussion, the objective function is characterized by only two parameters $q_1$ and $q_2$, i.e. $J(\mathit{x},\mathit{u})=f_0(\mathit{x},\mathit{u})+q_1{f}_1(\mathit{x},\mathit{u})+q_2{f}_2(\mathit{x},\mathit{u}).$The following discussion and the derived algorithm are extended to more general ℓ-parameters cases in a straightforward manner.

We aim at deriving the algorithm of estimating the parameters. Since the user rating is modelled by a quantized function as (3(3) $r=\{\begin{array}{ll}{M}_1& \mathrm{i}\mathrm{f}|E_J|\in [0,a_1]\\ M_2& \mathrm{i}\mathrm{f}|E_J|\in (a_1,a_2]\\ ⋮& \\ M_{m-1}& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-2},a_{m-1}]\\ M_m& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-1},\mathrm{\infty }),\end{array}$(3) ), the parameter estimation can be reduced to a class of set-membership estimation [17] as studied for state estimation problems [18,19]. Section 3.1 is devoted to the estimation of a parameter region. In Section 3.2, the parameter estimate is given from the region, and the estimation algorithm is presented.

3.1. Estimate of parameter region

Recall first that user rating R, given in (4(4) $\begin{array}{rl}& H:R({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})\\ & =\left\{r\right({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}),\mathrm{sgn}\left(\hat{J}\right({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})-J^{\ast }({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\left)\right)\},\end{array}$(4) ), includes rough information on his/her utility gained from experience $\{{\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}\}$. We suppose that $r=M_s$ holds for the experience, which implies the user gives s-grade for a current control system. Further supposing $\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})\ge 0$, we have the following inequality (5) $a_{s-1}<E_{J^{\left(d\right)}}\le a_s,$(5) where $E_{J^{\left(d\right)}}:=E_J({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)})$. Further, we let $f_i^{\left(d\right)}:=f_i({\mathit{x}}^{\left(d\right)},{\mathit{u}}^{\left(d\right)}),i\in \{0,1,2\}$. Then, we see that $E_{J^{\left(d\right)}}$ is described by (6) $E_{J^{\left(d\right)}}=\frac{({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})}{f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)}},$(6) where ${\hat{q}}_{1,\tau -1}$ and ${\hat{q}}_{2,\tau -1}$ are the estimated parameters at $(\tau -1)$th trial on the controller update. Then, by substituting (6(6) $E_{J^{\left(d\right)}}=\frac{({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})}{f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)}},$(6) ) into (5(5) $a_{s-1}<E_{J^{\left(d\right)}}\le a_s,$(5) ), we have (7) $\begin{array}{rl}& a_{s-1}(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & <({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & \le a_s(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)}).\end{array}$(7) The set of the linear inequalities in (7(7) $\begin{array}{rl}& a_{s-1}(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & <({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & \le a_s(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)}).\end{array}$(7) ) represents the existence region of $q_1^{\ast }$ and $q_2^{\ast }$.

In a similar manner, supposing $\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})<0$, we see that (8) $\begin{array}{rl}& -a_s(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & \le ({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & <-a_{s-1}(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\end{array}$(8) holds.

Consider again the τth trial on the controller update for derive the parameter estimate algorithm: a control operation is performed, and the system-user rates the control system $(P,K_{\tau })$ based on his/her experience $\{{\mathit{x}}_{\tau }^{\left(d\right)},{\mathit{u}}_{\tau }^{\left(d\right)}\}$. We suppose here that in the rating $R({\mathit{x}}_{\tau }^{\left(d\right)},{\mathit{u}}_{\tau }^{\left(d\right)})$, $r({\mathit{x}}_{\tau }^{\left(d\right)},{\mathit{u}}_{\tau }^{\left(d\right)})=M_s$ holds, i.e. the user rates the system by s-grade. Then, letting (9) ${\mathcal{S}}_{\tau }=\left\{\right(q_1^{\ast },q_2^{\ast }\left)|\begin{array}{ll}\left(7\right)& \mathrm{i}\mathrm{f}\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})\ge 0\\ \left(8\right)& \mathrm{i}\mathrm{f}\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})<0\end{array}\right\}.$(9) Recalling the user rating given at the first to $(\tau -1)$th control operations, we show that $(q_1^{\ast },q_2^{\ast })\in {\mathcal{Q}}_{\tau },$where (10) ${\mathcal{Q}}_{\tau }={\mathcal{S}}_0\cap {\mathcal{S}}_1\cap {\mathcal{S}}_2\cap \cdots \cap {\mathcal{S}}_{\tau }$(10) and ${\mathcal{S}}_0$ the initial guess on the parameter region. The update of the parameter existence region is illustrated in Figure 3. In the figure, the region enclosed by the red line is ${\mathcal{S}}_0$, and the region enclosed by the blue line is ${\mathcal{S}}_1$. The coloured area represents the parameter existence region ${\mathcal{Q}}_1$. In this way, we contract the parameter existence region by taking the intersection repeatedly. In the next subsection, we define the parameter estimate $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })$ from $Q_{\tau }$.

Figure 3. Estimate of parameter existence region.

3.2. Update method of estimated parameters

One can define the estimate $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })$ by the “centre” of region ${\mathcal{Q}}_{\tau }$. A drawback of taking the centre is its complexity: when the number of the controller update increases, i.e. τ increases, it is difficult to find the centre from ${\mathcal{Q}}_{\tau }$ of (10(10) ${\mathcal{Q}}_{\tau }={\mathcal{S}}_0\cap {\mathcal{S}}_1\cap {\mathcal{S}}_2\cap \cdots \cap {\mathcal{S}}_{\tau }$(10) ), which is a polytope. To find the estimate $({\hat{q}}_{2,\tau },{\hat{q}}_{2,\tau })$ in a computationally tractable way, we apply an approximation of ${\mathcal{Q}}_{\tau }$. Let ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$ be the rectangle region that approximates ${\mathcal{Q}}_{\tau }$, and is defined by (11) ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }=\left\{\right(q_1^{\ast },q_2^{\ast })|q_i^{\ast }\in [{\underset{\_}{\mathit{q}}}_{i,\tau },{\overline{q}}_{i,\tau }],i\in \{1,2\left\}\right\},$(11) where ${\underset{\_}{\mathit{q}}}_{i,\tau }$ and ${\overline{q}}_{i,\tau }$ are defined by ${\underset{\_}{\mathit{q}}}_{i,\tau }=\underset{q_i\in {\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau -1}\cap {\mathcal{S}}_{\tau }}{min}{q}_i,{\overline{q}}_{i,\tau }=\underset{q_i\in {\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau -1}\cap {\mathcal{S}}_{\tau }}{max}{q}_i,$respectively. Note that region ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$ is an “outer” approximation of ${\mathcal{Q}}_{\tau }$, i.e. it holds that ${\mathcal{Q}}_{\tau }\subseteq {\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }.$By taking the approximation of ${\mathcal{Q}}_{\tau }$ as (11(11) ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }=\left\{\right(q_1^{\ast },q_2^{\ast })|q_i^{\ast }\in [{\underset{\_}{\mathit{q}}}_{i,\tau },{\overline{q}}_{i,\tau }],i\in \{1,2\left\}\right\},$(11) ) at every controller update τ, we can find the estimate $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })$ in a simplified manner as (12) $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })=C\left({\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }\right),$(12) where $C(\cdot )$ represents the centre of gravity, i.e. ${\hat{q}}_{1,\tau }=\frac{{\underset{\_}{\mathit{q}}}_{1,\tau }+{\overline{q}}_{1,\tau }}{2},{\hat{q}}_{2,\tau }=\frac{{\underset{\_}{\mathit{q}}}_{2,\tau }+{\overline{q}}_{2,\tau }}{2}.$The approximation of the parameter existence region and parameter estimate in (12(12) $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })=C\left({\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }\right),$(12) ) are illustrated in Figure 4. In the figure, the black dotted line represents the rectangle region ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},1}$ while the blue triangle represents the estimated parameters $({\hat{q}}_{1,1},{\hat{q}}_{2,1})$.

Figure 4. Outer approximation of ${\mathcal{Q}}_{\tau }$ to define ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$, and the centre of ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$ to define $({\hat{q}}_{1,\tau },{\hat{q}}_{2,\tau })$.

Finally, the algorithm of estimating parameters $\{q_1^{\ast },q_2^{\ast }\}$ is summarized in Algorithm 1.

4. Analysis of algorithm

In this section, we address the convergence analysis of the proposed algorithm. We present the following theorem, which states the contraction of region ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$.

Theorem 4.1

In Algorithm 1, it holds that (13) $vol\left({\mathcal{Q}}_{rect,\tau +1}\right)<vol\left({\mathcal{Q}}_{rect,\tau }\right),$(13) where $\mathrm{v}\mathrm{o}\mathrm{l}\left(S\right)$ represents the volume of S.

Proof.

We consider that by the user rating, (14) $a_{s-1}<|E_{J^{\left(d\right)}}|\le a_s$(14) holds for some $s\in \{1,2,\dots ,m-1\}$ and ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau +1}$ is obtained as $\begin{array}{rl}& {\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau +1}\\ & =\left\{\right(q_1^{\ast },q_2^{\ast })|q_i^{\ast }\in [{\underset{\_}{\mathit{q}}}_{i,\tau +1},{\overline{q}}_{i,\tau +1}],i\in \{1,2\left\}\right\}.\end{array}$Consider here the following two cases: $\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})>0$ and $\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})<0$ in user rating R. First, suppose $\mathrm{s}\mathrm{g}\mathrm{n}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})>0$ implying that (14(14) $a_{s-1}<|E_{J^{\left(d\right)}}|\le a_s$(14) ) is reduced to $a_{s-1}<E_{J^{\left(d\right)}}\le a_s$.

To prove (13(13) $vol\left({\mathcal{Q}}_{rect,\tau +1}\right)<vol\left({\mathcal{Q}}_{rect,\tau }\right),$(13) ), we show that one of the following four conditions holds. (15) $\begin{array}{rl}& \left(\mathrm{i}\right){\underset{\_}{\mathit{q}}}_{1,\tau }<{\underset{\_}{\mathit{q}}}_{1,\tau +1},\left(\mathrm{i}\mathrm{i}\right){\overline{q}}_{1,\tau }>{\overline{q}}_{1,\tau +1},\\ & \left(\mathrm{i}\mathrm{i}\mathrm{i}\right){\underset{\_}{\mathit{q}}}_{2,\tau }<{\underset{\_}{\mathit{q}}}_{2,\tau +1},\left(\mathrm{i}\mathrm{v}\right){\overline{q}}_{2,\tau }>{\overline{q}}_{2,\tau +1}.\end{array}$(15) The graphical interpretation of conditions (i)–(iv) is given in Figure 5. We see that (i) holds if and only if line $E_{J^{\left(d\right)}}=a_s$ intersects segment $\left\{\right(q_1^{\ast },q_2^{\ast })\mid q_1^{\ast }\in ({\underset{\_}{\mathit{q}}}_{1,\tau },{\overline{q}}_{1,\tau }),q_2^{\ast }={\overline{q}}_{2,\tau }\}$, which is a part of the boundary of ${\mathcal{Q}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t},\tau }$. In the same way as the derivation of (7(7) $\begin{array}{rl}& a_{s-1}(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & <({\hat{q}}_{1,\tau -1}{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau -1}{f}_2^{\left(d\right)})-(q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)})\\ & \le a_s(f_0^{\left(d\right)}+q_1^{\ast }{f}_1^{\left(d\right)}+q_2^{\ast }{f}_2^{\left(d\right)}).\end{array}$(7) ) from (5(5) $a_{s-1}<E_{J^{\left(d\right)}}\le a_s,$(5) ), the condition for the intersection is described by ${\left(\mathrm{i}\right)}^{\prime }{\underset{\_}{\mathit{q}}}_{1,\tau }{f}_1^{\left(d\right)}+{\overline{q}}_{2,\tau }{f}_2^{\left(d\right)}<X,$where $X=\frac{{\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}-a_s{f}_0^{\left(d\right)}}{1+a_s}.$In a similar manner, (ii)–(iv) are equivalently reduced to $\begin{array}{rl}& {\left(\mathrm{i}\mathrm{i}\right)}^{\prime }{\underset{\_}{\mathit{q}}}_{1,\tau }{f}_1^{\left(d\right)}+{\overline{q}}_{2,\tau }{f}_2^{\left(d\right)}>Y,\\ & {\left(\mathrm{i}\mathrm{i}\mathrm{i}\right)}^{\prime }{\overline{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\underset{\_}{\mathit{q}}}_{2,\tau }{f}_2^{\left(d\right)}<X,\\ & {\left(\mathrm{i}\mathrm{v}\right)}^{\prime }{\overline{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\underset{\_}{\mathit{q}}}_{2,\tau }{f}_2^{\left(d\right)}>Y,\end{array}$respectively, where $Y=\frac{{\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}-a_{s-1}{f}_0^{\left(d\right)}}{1+a_{s-1}}.$Now, we suppose that none of (i)${}^{\prime }$–(iv)${}^{\prime }$ holds to prove (13(13) $vol\left({\mathcal{Q}}_{rect,\tau +1}\right)<vol\left({\mathcal{Q}}_{rect,\tau }\right),$(13) ) by contradiction. Then, it follows that both of the following inequalities $\begin{array}{rl}X& \le {\underset{\_}{\mathit{q}}}_{1,\tau }{f}_1^{\left(d\right)}+{\overline{q}}_{2,\tau }{f}_2^{\left(d\right)}\le Y,\\ X& \le {\overline{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\underset{\_}{\mathit{q}}}_{2,\tau }{f}_2^{\left(d\right)}\le Y\end{array}$hold. then, we have $X\le {\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}\le Y.$Here, we note that ${\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}>Y$must hold since $a_{s-1}>0$ and $f_0\ge 0$. This contradicts that none of (i)${}^{\prime }$–(iv)${}^{\prime }$ holds.

Figure 5. Condition for contraction, given in (15(15) $\begin{array}{rl}& \left(\mathrm{i}\right){\underset{\_}{\mathit{q}}}_{1,\tau }<{\underset{\_}{\mathit{q}}}_{1,\tau +1},\left(\mathrm{i}\mathrm{i}\right){\overline{q}}_{1,\tau }>{\overline{q}}_{1,\tau +1},\\ & \left(\mathrm{i}\mathrm{i}\mathrm{i}\right){\underset{\_}{\mathit{q}}}_{2,\tau }<{\underset{\_}{\mathit{q}}}_{2,\tau +1},\left(\mathrm{i}\mathrm{v}\right){\overline{q}}_{2,\tau }>{\overline{q}}_{2,\tau +1}.\end{array}$(15) ).

Next, we consider the case $\mathrm{sgn}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})<0$, which implies that (14(14) $a_{s-1}<|E_{J^{\left(d\right)}}|\le a_s$(14) ) is reduced to $-a_s\le E_{J^{\left(d\right)}}<-a_{s-1}$.

In the same way as the case of $\mathrm{s}\mathrm{g}\mathrm{n}({\hat{J}}^{\left(d\right)}-J^{\ast \left(d\right)})>0$, to prove (13(13) $vol\left({\mathcal{Q}}_{rect,\tau +1}\right)<vol\left({\mathcal{Q}}_{rect,\tau }\right),$(13) ), we show that one of the following conditions holds. $\begin{array}{rl}& {\left(\mathrm{i}\right)}^{″}{\underset{\_}{\mathit{q}}}_{1,\tau }{f}_1^{\left(d\right)}+{\overline{q}}_{2,\tau }{f}_2^{\left(d\right)}<X^{\prime },\\ & {\left(\mathrm{i}\mathrm{i}\right)}^{″}{\underset{\_}{\mathit{q}}}_{1,\tau }{f}_1^{\left(d\right)}+{\overline{q}}_{2,\tau }{f}_2^{\left(d\right)}>Y^{\prime },\\ & {\left(\mathrm{i}\mathrm{i}\mathrm{i}\right)}^{″}{\overline{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\underset{\_}{\mathit{q}}}_{2,\tau }{f}_2^{\left(d\right)}<X^{\prime },\\ & {\left(\mathrm{i}\mathrm{v}\right)}^{″}{\overline{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\underset{\_}{\mathit{q}}}_{2,\tau }{f}_2^{\left(d\right)}>Y^{\prime },\end{array}$where $\begin{array}{rl}{X}^{\prime }& =\frac{{\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}+a_{s-1}{f}_0^{\left(d\right)}}{1-a_{s-1}},\\ Y^{\prime }& =\frac{{\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}+a_s{f}_0^{\left(d\right)}}{1-a_s}.\end{array}$Supposing that none of (i)${}^{″}$–(iv)${}^{″}$ holds, we have $X^{\prime }\le {\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}\le Y^{\prime }.$Recall $E_J>-1$ as stated in Remark 2.1. It follows that $-a_{s-1}>-1$ holds. Consequently, it holds that ${\hat{q}}_{1,\tau }{f}_1^{\left(d\right)}+{\hat{q}}_{2,\tau }{f}_2^{\left(d\right)}<X^{\prime }$This contradicts that none of (i)${}^{″}$–(iv)${}^{″}$ holds. This concludes the statement of the theorem holds.

Remark 4.1

As implied by Theorem 4.1, by iterating the controller update with Algorithm 1, the parameter region ${\mathcal{Q}}_{\tau }$ contracts monotonically in the sense of the outer approximation. The asymptotic convergence of ${\mathcal{Q}}_{\tau }$ to $(q_1^{\ast },q_2^{\ast })$ is not guaranteed in the analysis but is numerically verified in a demonstration given in Section 5.

5. Numerical experiment

In this section, we present a numerical experiment of the proposed control system with Algorithm 1. In the experiment, we demonstrate personalization using the proposed control system, considering two users.

5.1. Problem setting

We address the LQR problem. A plant system is given by a discrete-time linear state space equation: (16) $P:x(k+1)=Ax\left(k\right)+Bu\left(k\right),$(16) where system matrices A and B are given by $A=\left[\begin{array}{cc}0& 1\\ -10& 1\end{array}\right],B=\left[\begin{array}{c}-1\\ 1\end{array}\right].$The objective function of a system-user is described by $J=\sum _{k=0}^{\mathrm{\infty }}x(k+1{)}^{\mathrm{\top }}Qx(k+1)+u(k{)}^{\mathrm{\top }}Wu\left(k\right),$where Q and W are weighting matrices. The corresponding optimal control law is given by (17) $K:u\left(k\right)=-K(Q,W)x\left(k\right),$(17) where $K(Q,W)=W^{-1}{B}^{\mathrm{\top }}P$ is the optimal feedback gain and P is the solution to the Riccati equation $PA+A^{\mathrm{\top }}P-PB{W}^{-1}{B}^{\mathrm{\top }}P+Q=0$. We consider two users, user A and user B. The parameters in J for user A are given by $Q{\ast }_A=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{q_{1,A}^{\ast },q_{2,A}^{\ast }\}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{50,1\}$ and W = 5. Then, the parameters for user B are given by $Q{\ast }_B=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{q_{1,B}^{\ast },q_{2,B}^{\ast }\}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\{1,50\}$ and W = 5.

It should be noted again that J is private, i.e. the system-designer cannot access Q directly. In this experiment, we try to estimate $q_1^{\ast }$ and $q_2^{\ast }$ based on the following user rating; (18) $r=\{\begin{array}{ll}100& (|E_J|=0)\\ 80& (0<|E_J|\le 0.003)\\ 60& (0.003<|E_J|\le 0.006)\\ 40& (0.006<|E_J|\le 0.009)\\ 20& (0.009<|E_J|\le 0.012)\\ 0& (0.012<|E_J|).\end{array}$(18) The flow of the experiment is shown below.

1. Set some initial estimate $q_{1,0},q_{2,0}$.

2. A control experiment is performed where the initial state $x_0$ is determined by random values chosen from $[0,10{]}^2$

3. A system-user evaluates the temporal control system and gives a rating to a system-designer based on (18(18) $r=\{\begin{array}{ll}100& (|E_J|=0)\\ 80& (0<|E_J|\le 0.003)\\ 60& (0.003<|E_J|\le 0.006)\\ 40& (0.006<|E_J|\le 0.009)\\ 20& (0.009<|E_J|\le 0.012)\\ 0& (0.012<|E_J|).\end{array}$(18) ).

4. Apply Algorithm 1 to the user rating to update the estimate of $q_i,i\in \{1,2\}$.

5. Update control law (17(17) $K:u\left(k\right)=-K(Q,W)x\left(k\right),$(17) ) with the updated parameters.

6. Back to Stage 2.

The experiment was performed under $(q_{1,0},q_{2,0})=(60,5)$, which is the initial guess.

5.2. Experiment results

The result of the experiment is given in Figures 6–8. The transitions of the parameters $q_1$ and $q_2$ obtained from the experiment are shown in Figure 6. In the figure, the horizontal axis represents the number of parameter updates ϵ and the vertical axis represents the parameter value. We see that the parameter estimate $({\hat{q}}_1,{\hat{q}}_2)$ converges to $(q_{1,A}^{\ast },q_{2,A}^{\ast })=(50,1)$ for user A, and to $(q_{1,B}^{\ast },q_{2,B}^{\ast })=(1,50)$ for user B.

Figure 6. Transitions of parameter estimate. (a) user A and (b) user B.

Next, the transitions of the user rating are shown in Figure 7. In the figure, the horizontal axis represents the number of parameter updates ϵ and the vertical axis represents the value of the utility gained from the control system. For either user A or B, we see that the utility is maximized by the algorithm even if the system-designer cannot access full information on the objective function.

Finally, we show the state transitions of personalized control systems in Figure 8, which use converged estimated parameters for the control. In the figure, the horizontal axis represents the discrete-time k and the vertical axis represents the state of the plant system (16(16) $P:x(k+1)=Ax\left(k\right)+Bu\left(k\right),$(16) ). We see that the control behaviour is different for each user, which indicates that the control system is personalized.

Figure 7. Transitions of utility. (a) user A and (b) user B.

Figure 8. State transitions of personalized control system. (a) user A and (b) user B.

6. Conclusion

In this paper, we addressed the personalization of control systems where the optimal controller is updated according to users' private objective. We formulated a problem of estimating the individual objective function based on the user ratings and proposed its solution algorithm. The algorithm was analysed for a special case where the objective function is characterized by only two parameters. Finally, a numerical experiment showed the usefulness of the algorithm.

We have not proved that Theorem 4.1 holds for the case with more than three parameters, so future works include the extension of the analysis where a general objective function is addressed. Another future work is to model the user rating in a different manner to (3(3) $r=\{\begin{array}{ll}{M}_1& \mathrm{i}\mathrm{f}|E_J|\in [0,a_1]\\ M_2& \mathrm{i}\mathrm{f}|E_J|\in (a_1,a_2]\\ ⋮& \\ M_{m-1}& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-2},a_{m-1}]\\ M_m& \mathrm{i}\mathrm{f}|E_J|\in (a_{m-1},\mathrm{\infty }),\end{array}$(3) ).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Grant-in-Aid for Scientific Research (B), No. 20H02173 JSPS.

Notes on contributors

Tomotaka Nii

Tomotaka Nii received the B.E. degrees in the Department of Applied Physico-informatics from Keio University in 2022. His research interests include control theory for human-in-the-loop systems.

Masaki Inoue

Masaki Inoue received the M.E. and Ph.D. degrees in mechanical engineering from Osaka University in 2009 and 2012, respectively. He served as a Research Fellow of the Japan Society for the Promotion of Science from 2010 to 2012. From 2012 to 2014, he was a Project Researcher of FIRST, Aihara Innovative Mathematical Modelling Project, and also a Doctoral Researcher of the Graduate School of Information Science and Engineering, Tokyo Institute of Technology. Currently, he is an Associate Professor of the Department of Applied Physico-informatics, Keio University. His research interests include control theory for human-in-the-loop systems. He is a member of IEEE, SICE, and ISCIE.