Stability and monotone convergence of generalised policy iteration for discrete-time linear quadratic regulations

ABSTRACT

In this paper, we analyse the convergence and stability properties of generalised policy iteration (GPI) applied to discrete-time linear quadratic regulation problems. GPI  is one kind of the generalised adaptive dynamic programming methods used for solving optimal control problems, and is composed of policy evaluation and policy improvement steps. To analyse the convergence and stability of GPI, the dynamic programming (DP) operator is defined. Then, GPI and its equivalent formulas are presented based on the notation of DP operator. The convergence of the approximate value function to the exact one in policy evaluation is proven based on the equivalent formulas. Furthermore, the positive semi-definiteness, stability, and the monotone convergence (PI-mode and VI-mode convergence) of GPI are presented under certain conditions on the initial value function. The online least square method is also presented for the implementation of GPI. Finally, some numerical simulations are carried out to verify the effectiveness of GPI as well as to further investigate the convergence and stability properties.

Keywords:

• Adaptive dynamic programming
• generalised policy iteration
• heuristic dynamic programming
• linear quadratic regulation

Acknowledgements

The authors appreciate the associate editor and anonymous reviewers for their valuable suggestions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. In this paper, a (linear) policy μ(x) = −Kx, or simply, K is said to be stabilising (or a stabilising policy) if all eigenvalues of A − BK lie on the interior of the unit circle, centred at the origin, in the complex domain.

2. In the case of PI, the initial stabilising policy guarantees that all of the updated policies are stabilising which again guarantees the existence of the limit point $V^{{\mu }_i}\left(x\right)$ (μ_i = −K_ix) of (15(15) $$\begin{array}{c}\hfill V_{i+1}\left(x_k\right)={\mathcal{D}}_{K_i}^N{V}_i\left(x_k\right),\end{array}$$(15) ).

Additional information

Funding

This work was supported by Institute of BioMed-IT, Energy-IT and Smart-IT Technology (BEST), a Brain Korea 21 Plus Program, Yonsei University.