Modified value-function-approximation for synchronous policy iteration with single-critic configuration for nonlinear optimal control

Abstract

This study proposes a modified value-function-approximation (MVFA) and investigates its use under a single-critic configuration based on neural networks (NNs) for synchronous policy iteration (SPI) to deliver compact implementation of optimal control online synthesis for control-affine continuous-time nonlinear systems. Existing single-critic algorithms require stabilising critic tuning laws while eliminating actor tuning. This paper thus studies alternative single-critic realisation aiming to relax the needs for stabilising mechanisms in the critic tuning law. Optimal control laws are determined from the Hamilton–Jacobi–Bellman equality by solving for the associated value function via SPI in a single-critic configuration. Different from other existing single-critic methods, an MVFA is proposed to deal with closed-loop stability during online learning. Gradient-descent tuning is employed to adjust the critic NN parameters in the interests of not complicating the problem. Parameters convergence and closed-loop system states stability are examined. The proposed MVFA approach yields an alternative single-critic SPI method with uniformly ultimately bounded NN parameter convergence and asymptotic closed-loop system states stability throughout the process of online learning without the need for stabilising mechanisms in the tuning law for critic NN. The proposed approach is verified via simulations.

Keywords:

• Adaptive dynamic programming
• approximate dynamic programming
• neural networks
• nonlinear control
• optimal control
• policy iteration

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Sir Ross and Sir Keith Smith Fund and an Australian Government Research Training Program Scholarship.