Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimization

Abstract

Reinforcement learning (RL) is a powerful machine learning technique that enables an intelligent agent to learn an optimal policy that maximizes the cumulative rewards in sequential decision making. Most of methods in the existing literature are developed in online settings where the data are easy to collect or simulate. Motivated by high stake domains such as mobile health studies with limited and pre-collected data, in this article, we study offline reinforcement learning methods. To efficiently use these datasets for policy optimization, we propose a novel value enhancement method to improve the performance of a given initial policy computed by existing state-of-the-art RL algorithms. Specifically, when the initial policy is not consistent, our method will output a policy whose value is no worse and often better than that of the initial policy. When the initial policy is consistent, under some mild conditions, our method will yield a policy whose value converges to the optimal one at a faster rate than the initial policy, achieving the desired “value enhancement” property. The proposed method is generally applicable to any parameterized policy that belongs to certain pre-specified function class (e.g., deep neural networks). Extensive numerical studies are conducted to demonstrate the superior performance of our method. Supplementary materials for this article are available online.

KEYWORDS:

• Mobile health studies
• Offline reinforcement learning
• Semi-parametric efficiency
• Trust region optimization

Supplementary Materials

The supplementary materials provide proofs for all technical lemmas, theorems, corollaries, details on the proposed algorithm, and additional numerical studies.

Acknowledgment

The authors thank the Editor, Associate Editor, and anonymous reviewers for their suggestions and helpful feedback which improved the paper significantly.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Dr. Fan Zhou’s work is supported by National Natural Science Foundation of China (12001356), “Chenguang Program” supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission, Open Research Projects of Zhejiang Lab (No. 2022RC0AB06), Shanghai Research Center for Data Science and Decision Technology, Innovative Research Team of Shanghai University of Finance and Economics.