Reinforcement Learning in Latent Heterogeneous Environments

Abstract

Reinforcement Learning holds great promise for data-driven decision-making in various social contexts, including healthcare, education, and business. However, classical methods that focus on the mean of the total return may yield misleading results when dealing with heterogeneous populations typically found in large-scale datasets. To address this issue, we introduce the K-Value Heterogeneous Markov Decision Process, a framework designed to handle sequential decision problems with latent population heterogeneity. Within this framework, we propose auto-clustered policy evaluation for estimating the value of a given policy and auto-clustered policy iteration for estimating the optimal policy within a parametric policy class. Our auto-clustered algorithms can automatically identify homogeneous subpopulations while simultaneously estimating the action value function and the optimal policy for each subgroup. We establish convergence rates and construct confidence intervals for the estimators. Simulation results support our theoretical findings, and an empirical study conducted on a real medical dataset confirms the presence of value heterogeneity and validates the advantages of our novel approach. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

KEYWORDS:

• Bidirectional asymptotics
• Dynamic treatments
• Latent heterogeneity
• Markov decision process
• Reinforcement learning

Acknowledgments

We extend our sincere thanks to the referees and editors for their valuable feedback.

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

This work is partially funded by a grant from the European Research Council under the Synergy program.