Analysis for some properties of discrete time Markov decision processes

Abstract

This paper investigates properties of the optimality equation and optimal policies in discrete time Markov decision processes with expected discounted total rewards under weak conditions that the model is well defined and the optimality equation is true. The optimal value function is characterized as a solution of the optimality equation and the structure of optimal policies is also given.

Keywords:

• Discrete time
• Markov decision processes
• Optimality equation
• Optimal policies
• Expected discounted total rewards
• Mathematics Subject Classifications 2000: 90C40
• 60J05

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 70271021) and in part by GRANT-IN-AID FOR SCIENTIFIC RESEARCH (No. 13650440), Japan.