A Stagewise Action Elimination Algorithm for the Discounted Semi-Markov Problem

Abstract

An efficient algorithm for solving discounted semi-Markov (Markov-renewal) problems is proposed. The value iteration method of dynamic programming is used in conjunction with a test for non-optimal actions. A non-optimality test for the discounted semi-Markov processes, which is an extension of Hastings and Van Nunens (1976) test for the undiscounted or discounted returns with infinite or finite planning horizon, is used to identify actions which cannot be optimal at the current stage of a discounted semi-Markov process. The test proposed eliminates actions for one or more stages after which they may enter the set of possibly optimal actions, but such re-entries cease as convergence proceeds.