Title:A Generalized Reduced Linear Program for Markov Decision Processes

Abstract:Approximate Linear Program (ALP) is a useful method for solving Markov Decision Processes with large number of states. An attractive feature of the ALP is that it offers performance guarantees for the approximate value function as well as the corresponding greedy policy. Nevertheless, a serious shortcoming is the presence of a large number of constraints. Techniques in the past have addressed this shortcoming either by selecting a subset of the original constraints to formulate a Reduced Linear Program (RLP) or by making use of problem structure to intelligently eliminate the constraints. Theoretical performance guarantees are available in the case of RLP if its constraints are sampled from the original constraints using the stationary distribution with respect to the optimal policy.\\ In this paper we deal with a constraint approximation method that we call the Generalized Reduced Linear Program (GRLP) which approximates the feasible set of the ALP by a larger set described by a fewer number of linear constraints. Each constraint of the GRLP is obtained as a positive linear combination of the constraints of the ALP. Thus th GRLP is nothing but linear function approximation of the constraints. The main contribution of this paper is the analytical machinery we develop to obtain error bounds related to GRLP that are based on two novel $\max$-norm contraction operators $\Gamma$ and $\tilde{Gamma}$. The paper thus addresses the open question namely whether or not linear function approximation of the constraints can be justified analytically. As an example, we demonstrate the usage of the error bounds developed to come up with the right GRLP for a controlled queue. We discuss the implication of our result in the context of reinforcement learning algorithms and also point out interesting future directions.