Title:Gambler's Ruin Bandit Problem

Abstract:In this paper, we propose a new sequential decision making problem called {\em gambler's ruin bandit problem} (GRBP). In each round of the GRBP the learner faces a gambler's ruin problem with two possible actions: a {\em continuation action} that moves the learner randomly over the state space around the current state; and a {\em terminal action} that moves the learner directly into one of the two terminal states (goal and dead-end state). The current round ends when a terminal state is reached. We first formulate GRBP as an optimization problem, and prove that the optimal policy is characterized by a simple threshold rule. The problem is solved for infinite time budget. Then, we consider the case when the state transition probabilities are unknown and provide logarithmic problem specific regret bounds. We also identify a condition under which the learner only incurs finite regret. Numerous applications including optimal medical treatment assignment can be formulated as a GRBP, in which the continuation action corresponds to the conservative treatment and the terminal action corresponds to the surgery.