Title:On the Convergence Rate of MCTS for the Optimal Value Estimation in Markov Decision Processes

Abstract:A recent theoretical analysis of a corrected-version of a Monte-Carlo tree search (MCTS) method, so-called UCT, established an unexpected result, due to a great deal of empirical successes reported from heuristic usage of UCT with relevant adjustments for the problem domains in the literature, that its convergence rate in estimating the expected error relative to the optimal value of a finite-horizon Markov decision process (MDP) at an initial state is polynomial. We strengthen this dispiriting slow-convergence result by arguing within a simpler framework in the perspective of MDP, apart from the usual MCTS description, that just simpler UCB1 applied with the policy set as the arm set is actually competitive with or asymptotically faster than the corrected-version of UCT because of its logarithmic convergence-rate. We also point out that MCTS in general has the worst-case time and space complexities that depend on the state-set size which contradicts the original spirit of MCTS. Unless heuristically used, UCT-based MCTS has yet to have theoretical supports for its applicabilities.