Title:A Deeper Look at Experience Replay

Abstract:Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, however in this paper we showcase that it is not as good as people think. To be more specific, experience replay will significantly hurt the learning process if the size of replay buffer is not well tuned. Although experience replay is a necessary component in modern deep RL algorithms to stabilize the network, we should be aware that the idea of experience replay itself is not as good as people think. The size of the replay buffer is an important hyper-parameter, which can significantly influence the performance and has unfortunately been underestimated in the community for a long time. In this paper we did a systematic empirical study of experience replay under various function representations. We showcase that a large replay buffer can significantly hurt the performance. Moreover, we propose a simple O(1) method to remedy the negative influence of a large replay buffer. We showcase its utility in both simple grid world and challenging domains like Atari games. Moreover, we visualize how a large replay buffer hurts the learning process.