Title:Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces

Abstract:Meta-reinforcement learning (RL) addresses the problem of sample inefficiency in deep RL by using experience obtained in past tasks for a new task to be solved.
However, most meta-RL methods require partially or fully on-policy data, i.e., they cannot reuse the data collected by past policies, which hinders the improvement of sample efficiency.
To alleviate this problem, we propose a novel off-policy meta-RL method, embedding learning and evaluation of uncertainty (ELUE).
An ELUE agent is characterized by the learning of a feature embedding space shared among tasks.
It learns a belief model over the embedding space and a belief-conditional policy and Q-function.
Then, for a new task, it collects data by the pretrained policy, and updates its belief based on the belief model.
Thanks to the belief update, the performance can be improved with a small amount of data.
In addition, it updates the parameters of the neural networks to adjust the pretrained relationships when there are enough data.
We demonstrate that ELUE outperforms state-of-the-art meta RL methods through experiments on meta-RL benchmarks.