No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
Authors:
Skander Moalla,
Andrea Miele,
Daniil Pyatko,
Razvan Pascanu,
Caglar Gulcehre
Abstract:
Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy. Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets. However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed…
▽ More
Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy. Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets. However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed loss of plasticity, and eventually a collapse in performance. For off-policy deep value-based RL methods, this phenomenon has been correlated with a decrease in representation rank and the ability to fit random targets, termed capacity loss. Although this correlation has generally been attributed to neural network learning under non-stationarity, the connection to representation dynamics has not been carefully studied in on-policy policy optimization methods. In this work, we empirically study representation dynamics in Proximal Policy Optimization (PPO) on the Atari and MuJoCo environments, revealing that PPO agents are also affected by feature rank deterioration and capacity loss. We show that this is aggravated by stronger non-stationarity, ultimately driving the actor's performance to collapse, regardless of the performance of the critic. We ask why the trust region, specific to methods like PPO, cannot alleviate or prevent the collapse and find a connection between representation collapse and the degradation of the trust region, one exacerbating the other. Finally, we present Proximal Feature Optimization (PFO), a novel auxiliary loss that, along with other interventions, shows that regularizing the representation dynamics mitigates the performance collapse of PPO agents.
△ Less
Submitted 20 November, 2024; v1 submitted 1 May, 2024;
originally announced May 2024.
Asymptotics of the number of possible endpoints of a random walk on a directed Hamiltonian metric graph
Authors:
Daniil Pyatko,
Vsevolod Chernyshev
Abstract:
In this paper, the leading term of the asymptotics of the number of possible final positions of a random walk on a directed Hamiltonian metric graph is found. Consideration of such dynamical systems could be motivated by problems of propagation of narrow wave packets on metric graphs.
In this paper, the leading term of the asymptotics of the number of possible final positions of a random walk on a directed Hamiltonian metric graph is found. Consideration of such dynamical systems could be motivated by problems of propagation of narrow wave packets on metric graphs.
△ Less
Submitted 27 December, 2021;
originally announced December 2021.
