-
Oryx: a Performant and Scalable Algorithm for Many-Agent Coordination in Offline MARL
Authors:
Claude Formanek,
Omayma Mahjoub,
Louay Ben Nessir,
Sasha Abramowitz,
Ruan de Kock,
Wiem Khlifi,
Simon Du Toit,
Felix Chalumeau,
Daniel Rajaonarivonivelomanantsoa,
Arnol Fokam,
Siddarth Singh,
Ulrich Mbou Sob,
Arnu Pretorius
Abstract:
A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to directly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable and combines it with a sequential form of implicit constraint Q-l…
▽ More
A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to directly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable and combines it with a sequential form of implicit constraint Q-learning (ICQ), to develop a novel offline auto-regressive policy update scheme. This allows Oryx to solve complex coordination challenges while maintaining temporal coherence over lengthy trajectories. We evaluate Oryx across a diverse set of benchmarks from prior works (SMAC, RWARE, and Multi-Agent MuJoCo) covering tasks of both discrete and continuous control, varying in scale and difficulty. Oryx achieves state-of-the-art performance on more than 80% of the 65 tested datasets, outperforming prior offline MARL methods and demonstrating robust generalisation across domains with many agents and long horizons. Finally, we introduce new datasets to push the limits of many-agent coordination in offline MARL, and demonstrate Oryx's superior ability to scale effectively in such settings. We will make all of our datasets, experimental data, and code available upon publication.
△ Less
Submitted 28 May, 2025;
originally announced May 2025.
-
Breaking the Performance Ceiling in Complex Reinforcement Learning requires Inference Strategies
Authors:
Felix Chalumeau,
Daniel Rajaonarivonivelomanantsoa,
Ruan de Kock,
Claude Formanek,
Sasha Abramowitz,
Oumayma Mahjoub,
Wiem Khlifi,
Simon Du Toit,
Louay Ben Nessir,
Refiloe Shabe,
Arnol Fokam,
Siddarth Singh,
Ulrich Mbou Sob,
Arnu Pretorius
Abstract:
Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are…
▽ More
Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many digital or simulation-based applications allow for an inference phase that utilises a specific time and compute budget to explore multiple attempts before outputting a final solution. In this work, we show that such an inference phase employed at execution time, and the choice of a corresponding inference strategy, are key to breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain up to a 126% and, on average, a 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during execution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. Our experimental data and code are available at https://sites.google.com/view/inf-marl.
△ Less
Submitted 27 May, 2025;
originally announced May 2025.
-
Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Authors:
Omayma Mahjoub,
Sasha Abramowitz,
Ruan de Kock,
Wiem Khlifi,
Simon du Toit,
Jemma Daniel,
Louay Ben Nessir,
Louise Beyers,
Claude Formanek,
Liam Clark,
Arnu Pretorius
Abstract:
As multi-agent reinforcement learning (MARL) progresses towards solving larger and more complex problems, it becomes increasingly important that algorithms exhibit the key properties of (1) strong performance, (2) memory efficiency, and (3) scalability. In this work, we introduce Sable, a performant, memory-efficient, and scalable sequence modeling approach to MARL. Sable works by adapting the ret…
▽ More
As multi-agent reinforcement learning (MARL) progresses towards solving larger and more complex problems, it becomes increasingly important that algorithms exhibit the key properties of (1) strong performance, (2) memory efficiency, and (3) scalability. In this work, we introduce Sable, a performant, memory-efficient, and scalable sequence modeling approach to MARL. Sable works by adapting the retention mechanism in Retentive Networks (Sun et al., 2023) to achieve computationally efficient processing of multi-agent observations with long context memory for temporal reasoning. Through extensive evaluations across six diverse environments, we demonstrate how Sable is able to significantly outperform existing state-of-the-art methods in a large number of diverse tasks (34 out of 45 tested). Furthermore, Sable maintains performance as we scale the number of agents, handling environments with more than a thousand agents while exhibiting a linear increase in memory usage. Finally, we conduct ablation studies to isolate the source of Sable's performance gains and confirm its efficient computational memory usage.
△ Less
Submitted 26 May, 2025; v1 submitted 2 October, 2024;
originally announced October 2024.
