Introducing Milabench: Benchmarking Accelerators for AI
Authors:
Pierre Delaunay,
Xavier Bouthillier,
Olivier Breuleux,
Satya Ortiz-Gagné,
Olexa Bilaniuk,
Fabrice Normandin,
Arnaud Bergeron,
Bruno Carrez,
Guillaume Alain,
Soline Blanc,
Frédéric Osterrath,
Joseph Viviano,
Roger Creus-Castanyer Darshan Patil,
Rabiul Awal,
Le Zhang
Abstract:
AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, w…
▽ More
AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.
△ Less
Submitted 22 November, 2024; v1 submitted 18 November, 2024;
originally announced November 2024.
CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning
Authors:
Luke Rowe,
Roger Girgis,
Anthony Gosselin,
Bruno Carrez,
Florian Golemo,
Felix Heide,
Liam Paull,
Christopher Pal
Abstract:
Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or…
▽ More
Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning (RL) to efficiently generate reactive and controllable traffic agents. Specifically, we process real-world driving data through a physics-enhanced Nocturne simulator to generate a diverse offline RL dataset, annotated with various rewards. With this dataset, we train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours by modifying the desired returns for the various reward components. This capability enables the generation of a wide range of driving behaviours beyond the scope of the initial dataset, including adversarial behaviours. We show that CtRL-Sim can generate realistic safety-critical scenarios while providing fine-grained control over agent behaviours.
△ Less
Submitted 14 October, 2024; v1 submitted 28 March, 2024;
originally announced March 2024.
