-
FLARE: Robot Learning with Implicit World Modeling
Authors:
Ruijie Zheng,
Jing Wang,
Scott Reed,
Johan Bjorck,
Yu Fang,
Fengyuan Hu,
Joel Jang,
Kaushil Kundalia,
Zongyu Lin,
Loic Magne,
Avnish Narayan,
You Liang Tan,
Guanzhi Wang,
Qi Wang,
Jiannan Xiang,
Yinzhen Xu,
Seonghyeon Ye,
Jan Kautz,
Furong Huang,
Yuke Zhu,
Linxi Fan
Abstract:
We introduce $\textbf{F}$uture $\textbf{LA}$tent $\textbf{RE}$presentation Alignment ($\textbf{FLARE}$), a novel framework that integrates predictive latent world modeling into robot policy learning. By aligning features from a diffusion transformer with latent embeddings of future observations, $\textbf{FLARE}$ enables a diffusion transformer policy to anticipate latent representations of future…
▽ More
We introduce $\textbf{F}$uture $\textbf{LA}$tent $\textbf{RE}$presentation Alignment ($\textbf{FLARE}$), a novel framework that integrates predictive latent world modeling into robot policy learning. By aligning features from a diffusion transformer with latent embeddings of future observations, $\textbf{FLARE}$ enables a diffusion transformer policy to anticipate latent representations of future observations, allowing it to reason about long-term consequences while generating actions. Remarkably lightweight, $\textbf{FLARE}$ requires only minimal architectural modifications -- adding a few tokens to standard vision-language-action (VLA) models -- yet delivers substantial performance gains. Across two challenging multitask simulation imitation learning benchmarks spanning single-arm and humanoid tabletop manipulation, $\textbf{FLARE}$ achieves state-of-the-art performance, outperforming prior policy learning baselines by up to 26%. Moreover, $\textbf{FLARE}$ unlocks the ability to co-train with human egocentric video demonstrations without action labels, significantly boosting policy generalization to a novel object with unseen geometry with as few as a single robot demonstration. Our results establish $\textbf{FLARE}$ as a general and scalable approach for combining implicit world modeling with high-frequency robotic control.
△ Less
Submitted 21 May, 2025;
originally announced May 2025.
-
DreamGen: Unlocking Generalization in Robot Learning through Video World Models
Authors:
Joel Jang,
Seonghyeon Ye,
Zongyu Lin,
Jiannan Xiang,
Johan Bjorck,
Yu Fang,
Fengyuan Hu,
Spencer Huang,
Kaushil Kundalia,
Yen-Chen Lin,
Loic Magne,
Ajay Mandlekar,
Avnish Narayan,
You Liang Tan,
Guanzhi Wang,
Jing Wang,
Qi Wang,
Yinzhen Xu,
Xiaohui Zeng,
Kaiyuan Zheng,
Ruijie Zheng,
Ming-Yu Liu,
Luke Zettlemoyer,
Dieter Fox,
Jan Kautz
, et al. (3 additional authors not shown)
Abstract:
We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of famil…
▽ More
We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.
△ Less
Submitted 17 June, 2025; v1 submitted 19 May, 2025;
originally announced May 2025.
-
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
Authors:
NVIDIA,
:,
Johan Bjorck,
Fernando Castañeda,
Nikita Cherniadev,
Xingye Da,
Runyu Ding,
Linxi "Jim" Fan,
Yu Fang,
Dieter Fox,
Fengyuan Hu,
Spencer Huang,
Joel Jang,
Zhenyu Jiang,
Jan Kautz,
Kaushil Kundalia,
Lawrence Lao,
Zhiqi Li,
Zongyu Lin,
Kevin Lin,
Guilin Liu,
Edith Llontop,
Loic Magne,
Ajay Mandlekar,
Avnish Narayan
, et al. (18 additional authors not shown)
Abstract:
General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy in the human world. A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapi…
▽ More
General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy in the human world. A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapidly learn new tasks. To this end, we introduce GR00T N1, an open foundation model for humanoid robots. GR00T N1 is a Vision-Language-Action (VLA) model with a dual-system architecture. The vision-language module (System 2) interprets the environment through vision and language instructions. The subsequent diffusion transformer module (System 1) generates fluid motor actions in real time. Both modules are tightly coupled and jointly trained end-to-end. We train GR00T N1 with a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets. We show that our generalist robot model GR00T N1 outperforms the state-of-the-art imitation learning baselines on standard simulation benchmarks across multiple robot embodiments. Furthermore, we deploy our model on the Fourier GR-1 humanoid robot for language-conditioned bimanual manipulation tasks, achieving strong performance with high data efficiency.
△ Less
Submitted 26 March, 2025; v1 submitted 18 March, 2025;
originally announced March 2025.
