Video generation
Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation
NeurIPS 2026 · 2026
RESEARCH SUMMARY
Robust Dreamer improves long autoregressive video rollouts with action control by storing diffusion latents in a 3D Gaussian memory and training the generator to handle errors in its own historical memory.
Read the supporting paper ↗
THE RESEARCH QUESTION
Why does action-controlled video drift over long rollouts?
A generator repeatedly conditions on its earlier predictions. Small mistakes can accumulate, while repeatedly decoding latent features into RGB images and encoding them again loses information. Training with clean memory also leaves a mismatch with the imperfect memory encountered during generation. [Paper]
How the method works
- Latent Gaussian Memory anchors diffusion latents to 3D Gaussian primitives and retrieves view-aligned features through latent-space splatting, avoiding repeated latent-to-RGB cycling.
- Deviation Learning synthesizes rollout-like errors. A Dynamic Deviation Archive organizes these deviations by autoregressive stage and denoising timestamp.
- Injecting archived deviations into training memory exposes the model to imperfect histories. Evaluation covers ScanNet, DL3DV and OmniWorldGame.
RESEARCH INTERPRETATION
What this contribution means
The key idea is to improve both the representation of history and the generator's ability to use imperfect history. This is relevant to interactive world simulation, where a plausible individual frame is not enough: revisiting a location should preserve the scene. The reported gains concern the paper's evaluated rollouts, rather than an unrestricted guarantee of physical consistency.
This section explains the contribution in context. For experimental protocols, numerical results and comparisons, consult the original paper.
Scope and limitations
Memory quality depends on successful 3D reconstruction. Occlusion, reflective surfaces, rapid motion and limited viewpoints can still produce incomplete geometry. Memory construction and generation are separate modules, which limits joint optimization. [Paper]
Questions about this research
What does AR mean in Robust Dreamer?
AR means autoregressive: the model generates subsequent video content using earlier generated content and control signals.
Does Gaussian memory eliminate long-term drift?
It is designed to reduce drift, together with deviation-aware training. Reconstruction failures and accumulated prediction errors remain limitations.
Sources, paper & code
Primary research sources support the summary above. Project resources provide demonstrations or implementations; Google Scholar is a bibliographic index.
- Primary sourcearXiv — paper record and versions ↗arxiv.org
- Primary sourceFull paper — PDF ↗arxiv.org
- Research resourceProject — demos and resources ↗hlinchen.github.io
- Metadata indexGoogle Scholar — bibliographic listing ↗scholar.google.com
Publication venue: Confirmed by Yifu Wang. The research-note update date is separate from the paper's publication date.