Introduction

Live games do not wait for your policy. DashVMC (Florent Tariolle & Florian Yger, INSA Rouen / PTA workshop NeurIPS 2026) trains a compact discrete world model from roughly two hours of recorded Geometry Dash footage, then improves a lightweight controller with PPO entirely inside frozen latent dreams — no further live-game interaction during refinement. The paper dropped on arXiv Sep 30 (2609.40003); code, LFS checkpoints, and an in-browser dream player are open at Tariolle/dash-vmc with the project page at tariolle.github.io/dash-vmc (paper link commit landed Oct 1).

What shipped

  • Modality: action-conditioned discrete world model + reactive game agent for the original, non-paused Geometry Dash (screen pixels in, keyboard jump out)
  • Open vs closed: open-source training + eval + deploy scripts; reported V7 tag; Git LFS checkpoints under checkpoints_v7/; browser ONNX/WebGPU dream player baked into the project site
  • Where to run today: clone the repo and load the published checkpoints, or open the Play inside the dreams embed on the project page (#play-dream → player/?embed=1)
  • One hard number: the decoder-free live path sustains a 60 Hz capture-to-action loop on a consumer RTX 2060 (mean ~12 ms; p95 ~14 ms)

Why makers should care

Most world-model papers evaluate in simulators that pause for the agent. DashVMC flips the contract: Desktop Duplication → encode → act before the keystroke goes stale. The stack is deliberately small and explicit:

  1. iFSQ tokenizer — Sobel edge maps → 64 discrete tokens per frame
  2. Action-conditioned transformer dynamics — imagines the next grid (and alive/death) so PPO can practice offline
  3. Tiny actor–critic — BC warm-start from demos, then PPO in 45-step dream rollouts

Across three controller seeds, dream-refined policies outlive their exact BC parents on Stereo Madness, Back on Track, Polargeist, a ship-form copy, and a held-out community layout. That is the headline for engine/world-model teams: imagination-trained gains that transfer to a live binary-action game.

Maker notes

git clone https://github.com/Tariolle/dash-vmc
cd dash-vmc
# Conda env: PyTorch 2.11.0+cu126 / CUDA 12.6 per README
conda run -n <env> python -m pip install -r requirements.txt
# Reported training snapshot: git tag V7
# Live 60 FPS deploy (needs Geometry Dash + published checkpoints):
python scripts/deploy.py --config configs/deepdash/v7-phase0.yaml \
  --vae-checkpoint checkpoints_v7/fsq_best.pt \
  --transformer-checkpoint checkpoints_v7/transformer_best.pt \
  --controller-checkpoint checkpoints_v7/controller_ppo_best.pt \
  --preprocessor exact-cuda --fps 60
# Or explore the world model without the live game:
python scripts/play_dream.py --config configs/deepdash/v7-phase0.yaml \
  --vae-checkpoint checkpoints_v7/fsq_best.pt \
  --transformer-checkpoint checkpoints_v7/transformer_best.pt \
  --controller-checkpoint checkpoints_v7/controller_ppo_best.pt

Prefer clicking first? The project site ships an interactive dream player that runs the exported ONNX graphs on-device (WebGPU with WASM fallback). Use that to feel action-conditioned continuations before you wire Desktop Duplication.

Scope note: this is one deterministic binary-action title with offline demos — not a general open-world video generator like Matrix-Game / EditWorld. Steal the dream-PPO → live 60 Hz pattern; do not expect a drop-in NPC brain for Unreal.

Conclusion

DashVMC is a clean, runnable answer to a hard question: can a world model trained on a short offline archive improve a policy that must then keep up with a real game clock? With open code, published checkpoints, a browser dream player, and a measured 60 FPS live path on an RTX 2060, the answer for Geometry Dash is yes — and the recipe is small enough to port.

—Titus