Introduction

lucidml just put a playable world model on Hugging Face that's small enough to run on one consumer GPU. The lucidml/gta repo holds a 630M-parameter model plus its full inference runtime under Apache-2.0. You give it one start image, hold WASD, and it generates the video that follows one frame at a time, streaming it to your browser while you play. The checkpoint was trained from scratch on GG800, an 800-hour GTA V dataset. lucidml

on November 11 with a browser demo you can try right now.

What shipped

  • Model: the GTA world model, built on lucidml's own Summaryformer architecture
  • Vendor: lucidml, a small independent lab. The card is written in the first person by its builder
  • Modality: interactive image-to-video. One start image goes in, live action-conditioned frames come out. There is no text prompt
  • Open vs API: open weights and inference code under Apache-2.0. There's no training code, and the browser client isn't included
  • Where to run it today: the hosted demo, or locally with the repo's FastAPI streaming server on a CUDA GPU
  • Concrete fact: 630,205,008 parameters, 28 blocks, width 1280, served at 384x672 and streamed as H.264 over a WebSocket

How the memory works

The interesting part is how a frame remembers the past. Summaryformer is a causal transformer where each step is a whole video frame. Each frame enters as a "record": 1,008 raw latent tokens plus 96 learned summary tokens that go through the same 28 blocks in the same forward pass. There's no separate summarizer network. The summary is a by-product of generating the frame.

While generating frame t, the model attends to:

  • frame t itself, raw and summary tokens
  • the last 2 frames, kept whole at full resolution
  • older frames inside a 24-frame window, but only as their 96 summary tokens
  • the start image, kept whole and never evicted

That all lives in one preallocated key/value buffer per block, with no attention mask. When a frame ages out of the raw window, its 96 summary rows get copied into the summary region and its raw rows get overwritten. At the served resolution that's 6,528 keys instead of the 26,208 you'd need to keep every frame at full detail. Each latent frame decodes to 4 RGB frames, so the 24-frame window covers 96 video frames.

Two more tricks are worth stealing. The two slowest temporal RoPE planes are reused as a "level axis" so the model can tell a summary token from a pixel token of the same frame. And the start image's cached keys get rotated forward in place, so the anchor frame stays next to the moving window during long sessions.

The model is conditioned only on the diffusion timestep, the player's action (5 numbers), the frame rate and the grid shape. Each frame takes three denoising passes plus one clean pass that rewrites its entry in memory, so the cache only ever holds clean frames.

Try it

The fastest route is the browser demo. To self-host:

hf download lucidml/gta --local-dir gta
cd gta
python summary_server.py   # serves on 0.0.0.0:3002

You'll need torch with FlexAttention and torch.compile, plus numpy, fastapi, uvicorn, pillow, av and imageio. flash_attn is optional. The first start is slow because the server compiles the model and plays a warm-up session. One GPU process serves one player.

You bring your own client. You upload the start image to POST /z0/{user_id}, open WS /ws/{user_id}, and send the keys you're holding as JSON:

{"type": "keys", "pressed": {"w": true, "a": false, "s": false, "d": false, "shift": false, "space": false, "left": false, "right": true}}

The server sends back H.264 Annex-B frames at 384x672.

Why it matters for game devs

This is a compact, permissively licensed, from-scratch model whose memory design is documented in plain language, and lucidml says it runs on one consumer GPU. The summary-token cache gives you long-horizon memory at a fraction of the attention cost. That's directly useful if you're prototyping your own action-conditioned model on footage from your game, and the small streaming protocol makes it easy to wire into a custom client or an engine viewport.

The catch

It's GTA V footage, so this is a research toy, not something to ship. The card doesn't address rights to the training footage, and you can't fine-tune on your own game yet because the training code isn't released. Sessions are finite: positions top out at 4,096 frame ids, and the server cuts sessions at 180 seconds by default (you can disable that). It's CUDA-only, and the low 384x672 output plus the short memory window mean drift on long runs is still something to watch.

—Titus