Introduction

Interactive world models keep getting prettier. The hard part is still the same: leave a room, come back five minutes later, and find the door where you left it — while the camera obeys you and the protagonist does not smear into a ghost.

Matrix-Game 3.5 (Kunlun Tech / Riemann Dynamics — arXiv 2608.29910) is the open counter-move. Five-billion-parameter first-person and third-person checkpoints, Apache 2.0 code, Patch Memory for geometry-aware recall, and a clear promise of few-step real-time distillation. Training spans Unreal sims, open-world game footage, and internet video. For game-gen teams, that is infrastructure you can fork — not another closed demo reel.

The Magic: Patch Memory Over Pixel Amnesia

Most video backbones only “remember” what fits in a short context window. Pan away from a building and the model redraws it on return. Matrix-Game 3.5 treats memory as 3D patch retrieval, not “hope the last N frames were enough.”

Here is the loop in plain language:

  1. Lift past observations into localized 3D patches with depth and camera pose.
  2. When the viewpoint returns, retrieve the patches that should be visible now.
  3. Project them into a memory canvas the generator can condition on.
  4. Keep static scenery in Patch Memory; route moving subjects through separate multi-view reference tokens so parked cars do not ghost the street after they leave.

Warped PRoPE folds camera projection into spatiotemporal rotary embeddings — no extra backbone parameters — so attention understands rotation, translation, and perspective instead of faking motion with pretty interpolations. The base models sit on Wan2.2-TI2V-5B. Documented base inference wants roughly a 40GB GPU and 64GB system RAM at ~704×1280. First-person and third-person .safetensors are on Hugging Face today; the distilled few-step causal checkpoints that unlock the ~20 fps / 720p single-GPU claim are still queued for release.

Pro Tip

Do not confuse open weights with a shippable game. Matrix-Game predicts the next frames under camera and prompt control. Your engine still owns collisions, inventory, win conditions, and anything a player can sue you over. Use 3.5 as a navigable look-dev / sim stream, then pin authoritative state elsewhere — Magpie-style — before you promise mechanics.

How To Probe It Like A Studio

Treat the public package as a camera-and-memory stress lab, not a product pitch.

  1. Clone the repo and pull the first-person or third-person base checkpoint from Hugging Face (RiemannDynamics/Matrix-Game-3.5-Base).
  2. Feed an anchor image, an explicit camera trajectory, and a short world prompt. Default base inference uses ~25 denoising steps and emits ~80-frame blocks — measure latency honestly on your hardware.
  3. Script the evil tests: leave a landmark, wander, return; reverse the camera hard; park a dynamic subject, move it, check for doubles; occlude and re-reveal identity.
  4. Log GPU peak, end-to-end control lag, and whether scene recall survives after the memory canvas grows — not just first-clip beauty.
  5. When the distilled real-time weights land, re-run the same harness. That A/B is the only fair verdict on the 20 fps claim.

# Sketch — follow the repo’s current infer.py / infer_distilled.py flags
python infer.py \
  --checkpoint first-person.safetensors \
  --image ./anchor.png \
  --camera ./traj.json \
  --prompt "rainy neon alley, wet asphalt, distant sirens" \
  --outdir ./mg35_rollout

Keep a sidecar JSON of poses, prompts, and failure notes. When distilled weights drop, you want apples-to-apples — not vibes.

Conclusion

Matrix-Game 3.5 is the open world’s answer to closed playable demos: geometric Patch Memory, camera-aware PRoPE, static/dynamic split, and downloadable 5B bases aimed at long-horizon interactive streams. The real-time story is unfinished until the distilled causal checkpoints are public and reproducible. Steal the memory architecture now. Stress-test recall like a QA lead. Wire it beside an authoritative engine when you care about rules. That is how game-gen teams turn conference slides into a pipeline you can ship prototypes with.

—Titus