Introduction

Matrix-Game 4.0 is live from Skywork AI. The series moves from pure interactive video worlds into streaming neural rendering for executable games: a native engine owns gameplay, physics, and player input, while a causal video model re-renders clean low-poly RGB into detailed frames. Code, browser demo, and three scene models ship today under Apache-2.0 on GitHub (Matrix-Game-4/) with weights and game assets on Hugging Face. Modality: video-to-video / streaming game render (Wan2.2-Fun-5B-Control lineage). Open weights plus open game source. Where to run it today: local Linux with an NVIDIA GPU for inference and the WebRTC demo; macOS or Linux for the standalone low-poly games without the model. Project site: matrix-game-v4-0.github.io/page.

What shipped

  • Inference + games (Apache-2.0): single-GPU offline render, single-user WebRTC browser demo, and native source for three scenes — Medieval Castle and Ancient Egypt (third-person shooting) plus City Racing (driving with time trials and pursuit).
  • Weights: each scene checkpoint is 5,001,016,512 BF16 parameters (~5B). HF folders: fps-castle, fps-egypt, driving_game, plus shared base/taew2_2.pth (streaming TAEHV), game/racing assets, and SHA256 manifests under releases/.
  • Base stack: shared umT5 / Wan VAE pulled from a pinned alibaba-pai/Wan2.2-Fun-5B-Control revision; training code and training data are not in this drop.
  • Output contract: 480×864 RGB. Startup chunk is one RGB frame; later chunks are four frames. Quality preset = Wan VAE + two denoising steps (1000,500). Realtime preset = TAEHV + one step + CUDA graphs. CFG 1, scheduler shift 5, attention cache of 21 latent frames including the sink.

The parent README stamps the public release as October 15, 2026; the reviewed inference/game commit hit main at 13:07 UTC on October 14. Prior Matrix-Game 1.0–3.0 trees remain in the same org repo.

How it works

Earlier Matrix-Game releases generated interactive world video end-to-end. 4.0 splits the stack: the engine keeps collision and game state honest; the model only sees clean low-poly RGB (plus fixed or dynamic text prompts) and streams neural frames. HUD and audio stay frame-aligned presentation layers outside the DiT. Castle/Egypt use an identity sink with fixed-relative RoPE; Racing seeds frame zero from a supplied image with an absolute-position sink — those policies are not interchangeable.

This is not a VLM agent and not a camera-retrieved world-memory bank. One active game session and one scene model per server process.

Pro Tip

Prove the pipeline with --reference-only on the browser path (controls, HUD, audio, transport, no weights), then switch to --preset realtime --fps 40 once MediaMTX and the model service are up. Treat --fps 40 as a capture cadence target, not a latency SLA.

Run it

Requirements for model inference: Linux, NVIDIA GPU, Python 3.12. Skywork’s source deployment used A100 80 GB with PyTorch 2.12.0 / CUDA 13.2; consumer-GPU fit is not verified in the README. From Matrix-Game-4/ (not the parent repo root):

git clone https://github.com/SkyworkAI/Matrix-Game.git
cd Matrix-Game/Matrix-Game-4
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.12.0 torchvision==0.27.0 \
  --index-url https://download.pytorch.org/whl/cu132
python -m pip install -e '.[inference,server,download,test]'

Download scene weights + assets with the repo’s manifest downloader (pinned Hub revision, length + SHA256 checks) — see releases/README.md. Then offline render or play:

# Offline quality render from a low-poly take
matrix-game render --scene ancient-egypt --input lowpoly.mp4 \
  --resources resources --preset quality --output outputs/egypt

# Local low-poly play (no neural path): 48 FPS / 2× sim-time demo
bash scripts/build_game.sh
bash scripts/play.sh --scene ancient-egypt

# Browser demo (loopback WebRTC)
mediamtx configs/mediamtx.yml
CUDA_VISIBLE_DEVICES=0 bash scripts/serve.sh \
  --scene ancient-egypt --resources resources --preset realtime --fps 40
# open http://127.0.0.1:8025/

Racing uses scripts/build_racing.sh and docs/racing.md. Do not expose the demo ports to the public internet — it is a trusted single-user loopback setup.

Why game devs should care

If you have been waiting for open weights that respect a real game loop, this is the Matrix-Game drop that separates simulation from look. You can ship or study low-poly gameplay on its own, then bolt on neural polish with scene-specialized 5B renderers. Castle, Egypt, and Racing give three concrete camera/domain profiles instead of one generic dream. It sits next to Multics and Helix World in the open playable-world lane, but the headline here is engine-grounded streaming render, not multi-ego video or audio-visual world foundation models.

Caveats

  • A100-class baseline: no consumer VRAM or FPS guarantee; realtime vs quality presets trade fidelity for cost.
  • Scene-locked models: unseen geometry, hard camera motion, occlusion, and long sessions can drift; neural pixels do not change collision.
  • No training drop: fine-tune/retrain needs your own stack; weights are not Diffusers drop-ins.
  • Third-party terms: Apache-2.0 for project weights/assets; Wan/umT5 keep upstream Apache-2.0; TAEHV is MIT; racing audio includes CC0 / CC BY 3.0 (see credits).
  • Security: HTTP/RTSP/WebRTC bind to loopback by default — treat it as a lab demo, not cloud gaming.

Conclusion

Matrix-Game 4.0 is Skywork’s open engine-plus-neural-render kit: three playable scenes, ~5B scene checkpoints, single-GPU inference, and a browser demo you can wire up today. Clone Matrix-Game-4, pull the HF manifests, run a quality offline render, then decide whether realtime on your GPU is good enough for your project.

—Titus