Introduction

Code Video Model is a new open release from a Microsoft Research team (Zhiyang Liang, Jiawei Zhang, Guocheng Qian, Dongdong Chen, Ziyu Wan). It was announced on X on October 12 (00:46 UTC), and the code is in code-video-model/CodeVideoModel under Apache-2.0. The idea is film previs, with a game engine doing the blocking. You write a prompt. A coding agent turns it into an editable Three.js scene: layout, object motion, camera path, event timing, and physics computed from equations. That scene renders a low-poly proxy video in under a minute. Then a frozen MiniMax-H3 reference-to-video model turns the proxy plus one reference image into the final clip. Modality: text → Three.js program → proxy video → 960×544 video with audio. The code is open and the H3 weights are public, but the scene-writing and reference-image stages call paid models. Where to run it today: locally on Linux with an NVIDIA CUDA GPU. There's no hosted demo.

What shipped

  • Code (Apache-2.0): a cvm package plus inference/ entry points for text→Three.js, rendering, a browser scene viewer, reference-image generation, final video generation and edit pairs. The repo also bundles the Three.js runtime.
  • Five ready-to-run examples: sports_car, tank_boarding, loft_tour, robot_navigation and a chair_placement original/edited pair. Each comes with a proxy code_video.mp4, a reference.png and a prompt.txt. Example media is CC BY 4.0.
  • CodeVideoBench: 22 cases across eight categories on Hugging Face. Gaming, product cinematography, architectural cinematics, physics grounding and bullet time get two cases each; scene editing, robotics simulation and trajectory variation get four each. Every case ships its runnable Three.js program, so you can open it in the World Inspector.
  • Tech report and project page: tech_report.pdf and code-video-model.github.io.

How it works

If you feed a crude proxy straight into a video model, you get a crude-looking clip back: flat materials, low-poly edges. Code Video Model avoids this with what it calls early-step conditioning, which needs no training. H3 runs its native 20 denoising steps, and the proxy video is only attended to for the first 5 (the --keep-steps default, about 25%). That is when layout, viewpoint and motion lock in. After that the proxy tokens are dropped, and the text prompt plus the reference image set the final look. H3 itself is never fine-tuned, and no ControlNet-style module is added.

On the team's own benchmark, Code Video Model on H3 scores 85.2 / 82.1 / 85.8 with QwenVL-3.8-27B, GPT-5.6 and GPT-6 as judges. The strongest learned baseline they tested, VACE with depth control, scores 74.3 / 73.1 / 80.6.

Since the whole shot is one program, a revision is a code edit. In one of the team's demos, the robot gets a new route by changing its waypoints while everything else in the scene stays the same. You can scrub the proxy before spending GPU time on the final render.

Run it

Ubuntu 22.04, a CUDA 12.6-capable NVIDIA driver, and Python 3.10. Setup per the README:

git clone https://github.com/code-video-model/CodeVideoModel.git
cd CodeVideoModel
conda create -n cvm python=3.10 -y && conda activate cvm
conda install -c conda-forge "ffmpeg[version='>=6',build='gpl_*']" -y
python -m pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
  --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
python -m playwright install chromium
python download_h3.py --output models/MiniMax-H3-ref2va   # pinned, hash-checked H3 snapshot

# bundled proxy + reference + prompt → final clip (no API key needed)
example=tank_boarding
python -m inference.generate_video \
  --code-video "examples/${example}/code_video.mp4" \
  --reference-image "examples/${example}/reference.png" \
  --prompt-file "examples/${example}/prompt.txt" \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output "outputs/${example}.mp4"

  • GPUs: by default the text encoder goes on cuda:1 and the generator on cuda:0. The script exposes --text-device / --generation-device flags and has a code path for running both on the same device. The report doesn't publish VRAM numbers.
  • Your own scene: python -m inference.generate_threejs --prompt "..." --output outputs/threejs, then python -m inference.render video --scene outputs/threejs and python -m inference.viewer --scene outputs/threejs --port 8000 to scrub it. In the code, this stage shells out to the GitHub Copilot CLI with gpt-6-astra.
  • Reference image: inference.generate_reference (strict) or generate_reference_guided (looser, text-led) produce a 1536×864 frame with GPT-Image-2. That step needs OPENROUTER_API_KEY.
  • Fixed output contract: 960×544, 24 fps, 124 frames (~5 s), seed 42. Proxies that don't match are rejected.

The bundled tank_boarding prompt shows what the authors mean by gaming: "First-person POV in Halo. I leap from a dropship onto a moving alien tank, grab the hull as plasma bolts streak overhead, climb to the turret, and rip open the hatch while the battlefield races beneath me."

Why game devs should care

This is engine-first video. Three.js handles what engines are good at: exact blocking, camera rails, timing, and physics that is computed rather than imagined. The video model only supplies the look. For game teams, that maps onto trailer and cutscene previs, pitch shots, and greybox-to-concept-frame passes where you need the camera move to land exactly. It's also a cheap way to produce controlled, physics-true clips as world-model training data, one of the uses the authors list. It's the offline sibling of the code-driven world models we've been tracking, such as the Streaming Code World Model, but it isn't interactive: you get a rendered 5-second shot, not a playable stream.

Caveats

  • Not fully local: the text→Three.js agent (Copilot CLI / GPT-6) and the reference-image step (GPT-Image-2 via OpenRouter) are paid, closed services. You can skip both by authoring the Three.js scene and the reference frame yourself.
  • H3 terms apply: the CVM code is Apache-2.0, but the MiniMax-H3 checkpoint carries its own licence. Check it before commercial use.
  • Alignment limits: the authors say frame-exact, pixel-level matching is still hard. When the reference image and the proxy disagree on a subject's shape, H3 keeps the composition and motion but may not reproduce every pose or boundary.
  • Self-reported scores: CodeVideoBench is the team's own benchmark, judged by VLMs. There are no third-party evals yet.

Conclusion

Code Video Model is a practical recipe that needs no training: let a coding agent build the shot in Three.js, then let a frozen MiniMax-H3 paint it, with the proxy only active in the first 5 of 20 steps. The code is Apache-2.0, the examples run without an API key, and the benchmark scenes are inspectable. For game teams it's a way to get previs-accurate camera and physics into AI clips today, as long as you have a Linux CUDA box and accept two paid API stages in the full pipeline.

—Titus