Introduction

PAN 2 is an open world model you can download today: give it a scene and a list of actions, and it renders what happens next. The Institute of Foundation Models (IFM) at MBZUAI

on November 10 and put the weights on Hugging Face with training and inference code on GitHub. Weights and code are both Apache-2.0. It isn't a real-time playable world like a Doom-style model. It's an action-to-outcome simulator, and a 29B one, aimed at agents that want to test "what if I do this?" before they act.

What shipped

  • Model: PAN 2, a diffusion backbone checkpoint (IFM/PAN2) that handles both text-to-video and image-to-video
  • Vendor: Institute of Foundation Models, MBZUAI
  • Modality: action-conditioned world simulation. The input is an optional starting image plus text describing actions, and the output is video
  • Open vs API: open weights, training code and inference code under Apache 2.0
  • Where to run it today: the repo's generate.py, the Diffusers PAN 2 pipeline, vLLM, or FastVideo, which IFM lists in
  • Concrete fact: output is 704x1248 at 24 fps, about 10 seconds (241 frames) by default and up to 20 seconds (481 frames), with 50 sampling steps and CFG 2.5 by default

How it works

IFM splits the job in two, following the generative latent prediction design from its Critique of World Models paper.

  • Conceptual reasoner: an LLM reads the current scene and the actions you asked for. It writes the next world state as a structured, time-stamped caption that covers events, which actor does what and when, and how the camera moves. Any instruction-following model works. The report tested Claude Opus 5.5, GPT-6 Astra, Kimi K3 and Qwen 3.8 Flash.
  • Perceptual backbone: a 29B MMDiT trained from scratch on hundreds of millions of videos captioned in that same format. It fills in the visual detail. Qwen3.5-9B is the frozen text encoder and the Wan 2.2 VAE decodes the result.

The model card says objects keep their identity and state when the camera moves or they drop out of view. IFM's demo thread shows the order of actions changing the outcome, and milk, ink and oil each behaving differently in water.

Try it

You need the transformer weights plus the two frozen parts:

hf download IFM/PAN2 --include "transformer/*" --local-dir ./PAN2
hf download Qwen/Qwen3.5-9B
hf download Wan-AI/Wan2.2-TI2V-5B-Diffusers
python scripts/convert_checkpoint.py to-native --input ./PAN2/transformer --output ./pan2-native/transformer
bash inference/inference.sh ./pan2-native/transformer ./out

The default prompt file has one text-to-video example and one image-to-video example. The backbone expects structured captions, so run free-text ideas through the repo's prompt rewriter first. That can be inference/prompts_rewritter.py with a local model, or the bundled Conceptual Reasoner skill for Codex or Claude Code. Each caption has a soft limit of 1,480 text-encoder tokens.

Why it matters for game devs

For games, the useful idea is the split between what happens and how it looks. You describe a beat as timed actions bound to actors, something like "the guard turns at 2s, the crate tips at 4s," and the model renders it with consistent objects and a camera that stays put in the world. That makes PAN 2 a sandbox for previs, for blocking out cutscene beats, and for testing whether an agent's plan "looks right" before anyone builds it in an engine. It's also a large, permissively licensed base for anyone who wants to fine-tune an action-conditioned world model on their own game footage, since the training code ships too.

The catch

PAN 2 is not interactive. Each clip is an offline diffusion render, not a frame-by-frame playable loop, so it won't replace an engine or a real-time world model. It's also heavy. The DiT alone is 29B parameters in bf16, so plan on roughly 58 GB just for those weights, plus a ~19 GB text encoder. The inference script can stream blocks from CPU memory (--offload blocks) when the model doesn't fit, and the repo ships an H200 setup guide. IFM also recommends running it behind its separate pan2-guardrail safety filter. The training data and benchmarks lean toward everyday physical scenes and robotics, not stylized game art, so expect to fine-tune if you want your own look.

—Titus