Introduction

Most game world models still treat the opponent like wallpaper. You press left. The diffusion stack paints the next frames. The NPC? Absorbed into the background — a side effect of player-centric prompts, not an agent with a plan.

That is fine for a highlight reel. It is useless if you want a real fight.

ReactiveGWM (Tencent / NUS / PolyU, project page inv-wzq.github.io/ReactiveGWM) cuts the knot. Player control and NPC strategy stop sharing one entangled prompt. Buttons steer your character. High-level tactics — Offense, Control, Defense — steer the NPC. Same world model. Two separate control planes. And the NPC modules transfer across games without a fresh annotation farm.

If you are building generative playable opponents, this is the architecture to steal.

The Magic: Decouple the Fighter From the Backdrop

Vanilla game world models usually look like this: one observation, one action stream, one scene prompt that describes everything. The NPC’s punches, blocks, and zoning get baked into that prompt as scenery. The model never learns “Guile is on Offense.” It learns “the pixels happened to move this way while the player jumped.”

ReactiveGWM flips the contract:

  • Player actions inject into the DiT backbone through a lightweight additive bias (no heavy adapter tax).
  • NPC strategy lands in the cross-attention path as a structured prompt: active behaviors, passive reactions, and a strategy class.
  • Self-attention + FFN keep game-specific physics and look.

That split is the whole game. Strategy becomes a steerable signal. Pixel dynamics stay with the backbone that already knows the title.

They prove it on Street Fighter II: Champion Edition and Street Fighter Alpha 3 — ~10k strategy-aligned clips per game, recorded through stable-retro, annotated with Gemini into Offense / Control / Defense, then refined with a deterministic rule engine so the label is not a VLM vibes check.

Pro Tip

Do not put the whole fight in one prompt. Describe what the NPC is trying to do, not the entire frame chronology. Offense closes distance and pressures. Control holds space with projectiles. Defense reads and guards. Once that language is a first-class condition, you can retarget tactics without rewriting the world.

Zero-Shot Strategy Transfer: Plug Modules, Keep the Game

The killer feature is transfer. Train ReactiveGWM on Game 1 with full strategy supervision. For Game 2, keep the vanilla backbone (action module, self-attention, FFN) so the native look and physics stay intact — then swap in the learned cross-attention layers from Game 1.

Those layers carry a game-agnostic interaction prior. No new strategy labels required. You get steerable NPCs on an unannotated title in one modular splice.

Numbers that matter for pipeline people:

  • NPC instruction following jumps from ~43% to ~76%+ on SF2 (VLM referees), and from ~41–50% to ~79% on SF3 for the base model.
  • Player Move-Acc / Att-Acc stay near ceiling — autonomy does not eat your controllability.
  • Transfer retains most of the strategy lift while preserving visual fidelity (SSIM/LPIPS stays in the vanilla band).

Cross-attention is a low-bandwidth channel (~0.7% residual energy) but a high-direction signal. Enough to re-aim the NPC. Not enough to trash the scene.

What To Steal For Your Pipeline This Week

1. Split the conditioning vocabulary. Player buttons and NPC tactics are different languages. Wire them on different paths.

ReactiveGWM control splitplayer: discrete buttons → additive bias into DiT residualNPC: Strategy{Offense|Control|Defense} + active/passive tags → cross-attentiongame: self-attn + FFN keep title-specific physics / look

2. Build strategy data as facts, then classify. Two-stage annotation beats free-form labels: VLM observes closed-set facts (range, advances, projectile count, crouch-guard), then a rule engine maps to Offense / Control / Defense. Labels stay reproducible. Hallucinations die at the fact layer.

3. Design for transfer from day one. If NPC logic lives in a swappable module, your next title is a backbone reuse + module drop — not a six-month re-annotate.

4. Benchmark the three axes that ship. Action following, strategy following, visual quality. A pretty clip that ignores Offense is still a passive renderer.

5. Know the limits before you promise real-time. Fighting games are a sharp testbed; diffusion latency still blocks true playable loops. Distillation and autoregressive backbones are the next move if you want this in a live match, not a research rollout.

Project: inv-wzq.github.io/ReactiveGWM. Paper: arXiv 2605.15256.

Conclusion

Player-centric world models make gorgeous solitary timelines. Competitive generative games need opponents that can choose a plan. ReactiveGWM draws the line — additive bias for your thumbs, cross-attention for their tactics, and modules you can transplant across titles. Steal the split. Ship the strategy. Leave the wallpaper NPCs in the cutscene reel.

—Titus