Introduction

AlayaLab just open-sourced AnyGameRenderer, a video-to-video LoRA that takes a crude gameplay recording (gray blockouts, colored primitives, wireframes) plus a text prompt and re-renders the same gameplay in a finished art style, frame-aligned with the input. The code is on GitHub under Apache-2.0, the LoRA is on Hugging Face, and the technical report (arXiv 2610.31466) hit Hugging Face Daily Papers on October 23.

The pitch is aimed squarely at the agent-built-game wave. AI coding agents can now ship a small playable prototype in minutes, but it usually looks like programmer art. AnyGameRenderer is a "skin it later" pass that keeps every entity where it was.

What shipped

  • Model: AnyGameRenderer, a LoRA on MiniMax-H3 ref2va (a 33B reference-video-conditioned video model)
  • Vendor: AlayaLab (Ruicong Liu, Liangyang Ouyang, Fengbo Lan, Zhixiang Wang, Kaipeng Zhang)
  • Modality: video-to-video generative rendering for game footage, with generated audio
  • Access: open. Inference code is Apache-2.0; the 2.4 GB LoRA follows the MiniMax-H3 Community License, which you must accept before downloading
  • Where to run it today: locally from the GitHub repo on a single 80 GB-class GPU
  • Output: 1344×768 at 24 fps, about 5 seconds (124 frames) per clip as trained

How it works

You feed it a plain mp4, not engine data. No G-buffers, no per-object labels, no low-poly proxies. The input convention is pale-gray geometry with one flat, saturated proxy color per character, weapon, or vehicle class (red, orange, yellow, green, cyan, blue, purple, magenta and so on), and no HUD or effects.

The caption is a small JSON spec describing the setting, each proxy color's subject, how it moves, and the soundscape. The repo even ships a coding-agent skill (skills/anygamerenderer-caption/SKILL.md) that inspects your clip's colors and writes the spec for you, with 45 ready-made art styles from photoreal AAA to LEGO, ukiyo-e, crayon, and synthwave.

Training used roughly 10k paired clips from three sources: 38 agent-made games exported from the GEAR game-creation tool, Unreal Engine scenes with programmatically controlled crowds and entity counts, and real gameplay recordings paired back to boxes via 3D detection.

The numbers that matter

The team built an entity-level benchmark of 59 held-out cases with 753 ground-truth entities. Against the zero-shot MiniMax-H3 base:

  • Temporal match rises from 50.9% to 62.9%
  • Position match rises from 57.1% to 61.9%
  • Missed count changes fall from 34.7% to 21.3%
  • Hallucinated entities fall from 52.1% to 46.0%, and unrendered ones from 4.1% to 2.9%

In plain terms: when three enemies spawn on frame 40, you are more likely to actually see three enemies on frame 40.

Try it

git clone https://github.com/AlayaLab/AnyGameRenderer.git && cd AnyGameRenderer
bash setup_env.sh && conda activate agr
bash download_weights.sh   # ~123 GB base weights + 2.4 GB LoRA
bash run_example.sh        # renders the bundled WW1 shooter, heist and city-driving clips

For your own clip:

python make_caption.py my_clip/spec.json -o my_clip/caption.txt
echo '{"name":"my_clip","primitive":"my_clip/primitive.mp4","caption_file":"my_clip/caption.txt","out":"out/my_clip.mp4"}' > jobs.jsonl
CUDA_VISIBLE_DEVICES=0 python infer.py jobs.jsonl

Budget for it. The README lists one GPU with at least 80 GB, 128 GB of system RAM, about 125 GB of disk, and roughly 17 minutes per clip at 20 steps on H100-class hardware, plus about 6 minutes of weight loading per process. This is a cloud-GPU job, not a laptop one.

Why it matters for game pipelines

This is not a real-time renderer. At 17 minutes a clip, you will not ship it in a game loop. What it does well is the step between "my agent built a playable tower defense" and "here is a trailer, pitch video, or style test that shows what it could look like." One checkpoint covers tower defense, auto-battlers, survivors-likes, turn-based battles, kart racing, crowd runners, and shooters.

It also points at a cleaner split of labor for small teams: let agents and engines handle logic, layout, and timing in cheap primitives, then let a video model handle the look. The entity-level benchmark is the useful part here, because "the style looks great but a unit vanished" is exactly the failure that makes generative rendering useless for gameplay footage.

Original Source

https://github.com/AlayaLab/AnyGameRenderer

https://huggingface.co/AlayaLab/AnyGameRenderer

https://arxiv.org/abs/2610.31466

https://alaya-lab.github.io/AnyGameRenderer/

Conclusion

AnyGameRenderer is a heavy, offline tool, but it is open, documented, and built for exactly the gray-box prototypes AI agents are pumping out. If you have a playable blockout and access to an 80 GB GPU, you can see it in LEGO, Ghibli, or AAA style this afternoon.

Want a feel for the MiniMax-H3 base model behind it first? Try it on Gen → https://artrealmai.com/gen?utm_source=magazine&utm_campaign=gen&utm_content=anygamerenderer-open-lora-turns-gray-box-gameplay-into-styled-video

—Titus