Introduction

Scene-Designer is out with open code and weights. The OmniAI Group at Zhejiang University's ACES Lab released an 8B vision-language agent that builds 3D indoor scenes step by step from a text prompt, laying out the floor plan, then furniture, walls and ceiling, and checking its own work from five camera views as it goes. The GitHub repo went public on October 16 (UTC), and the Scene-Builder checkpoint is on Hugging Face. You run it locally on Linux on top of SceneSmith, an open (MIT) agentic scene generator that exports through Blender and the Drake physics toolkit.

If you build levels, this is a research release, not an app. But it's a real, downloadable agent for the hardest part of room generation: putting the right objects in sensible places and fixing mistakes without endless retries.

What shipped

  • Name and vendor: Scene-Designer, from the OmniAI Group of ZJU ACES Lab (Zhejiang University, with Universal Identification Technology)
  • Modality: agentic 3D indoor scene generation (text prompt in, a furnished room scene out, saved as Blender snapshots)
  • Open vs API: open code and weights. The released checkpoint, Scene-Builder-Qwen3-VL-8B-Instruct, is fine-tuned from Qwen3-VL-8B-Instruct and ships as four safetensors shards
  • Where to run it today: locally, by cloning the repo, patching a pinned SceneSmith checkout and downloading the model from Hugging Face
  • Also released: Scene-OTC-Bench, a held-out test set of 94 two-scene comparisons and 55 four-scene ranking questions for judging scene quality

How it works

The team's project page frames the problem as "design intelligence": a good scene builder has to both design a room and critique it. In their tests, general-purpose models like GPT-5.2 often took several corrective attempts to fix even simple layout errors.

So Scene-Designer learns from two kinds of experience:

  • Scene-OTA-35K: about 35,000 observation, thought and action steps from 1,000 scene builds, with wasted moves trimmed out so the model learns efficient fixes
  • Scene-OTC-15K: about 15,000 comparison examples made by deliberately breaking scenes (layout drift, broken function, physics violations) so the model learns to rank better and worse rooms

Joint fine-tuning on both produces Scene-Builder. A reinforcement-learning stage called Scene-GRPO then uses a frozen critic to reward edits that beat their sibling options and actually improve on the parent scene.

At run time the agent works in four stages: floor plan, furniture, walls, then ceiling. At every visual stage it gets five fresh renders of the room, one top-down and four oblique views from the south, east, west and north, and picks its next tool call from what it sees.

Running it

Once SceneSmith and its assets are installed, generation is two terminals. Start the local tool server:

python -m uvicorn src.tool_server:app --host 127.0.0.1 --port 18000

Then point the agent at it with a prompt:

python -m src.rollout_client --server-url http://127.0.0.1:18000 \
  --model-dir model/Scene-Builder-Qwen3-VL-8B-Instruct \
  --prompt 'Design a warm bedroom with a bed, bedside storage, and a reading corner.' \
  --config scene_designer --max-steps-per-stage 30 --keep-server-alive

Results and full tool traces land in results/. Adding --branch-critic turns on critic-guided branching, where the agent tries alternatives and keeps the better scene.

Why game teams should care

  • Set dressing at scale: interiors are where procedural levels look most fake. An agent that reasons about function (bed near storage, a chair actually facing the reading lamp) is a better starting point than random scatter
  • A critic you can reuse: the scene-ranking side is useful on its own as an automated check for generated rooms
  • Physics-checked layouts: SceneSmith validates placements, so you get fewer floating chairs and fewer props sunk into the floor before anything reaches your engine

Pro Tip

Before committing to the full install, run the cheaper model-only check. It runs a single inference step on existing observations, without SceneSmith's scene tools:

python -m src.infer_step --model-dir model/Scene-Builder-Qwen3-VL-8B-Instruct \
  --sft-root datasets/sft --repo-root . --stage furniture --limit 1

If you keep going, note that the release keeps the research setting of a 0.2 m floor-penetration tolerance, compared with 0.05 m upstream. Tighten physics_validation.floor_penetration_tolerance_m if props clipping into floors will bother you in-engine.

Caveats

  • Heavy setup: you need Linux, Python 3.11, a separately installed SceneSmith at a pinned commit, SAM3/SAM3D checkpoints, material and CLIP embeddings, articulated-object assets, and EGL for Drake export. The example config also points SceneSmith and the annotation helpers at an OpenAI-compatible API (the samples use gpt-5.2), separate from the local Scene-Builder model
  • SFT weights only: the published checkpoint is the Scene-Builder fine-tuning stage, before Scene-GRPO. The RL-trained Scene-Designer policy itself isn't in the release
  • Not fully proven end to end: the repo's own validation notes say a complete floor-plan-to-ceiling run from a brand-new prompt wasn't executed in the release check, and paper results weren't reproduced
  • License and paper: neither the repo nor the model card names a license yet, so ask before using it commercially. The arXiv paper is marked "soon"
  • Indoor rooms, not engines: output is Blender and Drake scenes, so plan your own export step into Unity, Unreal or Godot

Original Source

https://github.com/ZJU-OmniAI/Scene-Designer

https://huggingface.co/XuanyuLiu/Scene-Builder-Qwen3-VL-8B-Instruct

https://zju-omniai.github.io/Scene-Designer/

Conclusion

Scene-Designer is an early, open look at room generation that thinks like a level designer: build in stages, look from several angles, and fix the error once instead of five times. It takes real effort to stand up, but if you're working on procedural interiors, the weights, the critic benchmark and the trajectory tooling are all there to study and build on.

—Titus