Introduction

Game and sim teams keep hitting the same wall: you need dense, physics-valid indoor spaces at scale, and neither “ask a VLM to place every chair for three hours” nor “fire an image-to-3D net and pray nothing clips” gets you there. SceneMosaic just dropped into Hugging Face Daily Papers with a hybrid answer — and it is squarely aimed at interactive entertainment and embodied AI pipelines.

Submitted September 4, 2026 (arXiv:2609.05594) with open code at github.com/rxjfighting/SceneMosaic, the framework matches top agentic baselines on semantic layout quality while claiming a 24× speedup, zero collisions / out-of-bounds in their SceneEval-100 runs, and a combinatorial trick that turns one evolution pass into many playable layout variants.

The Magic: Prior Init + Agentic Evolution

SceneMosaic refuses to pick a side in the old trade-off.

  1. Image-based prior — start from a reference image (or text→image), run object-centric reconstruction (SAM3 + SAM3D style), and land a coarse but fast layout.
  2. Physics pre-stab — containment fixes + gravity simulation so floating garbage dies before the agents waste tokens.
  3. Critic–Actor evolution — local VLM loops refine poses using orthographic 2D views (not noisy perspective 3D edits).
  4. Local-unit Cartesian mosaic — split the room into independent anchor units, evolve variants per unit, then compose global scenes via Cartesian product and novelty-aware greedy selection.

That last piece is the game-dev hook. Real rooms rearrange. Training and playtest coverage need structured diversity — chairs that still face the desk, monitors that keep desk orientation — not random Gaussian jitter that breaks support relations.

Pro Tip

If you are building a sim or level-gen loop, treat local units (desk cluster, sofa cluster, wall-TV stack) as your authoring atoms. Evolve those independently, then compose. You get combinatorial coverage without re-running a full-room agent marathon for every variant.

Wire SceneMosaic Into a Prototype Loop

  1. Grab the paper and repo: project page https://rxjfighting.github.io/SceneMosaic, code https://github.com/rxjfighting/SceneMosaic, PDF https://arxiv.org/pdf/2609.05594.
  2. Feed a single reference image (or a SceneEval-style prompt rendered to an image).
  3. Let reconstruction + relation DAG build the scene tree (attach / contain / walls).
  4. Run physics stabilization, then Critic–Actor evolution on each local unit in parallel.
  5. Assemble m diverse variants with the novelty distance (relative position + absolute distance + rotation) and keep the base layout as seed.
  6. Export meshes + poses into your engine or robotics sim and stress-test navigation, interaction, and collision budgets.

Input: one room image (or text → image)
  → object inventory + masks
  → per-object meshes + coarse 6D poses
  → scene tree (anchors / local units)
  → physics settle
  → Critic–Actor orthographic edits per unit
  → Cartesian compose + novelty pick (base + m variants)
Output: simulation-ready layouts, collision-clean

# Start from the public release
git clone https://github.com/rxjfighting/SceneMosaic.git
cd SceneMosaic
# Follow the repo README for env / VLM keys / simulator deps
# Then point the pipeline at your reference room image

On SceneEval-100 they report base scenes at ~0.14h and variants amortized near ~0.03h, with POS/ROT competitive with SceneSmith while COL and OOB sit at 0.0 in their table — the opposite profile of raw SAM3D-style image-to-3D dumps that spray interpenetration.

Conclusion

SceneMosaic is the practical middle path for generative game worlds that have to obey gravity, not just look good in a trailer frame. Fast prior, agentic cleanup, local-unit diversity — that is a pipeline you can actually schedule against a playtest calendar. Clone the repo, feed it a room still, and see how many collision-clean layout variants you can bank before lunch.

—Titus