Introduction

Most world models still cheat the brief. They paint the next frame like a gifted cinematographer — and leave gravity, depth, and camera uprightness as ambient vibes. Fine for a trailer. Fatal once you ask a player, an agent, or a level designer to move through the space.

Puffin-World (NTU / MMLab, arXiv 2609.04196) refuses that shortcut. It treats a world as three native states at once: physics (gravity + latitude), geometry (depth), and appearance (RGB). One unified multimodal stack perceives, generates, and reconstructs — without bolting on a separate SfM or offline recon pass.

If you ship generative worlds into engines, this is the architecture argument you have been waiting for.

The Magic: Physics, Geometry, Appearance — Native

Puffin-World is a unified multimodal world model that keeps three complementary world states first-class:

  • Physics — gravity field and latitude map. Scene uprightness. Horizon. Which way is up in the real world, not just in the last clip.
  • Geometry — depth as an explicit state, sharing latent space with RGB via the same VAE. Bridge from multi-view generation straight into 3D reconstruction.
  • Appearance — images and sequences generated jointly with camera and geometry conditions, not as a lonely pixel dream.

Omni-Camera is the control surface. At every pixel it concatenates an absolute field (up vector + latitude) with a relative ray field (origin + direction). One dense 9-channel condition covers monocular camera-to-world understanding, camera-controllable text-to-image, image-conditioned multi-view, in-place roll/pitch/yaw, translation, compound motion, and joint appearance–geometry generation.

Pro Tip

Relative camera motion alone is not a physical frame. The same relative trajectory can land in multiple absolute orientations — horizon drift, uprightness flip, soft world collapse. Propagate gravity from the reference view through known relative rotations into every target view. Steal that loop before you wire a free-camera explorer.

Why Appearance-Only World Models Stall Game Pipelines

Games and previz need more than pretty futures. Large rolls, long trajectories, uncalibrated single images, and agent look-around all expose the same failure: models that only track how the camera moved, not how it sits in the world.

Puffin-World closes that loop with physics propagation: absolute gravity from the reference view is rotated into each future camera frame, then converted into target-view gravity and latitude maps. Relative motion says how you moved. Propagation says how the new camera is oriented relative to reality.

Scale matters. Puffin-16M fights the flat-camera bias of public datasets:

  • Puffin-Cam-15M — 15M vision-language-camera triplets from 900k panoramas, aggressive roll/pitch and FoV ranges, captions with spatial and physical reasoning.
  • Puffin-Traj-1M — 1M trajectories with continuous look-up/look-down, full 360° yaw, compound exploration.

They also annotate ~44.5M images across 28 public datasets with absolute roll, pitch, and vertical FoV — free fuel for anyone auditing viewpoint bias before the next finetune.

What To Steal For Your Pipeline This Week

1. Represent the world as a triple, not a clip. Appearance without geometry and physics is a screensaver with ambition. Bake depth and uprightness into the generative contract.

Native world-state checklistphysics: gravity_field + latitude_mapgeometry: depth (shared VAE with RGB)appearance: RGB / sequencescamera: Omni-Camera = absolute(up, latitude) ⊕ relative(ray origin, direction)

2. Unify tasks by masking inputs, not by forking models. Text, target cameras, reference images, and role masks compose the job: perception, free-viewpoint sim, text/image-to-3D world gen, joint reconstruction. One framework. Different recipes.

3. Close the loop for exploration. Mimic-world exploration expands different 3D worlds from a shared start under the same prescribed trajectory. Self-calibrated exploration starts gravity-misaligned, reasons about the physical state, predicts a corrective camera action, and imagines the fixed observation — perception → action → generation without a bolted calibrator.

4. Feed engine pipelines with reconstruction-ready views. Joint appearance + geometry across hard trajectories is the difference between “cool video” and meshes/splats you can drop into Unreal or Unity for layout and previz.

5. Grab the open stack. Paper, models, Puffin-16M, and the annotated public corpora are out on Hugging Face under the Puffin collection. Start with camera-to-world perception on your own captures, then graduate to trajectory-conditioned multi-view gen.

Blog: huggingface.co/blog/KangLiao/puffin-world. Paper: arXiv 2609.04196.

Conclusion

Pixel-only world models will keep dazzling timelines. Pipelines that need navigable, reconstructible, physically honest space need native world states. Puffin-World draws the line with Omni-Camera, physics propagation, and a dataset that actually stresses look-around. Steal the triple. Anchor gravity. Then generate worlds your engine can stand on.

—Titus