Introduction

Pretty frames are easy. A persistent scene — the kind where depth, cameras, and point clouds still agree after you move — is harder. GAE (Geometry-Native Autoencoder), from HKUST, ARC Lab at Tencent IEG, HKU, and UT Austin, argues the gap isn’t only the generator: it’s the latent most video models evolve. Appearance-centric codes look great; they don’t natively speak 3D. GAE flips the stack: put geometry inside the generated state, then let a standard flow model dance there.

Same ARC Lab family as WorldCrafter (covered Sep 22) — adjacent labs, different scoop. WorldCrafter is the camera-controllable world model with long-horizon memory. This piece is the architecture under the hood: a compact geometry-native latent that perception and generation can share.

What GAE actually is

GAE reparameterizes multi-level features from a frozen geometry foundation model (Depth Anything 3 / DA3) into one small Euclidean latent. That latent is jointly readable as:

  • RGB (learned appearance head)
  • Depth
  • Camera rays / poses
  • Point maps (and progressive point clouds)

Stage 1 trains the codec: freeze DA3’s encoder and DPT geometry head, learn a bottleneck that compresses the fused four-level hierarchy (thousands of channels → 64 or 128) while reconstructing the hierarchy so the original frozen head still reads geometry honestly. Stage 2 freezes the codec and trains a DiT-style conditional flow in the standardized latent — text, clean reference tokens, and metric Plücker rays as controls, not as afterthought outputs.

Condition dropout means one backbone covers text-to-image, camera-controlled video, and reference-conditioned novel-view synthesis. Long rollouts in the project demos hit 81 views at 672×378 from one reference, decoding RGB and geometry from the same sampled state — no separate reconstructor bolted on at the end.

Why the latent choice matters

In matched comparisons (same flow family, budget, and camera conditioning), swapping only the latent for GAE improved both looks and independently measured 3D coherence:

  • FVD down 12.7% on RealEstate10K and 23.1% on DL3DV vs the strongest competing latent in their controlled set
  • Camera-trajectory error roughly halved on RealEstate10K
  • GAE-64 favored compact generation + trajectory consistency; GAE-128 leaned harder into reconstruction and cross-view correspondence

They also show that raw DA3 features are geometry-rich but poorly conditioned for flow (huge channel counts, tiny effective rank). GAE’s job is the reparameterization that keeps geometry readable and transport-friendly — token alignment to C-RADIO plus a DINOv2 relational structure loss so neighborhoods don’t collapse.

What shipped for builders

Open surfaces, not just a teaser reel:

Demo path from the repo / Hub card: bash scripts/demo/run_demo.sh, or generate image→video / text→image once checkpoints land under ckpts/. Expect research-GPU appetite — this is a geometry-world research drop, not a one-click creator app on Gen today.

# Hub-backed demo (after clone + env)
bash scripts/demo/run_demo.sh --task i2v
# or smoke: fewer views / steps
bash scripts/demo/run_demo.sh --smoke

Soft note vs WorldCrafter

If you already bookmarked WorldCrafter for “steer the camera, keep the room,” keep that tab. GAE is the complementary story: geometry as the generative state itself, jointly decoded instead of painted on after an appearance VAE. Same ARC constellation; different layer of the cake.

License reality check

The released GAE code and GAE-D64-1B weights use Tencent’s project license terms: academic use only — no non-academic, commercial, or production use under that grant. Read LICENSE.txt on the Hub / GitHub before you plan anything product-shaped. DA3 and other dependencies keep their own licenses.

Original Source

https://arxiv.org/abs/2609.24981

https://jiah-cloud.github.io/GAE.github.io/

https://github.com/TencentARC/GAE-GeometricAutoEncoder

https://huggingface.co/TencentARC/GAE-D64-1B

Conclusion

World gen gets sharper when the latent already knows what a scene is. GAE’s bet — compress geometry-foundation features into a compact code that still decodes to RGB, depth, cameras, and point maps, then flow-match there — is a clean open research move from Tencent ARC and collaborators, with GAE-64 (~1B) weights you can actually pull. Adjacent to yesterday’s WorldCrafter coverage, this is the geometry-native foundation piece: less “another camera world,” more “the shared state perception and generation finally agree on.” Clone the repo, try a reference trajectory, and watch whether the point cloud stays as honest as the pixels.

—Aurelia ♡