Introduction

Video models love tokens the way a summer garden loves rain—until the bill arrives. Fixed-grid VAEs tax every patch the same, even when half the clip is a sleepy backdrop. Kakao’s KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), presented at ECCV 2026, lets the model decide which latent tokens earn their keep.

What KATok Changes

KATok is a transformer video VAE with a learned keep-or-drop selector at the latent bottleneck. Instead of a preset token budget or an inference-time search, content complexity sets the effective token count:

  • Busy motion and detail keep more tokens
  • Static or homogeneous regions get dropped
  • The same mask soft-gates decoder attention so dropped tokens stay out of the reconstruction path

An asymmetric coarse-to-fine design helps too: the encoder uses larger patches for a compact latent set, while the decoder queries a finer grid to recover detail.

Pro Tip

Think of KATok as a smart crop for compute. If your pipeline still pays for every empty sky tile, adaptive tokenization is the sunny fix worth watching.

Numbers That Matter for Creators

On Panda-70M at 256²×16, Kakao reports roughly 366 active tokens on average with strong reconstruction (about 31.24 PSNR / 5.12 rFVD). Downstream, when KATok feeds a flow-matching generator, Kakao cites about 6.9× faster generator training and 3.2× higher generation throughput versus a dense transformer tokenizer—plus a cascaded mask prior that restores spatial layout after token dropping (about 61.53 gFVD on UCF-101 in their sparse setup).

Sparse latents need positions: dropping tokens breaks the old “grid implies layout” assumption, so KATok studies cascaded mask-prior conditioning (predict occupancy, then content) and a joint content–position variant.

Why It Matters Now

High-res and long-form video keep climbing the cost curve. Adaptive keep-or-drop tokenization is the kind of systems win that makes longer clips and denser motion feel affordable again—especially if labs verify it across more generators, as Kakao says it plans to do.

Project page: https://kakao.github.io/KATok/

Paper (arXiv): https://arxiv.org/abs/2608.24293

Conclusion

Less redundant tokens, more motion where it counts, and a clearer path to cheaper high-res video. Keep an eye on KATok as ECCV week unfolds—and keep your own pipelines ready for content-aware compression.

—Aurelia ♡