Introduction

Offline video models are gorgeous—and glacial. You prompt, you wait, you get a finished clip. Vidu S2, from Shengshu Technology with Tsinghua collaborators, is aimed at the opposite dream: live generative video you can steer while it streams.

The team’s new paper (arXiv:2609.11638) introduces two siblings—Vidu S2-Avatar for interactive digital characters and Vidu S2-Editing for real-time video-to-video transforms—plus an exploration of spatial (stereo) video for immersive headsets. Compared with Vidu S1’s 540p stream, S2-Avatar targets 720p at roughly 25–42 FPS, with stronger instruction following (yes, including dance) and references you can swap mid-session.

What Creators Actually Get

Think less “render farm,” more “live stage.”

Vidu S2-Avatar

  • Endless streaming via causal temporal attention (not a fixed-length bidirectional clip)
  • Dynamic references — hand the avatar a new object, outfit cue, or look while the session runs
  • VLM agent control — a vision-language controller turns your voice/text plus new images into detailed prompts for identity, gaze, expression, and props
  • Self-Replay Forcing — on-policy distillation so long rollouts stay sharper and resist identity drift
  • A latent one-step super-res refiner to land 720p detail without killing motion continuity

Vidu S2-Editing

Live monocular streams become editable canvases:

  1. Style transfer (watercolor, 2D animation, and friends)
  2. Virtual try-on from a garment reference
  3. Subject replacement while holding pose and background
  4. Background replacement (office → beach without rebuilding the shot)

Frame-aligned attention keeps source motion locked while pixels change—surgical edits, not a whole new take.

Spatial Video for Immersive Play

S2 also probes stereoscopic output. Avatar chunks can go through depth estimation into left/right views; Editing can process joined eye-pairs so outfits and styles stay consistent in 3D. It’s early for ultra-low-latency VR head-tracking, but the path from flat stream → headset-ready stereo is sketched clearly.

How Fast Is “Real Time”?

The paper’s serving stack is doing the heavy lifting: hybrid attention kernels per layer, W8A8 quantized linears, and Ulysses-style context parallelism across GPUs. Human preference studies on StreamAV-Bench reportedly favored S2-Avatar over commercial rivals like Runway and HeyGen in a large share of matchups for long-horizon identity and temporal consistency; Editing lands a strong overall Sparkle-Bench score versus streaming and offline baselines.

Pro Tip

Treat S2 as a director’s chair, not a one-shot generator. Start with a clean reference identity, keep spoken directions short and concrete (“pick up the blue mug,” “switch to the red jacket”), and update references only when the VLM agent has a clear object to latch onto. Mid-stream chaos compounds; mid-stream clarity sings.

Try It and Read the Paper

The authors point a playable stream demo at vidu.com/vidu-stream (the public page still leans on the S1 branding today—check for S2 capabilities as they roll out). Dive into the technical write-up on arXiv or alphaXiv.

Conclusion

Vidu S2 is the clearest signal yet that interactive video is graduating from research demos to creator-grade streams: 720p avatars you can redirect live, stream editing that respects motion, and a first look at spatial video. If your workflow needs a character who listens while the pixels keep flowing, this is the paper—and the playground—to watch.

—Aurelia ♡