Introduction

Most voice assistants still take turns like a polite radio play: you speak, they freeze, they answer, you wait. Realtime-Venus, a fresh Hugging Face drop from inclusionAI (Ant Group) with Tsinghua University, aims at something closer to a real conversation — eyes open, ears open, and willing to start talking when the moment calls for it.

Two 9B checkpoints landed together: Realtime-Venus-Omni (audio-visual) and Realtime-Venus-Audio (audio-first). The technical report is arXiv:2609.13814. Weights and custom Transformers code live on the Hugging Face model card; the async Realtime-Venus-Harness and cookbooks live on GitHub.

What “full-duplex” actually means here

Native full-duplex is the headline: the model keeps perceiving while it speaks. It is trained to tell apart backchannels, interruptions, corrections, and redirections — the messy social layer that turn-taking systems usually paper over.

On top of that sits Omni-Proactive behavior. Instead of waiting for a wake phrase, the Omni checkpoint continuously watches temporally aligned video and audio and can initiate when an event warrants a response. Think “the kettle just boiled” energy, not “say the magic word.”

Architecture at a glance

Both checkpoints adapt MiniCPM-o 4.5 / Omni-Flow, with:

  • Visual encoder: SigLIP2 (Omni; unused at Audio inference)
  • Audio encoder: Whisper-Medium
  • Language backbone: Qwen3-8B
  • Speech: discrete S3 tokens plus a streaming flow-matching decoder
  • Context: 40,960 tokens · dtype: BF16

Omni takes video/images, audio, and text; Audio takes audio and text. Both can emit text plus native speech (via bundled Token2wav / reference voice assets).

Delegation without freezing the chat

A distinctive piece is in-stream asynchronous delegation. On the shared causal timeline the model can emit harness requests and later consume backend results the same way — so an external tool call does not have to stall the ongoing conversation. Executing those requests needs the Realtime-Venus-Harness runtime from the GitHub repo, not just the weight shards.

There is also training-free long-video Memory: archive visually informative moments, retrieve query-relevant non-redundant evidence, and reassemble the matching audio-visual context — no extra finetune required for that memory path.

How to try it (and what to watch)

Expect a serious local stack: Python 3.10, CUDA, FFmpeg, plus trust_remote_code Transformers loads for the custom Omni/Audio modules. The README walks duplex streaming (as_duplex / streaming_prefill / streaming_generate), half-duplex chat(), memory-enabled modes, and audio-only duplex. Project demos also sit at realtime-venus.github.io.

License note: the repo ships Apache 2.0, and the card asks you to also respect upstream MiniCPM-o / Qwen / Whisper / SigLIP and any data terms you mix in. Read LICENSE / LEGAL.md before you ship a product on top.

Why creators should care

If you build companions, live tutors, AV agents, or “watch with me” tools, Realtime-Venus is a concrete open checkpoint family for listen-while-speaking, proactive AV, and tool delegation that does not mute the room. It will not replace every cloud realtime API tomorrow — but as a research-grade, downloadable Omni/Audio pair with a harness story, it is one of the more complete full-duplex drops on the public shelf this week.

Original Source

https://huggingface.co/inclusionAI/Realtime-Venus

https://arxiv.org/abs/2609.13814

https://github.com/inclusionAI/Realtime-Venus

—Aurelia ♡