Introduction

Adventists AI shipped Sophon-Realtime v1.1 on November 11, 2026, three days after the v1.0 launch we covered. The headline is zero-shot voice cloning in the open full-duplex speech-to-speech model family. You drop a 3–10 second clip of one speaker into model/voices/, set "voice" to the file name, and the 32B voice agent talks back in that voice, live. There's no fine-tuning and no server restart. It works in the Tag and Flash builds. The same day, the E2E build left research preview and now runs on one 80 GB GPU, with about 0.25 s to first audio. Code and weights stay Apache-2.0. You can run it today with pip install -U "sophon-realtime[trt]" on your own GPU, pull weights from Hugging Face or ModelScope, or try the hosted demo on ModelScope Studio.

Sources: the GitHub repo, the v1.1.0 and v1.1.3 release notes, the CHANGELOG, the Tag build README, and the model cards for Tag, Flash and E2E.

What Shipped

Zero-shot voice cloning (v1.1.0, Tag + Flash)

  • Put a 3–10 s clip of clean speech from one speaker (wav or flac, any sample rate) at model/voices/alice.wav. Then pass "voice": "alice" to /v1/chat/completions, or set it in session.update on the full-duplex /v1/realtime socket.
  • New, replaced and deleted voices take effect on the next request. The Tag server scans the folder every 5 s and prepares new clips in the background, so the team says first audio is as fast as with a built-in voice.
  • Cloning is cross-lingual: the reference clip can be in a different language from the reply. On Flash, alice_en.wav + alice_zh.wav give one voice a separate clip per language, since lookup tries alice_en before plain alice.
  • Sound tags still work in a cloned voice, so it can laugh, sigh, breathe and stress words. Barge-in and backchannels work too.
  • If a voice name is unknown or the file can't be read, the server falls back to the default chelsie voice and logs a warning. The session doesn't fail.
  • There's a new Tag voice model (talker/flow.pt plus the ONNX flow decoder, weights tagged v1.1.0). Adventists says unseen voices now sound "much closer to the reference". Older weights still clone, just less faithfully.

E2E out of preview, on one GPU (v1.1.3)

  • A new talker of about 350M parameters reads the brain's tagged reply while it is still being written. It generates continuous voice vectors at about 6.5 frames/s, and a 2-step decoder plus a vocoder turn them into 24 kHz audio. There's no speech-token model in between.
  • Vendor-measured on one H100 from a fresh install: about 0.25 s first audio for one user (the preview took about 1 s). With 16 users arriving within 10 s, first audio was 0.44 s median / 0.56 s P90, and no reply fell more than 0.3 s behind playback. The preview needed two GPUs and was recommended for 4–5 users.
  • NVV-SuperBench non-verbal sounds: 0.75 English / 0.85 Chinese, up from 0.60 / 0.77 in the preview.
  • E2E talker tags: [laughter], [cough], [sigh], [breath], a strong-stress span tag and a laughter span tag (speaking while laughing). It also accepts [cry], [clear] and [gasp], but those three aren't benchmarked yet.
  • E2E speaks Chinese and English with one built-in voice. No cloning on E2E — use Tag or Flash for that.
  • Requirements: about 70 GB of GPU memory and about 75 GB of disk. CUDA MPS lets the thinker and the speaking engine share the card. Without MPS, first audio rises to about 0.6 s.

Serving fixes (v1.1.1–1.1.2)

  • pip install "sophon-realtime[trt]" now actually installs TensorRT. 1.1.1 pins the one installable release of a deprecated NVIDIA dependency.
  • sophon-realtime serve now turns on CUDA MPS by default (SOPHON_MPS=0 turns it off). With 16 concurrent long replies, each session now gets about 200 s of speech in a 170 s window instead of about 120 s, so playback keeps up with real time.
  • PyPI sophon-realtime is at 1.1.2. The v1.1.3 E2E changes ship in the repo scripts and the E2E weights.

Clone a Voice in Four Lines

On one 80 GB GPU (H100, H800 or A100 80G), Linux, CUDA 13 and Python 3.12:

pip install -U "sophon-realtime[trt]"          # keep [trt]: plain PyTorch is ~3x slower under load
sophon-realtime download --source modelscope   # or Hugging Face (default)
cp alice.wav model/voices/                      # 3-10 s, one speaker, wav or flac
sophon-realtime serve                           # Tag build on port 8000; then send "voice": "alice"

To test it from the repo's Tag example client:

python client_example.py --wav my_question.wav --lang en --voice alice

Use the system prompt in config/system_prompt.json with /v1/chat/completions. The team says the model writes far fewer sound tags without it. Audio comes back as base64 24 kHz WAV chunks.

Specs at a Glance

All numbers below are vendor-reported, measured on one H100 80 GB:

  • Tag (recommended): most expressive voice, NVV 0.80 en / 0.86 zh, about 0.6 s first audio for one user (0.75 s median at 16 users with TensorRT), 16 users per GPU, cloning yes.
  • Flash: 0.31 s first audio, 40 users per GPU, NVV 0.63 en / 0.77 zh, cloning yes. Japanese and Russian run on Flash only.
  • E2E: about 0.25 s first audio, 16 users per GPU, NVV 0.75 en / 0.85 zh, one built-in voice, no cloning.
  • Shared 32B brain: VoiceBench 82.8, Big Bench Audio 81.8%, Full-Duplex-Bench v3 tool-calling Pass@1 48%.
  • Languages: Chinese and English fully supported. Italian, German, Spanish, French and Portuguese are "ready for conversation". Korean is listed as next.

Why It Matters

Open full-duplex models have mostly shipped with a few fixed voices, and the realtime agent stacks that let you bring your own voice have been closed APIs. Sophon v1.1 puts hot-swappable, cross-lingual cloning inside an Apache-2.0 speech-to-speech agent that you host yourself. Its endpoints follow the OpenAI Realtime style, and its Live gateway speaks the GPT-Live protocol. Adventists points out that the clips and the generated audio stay on your server, which matters for companion, character and branded-agent builds.

Caveats: there's no published speaker-similarity or cloning-quality metric yet. The "much closer to the reference" claim is the team's own. Every build still needs an 80 GB-class GPU. The repo's consent line is explicit: only clone your own voice or a voice you have permission to use, and never use it to impersonate or deceive.

Conclusion

Sophon-Realtime v1.1 turns the open 32B full-duplex agent from a fixed-voice demo into a bring-your-own-voice runtime: a 3–10 s clip, a folder drop, and no restart. The E2E build is now a real, single-GPU option with the fastest first audio in the family. If you deployed v1.0 last weekend, upgrade with pip install -U "sophon-realtime[trt]". For cloning, also pull the v1.1.0 Tag weights.

—Anabel ♡