Introduction

Tencent Hunyuan just open-sourced a live interpreter. Hy-SimulMT, released on October 26, listens to Chinese or English audio as it streams in and translates it into the other language while the speaker is still talking. Pair it with a TTS service and it reads the translation back in a clone of the original speaker's voice.

The weights are on Hugging Face under Apache 2.0, with a GitHub repo that ships a web UI, a WebSocket API and training tools. Tencent also runs it in its Hy Translate mobile app if you just want to hear it.

The number to remember: on Tencent's long-form RealSI test, Hy-SimulMT ran at a StreamLAAL of about 3 seconds, the lowest lag among the systems it compared, including commercial live-translation APIs.

What Shipped

From the model card and repo:

  • Model: Hy-SimulMT-30B-A3B, a mixture-of-experts model with 30B total parameters and about 3B active per token.
  • Directions: Chinese to English and English to Chinese, speech in, translated text out, in real time.
  • Two flavors: the standard model outputs only the translation. The Interleave model outputs the source transcript and the translation together, which is handy for live captions.
  • Formats: full weights, FP8, and GGUF for each flavor, six repos in total.
  • Voice output: speech and voice cloning come from a separate TTS service. Tencent's docs use Qwen3-TTS-12Hz-0.6B-Base on vLLM-Omni. Cloning starts after a short sample of the speaker and refreshes every 10 seconds.
  • License: Apache 2.0 on the Hugging Face model card.

The Benchmarks, With Caveats

On short-form audio, Tencent reports these quality scores (higher is better):

  • FLEURS: Hy-SimulMT 90.3, Gemini3.5-LiveTranslate 92.4, Qwen3.8-LiveTranslate 88.4, GPT-Realtime-Translate 77.7.
  • CoVoST2: Hy-SimulMT 75.5, Qwen3.8-LiveTranslate 82.1, Gemini3.5-LiveTranslate 74.4.
  • RealSI-Sentence: Hy-SimulMT 84.7, the top score on Tencent's list.

So it isn't a clean sweep. Gemini and Qwen still win some columns. But it beats the other open models Tencent lists, like SeamlessStreaming, on FLEURS, ACL6060 and RealSI-Sentence, and it does that with lower lag.

Tencent also ran a small test against six certified human interpreters on 18 fresh recordings of about six minutes each, and says the model matched their quality while often responding faster. Tencent itself says those results only cover its own samples and settings, and they don't measure voice similarity. Treat them as a promising sign, not proof.

How to Run It Today

You need a Linux box with an NVIDIA GPU and Python 3.12+. The one-command path installs vLLM and the SimulInfer web service:

git clone https://github.com/Tencent-Hunyuan/Hy-SimulMT.git
cd Hy-SimulMT/SimulInfer
bash deploy.sh --install-server --model tencent/Hy-SimulMT-30B-A3B-Interleave

When it prints Ready: http://127.0.0.1:8080/, open that page, pick a direction, and upload audio or use your mic. From the command line:

curl -N http://127.0.0.1:8080/translate/stream \
  -F direction=en-zh -F file=@audio.wav

For the cloned voice, start a Qwen3-TTS service and point SimulInfer at it, following the repo's TTS setup guide. Then pick Clone voice and read translation on the page, or add --tts to the client, and the translated speech is saved to outputs/translated.wav.

  • GGUF note: Tencent says llama.cpp support isn't merged upstream yet and recommends waiting for an official llama.cpp release before using the GGUF files.
  • Mic over the network: remote microphone access needs HTTPS, so use an HTTPS port-forward if the server isn't local.

Why It Matters for Audio Creators

Open speech-to-speech translation has mostly meant waiting for a sentence to finish, then translating it. Hy-SimulMT starts speaking early and corrects itself as it goes, the way a human interpreter does. That makes it useful for live streams, bilingual podcasts, interviews and meetups, not just batch dubbing.

The voice-cloning setup is the part to watch. Because the TTS stage is a separate, swappable service, you can choose your own voice model, keep a fixed narrator voice instead of cloning, or route the text into your own dubbing pipeline. And the Interleave model gives you live bilingual captions for free.

Keep the limits in mind. It's only Chinese and English for now, it needs a real GPU server, and voice cloning raises the usual consent questions. Clone only voices you have permission to use.

Conclusion

Hy-SimulMT is the most complete open live-interpreter release I've seen: Apache-2.0 weights, about 3 seconds of lag, a ready web app, and voice cloning through Qwen3-TTS. If you make bilingual Chinese and English content, it's worth a weekend on a rented GPU. I'll be watching for more languages and that llama.cpp merge.

—Anabel ♡