Introduction

Tiny voice models usually pick one language and do it well. M5-TTS, from the Audio, Speech and Language Processing Lab (ASLP Lab) at Northwestern Polytechnical University, aims wider. Its language model has only 50 million parameters, yet it clones a voice from a short clip, speaks Chinese, English, German, French, Japanese and Korean, handles Cantonese and Sichuanese dialects, and streams faster than real time on four CPU cores.

The paper went up on arXiv this week, and on November 10 the lab added a Custom Voice checkpoint and an audio tokenizer to its Hugging Face collection, next to the voice-cloning weights. The checkpoints are open under Apache-2.0, and there's a free online demo you can use right now.

What Shipped

  • Model: M5-TTS by ASLP Lab (Northwestern Polytechnical University), with a technical report and arXiv paper (2611.10162).
  • Size: a 50M-parameter language model, made of a 30M "temporal" backbone and a 20M local decoder. The speech codec isn't included in that count, and each checkpoint file holds about 150M parameters in total once the text embeddings are included.
  • Checkpoints: M5-TTS-50M-VoiceClone for zero-shot cloning from a reference clip, M5-TTS-50M-CustomVoice for built-in speaker voices, a base M5-TTS repo and an audio tokenizer.
  • Languages: Chinese, English, German, French, Japanese and Korean, plus Cantonese (Yue) and Sichuanese (Chuan) accents.
  • Audio: 24 kHz speech built from 16 codebooks at 12.5 frames per second, decoded with a causal, streaming-friendly codec.
  • License: Apache-2.0 on the Hugging Face model repos.
  • What's open: inference code and the voice-cloning and custom-voice checkpoints. The training, continued-pretraining and GRPO fine-tuning code are not part of the open release.

How It Stays So Small

Most speech language models predict one audio token at a time. M5-TTS uses multi-token prediction: the main backbone steps forward through time, and a small local decoder fills in all 16 codebook tokens for each frame. That split keeps the model light enough for a CPU.

After pretraining, the lab polished it with GRPO (a reinforcement learning method). The rewards scored how close the voice sounds to the reference speaker and whether the pauses feel natural, and they also flagged the exact characters the model got wrong, so the model learned from each mistake rather than just a pass/fail score per sentence.

The Numbers

These figures come from ASLP Lab's report. Lower error rates are better.

  • Seed-TTS Eval (voice cloning): 1.37% character error on Chinese and 2.91% word error on English. MOSS-TTS-Nano, a 100M model, scores 3.62% and 4.50%, so M5-TTS cuts errors by 62.2% and 35.3% with half the parameters.
  • Custom Voice mode: 1.26% Chinese and 1.88% English error.
  • Hard Chinese sentences: 17.48% error, worse than MOSS-TTS-Nano's 15.77%. Tongue twisters and tricky text are still a weak spot, and the lab says so.
  • Six-language test (MiniMax set): fewer errors than MOSS-TTS-Nano in five of six languages. Japanese dropped from 40.22% to 9.70% error and Korean from 8.21% to 2.88%.
  • Cantonese: 8.55% (easy) and 8.70% (hard) mixed error rate, lower than CosyVoice2-Yue and Llasa-1B-Yue.
  • Sichuanese: 7.78% error on the hard set, beating CosyVoice2-WSC, though it trails that model on the easy set.
  • CPU streaming: a real-time factor of 0.653 and 225 ms to the first audio chunk on four Intel Xeon Gold 6530 cores. The lab hasn't published phone or ARM timings.

One caveat: in a 10-listener test, people rated M5-TTS higher than MOSS-TTS-Nano for speaker similarity and clarity, but lower for naturalness. The lab's own automatic judge preferred M5-TTS, so treat that win as promising but not settled.

We tried the official Space on November 10. We gave it the lab's Mandarin sample voice and asked for an English sentence. It returned a 5.5-second, 24 kHz English clip in about 10 seconds, including the wait in the free GPU queue.

How to Try It

The quickest route is the M5-TTS Space:

  • Upload or record a short reference clip.
  • Type its exact transcript into the reference box. Cloning depends on it.
  • Choose the language you want it to speak (zh, en, de, fr, ja or ko).
  • For dialect, set Accent to yue (Cantonese) or chuan (Sichuanese).
  • Each request can be up to 250 characters.

To run it on your own machine, the Space's code works locally. It pulls the speech codec from Qwen3-TTS, and you'll need PyTorch 2.9 and WeTextProcessing:

git clone https://huggingface.co/spaces/ASLP-lab/M5-TTS-Space m5-tts
cd m5-tts
pip install -r requirements.txt
hf download Qwen/Qwen3-TTS-12Hz-1.7B-Base --include "speech_tokenizer/*" --local-dir qwen3-codec
python m5_infer.py \
  --text "Tonight the city hums in lilac light." \
  --language en \
  --reference-audio assets/aslp_default_voice.wav \
  --reference-text "猪携带该疾病，然后通过蚊子传染给人类。" \
  --reference-language zh \
  --checkpoint model/model.safetensors \
  --codec qwen3-codec/speech_tokenizer \
  --device cpu \
  --output out.wav

Pro Tip

Keep each line short. The model reads in chunks of about 96 text tokens, and fixed seeds (the default is 42) give repeatable takes. If a long paragraph sounds rushed, split it into sentences and generate them one by one.

Where It Fits

M5-TTS is a great fit for offline voice assistants, game and visual-novel prototypes, language-learning apps, and anything that needs Cantonese or Sichuanese without a cloud bill. The Apache-2.0 license makes commercial experiments easy. As always, only clone voices you have permission to use.

If you want an even smaller English-only option, compare it with Kyutai's Pocket TTS 4.0 tiny models. For a bigger multilingual cloner, see Audio8-TTS.

Conclusion

M5-TTS fits voice cloning, six languages and two Chinese dialects into a 50M model that streams on a CPU, and its error rates beat a model twice its size. Hard Chinese text and naturalness still need work, but you can test it free today. Record ten seconds of your own voice and hear it speak Cantonese.

—Anabel ♡