Introduction

Almost every voice cloner you've used works in two steps. First a model draws a sketch of the speech, such as a Mel spectrogram, codec tokens or a latent. Then a second model, the vocoder or decoder, turns that sketch into sound. WaveGen skips the second step and draws the sound wave itself.

On November 10, Sang-Hoon Lee (Ajou University) and Ha-Yeong Choi (KT Corp.) released WaveGen's code and checkpoints. The model was accepted to NeurIPS 2026, and the paper went up on arXiv the day before. It's a zero-shot voice cloner for English and Mandarin, trained only on the open Emilia dataset. You can try it right now in a free Hugging Face Space.

What Shipped

  • Model: WaveGen, a diffusion-based model that goes straight from text to waveform, built on a method called Spectral Forcing. Paper: arXiv 2611.12527.
  • Checkpoints: three on Hugging Face. There's the pre-trained 1300k-step model, the recommended V-Spec fine-tune (1400k steps) and WaveGen-Flash, a four-step distilled version.
  • Size: the paper's largest model has about 0.6B parameters. Each checkpoint is a 2.7 GB FP32 Safetensors file.
  • Audio: 24 kHz speech, generated as 50 waveform patches per second (480 samples each).
  • Languages: English and Mandarin Chinese.
  • Code: training, inference and evaluation code on GitHub, plus a local Gradio app and a Dockerfile. Flash ships with inference code only. Its distillation recipe is still "coming soon."
  • License: the source code is MIT, but the checkpoints are CC BY-NC 4.0, so you can't use the released weights commercially.

Why "No Vocoder" Matters

Two-stage systems stack two models, and each one adds its own errors. If the vocoder never saw a certain kind of voice in training, it can smear the high frequencies or add a metallic buzz, even when the first stage got the words right.

WaveGen trains a single model to predict a rough spectral picture of the speech and the waveform at the same time. It uses two separate diffusion heads that share one backbone. The spectral path acts as a guide rail, and the waveform path does the real work, so the output never passes through a separately trained decoder. The model also learns the text-to-speech timing on its own, so it needs no duration predictor and no helpers like BERT or wav2vec 2.0 features.

The authors tested this directly. At the same compute budget, their end-to-end model beat a two-stage version of itself on word error, speaker similarity and audio quality.

The Numbers

These figures are from the authors' model card and paper, measured on the full Seed-TTS test sets (1,088 English and 2,020 Chinese lines). Lower error is better, and higher similarity (SIM) is better.

  • WaveGen (pre-trained): 1.47% English WER, 0.755 SIM. Chinese CER is 1.75%.
  • WaveGen V-Spec (recommended): 1.39% English WER, 0.743 SIM, 1.65% Chinese CER.
  • WaveGen-Flash (4 steps): 1.49% English WER, 0.727 SIM, 1.59% Chinese CER.
  • Same-data rivals (English): F5-TTS scores 1.83% WER and 0.67 SIM, ZipVoice 1.70% and 0.69, and OmniVoice trained on Emilia 1.72% and 0.71. WavTTS, ByteDance's waveform model, gets 1.50% and 0.65.
  • Chinese is the weak spot: on Seed-zh, F5-TTS (1.56%), ZipVoice (1.40%) and OmniVoice-Emilia (0.89%) all make fewer character errors than WaveGen. Its "state of the art" claim rests on English and speaker similarity.

Keep in mind that these are the lab's own numbers, and the comparison only covers models trained on Emilia. Commercial models trained on much more data aren't part of it.

We tested the official Space on November 10 with the English sample voice from F5-TTS's repo and a 20-word prompt. Both variants returned an 11-second, 24 kHz clip in about 11 seconds, counting the wait in the free GPU queue. The Space reported 2.5 seconds of compute for V-Spec (50 steps) and 0.8 seconds for Flash (4 steps).

How to Try It

The easiest way in is the WaveGen Space:

  • Upload a short, clean reference clip.
  • Type its exact transcript into the reference text box.
  • Enter the English or Chinese line you want spoken.
  • Choose V-Spec for the best quality or Flash for speed. Speed, seed and Mel temperature are adjustable.

To run it locally, you'll need Python 3.10+, PyTorch with torchaudio, and FFmpeg:

git clone https://github.com/sh-lee-prml/WaveGen
cd WaveGen
pip install -e .
hf download LeeSangHoon/WaveGen --include "WaveGen_VSpec/*" --local-dir ckpts
wavegen_infer-cli \
  --ckpt_file ckpts/WaveGen_VSpec/model_1400000.safetensors \
  --model_cfg ckpts/WaveGen_VSpec/WaveGen_VSpec.yaml \
  --vocab_file ckpts/WaveGen_VSpec/vocab.txt \
  --ref_audio reference.wav \
  --ref_text "The transcript of the reference audio." \
  --gen_text "Tonight the city hums in lilac and gold."

You can also run python app.py to open the same Gradio interface at http://127.0.0.1:7860.

Pro Tip

Start with Flash to draft your lines, since four steps is roughly three times faster in the Space. Then switch to V-Spec for the final take. Keep Mel temperature at 0 (the default) for steady, repeatable reads, and fix the seed whenever you like a take.

Where It Fits

WaveGen is mostly a research release, but it's a useful one. The full training code and the open Emilia recipe mean labs and hobbyists can study or retrain an end-to-end TTS system without wiring up a separate vocoder. Creators can use it for non-commercial narration, voice-acting drafts and comparison tests against two-stage cloners. Because of the CC BY-NC weights, commercial projects will need something else. As always, only clone voices you have permission to use, and label AI-generated audio.

If you need a commercially friendly cloner, look at ZONOS3-Mini or the Apache-2.0 M5-TTS.

Conclusion

WaveGen shows that a single model can go from text straight to sound wave and still beat F5-TTS-style two-stage systems on English accuracy and voice similarity. It's weaker on Chinese, and the weights are non-commercial, but it's open, documented and free to try today. Load a ten-second clip in the Space and compare Flash with V-Spec yourself.

—Anabel ♡