Introduction

The Saruwatari Lab at the University of Tokyo (sarulab-speech on Hugging Face) has released DialogueSidon2, an open-weight audio-to-audio model for two-speaker dialogue separation and restoration. You give it a degraded mono recording of two people talking, with phone-line bandwidth, background noise, room echo or codec artifacts. It gives you back one clean 24 kHz track per speaker. The model card and training/inference code went up on November 14 (about 22:30 MYT). The weights are under CC BY-NC 4.0 (research and non-commercial only) and the code is MIT. You can try it today in a free Hugging Face ZeroGPU demo, or run it locally on GPU or CPU.

It replaces DialogueSidon (arXiv:2604.09344), which used a diffusion model to sample each speaker's audio. Version 2 drops diffusion entirely. It reads both speakers' latents straight out of the mix in one forward pass with no sampling, so the same input always gives the same output.

What Shipped

  • Model: DialogueSidon2 from sarulab-speech. It's a fully fine-tuned w2v-BERT 2.0 (13 layers) with a readout head, plus an SSL-VAE decoder that turns the predicted latents back into 24 kHz audio.
  • Input: a mono or stereo recording of exactly two speakers, at any sample rate.
  • Output: two 24 kHz tracks, one per speaker (speaker 1 and speaker 2, in arbitrary order).
  • Length: clips up to 30 s are separated in one shot. Longer files are processed in 30 s windows with 5 s of overlap, re-aligned using the overlap and cross-faded.
  • Training: 200,000 steps on 30 s segments. The data mixes telephone conversations (CallHome, Fisher), OpenYap-1k, Basis Conversations 1500 and simulated LibriTTS-R mixtures. The full data recipes are in the repo.
  • Packaging: model.safetensors for the Sidon repo, plus torch.export programs (separator.pt2, vae_decoder.pt2) and a standalone ds2_infer.py that needs only torch and torchaudio.

How It Compares to DialogueSidon

The lab scored both models in the same run, with 95% bootstrap confidence intervals:

  • Switchboard (100 dialogues): WER 14.43% → 12.71%, NISQA 3.45 → 3.63, DNSMOS 3.64 → 3.76. Speaker similarity and voice-activity accuracy also went up slightly.
  • OpenDialog (100 real recordings): WER 14.07% → 9.95%, NISQA 3.57 → 3.82, DNSMOS 3.60 → 3.71. Voice-activity accuracy dropped 1.5 points.
  • CallFriend (German, French, Japanese, Spanish, Mandarin): perceived quality (NISQA and DNSMOS) improved in every language. Character error rate in Japanese and Mandarin was higher, but the confidence intervals are too wide to call it either way.

Where to Run It Today

  • Browser: the DialogueSidon2 demo Space runs on ZeroGPU and comes with a sample telephone-dialogue clip. Upload your own two-person recording and download the separated tracks.
  • Local, no repo clone: pin torch==2.8.0 and torchaudio==2.8.0, because the exported programs were built with PyTorch 2.8.0.

import sys, soundfile as sf, torch
from huggingface_hub import snapshot_download

model_dir = snapshot_download("sarulab-speech/DialogueSidon2",
    allow_patterns=["separator.pt2", "vae_decoder.pt2", "metadata.json", "ds2_infer.py"])
sys.path.insert(0, model_dir)
import ds2_infer

models = ds2_infer.load(model_dir, "cuda")  # or "cpu"
audio, rate = sf.read("interview.wav", always_2d=True)
speakers = ds2_infer.separate(torch.from_numpy(audio.T).float(), rate, models)
sf.write("speakers.wav", speakers.T.numpy(), ds2_infer.SAMPLE_RATE)  # L = speaker 1, R = speaker 2

  • With the Sidon repo: python -m sidon.dialoguesidon2.infer --model ./DialogueSidon2 --output-dir out dialogue.wav writes dialogue_speaker1.wav and dialogue_speaker2.wav.

Caveats

  • Non-commercial license. CC BY-NC 4.0 rules out paid podcast-cleanup or dubbing services. If you need commercial terms, look at Kezek v1 from Manifestro (Apache-2.0, Nov 11). It splits a mono two-person mix into per-speaker tracks that sum exactly to the input, but it doesn't restore the audio and needs a CUDA GPU with bf16.
  • Exactly two speakers. Three-person panels are out of scope, and the order of the output tracks is arbitrary.
  • Track swaps on long files. If one speaker is silent through a 5 s overlap, the windowed re-alignment can swap the two tracks from that point on. Check long interviews by ear.
  • Use policy. The card says not to use it for voice cloning or speaker identification, or on recordings of people who haven't consented.
  • Brand-new. It had zero downloads at release, and there's no standalone DialogueSidon2 paper yet. The numbers above come from the model card.

Conclusion

DialogueSidon2 is the most direct open tool I've seen for a common audio headache: a phone or single-mic recording of two people that you want as two clean, separate stems. Think podcast interviews recorded on one phone, archival call recordings, or two-speaker data for training full-duplex voice models (that last one is clearly the lab's own use case, given its Sidon dataset-cleaning roots). Single-pass and deterministic makes it far easier to batch than the diffusion-based v1. Try the demo with your worst recording first, and keep the non-commercial license in mind before you build anything on it.

—Anabel ♡