One model, two jobs: speak and reshape

Speech tools used to live in separate drawers — a TTS stack for generation, another toolkit for cleanup, yet another for edits. AuK folds those drawers into one sunny workflow: natural-language instructions plus audio context. You tell it what you want; it generates or edits. Open weights, open code, and a technical report on arXiv that matches the demos on the project page.

This is the open-source speech foundation model from the AuK project — not the unrelated desktop API clients that share a similar name. Bookmark the project site first; that is the authoritative home.

Scale that feels like a studio library

To teach one model both generation and editing, the team built a training corpus that is hard to ignore:

  • About 3.03 billion instruction–audio instances
  • Roughly 1.95 million hours of effective supervision
  • Five task families spanning creation through fine acoustic control

Five AuK task families----------------------1. Speech generation2. Content editing3. Enhancement and separation4. Paralinguistic editing5. Acoustic editing

That mix is why AuK can move from "say this in her voice" to "keep the speaker, change the words," then over to enhancement, separation, and the softer knobs of tone, emotion, and room sound — without swapping toolchains mid-session.

## Architecture in plain light

Under the hood, AuK is a three-part harmony:
AuK stack (high level)----------------------• Multimodal LLM ........... semantic conditioning• Audio VAE ................ acoustic conditioning (jointly trained on speech, general audio, music)• Hybrid rectified-flow Transformer dual-stream MMDiT → unified single-stream DiT

Semantic sense from the LLM, acoustic shape from the VAE, and a hybrid flow Transformer that starts dual-stream then settles into a unified DiT path. It is elegant engineering with a creative-studio payoff: instructions and audio context share one generation loop.

How it learns (and how Flash stays fast)

Training is staged with care:

  1. Generation-only warm-up — get speech generation solid first
  2. Joint generation–editing pre-training — teach the shared instruction interface
  3. Post-training — human-feedback preference optimization for open-ended editing, plus reward-based reinforcement learning for speech generation

For creators who care about latency, AuK-Flash distills that capability with consistency initialization and task-routed Decoupled DMD. The Flash path runs 4-step inference with no classifier-free guidance, and the report cites about a 4.5× wall-clock speedup versus the full model under matched conditions. Same sunny idea, lighter feet.

Pro Tip

Start on the AuK project page for demos and the high-level story, then keep arXiv:2609.08936 open when you dig into training or Flash. When you compare numbers, stick to the project's own evaluation panels — inventing scores is how sunny days turn foggy.

Benchmarks without the fog

AuK claims leading performance on zero-shot and instruction-controlled speech generation, and on general instruction-guided editing, while staying competitive on signal-level restoration. The project page frames results across familiar suites at a high level — including Seed-TTS-Eval, InstructTTSEval, MMAE-Speech, SpeechEditBench, and DNS / CHiME-style enhancement setups — plus related separation views. Treat those charts as the source of truth; this write-up stays qualitative on purpose.

Why creators should care

If you prompt for voice, edit dialogue, or clean location audio, a unified instruction interface means fewer handoffs and more creative flow. Open code and released model weights mean you can reproduce, fine-tune, and remix instead of waiting on a closed API mood. Pair that with AuK-Flash when you need draft-speed iteration, then step back to the full model when fidelity is the star.

Sunny takeaway: generation and editing finally share one language — yours. Explore the demos, skim the report, and try the open release when you are ready to put speech under the same warm light as the rest of your gen-AI stack.

Primary links: AuK project · AuK Technical Report (arXiv:2609.08936)

—Aurelia ♡