Introduction

Fish Audio, the team behind the Drama 3 voice model, just opened a preview of Realtime Speech to Text. The new model, transcribe-stream-1,

on October 28 (MYT), with full docs already live.

It's not just another live captioner. Fish built it for voice agents, so it tries to answer the hard question every talking app gets wrong: has this person actually finished speaking? It also tries to tell a quick "mm-hm" apart from a real interruption, and your user's voice apart from a TV in the background.

It's a closed, paid API, and you can use it today with a Fish Audio API key.

What Shipped

From the Realtime STT guide:

  • Model: transcribe-stream-1, marked preview.
  • Endpoint: a WebSocket at wss://api.fish.audio/v1/asr/realtime. Messages use MessagePack, and the official SDKs don't wrap it yet.
  • Input: 16-bit mono PCM at 8 kHz up to 48 kHz, sent in 80 to 100 ms chunks. The server resamples for you. MP3 and Opus aren't accepted here.
  • Languages: language tags for 29 languages, and turn detection works on speech in any language.
  • Output: live drafts, turn signals, and one final transcript per turn, with speaker tags and sound tags like [laughter] and [pause].
  • Limits: sessions up to 60 minutes, single turns up to 60 seconds.
  • Price: $0.36 per hour, billed per second of session time.

For recordings you already have, Fish points you to its file model, transcribe-1-pro, instead.

The Smart Turn Detection Part

This is the reason to care. In the default auto mode, every stretch of sound becomes a turn, and the server labels what kind it was:

  • User turn: the server sends user_turn_taken, then often an early eager_end_of_turn, then the final committed text.
  • Backchannel: if your agent is talking and the user says "yeah" or "right," you get user_backchannel. That means keep talking, don't stop.
  • Other sound: noise, a TV, or another voice commits as a non-user turn, so your agent doesn't answer the room.

To make the backchannel trick work, your app tells the server when the agent is speaking with an agent_speaking message. You can also feed it conversation history (up to 32 lines) and up to 100 custom terms, like product names, so it spells them right.

If you'd rather control turns yourself, there are also vad and manual modes.

The Numbers, With Caveats

Fish measured these latencies on its production service using synthetic speech:

  • First draft text: about 0.5 seconds after speech starts.
  • Early end-of-turn signal: about 0.6 seconds after the speaker stops.
  • Final transcript: about 1.1 seconds after the speaker stops.

Fish also says it ranked #1 of 20 on Cekura's STT benchmark with a 2.21% overall word error rate. That's Fish's own launch claim, and I haven't seen the full benchmark breakdown yet, so test it on your own audio, especially accents and noisy rooms.

How to Try It Today

  1. Grab an API key from Fish Audio.
  2. Open the WebSocket from your server, never a web page, with Authorization: Bearer <API key>. For a browser mic, relay audio through your backend.
  3. Send a config message like this, then wait for session_started:

{
  "type": "config",
  "audio": { "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 },
  "turn_detection": { "mode": "auto", "eager": true },
  "results": { "drafts": true }
}

  1. Stream audio chunks as you capture them, and answer each committed message where user_turn is true.
  2. Send audio_done to close the session.

Fish's docs include tested Python and Node.js clients, plus a browser microphone example. One gotcha: every open connection counts against your API concurrency limit, even while it's silent.

Why It Matters for Audio Creators

Most voice-agent awkwardness comes from turn-taking, not voice quality. Bots cut you off mid-thought, or they stop dead the moment you say "uh-huh." Fish now offers both ends of the loop, this listener plus its streaming TTS voices like Drama 3, so you can build an interactive character, a voice-driven game NPC, or a live-captioned stream on one vendor.

At $0.36 an hour, it's also cheap enough to leave running for a long podcast-style session or a livestream caption track. Just remember it's a preview, so APIs and behavior may change.

Conclusion

transcribe-stream-1 is Fish Audio's bet that live transcription should understand conversation, not just words: turn detection, backchannel awareness, background filtering and context, at $0.36 per hour. If you're building anything that talks back, it's worth an afternoon of testing against your current STT.

—Anabel ♡