Architecting Low-Latency Voice AI: From Audio Frames to Streaming LLM Tokens

The Anatomy of Voice Latency

When building conversational voice AI, latency is conversational physics. When two humans converse, the typical gap between turns is 200 to 250 milliseconds. Once machine latency crosses 700ms, callers intuitively begin talking over the system or assume the line is dead.

At Tunk.ai, our mission has been to consistently hit sub-400ms conversational turn-around on production telephony networks.

The Three Bottlenecks in Traditional Voice Pipelines

Traditional voice architectures process calls sequentially in three discrete phases:

[Telephony Audio Chunk] ──> [Full Audio STT Buffer] ──> [LLM API Request] ──> [Full TTS Synthesis] ──> [Speaker Output]

1. Acoustic chunking lag: Buffering audio until a pause is detected adds 400–600ms before transcription even starts.
2. LLM Time-to-First-Token (TTFT): Standard non-streaming LLM requests wait for the complete answer before synthesizing speech.
3. TTS sentence-level synthesis: Synthesizers typically wait for a complete sentence or clause before emitting PCM audio.

The Streaming Pipeline Solution

To break this barrier, we restructured the stack into a continuous dual-simplex pipeline:

// Bi-directional WebSocket frame streamer
export async function streamAudioFrame(
  socket: WebSocket,
  frame: Buffer,
  vadState: VoiceActivityDetector
) {
  if (vadState.isSpeech(frame)) {
    sttStream.write(frame);
  }
}

Key Architecture Principles:

• Bi-directional WebSockets: We stream raw 16-bit 8kHz/16kHz PCM audio directly into our deep learning transcription worker over encrypted WebSocket tunnels.
• Incremental Speculative Completion: We feed partial STT hypotheses into streaming LLMs to prepare cached tool contexts before the user has even finished their sentence.
• Chunked Text-to-Speech: TTS audio synthesis initiates on the very first 3–4 words emitted by the LLM, piping synthesized bytes back to the caller while the remaining response is still being reasoned out.

Key Takeaway: Do not process voice in serial steps. Pipe audio frames directly through asynchronous generators from microphone to speaker.