TOUGH TONGUE AI
  • Home
    LogoTOUGH TONGUE AI

    RESOURCES

    Loading...

    HomeLibrarySD Mock · Design a Real-Time Voice AI

    SD Mock · Design a Real-Time Voice AI

    SD Mock · Design a Real-Time Voice AI

    Overview

    Design a real-time voice AI agent for a support line on a live architecture board: stream speech in, reason, speak back in under half a second, and defend every millisecond to an expert interviewer who watches every move.

    Your Instructions

    Design a Real-Time Voice AI

    A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.

    The question: a customer calls a support line and talks to an AI that understands them, looks things up on their account, and answers out loud. Speech is far less forgiving than chat. A pause longer than about a second feels broken.

    Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):

    • Notepad. Shared notes. Requirements, capacity math, the latency budget, schema, and end-of-session feedback live here. You can type in it too.
    • Architecture board. Your design canvas. Drag typed components, wire them, label edges, mark scaled tiers as distributed. If it is not on the board, it is not in your design.

    Build incrementally while you talk. Trace one caller turn end to end through your diagram, with rough timestamps. Argue the tradeoff, not "industry standard."

    Design choices interviewers probe

    Read this before you go in. These are the decisions, and why they matter.

    The latency budget: overlap, don't pipeline

    Write the budget first. The obvious design chains three calls, speech-to-text, then an LLM, then text-to-speech, each run to completion: over 3 seconds of dead air once you include the silence you waited through. Even with every stage streaming, running them in sequence costs about 650 ms: speech-end detection 100 ms + final transcription 150 ms + model first token 300 ms + synthesis start 100 ms. That already misses 500 ms, so the stages must overlap. Transcribe continuously, so the final transcript is a confirmation, not a computation. When the turn detector is fairly sure the caller is finishing (around 50 ms), start retrieval and a speculative model call on the partial transcript; if the last words change the meaning, cancel and restart. Send the first clause to synthesis before the model has finished. The critical path becomes 50 + 300 + ~30 + 100, about 480 ms. What the caller feels is time to the first syllable, not time to the full answer. Say where you start and stop the clock: phone carrier legs add a few hundred milliseconds you cannot control.

    Turn detection: when has the caller finished?

    A fixed silence timer is the naive answer. It fails both ways: people pause mid-sentence (reading out an account number), so a short timer cuts them off, and whatever you choose is added to every reply. Layer it instead: voice activity detection says whether anyone is speaking, a small model reads the partial transcript and judges whether the thought is complete, and a short silence floor confirms. Be more patient right after you asked for a number. An "eager" signal lets you start the LLM early, at the cost of discarded drafts when the caller keeps talking. Silence plus VAD only is equally defensible for a simple bot if you can say what it costs in latency.

    Barge-in: the caller can interrupt at any time

    Keep listening while the agent speaks. When the caller talks for more than about a quarter second, and it is not a "mm-hm", treat it as an interruption: flush the audio already queued at the carrier (a clear message), cancel the TTS stream and the LLM generation, and rewrite the agent's last message to only the words the caller actually heard. Carrier playback confirmations (mark messages) tell you where playback reached. Skip the history fix and the model believes the caller heard things they never did. The main cause of false interruptions is echo: the agent's own voice leaking back into a speakerphone mic.

    A stateful orchestrator per call

    A call is a multi-minute session, not a request. Something must own its state machine (Listening, Thinking, Speaking, Interrupted), its conversation history, and its open streams. Put one session orchestrator per call, pinned for the whole call, with the media gateway in front of it. Snapshot state to Redis at every turn boundary: if the node dies, the call reattaches to a new orchestrator and loses at most one turn. Stateless handlers that reload everything per audio frame are too slow; a single giant stateful server is a blast-radius problem.

    Pipeline versus one speech-to-speech model

    The pipeline (separate recognition, model, and synthesis) is swappable, debuggable, and loggable: you can keep a transcript, bias recognition, and enforce guardrails at the seams. It costs latency across three hops, and tone and hesitation are lost at transcription. A speech-to-speech model is lower latency and sounds far more natural, but gives up exactly those seams: no transcript to audit, no place to bias vocabulary, no clean guardrail point. Choose from the requirements. A regulated support line with specialized vocabulary favours the pipeline; a consumer product where naturalness is the product favours speech-to-speech; a hybrid uses speech-to-speech for conversation and a pipeline path for tool calls and audit records.

    Actions inside the budget

    Tool calls take 200 ms to seconds, and some move money. Keep the caller informed: load the account summary at call start so most turns need no tool, speak a short honest filler ("Let me check that") the moment a tool starts, and time out with a holding line instead of silence. Keep actions safe: tier tools (read, low-risk write, high-risk write), put an idempotency key on every call so a retry cannot refund twice, and read back anything consequential before doing it. The tool gateway enforces this, not the prompt.

    Domain accuracy: three different embeddings

    A specialized domain uses embeddings in three separate roles; keep them apart so you can measure and fix each one. Recognition biasing: a general recognizer confidently turns product names into common words. Give it a vocabulary, but make it contextual: the ~50 terms plausible at this point in the call beat all 10,000 product names, which dilute the bias. Retrieval embeddings: off-the-shelf embedders don't know your enterprise plan and individual plan are different. Fine-tune on real question-to-passage pairs, and train against hard negatives, the look-alike wrong passages. Speaker embeddings: these map a voice, not meaning. Use them to tell two people on the line apart, and for voice verification only as one signal among several, because voices can be cloned. Then evaluate separately: recognition by error rate on domain terms (5% overall error is useless if every product name is wrong), retrieval by recall@k on a golden set. Phone audio is also 8 kHz and noisy, so the recognizer should be telephony-tuned.

    Reference solution

    One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.

    Requirements (lock these or take the defaults)

    • Inbound phone calls to a support line; optional in-app mic button.
    • Understand domain vocabulary, look up account data and policy answers, perform a few account actions with confirmation.
    • Caller can interrupt at any time. Hand off to a human with context when stuck.
    • Record calls with consent; transcripts with PII redacted.
    • Scale: ~1M AI-handled calls/day, ~4 minutes each, telephone-quality audio.
    • Latency: first reply audio within ~500 ms of the caller finishing, measured at our gateway.
    • Out of scope: training speech or language models, outbound campaigns.

    Capacity

    • Concurrency: 10K concurrent calls at peak, each holding an open pipeline (a recognition stream, model context, a synthesis stream) for the whole call. Cross-check: 1M calls/day × 240 s ÷ 86,400 s × 3 peak ≈ 8,300.
    • Audio: 10K × 32 kbps ≈ 320 Mbps each way, which is small. (Raw telephone G.711 is 64 kbps, so ~640 Mbps; still small.) What is not small is the message rate: 20 ms frames mean ~1M messages/s across the gateways.
    • Turns: 10K calls × 1 turn every 10 s = 1K turns/s through the pipeline. With 30% needing a tool round, ~1,300 model calls/s; at ~4K prompt tokens each, ~5.2M input tokens/s, half a cacheable prefix.
    • Latency: 100 + 150 + 300 + 100 ≈ 650 ms in sequence, so the stages must overlap to land near 480 ms.
    • Cost: the pipeline runs cents per minute against roughly $1 per minute for a human agent. Cost under concurrency is the thing to watch: 10K open pipelines cost money even when nobody is speaking.

    Rough numbers are fine; the method is what gets graded. The point: this system is sized by concurrent calls and bounded by latency, not by bandwidth or storage.

    Components to put on the board

    Audio flows in through telephony to a media gateway, which pins each call to one session orchestrator. The orchestrator is the hub: it streams audio to the turn detector and the recognizer, streams text to and from the LLM, and streams text to TTS and audio back out. Tools, recordings, and analytics hang off to the side, never on the path of the next spoken sentence.

    Mobile Client (phone caller) → Third-party API (Telephony / SIP) → Load Balancer (sticky by call) → WebSocket Server (Media Gateway, distributed)
    Mobile Client (app) → Load Balancer → WebSocket Server (Media Gateway)
    WebSocket Server ⇄ Service (Session Orchestrator, distributed)
        Service ⇄ ML Model (Turn Detector)
        Service ⇄ ML Model (Streaming STT)
        Service ⇄ ML Model (LLM)
        Service ⇄ ML Model (Streaming TTS)
        Service → Service (Tool Gateway) → Third-party API (Account / billing APIs)
                                       → Postgres (tool audit, tenant keyterms)
        Service → Elasticsearch (Retrieval index, queried on the partial transcript)
        Service ⇄ Redis (session snapshots)
        Service → Kafka (turn + timing events) → Worker (recording + redaction) → Blob Store
        Service → Queue (human handoff)
    

    Components:

    • Third-party API (Telephony / SIP). Terminates the phone call and streams 8 kHz audio both ways over a media WebSocket, with mark and clear messages. Without it there is no phone channel.
    • Load Balancer. Places a new call on a gateway and keeps every frame of that call on it. Sticky by call id.
    • WebSocket Server (Media Gateway). Decodes, jitter-buffers, denoises, and relays audio frames between the caller and the call's orchestrator. Handles ~1M small messages/s across the tier, so it is distributed.
    • Service (Session Orchestrator). One per call. Owns the turn, retrieval, tool calls, and cancellation; holds the history and every open stream; chunks model tokens into TTS; runs barge-in. The load-bearing component: without it there is nobody to overlap stages or cancel them.
    • ML Model (Turn Detector). Voice activity plus a semantic completeness model. Emits eager and confirmed end-of-turn, and detects interruptions.
    • ML Model (Streaming STT). Partial and final transcripts, biased with a contextual vocabulary. One open stream per call.
    • Elasticsearch (Retrieval index). Domain knowledge embedded with a domain-tuned model. Queried on the partial transcript, in parallel with the model call, not as a step the model waits on.
    • ML Model (LLM). Small, fast model on the hot path; streams tokens and emits tool calls. Cached prompt prefix.
    • ML Model (Streaming TTS). Audio for each clause as soon as it arrives.
    • Service (Tool Gateway). Enforces tiers, allowlists, timeouts, and idempotency keys for account actions; writes the audit log.
    • Redis. Per-turn session snapshot so a crashed orchestrator loses one turn, not the call.
    • Kafka. Every turn, timing, and tool event. Takes recordings, transcripts, and metrics off the hot path.
    • Worker → Blob Store. Writes stereo recordings and PII-redacted transcripts; 90-day lifecycle.
    • Queue (human handoff). Warm transfer with transcript and summary.

    Walk one caller turn: "I was charged twice for my July bill"

    1. Caller audio crosses the carrier to Telephony, which streams 20 ms frames to the call's Media Gateway through the Load Balancer.
    2. The gateway cleans the audio and relays it to the call's Session Orchestrator.
    3. The orchestrator streams frames to Streaming STT and the Turn Detector; partial transcripts arrive continuously, so the transcript is essentially complete when the caller stops.
    4. About 50 ms after the caller stops, the Turn Detector says "probably done." The orchestrator starts retrieval and a speculative model call on the partial transcript, in parallel. At ~100 ms the turn is confirmed; the final transcript (~130 ms) matches, so the draft stands.
    5. The model asks for get_bill(July). The orchestrator plays a short filler through TTS and calls the Tool Gateway with an idempotency key; the account was already loaded at call start.
    6. The tool returns; the model streams the answer. The orchestrator sends the first clause to Streaming TTS. On a turn with no tool, first audio leaves the gateway around 480 ms after the caller stopped.
    7. Carrier mark echoes confirm what has played. Every step is logged to Kafka with timestamps.

    Caller interrupts ("No, the June one"): the Turn Detector flags sustained speech that is not a backchannel. The orchestrator sends clear to the carrier, cancels TTS and the LLM stream, trims its last message to the words up to the last confirmed mark, and goes back to listening. Orchestrator crashes mid-call: the gateway reattaches the call to a new orchestrator, which loads the Redis snapshot and asks the caller to repeat. Pending tool calls are safe to re-issue because they are idempotent. Slow tool (2 s): the filler covers the first second; past a timeout the agent says it is still checking rather than going silent.

    Schema

    CallSession(session_id PK, tenant_id, caller_ani, state, turn_seq, history, last_played_mark, pending_tools, node_id)   -- Redis, TTL
    TurnEvent(session_id, turn_seq, user_text, t_speech_end, t_eot, t_llm_first_token, t_tts_first_byte, interrupted, played_text)   -- Kafka → warehouse
    ToolCall(idempotency_key PK, session_id, tool, args, tier, status, result, confirmed_by_caller)   -- Postgres
    TenantConfig(tenant_id PK, keyterms, prompt_version, voice_id, tool_allowlist)   -- Postgres, cached
    Recording(session_id, blob_path, retention_until)   -- Blob Store
    
    • Session state is keyed and routed by session_id; Redis holds it only for the life of the call.
    • Tool calls are keyed by session_id:turn_seq:tool_call_id, so retries return the first result.
    • Turn events are append-only and partitioned by session_id in Kafka, which preserves per-call order.

    Scaling and wrap-up

    • Gateways and orchestrators scale out by open-call count behind sticky load balancing; STT by open streams; LLM and TTS pools by queue depth and first-token p95.
    • Co-locate gateway, orchestrator, and model services in one region; cross-region hops eat the budget.
    • What breaks first, in order: recognition on domain terms, turn detection, a slow tool call blowing the budget, network jitter (jitter buffer), cost under concurrency.
    • First metrics: per-stage p95 (end-of-turn, model first token, TTS first audio), domain-term error rate, retrieval recall@k, handoff rate.
    • If time: hybrid speech-to-speech, multilingual callers, outbound calling rules, admission control at peak.

    Equally defensible variants

    SIP/RTP media servers instead of a provider media WebSocket (lower overhead at volume, more to operate). WebRTC for the app channel is the default; a WebSocket works on good networks. A speech-to-speech model for app-only, answer-only flows. Silence plus VAD turn detection for a simple bot, if you account for the latency. Human approval instead of step-up verification for high-risk actions. What matters is that you can argue the tradeoff.

    More in System Design Interview Prep: AI Coach Edition

    • SD Coach · Media Streaming
    • SD Mock · Design YouTube / Netflix
    • SD Mock · Design Instagram
    • SD Coach · Agentic AI
    • SD Coach · Agentic AI Architectures (Slides)
    • SD Mock · Design ChatGPT
    • SD Mock · Design Claude Code
    • SD Mock · Design an AI Customer Support System
    • SD Mock · Design a File Uploader for an AI Chat App
    • SD Mock · Design an LLM Query Batching System
    • SD Coach · Reliability and Security
    • SD Mock · Design It Live
    View the full course (43 sessions) →