Loading...

A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.
The question: a customer calls a support line and talks to an AI that understands them, looks things up on their account, and answers out loud. Speech is far less forgiving than chat. A pause longer than about a second feels broken.
Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):
Build incrementally while you talk. Trace one caller turn end to end through your diagram, with rough timestamps. Argue the tradeoff, not "industry standard."
Read this before you go in. These are the decisions, and why they matter.
Write the budget first. The obvious design chains three calls, speech-to-text, then an LLM, then text-to-speech, each run to completion: over 3 seconds of dead air once you include the silence you waited through. Even with every stage streaming, running them in sequence costs about 650 ms: speech-end detection 100 ms + final transcription 150 ms + model first token 300 ms + synthesis start 100 ms. That already misses 500 ms, so the stages must overlap. Transcribe continuously, so the final transcript is a confirmation, not a computation. When the turn detector is fairly sure the caller is finishing (around 50 ms), start retrieval and a speculative model call on the partial transcript; if the last words change the meaning, cancel and restart. Send the first clause to synthesis before the model has finished. The critical path becomes 50 + 300 + ~30 + 100, about 480 ms. What the caller feels is time to the first syllable, not time to the full answer. Say where you start and stop the clock: phone carrier legs add a few hundred milliseconds you cannot control.
A fixed silence timer is the naive answer. It fails both ways: people pause mid-sentence (reading out an account number), so a short timer cuts them off, and whatever you choose is added to every reply. Layer it instead: voice activity detection says whether anyone is speaking, a small model reads the partial transcript and judges whether the thought is complete, and a short silence floor confirms. Be more patient right after you asked for a number. An "eager" signal lets you start the LLM early, at the cost of discarded drafts when the caller keeps talking. Silence plus VAD only is equally defensible for a simple bot if you can say what it costs in latency.
Keep listening while the agent speaks. When the caller talks for more than about a quarter second, and it is not a "mm-hm", treat it as an interruption: flush the audio already queued at the carrier (a clear message), cancel the TTS stream and the LLM generation, and rewrite the agent's last message to only the words the caller actually heard. Carrier playback confirmations (mark messages) tell you where playback reached. Skip the history fix and the model believes the caller heard things they never did. The main cause of false interruptions is echo: the agent's own voice leaking back into a speakerphone mic.
A call is a multi-minute session, not a request. Something must own its state machine (Listening, Thinking, Speaking, Interrupted), its conversation history, and its open streams. Put one session orchestrator per call, pinned for the whole call, with the media gateway in front of it. Snapshot state to Redis at every turn boundary: if the node dies, the call reattaches to a new orchestrator and loses at most one turn. Stateless handlers that reload everything per audio frame are too slow; a single giant stateful server is a blast-radius problem.
The pipeline (separate recognition, model, and synthesis) is swappable, debuggable, and loggable: you can keep a transcript, bias recognition, and enforce guardrails at the seams. It costs latency across three hops, and tone and hesitation are lost at transcription. A speech-to-speech model is lower latency and sounds far more natural, but gives up exactly those seams: no transcript to audit, no place to bias vocabulary, no clean guardrail point. Choose from the requirements. A regulated support line with specialized vocabulary favours the pipeline; a consumer product where naturalness is the product favours speech-to-speech; a hybrid uses speech-to-speech for conversation and a pipeline path for tool calls and audit records.
Tool calls take 200 ms to seconds, and some move money. Keep the caller informed: load the account summary at call start so most turns need no tool, speak a short honest filler ("Let me check that") the moment a tool starts, and time out with a holding line instead of silence. Keep actions safe: tier tools (read, low-risk write, high-risk write), put an idempotency key on every call so a retry cannot refund twice, and read back anything consequential before doing it. The tool gateway enforces this, not the prompt.
A specialized domain uses embeddings in three separate roles; keep them apart so you can measure and fix each one. Recognition biasing: a general recognizer confidently turns product names into common words. Give it a vocabulary, but make it contextual: the ~50 terms plausible at this point in the call beat all 10,000 product names, which dilute the bias. Retrieval embeddings: off-the-shelf embedders don't know your enterprise plan and individual plan are different. Fine-tune on real question-to-passage pairs, and train against hard negatives, the look-alike wrong passages. Speaker embeddings: these map a voice, not meaning. Use them to tell two people on the line apart, and for voice verification only as one signal among several, because voices can be cloned. Then evaluate separately: recognition by error rate on domain terms (5% overall error is useless if every product name is wrong), retrieval by recall@k on a golden set. Phone audio is also 8 kHz and noisy, so the recognizer should be telephony-tuned.
One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.
Rough numbers are fine; the method is what gets graded. The point: this system is sized by concurrent calls and bounded by latency, not by bandwidth or storage.
Audio flows in through telephony to a media gateway, which pins each call to one session orchestrator. The orchestrator is the hub: it streams audio to the turn detector and the recognizer, streams text to and from the LLM, and streams text to TTS and audio back out. Tools, recordings, and analytics hang off to the side, never on the path of the next spoken sentence.
Mobile Client (phone caller) → Third-party API (Telephony / SIP) → Load Balancer (sticky by call) → WebSocket Server (Media Gateway, distributed)
Mobile Client (app) → Load Balancer → WebSocket Server (Media Gateway)
WebSocket Server ⇄ Service (Session Orchestrator, distributed)
Service ⇄ ML Model (Turn Detector)
Service ⇄ ML Model (Streaming STT)
Service ⇄ ML Model (LLM)
Service ⇄ ML Model (Streaming TTS)
Service → Service (Tool Gateway) → Third-party API (Account / billing APIs)
→ Postgres (tool audit, tenant keyterms)
Service → Elasticsearch (Retrieval index, queried on the partial transcript)
Service ⇄ Redis (session snapshots)
Service → Kafka (turn + timing events) → Worker (recording + redaction) → Blob Store
Service → Queue (human handoff)
Components:
mark and clear messages. Without it there is no phone channel.get_bill(July). The orchestrator plays a short filler through TTS and calls the Tool Gateway with an idempotency key; the account was already loaded at call start.mark echoes confirm what has played. Every step is logged to Kafka with timestamps.Caller interrupts ("No, the June one"): the Turn Detector flags sustained speech that is not a backchannel. The orchestrator sends clear to the carrier, cancels TTS and the LLM stream, trims its last message to the words up to the last confirmed mark, and goes back to listening.
Orchestrator crashes mid-call: the gateway reattaches the call to a new orchestrator, which loads the Redis snapshot and asks the caller to repeat. Pending tool calls are safe to re-issue because they are idempotent.
Slow tool (2 s): the filler covers the first second; past a timeout the agent says it is still checking rather than going silent.
CallSession(session_id PK, tenant_id, caller_ani, state, turn_seq, history, last_played_mark, pending_tools, node_id) -- Redis, TTL
TurnEvent(session_id, turn_seq, user_text, t_speech_end, t_eot, t_llm_first_token, t_tts_first_byte, interrupted, played_text) -- Kafka → warehouse
ToolCall(idempotency_key PK, session_id, tool, args, tier, status, result, confirmed_by_caller) -- Postgres
TenantConfig(tenant_id PK, keyterms, prompt_version, voice_id, tool_allowlist) -- Postgres, cached
Recording(session_id, blob_path, retention_until) -- Blob Store
session_id; Redis holds it only for the life of the call.session_id:turn_seq:tool_call_id, so retries return the first result.session_id in Kafka, which preserves per-call order.SIP/RTP media servers instead of a provider media WebSocket (lower overhead at volume, more to operate). WebRTC for the app channel is the default; a WebSocket works on good networks. A speech-to-speech model for app-only, answer-only flows. Silence plus VAD turn detection for a simple bot, if you account for the latency. Human approval instead of step-up verification for high-risk actions. What matters is that you can argue the tradeoff.