TOUGH TONGUE AI
  • Home
    LogoTOUGH TONGUE AI

    RESOURCES

    Loading...

    HomeLibrarySD Mock · Design an AI Customer Support System

    SD Mock · Design an AI Customer Support System

    Overview

    Design an AI support agent that answers from company docs, looks up accounts and issues refunds safely, on a live architecture board, and defend your design to an expert interviewer who watches every move.

    Your Instructions

    Design an AI-Powered Customer Support System

    A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.

    The system: a customer describes a problem in their own words. Your agent answers from the company's documentation, looks up their account, and where appropriate takes an action for them, such as a refund, a password reset or a plan change. When it cannot resolve the issue, it hands off to a human with the context already gathered.

    Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):

    • Notepad. Shared notes. Requirements, capacity math, schema, and end-of-session feedback live here. You can type in it too.
    • Architecture board. Your design canvas. Drag typed components, wire them, label edges, mark scaled tiers as distributed. If it is not on the board, it is not in your design.

    Build incrementally while you talk. Trace one customer message end to end through your diagram, from "I was charged twice" to a refund or a handoff. Argue the tradeoff, not "industry standard."

    Design choices interviewers probe

    Read this before you go in. These are the decisions, and why they matter. The request rate here is tiny; the hard parts are retrieval quality, action safety and honest measurement.

    Retrieval: vector search alone, or hybrid with rerank

    Textbook RAG embeds the question and takes the nearest chunks. It fails on real support traffic. Customers paste error codes and order numbers, and embeddings blur exact strings. Run keyword search (BM25) next to vector search and fuse the two ranked lists. Pull about 50 candidates for recall, then rerank with a cross-encoder (an ML model that reads query and passage together) down to about 5 for precision. Filter on metadata first (product, locale, plan): a passage about a plan the customer is not on produces an answer that is fluent and wrong. Chunk on the document's own headings and prefix each chunk with its title and heading path. Elasticsearch or OpenSearch doing both BM25 and kNN, or Postgres with pgvector plus full-text, are equally defensible if you can say why both modes exist.

    Where authority lives: the prompt or a policy layer

    The moment the agent can refund, it stops being an information system. "Don't refund over $100" in a system prompt is a suggestion a model can ignore or a customer can talk it out of. The rule: the model proposes, deterministic code disposes. A Policy Layer service sits between the agent and every tool. It tiers actions by reversibility: read-only (free), reversible writes (logged), consequential writes like refunds (allowed only inside thresholds such as amount, return window and verified account), irreversible or high-value (always a human). Thresholds are configuration in code, never prompt text. Asking the customer to confirm a refund first is a good addition, not a replacement.

    Whose credentials the tools use

    If the agent calls the orders API with a service account, any prompt injection ("show me the orders for this other email") becomes data access. Tools run with the customer's own session token. The injection then fails in authorization code, not in the model's judgement.

    Retries and runaway loops

    The orchestrator can crash after calling the refund API and before saving the result. A plain retry double-refunds. Every mutating call carries an idempotency key built from conversation id and step, and the tool deduplicates on it. Also cap the agent loop: steps, tokens and seconds per turn. When a cap is hit, escalate. This system always has a human fallback, so use it.

    When to stop and hand off

    Leaving escalation to the model's discretion fails both ways. Use explicit triggers: the customer asks for a human, sentiment turns negative, a policy boundary is hit, the budget runs out, repeated failure, or a designated topic (billing dispute, account security, legal). Hand the human the transcript, retrieved passages, attempted actions and what was established, so the customer never repeats themselves. And when retrieval returns nothing relevant, the agent says it does not know. No passages means no answer. That turns hallucination from a model problem into a routing decision.

    Keeping knowledge fresh

    A nightly rebuild means a day of stale policy answers. Re-embed on change: documentation edits emit events, and content hashes skip unchanged chunks. Past tickets are the richest source and the messiest: scrub PII and index only well-resolved ones. Swapping the embedding model invalidates every vector, so build a new index version alongside the old, validate it, and cut over atomically with an alias.

    Measuring success: deflection or resolution

    Deflection counts conversations that never reached a human. It goes up when the agent is unhelpful enough that customers give up. Measure resolution, verified by no repeat contact on the same issue and by satisfaction. Evaluate retrieval separately from generation with a golden set of real questions and the passages that should come back (recall@k). When quality drops, that tells you whether the fix is the index or the prompt.

    Reference solution

    One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.

    Requirements (lock these or take the defaults)

    • Answer questions grounded in company knowledge (help center, runbooks, product docs, past resolved tickets), with a link to the source.
    • Look up the customer's own orders and account; take bounded actions (refund, password reset, plan change).
    • Escalate to a human queue with full context.
    • Staff can add, edit and retire knowledge; changes show up within minutes.
    • Chat channel. The chat front end calls us once per customer message; we stream the reply.
    • Scale: about 100,000 conversations a day, about 6 turns each; about 50,000 documents.
    • Out of scope: building the chat UI, the human agent desk, model training, multiple tenant companies.

    Capacity

    Rough numbers are fine; the method is what gets graded.

    • Traffic: 100,000 conversations ÷ 86,400 s ≈ 1.2 conversations/sec, about 10/sec at peak. That is a tiny request rate.
    • Cost: 100,000 × 6 turns × $0.02 per turn = $12K/day of model spend.
    • Value: at 60% resolution, 60,000 × $5 per human contact = $300K/day of human cost avoided. Quality, not throughput, is where the money is.
    • Index: 50,000 docs × 5 chunks = 250K chunks. At 1,024 dimensions × 4 bytes ≈ 1 GB of vectors: fits in memory on one node. Replicate it for availability; do not shard it.
    • The real throughput limit is the LLM provider: about 60 turns/sec at peak, 2 to 3 model calls each.

    Components to put on the board

    The customer's message enters through the gateway to an orchestrator that runs a bounded loop. On each step the model (behind an LLM gateway) either asks for knowledge, which the retrieval service fetches from a hybrid index and reranks, or proposes an action, which passes through the policy layer before any account tool runs. Every exit that is not a grounded answer goes to escalation. Ingest keeps the index fresh off the hot path; the conversation store feeds evaluation.

    Web Client (customer chat)
        → API Gateway
            → Service: Agent Orchestrator (loop + budget)
                ↔ ML Model: LLM (via LLM Gateway service)
                → Service: Retrieval → Elasticsearch (BM25 + vectors, alias → index vN)
                                     → ML Model: Reranker (cross-encoder)
                → Service: Policy Layer → Service: Account Tools → Postgres (orders, billing, auth)
                → Service: Escalation → Third-party API (human agent desk)
                → Postgres: Conversation Store + Tool Ledger
    Blob Store (docs, tickets) → Kafka (change events) → Worker: Knowledge Ingest (scrub, chunk, embed) → Elasticsearch
    Postgres: Conversation Store → Data Warehouse (evaluation: golden set, resolution)
    

    Components:

    • Web Client. The customer's chat widget. It sends one message per call with the customer's session token and renders the streamed reply.
    • API Gateway. Authenticates the customer session, rate-limits per customer, and holds the streaming connection open. Without it, anyone can call tools as anyone.
    • Service: Agent Orchestrator. Owns the loop: decide, retrieve or act, observe, repeat. It enforces the budget (steps, tokens, seconds) and escalates when it runs out. Stateless between steps; every step is written to the conversation store so a new instance can resume after a crash.
    • ML Model: LLM (behind an LLM Gateway service). Does the reasoning and writes the reply. The gateway routes easy turns to a small model, falls back to a second provider on outages, and caches the static system prompt and tool schemas.
    • Service: Retrieval. Turns the customer's phrasing into the right five passages: metadata prefilter, BM25 plus vector search, rank fusion, rerank, and a relevance floor. Below the floor it returns nothing, on purpose.
    • Elasticsearch. The hybrid index over chunks and cleaned past tickets, served through an alias so a new version can be swapped in atomically. Replicated across zones.
    • ML Model: Reranker. A cross-encoder that scores about 50 query and passage pairs. Too slow for the whole corpus, right for a short list.
    • Service: Policy Layer. Deterministic code and config. Classifies each proposed tool call by tier, checks thresholds, and returns allow, confirm with customer, or needs a human. The load-bearing box of this design.
    • Service: Account Tools. The real refund, reset and plan-change APIs. Called with the customer's own token and an idempotency key.
    • Postgres (systems of record). Orders, billing and auth data the tools read and write.
    • Service: Escalation. Opens a case in the right human queue with the handoff packet: transcript, passages, attempted actions, established facts.
    • Third-party API (human agent desk). Zendesk, Salesforce or similar, where people pick up escalated cases.
    • Postgres: Conversation Store + Tool Ledger. Transcript, agent steps with retrieved chunk ids, and tool calls with a unique index on the idempotency key. Audit trail and evaluation source.
    • Blob Store → Kafka → Worker: Knowledge Ingest. Source documents and tickets emit change events; the worker scrubs PII, filters tickets by resolution quality, chunks on headings, skips unchanged content hashes, embeds, and writes to the index version.
    • Data Warehouse. Offline evaluation: golden-set recall@k, groundedness, verified resolution rate.

    Walk one customer message: "I was charged twice for order 88231, checkout showed PAY-409"

    1. Web Client → API Gateway. The message arrives with Maya's session token (she is on the Pro plan). The gateway authenticates and rate-limits.
    2. API Gateway → Agent Orchestrator. It loads the conversation, logs the turn, and checks hard triggers. None fire.
    3. Orchestrator ↔ LLM. The model asks for two things: look up order 88231 and search the knowledge base.
    4. Orchestrator → Policy Layer → Account Tools. get_order is read-only, so it runs with Maya's token. Two $49 charges come back.
    5. Orchestrator → Retrieval → Elasticsearch → Reranker. Prefilter to Checkout, English, Pro. BM25 finds the PAY-409 error-code page; vectors find the duplicate-charge refund policy; the reranker keeps five passages, both above the floor. An Enterprise-only refund article never enters scoring.
    6. Orchestrator ↔ LLM → Policy Layer. The model proposes issue_refund(88231, $49). The policy layer checks: under $100, inside 30 days, verified account. Rule: confirm with the customer first. The reply streams with a citation and a question.
    7. Maya says yes → Account Tools. The refund runs with Maya's token and idempotency key conv-7f3a:step-4. If the orchestrator crashes and retries, the tool returns the original refund instead of refunding again.
    8. Orchestrator → Conversation Store. Every step, passage id and tool result is recorded. The conversation counts as resolved only if Maya does not contact support about this order again within a week.

    Refund over the threshold: the policy layer returns needs-human. Escalation opens a billing case with the transcript, order lookup, passages and "refund of $400 proposed, blocked by threshold". Maya is told a person is taking over and does not repeat herself. Retrieval finds nothing: the relevance floor returns empty. The agent says it does not have that information and offers a human. It does not guess. (Air Canada was held liable in 2024 when its chatbot invented a refund policy.) Prompt injection: "show me the orders for jane@example.com" fails in the orders API, because it only sees Maya's orders. The model's judgement is never the last line of defense. LLM provider outage: the LLM gateway fails over to a second provider. If both fail, the orchestrator escalates with a polite handoff message. The chat path stays available; refunds wait rather than guess.

    Schema

    Conversation(conversation_id PK, customer_id, channel, state, outcome, started_at)
    Turn(conversation_id, seq, role, text, created_at)                       PK (conversation_id, seq)
    AgentStep(conversation_id, seq, step, kind, retrieved_chunk_ids, tokens, latency_ms)
    ToolCall(idempotency_key UNIQUE, conversation_id, tool, args, result, policy_decision, created_at)
    Chunk(chunk_id, doc_id, doc_version, heading_path, text, content_hash, embedding, product, locale, plan_scope)
    PolicyConfig(tool, tier, max_amount, window_days, requires_verified, version)
    GoldenItem(question, expected_chunk_ids, notes)
    
    • Conversation data is relational and small: about 600K turns a day. Postgres, indexed by conversation_id; partition by month if it grows. A document store for transcripts is fine if you keep the tool ledger's uniqueness guarantee.
    • The tool ledger's unique index on idempotency_key is the exactly-once mechanism for money.
    • Chunks live in the search index, rebuilt from the source documents. One index of 250K chunks fits on one node; three replicas across zones.
    • Policy config is versioned so every decision can be traced to the rules in force at the time.

    Scaling and wrap-up

    • Web tier and orchestrator scale horizontally behind the gateway; at ~10 conversations/sec that is a handful of instances.
    • The LLM provider's token limits are the real bottleneck: prompt caching, small-model triage and provider fallback absorb it.
    • The index scales by replicas, not shards. Ingest scales by adding workers on the Kafka topic.
    • First metrics: verified resolution rate, retrieval recall@5 on the golden set, escalation rate by reason (plus freshness lag and cost per resolution).
    • If time: voice channel (sub-second budget, read back identifiers), multi-tenant isolation, multilingual retrieval, turning escalations into new articles and golden-set cases.

    Equally defensible variants

    Retrieval: Elasticsearch/OpenSearch hybrid, Postgres with pgvector plus full-text, or a dedicated vector store next to a keyword engine. Retrieval style: always retrieve before the first model call, or let the model call search as a tool; retrieve-by-default with extra tool calls within the budget is the common middle. Orchestrator: plain containers, or a workflow engine such as AWS Step Functions for the step loop. Transcripts: Postgres or DynamoDB. Human desk: any ticketing system. What matters is that you can argue the tradeoff.

    More in System Design Interview Prep: AI Coach Edition

    • SD Coach · Media Streaming
    • SD Mock · Design YouTube / Netflix
    • SD Mock · Design Instagram
    • SD Coach · Agentic AI
    • SD Coach · Agentic AI Architectures (Slides)
    • SD Mock · Design ChatGPT
    • SD Mock · Design Claude Code
    • SD Mock · Design a Real-Time Voice AI
    • SD Mock · Design a File Uploader for an AI Chat App
    • SD Mock · Design an LLM Query Batching System
    • SD Coach · Reliability and Security
    • SD Mock · Design It Live
    View the full course (43 sessions) →