TOUGH TONGUE AI
  • Home
    LogoTOUGH TONGUE AI

    RESOURCES

    Loading...

    HomeLibrarySD Mock · Design ChatGPT

    SD Mock · Design ChatGPT

    SD Mock · Design ChatGPT

    Overview

    Design a consumer AI chat product like ChatGPT on a live architecture board: drag components, wire them up, and defend your design to an expert interviewer who watches every move.

    Your Instructions

    Design ChatGPT

    A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.

    Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):

    • Notepad. Shared notes. Requirements, capacity math, schema, and end-of-session feedback live here. You can type in it too.
    • Architecture board. Your design canvas. Drag typed components, wire them, label edges, mark scaled tiers as distributed. If it is not on the board, it is not in your design.

    Build incrementally while you talk. Trace one message end to end through your diagram, from Enter to the last streamed word. Argue the tradeoff, not "industry standard."

    Design choices interviewers probe

    Read this before you go in. These are the decisions, and why they matter.

    Size in tokens, not requests

    A "thanks" and a pasted 50-page contract are both one request. They differ in GPU cost by three orders of magnitude, so any limit or capacity plan built on requests per second is wrong. Split tokens into input and output. Output is generated one token at a time and sets how long an answer takes. Input is the whole prompt, and because every turn re-sends the conversation, it runs about six times the output. That ratio is the most important number in the interview: it says the money is in the prompt.

    Stream the answer, and make the stream behave

    A 500-token answer takes around ten seconds. Delivered at once, it feels broken. With the first token in half a second, it feels instant. So stream. Server-Sent Events is the natural fit: the server pushes, the client only listens, it runs over plain HTTP and reconnects on its own. WebSockets are equally defensible if you want two-way traffic such as voice or interruptions, but they carry more operational weight. Three things follow. Time to first token is the latency that matters, because total time is fixed by the model. Requests live for seconds to minutes, so open connections, not request rate, measure the serving tier. And when the user closes the tab or presses Stop, the cancel must travel all the way to the running model, or you pay for tokens nobody reads. Save whatever was generated so a reconnect shows a partial answer, not an empty turn. A short grace period before cancelling lets a phone that switches networks catch up.

    Context: what goes into the prompt each turn

    The model remembers nothing, and the context window is finite. Every turn you rebuild the prompt from stored history. A sliding window of the last N turns is simple but quietly drops the rule the user set at the start. A rolling summary of older turns plus recent turns verbatim is the usual workhorse; it costs an extra model call when the summary refreshes. Retrieving relevant old turns by embedding helps in very long threads. Whatever you choose, always keep the system prompt and the user's first message, which usually defines the task.

    Cost: order the prompt for prefix caching

    Between turn 7 and turn 8, most of the prompt is identical. Inference providers can keep the computed state for a repeated prefix and skip recomputing it, usually at a fraction of the price. That only works if stable content comes first and variable content last: system prompt, memory facts, first message, summary, older turns, then the new message. A timestamp near the top, or notes interleaved between history blocks, breaks the match for everything after it. With input six times output, this ordering is the biggest cost lever you have. Refreshing the summary rewrites the top of the prompt, so refresh it rarely.

    Model choice: route by difficulty

    Most turns are easy: "make it shorter", "thanks", a quick fact. A smaller model answers them faster and far more cheaply. Put a lightweight classifier at the inference gateway that sends hard turns to the large model, or let the small model escalate when unsure. The risk is misrouting a hard question; keep thresholds conservative and let paid users pin the large model. Sending every message to the biggest model is the naive answer.

    Overload: GPUs cannot be autoscaled

    GPU capacity is fixed on any timescale shorter than buying hardware, so demand above supply is normal. Decide who gets served. A higher message cap for paid users does nothing when everyone waits in one line; give paid traffic priority in the queue instead, with a minimum share so free users slow down rather than starve. Pick one visible degradation for the product you scoped: show a queue position, fall back to the smaller model with a label, or reject with a retry time. Enforce per-user and per-tier budgets in tokens at the inference gateway, before a request occupies a GPU. Cache exact repeats; cache near-duplicates only with a conservative similarity threshold and only for prompts that do not depend on history, because a wrong cached answer is worse than recomputing.

    Memory is not context

    In-conversation context is the thread's own history. Cross-conversation memory is a small set of durable facts about the user (preferences, ongoing projects, stated constraints), extracted by a background job after a conversation and pulled into future prompts. Keep it small, because every fact costs tokens on every future turn. Make it visible and editable, because a wrong fact silently degrades everything afterward. Scope it: users must be able to turn it off and delete it. Treating "all history" as memory is a common mistake.

    Safety and abuse

    Classify input before any GPU time is spent. Output classification clashes with streaming: tokens are on screen before a problem is detected. Either buffer a small window before display, or classify incrementally and stop the stream. Most abuse is automated scraping through free accounts; it shows up as high rates, regular timing and no reading pauses. Per-account rate limits combined with token budgets cap what any one account is worth.

    Reference solution

    One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.

    Requirements (lock these or take the defaults)

    • Multi-turn chat with occasional tool calls; streamed answers; stop, regenerate, edit.
    • List, open, rename, delete conversations. Remember durable user facts across conversations; users can view, edit, delete them.
    • 200M weekly users, 50M daily, ~10 messages each per day.
    • Answers ~500 tokens; prompts ~3,000 tokens because history is re-sent; conversations outgrow the context window.
    • Free and paid tiers. First token well under a second.
    • Inference is a service with known latency and cost. Out of scope: training, running the model internals.

    Capacity

    • Messages: 50M × 10 = 500M/day ≈ 6K/sec average, ~20K/sec at peak.
    • Tokens: 500M × 500 = 250B output/day; 500M × 3,000 = 1.5T input/day. Input is ~6× output because history is re-sent.
    • Connections: 20K/sec arriving × ~10 sec per answer ≈ 200K open streams at peak.
    • Storage: 500M × 2 KB ≈ 1 TB/day. Trivial next to the compute.

    Rough numbers are fine; the method is what gets graded.

    Components to put on the board

    Two halves joined by an inference gateway. The conversation half is ordinary web-scale plumbing: gateway, chat service, stores, filters, async workers. The inference half is a scarce GPU fleet behind a gateway that picks a model tier and enforces budgets, and a queue that admits paid traffic first. The hot path is client → API gateway → chat service → inference gateway → queue → model → tokens streamed back. Storage writes, memory extraction and billing stay off it.

    Web Client / Mobile Client
        -> CDN -> API Gateway -> Chat Service (Service)
                                   -> Cassandra (conversations, messages)
                                   -> Postgres (memory facts)
                                   -> ML Model (safety filters)
                                   -> Inference Gateway (Service)
                                          -> Rate Limiter (token budgets, Redis)
                                          -> Cache (response cache)
                                          -> Queue (tiered: paid > free)
                                                -> ML Model (small model)
                                                -> ML Model (large model)
                                   <- token stream from the models
                                   -> Redis (stream buffer for reconnects)
                                   -> Kafka -> Worker (title, summary, memory extraction) -> Postgres
        <- SSE stream from Chat Service via API Gateway; cancel flows back
    

    Components:

    • Web Client / Mobile Client. Opens the stream, renders tokens, sends an idempotency key, cancels on Stop or close.
    • CDN. Static assets and TLS close to the user; long-lived streams terminate at the edge.
    • API Gateway. Authentication, per-user rate limits, and tier identification. Proxy timeouts set for minute-long responses.
    • Chat Service. The orchestrator. Persists the user message, loads memory, assembles the prompt stable-first, runs tool calls, relays tokens, saves the answer or the partial. Stateless and horizontally scaled.
    • Cassandra. Conversations and messages, partitioned by conversation and ordered by a time-sortable ID. Write-heavy and simple to query.
    • Postgres (memory). A handful of durable facts per user, editable by the user. Small by design.
    • ML Model (safety filters). Input check before GPU spend; output check on a small buffered window of the stream.
    • Inference Gateway. Picks the model tier by difficulty, enforces token budgets, checks the response cache, and rejects before a GPU is touched. Without it, every message hits the big model and overload is first come, first served.
    • Rate Limiter. Token buckets per user and tier.
    • Cache. Exact and conservative semantic matches for context-free prompts.
    • Queue. Tiered admission that batches requests for the GPUs. Paid traffic goes first under saturation.
    • ML Model (small / large). Two GPU pools. Easy turns go small; hard turns go large.
    • Redis. Short-lived delta buffer per message so a reconnecting client catches up.
    • Kafka + Worker. Usage and billing, titles, summary refresh, and memory extraction after a conversation goes idle.

    Walk one message: "Plan a 3-day trip to Jaipur. I'm vegetarian."

    1. The Mobile Client posts the message with an idempotency key and opens an SSE stream.
    2. The API Gateway authenticates and tags the tier. The Chat Service saves the message to Cassandra before anything expensive.
    3. The safety ML Model checks the text. The Chat Service loads the user's memory facts from Postgres and builds the prompt: system, memory, first message, summary, recent turns, new message.
    4. The Inference Gateway debits the token budget, misses the cache (the prompt depends on history), and classifies the turn as hard. It routes to the large model.
    5. The Queue admits it ahead of free traffic. The model reuses the cached prefix and emits the first token.
    6. The Chat Service streams tokens over SSE, checks output on a small buffer, and appends to the partial answer.
    7. At the end, the answer is saved and a usage event goes to Kafka. When the conversation goes idle, a Worker extracts "vegetarian" as a memory fact.

    Tab closed mid-answer: after a short grace period with no reconnect, the Chat Service cancels at the gateway, the GPU stops, and the partial answer is saved. Phone switches networks: the client reconnects with its last event ID and catches up from the Redis buffer. Launch-day spike: paid traffic keeps priority; free traffic gets the stated degradation (for example the small model, labelled) or a retry time.

    Schema

    User(user_id, plan, created_at)                                      -- Postgres
    MemoryFact(user_id, fact_id, text, source_conversation_id, created_at) -- Postgres
    Conversation(user_id, updated_at, conversation_id, title,
                 current_leaf_id, summary)                               -- Cassandra
    Message(conversation_id, message_id, parent_id, role, content,
            status, model, input_tokens, cached_tokens, output_tokens)   -- Cassandra
    StreamBuffer(message_id -> [event_id, text], TTL 10m)                -- Redis
    
    • Messages: partition key conversation_id, clustering key message_id (time-ordered).
    • Conversation list: partition key user_id, clustered by updated_at descending.
    • parent_id makes each conversation a tree; regenerate and edit add branches; current_leaf_id marks the active path.
    • Replication factor 3; status covers generating, complete, partial.

    Scaling and wrap-up

    • API gateway, Chat Service and workers scale horizontally; Cassandra scales by partition.
    • Prefix caching absorbs input tokens; the small model absorbs easy turns; the queue absorbs bursts; the Redis buffer absorbs flaky networks.
    • First metrics: time to first token by tier, queue wait by tier, share of traffic on the small model.
    • If time: prefix-cache hit rate and cache-aware routing, inference internals, multi-step tools, deletion of a conversation and its derived memory.

    Equally defensible variants

    WebSockets instead of SSE. Always finishing generation in the background instead of cancelling. DynamoDB or sharded Postgres instead of Cassandra. A key-value store for memory facts. Small-model escalation instead of an upfront classifier. A queue-position screen instead of a model fallback as the degradation behavior. What matters is that you can argue the tradeoff.

    More in FDE Interview Prep: AI Coach Edition

    • FDE Mock · Palantir-style Room
    • FDE Mock · AI Lab Room
    • FDE Mock · Startup Room
    • FDE Mock · Cloud Room
    • SD Coach · The System Design Interview
    • SD Coach · Agentic AI
    • Interview Coach: STAR Method
    • FDE Drill · Clarify
    • FDE Drill · Decompose
    • FDE Drill · MVP Cut
    • FDE Drill · Explain It to an Exec
    • FDE Drill · Debug First Moves
    View the full course (43 sessions) →