Loading...
A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.
Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):
Build incrementally while you talk. Trace one message end to end through your diagram, from Enter to the last streamed word. Argue the tradeoff, not "industry standard."
Read this before you go in. These are the decisions, and why they matter.
A "thanks" and a pasted 50-page contract are both one request. They differ in GPU cost by three orders of magnitude, so any limit or capacity plan built on requests per second is wrong. Split tokens into input and output. Output is generated one token at a time and sets how long an answer takes. Input is the whole prompt, and because every turn re-sends the conversation, it runs about six times the output. That ratio is the most important number in the interview: it says the money is in the prompt.
A 500-token answer takes around ten seconds. Delivered at once, it feels broken. With the first token in half a second, it feels instant. So stream. Server-Sent Events is the natural fit: the server pushes, the client only listens, it runs over plain HTTP and reconnects on its own. WebSockets are equally defensible if you want two-way traffic such as voice or interruptions, but they carry more operational weight. Three things follow. Time to first token is the latency that matters, because total time is fixed by the model. Requests live for seconds to minutes, so open connections, not request rate, measure the serving tier. And when the user closes the tab or presses Stop, the cancel must travel all the way to the running model, or you pay for tokens nobody reads. Save whatever was generated so a reconnect shows a partial answer, not an empty turn. A short grace period before cancelling lets a phone that switches networks catch up.
The model remembers nothing, and the context window is finite. Every turn you rebuild the prompt from stored history. A sliding window of the last N turns is simple but quietly drops the rule the user set at the start. A rolling summary of older turns plus recent turns verbatim is the usual workhorse; it costs an extra model call when the summary refreshes. Retrieving relevant old turns by embedding helps in very long threads. Whatever you choose, always keep the system prompt and the user's first message, which usually defines the task.
Between turn 7 and turn 8, most of the prompt is identical. Inference providers can keep the computed state for a repeated prefix and skip recomputing it, usually at a fraction of the price. That only works if stable content comes first and variable content last: system prompt, memory facts, first message, summary, older turns, then the new message. A timestamp near the top, or notes interleaved between history blocks, breaks the match for everything after it. With input six times output, this ordering is the biggest cost lever you have. Refreshing the summary rewrites the top of the prompt, so refresh it rarely.
Most turns are easy: "make it shorter", "thanks", a quick fact. A smaller model answers them faster and far more cheaply. Put a lightweight classifier at the inference gateway that sends hard turns to the large model, or let the small model escalate when unsure. The risk is misrouting a hard question; keep thresholds conservative and let paid users pin the large model. Sending every message to the biggest model is the naive answer.
GPU capacity is fixed on any timescale shorter than buying hardware, so demand above supply is normal. Decide who gets served. A higher message cap for paid users does nothing when everyone waits in one line; give paid traffic priority in the queue instead, with a minimum share so free users slow down rather than starve. Pick one visible degradation for the product you scoped: show a queue position, fall back to the smaller model with a label, or reject with a retry time. Enforce per-user and per-tier budgets in tokens at the inference gateway, before a request occupies a GPU. Cache exact repeats; cache near-duplicates only with a conservative similarity threshold and only for prompts that do not depend on history, because a wrong cached answer is worse than recomputing.
In-conversation context is the thread's own history. Cross-conversation memory is a small set of durable facts about the user (preferences, ongoing projects, stated constraints), extracted by a background job after a conversation and pulled into future prompts. Keep it small, because every fact costs tokens on every future turn. Make it visible and editable, because a wrong fact silently degrades everything afterward. Scope it: users must be able to turn it off and delete it. Treating "all history" as memory is a common mistake.
Classify input before any GPU time is spent. Output classification clashes with streaming: tokens are on screen before a problem is detected. Either buffer a small window before display, or classify incrementally and stop the stream. Most abuse is automated scraping through free accounts; it shows up as high rates, regular timing and no reading pauses. Per-account rate limits combined with token budgets cap what any one account is worth.
One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.
Rough numbers are fine; the method is what gets graded.
Two halves joined by an inference gateway. The conversation half is ordinary web-scale plumbing: gateway, chat service, stores, filters, async workers. The inference half is a scarce GPU fleet behind a gateway that picks a model tier and enforces budgets, and a queue that admits paid traffic first. The hot path is client → API gateway → chat service → inference gateway → queue → model → tokens streamed back. Storage writes, memory extraction and billing stay off it.
Web Client / Mobile Client
-> CDN -> API Gateway -> Chat Service (Service)
-> Cassandra (conversations, messages)
-> Postgres (memory facts)
-> ML Model (safety filters)
-> Inference Gateway (Service)
-> Rate Limiter (token budgets, Redis)
-> Cache (response cache)
-> Queue (tiered: paid > free)
-> ML Model (small model)
-> ML Model (large model)
<- token stream from the models
-> Redis (stream buffer for reconnects)
-> Kafka -> Worker (title, summary, memory extraction) -> Postgres
<- SSE stream from Chat Service via API Gateway; cancel flows back
Components:
Tab closed mid-answer: after a short grace period with no reconnect, the Chat Service cancels at the gateway, the GPU stops, and the partial answer is saved. Phone switches networks: the client reconnects with its last event ID and catches up from the Redis buffer. Launch-day spike: paid traffic keeps priority; free traffic gets the stated degradation (for example the small model, labelled) or a retry time.
User(user_id, plan, created_at) -- Postgres
MemoryFact(user_id, fact_id, text, source_conversation_id, created_at) -- Postgres
Conversation(user_id, updated_at, conversation_id, title,
current_leaf_id, summary) -- Cassandra
Message(conversation_id, message_id, parent_id, role, content,
status, model, input_tokens, cached_tokens, output_tokens) -- Cassandra
StreamBuffer(message_id -> [event_id, text], TTL 10m) -- Redis
WebSockets instead of SSE. Always finishing generation in the background instead of cancelling. DynamoDB or sharded Postgres instead of Cassandra. A key-value store for memory facts. Small-model escalation instead of an upfront classifier. A queue-position screen instead of a model fallback as the degradation behavior. What matters is that you can argue the tradeoff.