Loading...
One GPU can run up to 100 inputs in a single pass. Requests arrive continuously, and every caller waits on an open connection for its own answer. Design the system that batches them.
A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.
Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):
Build incrementally while you talk. Trace one prompt end to end through your diagram: from the caller, into a batch, onto a GPU, and back to the same open connection. Argue the tradeoff, not "industry standard."
Read this before you go in. These are the decisions, and why they matter.
Open by asking whether you can treat the model as a black box: it takes up to N inputs and returns N outputs in roughly fixed time. Then the interview is about the API, the queue, the batching policy, the worker pool, and failures. Also ask two things. Is the caller waiting, or is this an offline scoring job? And which wins, latency or utilization, with what ceiling on the other? Offline means huge batches and throughput only. Synchronous means a latency budget drives every number.
Sending each request straight to the GPU gives a batch of 1. At 500 ms per pass that is 2 requests per second per GPU instead of 200, so you would need 100 times the GPUs. Waiting for a full batch fails the other way: at 50 requests per second overnight, a request waits 2 seconds just to fill it. The sound default is flush when the queue holds N, or when the oldest request has waited W, whichever comes first. At peak the size rule fires; off-peak the timer does. Keep filling a formed batch while you wait for a free GPU. Adaptive W is equally defensible if you can say why.
Treat the wait timer as a tunable dial, not a fixed number, and tune it against the p99 target. It starts when the first request in a batch arrives. Work backwards from the SLO. Worst case, a request waits W, then waits for a busy GPU to finish its current batch (T), then runs its own batch (T), plus network overhead. With a 1.5 s p99, T = 500 ms and about 100 ms of overhead, W can be at most 400 ms. Pick 200 ms and keep slack for a retry. With a faster model the same math lands far lower; 20 ms is a common example. The method is what gets graded. Adaptive waits are a good refinement: shrink the wait when the queue is deep, stretch it when traffic is thin so fewer GPUs can serve it.
The caller's connection lives on one API server. The answer is produced on a GPU worker that knows nothing about HTTP. The naive answers fail: the worker cannot call the client back, and polling a result store adds latency and a read storm. Broadcasting every result to every server works for a modest fleet, but each server throws away most of what it receives. The default: the API server registers a future under a request_id, enqueues the request with reply_to = its own server id, and the worker returns results tagged with both. A reply channel per API server delivers each result to the one server holding the connection, which resolves the future. Co-locating the batcher inside each GPU host (in-process futures, no reply hop) is equally defensible if you can say why; its cost is that each host sees only a slice of traffic, so batches fill slower.
With a pool of GPUs, assigning batches is a scheduling problem. Round robin fails because batches take different amounts of time and workers drift out of sync: some GPUs pile up work while others sit idle. Put a Scheduler between the batcher and the workers. Workers heartbeat busy or free, queue depth, and time remaining; the scheduler sends each batch to the worker with the least outstanding work. A worker that stops heartbeating is treated as dead and its batch is reassigned. Pull-based workers that ask for a batch when free are equally defensible if you can say why.
Past capacity, an unbounded queue makes every request late. The GPUs stay busy computing answers nobody is waiting for; goodput goes to zero. Bound the queue by what can still meet the deadline, reject the rest in milliseconds with a retryable 429, carry an absolute deadline on every request, and drop expired requests before they reach a GPU. Add per-tenant concurrency caps and weighted selection when forming batches, so one caller with ten thousand requests cannot fill every batch.
A worker can die mid-batch, and one malformed input can fail the whole batch. For a crash you must choose: re-enqueue every request (they must be re-runnable) or fail them individually. Say which and why. Re-enqueue works here because inference is read-only. Give every dispatch a lease with heartbeats; when the lease expires, put the batch back at the head of the queue. When a batch fails as a whole, bisect it: split and retry the halves until the one bad input is isolated, so 99 callers succeed and one gets a 400.
One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.
Rough numbers are fine; the method is what gets graded.
The graph has two halves. A synchronous front door accepts calls and holds connections open. An asynchronous core queues requests, forms batches, runs them on GPUs, and routes each result back. The bridge between the halves is a request_id plus a reply_to address. The GPU tier is the only expensive, scarce resource; everything else exists to keep it fed with full batches without breaking the latency budget.
Web Client
-> API Gateway (auth, per-tenant Rate Limiter)
-> Load Balancer
-> Service: API Server [distributed] holds connection + pending map
-> Queue: Request Queue (bounded, per model)
-> Service: Batcher (flush on N=100 or oldest waited W)
-> Scheduler (least outstanding work, leases, heartbeats)
-> Worker: GPU Worker [distributed] -> ML Model
-> Pub/Sub Topic: Reply Channel (keyed by API server id)
-> Service: API Server (resolve future, respond)
Redis: lease table + worker registry (read/written by Scheduler)
Components:
Worker dies mid-batch: heartbeats stop, the lease expires at 750 ms, and b9 goes back to the head of the queue. Expired requests get 504; the rest re-run. Safe because inference is read-only. The retry lands at about 1.55 s, just past the SLO, so a crash is a p99.9 event. One poison input: the whole batch errors. Bisect: split, retry the halves, recurse on the failing half. The bad input is isolated in about log2(57) ≈ 6 rounds; 56 succeed and one gets 400. Caller hangs up: a7 drops the pending entry. If r42 was already dispatched, its result arrives and is discarded. API server dies: its connections and pending map vanish. Clients see a reset and retry with an idempotency key. Results addressed to a7 are dropped; nothing durable was lost.
PendingRequest(request_id PK, reply_to, model, tenant, priority, deadline, input, attempts) -- Request Queue
PendingMap(request_id -> future) -- API Server memory
BatchLease(batch_id PK, worker_id, request_ids[], leased_at, lease_expires) -- Redis
Worker(worker_id PK, model, state: loading|ready|busy|draining, last_heartbeat) -- Redis
Result(request_id, reply_to, status, output) -- Reply Channel message
Co-located batching (queue and batcher inside each GPU host, as Triton and vLLM do) instead of a shared queue: fewer hops, but each host sees a slice of traffic. Direct RPC to the reply_to address instead of a pub/sub channel. Pull-based workers instead of a scheduler pushing. A shared result topic that every API server filters, instead of a channel per server, for a modest fleet. Redis Streams or an in-memory queue with a standby instead of a heavier broker. What matters is that you can argue the tradeoff.