TOUGH TONGUE AI
  • Home
    LogoTOUGH TONGUE AI

    RESOURCES

    Loading...

    HomeLibrarySD Mock · Design an LLM Query Batching System

    SD Mock · Design an LLM Query Batching System

    SD Mock · Design an LLM Query Batching System

    Overview

    Design the batching layer behind an LLM API on a live architecture board: queue requests, fill GPU batches, and route every answer back to the caller still waiting, while an expert interviewer watches every move.

    Your Instructions

    Design an LLM Query Batching System

    One GPU can run up to 100 inputs in a single pass. Requests arrive continuously, and every caller waits on an open connection for its own answer. Design the system that batches them.

    A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.

    Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):

    • Notepad. Shared notes. Requirements, capacity math, the request record, and end-of-session feedback live here. You can type in it too.
    • Architecture board. Your design canvas. Drag typed components, wire them, label edges, mark scaled tiers as distributed. If it is not on the board, it is not in your design.

    Build incrementally while you talk. Trace one prompt end to end through your diagram: from the caller, into a batch, onto a GPU, and back to the same open connection. Argue the tradeoff, not "industry standard."

    Design choices interviewers probe

    Read this before you go in. These are the decisions, and why they matter.

    Scope: treat the model as a black box

    Open by asking whether you can treat the model as a black box: it takes up to N inputs and returns N outputs in roughly fixed time. Then the interview is about the API, the queue, the batching policy, the worker pool, and failures. Also ask two things. Is the caller waiting, or is this an offline scoring job? And which wins, latency or utilization, with what ceiling on the other? Offline means huge batches and throughput only. Synchronous means a latency budget drives every number.

    When to flush a batch

    Sending each request straight to the GPU gives a batch of 1. At 500 ms per pass that is 2 requests per second per GPU instead of 200, so you would need 100 times the GPUs. Waiting for a full batch fails the other way: at 50 requests per second overnight, a request waits 2 seconds just to fill it. The sound default is flush when the queue holds N, or when the oldest request has waited W, whichever comes first. At peak the size rule fires; off-peak the timer does. Keep filling a formed batch while you wait for a free GPU. Adaptive W is equally defensible if you can say why.

    The timer is a latency knob: tune it to the budget

    Treat the wait timer as a tunable dial, not a fixed number, and tune it against the p99 target. It starts when the first request in a batch arrives. Work backwards from the SLO. Worst case, a request waits W, then waits for a busy GPU to finish its current batch (T), then runs its own batch (T), plus network overhead. With a 1.5 s p99, T = 500 ms and about 100 ms of overhead, W can be at most 400 ms. Pick 200 ms and keep slack for a retry. With a faster model the same math lands far lower; 20 ms is a common example. The method is what gets graded. Adaptive waits are a good refinement: shrink the wait when the queue is deep, stretch it when traffic is thin so fewer GPUs can serve it.

    The async-to-sync bridge

    The caller's connection lives on one API server. The answer is produced on a GPU worker that knows nothing about HTTP. The naive answers fail: the worker cannot call the client back, and polling a result store adds latency and a read storm. Broadcasting every result to every server works for a modest fleet, but each server throws away most of what it receives. The default: the API server registers a future under a request_id, enqueues the request with reply_to = its own server id, and the worker returns results tagged with both. A reply channel per API server delivers each result to the one server holding the connection, which resolves the future. Co-locating the batcher inside each GPU host (in-process futures, no reply hop) is equally defensible if you can say why; its cost is that each host sees only a slice of traffic, so batches fill slower.

    Choosing a GPU: capacity, not round robin

    With a pool of GPUs, assigning batches is a scheduling problem. Round robin fails because batches take different amounts of time and workers drift out of sync: some GPUs pile up work while others sit idle. Put a Scheduler between the batcher and the workers. Workers heartbeat busy or free, queue depth, and time remaining; the scheduler sends each batch to the worker with the least outstanding work. A worker that stops heartbeating is treated as dead and its batch is reassigned. Pull-based workers that ask for a batch when free are equally defensible if you can say why.

    Overload: bound the queue, reject fast

    Past capacity, an unbounded queue makes every request late. The GPUs stay busy computing answers nobody is waiting for; goodput goes to zero. Bound the queue by what can still meet the deadline, reject the rest in milliseconds with a retryable 429, carry an absolute deadline on every request, and drop expired requests before they reach a GPU. Add per-tenant concurrency caps and weighted selection when forming batches, so one caller with ten thousand requests cannot fill every batch.

    Failures touch 100 callers at once

    A worker can die mid-batch, and one malformed input can fail the whole batch. For a crash you must choose: re-enqueue every request (they must be re-runnable) or fail them individually. Say which and why. Re-enqueue works here because inference is read-only. Give every dispatch a lease with heartbeats; when the lease expires, put the batch back at the head of the queue. When a batch fails as a whole, bisect it: split and retry the halves until the one bad input is isolated, so 99 callers succeed and one gets a 400.

    Reference solution

    One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.

    Requirements (lock these or take the defaults)

    • Accept one inference request over a synchronous API and return its output on the same call.
    • Batch up to N = 100 inputs per GPU pass; the model is a black box that takes about T = 500 ms per batch regardless of fill.
    • Return each output to exactly the caller that sent it, including when part of a batch fails.
    • Peak 1,000 requests/s, overnight about 50/s. Inputs and outputs about 2 KB each.
    • Latency wins with a ceiling: p99 end to end at or under 1.5 s. Within that, maximize GPU utilization.
    • Out of scope: model internals, training, token streaming (mention as an extension).

    Capacity

    • Per GPU: 100 inputs ÷ 0.5 s = 200 req/s with full batches, versus 2 req/s unbatched (a 100x gap).
    • Fleet: 1,000 ÷ 200 = 5 GPUs at 100%. Run at 70%: 1,000 ÷ 140 ≈ 7.1, so 8 GPUs, plus one spare = 9. Pool capacity with 8 serving: 8 × 200 = 1,600 req/s.
    • Fill time: at peak a batch of 100 fills in 100 ÷ 1,000 = 100 ms, so the size rule fires. Overnight it takes 100 ÷ 50 = 2 s, so the 200 ms timer fires and batches hold about 10.
    • Latency budget: 50 ms in + W + up to 500 ms waiting for a GPU + 500 ms own batch + 50 ms out = 1,100 ms + W. So W ≤ 400 ms; pick 200 ms.
    • Cost: about $2 per GPU-hour at about 1,000 tokens/s is 3.6M tokens/hour, a floor near $0.55 per million tokens. At 30% utilization the real cost roughly triples. That is why batching is an economic decision.
    • Queue bound: 1,600 req/s drain × 0.9 s usable wait ≈ 1,400 requests. In flight at peak (Little's law): 1,000 × about 0.6 s ≈ 600 open connections. Bandwidth is about 4 MB/s, irrelevant.

    Rough numbers are fine; the method is what gets graded.

    Components to put on the board

    The graph has two halves. A synchronous front door accepts calls and holds connections open. An asynchronous core queues requests, forms batches, runs them on GPUs, and routes each result back. The bridge between the halves is a request_id plus a reply_to address. The GPU tier is the only expensive, scarce resource; everything else exists to keep it fed with full batches without breaking the latency budget.

    Web Client
      -> API Gateway            (auth, per-tenant Rate Limiter)
      -> Load Balancer
      -> Service: API Server    [distributed]  holds connection + pending map
           -> Queue: Request Queue   (bounded, per model)
                -> Service: Batcher  (flush on N=100 or oldest waited W)
                     -> Scheduler        (least outstanding work, leases, heartbeats)
                          -> Worker: GPU Worker [distributed] -> ML Model
                          -> Pub/Sub Topic: Reply Channel (keyed by API server id)
                               -> Service: API Server  (resolve future, respond)
    Redis: lease table + worker registry (read/written by Scheduler)
    

    Components:

    • Web Client. Calls the inference endpoint and blocks until the answer or its timeout. It never learns there was a batch.
    • API Gateway. Authenticates, validates input size, and enforces per-tenant limits through a Rate Limiter. Stops abuse and oversized prompts before they cost GPU time.
    • Load Balancer. Spreads connections across API servers. Round robin works because API servers hold no batching state.
    • Service: API Server. Assigns a request_id, computes an absolute deadline, registers a future in an in-memory pending map, and enqueues. It parks the connection on the future without blocking a thread. Without it there is nowhere for the answer to land.
    • Queue: Request Queue. Bounded, one per model (and per priority tier). Rejects with a fast 429 when full. Without the bound, overload turns into congestion collapse.
    • Service: Batcher. Runs the flush rule, drops expired requests, and hands each formed batch to the Scheduler. This is the heart of the design.
    • Scheduler. Knows each worker's real capacity from heartbeats and sends each batch to the least-loaded one. Holds a lease on every dispatched batch and reassigns a batch whose worker goes silent. Without it, round robin piles work on busy GPUs.
    • Worker: GPU Worker + ML Model. Loads the model once, runs a batch in about T, returns one tagged result per input in input order, and heartbeats while busy.
    • Pub/Sub Topic: Reply Channel. One channel per API server. Each result is published on its reply_to channel, so it reaches only the server holding that caller's connection.
    • Redis. Holds the lease table and worker registry so a standby scheduler can take over.

    Walk one prompt: request r42 from caller to GPU and back

    1. The client sends a prompt with a 1.5 s timeout. The API Gateway authenticates tenant B and checks its rate limit.
    2. The Load Balancer routes it to API Server a7, which assigns request_id r42, sets a deadline about 1.4 s out, and registers a future in its pending map.
    3. a7 enqueues {r42, reply_to: a7, deadline} on the Request Queue. If the queue is full, a7 returns 429 immediately.
    4. The Batcher sees 57 requests queued when the oldest reaches W = 200 ms. It drops anything past its deadline and forms batch b9.
    5. The Scheduler picks GPU Worker g4, the one with the least outstanding work, records a 750 ms lease for b9, and dispatches it.
    6. g4 runs the model in about 500 ms and returns 57 results, each tagged with request_id and reply_to.
    7. r42 is published on Reply Channel a7. a7 resolves pending[r42], deletes the entry, and writes 200 OK. Total about 800 ms.

    Worker dies mid-batch: heartbeats stop, the lease expires at 750 ms, and b9 goes back to the head of the queue. Expired requests get 504; the rest re-run. Safe because inference is read-only. The retry lands at about 1.55 s, just past the SLO, so a crash is a p99.9 event. One poison input: the whole batch errors. Bisect: split, retry the halves, recurse on the failing half. The bad input is isolated in about log2(57) ≈ 6 rounds; 56 succeed and one gets 400. Caller hangs up: a7 drops the pending entry. If r42 was already dispatched, its result arrives and is discarded. API server dies: its connections and pending map vanish. Clients see a reset and retry with an idempotency key. Results addressed to a7 are dropped; nothing durable was lost.

    Schema

    PendingRequest(request_id PK, reply_to, model, tenant, priority, deadline, input, attempts)   -- Request Queue
    PendingMap(request_id -> future)                                                              -- API Server memory
    BatchLease(batch_id PK, worker_id, request_ids[], leased_at, lease_expires)                    -- Redis
    Worker(worker_id PK, model, state: loading|ready|busy|draining, last_heartbeat)               -- Redis
    Result(request_id, reply_to, status, output)                                                  -- Reply Channel message
    
    • Everything is in-flight state with a life of about 1.5 s. Nothing needs durable storage; loss is recovered by client retry.
    • One queue per batch key (model + version + priority tier); workers are pinned to a model.
    • Redis replicated (primary plus replica) only so a standby scheduler can rebuild leases quickly.

    Scaling and wrap-up

    • API servers scale horizontally behind the Load Balancer; GPU workers scale by count.
    • The Batcher and Scheduler are one logical process each per model: leader plus warm standby.
    • Autoscale GPUs on the age of the oldest queued request (it maps to what callers feel), plus batch fill; not queue depth or raw GPU utilization. New GPUs take minutes, so keep the N+1 spare warm.
    • First metrics: oldest-request age, batch fill (average batch ÷ N), goodput (answers delivered within deadline).
    • If time: continuous batching when outputs vary in length, KV-cache memory as the real limit instead of a count of 100, prefix caching for a shared system prompt, token streaming, an offline batch tier at a discount.

    Equally defensible variants

    Co-located batching (queue and batcher inside each GPU host, as Triton and vLLM do) instead of a shared queue: fewer hops, but each host sees a slice of traffic. Direct RPC to the reply_to address instead of a pub/sub channel. Pull-based workers instead of a scheduler pushing. A shared result topic that every API server filters, instead of a channel per server, for a modest fleet. Redis Streams or an in-memory queue with a standby instead of a heavier broker. What matters is that you can argue the tradeoff.

    More in System Design Interview Prep: AI Coach Edition

    • SD Coach · Media Streaming
    • SD Mock · Design YouTube / Netflix
    • SD Mock · Design Instagram
    • SD Coach · Agentic AI
    • SD Coach · Agentic AI Architectures (Slides)
    • SD Mock · Design ChatGPT
    • SD Mock · Design Claude Code
    • SD Mock · Design an AI Customer Support System
    • SD Mock · Design a Real-Time Voice AI
    • SD Mock · Design a File Uploader for an AI Chat App
    • SD Coach · Reliability and Security
    • SD Mock · Design It Live
    View the full course (43 sessions) →