Loading...
A 20-minute mock system design interview. You drive the pace. Jump to the board when you are ready; skip capacity math if you want. The interviewer grades your thinking, not a template match.
The trap is "photos in S3, rows in MySQL, done." That sentence says nothing about who serves reads while uploads hog connections, how 3.7 TB of metadata is partitioned, or how "newest photos from the 500 people I follow" becomes one fast query. The design's substance is exactly what the quip skips.
Two shared surfaces (only one is on screen at a time; use the panel arrow to switch):
Build incrementally while you talk. Trace one upload end to end through your diagram, then one feed load. Argue the tradeoff, not "industry standard."
Read this before you go in. These are the decisions, and why they matter.
Uploads are large, slow, disk-bound writes. Views are small, cache-served reads. Put both on one fleet and the slow path eats the fast one. A web server holds about 500 concurrent connections. An upload holds one for seconds; a cached read holds one for milliseconds. A modest number of uploaders occupies every slot and reads starve.
The write rate is only ~23 photos/sec, so this is a connection-occupancy problem, not a throughput problem. Say that distinction out loud. The fix is dedicated upload servers and dedicated read servers behind their own load balancers, scaled and tuned independently: long timeouts and large body limits on the write side, keep-alive and cache clients on the read side.
Pre-signed direct-to-blob uploads that shrink the write fleet to a metadata-only service are equally defensible if you can say why.
Ten years of photo bytes is ~1,425 TB. Ten years of metadata is ~3.7 TB. A 400× gap means two different stores: bytes in a Blob Store (HDFS or S3) with multiple copies on different servers; rows in a sharded, replicated metadata store. Blobs inside the database make the metadata tier 400× larger than it needs to be and ruin every index.
The second surprise is that the follow graph (1.82 TB) is almost as large as the photo table (1.88 TB). At social scale, relationships weigh as much as content, and feed generation reads the follow graph for every user.
PhotoID % 10. A user's photos scatter evenly. No user can make a shard hot or large, and a shard outage costs everyone about a tenth rather than someone everything. "All of a user's photos" becomes a scatter query. That is acceptable because it is not the hot path; the feed is.The catch: you need the PhotoID before you know the shard, so per-shard auto-increment no longer works. That is the chicken-and-egg problem the next decision solves.
One dedicated key-generation database with a single 64-bit auto-increment column works, and it is a single point of failure. If it dies, nobody can upload anywhere.
Run two. One issues odd IDs (auto-increment-increment = 2, offset = 1), the other even (offset = 2), behind a load balancer. They drift apart harmlessly; uniqueness only requires that they never issue the same number.
Then put time in the ID. The feed must find the newest photos on every shard. If the PhotoID starts with creation time, the primary index on PhotoID is a time index, and "newest N" on each shard is a short index scan.
PhotoID = [ epoch seconds : 31 bits ] [ per-second sequence : 9 bits ]
=> 40 bits = 5 bytes
31 bits covers 50 years (1.6B seconds against 2.1B). 9 bits gives 512/sec, which is 22× the expected 23/sec and lands the ID on a byte boundary. The sequence resets every second; the epoch prefix keeps IDs unique across seconds. Shard is still PhotoID % 10, and consecutive sequence numbers walk through the shards in turn, so load spreads by construction.
Snowflake (41 bits of milliseconds, 10 bits of machine identity, 12 bits of sequence, minted on each app server) is equally defensible. Instagram's own scheme generates a 64-bit ID inside each Postgres shard. The one idea that matters: time in the high bits so the ID sorts by creation order.
These are not the same guarantee. A photo appearing in a feed a few seconds late is fine; asynchronous builders and caches without invalidation storms are how you buy availability. A lost photo is not fine: acknowledge an upload only after multiple copies exist on different storage servers, replicate every metadata shard, and keep a hot standby for any single-instance service. Durability is built at write time, not restored afterwards.
Changing the modulus from 10 to 20 moves almost every row. Instead, create many logical partitions up front and map several onto each physical server through a config map. When a server fills, copy some partitions to a new server, replay writes until it catches up, flip the map, drain the old copy. No PhotoID changes and no application change. Consistent hashing is the alternative when you cannot pre-decide the partition count.
One complete design you can study, then rebuild in your own words. Do not recite it. In the interview, draw your graph and defend the choices.
Rough numbers are fine; the method is what gets graded.
How the graph works: two entry paths that never share a server. Uploads go through the upload load balancer to the write fleet, which mints a PhotoID, writes bytes to the blob store, waits for replicas, then inserts the row into shard PhotoID % 10. Feed loads go through the read load balancer to the read fleet, which answers from Memcached and the CDN and only falls through to the shards and blob store on a miss. Feed builders run off to the side, continuously, and never sit on either request path.
Mobile Client, Web Client
→ Load Balancer (upload) → Service: Upload Service (mark distributed)
→ Load Balancer (key-gen) → MySQL (key-gen, odd) | MySQL (key-gen, even)
→ Blob Store (photo bytes, multiple copies)
→ MySQL (metadata shards, PhotoID % 10, mark distributed)
→ Load Balancer (read) → Service: Read Service (mark distributed)
→ Memcached (hot Photo rows, UserNewsFeed rows)
→ MySQL (metadata shards)
→ CDN → Blob Store (miss only)
Worker: Feed Builder (mark distributed)
→ MySQL (metadata shards): newest PhotoIDs since last build
→ UserNewsFeed row (metadata store, cached in Memcached)
Components:
PhotoID % 10, each replicated. Holds User, Photo, UserFollow, and UserNewsFeed. The Photo primary index on PhotoID is a time index because the ID is epoch-prefixed. A distributed KV (Cassandra, DynamoDB) with PhotoID as partition key is the book's other recommendation; it keeps replicas by default and defers deletes for undelete.Ana (UserID 4471) follows Bo (8812) and Cy (9903). Bo uploads three photos at 10:00:01, 10:00:02, 10:00:02; Cy uploads one at 10:00:03.
POST /v1/photos with the bytes and title. The upload Load Balancer routes it to an Upload Service instance. Read servers never see it.PhotoID = (epoch seconds << 9) | 43, i.e. 1789580402·043.PhotoID % 10 = 3. It inserts the Photo row (PhotoPath, lat/long, CreationDate) into shard 3; the shard's replicas copy it.Then Ana opens the app:
GET /v1/feed hits the read Load Balancer and a Read Service instance.PhotoID % 10, fast because the primary index is already in time order.Celebrity upload: an account with 50M followers must not trigger 50M feed-row pushes. Hybrid delivery: their followers pull that account's recent PhotoIDs and merge at read time; small accounts are pushed into UserNewsFeed rows as above.
Storage server dies holding 100K photos: nothing is lost, because a single copy never existed. The upload was acknowledged only after multiple copies on different servers; reads fall to another copy. Metadata shards have replicas; single-instance services have a hot standby that takes no traffic until failover.
Shard at 95% disk: logical partitions over physical servers. Copy some partitions to a new server, replay writes until caught up, flip the config map, drain the old copy. Zero downtime, no ID change.
User deletes a photo: mark it deleted in metadata immediately so the Read Service and Feed Builders skip it. After the undelete window, purge bytes from the Blob Store and CDN and scrub it from UserNewsFeed rows. Takedowns use the same path with a zero window. Edits create a new PhotoID rather than mutating bytes, which keeps long CDN TTLs safe.
User (68 B): UserID, Name, Email, DateOfBirth, CreationDate, LastLogin
Photo (284 B): PhotoID, UserID, PhotoPath, PhotoLat, PhotoLong, UserLat, UserLong, CreationDate
UserFollow (8 B): UserID, FolloweeID
UserNewsFeed: UserID, PhotoIDs[], LastGeneratedAt — one row per user
KeyGen: ID auto-increment — two MySQL, odd / even
Photo rows sharded by PhotoID % 10; index on (PhotoID, CreationDate) so "newest first" is an index scan on every shard
UserFollow keyed by UserID so a follower's list is one read; UserNewsFeed is one row per user
Every shard replicated; a KV variant keeps replicas by default and defers deletes for undelete
Bytes never live in a row; PhotoPath points at the Blob Store and is served through the CDN
The 4-byte PhotoID in the row sizing is smaller than the 5-byte ID you design; noticing that mismatch out loud earns a point
Cassandra or DynamoDB with PhotoID as the partition key instead of MySQL shards. Snowflake IDs minted on each app server instead of the key-gen pair. Pre-signed direct-to-S3 uploads that leave the write fleet metadata-only. Redis for UserNewsFeed rows instead of a table behind Memcached. Consistent hashing instead of a logical-partition map. What matters is that you can argue the tradeoff.