Foundations of System Design

One box, three million visitors

Here is a design that has launched thousands of products:

 browser ──► [ one server: web app + Postgres + images on its disk ]

It is a good design. A recipe site doing 30 page views per second at its busiest barely wakes this machine up. Adding load balancers and caches now would cost money and attention and buy no capacity you need. (What it does need from day one is backups; one box is also one failure, which comes up later.)

Then a TV cooking show mentions one of your recipes. Three million people open that page within ten minutes. That is 3,000,000 ÷ 600 s = 5,000 page views per second, about 170 times your normal peak.

To see what happens, you need three facts about a page view and three about the machine. They are assumptions. In an interview you state them out loud, and the interviewer corrects any they dislike.

Predict first: which resource runs out first, and by how much? Most people guess the database.

The move: stop repeating the same work

The reflex is to buy capacity: a bigger box, then eight boxes. But look at what the server is actually doing. It builds the same page three million times, runs the same four queries three million times and sends the same 1.9 MB of images, CSS and JS three million times. After the first visitor, nobody asked for anything new.

So stop repeating the work. Put a CDN (a content delivery network: caching servers in many cities) in front of the site. Let it keep the images, CSS and JS, and let it keep the HTML of public pages for 60 seconds.

• The static files, 95% of the bytes, now come from edge servers near each reader.
• Each edge location asks your server for the recipe about once a minute. Even with 500 edge locations, that is about 8 requests and 33 database queries per second.

One move fixed the network, the CPU and the database, because all three loads came from the same repeated work.

What it costs. If you fix a typo in the recipe, readers can see the old version for up to a minute unless you purge the CDN. Anything personal ("Hi, Sam") cannot be part of the shared cached HTML. And the CDN bills per gigabyte. For a recipe, a minute of staleness is fine. That pair, the move and the price you accept, is the basic unit of system design.

💡 The whole course in one sentence: find the resource that runs out first, pick the cheapest move that relieves it, and say out loud what that move costs.

Pressure, move, price: the map of this course

Every system in this roadmap is a mix of a few pressures. Each pressure has a standard move, and each move has a price. You will meet every row again in depth.

The roadmap itself runs like this: foundations, then the building blocks (databases, caches, queues), then the interview method (requirements, then design steps), then complete designs (a URL shortener and rate limiter, a news feed, video streaming, chat), and finally advanced topics and mock interviews.

This lesson teaches the three things everything else builds on: what the interview grades, the life of a request (the boxes in that table, drawn as a path), and whiteboard arithmetic. Then it uses them to grow a design and to plan for failure. It assumes you have built a web app and know what an HTTP request and a SQL query are.

Your turn: the page that cannot be shared

Same spike, same numbers, but the popular page is the logged-in "My cookbook" page: every reader sees their own saved recipes. What does the CDN still do for you, and what do you do about the rest?

What the interviewer is actually grading

"Design a food delivery app" in 45 minutes has no answer key. Two strong engineers can draw different diagrams and both pass. What the interviewer grades is judgment: whether your choices follow from the requirements, and whether you know what each one costs.

Not a coding interview

Five signals, and what they sound like

One sentence shape is worth rehearsing until it comes out automatically:

"I'll use X because [requirement]. It costs Y. If [requirement] changes to Z, I'd switch to W."

Interviewers push on exactly the last part. Expect follow-ups such as "what if traffic is 10× higher?", "what happens when that node dies?" and "how would you know it's broken?" If you have already named the price, those questions are easy.

Most trade-offs pull between three goals that Martin Kleppmann names in Designing Data-Intensive Applications: scalability (coping with growth), reliability (working correctly when parts fail) and maintainability (people can run and change it). A design that chases the first two by adding ten services usually loses the third.

Requirements first, because they change everything

"Design a messaging app" could mean a consumer messenger for hundreds of millions of people, whose phones go offline for days and whose messages are end-to-end encrypted. It could also mean a notification feed for the 1,000 employees of one company. The first needs a whole lesson (Chat Application (WhatsApp)); the second is one server and a table. You cannot tell which until you ask.

Requirements come in two kinds:

• Functional: what the system does. "Users send messages to a group; messages sent while you are offline arrive when you reconnect."
• Non-functional: how well it does it: scale, latency, availability, durability, consistency, cost. "99% of messages arrive within a second; an accepted message is never lost."

The non-functional ones decide the architecture, so those are the ones to turn into numbers. This is the Scope phase of the interview (below). Requirements Gathering shows how to ask for them quickly.

Predict first: the prompt is "Design a leaderboard for a mobile game." Two candidates open like this.

• A: "I'll keep scores in Redis sorted sets, stream score events through Kafka and store history in Cassandra."
• B: "A few questions first. One global ranking, or rankings among friends? How many players, and how many score updates per second at peak? Must a player see their new rank the moment a match ends, or is a minute later fine?"

Which opening is stronger, and why can A's design be good and still score badly?

The shape of the session

This course uses one interview framework throughout, in four phases: Scope → Sketch → Deep dive → Wrap-up.

Everything in this lesson has a slot in it. Requirements and estimates belong to Scope, and only the numbers that decide something. The life of a request is what you draw and trace in the Sketch. Failure thinking fills the Deep dive and comes back in the Wrap-up. A longer slot buys depth, a second deep-dive topic, not a longer requirements phase. Interview Framework & Strategy teaches the framework, its checkpoints and what to do when you fall behind; Design Process Steps covers the Sketch and the Deep dive.

The life of a request

Every design in this course is a variation on one path. Learn it once and you have a map: when a prompt mentions a pressure, you will know which box it lands on.

 client (browser or app)
   │  1. DNS: shop.example.com → an address of the CDN
   ▼
 CDN edge ─────────── 2. static files and cacheable pages are answered here
   │  3. everything else is forwarded to your origin
   ▼
 load balancer ────── 4. picks a healthy app server
   │
   ▼
 app server (one of many identical, stateless copies)
   ├──► cache ─────────────── 5. look here first
   ├──► database ──────────── 6. on a miss: read here, then fill the cache
   │      └── replicas          (copies of the database that serve reads)
   ├──► object storage ────── big files: photos, PDFs, backups
   ├──► queue ──► workers ─── 7. slow work, finished after the response
   │
   ▼
 8. the response travels back along the same path

Without a CDN, step 1 returns the load balancer's address and steps 2 and 3 disappear.

Hop by hop

Deep dives: the load balancer, app tier and object storage in Core Building Blocks; caches and databases in Databases & Storage; queues in Messaging & Queues. Three details on the path trip people up.

Stateless means "any server can take any request". If server 2 keeps your cart in its own memory, the load balancer must keep sending you to server 2 (a sticky session), and a restart or crash of server 2 empties your cart. Keep that state in a shared store and the servers become interchangeable: you can add them, kill them and deploy them one at a time. Sticky sessions are not forbidden, but they have a price: uneven load, and lost state when that server goes.

Cache-aside: the app talks to both. On a read, the app asks the cache. On a miss, the app reads the database and writes the answer into the cache with a TTL (time to live). The cache never talks to the database in this pattern, which is why the diagram draws two separate arrows from the app server. Other strategies and invalidation are in Databases & Storage.

202 Accepted, not 200 OK. When work goes on a queue, the honest answer is "accepted, not done yet", and HTTP has a status code for exactly that: 202. The client then polls, gets notified, or never needs to know.

Two transport facts you will use constantly

• HTTP runs over TCP, and HTTPS adds TLS in between. HTTP/1.1 and HTTP/2 use TCP. HTTP/3 runs over QUIC, which runs over UDP and combines the transport and encryption handshakes. A new TCP connection costs one round trip before the request can leave, and a TLS 1.3 handshake costs one more (RFC 8446). QUIC does both in one round trip (RFC 9000).
• Plain UDP is for real time. Voice and video calls and multiplayer games use UDP, because a late packet is worthless and waiting for a retransmission causes a stall. On-demand video, such as a film or a recorded clip, is delivered over HTTP, so over TCP or QUIC.

Networking Basics covers DNS, HTTP, CDNs, TCP, UDP and QUIC properly.

Trace: one request from Frankfurt

Your shop's servers are in Virginia, and a customer in Frankfurt opens a product page for the first time. The assumptions:

• A Frankfurt–Virginia round trip takes about 90 ms. The straight-line distance, about 6,500 km, sets a floor near 65 ms in fiber, and real routes are longer.
• The DNS answer is not cached yet, and the lookup takes 30 ms.
• The server's own work (load balancer, app code, one cache hit) takes 8 ms.

Predict first: of the roughly 300 ms this request takes, how much is your server's work? Which single change would save the most?

Why it works. On an idle system, a request's latency is roughly round trips × distance, plus work. Physics sets the first term. Light in fiber covers about 200,000 km per second, so every 1,000 km between client and server adds about 10 ms to each round trip. You cannot speed up light. You can only move the answer closer, or cross the distance fewer times. Under heavy load a third term appears, waiting in queues, and it grows sharply as servers approach full utilization. Key Concepts & Terminology covers that and tail latency.

Whiteboard arithmetic

Estimates exist to choose boxes. "32 petabytes of photos" tells you the photos cannot live in the database. "125 Gbps at peak" tells you a CDN is not optional. An estimate that does not change a decision is time you could have spent designing, so after each number, say what it decides.

Numbers worth memorizing

These are the classic "latency numbers every programmer should know" from Peter Norvig and Jeff Dean, rounded (the list). Modern hardware beats some of them (NVMe SSDs and newer datacenter networks are faster), but you reason with the ladder of orders of magnitude, and that has not changed.

Three consequences you will use:

• Memory is about 100,000× faster than a disk seek (100 ns against 10 ms). A database whose hot data fits in RAM behaves very differently from one that has to seek.
• A cache hit on another machine costs a network round trip, about 0.5 ms, not 100 ns. Ten sequential cache calls in one request add about 5 ms, so batch them into one round trip.
• Crossing an ocean costs more than almost anything you do inside the datacenter. Count the long round trips first.

Units that bite

Formulas you will reuse

Worked example: a photo app

The interviewer offers these numbers. They are assumptions for the exercise, not any real company's figures.

On the whiteboard:

photo views   50M × 10 × 6  = 3B/day       ≈ 35k/s avg    ≈ 104k/s peak
feed calls    50M × 10      = 500M/day     ≈ 5.8k/s avg   ≈ 17k/s peak
uploads       5M/day                       ≈ 58/s avg     ≈ 174/s peak
photo bytes   5M × 3.5 MB   = 17.5 TB/day  ≈ 6.4 PB/year  ≈ 32 PB in 5 years
egress        35k/s × 150 KB × 8 bits      ≈ 42 Gbps avg  ≈ 125 Gbps peak
ingress       58/s × 3 MB × 8 bits         ≈ 1.4 Gbps avg (egress is ≈ 30× this)
metadata      5M × 1 KB     = 5 GB/day     ≈ 1.8 TB/year

Now the part that earns the points: what each number decides.

In the room you say only the numbers that decide something. Say it aloud in about this many words: "Roughly 17,000 feed requests a second at peak against under 200 uploads a second, so it's heavily read-dominated. Photos add 17 terabytes a day, over 30 petabytes in five years, so they go to object storage behind a CDN, which carries about 125 gigabits at peak. Metadata is small, under two terabytes a year, so I'd start with one database, a standby and a cache in front." Then move on. Precision beyond one or two significant figures rarely changes a decision.

The classic slips

Your turn: swap photos for video

Product wants 15-second video clips instead of photos, encoded at 2 Mbps. The numbers of uploads and views stay the same, and every clip in the feed autoplays in full. Which number changes most, and what do you do about it?

Earn every box

A single server is a real design

A school's homework portal serves 800 students. Say each makes 30 requests on a busy day: 24,000 requests, or 0.3 per second on average. Even if every student submits in the last hour before a deadline, that is about 7 per second. One server with a managed database and backups handles it with plenty of room to spare.

In an interview, drawing Kubernetes, Kafka and six microservices for this is not ambition. It is a design that costs more, fails in more ways and needs more people, to solve problems that do not exist. Every box should be the answer to a number.

"Monolith" means one deployable unit, not one machine: a monolith can run as 20 identical copies behind a load balancer. You split out a separate service when one part must scale, deploy or fail independently, or when teams keep blocking each other. You pay for it with network calls that can fail, harder debugging and more to operate. Shopify has described deliberately keeping a modular monolith, with enforced boundaries inside one codebase, instead of moving to microservices (Shopify Engineering, 2019). Advanced Topics & Final Prep covers microservices.

Traffic grows 10×: who breaks first?

Your shop API runs on four stateless app servers, one cache node, and one Postgres primary with a standby. Here is peak load today, from your dashboards:

A new market launches, and peak traffic will be 10× higher with the same mix of requests.

Predict first: which tier breaks first, and which is hardest to fix?

Why the stateful tier is the hard part

App servers are easy to multiply because they hold nothing: any copy can serve any request, so ten copies do ten times the work. A database is its state. Copies must agree on what is true, so either every write reaches every copy (replication), or each node owns a different slice of the data (partitioning). Both have a price: lag, coordination, and queries that span slices. That is why the database is usually the first hard bottleneck in a growing web service, and why so much of this course is about data.

🧠 Name the bottleneck before the move. "At 10× the primary is at 350%: 250 points from reads, 100 from writes. The cache fixes the reads. Writes need a bigger primary now and partitioning by customer later, because replicas don't take writes." That is the difference between adding boxes by instinct and adding them because a number demanded it.

Scale up, then out

Buying a bigger machine (vertical scaling) needs no code changes, and it is a perfectly good first move. Its limits are real, but they are mostly not about price. In the cloud, the hourly price within one instance family grows roughly in proportion to size, so one 64-vCPU machine costs about the same as eight 8-vCPU machines of that family. The limits are these:

• There is a biggest machine. Past it, you must split.
• One machine is one failure. A bigger box is no more redundant than a small one, so you need a standby anyway.
• Resizing usually means a restart or a failover.

Adding machines (horizontal scaling) has no hard ceiling for stateless servers, and it gives you redundancy along the way. For the database it means replicas and partitioning, with the prices above. So keep the app tier stateless from day one, so you can scale it out the moment a number demands it. Scale the database up while a bigger machine exists and the step is easy, and plan partitioning before you run out of bigger machines. Key Concepts & Terminology treats scalability properly.

Failure is the normal case

Netflix has described how its move to the cloud began: in August 2008 a major database corruption stopped it from shipping DVDs for three days (Netflix, 2016). It later built Chaos Monkey, which randomly terminates production instances so that engineers must build services that survive losing one. Its schedule runs on weekdays, between 9:00 and 15:00 by default, which keeps the breakage to hours when engineers are at work to respond (configuration docs).

At scale, something is always broken: a disk, a server, a network link, a dependency. The design question is never whether a box fails but what the user sees when it does.

Three questions for every box

Point at each box on your diagram and ask:

1. What if it dies? Does everything stop, or does the system degrade?
2. What if it is slow? Do callers wait forever, or time out and fall back?
3. What if a request to it is retried? Does the retry repeat a side effect, such as a second charge or a second email?

Failure modes along the path

The PostgreSQL documentation states the asynchronous-replication risk plainly: committed transactions that had not reached the standby are lost on failover (docs). Retries deserve their own warning: if three layers each try a call four times (once plus three retries), one user request can become 4 × 4 × 4 = 64 calls at the bottom layer. Retry at one layer, back off, and add random jitter so clients do not retry in lockstep.

The retry that charges twice

The third question catches the most expensive bugs. Your checkout calls a payment provider. The provider charges the card, but the reply is lost: the network drops, or your HTTP client gives up after 5 seconds while the provider finishes after 6. A timeout means "I don't know", not "it failed". If you retry, you may charge twice. If you don't, you may not have charged at all.

The fix is an idempotency key: a unique ID for the operation (this checkout), created once and sent with every retry. A provider that supports keys charges at most once per key, for as long as it remembers the key. Stripe's API, for example, accepts an Idempotency-Key header and returns the saved result when the same key comes back (docs). A toy provider shows the difference:

class PaymentProvider:
    """A toy provider: the charge really happens, but a reply can get lost."""
    def __init__(self, lost_replies=0):
        self.charges = []             # what the bank actually did
        self.replies = {}             # idempotency key -> original reply
        self.lost_replies = lost_replies

    def charge(self, amount, key=None):
        if key is not None and key in self.replies:
            return self.replies[key]  # a retry: replay the first reply
        reply = f"charge-{len(self.charges) + 1}"
        self.charges.append(amount)
        if key is not None:
            self.replies[key] = reply
        if self.lost_replies > 0:     # charged, but the caller never hears back
            self.lost_replies -= 1
            raise TimeoutError("no reply")
        return reply


def pay(provider, amount, key=None, attempts=3):
    for _ in range(attempts):
        try:
            return provider.charge(amount, key=key)
        except TimeoutError:
            pass                      # outcome unknown, so try again
    raise RuntimeError("gave up")


plain = PaymentProvider(lost_replies=1)
pay(plain, 40)
print("no key:  ", plain.charges)     # [40, 40]: charged twice

keyed = PaymentProvider(lost_replies=1)
pay(keyed, 40, key="order-7731")
print("with key:", keyed.charges)     # [40]: charged once

Your own service needs the same guarantee, because your clients retry too, and people double-click "Pay":

1. Reserve the key first. Insert a row with the key and the status pending into a table where the key is UNIQUE. A duplicate request collides with that row. If the row says done, return the stored result. If it says pending, answer "still processing", or carry on at step 2, which is safe because of the key.
2. Call the provider with the same key.
3. Record the result on that row, then return it.
4. If you crash between steps 2 and 3, the row stays pending. A retry, or a reconciliation job, calls the provider again with the same key, and the provider replays the original result instead of charging again. That works only within the provider's key-retention window (Stripe may prune keys once they are 24 hours old). After that, look the charge up first, for example by the order ID you stored in its metadata, before trying again.

The price is a table, a pending state and a reconciliation job. The classic broken version looks for an existing row first and inserts one only after charging, with no unique constraint. Two concurrent requests both see "no row" and both charge; a crash after the charge leaves no row, so the retry charges again. Queues redeliver jobs too, and the same idea makes a queue consumer safe to run twice; Messaging & Queues covers that.

Not all data needs the same guarantee

During a failure, some reads can only be answered from a copy that might be out of date. Whether that is acceptable depends on the operation, not the whole system. A like count a few seconds behind hurts nobody, so serve it from any replica. "Is seat 14C still free?" must be decided by one authority, a single primary with a transaction or a unique constraint, even if that means refusing to sell seats while the authority is unreachable. Decide per operation, and say which operations get which treatment. Key Concepts & Terminology states the CAP theorem and the consistency models precisely.

Your turn: the region goes dark

Your shop runs in one cloud region, spread across three availability zones. The interviewer asks: "What happens if the whole region goes down? Would you go multi-region?" Answer in two or three sentences, as you would in the room.

Final round: no label on the problem

Real prompts don't say which move they want. Start from the requirements, put numbers on them, find the pressure, then pick the move and its price.

Challenge 1: results day

A national exam board publishes results for 3 million students at 09:00. In past years 60% checked in the first ten minutes, and the first minute was three times busier than the ten-minute average. Each student logs in and sees only their own result. Results never change after publication.

Before peeking, answer these:

1. What is the peak request rate, and how many app servers does it take if one handles 1,000 requests per second?
2. Where do the results live at 09:00, and why?
3. Which caching mistake here would be a scandal rather than a slowdown?
4. Why is autoscaling alone not enough?

Challenge 2: the overnight report run

Every night at 01:00, a B2B product generates a PDF report for each of its 20,000 customers and emails it. One report takes 3 seconds of CPU. Reports must arrive by 08:00, and no customer should get the same report twice. How many workers do you need, and what happens when one crashes halfway?

Cheat sheet: pressure → move → price

Before moving on, pick a product you used today (a food-delivery app, your bank, a video site) and explain it aloud as in an interview, in five minutes. Scope it with three clarifying questions and two or three numbers, each with the decision it drives. Sketch the path from DNS to database, tracing one read and one write. Go deep on one failure and the price of its fix. Then wrap up in two sentences: the top risk, and what changes at 10×. If you can do that without notes, the rest of this course adds depth to a map you already hold.

Next: Key Concepts & Terminology states scalability, latency, availability and consistency precisely, and Networking Basics covers DNS, HTTP, CDNs, TCP, UDP and QUIC. Then Core Building Blocks opens up each box on the path.