Core Building Blocks

One box, launch day

You are building a secondhand marketplace. People photograph things they want to sell, post listings, and browse everyone else's. Version one runs on one machine: the web app, PostgreSQL and the uploaded photos all live on a 16-core VM with a 2 TB SSD and a 10 Gbit/s network card. The beta runs fine on it.

Then marketing does its job. Here are the assumptions for launch month and what they imply:

Predict first: which parts of the one box fail on launch day, in what order, and which part that people usually blame first survives?

In a product full of photos, the app servers and the bytes (network and disk) break the design long before the queries do. And one problem shows up in no traffic number at all: the machine is a single point of failure. A kernel update that needs a reboot takes the whole marketplace offline.

Each of these failures has a standard fix, and each fix is a building block: a component whose behavior, limits and failure modes are well known, so you can use it without inventing it.

This lesson teaches the first three in depth and ends with the map of where the other three sit. It assumes the life of a request and the estimation habits from Foundations of System Design, and the case for scaling out from Key Concepts & Terminology.

💡 Habit worth building: name the pressure before you name the box. "Photos arrive at 1.8 TB a day, so the bytes go to object storage" earns more credit than "I'll add S3". In an interview, every block you draw should come with three things: the pressure that put it there, the move it makes against that pressure, and what it costs.

Load balancers: one address, many servers

Pressure: more requests than one server can handle, and servers that die.

A load balancer gives clients one stable address and spreads their requests across a pool of interchangeable servers (the backends, or targets). It has three jobs:

1. Spread the load so no backend drowns while another idles.
2. Detect dead backends and send traffic elsewhere.
3. Decouple clients from the pool, so you can add, remove and replace servers without anyone noticing.

                  clients
                     |
            api.market.example
                     |
             +---------------+
             | load balancer |
             +---------------+
             /       |       \
      +-------+  +-------+  +-------+
      | app 1 |  | app 2 |  | app 3 |
      +-------+  +-------+  +-------+

L4 or L7: how much of the request can it see?

Balancers differ in how much of the traffic they read.

• An L4 (transport-layer) balancer sees TCP or UDP: addresses and ports. It chooses a backend when a connection starts and sends every packet of that connection to the same backend. It never parses the HTTP inside. Packet-forwarding designs such as Linux IPVS and Google's Maglev do not even end the client's TCP connection; the backend does. AWS Network Load Balancer works the same way unless you ask it to terminate TLS.
• An L7 (application-layer) balancer ends the client's connection, reads each HTTP request, and sends it over its own connection to a backend. Because it sees the method, path, headers and cookies, it can send /api/* and /images/* to different pools, choose a backend per request, and retry a failed request elsewhere. AWS Application Load Balancer, NGINX, Envoy and HAProxy in HTTP mode work this way.

Follow one HTTPS request through each:

What each costs you:

• L4 is cheap per packet, handles enormous throughput, and works for any TCP or UDP protocol (databases, MQTT, game traffic). But it balances connections, not requests, and it cannot route by URL or retry.
• L7 gives you content-based routing and per-request balancing. It pays with CPU for parsing and TLS on every request, with a proxy hop (two TCP connections instead of one), and it becomes a place that holds your TLS keys and sees plaintext.

Predict: four API-gateway instances call an internal recommendation service over gRPC. The service runs 20 pods behind an L4 balancer. gRPC uses HTTP/2, which keeps one long-lived connection per client and sends every request over it. What does the CPU graph of the 20 pods look like?

The same trap catches every long-lived connection. WebSockets stay on the server they first reached, so after a scale-out the new servers receive only new connections while the old ones stay hot.

Reverse proxy, L7 balancer, API gateway: one family

These terms blur because they describe the same kind of machine configured for different jobs.

• A reverse proxy accepts connections on behalf of servers and forwards requests to them. Clients see the proxy, never the backends. Typical extras: TLS termination, compression, response caching, and buffering slow clients so a backend worker is not tied up by someone on a bad mobile connection.
• An L7 load balancer is a reverse proxy whose main job is spreading requests over a pool and routing around dead members.
• An API gateway is a reverse proxy at the edge of a set of services, with API concerns added: authentication, per-client quotas and rate limits (rate limiters get their own lesson, Design URL Shortener & Rate Limiter), routing /orders and /users to different services, and sometimes combining several backend calls into one response.
• An L4 packet-forwarding balancer is not a proxy, because it never ends the client's connection.
• A forward proxy is the mirror image: it acts for clients going out, such as a corporate egress proxy.

NGINX, Envoy and HAProxy can each be configured as a reverse proxy, an L7 balancer or a gateway, and each can also run as an L4 proxy that ends the TCP connection; the role comes from the configuration, not the product. Packet-forwarding L4 is the territory of IPVS, Maglev and NLB. Terminating TLS at the balancer is normal practice, and both ALB and NLB support it. If policy requires encryption inside your network, the balancer re-encrypts to the backends. If the backends themselves must hold the keys, use L4 pass-through.

A common layout: DNS sends image and static-file requests to a CDN, and API calls to a load balancer → API gateway → services. You will also see a gateway fleet acting as the edge proxy itself, with no separate balancer in front. Both are sound. The balancer spreads load over identical instances; the gateway decides which service a request belongs to.

⚠️ Keep the gateway thin. It sits on every request's path, so business logic there turns it into a bottleneck and a shared deploy risk. It must scale out and be redundant like any other tier.

Choosing a backend

Real products offer these directly. AWS Application Load Balancer offers round robin (the default), least outstanding requests and weighted random (ALB docs); Envoy's least-request balancer uses power of two choices by default.

Does the choice matter? Simulate it. Ten servers each handle one request at a time. Nine requests in ten are cheap (5 ms, a listing fetch) and one in ten is expensive (100 ms, a search). Requests arrive at random, keeping the servers 80% busy on average. Every policy sees exactly the same stream.

import heapq
import random

def simulate(policy, servers=10, requests=200_000, load=0.8, seed=7):
    rng = random.Random(seed)
    cheap, costly = 5, 100                      # ms of work per request
    mean = 0.9 * cheap + 0.1 * costly           # 14.5 ms
    rate = load * servers / mean                # arrivals per ms
    arrivals, work, t = [], [], 0.0
    for _ in range(requests):                   # same workload for every policy
        t += rng.expovariate(rate)
        arrivals.append(t)
        work.append(cheap if rng.random() < 0.9 else costly)

    pick = random.Random(seed + 1)
    free_at = [0.0] * servers                   # when each server's queue drains
    busy = [[] for _ in range(servers)]         # finish times of unfinished requests
    latencies, turn = [], 0
    for now, w in zip(arrivals, work):
        for q in busy:                          # forget requests that have finished
            while q and q[0] <= now:
                heapq.heappop(q)
        if policy == "round robin":
            s = turn % servers
            turn += 1
        elif policy == "random":
            s = pick.randrange(servers)
        elif policy == "least outstanding":
            s = min(range(servers), key=lambda i: len(busy[i]))
        else:                                   # power of two choices
            a, b = pick.sample(range(servers), 2)
            s = a if len(busy[a]) <= len(busy[b]) else b
        done = max(now, free_at[s]) + w         # one request at a time, in order
        free_at[s] = done
        heapq.heappush(busy[s], done)
        latencies.append(done - now)
    latencies.sort()
    p = lambda q: latencies[int(q * len(latencies)) - 1]
    return p(0.50), p(0.99)

for policy in ["round robin", "random", "power of two choices", "least outstanding"]:
    p50, p99 = simulate(policy)
    print(f"{policy:22} p50 {p50:6.1f} ms   p99 {p99:6.1f} ms")

Other seeds move individual numbers by up to about 25% (round robin's p99 ranged from 576 to 728 ms over 30 seeds), but the order never changes. Real servers run several requests at once, which narrows the gaps: rerun with four request slots per server and p99 is 167 ms for round robin against 100 ms for least outstanding; with sixteen slots the gap almost disappears. The algorithm matters most when each backend can run only a few requests at a time (single-threaded runtimes, GPU inference, small connection pools) and when request costs are very uneven.

Why it works. A request's latency is mostly time spent waiting behind other requests. Round robin hands the next request to a server stuck behind a 100 ms search while other servers sit idle. Least outstanding sends it to an idle server whenever one exists, so the median request does not wait at all. Two random samples are enough to avoid the longest queues most of the time, and that matters in production, where you run many balancer nodes and each sees only its own in-flight counts. If they all chase "the least loaded backend" using stale counts, they pile onto the same one together; two random choices spread them out.

🧠 Say it like this: "Request costs vary a lot here, searches versus point reads, so I'd use least outstanding requests, or power of two choices across a fleet of balancers. Round robin would queue cheap requests behind expensive ones." A likely follow-up: "What if one backend is a bigger machine?" Weights, or least outstanding, which adapts on its own because the bigger machine finishes requests sooner.

Affinity: when the same key must reach the same server

Sometimes you want a key to land on the same backend every time: a server holding a large per-user cache, or a shard of in-memory state. Hash the key and pick a server from the hash.

The naive hash(key) % N has a nasty property. When N changes, almost every key moves: going from N to N + 1 servers remaps N/(N + 1) of the keys. From 9 servers to 10 that is 90% of the keys, and every moved key starts cold on its new server. Consistent hashing moves only the new server's share, about 1/10 here (9.6% in a 200,000-key run). How the ring and its virtual nodes work belongs to Advanced Topics & Final Prep. Also watch the key you hash: hashing by source IP puts an entire office, or a mobile carrier's shared address, onto one backend.

Health checks: noticing that a server died

Balancers learn about dead backends in two ways:

• Active checks. The balancer probes each backend on a schedule, such as GET /healthz every 10 seconds with a 2-second timeout. After a number of consecutive failures (the unhealthy threshold) it stops sending traffic; after a number of consecutive successes (the healthy threshold) it resumes. Requiring several consecutive results both to leave and to return gives hysteresis, so one lost probe doesn't bounce a server in and out of the pool. A threshold of 1 does exactly that bouncing, called flapping.
• Passive checks, also called outlier detection. The balancer watches real traffic and ejects a backend after, say, five consecutive errors or connection failures. Open-source NGINX's max_fails and fail_timeout settings work this way. Passive checks react faster to hard failures, but only for backends that are receiving traffic.

Follow a hang with active checks every 10 s, a 2 s timeout and an unhealthy threshold of 3. The server freezes at t = 0, just after a successful probe:

For 32 seconds the balancer keeps sending this server its share of traffic. With the marketplace's peak of 6,944 requests/s spread over 9 servers (the fleet sized in "Sizing the tier" below), that is about 772 requests/s, so roughly 24,700 requests hang until their clients time out. Defaults can be slower: an AWS Application Load Balancer checks every 30 s with a 5 s timeout and removes a target after 2 failures (ALB health check docs), up to about 65 s. Shorten the window with faster probes, passive checks, and a per-request timeout at the balancer that retries idempotent requests on another backend.

Predict: to be thorough, your /healthz runs SELECT 1 against the database, with the probe settings above. The database fails over and is unreachable for two minutes, while most pages could still be served from the cache. What happens to the site?

Two more failure modes to name in an interview:

• Draining. When you remove a backend on purpose (a deploy or a scale-in), the balancer should stop sending it new requests and let in-flight ones finish. AWS calls this the deregistration delay; it defaults to 300 seconds. Long-lived connections make draining slow, so WebSocket servers close connections in batches and let clients reconnect elsewhere.
• Retry storms. Retries multiply across layers. If the client, the balancer and the service each try three times, one slow database query can turn into 3 × 3 × 3 = 27 attempts at the bottom, exactly when the database is least able to cope. Retry at one layer, cap retries with a budget, back off with jitter, and never blindly retry a request that is not idempotent, such as a payment (Networking Basics covers idempotency).

Who balances the balancer?

A single balancer is a single point of failure in front of everything you made redundant. The standard answers:

• Active-passive pair with a floating IP. Two balancer machines run VRRP (keepalived is the usual tool). The active one owns a virtual IP address. If its heartbeats stop (one per second by default, with takeover after about three missed), the standby claims the address. Connections that were open through the failed machine are normally lost, and clients reconnect.
• Active-active behind DNS or anycast. Several balancers each carry traffic. DNS returns several addresses, or one address is announced from several places (anycast) and routers spread the flows. Capacity grows with each balancer you add. DNS-based failover is only as fast as clients honor the record's TTL (see Networking Basics).
• Managed cloud balancers run as a fleet across availability zones and scale themselves. Your job is to spread your backends across zones too.
• Across regions, DNS that answers by geography or measured latency, or anycast, sends each user to a nearby healthy region, and each region runs its own balancers.

                    clients
                       |
        virtual IP 203.0.113.10 (VRRP)
                       |
          +------------+------------+
          |                         |
   +-------------+          +---------------+
   | LB-a ACTIVE |          | LB-b STANDBY  |
   | owns the IP |<-------->| takes the IP  |
   +-------------+ heartbeat| if LB-a dies  |
          |                 +---------------+
          v
   app servers (both LBs know the whole pool)

Your turn: sellers can now chat live with buyers over WebSockets: 300,000 concurrent connections at peak across 20 chat servers. Which balancer and algorithm do you choose, and what happens during a deploy and after a scale-out?

The stateless service tier

Pressure: you must add and remove servers freely, for peaks, deploys and failures, and the balancer must be able to send any request to any server.

A service tier is stateless when no request depends on something that only one instance holds: no session in local memory, no uploaded file on local disk, no in-memory counter that is the only copy. Instances still hold state, such as caches and connection pools, but only copies that can be rebuilt. That is what lets the balancer treat servers as interchangeable, and it is what makes autoscaling and rolling deploys routine.

Predict: on the one box, login stored the session in the app's memory. You add a second server behind a round-robin balancer. What do users see?

Where the state goes

Signed tokens (a JWT in a cookie or header) let the server keep nothing: it checks the signature and trusts the claims inside. The price is revocation. A stolen token, or one belonging to a user who just logged out, stays valid until it expires. The usual compromise is short-lived access tokens (minutes) plus a refresh token that is checked against a store.

Sticky sessions: the tempting shortcut

A balancer can pin each user to one server with a cookie (AWS calls this stickiness). It hides the state problem without solving it:

• when the server dies, its users lose their sessions;
• deploys and scale-ins must either wait for sessions to end or break them;
• load becomes uneven: one server collects a few very heavy users, and after a scale-out the existing users stay on the old servers.

Stickiness is fine as a performance hint when the state can be rebuilt, such as a warm per-user cache. It is a trap when the pinned server holds the only copy.

Sizing the tier

How many app servers does the marketplace need at peak? Assumptions: 6,944 requests/s at peak; 8 ms of CPU per request; 16 cores per server; plan for at most 60% CPU so a spike or a lost server doesn't tip you over; three availability zones, and you must survive losing one.

The last row is Little's law: requests in flight = arrival rate × time in the system. A request that spends 60 ms mostly waiting on the cache and the database still occupies a worker for those 60 ms. A pool of 32 threads therefore caps a server at 32 ÷ 0.06 s ≈ 533 requests/s while its CPU sits mostly idle. Size worker pools, or use asynchronous I/O, from this number rather than from the core count.

⚠️ Autoscaling is not instant. A new instance takes minutes to boot, pass health checks and warm its caches. That is why you keep headroom (the 60%) and scale on a leading signal such as requests per instance, not only on CPU. Some balancers also ramp a new backend up gradually instead of handing it a full share of traffic cold.

Your turn: product wants "log out of all devices" to take effect within 5 seconds, and the platform team refuses to add a network lookup to every request. Sessions in Redis add a lookup; signed tokens that live 15 minutes break the 5-second rule. What do you propose?

Object storage: where the bytes live

Pressure: large blobs (photos, video, PDFs, backups, logs) that grow without limit, are written once and read many times.

Object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compatible systems such as MinIO and Ceph) stores objects: a blob of bytes plus metadata, addressed by a key inside a bucket, and read and written over HTTP.

What you get (S3's documented behavior):

• Capacity and object counts that are effectively unlimited. You pay per GB stored per month, per request, and per GB sent out.
• Durability designed for 99.999999999% ("eleven nines"), achieved by storing the data redundantly across several facilities.
• Strong read-after-write consistency: once a PUT succeeds, every later GET or LIST sees it (S3 consistency model).
• Request rates that scale by key prefix: at least 3,500 writes and 5,500 reads per second per prefix, with S3 adding capacity as load grows.

What you give up:

• It is not a filesystem. You write an object whole; you don't edit bytes in the middle of it. A "rename" is a copy plus a delete, and "folders" are only key prefixes.
• Latency. AWS quotes roughly 100–200 ms for small objects or the first byte of a large one. That is fine for images behind a CDN and wrong for a hot key-value lookup.
• No queries. You can list keys by prefix and that is all. Anything you filter or sort by lives in a database.

The pattern: facts in the database, bytes in the bucket

The database row holds what you query: photo ID, listing ID, owner, status, and the object key, such as listings/42/7156281536.jpg. The bucket holds the bytes. Putting 3 MB blobs in the database bloats its backups and replication; keeping queryable facts only inside objects makes them unsearchable.

Store the key, not the URL. Build URLs when you read, from configuration: the CDN's domain plus the key. When you change CDN provider, rename the domain or move buckets, you edit one setting instead of rewriting 600 million rows.

The upload path: let clients talk to the bucket

client           API         database     object store   resize worker
  |               |              |              |              |
  | 1 add photo   |              |              |              |
  |-------------->|              |              |              |
  |               | row: pending |              |              |
  |               |------------->|              |              |
  | 2 presigned   |              |              |              |
  |   PUT URL     |              |              |              |
  |<--------------|              |              |              |
  | 3 PUT 3 MB    |              |              |              |
  |-------------------------------------------->|              |
  |               |              |              | 4 created    |
  |               |              |              |   event,     |
  |               |              |              |   via queue  |
  |               |              |              |------------->|
  |               |              |              | 5 resized    |
  |               |              |              |   copies     |
  |               |              |              |<-------------|
  |               |              | 6 ready      |              |
  |               |              |<----------------------------|

1. The client asks to add a photo. The API checks that the user may edit listing 42 and inserts a photo row with status pending and a key.
2. The API returns a presigned URL: a URL signed with the server's credentials that allows one kind of operation (a PUT to this key) until it expires, here in 15 minutes.
3. The client uploads straight to object storage. The 3 MB never pass through your app servers.
4. Object storage emits an "object created" event, and a queue delivers it to a resize worker.
5. The worker writes the thumbnail, card and full-size copies back to object storage, under keys derived from the photo ID.
6. The worker marks the row ready. Readers now receive CDN URLs built from the keys.

Why bother? Routed through the app tier, uploads would carry 1.8 TB a day, about 170 Mbit/s on average and 500 Mbit/s at peak, through servers sized for small JSON requests. Worse, each slow phone upload holds a worker. At the peak of about 21 photos per second, 20-second uploads keep roughly 420 requests in flight, about 46 per server, most of the 72-request budget computed above.

Failure modes, and what each costs:

• The client never uploads. The row stays pending forever, so a cleanup job deletes rows still pending after a day.
• The photo lands but the worker never finishes. The event is lost or the worker keeps crashing, so the bytes sit in the bucket while the row says pending. Queue retries and a dead-letter queue handle most of it; the same cleanup job deletes the object together with the stale row, so nothing is orphaned.
• A presigned URL leaks. It is a bearer token: whoever holds it can use it, even repeatedly, until it expires. Keep expiry short, sign one key per URL, and check the object's type and size before marking it ready. S3's presigned POST policies can also cap the size with a content-length-range condition.
• A huge file. One S3 PUT accepts up to 5 GB; AWS recommends multipart upload from about 100 MB. Multipart splits the file into parts of 5 MiB to 5 GiB, up to 10,000 of them, uploaded in parallel and retried one by one, for objects up to about 50 TB (S3 upload limits). A video on a flaky connection resumes from the last good part instead of starting over.

Serving: put a CDN in front

At peak the marketplace serves about 27,800 image requests a second, 33 Gbit/s. You don't serve that from the bucket. A CDN caches images near users; at a 95% hit ratio the bucket sees about 1,400 requests/s and 1.7 Gbit/s at peak. Because a replaced photo gets a new key, an image's bytes never change under its URL, so the CDN can cache it for a very long time and never needs invalidating for updates. Deletions are different: a takedown or a seller removing a photo of their home still needs a CDN purge, or a TTL short enough to live with. Private files, such as invoices, use short-lived signed URLs instead of public caching. Pull versus push CDNs, cache headers and invalidation are in Networking Basics.

Your turn: sellers may now attach a 2–10 minute video (up to 2 GB) to a listing. The app servers keep a 30-second request timeout. Sketch the upload and what happens after it.

Unique IDs without a single counter

Pressure: many servers create records at the same time, and each record needs a unique key without a round trip to one central counter.

On the one box, PostgreSQL's bigserial hands out 1, 2, 3, and that goes surprisingly far: one primary database can issue thousands of IDs a second. It stops being enough when:

• writes are split across databases, and two shards would both hand out 1,001;
• you need the ID before the row exists: the photo key in the upload flow, a record created offline on a phone, a message ID the client retries with;
• writers sit in several regions, and a cross-region call on every insert costs tens of milliseconds.

Decide what the ID must do before choosing how to make it:

The options:

Flickr has described its ticket servers: two MySQL servers, one issuing odd numbers and one even, so that neither is a single point of failure (Flickr engineering, 2010). UUIDv7 was standardized in RFC 9562 in May 2024; PostgreSQL 18 ships uuidv7() and Python 3.14 ships uuid.uuid7().

Are random UUIDs safe from collisions? With n IDs drawn from 2¹²² values, the chance of any collision is about n² ÷ 2¹²³. For a trillion IDs that is about 9 × 10⁻¹⁴. A broken random number generator is a far likelier source of duplicates than the math.

Snowflake: 64 bits, three fields

Twitter has described Snowflake, its ID service from 2010: a 41-bit millisecond timestamp counted from a custom epoch (about 69 years of range), a 10-bit machine ID (1,024 machines) and a 12-bit sequence number (Snowflake README).

 bit 63   bits 62..22                    bits 21..12         bits 11..0
 +----+-------------------------------+-------------------+------------------+
 | 0  | ms since custom epoch (41)    | worker ID (10)    | sequence (12)    |
 +----+-------------------------------+-------------------+------------------+

Here is a generator. The clock is a parameter so the demo is deterministic:

import threading
import time

EPOCH_MS = 1_704_067_200_000        # custom epoch: 2024-01-01 00:00:00 UTC
WORKER_BITS, SEQ_BITS = 10, 12
MAX_SEQ = (1 << SEQ_BITS) - 1       # 4095

def wall_clock_ms():
    return time.time_ns() // 1_000_000

class Snowflake:
    def __init__(self, worker_id, clock=wall_clock_ms):
        if not 0 <= worker_id < (1 << WORKER_BITS):
            raise ValueError("worker id must fit in 10 bits")
        self.worker_id, self.clock = worker_id, clock
        self.last_ms, self.seq = -1, 0
        self.lock = threading.Lock()

    def next_id(self):
        with self.lock:
            now = self.clock()
            if now < self.last_ms:                 # NTP stepped the clock back
                raise RuntimeError(f"clock went back {self.last_ms - now} ms")
            if now == self.last_ms:
                self.seq = (self.seq + 1) & MAX_SEQ
                if self.seq == 0:                  # 4,096 IDs this ms: wait
                    while now <= self.last_ms:
                        now = self.clock()
            else:
                self.seq = 0
            self.last_ms = now
            return ((now - EPOCH_MS) << (WORKER_BITS + SEQ_BITS)
                    | self.worker_id << SEQ_BITS
                    | self.seq)

def decode(snowflake_id):
    return {"ms_since_epoch": snowflake_id >> (WORKER_BITS + SEQ_BITS),
            "worker": (snowflake_id >> SEQ_BITS) & ((1 << WORKER_BITS) - 1),
            "seq": snowflake_id & MAX_SEQ}

# Deterministic demo: three calls in one millisecond, then one in the next.
ticks = iter([1_750_000_000_000] * 3 + [1_750_000_000_001])
gen = Snowflake(worker_id=37, clock=lambda: next(ticks))
for _ in range(4):
    i = gen.next_id()
    print(i, decode(i))

Output:

192656126771351552 {'ms_since_epoch': 45932800000, 'worker': 37, 'seq': 0}
192656126771351553 {'ms_since_epoch': 45932800000, 'worker': 37, 'seq': 1}
192656126771351554 {'ms_since_epoch': 45932800000, 'worker': 37, 'seq': 2}
192656126775545856 {'ms_since_epoch': 45932800001, 'worker': 37, 'seq': 0}

Within one millisecond the sequence counts up. In the next millisecond it resets and the ID jumps by 2²² − 2, because the timestamp sits above the other 22 bits. Numeric order is time order.

Why it works. Two IDs from one worker never collide while the process runs: the pair (timestamp, sequence) never repeats, because the sequence counts within a millisecond and the generator refuses to go back in time. The guard lives in memory, so a process that restarts after the clock stepped back (a VM restored from a snapshot, say) must first wait until the clock passes the last timestamp it persisted, or take a fresh worker ID. IDs from different workers differ in the worker bits. Across workers the order is only rough: if two servers' clocks disagree by 3 ms, an ID minted later on one can be smaller than an ID minted earlier on the other. Snowflake's own documentation promises only rough ordering (IDs "k-sorted"), not a strict global sequence.

The capacity math: 4,096 IDs per millisecond is about 4.1 million per second per worker, and 2⁴¹ ms is about 69.7 years, so an epoch of 2024 runs out in September 2093.

Failure modes

• The clock goes backwards. An NTP correction or a VM migration steps the clock back 6 ms. Without the guard, the generator would reuse timestamps it already used, restart the sequence at 0 and mint duplicates. So it refuses, or waits out small steps and alerts on large ones. Configuring time sync to slew (adjust gradually) rather than step makes this rare.
• Two workers share an ID. Two instances with worker ID 37 mint identical IDs whenever they issue one in the same millisecond at the same sequence number. Worker IDs must be assigned, from a lease in a coordination service such as etcd or ZooKeeper, a database table, or a stable ordinal from the orchestrator. Picking one at random at boot is the birthday problem: 40 instances choosing among 1,024 IDs collide with probability 54%.
• More than 4,096 IDs in one millisecond. The generator waits for the next millisecond; that caps one worker at about 4.1 million IDs per second.
• A hot range partition. Time-ordered IDs are excellent keys inside one database's B-tree, and a poor key for splitting data into ranges across machines, because every new write lands in the newest range. Hash the ID to choose a shard instead (hot keys and partitioning are in Databases & Storage).
• Information leaks. Time-ordered IDs reveal when a record was created, and sequential ones reveal how many exist: a competitor places two orders a day apart and subtracts. If that matters, expose a separate random public ID. An unguessable ID is not access control either way; the server must still check that the caller may read /orders/….

Random keys in a B-tree

Most databases keep the primary key in a B-tree, and MySQL's InnoDB stores the whole table in primary-key order. Time-ordered keys arrive in increasing order, so inserts go to the right edge: the same few pages stay in memory and fill up completely. Random UUIDv4 keys land anywhere, so each insert touches a random page that may not be in memory, and pages split and stay partly empty. Once the index is bigger than memory, the difference shows up as disk reads on every insert. In InnoDB every secondary index also stores the primary key, so a 16-byte key instead of an 8-byte one makes every index bigger. Indexes in depth are in Databases & Storage.

🧠 Say it like this: "I'd use Snowflake-style 64-bit IDs. There's no coordination per insert, they fit a BIGINT, and they sort by time, so 'newest first' is an index scan. The costs are a dependency on the clock, which the generator guards against, and assigning worker IDs, which I'd lease from etcd." A likely follow-up: "Why not UUIDv7?" It would also work. It needs no worker IDs at all, and costs 16 bytes instead of 8.

Your turn: order IDs appear in customer-facing URLs, and the business does not want competitors estimating order volume. Orders live in one MySQL (InnoDB) primary, with a peak of 3,000 inserts a second and about 1,000 a second on average. What do you use?

The map: where caches, databases and queues fit

Put the pieces together for the marketplace. Uploads, not drawn here, go straight from the client to object storage with presigned URLs.

   DNS answers "where is img.market.example / api.market.example?"
   and then steps aside: it carries none of the traffic below.

              browser / mobile app (its own HTTP cache)
                |                                  |
          image requests                       API calls
                |                                  |
            CDN edge                       load balancer (L7)
                | on a miss                        |
                v                        stateless app servers
         object storage                   (in-process cache)
           |        ^                     |         |         |
           |        |                   cache   database    queue <---+
           |        |                  (Redis)  (primary +    |       |
           |        |                            replicas)    v       |
           |        +------------------------------------- workers    |
           |             resized photos             (resize photos,   |
           |                                         send emails,     |
           |                                         update search)   |
           +-------------------- "object created" events -------------+

The app servers look in the cache first and go to the database on a miss (the cache-aside pattern). Two sources feed the queue: object storage publishes an "object created" event when an upload lands, which triggers resizing, and the API publishes jobs such as emails and search-index updates. The workers write resized photos back to object storage.

Caches: pick the layer

Cache as close to the user as the data's audience allows: public and unchanging data goes to the CDN; per-user data can live in the user's browser (private caching) or in the app tier, never in a shared CDN cache; fast-changing data gets a short TTL or is deleted on write.

Predict: the listing API serves 6,944 reads a second at peak through Redis. The hit ratio rises from 90% to 99%. By what factor does database read load fall?

Databases: the source of truth

The map has two sources of truth: the database for facts and the original photos in object storage for bytes. Everything else is derived from them (cache entries, CDN copies, resized photos, search indexes) and can be rebuilt, while nothing can rebuild either source. The database is also the hardest tier to scale out, because every row has a home: read replicas carry read-heavy load (and lag behind), sharding splits large or write-heavy data. In the marketplace, one primary with replicas is plenty, because 1.2 listing writes a second and small metadata rows are all it carries. The bytes went to object storage. Choosing and scaling databases is Databases & Storage.

Queues: take slow work off the request path

Ask one question: does the user need this result before the response? If not, queue it. In the marketplace that covers resizing photos (seconds of CPU each, triggered by object storage's upload event), sending "your item sold" emails and updating the search index (published by the API). The request that caused the work answers in milliseconds; workers catch up, and a spike becomes a backlog instead of timeouts.

What you pay: the work finishes later (the listing shows a "processing" placeholder), a message may be delivered more than once, so workers must be idempotent (write resized copies to keys derived from the photo ID, so a repeat overwrites the same objects), and the backlog needs monitoring. Delivery guarantees, ordering and dead-letter queues are in Messaging & Queues.

Every box has a price

Every hop is another network round trip (well under a millisecond inside one data center, but they add up), another thing to deploy, monitor and page someone about, and another way to fail. The marketplace earned each block with a number; a prototype with a hundred users earns almost none of them. Interviewers reward "I'd add a cache if reads outgrow one database" over boxes drawn by reflex, and they discount "because big company X uses it".

Final round: no labels

Real prompts don't say which block they want. Find the pressure first, then pick the move and name its price.

Challenge 1: invoices at month end

A billing service must generate 1.2 million invoice PDFs on the first of each month (2 seconds of CPU each), email each customer a link, and keep every invoice downloadable for 7 years. The API tier is six small servers with a 30-second timeout. Name the blocks and the numbers that justify them.

Challenge 2: the internal tool

An expense tool for 200 employees, at most 5 requests a second during office hours. It must be deployable without downtime and survive the loss of one server. Draw the design and say what you would add later, and when.

Challenge 3: a flash sale

A ticketing site sells 50,000 concert tickets at 10:00 sharp. 400,000 people have the page open, and within ten seconds of 10:00 they send 400,000 "buy" requests, about 40,000 a second. You scaled the load balancer and the app tier ahead of time: 40 servers, enough for 48,000 requests a second. Which block fails first, and what do you put in front of it?

Cheat sheet: pressure → move → cost

Before moving on, explain the marketplace aloud as you would at a whiteboard, without notes: follow one photo upload and one listing view end to end, and at every box say the pressure that put it there and what it costs. Then answer three follow-ups: one app server dies mid-request; a server's clock steps back 50 ms; the active load balancer's machine loses power. If each answer names the mechanism that saves you (or doesn't), you are ready.

Next: Databases & Storage goes deep on the database and cache boxes, and Messaging & Queues on the queue. To see ID generation put to work in a complete design, try URL Shortener (e.g. TinyURL).