Twitter/Instagram News Feed
Design feed generation using push vs pull models, ranking algorithms, and fanout-on-write strategies.
SPACED REPETITION Β· 18 practice questions
Make this lesson stick.
Try 3 questions now. No account needed. Sample answers aren't saved.
or sign in to practice all 18The feed is a query you cannot afford to run
You open the app and expect the newest posts from everyone you follow, in well under a second. The obvious design keeps posts in one table and builds the feed when you ask for it:
SELECT p.post_id, p.author_id, p.body, p.created_at
FROM posts p
JOIN follows f ON f.followee_id = p.author_id
WHERE f.follower_id = :me
ORDER BY p.created_at DESC
LIMIT 20;
For a startup with 10,000 users and an index on posts(author_id, created_at), this is the right answer. Now give it real load. Assume 200 million daily active users who each load 10 feed pages a day and follow 200 accounts on average. That is about 23,000 feed pages per second on average and 69,000 at a 3Γ peak. Every page reaches into 200 authors' recent posts, merges them and sorts them. That is 4.6 million author lookups per second on average and 13.9 million at peak. Most of those pages look almost exactly like the page the same user saw ten minutes earlier.
The SQL is not what breaks first. One database cannot hold the data, so you shard posts by author, and now every feed page is a scatter-gather. With 100 shards, a user who follows 200 authors touches about 87 of them. By definition each shard is slower than its own p99 1% of the time, so if those slow moments are independent, a request that waits for 87 shards hits at least one slow shard about 58% of the time. Your median feed load now looks like your shards' tail latency.
The fix is to stop recomputing the same merge on every read. Do the work once, when a post is written, so that a read is a lookup. That move creates new problems, and each has a standard answer. This lesson teaches six moves, and each one starts with a pressure in the requirements.
| # | Pressure in the requirements | Move | What you pay |
|---|---|---|---|
| 1 | Reads outnumber posts, and each read merges hundreds of authors | Fan out on write into precomputed timelines | Write amplification, RAM |
| 2 | A few authors have tens of millions of followers | Hybrid: skip fan-out for them, merge at read | A read-time merge, hot keys |
| 3 | Users Γ timeline entries outgrow RAM | IDs only, capped, active users only | Rebuilds for returning users |
| 4 | New posts arrive while the user scrolls | Cursor pagination | No "jump to page 37" |
| 5 | "Best first", not newest first | Retrieve candidates, rank at read time | A latency budget and a fallback |
| 6 | Posts are deleted and authors blocked after fan-out | Filter at read, clean up later | Extra work on every read |
This lesson assumes the building blocks. Databases & Storage covers caching, sharding and hot keys. Messaging & Queues covers delivery guarantees and idempotent consumers. Design Social & Streaming Systems is the shared toolkit (fan-out, celebrities, media, real-time); this lesson applies it to one problem in depth.
Before any move: pin down the requirements
"Design Twitter's feed" is several different problems until you ask a few questions. These are the ones whose answers change the design:
| Ask | Why it changes the design |
|---|---|
| Newest first, or ranked? | Ranking adds a read-time stage and changes pagination |
| How soon must followers see a new post? | Sets the fan-out delivery target: seconds or minutes |
| What is the largest follower count, and how skewed is the distribution? | Decides whether you need the hybrid |
| Must authors see their own post immediately? | Adds a read-your-own-writes path |
| DAU, feed loads per user, posts per day, follows per user | Gives QPS, fan-out rate and memory |
| How far back can users scroll? | Sets the timeline length and the deep-scroll fallback |
Photos and video change storage (object storage behind a CDN), not the feed logic, so park them. Requirements Gathering covers how to run this conversation. For the rest of this lesson the agreed scope is:
- Functional: create a post; follow and unfollow; read the home timeline, meaning posts from accounts you follow, newest first. Ranking arrives in Move 5.
- Non-functional: first page p99 under 200 ms; a new post appears in 95% of followers' feeds within 5 seconds; authors always see their own new post; availability matters more than instant freshness.
That 5-second window is a consistency decision you make during normal operation, not only when the network partitions. In PACELC terms it is the "else" branch: when there is no partition, you trade consistency for latency. Key Concepts & Terminology states CAP and PACELC precisely. Say it that way in the interview, instead of "the feed is AP".
The numbers. State the assumptions aloud, then compute. These were computed with python3 and rounded:
| Assumption | Value |
|---|---|
| Daily active users | 200M |
| Feed page loads per user per day | 10, with 20 posts per page |
| New posts per day | 100M |
| Accounts followed per user, and followers reached per ordinary post | 200 |
| Peak Γ· average | 3Γ |
| Quantity | Arithmetic | Average | Peak |
|---|---|---|---|
| Feed page loads | 200M Γ 10 Γ· 86,400 | 23k/s | 69k/s |
| New posts | 100M Γ· 86,400 | 1.2k/s | 3.5k/s |
| Pull: author lists read | 23k Γ 200 | 4.6M/s | 13.9M/s |
| Push: timeline inserts | 1.2k Γ 200 | 231k/s | 694k/s |
| Posts fetched to fill pages | 69k Γ 20 | 1.4M/s |
Move 1: Fan out on write
Pressure: every read repeats a merge that only changes when someone posts.
Flip the work around. When a post is written, push its ID into a precomputed home timeline for each follower. Reading the feed becomes: read your timeline, fetch those posts by ID, return them. This is fan-out on write, or push. Building the merge on every read is fan-out on read, or pull.
The write path, step by step
client ββPOST /postsβββΊ Post service ββ2. append post_idβββΊ Author timeline
β (the author's own posts)
β 1. one transaction: post row + outbox row
βΌ
Posts DB ββ3. relayβββΊ queue (post_created)
β 4. at-least-once
βΌ
Graph store βββ5. pagesββ Fan-out workers
(follower lists) β 6. insert post_id + author_id
βΌ
Timeline cache (one key per active user)
| Step | What happens | What is true afterwards |
|---|---|---|
| 1 | The Post service writes the post and an outbox row in one transaction | The post is durable |
| 2 | It appends the post ID to the author's own author timeline, a short list of their recent posts under one key, then returns 201 | The author can see the post; no follower sees it yet |
| 3 | A relay publishes the outbox row to the queue | The fan-out job exists, even if the service crashed right after step 1 |
| 4 | A fan-out worker takes the job | It may receive the same job twice |
| 5 | It reads the author's follower list a page at a time, a few thousand IDs per page | Checkpointing the page cursor lets another worker resume after a crash |
| 6 | It inserts the post ID and author ID into each follower's timeline, then trims it to the cap | Followers see the post on their next read |
Three details decide whether this is correct:
- Why the outbox. Saving the post and publishing to the queue touch two separate systems. If the service crashes between them, the post exists but is never fanned out. Writing the event into the same database transaction, and relaying it afterwards, closes that gap. Change data capture from the database log does the same job. Advanced Topics & Final Prep covers the outbox pattern.
- Why step 2 happens before the 201. It is the one write the author's next read depends on, and it is a single small insert. The fan-out worker repeats it as a backstop (the insert is idempotent), so if step 2 fails after the commit, the author sees the post a few seconds late instead of never.
- Why the insert must be idempotent. At-least-once delivery means a worker that crashes after writing 3,000 of 5,000 timelines gets redone from the start or from its last checkpoint. The second insert of the same post must change nothing. Move 3 shows which data structure gives you that for free.
Predict first: the author's own home timeline is just one of the thousands of keys the workers will write, maybe seconds from now. The author reloads the moment the 201 arrives. How does this design still meet "authors always see their own post"?
Check your answer
Through step 2. The read path merges the reader's own author timeline into every page, and step 2 wrote the post there before the 201, so it is on the very next read, from any device. Writing the author's own home timeline synchronously in step 2 would also work. Either way it is exactly one extra synchronous write, never the whole fan-out. Relying on the workers does not work: their lag is seconds normally and minutes during a backlog.
Why it works: the work moves to where it is cheap
Push costs (posts per second) Γ (followers per post). Pull costs (reads per second) Γ (accounts followed per read). Every follow edge has exactly one follower and one followee, so across the whole graph the average number of followers equals the average number of followees. The ratio of the two costs is therefore roughly the ratio of reads to posts: 20 to 1 here, or 231k inserts per second instead of 4.6M author reads per second.
The latency win does not depend on that ratio. A pushed read touches one timeline key plus one batched fetch of 20 posts, instead of waiting on 87 shards.
The cost argument has one hole, and it is the next move. An average hides the distribution. If heavy readers follow more accounts than average, pull gets worse. If heavy posters have more followers than average, push gets worse, and on social networks they do.
π‘ Say it like this: "Reads are about 20 times posts, and a pull feed would scatter-gather across most shards on every page. So I'll precompute each user's home timeline with an asynchronous fan-out on write. Followers see a post within a few seconds, which meets the 5-second target, and the author sees it at once, because the Post service appends it to the author's own post list before returning and every read merges that list."
Likely follow-up: "What happens when someone with 100 million followers posts?"
Move 2: The celebrity problem
Pressure: follower counts are extremely skewed.
Take an account with 100 million followers, 40% of whom are active and have a timeline. One post means 40 million timeline inserts. Size the fan-out tier at 1 million inserts per second, comfortably above the 694k/s peak. Even if you spread that one job across the entire tier, it occupies the tier for 40 seconds. At peak, about 139,000 ordinary posts arrive during those 40 seconds and wait. The 5-second target is gone, for everybody.
Twitter has described exactly this. Delivery to a million followers took about 3.5 seconds at p50, but p99 could reach 5 minutes. Queues backed up "all the time for high value fanouts". Followers routinely saw replies to a celebrity's tweet before the tweet itself: a reply from a small account could finish its own fan-out while the celebrity's was still running. The source is High Scalability's summary of Raffi Krikorian's "Timelines at Scale" talk; the numbers are from 2012β2013.
β οΈ This is write amplification, not a hot key. A hot key is heavy traffic concentrated on one key. Here one write turns into 40 million. Keep the two terms apart, because the fixes differ.
The move: don't fan out the big accounts
Pick a threshold. Authors below it are fanned out as before. Authors above it are not fanned out at all: their posts stay in the author timeline that step 2 of the write path filled before the 201. (Every post, big or small, goes there; the same list serves the reader's own posts and rebuilds.) At read time, the feed service merges the reader's precomputed timeline with the author timelines of the big accounts the reader follows.
Bob follows Ana (300 followers), Raj (2k) and @league (100M)
Ana posts ββββββΊ fan-out βββΊ timeline:bob
Raj posts ββββββΊ fan-out βββΊ timeline:bob
@league posts ββΊ author:league only (no fan-out)
Bob reads: merge(timeline:bob, author:league, author:bob) βββΊ newest 20
author:bob is Bob's own post list, the read-your-own-writes merge from Move 1. In the same talk, under "the future", Twitter described moving to this fix: for accounts like Taylor Swift's, "don't bother with fanout anymore, instead merge in her timeline at read time."
Predict: why does the hybrid also cure "replies before the original"?
Check your answer
The celebrity's post no longer travels through the fan-out queue. It is in the author timeline before the Post service even returns, and every follower's read merges that key. A reply from an ordinary account is fanned out within seconds, so it can no longer overtake the post it answers.
Why it works
Pull was expensive because every reader reads different keys: 200 authors each, with little sharing between readers. For a big account the opposite is true: millions of readers read the same key, which is the best case for a cache. Push is expensive only when the follower list is long. The hybrid puts each author on the side where they are cheap.
The cost moves; it does not disappear. Every feed read now also reads a few author timelines. If the average reader follows 5 big accounts, that is 116k extra author-timeline reads per second, but they all land on a few thousand shared keys.
Choosing the threshold
The threshold is a tunable, not a magic number, and you can derive it from the fan-out target. Say you let a single post use at most 100k inserts per second, so one post cannot starve the others, and each post must finish within 5 seconds. Then the cutoff is 100k Γ 5 = 500k active followers. Count active followers, since inactive users get no inserts anyway (Move 3).
Two details matter in practice:
- Hysteresis. An account close to the threshold would flip between push and pull every day. Switch to pull at 500k and back to push only below, say, 400k.
- The crossing, both ways. Push to pull: the account's older posts are already in timelines and its new ones come from its author timeline, so a post can briefly arrive from both sources and the merge deduplicates by post ID. Pull to push: the posts made while the account was pulled exist only in its author timeline. Keep merging that timeline until those posts are older than anything timelines still hold, or backfill them into active followers' timelines; otherwise they vanish from feeds.
The new hot spot
The big author's timeline is now a hot key. Back to the account with 100 million followers: take its 40 million followers who have timelines, two thirds of them active on a given day, each loading 10 pages a day. That one key gets about 3,100 reads per second on average and 9,300 at peak. The post itself is hot as well, since every one of those reads fetches it.
The standard fix is a small in-process cache in each feed server with a 1-second TTL. With 500 feed servers, the key sees at most about 500 reads per second: one per server per second, so its load grows with the size of the fleet, not with the number of readers. One second of staleness fits inside the 5-second target. Splitting the shard does not help, because a single key cannot be split.
Your turn: the interviewer asks, "What about a user who follows 3,000 big accounts?" Their read would merge 3,000 author timelines. What do you do?
Check your answer
For this one user, the read-time merge is now the expensive part. Reasonable answers: merge only the big accounts this user actually engages with, say the top 50 by affinity, and let the rest compete as ranking candidates or not appear at all; or keep a separate "big accounts" timeline for heavy followers like this and refresh it every minute or so. Either way, name the cost: some freshness or completeness for a rare kind of user, in exchange for keeping every other read cheap. There is no free option, and saying so is the point.
Facebook has described a design at the other end of the spectrum. Its Multifeed system builds News Feed when it is requested: an aggregator "fanned out the request to all the leaves", which hold recent activity in memory, then ranked and filtered the results (Facebook Engineering, 2015). Both approaches work. The workload decides, and reasoning about the workload is what the interviewer wants to hear.
π‘ Say it like this: "I won't fan out accounts above about half a million active followers; the number comes from my per-post fan-out budget. Their posts are merged at read time from their author timelines, which are hot keys, so I cache them in each feed server for a second."
Likely follow-up: "What changes when an account crosses the threshold?"
Move 3: Timeline storage: IDs only, capped, active users only
Pressure: hundreds of millions of timelines, each hundreds of entries long, all in RAM.
What goes in an entry
Store IDs, not posts: the post ID, the author ID and a few flag bits. The post body lives once, in the post store. Reading a page fetches the 20 bodies in one batched lookup; this step is usually called hydration. Copying bodies into timelines would multiply storage by the fan-out, and every edit or delete would have to find every copy.
The author ID earns its bytes. It lets the read path drop posts from blocked or unfollowed authors without fetching them (Move 6). Twitter has described its home-timeline entries this way: the tweet ID, the author's user ID and 4 bytes of flags, at most 800 entries per user, stored as Redis lists (same High Scalability source).
Which structure, and what it costs
Two reasonable choices in Redis:
Sorted set (ZADD) |
List (LPUSH + LTRIM) |
|
|---|---|---|
| Inserting the same post twice | No-op: same member | A duplicate entry |
| Order | By score, whatever the arrival order | By arrival |
| "Posts older than X" | A range query by score | Scan and filter |
| Memory per entry, measured | 113 B | 22 B |
The memory row was measured on Redis 7.4 with 2,000 timelines of 800 entries each, from the change in used_memory. The sorted set held post_id:author_id strings with a 19-digit post ID and an author ID of up to 10 digits; with 19-digit author IDs it rises to 129 B. The list held 20-byte packed entries (8 bytes of post ID, 8 of author ID, 4 of flags). Redis stores small sorted sets, 128 entries or fewer by default, in a compact encoding, so a test with 100 entries would badly under-measure a full timeline.
Now size the tier. Keep timelines for the 300 million users active in the last 30 days, 800 entries each, two copies (primary and replica), and fill nodes only to 70% to leave room for fragmentation and fork overhead:
| Structure | Raw | Γ 2 copies | Γ· 0.7 | Nodes with 64 GB |
|---|---|---|---|---|
| Packed list, 22 B | 5.3 TB | 10.6 TB | 15.1 TB | 236 |
| Sorted set, 113 B | 27.1 TB | 54.2 TB | 77.5 TB | 1,211 |
(GB and TB here are 10βΉ and 10ΒΉΒ² bytes.) Two conclusions. First, at this size the data structure is a 5β6Γ memory decision. The sorted set is the clearer interview answer: idempotent, ordered and easy to page. The list is what you choose when RAM dominates, and then you remove duplicates and sort each page on read. Second, memory sizes this cluster, not write traffic. Each insert is applied on a primary and on its replica, so the 694k inserts per second at peak become about 5,900 writes per second on each of the 236 nodes, a light load for a Redis node.
β οΈ Sorted-set scores are 64-bit floating-point numbers, exact only up to 2β΅Β³. A 64-bit time-sortable post ID, around 1.8 Γ 10ΒΉβΈ, does not fit: IDs that differ by 1 become the same score. Use the post time in milliseconds as the score and the post_id:author_id string as the member.
Only active users get a timeline
On any given day most registered users do not open the app, and fanning out to them wastes work and RAM. Twitter has described skipping timeline inserts for users who have not logged in within the last 30 days. The rules:
- the fan-out writes only into timelines that already exist, and never creates one;
- a timeline's TTL is refreshed when its owner reads it, never when a followee posts;
- a missing timeline is rebuilt when its owner comes back.
Predict: a worker does ZADD and then sets a 7-day expiry on the follower's timeline for every fan-out. User U stopped opening the app in March but follows one account that posts daily. When does U's timeline expire? And what goes wrong if the worker creates missing timelines?
Check your answer
Never. Each daily post resets the 7-day clock, so the timeline of an inactive user lives forever. That is the opposite of the intent, and it quietly keeps every inactive but followed user in RAM.
If the worker creates missing keys, a returning user's timeline exists but holds only the posts made since it expired. The read path sees a key, treats it as warm and shows an almost empty feed, so the rebuild never runs. "The key exists" means "the timeline is warm" only if nothing but the rebuild creates keys, and only once that rebuild has finished (see below).
In Redis, "insert only if the timeline exists" has to be atomic, so it runs as a short Lua script:
-- KEYS[1] = timeline key; ARGV[1] = score (ms), ARGV[2] = member, ARGV[3] = cap
if redis.call('EXISTS', KEYS[1]) == 0 then
return 0 -- no timeline: inactive user, do not create one
end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2]) -- same member again = no-op
redis.call('ZREMRANGEBYRANK', KEYS[1], 0, -(tonumber(ARGV[3]) + 1)) -- keep newest cap
return 1
On Redis 7.4 this returns 0 for a missing key without creating it, and after five posts plus one redelivered post with a cap of 3, the timeline holds exactly the three newest posts. Note what it does not do: it never touches the TTL.
Rebuilding a returning user
A rebuild is a pull, done once: read the user's following list, take recent posts from each followee's author timeline, keep the newest 800, and write the timeline. Three rules keep it safe:
- Serve the first page from the merge itself. The rebuild has already computed it, so the user does not wait twice.
- Make the key exist before backfilling. Redis deletes a sorted set when its last member goes, so an "empty" timeline needs a placeholder member: score 0, skipped by every read, and evicted by the trim once real entries fill the cap. With the key in place, posts fanned out during the rebuild land in it instead of being skipped. A user whose followees have posted nothing still has a key, so they are not rebuilt on every read. Because inserts are idempotent, overlap between fan-out and backfill is harmless.
- One rebuild per user at a time, and a global rate limit. A per-user lock ("single flight") stops ten parallel requests from rebuilding ten times. While the lock is held the key exists but is half-filled, so readers check the lock alongside the timeline and wait for the rebuild's result instead of serving the partial key. If a cache shard is lost, a million users rebuild at once: a thundering herd onto the post and graph stores.
The other stores, briefly
- Post store, keyed by post ID. It serves hydration, 1.4 million lookups per second at peak, so a post cache (Memcached or Redis) sits in front of it.
- Author timelines, keyed by author: each author's recent post IDs, written by the Post service before it returns. They serve the big-account merge, the author's own posts and rebuilds.
- Graph store, with two adjacency lists per user: followers, used by the fan-out, and following, used by rebuilds and read-time filters, both sharded by user ID. A feed needs one-hop lookups, not graph traversal, so a general-purpose graph database buys you nothing here. Twitter has described its fan-out as querying a social-graph service built on Flock, which "maintains the follower and followings lists". A follower list with 100 million entries has to be split; in a wide-column store you add a bucket column to the partition key instead of putting it all in one partition, as the Cassandra data-modeling guide describes.
Move 4: Pagination that survives new posts
Pressure: the feed changes while the user is reading it.
Offset pagination says "skip 20, take 20". A feed grows mostly at the top, so every new post shifts the offsets under the reader:
feed = list(range(100, 90, -1)) # post ids, newest first: 100 .. 91
page1 = feed[0:5] # [100, 99, 98, 97, 96]
feed = [103, 102, 101] + feed # three new posts arrive at the top
offset_page2 = feed[5:10]
cursor_page2 = [p for p in feed if p < page1[-1]][:5]
print("offset:", offset_page2) # offset: [98, 97, 96, 95, 94]
print("cursor:", cursor_page2) # cursor: [95, 94, 93, 92, 91]
With offsets, page 2 repeats 98, 97 and 96. A deletion above the page boundary does the reverse and silently skips a post. A cursor says "20 posts older than the last one you showed me". Post IDs are unique and time-sortable, so "older than 96" means the same thing whatever arrives on top.
Checking for new posts needs more care. Because of fan-out lag, a post can be inserted seconds after a newer post the client already shows, so "newer than the first post I showed" would skip it for good. Instead, refetch the first page and let the client merge it with what it has, by post ID. A late post that lands below the first page appears only after a full refresh; accept that, and say so.
Make the cursor the post ID, or the pair (timestamp, post ID). A timestamp alone is ambiguous. When several posts share the boundary timestamp, "strictly older than" skips the ones not yet shown, and "older or equal" shows some twice. With the sorted set from Move 3, whose score is the post time in milliseconds, that means reading by score from the cursor's timestamp downwards and skipping entries whose ID is not older than the cursor's ID. Time-sortable IDs are covered in Core Building Blocks.
One cursor across several sources
The hybrid read has several sources: the reader's timeline, a few author timelines and the reader's own posts. Apply the same cursor to every source, merge newest first, filter, and stop at the page size:
import heapq
def read_page(sources, cursor=None, size=20, visible=lambda entry: True):
"""sources: lists of (post_id, author_id), each sorted newest first.
Returns up to `size` post ids older than `cursor`, and the next cursor."""
older = [(e for e in src if cursor is None or e[0] < cursor) for src in sources]
page, seen = [], set()
for entry in heapq.merge(*older, key=lambda e: e[0], reverse=True):
post_id = entry[0]
if post_id in seen or not visible(entry):
continue # pushed and pulled copy of one post, or filtered out
seen.add(post_id)
page.append(post_id)
if len(page) == size:
break
return page, (page[-1] if page else cursor)
timeline = [(96, 7), (94, 8), (93, 1), (90, 7)] # pushed; 93 arrived before @league crossed the threshold
league = [(97, 1), (95, 1), (93, 1), (91, 1)] # @league's author timeline, pulled at read
mine = [(98, 42)] # the reader's own posts
blocked = {8}
cursor = None
for _ in range(3):
page, cursor = read_page([timeline, league, mine], cursor, size=3,
visible=lambda e: e[1] not in blocked)
print(page, "next cursor:", cursor)
# [98, 97, 96] next cursor: 96
# [95, 93, 91] next cursor: 91
# [90] next cursor: 90
Trace the second page. Everything older than 96 is merged: 95 from league; 94 is dropped because author 8 is blocked; 93 appears in both sources and is kept once; then 91. The filter runs before the page is counted, so a filtered post does not leave a short page.
Pagination in a ranked feed
A ranked order is not sorted by post ID, so "older than X" no longer describes the next page, and re-ranking on every request would reshuffle posts the user has already scrolled past. A common approach: when a session starts (or on pull-to-refresh), rank one batch of candidates, say 500 post IDs. Store that ranked list server-side for the session, and page through it with a cursor of (session ID, position). New posts go into a "new posts" button at the top instead of reshuffling what is below. You give up freshness within a session and pay a little memory per active session: 500 IDs Γ 8 bytes is 4 KB.
Deep scroll. Past the 800th entry the timeline has nothing left. Either stop ("you're all caught up") or fall back to a slow pull for older posts. That is a product decision; say which one you chose.
Move 5: Ranking at interview depth
Pressure: the product wants the best posts first, not just the newest.
Keep the interview conversation at the level of stages:
candidates βββΊ features βββΊ score βββΊ filters and heuristics βββΊ page
timeline, affinity, model or blocked or muted, already seen,
big accounts, engagement, formula author diversity
recommended age
- Candidate retrieval. The timeline plus the big-account merge from Moves 1 and 2 is the in-network source: a few hundred candidates. A "For you" feed adds out-of-network sources, such as posts liked by people you follow.
- Features. Author affinity (how often you interact with this author), the post's engagement so far, its age and media type. Stream and batch jobs compute them ahead of time; the request fetches them in one batch.
- Score. A trained model at scale; a weighted formula on the whiteboard.
- Filters and heuristics. Visibility (blocked, muted, deleted), posts already seen, and no five posts in a row from the same author.
X's open-sourced home-timeline service has this shape: candidate sources, hydration of about 6,000 features, an ML ranker, then filters such as author diversity, removal of previously seen posts and visibility filtering for blocked and muted accounts (home-mixer README).
Rank at read time, precompute the inputs
Why not score each post once, at fan-out time, and store the score as the timeline's sort key? The worker does know the viewer at that moment. And with a pure exponential decay like the formula below, age alone would not reorder posts, because it multiplies every score by the same factor. Three things break the idea. The inputs keep changing after the insert: engagement accumulates for hours and affinity drifts. Real models do not treat age as a simple multiplier. And you would run the model on 694k inserts per second at peak, mostly for timelines nobody opens that day. So precompute the candidates, which is the timeline, and the features. The ranker runs on each request over a few hundred candidates within a latency budget. Give it a timeout, and if it misses, return the candidates in reverse-chronological order. A slightly worse feed beats an error page.
A whiteboard score, and its classic bug
score = (0.6 Γ affinity + 0.4 Γ engagement) Γ 0.5 ^ (age_hours / 6)
The last factor halves a post's score every 6 hours. The weights are placeholders you would tune with experiments, and you should say so.
The bug hides in engagement. The raw rate, likes Γ· impressions, is noisy for new posts: 1 like from 2 impressions is 0.5, which beats 2,000 likes from 10,000 impressions (0.2). Every new post with one lucky early like jumps to the top of the feed. The fix is to shrink the rate toward the platform average until the post has been seen enough times:
def smoothed_rate(likes, impressions, prior_rate=0.05, prior_weight=100):
# as if every post started with 100 impressions at the 5% average rate
return (likes + prior_rate * prior_weight) / (impressions + prior_weight)
for likes, shown in [(1, 2), (2_000, 10_000)]:
print(f"{likes:>5} of {shown:<6} raw {likes / shown:.3f} smoothed {smoothed_rate(likes, shown):.3f}")
# 1 of 2 raw 0.500 smoothed 0.059
# 2000 of 10000 raw 0.200 smoothed 0.199
A rate already ignores the absolute count, so a post with 10 million likes does not dominate because of its count. What distorts a rate is a small denominator. A logarithm compresses large values, but it does not fix a small sample: log(1 + 0.5) still beats log(1 + 0.2).
π‘ Say it like this: "Ranking runs at read time over a few hundred candidates, using features computed offline. It has a timeout, and on a miss I fall back to chronological order."
Likely follow-up: "How would you train the model?" Log impressions and engagements, train offline, and compare versions with A/B tests. The details belong to an ML design round; say so and bring the conversation back to the system.
Move 6: After fan-out, the world changes
Pressure: a timeline is a copy, and copies go stale.
| Event after fan-out | Enforce at read | Clean up later |
|---|---|---|
| Post deleted or edited | Hydration returns the current post; deleted ones are dropped | Optionally remove the ID from timelines |
| Reader unfollows an author | Drop entries whose author is neither the reader nor in the reader's following list | Remove that author's entries |
| Reader blocks or mutes an author | Drop entries by author ID; required, since blocking is a safety feature | Same cleanup job |
| Reader follows someone new | Backfill that author's recent posts into the timeline | |
| Author makes the account private | Hydration checks whether this reader may see the post |
The rule: the timeline is a list of candidates; the source of truth decides what is shown. Hydration reads the post store anyway, so deletes and edits cost nothing extra. Blocks and unfollows need the author ID in the entry, which is why Move 3 stored it. Background cleanup stops timelines from filling up with dead entries, but it lags, so it never replaces the read-time check.
β οΈ Do not build "recently deleted posts" as one Redis set with a TTL. Members of a Redis set cannot expire individually, so a TTL on the key forgets every entry at once. Checking deletion at hydration makes such a set unnecessary for correctness anyway.
Your turn: a user blocks an author at 10:00, and the cleanup job removes that author's 30 entries from the user's timeline at 10:07. What does the user see at 10:02 in each design: (a) cleanup only, or (b) cleanup plus a read-time filter?
Check your answer
(a) The blocked author's posts, for seven minutes. If the block was about harassment, that is an incident, not acceptable lag. (b) Nothing from that author: the read path drops entries by author ID using the reader's current block list, and the cleanup only reclaims space.
Keeping the feed live
An open app should show "3 new posts" without a manual refresh. Two cheap options:
- Poll by refetching the first page every 30 to 60 seconds while the feed is on screen, and show "3 new posts" for IDs the client has not seen. Simple and stateless; it costs requests.
- Signal over a connection the app already holds for messages or notifications: a tiny "new posts for you" message, after which the client refetches the first page.
Do not stream full posts to every online follower. A big account's post would turn into millions of pushes, carrying content that skipped ranking and visibility filters. Online followers get the timeline insert like everyone else; the signal is an extra, so a missed signal costs nothing. Networking Basics compares server-sent events, WebSockets and long polling, and Chat Application (WhatsApp) covers holding millions of persistent connections.
The whole design, and how it fails
WRITE client ββΊ Post service β1ββΊ Posts DB + outbox β3ββΊ queue ββΊ Fan-out workers
β β
ββ2ββΊ Author timeline, then 201 (worker re-appends)β
β
4. big account? yes: stop here β
no: page through its followers in the Graph store βββββββββ€
β
5. insert into each follower's timeline in the βββββββββββββββ
Timeline cache, only if that timeline exists
READ client ββΊ Feed service β1ββΊ Timeline cache (the reader's timeline)
β β2ββΊ Author timelines (big accounts via a 1 s in-process
β cache; the reader's own, direct)
β β3ββΊ Ranker (features precomputed)
β β4ββΊ Post cache ββΊ Posts DB (hydration + visibility)
βΌ
page + next cursor
| Event | What happens | Mitigation | What you give up |
|---|---|---|---|
| Timeline node dies; replica promoted | Inserts not yet replicated are lost, so a few posts are missing from some timelines | Accept it, or replay the last few minutes of fan-out events; idempotent inserts make replay safe | Replay work, or a few missing posts |
| A timeline shard is lost with no replica | Its users need rebuilds on their next read | Single-flight, rate-limited rebuilds; serve a pull page meanwhile | Slower first loads for those users |
| The fan-out queue backs up during a big event | Timelines fall behind the 5-second target | Alert on consumer lag, add workers, keep big jobs off the queue that small posts use | Freshness, temporarily |
| A worker crashes mid-job | The job is redelivered | Idempotent inserts; checkpoint the follower-page cursor | Some repeated work |
| A big account posts | Hot author-timeline key and hot post key | In-process caches with a 1-second TTL | 1 second of staleness |
| The ranker is slow or down | Feed latency blows the budget | Timeout, then chronological fallback | Relevance, briefly |
| Post saved, event never published | The post is never fanned out | Outbox or change data capture | A relay process to run |
Your turn: change one requirement
Decide before opening each answer: what changes, what you do, and what you pay.
1. Reads grow 10Γ; posts stay the same.
Check your answer
Push becomes even more attractive. Pull work would grow to 46M author reads per second on average, while the fan-out stays at 231k inserts per second, and timeline memory does not change at all. What grows is the read tier: 694k page loads per second at peak, 13.9M post fetches per second for hydration, and ten times the ranking compute. So you add timeline replicas for reads, grow the post cache and add ranking servers. Hot big-account keys stay at about one read per second per feed server, so their load grows with the servers you add, not tenfold with the readers.
2. The product launches in Europe, and each region must keep serving feeds if the other one is down.
Check your answer
Give each user a home region and keep their timeline there. Replicate the post log between regions asynchronously, and run fan-out in each region for the followers who live there. A US post reaches European followers after the cross-region replication lag plus local fan-out, so state a looser freshness target across regions. If a region fails, its users move to the other region, which holds no timelines for them. Either rebuild on demand (a burst of rate-limited rebuilds and slow first loads) or keep every timeline in both regions (double the timeline RAM, about 15 TB more with packed lists). That trade-off is the decision to name.
3. Posts become stories that disappear 24 hours after they are posted.
Check your answer
Fan-out is unchanged, but each entry now has its own expiry. Redis sorted-set members cannot expire individually, so use the post time as the score, filter "newer than now minus 24 hours" at read, and trim older entries with a delete by score range. A TTL on the key would be wrong: it would expire the whole timeline at once. Since nothing older than a day is useful, the cap can shrink a lot, and memory shrinks with it.
Final round: no label on the prompt
Real prompts don't say which move they need. Find the pressure first.
Challenge 1: the company feed
An internal social app for a company with 20,000 employees. About 3,000 use it daily, loading the feed 10 times each. People follow teams and colleagues. p99 under 500 ms.
Check your answer
Pull, using the SQL query from the start of this lesson with an index on posts(author_id, created_at). The load is 3,000 Γ 10 Γ· 86,400, about 0.35 feed loads per second, which one Postgres instance handles easily. Fan-out workers, timeline caches and a threshold would add cost and failure modes that no requirement asks for. Then say what would change your mind: daily users growing by orders of magnitude, or follow counts reaching the thousands.
Challenge 2: group feeds
Users join groups instead of following people, and the home feed shows posts from your groups, newest first. Most groups have under 1,000 members, but some have 5 million. A typical user is in 30 groups; some are in 300.
Check your answer
This is the celebrity problem with groups as the authors. Fan out posts from small groups into members' timelines. Don't fan out large groups; merge their recent posts at read time from a per-group timeline, cached in each feed server. Derive the size cutoff from your per-post fan-out budget, with hysteresis. The user in 300 groups is the "follows 3,000 big accounts" case: merge only the large groups they engage with most, or keep a periodically refreshed large-group timeline for such users. The costs: a merge on every read, and some completeness for rare heavy users.
Cheat sheet: pressure β move β cost
| Pressure | Move | Cost |
|---|---|---|
| Reads β« posts; pull scatter-gathers across shards | Fan-out on write into per-user timelines, asynchronously | Write amplification, RAM, seconds of lag |
| Authors with huge follower counts | Skip fan-out above a threshold derived from the fan-out budget, with hysteresis; merge author timelines at read | A read-time merge; hot keys |
| A hot author-timeline key | In-process cache with a short TTL | Seconds of staleness |
| Timelines Γ entries in RAM | IDs only, capped length, active users only; measure bytes per entry | Rebuilds; a deep-scroll limit |
| Inactive users | Fan-out never creates keys; TTL refreshed on the owner's read | Rebuild on return |
| At-least-once delivery | Idempotent inserts (sorted-set member = post) and checkpoints | 5 to 6 times the memory of a packed list |
| Posts arrive while the user scrolls | Cursor pagination by post ID; a per-session ranked list for ranked feeds | No random page access |
| Checking for new posts while fan-out lags | Refetch the first page and merge by post ID | Late posts below page 1 need a refresh |
| Best-first feed | Candidates β features β ranker at read time, timeout β chronological | A latency budget |
| Deletes, blocks and unfollows after fan-out | Filter at read, clean up asynchronously | Read-time work |
| Author must see their own post | Post service appends to the author timeline before the 201; every read merges the reader's own | One synchronous write; one more key per read |
Before moving on, explain the design aloud as you would in an interview, from a blank whiteboard, in about five minutes: the questions you asked, the numbers, why push beats pull for this workload, where push breaks and how the hybrid fixes it, what a timeline entry holds and how much RAM the tier takes, how a page is read with a cursor, and what happens when a big account posts during a final. Then ask a friend to throw this lesson's follow-up questions at you. Mock Interviews & Communication has a rubric for that.
Next: Chat Application (WhatsApp) delivers messages to specific devices over persistent connections, and YouTube / Netflix Streaming covers the media pipeline this lesson parked.