Performance Optimization

The answer that arrived all at once

A biomedical research team gets an assistant over 5,183 scientific abstracts, the SciFact collection from the BEIR benchmark. A researcher types a claim, and the assistant answers from the abstracts, with citations. The pipeline is the one earlier lessons build:

claim ─► embed (MiniLM, CPU) ─► HNSW, top 100 ─► cross-encoder reranks 100 ─► top 8 ─► LLM ─► answer

The product promise fits in one sentence: the first words of an answer appear within 1.5 seconds for 95 of every 100 questions. Retrieval quality passed its evaluation. Then the first traces came in. Here is one, for the claim "Individuals with low serum vitamin D concentrations have increased risk of multiple sclerosis.":

What should have come back: the first words by 1.5 s. What came back: a spinner for 5.3 seconds, then the whole answer at once, because the app waits for the complete answer before it shows anything.

Predict first: you may change one thing. Which gets this query's first word closest to 1.5 s: a 10× faster vector index, streaming the answer as it is written, or reranking 30 candidates instead of 100?

Three habits produced that trace. Every stage sits on the path to the first word. The app shows nothing until the last word exists. And nobody had measured where the time went, so the team's first plan was to tune the vector index, a stage that takes about a millisecond.

This lesson is about keeping a latency promise without giving up retrieval quality or blowing the budget. Each move starts from a symptom you can see in a trace or on a dashboard:

The lesson assumes the pipeline from Classic RAG Pipeline and the budget rules from Foundations of Modern AI Search. Skipping work with caches has its own lesson, Caching Strategies. SLOs, rollouts and capacity planning live in Production Deployment & Optimization.

Before any move: two clocks and a budget

Two numbers describe how long a user waits, and different fixes move them:

• Time to first token (TTFT): from pressing Enter to the first word of the answer on screen. Everything before the LLM counts, plus the model's own delay before its first token.
• Time to full answer: until the last word. It adds the time the model spends writing, which grows with the length of the answer.

A latency budget splits a target into allowances per stage, at a stated percentile. Foundations of Modern AI Search built one and gave the rules: parallel stages cost the slowest of them, sequential stages cost the sum, and adding up stage p95s gives a plan, not a guarantee.

Percentiles count requests, not people. A p95 of 2 s means 95% of requests finish within 2 s. A researcher who asks 20 questions in a session meets at least one slower answer with probability 1 − 0.95**20 ≈ 0.64. Most sessions see the tail.

Where do targets come from? Jakob Nielsen's three response-time limits (1993, drawing on Miller 1968 and Card et al. 1991) are a good start: about 0.1 s feels instantaneous, about 1 s keeps the user's flow of thought, and about 10 s is the limit for holding attention. A cited answer that takes 6 s to write is fine if words start within about a second. Six seconds of spinner is not.

Four questions to ask before you touch the pipeline:

💡 Habit worth building: before optimizing anything, write the promise as "clock, percentile, target, load", for example "first token, p95, 1.5 s, at 20 queries per second". Half of all latency arguments are two people using different clocks.

Move 1: Measure the distribution, stage by stage

Symptom: the dashboard's average looks acceptable, and users still say the assistant is slow.

The code below runs the pipeline over SciFact's 300 test claims and times each stage. Everything before the LLM runs for real: MiniLM on the CPU, an hnswlib HNSW index, and the MiniLM cross-encoder from Reranking Models. First, the setup (about a minute to embed the abstracts on a laptop CPU):

import csv, io, json, os, time, urllib.request, zipfile
import numpy as np
import hnswlib
from sentence_transformers import SentenceTransformer, CrossEncoder

URL = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip"
if not os.path.exists("scifact"):
    zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(URL).read())).extractall(".")

corpus = [json.loads(line) for line in open("scifact/corpus.jsonl")]
TEXTS = [d["title"] + ". " + d["text"] for d in corpus]                  # 5,183 abstracts
queries = {q["_id"]: q["text"] for q in map(json.loads, open("scifact/queries.jsonl"))}
test_ids = sorted({r["query-id"] for r in csv.DictReader(open("scifact/qrels/test.tsv"),
                                                          delimiter="\t")}, key=int)
QUESTIONS = [queries[i] for i in test_ids]                                # 300 claims

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cpu")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", device="cpu")
TOKENS = [len(embedder.tokenizer(t, verbose=False)["input_ids"]) for t in TEXTS]

X = embedder.encode(TEXTS, batch_size=64)             # unit-length 384-d rows; about a minute on CPU
index = hnswlib.Index(space="ip", dim=X.shape[1])
index.init_index(max_elements=len(X), M=16, ef_construction=200, random_seed=0)
index.add_items(X)
index.set_ef(128)                                     # query-time efSearch

The LLM stage is modeled, so the block runs without an API key or a GPU. Its first token arrives after a fixed wait plus the time to read the prompt, and then it writes at a fixed speed. Answer lengths come from a skewed distribution with a median of 180 tokens. All of these are assumptions, stated in the first lines; Move 6 shows how to measure a real model's numbers.

# The LLM stage is modeled, not run. Assumed hosted model (round numbers; measure your own):
LLM_WAIT_S = 0.30        # network and queueing before the model starts reading the prompt
PREFILL_TOK_S = 10_000   # prompt tokens read per second (prefill)
DECODE_TOK_S = 100       # answer tokens written per second (decode)
PROMPT_BASE = 250        # instructions + question, in tokens
rng = np.random.default_rng(7)
ANSWER_TOKENS = np.clip(rng.lognormal(np.log(180), 0.5, len(QUESTIONS)), 40, 600).astype(int)

def answer(i, pool=100, k_context=8):
    """One query through the pipeline; returns seconds per stage."""
    question, t = QUESTIONS[i], {}
    t0 = time.perf_counter()
    qv = embedder.encode(question)
    t1 = time.perf_counter()
    ids, _ = index.knn_query(qv, k=pool)
    t2 = time.perf_counter()
    scores = reranker.predict([(question, TEXTS[j]) for j in ids[0]], batch_size=32)
    top = ids[0][np.argsort(-scores)[:k_context]]
    t3 = time.perf_counter()
    prompt_tokens = PROMPT_BASE + sum(TOKENS[j] for j in top)
    return {"embed": t1 - t0, "ann": t2 - t1, "rerank": t3 - t2,
            "llm_ttft": LLM_WAIT_S + prompt_tokens / PREFILL_TOK_S,
            "llm_decode": ANSWER_TOKENS[i] / DECODE_TOK_S}

for i in range(5):                      # warm-up: first calls pay one-off costs
    answer(i)
runs = [answer(i) for i in range(len(QUESTIONS))]
T = {s: 1000 * np.array([r[s] for r in runs]) for s in runs[0]}      # milliseconds
T["first token"] = T["embed"] + T["ann"] + T["rerank"] + T["llm_ttft"]
T["full answer"] = T["first token"] + T["llm_decode"]

print(f"{'ms':12}{'p50':>7}{'p95':>7}{'p99':>7}{'mean':>7}")
for stage, v in T.items():
    print(f"{stage:12}" + "".join(f"{x:7.0f}" for x in (*np.percentile(v, [50, 95, 99]), v.mean())))

One run printed:

ms              p50    p95    p99   mean
embed             5      7     21      6
ann               1      1      1      1
rerank         1143   1418   4095   1227
llm_ttft        619    734    833    628
llm_decode     1735   3554   4603   1864
first token    1778   2170   4750   1861
full answer    3486   5562   6941   3725

These numbers come from an Apple M2 Max laptop CPU that was running other jobs (load average between 3 and 7 on 12 cores). An earlier run, with the machine far busier, measured a reranker median of 2,778 ms instead of 1,143 ms: the same code and data, 2.4 times slower. Compare rows with each other, and measure on the hardware you will deploy.

Read the table from the bottom up:

1. The first word waits for the whole answer. The app shows nothing until the end, so the researcher waits for the full-answer column: a median of 3.5 s and a p95 of 5.6 s. The promise is broken for nearly every question, not just the tail.
2. Before the LLM, the reranker is nearly all the time. Its median is 1,143 ms, about 11 ms per (claim, abstract) pair. Embedding the claim takes 5 ms and the HNSW search 1 ms, so of the roughly 1,150 ms before the LLM starts, 99% is reranking.
3. The mean hides the tail. The reranker's mean (1,227 ms) sits close to its median, yet its p99 is 4,095 ms: a few requests hit a stall, most likely other jobs taking the CPU. A dashboard that shows only the mean or the median never sees the requests that took four seconds to rerank.
4. Below the p99, the reranker's spread is mostly the data. Each batch of 32 pairs is padded to its longest pair, and some claims pull in pools of long abstracts: the pool at the p95 carries about 16% more padded tokens than the median pool. In a rerun on the first 100 claims that kept each query's fastest of three tries, which strips out most of the machine's noise, rerank time followed the padded tokens almost exactly (correlation 0.97), and its p95 was still 1.2 times its median, close to the 1,418 / 1,143 = 1.24 in the table. The p99 is the machine (item 3). The writing time's spread is the data too: long answers take longer to write.

Percentiles do not add up. Continuing in the same session:

stages = ["embed", "ann", "rerank", "llm_ttft"]
print("sum of stage p95s:", round(sum(np.percentile(T[s], 95) for s in stages)), "ms")
print("p95 of the sum:   ", round(np.percentile(T["first token"], 95)), "ms")
boot = np.random.default_rng(0).choice(T["first token"], (2000, len(runs)))
print("p99 from 300 queries: 95% interval",
      np.percentile(np.percentile(boot, 99, axis=1), [2.5, 97.5]).round(), "ms")

sum of stage p95s: 2160 ms
p95 of the sum:    2170 ms
p99 from 300 queries: 95% interval [2300. 5448.] ms

Here the sum of the stage p95s (2,160 ms) lands close to the p95 of the total (2,170 ms), because one stage dominates. With several stages of similar size, the sum can miss in either direction:

• Everyday spread: the sum overestimates. When each stage's slow moments are its own, they rarely coincide. Four independent stages with a median of 100 ms and a p95 of 164 ms each add up to 656 ms, yet the p95 of their total is about 530 ms.
• Rare stalls: the sum can underestimate. Say four stages take 100 ms, or 400 ms on 2% of requests, independently. Each stage's p95 is 100 ms, so the sum says 400 ms. But 1 − 0.98**4 ≈ 7.8% of requests hit at least one stall, so the p95 of the total is 700 ms. Stalls too rare to show in any one stage's p95 add up to more than 5% of requests.
• A shared cause pulls the two together. If one cause, such as a noisy neighbour on the machine, stalls all four stages on the same 2% of requests, the p95 of the total is exactly 400 ms, the sum. Stages in perfect lockstep, each at the same percentile of its own times on every request, always add up exactly. With independent noise on top of a shared cause, the sum can still miss either way.

The sum is a planning number. The percentile you promise must be measured end to end.

Small samples make shaky tails. With 300 queries, the p99 rests on the three or four slowest. The bootstrap interval above says a rerun could plausibly report anything in that range. Collect thousands of requests before you trust a p99, and compare p99s only across runs of similar size.

Never average percentiles. A dashboard that averages the per-server p95s, or the per-minute p95s over an hour, shows a number that is not the p95 of anything. One hot server can hold most of the slow requests and still be averaged away. Aggregate raw samples, or histograms with fixed buckets that merge exactly, and compute the percentile from the merged data.

In production, record every request's per-stage timings as histogram metrics, and keep full traces (one span per stage) for a sample of requests that deliberately includes the slow ones. Give the spans the attributes that explain the time: pool size, prompt tokens, answer tokens, model id, cache hit, index version. OpenTelemetry's GenAI conventions name the LLM's clocks, for example gen_ai.client.operation.time_to_first_chunk for streaming calls and gen_ai.server.time_to_first_token on the serving side; they were still marked "Development" in September 2026 (conventions). Tracing and logging in depth are part of Infrastructure & Security.

Three measurement habits:

• Measure where the user is. Server spans miss DNS, TLS setup, the network, proxies and the browser. Record time to first token on the client as well.
• Keep warm-up out of steady state, and measure it separately. The loop above discards five warm-up queries. Cold starts are real latency, just a different kind (Move 7).
• Gate releases on latency too. Replay the labeled question set in CI next to the quality check, and fail the build when the p95 of either clock regresses by more than a margin you chose in advance.

Your turn. The p95 of time to first token is 1.4 s, inside the promise. A typical session has 12 questions. What share of sessions contain at least one first word slower than 1.4 s? What would you add to the dashboard?

Move 2: Stream, and budget the first token

Symptom: a spinner, then the whole answer at once.

With streaming, the app shows each token as the model writes it. In the table, the researcher's wait moves from the full-answer row to the first-token row: a median of 3,486 ms becomes 1,778 ms, and the p95 falls from 5,562 ms to 2,170 ms. The model writes the same tokens at the same speed, so the total time and the bill do not change. What changes is what the user waits for.

Streaming has costs of its own:

• Checks that need the whole answer run afterwards. Classic RAG Pipeline checks citations and refusals on the complete text. Either stream and withdraw or flag an answer that fails, or buffer when an unchecked answer must never be shown (Grounding & Hallucination Control chooses per risk tier).
• Every hop must pass the chunks through. nginx buffers proxied responses by default (proxy_buffering on). For a streaming route, switch it off, or send the response header X-Accel-Buffering: no (nginx docs). Gateways and compression middleware can buffer too. The classic symptom: streaming works on your laptop and arrives in one piece in production.
• Reasoning models think before they write. Thinking tokens come before the first visible word, so they count toward time to first token. Some current models cannot switch thinking off: Anthropic's Claude Opus 5.5, for example, rejects requests that disable it and decides per request how much to think (adaptive thinking). Its effort setting, medium by default, is the control (checked September 2026). On a latency-sensitive route, lower the effort or pick a model that answers directly, show progress ("Searching 5,183 abstracts…"), and time the first text token, not the first event on the stream.

Measure time to first token on the stream itself. This helper times any iterator of text pieces:

def time_stream(pieces):
    """Time any iterator of text pieces: (s to first non-empty piece, s to the last, full text)."""
    t0, first, parts = time.perf_counter(), None, []
    for piece in pieces:
        if piece and first is None:
            first = time.perf_counter() - t0
        parts.append(piece)
    return first, time.perf_counter() - t0, "".join(parts)

With a hosted model, the same helper wraps the SDK's stream. A sketch with Anthropic's Python SDK (it needs an API key, so it is not part of the runnable session):

import anthropic

MODEL = "claude-haiku-4-5-20251001"   # Haiku 4.5, an example id as of September 2026
client = anthropic.Anthropic()        # reads ANTHROPIC_API_KEY from the environment

def claude_stream(context, question):
    with client.messages.stream(
        model=MODEL, max_tokens=1024,
        system="Answer from the sources. Cite them like [S1].",
        messages=[{"role": "user", "content": f"{context}\n\nQuestion: {question}"}],
    ) as stream:
        yield from stream.text_stream   # text only, so thinking time counts toward ttft

ttft, total, text = time_stream(claude_stream(context, question))   # context: the chosen abstracts

Now budget the first-token clock. The promise is 1.5 s at p95. The modeled LLM needs 734 ms to its first token at p95, which leaves about 770 ms for everything before it. Embedding and search take under 10 ms. The reranker's p95 is 1,418 ms, almost twice the whole allowance. Streaming was necessary, and it is not enough.

Your turn. Product adds a second promise: the full answer within 8 s at p95. Which stages count toward it, and does streaming help? Does the current pipeline keep it?

Move 3: Cut the biggest slice first

Symptom: two weeks spent making a stage faster, and the dashboard does not move.

Amdahl's law: if a stage takes a fraction p of the time and you make it s times faster, the whole becomes 1 / ((1 − p) + p / s) times faster, and you save p × (1 − 1/s) of the total. Here it is on the first-token clock, with the medians from Move 1 typed in:

median_ms = {"embed": 5, "ann": 1, "rerank": 1143, "llm_ttft": 619}   # Move 1
total = sum(median_ms.values())         # sum of stage medians: a typical wait for the first token
for stage, s in [("ann", 10), ("embed", 2), ("llm_ttft", 2), ("rerank", 3)]:
    new = total - median_ms[stage] * (1 - 1 / s)
    print(f"{stage:8} {s:>2}x faster: {total:.0f} -> {new:.0f} ms, "
          f"{1 - new / total:5.1%} lower, speedup {total / new:.2f}x")

ann      10x faster: 1768 -> 1767 ms,  0.1% lower, speedup 1.00x
embed     2x faster: 1768 -> 1766 ms,  0.1% lower, speedup 1.00x
llm_ttft  2x faster: 1768 -> 1458 ms, 17.5% lower, speedup 1.21x
rerank    3x faster: 1768 -> 1006 ms, 43.1% lower, speedup 1.76x

The 10× faster index, the plan from the hook, saves under a millisecond. Halving the LLM's time to first token saves 17.5%. Cutting the reranker's time by 3×, close to the 2.8× that a pool of 30 instead of 100 buys in Move 4, saves 43%. The rule: rank stages by their share of the clock you promised, and start at the top.

Keep speedup and latency reduction apart. They are different numbers for the same change. Make a stage that is 80% of the time twice as fast: the speedup is 1 / (0.2 + 0.8/2) ≈ 1.67×, and the latency falls by 0.8 × (1 − 1/2) = 40%, not 67%.

The bottleneck depends on the clock. On the full-answer clock, writing the answer (median 1,735 ms of 3,486 ms here) is the largest slice. On the first-token clock it does not count at all, once you stream. Shorter answers or a faster output tier help the full-answer clock and do nothing for the first-token clock; the reranker sits on both. Choose the clock (Move 2) before you choose the stage, and measure again after every fix, because the bottleneck moves.

Your turn. Another team's median trace: embed 50 ms, vector search 150 ms, fetch chunk text 60 ms, rerank 80 ms, assemble the prompt 70 ms, LLM 1,100 ms, 1,510 ms in total, all sequential. Team A proposes a 10× faster search. Team B proposes a model that is 2× faster on the LLM part. What does each save, and what must Team B prove?

Move 4: Rerank fewer pairs, or cheaper ones

Symptom: the reranker is the largest stage before the first token.

Reranking Models gave the estimate: reranker latency ≈ pairs × tokens per pair ÷ tokens per second. Three factors, three levers: the pool size, the pair length and the hardware. Measure each on the session from Move 1:

import torch

def rerank_ms(model, question, ids):
    pairs = [(question, TEXTS[j]) for j in ids]
    t0 = time.perf_counter()
    model.predict(pairs, batch_size=32)
    return 1000 * (time.perf_counter() - t0)

gpu = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else None
on_gpu = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", device=gpu) if gpu else None
settings = {                          # name: (model, pool size, max tokens per pair)
    "CPU, pool 10": (reranker, 10, 512),
    "CPU, pool 30": (reranker, 30, 512),
    "CPU, pool 100": (reranker, 100, 512),
    "CPU, pool 30, 256 tokens": (reranker, 30, 256),
}
if on_gpu:
    settings[f"{gpu.upper()}, pool 100"] = (on_gpu, 100, 512)
    on_gpu.predict([(QUESTIONS[0], TEXTS[0])])                      # warm-up

times = {name: [] for name in settings}
for question in QUESTIONS[:50]:
    ids, _ = index.knn_query(embedder.encode(question), k=100)
    for name, (model, pool, max_tokens) in settings.items():      # interleaved, so load drift
        model.max_seq_length = max_tokens                          # hits every setting alike
        times[name].append(rerank_ms(model, question, ids[0][:pool]))
reranker.max_seq_length = 512
for name, t in times.items():
    print(f"{name:25} p50 {np.percentile(t, 50):5.0f}   p95 {np.percentile(t, 95):5.0f} ms")

CPU, pool 10              p50   177   p95   216 ms
CPU, pool 30              p50   451   p95   526 ms
CPU, pool 100             p50  1251   p95  1551 ms
CPU, pool 30, 256 tokens  p50   197   p95   251 ms
MPS, pool 100             p50   223   p95   245 ms

This output comes from a later session on the same laptop than Move 1's table, with the load average between 7 and 9. Another run under similar load printed medians of 157, 435, 1,193, 190 and 226 ms, and a GPU p95 of 357 ms: on a shared machine, the tail moves from run to run.

Time grows with the pool, a little less than in proportion: 177, 451 and 1,251 ms at the median for 10, 30 and 100 pairs. Capping pairs at 256 tokens instead of 512 cut the pool-30 median from 451 to 197 ms. Each batch is padded to its longest pair, so the cap roughly halves the tokens actually processed, and attention's cost grows faster than linearly with length. The laptop's GPU reranked all 100 pairs in 223 ms, faster than its CPU reranked 30.

Now the quality side. A faster reranker that ranks worse is not a win. Reranking Models measured this exact pair of models on SciFact, with exact search as the first stage: nDCG@10 of 0.682, 0.689, 0.692, 0.688 and 0.687 for pools of 10, 20, 30, 50 and 100. Re-running it with this lesson's HNSW index gave 0.682, 0.688, 0.692, 0.687 and 0.690, against 0.648 with no reranker, while first-stage recall climbed from 0.788 at 10 to 0.928 at 100. (SciFact's labels are binary, so every relevant abstract in the top 10 earns 1 / log2(rank + 1), and the ideal ordering puts all of a claim's labeled abstracts first. Recall here is the share of a claim's labeled abstracts in the pool, averaged over the 300 claims.) Pools of 20 to 100 all score within 0.005 of each other: on this data, a pool of 30 buys the quality of 100 for about a third of the pairs. The recall the bigger pool adds does not survive reranking here, which is the same finding Reranking Models reported.

That is a measurement for this model on this corpus, not a rule. Pools of 50 to 200 are common, and on data where first-stage recall keeps paying past 30 candidates you keep the bigger pool and buy speed elsewhere. The reranker cannot promote a chunk that never reached the pool.

⚠️ Over-retrieval means the prompt, not the pool. Reranking 100 candidates and putting the best 8 in the prompt is the design this course recommends. Putting 100 chunks into the prompt is the mistake: each one costs prefill time and input tokens (Move 6), and buries the evidence.

Truncation has a quality price. At 256 tokens the reranker no longer sees the end of longer abstracts. On SciFact, a pool of 30 at 256 tokens scored nDCG@10 0.686 against 0.692 at 512: a small loss here. When the evidence sits late in a chunk it can be large (Reranking Models, "Mind the length limit"), so measure it on your data.

More levers, each with its price:

• A GPU. On this laptop the GPU (device="mps") reranked a pool of 100 at a p95 of 245 ms (357 ms in another run), against 1,551 ms on the CPU. A server GPU is faster still. The price is the hardware, and idle capacity at quiet hours.
• A smaller cross-encoder. The MiniLM cross-encoder card lists the 2-layer model at 4,100 pairs per second against 1,800 for the 6-layer one, measured on a V100 GPU with short MS MARCO passages, not on a CPU like the timings above. The card gives NDCG@10 on TREC DL 2019 of 71.01 against 74.30 (Reranking Models).
• Skip reranking when the first stage is confident, for example for exact identifier matches. It saves money and mean latency, but little at the p95. As long as more than 5% of queries are reranked, the p95 request is still a reranked one; it only moves down within them (with 70% reranked, it is the reranked queries' own p93). On Move 1's run, skipping a random 30% of queries cuts the mean time to first token from 1,861 to about 1,490 ms, and the p95 only from 2,170 to about 2,120 ms.
• A timeout with a fallback. If the reranker has not answered in time, use the first-stage order. You lose some quality on those queries instead of breaking the promise.
• Batching across concurrent queries raises throughput per GPU, not the speed of one query (Move 7).

Your turn. The assistant now streams. The promise is still first words within 1.5 s at p95; the modeled LLM takes 734 ms at p95, and embedding plus search under 10 ms. Using the two tables above, which reranker settings fit, and which would you ship?

Move 5: Overlap what does not depend

Symptom: the trace shows stages that do not use each other's output waiting in line.

Only a dependency forces an order. Lexical search needs the query text and the user's permission filter; it does not need the embedding. Write the pipeline as stages with dependencies and compute the critical path, the chain of stages that sets the finish time:

from functools import cache

# stage: (milliseconds, stages it must wait for)
PLAN = {
    "understand": (2,   []),
    "acl":        (15,  ["understand"]),          # fetch the user's groups for the filter
    "embed":      (30,  ["understand"]),
    "dense":      (8,   ["embed", "acl"]),
    "bm25":       (25,  ["acl"]),                 # needs the filter, not the embedding
    "fuse":       (2,   ["dense", "bm25"]),
    "rerank":     (150, ["fuse"]),
    "llm_ttft":   (600, ["rerank"]),
}

@cache
def finish(stage):
    ms, deps = PLAN[stage]
    return ms + max((finish(d) for d in deps), default=0)

print("one after another:", sum(ms for ms, _ in PLAN.values()), "ms")
print("dependencies only:", finish("llm_ttft"), "ms")
path, s = [], "llm_ttft"
while s:
    path.append(s)
    deps = PLAN[s][1]
    s = max(deps, key=finish) if deps else None
print("critical path:", " <- ".join(path))

one after another: 832 ms
dependencies only: 794 ms
critical path: llm_ttft <- rerank <- fuse <- bm25 <- acl <- understand

Running everything as early as its inputs allow saved 38 ms of 832, under 5%. Reranking and the LLM sit on every path, so no amount of overlap removes them (Move 3 again). Overlap is still worth doing because it is nearly free. And the critical path tells you which stage to shorten next: making embed faster here saves nothing, because the BM25 branch finishes later.

Overlaps that usually pay in a RAG service:

• The embedding and the permission lookup at once, then lexical and dense search in parallel, each with the permission filter applied inside the search. The fusion step waits for the slower list (Hybrid Retrieval Systems). Do not start the searches before the filter is ready and drop forbidden hits afterwards: that is post-filtering, and it starves the results (Metadata & Filtering).
• Retrieval on the raw message while a rewriter runs. If the rewrite comes back unchanged, the results are already there. The price is extra retrieval load (Query Understanding & Intent).
• Independent tool calls in one agent turn, each with its own timeout (Agentic RAG Systems, Move 3).

Async buys throughput, not a shorter chain

async lets one process keep many requests in flight while each waits on the network. It shortens a single request only when that request has independent waits to overlap. One blocking call hidden inside an async service does the opposite:

import asyncio, time

def search_blocking(query):
    time.sleep(0.2)                   # a synchronous client call: 200 ms of network wait
    return [query]

async def handler_bad(query):
    return search_blocking(query)     # blocks the event loop; nobody else can run

async def handler_good(query):
    return await asyncio.to_thread(search_blocking, query)   # waits in a worker thread

async def serve(handler, n=10):
    t0 = time.perf_counter()
    await asyncio.gather(*(handler(f"q{i}") for i in range(n)))
    return time.perf_counter() - t0

for handler in (handler_bad, handler_good):
    print(f"{handler.__name__}: 10 concurrent requests took {asyncio.run(serve(handler)):.1f} s")

handler_bad: 10 concurrent requests took 2.1 s
handler_good: 10 concurrent requests took 0.2 s

The blocking handler served the ten requests one after another: ten 0.2 s calls in a row, about 2 s for the last one (the extra 0.1 s is sleep overshoot on a busy machine). The other handler let the ten waits overlap. Neither made any request faster than 0.2 s. A synchronous database driver, requests.get, or a slow tokenizer called inside an async handler adds its time to every request in flight. The trace shows it as mysteriously long gaps that grow with traffic.

Hidden round trips add up the same way. Fetch the text of the 8 chosen chunks in one call, not one query per chunk. Reuse HTTP clients and connection pools, because a new connection per request pays DNS, TCP and TLS handshakes every time. Keep the retriever, the reranker and the LLM endpoint in one region: a cross-region round trip costs tens of milliseconds, on every call.

Threads help with waiting, not with Python computation. CPU-bound pure-Python code, such as a pure-Python BM25 scorer, holds the interpreter lock, so two threads running it take turns. Move that work to a separate process or service, or use native code that releases the lock while it computes.

Your turn. The permission lookup starts taking 120 ms because the directory service is slow. Recompute the plan. What is the critical path now, and what do you change?

Move 6: Make the model quicker to wait for

Symptom: everything before the LLM fits its budget, and the model's own delay and writing time dominate both clocks.

An LLM call has two phases:

• Prefill reads the whole prompt, processing its tokens in parallel, before the first output token. Its time grows with the prompt.
• Decode writes the answer one token at a time, because each token depends on the ones before it. For every new token the hardware reads the model's weights and the growing key-value cache (the attention state of the prompt and of the tokens written so far), so decode speed is usually limited by memory bandwidth, not arithmetic. Writing time ≈ answer tokens ÷ tokens per second.

Measure both on a local model, continuing the session. This uses Qwen3-1.7B, the small open-weight model from Classic RAG Pipeline, and prompts built from 1, 8, 16 and 32 retrieved abstracts:

import threading
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

LLM_ID = "Qwen/Qwen3-1.7B"     # example small open-weight model (about 4 GB); any chat model works
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
llm_tok = AutoTokenizer.from_pretrained(LLM_ID)
llm = AutoModelForCausalLM.from_pretrained(LLM_ID, dtype=torch.bfloat16).to(device)

def chat_inputs(context, question):
    messages = [{"role": "system", "content": "Answer from the sources. Cite them like [S1]."},
                {"role": "user", "content": f"{context}\n\nQuestion: {question}"}]
    prompt = llm_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                         enable_thinking=False)
    return llm_tok(prompt, return_tensors="pt").to(device)

def local_stream(inputs, max_new_tokens=128):
    """Start generating in a background thread; yield text pieces as they arrive."""
    streamer = TextIteratorStreamer(llm_tok, skip_prompt=True, skip_special_tokens=True)
    threading.Thread(target=llm.generate, kwargs=dict(**inputs, streamer=streamer,
                     do_sample=False, max_new_tokens=max_new_tokens)).start()
    yield from streamer

question = QUESTIONS[0]
ids, _ = index.knn_query(embedder.encode(question), k=32)
def context(k):
    return "\n\n".join(f"[S{n + 1}] {TEXTS[j]}" for n, j in enumerate(ids[0][:k]))

time_stream(local_stream(chat_inputs(context(8), question), max_new_tokens=8))   # warm-up
for k in (1, 8, 16, 32):
    inputs = chat_inputs(context(k), question)
    runs = []
    for _ in range(3):                                  # median of 3: the laptop is noisy
        ttft, total, text = time_stream(local_stream(inputs, max_new_tokens=64))
        out = len(llm_tok(text)["input_ids"])
        runs.append((ttft, (out - 1) / (total - ttft)))
    ttft, speed = np.median(runs, axis=0)
    print(f"{k:2} chunks, {inputs['input_ids'].shape[1]:5} prompt tokens: "
          f"first token {ttft:4.2f} s, then {speed:4.1f} tokens/s")

 1 chunks,   170 prompt tokens: first token 0.19 s, then 44.2 tokens/s
 8 chunks,  1720 prompt tokens: first token 1.05 s, then 41.5 tokens/s
16 chunks,  3240 prompt tokens: first token 2.00 s, then 39.8 tokens/s
32 chunks,  7150 prompt tokens: first token 4.88 s, then 35.4 tokens/s

Two things to read. The first token grows roughly in proportion to the prompt: from 0.19 s at 170 tokens to 4.88 s at 7,150, about 1,500 prompt tokens per second on this laptop. The writing speed changes much less (44 down to 35 tokens per second): each new token reads a longer key-value cache, so it slows a little. A hosted model is much faster at both. The shape is the same, and it is why Move 1 models the LLM with a prefill rate and a writing rate.

The levers, and which clock each one moves:

A few notes on the table:

• Anthropic's latency guide recommends the first three levers and streaming. An answer that hits max_tokens is cut off, so check the stop reason before showing it (Classic RAG Pipeline).
• As a dated example of a faster tier, Anthropic's fast mode (a research preview in September 2026) gives "up to 2.5x higher output tokens per second" on Claude Opus 5.5 at twice the price. Its docs say the gain is in output speed, not time to first token, so it helps the full-answer clock only.
• On your own GPUs, a serving engine such as vLLM provides continuous batching, chunked prefill, prefix caching and speculative decoding. In speculative decoding, a small draft model proposes several tokens and the large model checks them in one pass. The output follows the large model's own distribution, so quality does not change, and Leviathan et al. (ICML 2023) reported 2–3× faster generation on T5-XXL. Quantization shrinks the weights that every decode step must read, and it can cost accuracy: rerun your evaluation.

Routing: the average gains more than the p95

Send easy questions to a small, fast model and the rest to a large one. Here is what that does to both clocks and the bill, with two assumed model profiles and 3,000-token prompts:

import numpy as np

# Two assumed hosted models. Prices: Anthropic list prices checked September 2026 for
# Claude Opus 5.5 (L) and Claude Haiku 4.5 (S), USD per million input / output tokens.
# Speeds are round-number assumptions, not measurements of those models.
MODELS = {"L": dict(wait=0.30, prefill=10_000, decode=100, usd_in=4.0, usd_out=20.0),
          "S": dict(wait=0.20, prefill=20_000, decode=200, usd_in=1.0, usd_out=5.0)}
PROMPT_TOKENS = 3_000
rng = np.random.default_rng(7)
answer_tokens = np.clip(rng.lognormal(np.log(180), 0.5, 10_000), 40, 600)

def run(share_small, pre_llm=0.35):          # assumed seconds spent before the LLM call
    small = rng.random(answer_tokens.size) < share_small
    ttft, full, usd = [], [], []
    for is_small, out in zip(small, answer_tokens):
        m = MODELS["S" if is_small else "L"]
        first = pre_llm + m["wait"] + PROMPT_TOKENS / m["prefill"]
        ttft.append(first)
        full.append(first + out / m["decode"])
        usd.append((PROMPT_TOKENS * m["usd_in"] + out * m["usd_out"]) / 1e6)
    return np.array(ttft), np.array(full), 1000 * np.mean(usd)

print("small share  TTFT p50  p95   full p50  p95   USD/1k")
for share in (0.0, 0.7, 0.9, 0.97):
    ttft, full, usd = run(share)
    print(f"{share:10.0%}  {np.percentile(ttft, 50):8.2f} {np.percentile(ttft, 95):4.2f}"
          f"  {np.percentile(full, 50):8.2f} {np.percentile(full, 95):4.2f}  {usd:7.2f}")

small share  TTFT p50  p95   full p50  p95   USD/1k
        0%      0.95 0.95      2.74 5.02    16.03
       70%      0.70 0.95      1.84 3.82     7.70
       90%      0.70 0.95      1.65 3.24     5.25
       97%      0.70 0.70      1.61 2.86     4.33

Sending 70% of questions to the small model cuts the bill by about half (USD 16.03 to 7.70 per 1,000 queries) and cuts the median full answer from 2.74 s to 1.84 s. The p95 gains less. The rule is the one from skipping the reranker in Move 4: while more than 5% of requests take the slow path, the p95 request is a slow-path request, and it can only slide down that path's own distribution (with 30% of questions on the large model, to the large model's own p83). In this simulation each model's first token takes a fixed time, so the first-token p95 has nowhere to slide: it stays at 0.95 s until more than 95% of questions go small. Answer lengths vary, so the full-answer p95 does slide, from 5.02 s to 3.82 s, about the large model's own p83: a 24% cut, against 33% at the median. The router itself can be cheap (rules, or a classifier on the query embedding you already computed). The real cost is the hard question sent to the small model, so measure accuracy per route on a labeled question set. A cascade that tries the small model first and escalates when a check fails is a different trade: the escalated questions pay for both calls, one after the other.

Prompt caching: only the part that repeats

Provider prompt caching reuses the already-processed start of a prompt. Anthropic's prompt caching docs, checked September 2026: cache reads bill at 0.1× the input price on most models (0.05× on Claude Opus 5.5), writes to the default 5-minute cache at 1.25×, each hit refreshes the lifetime, and "you will generally see improved time-to-first-token for long documents". The prefix must be byte-identical, and it must reach a minimum length that depends on the model, from 512 to 4,096 tokens (4,096 for Claude Haiku 4.5). A shorter prefix is simply processed without caching, with no error. OpenAI caches automatically from 1,024 tokens on its current models (docs).

Two consequences for RAG:

• In our assistant, caching does nothing. The fixed part is about 250 tokens of instructions, below every minimum. The roughly 3,000 tokens of abstracts change with every claim, so they cannot be a shared prefix (Context Augmentation makes the same point).
• It pays when a long, stable block leads every prompt, such as 5,000 tokens of policies and worked examples. At Claude Haiku 4.5's USD 1 per million input tokens, that block costs USD 5.00 per 1,000 queries uncached and USD 0.50 as cache reads, plus a write at 1.25× whenever the cache expires. Put stable content first and variable content after it; a timestamp at the top of the system prompt defeats the whole thing. The rest is in Caching Strategies.

Your turn. Cut the context set from 8 abstracts to 5, at about 380 tokens each (the average of the chosen abstracts in Move 1's run). How much sooner does the first token come with the modeled hosted LLM (prefill 10,000 tokens per second) and with the laptop model above (about 1,500 per second)? What does it save per 1,000 queries at USD 4 per million input tokens, and what must you check before shipping it?

Move 7: Survive the peak

Symptom: the p95 was fine in testing. At the Monday-morning peak it triples, and nothing in the code changed.

Queues: waiting explodes near capacity

Take one reranker worker whose service time has a median of 200 ms, and let queries arrive at random. Utilization is the arrival rate divided by the rate the worker can sustain:

import numpy as np

rng = np.random.default_rng(0)
N = 200_000
service = rng.lognormal(np.log(0.200), 0.3, N)          # one worker; median 200 ms per query
capacity = 1 / service.mean()                           # queries per second it can sustain
print(f"mean service {1000 * service.mean():.0f} ms, capacity {capacity:.2f} q/s")

for load in (0.5, 0.7, 0.8, 0.9, 0.95):
    gaps = rng.exponential(1 / (load * capacity), N)    # random (Poisson) arrivals
    wait, total = 0.0, np.empty(N)
    for i in range(N):                                  # FIFO queue, one request at a time
        total[i] = wait + service[i]                    # time in system = queueing + service
        wait = max(0.0, total[i] - gaps[i])             # what the next arrival finds ahead of it
    p50, p95 = np.percentile(total, [50, 95])
    print(f"utilization {load:.0%}: p50 {1000 * p50:5.0f} ms   p95 {1000 * p95:6.0f} ms")

mean service 209 ms, capacity 4.78 q/s
utilization 50%: p50   265 ms   p95    703 ms
utilization 70%: p50   370 ms   p95   1181 ms
utilization 80%: p50   501 ms   p95   1741 ms
utilization 90%: p50   898 ms   p95   3386 ms
utilization 95%: p50  1579 ms   p95   6544 ms

No request got slower to process. They got slower to start, because they waited for each other. At 50% utilization the p95 is already 3.5 times the median service time, at 90% it is 17 times, and the last few percent before capacity are catastrophic. So keep headroom, measure queue wait as its own span, and scale on queue wait or latency rather than on average CPU. How much capacity to buy is Production Deployment & Optimization's question; this curve is why the answer is never "exactly the peak".

Batching: more throughput, not a faster request

batch = QUESTIONS[:64]
t0 = time.perf_counter()
for q in batch:
    embedder.encode(q)                           # one request at a time
one_by_one = time.perf_counter() - t0
t0 = time.perf_counter()
embedder.encode(batch, batch_size=64)            # all 64 in one forward pass
together = time.perf_counter() - t0
print(f"one at a time: {1000 * one_by_one / 64:5.1f} ms each, {64 / one_by_one:5.0f} queries/s")
print(f"one batch:     {1000 * together:5.1f} ms for all, {64 / together:5.0f} queries/s")

one at a time:   4.9 ms each,   204 queries/s
one batch:      63.8 ms for all,  1002 queries/s

The batch raised throughput about five times, from 204 to 1,002 queries per second. It also made every query in it slower: each one waited 63.8 ms for the whole batch, against 4.9 ms alone.

Serving systems batch dynamically: they collect requests for up to some window, or until the batch is full, then run them together. Every request can pay up to that window in waiting. Batching pays when the device is the bottleneck at peak, because it raises queries per second per GPU and cuts the cost per query. At low traffic it only adds waiting. Reranking Models made the same point for reranking. Work that nobody is waiting for, such as re-embedding a corpus or offline evaluation, belongs in batch jobs: Anthropic's Batch API bills at 50% of the normal price (September 2026).

Quote throughput honestly. 100 queries at 50 ms each, one at a time, take 5.0 s: 20 queries per second. In batches of 32 at 200 ms per batch, the 100 queries need four batches (32 + 32 + 32 + 4) and take 0.8 s: 125 queries per second, 6.25×. The 160 queries per second of a full batch holds only when every batch is full.

Load tests that tell the truth

• Send requests at a fixed arrival rate (an open model), the way independent users do. A closed-loop generator waits for each response before sending the next request, so when the system slows down it sends less and hides the slowdown. This is coordinated omission (wrk2 README). k6's constant-arrival-rate and ramping-arrival-rate executors implement the open model (k6 docs).
• Use the real mix: real question lengths, answer lengths and cache hit rates. A test where every answer is 50 tokens long measures a different system.
• Warm up, then hold the peak rate long enough for the tail to show. A longer soak run finds leaks and caches that grow without bound.
• Record per-stage percentiles, errors, timeouts and fallbacks, not just the end-to-end mean.

Tails: fan-out, timeouts and hedges

A query that waits for many parts waits for the slowest. Here is a vector index split into 8 shards, where each shard's search takes about 20 ms but 1 call in 100 hits a 250 ms stall:

import numpy as np

rng = np.random.default_rng(1)
Q, SHARDS = 100_000, 8

def shard_ms(shape):
    """One shard's search: about 20 ms, but 1 call in 100 hits a 250 ms stall."""
    base = rng.lognormal(np.log(20), 0.25, shape)
    return base + np.where(rng.random(shape) < 0.01, 250, 0)

first = shard_ms((Q, SHARDS))
plain = first.max(axis=1)                            # a query waits for its slowest shard

hedge_at = np.percentile(first, 95)                  # a shard's own p95
backup = hedge_at + shard_ms((Q, SHARDS))            # replica asked once the first is late
hedged = np.where(first > hedge_at, np.minimum(first, backup), first).max(axis=1)

print(f"one shard: p95 {hedge_at:.0f} ms, p99 {np.percentile(first, 99):.0f} ms")
print(f"queries slower than 100 ms: {np.mean(plain > 100):.1%} (1 - 0.99**8 = {1 - 0.99**8:.1%})")
for name, t in (("fan-out", plain), ("fan-out, hedged", hedged)):
    print(f"{name:16} p50 {np.percentile(t, 50):4.0f}  p95 {np.percentile(t, 95):4.0f}  "
          f"p99 {np.percentile(t, 99):4.0f} ms")
print(f"extra shard calls: {np.mean(first > hedge_at):.1%}")

one shard: p95 31 ms, p99 262 ms
queries slower than 100 ms: 7.9% (1 - 0.99**8 = 7.7%)
fan-out          p50   29  p95  269  p99  277 ms
fan-out, hedged  p50   29  p95   49  p99   58 ms
extra shard calls: 5.0%

One shard's p95 is 31 ms, yet the query's p95 is 269 ms: 7.9% of queries hit at least one stall, which matches 1 − 0.99**8 ≈ 7.7%. Dean and Barroso's The Tail at Scale (CACM 2013) gives the extreme version: if each of 100 servers is slow on 1 request in 100, "63% of user requests will take more than one second". Their remedy is the hedged request: if a part has not answered by its p95, send the same request to a replica and take whichever answers first. Deferring the hedge to the p95 "limits the additional load to approximately 5%". In the simulation, hedging cost 5.0% extra shard calls and cut the p99 from 277 ms to 58 ms. Their own example: in a Google BigTable benchmark, hedging after 10 ms cut the 99.9th-percentile latency from 1,800 ms to 74 ms while sending 2% more requests.

Rules for the tail:

• Hedge cheap, idempotent reads, such as a shard search or an embedding call. A hedged LLM generation is a second bill; do it only with the arithmetic in hand.
• Every timeout needs a fallback. Reranker late: use the first-stage order. For the LLM, time out on the first token, not the whole answer, because a long answer is legitimately slow. A fallback that starts only after the timeout adds to the wait, so budget timeout plus fallback, or run the fallback alongside.
• Stop waiting on a dependency that is down. A circuit breaker stops sending requests to a failing service for a while and answers from the fallback at once, instead of making every request wait out the timeout.
• Under overload, shed work before requests. When queue wait passes a threshold, skip reranking or route to the small model, and record that you did. A slightly worse answer on time beats a timeout.
• Retries multiply load exactly when the system is struggling. Cap them and back off. Know your client's defaults: Anthropic's Python SDK retries twice by default and retries timeouts too, so one call can take up to three times its timeout.

Cold starts

A new replica must load its models before it can answer, and its first calls are slower than the rest:

import subprocess, sys

REPLICA = """
import time
t0 = time.perf_counter()
from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", device="cpu")
t1 = time.perf_counter()
pairs = [("Is obesity heritable?", "Twin studies estimate that body mass index is highly heritable.")] * 30
model.predict(pairs)
t2 = time.perf_counter()
model.predict(pairs)
t3 = time.perf_counter()
print(f"start and load {t1 - t0:.1f} s, first 30 pairs {1000 * (t2 - t1):.0f} ms, "
      f"next 30 pairs {1000 * (t3 - t2):.0f} ms")
"""
print(subprocess.run([sys.executable, "-c", REPLICA], capture_output=True, text=True).stdout.strip())

start and load 3.0 s, first 30 pairs 23 ms, next 30 pairs 18 ms

A fresh process needed 3.0 s to import its libraries and load the cross-encoder before it could score anything, and its first batch (23 ms) was slower than the second (18 ms). Bigger models take longer to load, and the gap between first and later calls depends on the hardware, so measure it on yours. Autoscaling adds replicas exactly at the peak, so cold starts land on the busiest minutes. Load the models and run a warm-up request before a replica reports ready, keep a warm minimum, and never load a model lazily inside the first user's request. Measure the p99 in the minutes after a deploy or a scale-out, not only at steady state.

Your turn. The reranker worker from the queue simulation runs at 60% utilization at peak. A marketing launch will bring 40% more traffic. What happens to the p95, and what are your options?

Final round: no label on the trace

Real incidents don't say which move they need. Read the symptom, pick the move, and say what you would measure.

Challenge 1: the few-shot examples

A teammate adds 12 worked examples to the system prompt, 3,500 tokens in all, to improve answer format. Answer quality rises slightly. The p95 of time to first token goes from 1.2 s to 1.6 s; answer lengths did not change.

Challenge 2: the shared GPU

To save money, the reranker moves from a dedicated GPU to a shared GPU pool that batches requests from several services. The median drops by 20 ms. At the daily peak, the p99 of the rerank stage goes from 180 ms to 900 ms.

Challenge 3: hybrid made it slower

Adding BM25 next to dense search improved recall@10 on the labeled set, and the p95 of retrieval rose by 40 ms, about the BM25 call's own time. The trace shows the BM25 span starting when the dense span ends.

Cheat sheet: symptom → move → cost

Before moving on, take your own pipeline and answer, from a blank page: which clock and percentile the promise names; the p50 and p95 of each stage on that clock at peak load; which stage Amdahl says to fix first and how much it can save; what that fix costs in quality, and on which labeled set you will check it. Then run the loop: 100 to 300 real questions with labels, replayed until you have thousands of requests, per-stage spans, both clocks at realistic load, and for every change the quality metric, the latency percentile and the cost per 1,000 queries side by side. Keep a change only if quality holds.

Next: Caching Strategies for skipping work entirely, Production Deployment & Optimization for SLOs, rollouts and capacity, and Infrastructure & Security for tracing and cost management in production.