Caching Strategies

The refund answer that belonged to another plan

A help-desk assistant answers questions about a subscription software product with a RAG pipeline: retrieve help-center chunks, put them in a prompt, generate an answer. Generation dominates the bill, and the logs show the same questions again and again. So the team adds a semantic answer cache: embed each incoming question, find the most similar question answered before, and reuse its answer when the cosine similarity is at least 0.85, "a good starting point" according to a blog post.

Here is that cache with three answered questions, using the small open model all-MiniLM-L6-v2:

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")  # 384-d, unit-length output

CACHE = {  # question answered earlier -> the answer that was generated for it
    "Can I get a refund on my annual plan?":
        "Yes. Annual plans are refundable in full within 30 days of purchase.",
    "How do I reset my password?":
        "Click 'Forgot password' on the sign-in page. The link is valid for one hour.",
    "Which SSO providers do you support?":
        "Any SAML 2.0 or OIDC provider, for example Okta or Microsoft Entra ID.",
}
KEYS = list(CACHE)
KEY_VECS = model.encode(KEYS)
THRESHOLD = 0.85  # "a good starting point", according to a blog post

def lookup(question):
    sims = KEY_VECS @ model.encode(question)       # cosine: all vectors are unit length
    best = int(np.argmax(sims))
    verdict = "HIT " if sims[best] >= THRESHOLD else "miss"
    print(f"{verdict} {sims[best]:.3f}  {question!r} -> nearest {KEYS[best]!r}")

lookup("Can I get a refund on my monthly plan?")
lookup("I bought a yearly subscription last week, can I get my money back?")
lookup("How can I reset my password?")

Predict first: which of the three new questions get a cached answer?

HIT  0.893  'Can I get a refund on my monthly plan?' -> nearest 'Can I get a refund on my annual plan?'
miss 0.562  'I bought a yearly subscription last week, can I get my money back?' -> nearest 'Can I get a refund on my annual plan?'
HIT  0.992  'How can I reset my password?' -> nearest 'How do I reset my password?'

The cache served the wrong answer and missed the right one. An embedding measures closeness of topic and wording: "annual" → "monthly" changes one word, so the vectors stay close, while a real paraphrase shares almost no words. The score does not know which word decides the answer.

This lesson is about caching in a retrieval system without serving answers that are wrong, stale or someone else's. It rests on one rule:

A cache entry is the result of a function. Its key must contain every input that can change the result.

For a RAG answer, those inputs are the question (its meaning, not its spelling: the hard part), who is asking (tenant, permissions, sometimes the user), the documents (captured as an index version), the prompt template and model, and anything else the prompt uses, such as today's date. Every bug in this lesson is an input left out of a key, or a key that matches more than it should.

Where caches sit in a RAG system

indexing:  chunk ──▶ [document-embedding cache] ──▶ vector ──▶ index

serving:   question + scope ──▶ [exact answer cache] ──hit──▶ answer
                │ miss
                ▼
           [query-embedding cache] ──▶ query vector
                │
           [semantic answer cache] ──hit, checks pass──▶ answer
                │ miss
                ▼
           [retrieval-result cache] ──hit──▶ ranked chunk ids ──┐
                │ miss                                          │
                ▼                                               │
           search ──▶ rerank ──▶ ranked chunk ids ──────────────┤
                                                                ▼
                                  fetch text by id (permission-filtered)
                                                                │
           prompt = [ stable prefix | passages | question ] ◀───┘
                      └─ provider prompt cache
                │
                ▼
               LLM ──▶ answer

This lesson assumes the pipeline from RAG Architecture & Implementation; the rest of the latency budget is in Performance Optimization. The code uses Redis 8 through redis-py, and every block was run against Redis 8.10.

Before any cache: price a hit, and price a false hit

A cache is worth what its hits save minus what its mistakes cost. Put numbers on both:

import numpy as np

# Assumptions for one support assistant (round numbers; use your own)
PRICE_IN, PRICE_OUT = 3.00, 15.00   # USD per million input / output tokens
PROMPT, OUTPUT = 4_000, 300          # tokens per generated answer
EMBED_PRICE, QUERY_TOKENS = 0.02, 20 # a small hosted embedding model; tokens per query
WRONG_ANSWER = 5.00                  # USD: what one wrong answer costs you (a ticket, a refund)

llm = (PROMPT * PRICE_IN + OUTPUT * PRICE_OUT) / 1e6     # USD per generated answer
emb = QUERY_TOKENS * EMBED_PRICE / 1e6
print(f"generate an answer: USD {1000 * llm:.2f} per 1,000 queries")
print(f"embed the query:    USD {1000 * emb:.4f} per 1,000 queries")

def usd_per_1000(hit_rate, false_hit_rate=0.0):
    misses = (1 - hit_rate) * llm
    wrong = hit_rate * false_hit_rate * WRONG_ANSWER     # false hits are served, not retried
    return 1000 * (misses + wrong)

for h, f in [(0.0, 0.0), (0.3, 0.0), (0.3, 0.01), (0.6, 0.002)]:
    print(f"hit rate {h:.0%}, false hits {f:.1%} of hits: USD {usd_per_1000(h, f):6.2f}")
print(f"break-even false-hit rate: {llm / WRONG_ANSWER:.2%}")

# Latency: exact-cache hits take ~30 ms, misses run the pipeline (median ~2 s, long tail)
rng = np.random.default_rng(0)
misses = rng.lognormal(np.log(2.0), 0.4, 100_000)
for h in (0.0, 0.3, 0.6, 0.9, 0.96):
    lat = np.where(rng.random(100_000) < h, 0.03, misses)
    print(f"hit rate {h:4.0%}: p50 {np.percentile(lat, 50):5.2f} s   p95 {np.percentile(lat, 95):5.2f} s")

generate an answer: USD 16.50 per 1,000 queries
embed the query:    USD 0.0004 per 1,000 queries
hit rate 0%, false hits 0.0% of hits: USD  16.50
hit rate 30%, false hits 0.0% of hits: USD  11.55
hit rate 30%, false hits 1.0% of hits: USD  26.55
hit rate 60%, false hits 0.2% of hits: USD  12.60
break-even false-hit rate: 0.33%
hit rate   0%: p50  2.00 s   p95  3.85 s
hit rate  30%: p50  1.59 s   p95  3.58 s
hit rate  60%: p50  0.03 s   p95  3.16 s
hit rate  90%: p50  0.03 s   p95  2.00 s
hit rate  96%: p50  0.03 s   p95  0.03 s

Four lessons in those lines:

1. The money is in the answer. At these assumed prices the query embedding costs about 40,000 times less than the generation. A query-embedding cache is for latency; an answer cache is for money.
2. A false hit is not priced like a miss. With a wrong answer costing USD 5, one false hit erases the savings of about 300 correct hits. A 30% hit rate with 1% false hits costs more than no cache, and a cache that is never wrong at 30% (USD 11.55) beats one that is sometimes wrong at 60% (USD 12.60).
3. Caches fix the median before the tail. At a 60% hit rate the median request is a hit, but p95 only drops from 3.85 s to 3.16 s. While fewer than 95% of requests hit, the 95th-percentile request is a miss.
4. Hit rate belongs to your traffic. The MeanCache paper (Gill et al., IPDPS 2025) found that about 31% of the queries of 20 ChatGPT users in an academic setting were similar to one the same user had asked before, and its authors warn that the ratio may vary elsewhere. Yours may show 5% or 60%.

Measure before you build. Normalize a week of real questions the way your key would, and count how many repeat an earlier one within your planned TTL and scope: that is your exact-cache hit rate. For a semantic cache, label a sample of "nearest earlier question" pairs as same answer or different answer. Now you know the hit rate and the false-hit rate before anything ships.

Your turn: with the same simulated latencies, the hit rate reaches 70%. Does p95 become a hit?

Move 1: Exact caches with complete keys

Symptom: a nightly re-index re-embeds all 1,000 chunks although one changed, and after each deploy the cache starts cold.

Exact caches are the safe ones. If the key contains every input, a changed input produces a new key and nothing stale can be served. Here is a key builder and an embedding cache that serves both sides of the pipeline:

import hashlib, json
import numpy as np
import redis
from sentence_transformers import SentenceTransformer

r = redis.Redis(protocol=2)                   # localhost:6379
PREFIX = "rag:"                               # one namespace per app and environment
MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
REVISION = "1110a243fdf4706b3f48f1d95db1a4f5529b4d41"   # pin the exact weights you load
model = SentenceTransformer(MODEL_NAME, revision=REVISION)
# Everything that changes the vector: model, revision, dimension, prefix, normalization.
EMBED_ID = {"model": MODEL_NAME, "revision": REVISION, "dim": 384, "prefix": "", "normalized": True}

def cache_key(kind, **inputs):
    """Same inputs -> same key, in every process, forever."""
    blob = json.dumps(inputs, sort_keys=True, ensure_ascii=False)
    return f"{PREFIX}{kind}:{hashlib.sha256(blob.encode()).hexdigest()[:32]}"

stats = {"embedded": 0}

def embed_many(texts, ttl=None):
    """Embedding cache: look everything up in one round trip, embed only the misses."""
    keys = [cache_key("emb", model=EMBED_ID, text=t) for t in texts]
    cached = r.mget(keys)
    todo = [i for i, raw in enumerate(cached) if raw is None]
    if todo:
        vecs = model.encode([texts[i] for i in todo]).astype(np.float32)
        stats["embedded"] += len(todo)
        with r.pipeline() as p:
            for i, v in zip(todo, vecs):
                p.set(keys[i], v.tobytes(), ex=ttl)
                cached[i] = v.tobytes()
            p.execute()
    return np.stack([np.frombuffer(raw, dtype=np.float32) for raw in cached])

chunks = [f"Section {i}: refunds, invoices and plan changes, part {i}." for i in range(1000)]
embed_many(chunks)                                 # first indexing run
print("first run embedded:", stats["embedded"])
chunks[17] = "Section 17: annual plans are refundable within 14 days."   # one chunk edited
stats["embedded"] = 0
embed_many(chunks)                                 # re-index after the edit
print("second run embedded:", stats["embedded"])

q = embed_many(["How do I get a refund?"], ttl=7 * 24 * 3600)[0]   # query side: same cache
print("query vector:", q.shape, q.dtype)

first run embedded: 1000
second run embedded: 1
query vector: (384,) float32

Why it works

The key is derived from the inputs, so it is content-addressed. Edit chunk 17 and its text hash changes: one new key, one embedding call. Upgrade the model and EMBED_ID changes, so every key is new, which is what you want, because old vectors are useless to a new model. Content-addressed entries never become wrong; they only become unused. Eviction reclaims them once Redis has a memory limit and an allkeys-* policy (Move 6). embed_many gives document vectors no TTL, so on a default Redis, which has neither, they stay forever. For a large corpus, keep document vectors in a durable, disk-backed store instead: Embedding Pipeline keeps them in a SQLite file, so a crashed run or a rollback re-embeds nothing. An evicting Redis suits query vectors and small corpora.

That also decides where entries may live. Content-addressed entries are safe anywhere, even in an in-process dictionary on each replica. Entries that can become wrong, such as answers, belong in one shared store where invalidation can reach them.

⚠️ Python's hash() is not a cache key. String hashing is randomized per process (Python 3.3 and later): run python3 -c 'print(hash("How do I get a refund?"))' three times and you get three numbers. Every replica and restart then writes under new keys, and nothing can recompute a key to delete it. Use hashlib over json.dumps(..., sort_keys=True), so dictionary order cannot change the key either.

⚠️ Normalize only what cannot change the output. Collapsing whitespace is safe. Lowercasing is free for the embedding key when the model lowercases anyway: all-MiniLM-L6-v2's tokenizer does, so "XK-4471" and "xk-4471" embed identically (cosine 1.000 in our run). For a case-sensitive model, or exact identifiers in an answer key, check first. And if the prompt uses today's date ("is the sale still on?"), the date is an input.

⚠️ The silent model change. A new revision of the same model name, or a changed query prefix, keeps the dimension, so stale cached vectors raise no error and quietly degrade retrieval. Put the revision and prefix in the key. Re-embedding a corpus under a new model (new index, alias switch) is in Vector Embeddings; doing it at scale is in Embedding Pipeline.

Caching query rewrites. A single-turn rewrite (spelling fixes, expansion) is a function of the query and the rewriter, so popular queries repeat and the cache pays. A conversational rewrite ("and for the monthly plan?") depends on the history, so its key must include the history and it almost never repeats. Cache the answer under the rewritten standalone question instead, where repeats do happen (Query Understanding & Intent covers the rewriting).

Your turn: you switch the embedding model to a new revision with the same 384 dimensions. What happens to the embedding cache's hit rate, and is that a bug?

Move 2: Semantic caching, and the threshold that isn't there

Symptom: a question that differs by one word gets a confident answer to the other question. The hook's monthly-plan customer.

One example proves nothing, so label some pairs. For each pair, ask the only question that matters: does the cached answer to the first question also answer the second?

import numpy as np
from sentence_transformers import SentenceTransformer

# Labeled pairs: does the cached answer for the first question also answer the second?
REWORDED = [  # same answer, light rewording
    ("How do I reset my password?", "How can I reset my password?"),
    ("How do I cancel my subscription?", "how to cancel subscription"),
    ("What payment methods do you accept?", "Which payment methods do you accept?"),
    ("Can I get a refund on my annual plan?", "Can I get a refund for my annual plan?"),
    ("How do I export my data to CSV?", "How to export data as CSV"),
    ("Does the API have rate limits?", "Are there rate limits on the API?"),
    ("How do I add a new user to my workspace?", "How do I add a user to my workspace?"),
    ("Is there a mobile app?", "Do you have a mobile app?"),
]
PARAPHRASED = [  # same answer, different words
    ("Can I get a refund on my annual plan?", "I bought a yearly subscription last week, can I get my money back?"),
    ("How do I reset my password?", "I forgot my password and can't log in, what do I do?"),
    ("What is the refund window for annual plans?", "How many days do I have to ask for a refund on a yearly plan?"),
    ("How do I export my data to CSV?", "Can I download my records as a spreadsheet?"),
    ("Does the API have rate limits?", "How many requests per minute can I send to the API?"),
    ("How do I add a new user to my workspace?", "How can I invite a teammate?"),
    ("Which SSO providers do you support?", "Can I sign in with Okta or Azure AD?"),
    ("How do I cancel my subscription?", "I want to stop paying, how do I end my plan?"),
    ("What payment methods do you accept?", "Can I pay by credit card or bank transfer?"),
    ("How do I change the email address on my account?", "I need to update my login email"),
    ("Is there a mobile app?", "Can I use the product on my phone?"),
    ("How do I delete my account?", "Please remove my account and all my data"),
]
ONE_SLOT = [  # one word or number changed -> a DIFFERENT answer
    ("Can I get a refund on my annual plan?", "Can I get a refund on my monthly plan?"),
    ("How do I cancel my subscription?", "How do I renew my subscription?"),
    ("What is the API rate limit on the Pro plan?", "What is the API rate limit on the Free plan?"),
    ("How do I add a user to my workspace?", "How do I remove a user from my workspace?"),
    ("How do I export my data to CSV?", "How do I import my data from CSV?"),
    ("Is SSO available on the Business plan?", "Is SSO available on the Starter plan?"),
    ("What is the status of order 10432?", "What is the status of order 10433?"),
    ("How do I enable two-factor authentication?", "How do I disable two-factor authentication?"),
    ("Can I upgrade my plan in the middle of the month?", "Can I downgrade my plan in the middle of the month?"),
    ("What was the reporting deadline for Q3?", "What was the reporting deadline for Q4?"),
    ("What is the ibuprofen dose for adults?", "What is the ibuprofen dose for children?"),
    ("Does the Starter plan include phone support?", "Does the Starter plan include email support?"),
]

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def cosines(pairs):
    a = model.encode([p[0] for p in pairs])
    b = model.encode([p[1] for p in pairs])
    return (a * b).sum(axis=1)                 # row-wise cosine (unit vectors)

groups = {"reworded": cosines(REWORDED), "paraphrased": cosines(PARAPHRASED),
          "one-slot": cosines(ONE_SLOT)}
for name, s in groups.items():
    print(f"{name:12} min {s.min():.3f}  median {np.median(s):.3f}  max {s.max():.3f}")
print("threshold  reworded  paraphrased  one-slot (false hits)")
for t in (0.80, 0.85, 0.90, 0.95):
    row = [f"{int((s >= t).sum())}/{len(s)}" for s in groups.values()]
    print(f"{t:9.2f}  {row[0]:>8}  {row[1]:>11}  {row[2]:>8}")

Predict: which group scores higher on average, the true paraphrases or the one-word changes that need a different answer?

reworded     min 0.844  median 0.970  max 0.993
paraphrased  min 0.280  median 0.564  max 0.818
one-slot     min 0.692  median 0.847  max 0.948
threshold  reworded  paraphrased  one-slot (false hits)
     0.80       8/8         1/12      8/12
     0.85       7/8         0/12      6/12
     0.90       7/8         0/12      3/12
     0.95       7/8         0/12      0/12

Swap the model name and run it again. The pattern held on all five models we tried, in six configurations:

• One-slot changes outscore true paraphrases on every model. No threshold catches paraphrases without also accepting wrong answers. Raise it until the false hits stop, and what remains is a cache for light rewordings: 7 of 8 on MiniLM at 0.95.
• Thresholds do not transfer, not even between two prompts for the same model: Qwen3 lets 4 of 12 wrong answers through at 0.90 without its instruction and none with it. On e5-small-v2 everything bunches between 0.84 and 1.0, and 0.95 still lets 8 of 12 through. The order-number pair scores 0.948 on MiniLM, just under 0.95, and 0.974 on mpnet, above it. A threshold from a blog post or another model means nothing (Cosine Similarity & Distance Metrics explains why scores are uncalibrated).
• Thirty-two pairs is a demonstration, not a tuning set. Tune on a few hundred pairs from your own logs.

Why it happens. A bi-encoder compresses the question into one vector without knowing which word decides the answer. "Annual" vs "monthly" is a small part of an otherwise identical sentence, so it moves the vector a little; a paraphrase changes nearly every word, so it moves the vector a lot. Similarity measures overlap. The cache needs equivalence.

Checks that know the difference

Use the embedding only to find a candidate, and decide with checks that look at the words that matter:

import re, time
from sentence_transformers import CrossEncoder

PLANS = {"free", "starter", "pro", "business", "annual", "monthly"}
SYNONYMS = {"yearly": "annual"}

def slots(q):
    """Parts of a question that must match exactly: numbers, ids, plan names."""
    words = [SYNONYMS.get(w, w) for w in re.findall(r"[a-z0-9][a-z0-9-]*", q.lower())]
    return {w for w in words if any(c.isdigit() for c in w) or w in PLANS}

nli = CrossEncoder("cross-encoder/nli-deberta-v3-base", device="cpu")
ENTAILMENT = nli.model.config.label2id["entailment"]   # 1 for this checkpoint; read it, never copy it

def mutual_entailment(a, b):
    p = nli.predict([(a, b), (b, a)], apply_softmax=True)
    return float(min(p[0][ENTAILMENT], p[1][ENTAILMENT]))

def accept(a, b, cos, t=0.80):
    if cos < t:
        return "below threshold"
    if slots(a) != slots(b):
        return "slot mismatch"
    return "accept" if mutual_entailment(a, b) >= 0.5 else "not equivalent"

for name, pairs in [("reworded", REWORDED), ("paraphrased", PARAPHRASED), ("one-slot", ONE_SLOT)]:
    verdicts = [accept(a, b, c) for (a, b), c in zip(pairs, cosines(pairs))]
    print(f"{name:12}", {v: verdicts.count(v) for v in sorted(set(verdicts))})

t0 = time.perf_counter()
for _ in range(20):
    mutual_entailment(*ONE_SLOT[0])
print(f"verifier: {(time.perf_counter() - t0) / 20 * 1000:.0f} ms per check (CPU)")

reworded     {'accept': 8}
paraphrased  {'below threshold': 11, 'not equivalent': 1}
one-slot     {'below threshold': 4, 'not equivalent': 4, 'slot mismatch': 4}
verifier: 370 ms per check (CPU)

Three layers, three jobs:

1. A low embedding threshold (0.80 for this model) only finds a candidate.
2. Slot matching rejects candidates whose numbers, ids or known product terms differ: 10432 vs 10433, Q3 vs Q4, Pro vs Free, annual vs monthly. The vocabulary comes from your product catalog, and it knows only the slots you listed.
3. A pair model judges the rest: this natural-language-inference cross-encoder must find that each question entails the other. It caught enable/disable, import/export, upgrade/downgrade and adults/children, which no slot list here contained. It took about 45 to 590 ms per check on a laptop CPU in our runs, depending on load (about 20 ms on the laptop's GPU), and runs only on candidates that passed the first two layers. So a semantic hit costs the query embedding plus the verifier: on a CPU that is tens to hundreds of milliseconds, not the few milliseconds of an exact hit, though still far below a pipeline run. It makes its own mistakes: it scored "Which MFA provider do we use?" against "Which SSO provider do we use?" at 0.857, a pass; only the embedding threshold (cosine 0.497) stopped that pair.

On these 32 pairs the combination kept all 8 rewordings, rejected all 12 one-slot changes, and still missed all 12 real paraphrases. That is the honest shape of a safe semantic cache: it turns rewordings into hits. Catching real paraphrases safely takes a different design:

• Canonical answers. Route questions to a curated set of FAQ intents with a classifier trained on labeled examples, and keep one reviewed answer per intent. The hit rate then comes from routing accuracy, which you can measure.
• Route rules. Some routes never get a semantic cache: dosages, legal deadlines, prices, anything personal or live. There, one false hit costs more than a month of savings.

💡 Know your score's scale. Redis vector sets report a similarity from 0 to 1 that equals (1 + cosine) / 2 (checked on Redis 8.10: cosine 0.8 scores 0.9). RedisVL's SemanticCache (0.27.2) takes a distance_threshold in cosine distance, 1 − cosine (0 to 2), default 0.1. So cosine 0.9 is a vector-set score of 0.95 and a RedisVL distance_threshold of 0.1, while a bare "0.9" means cosine 0.8 to a vector set and cosine 0.1 to RedisVL. Convert before you compare.

Your turn: your service uses bge-small-en-v1.5, and a teammate sets the threshold to 0.90 "because MiniLM at 0.90 only let 3 of 12 through". Using the table, what happens, and what should the teammate do instead?

Move 3: Scope every entry to who may see it

Symptom: a user at one customer company asks "Which SSO provider do we use?" and is told that the company signs in with another customer's Okta tenant.

This is not a threshold problem. The two users typed the same words, so the similarity is 1.0 on any model, and no threshold, verifier or model upgrade stops it. The answer depends on who asks, so who asks goes into the key.

Continuing from the blocks above, here is a semantic cache on Redis vector sets with the scope in the key name:

import hashlib, json, uuid
import redis

r = redis.Redis(protocol=2)  # RESP2: redis-py 8.1 on RESP3 raises on VSIM WITHSCORES for a missing key
PREFIX = "rag:"
VERSIONS = "index=docs-v42|prompt=answer-v7|llm=llm-2026-09"

def scope_of(tenant, groups=(), user=None):
    """Who may see an answer. Resolved on the server from the session, never from the request."""
    blob = json.dumps({"tenant": tenant, "groups": sorted(set(groups)), "user": user})
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def vset_key(scope):        # scope and versions live in the key name: there is no filter to forget
    return f"{PREFIX}sem:{scope}:{hashlib.sha256(VERSIONS.encode()).hexdigest()[:8]}"

def store(question, answer, scope, ttl=3600):
    entry = uuid.uuid4().hex
    r.set(f"{PREFIX}ans:{entry}", json.dumps({"q": question, "a": answer}), ex=ttl)
    r.vset().vadd(vset_key(scope), model.encode(question).tolist(), entry)
    r.expire(vset_key(scope), ttl, nx=True)      # the set lives as long as its newest entry:
    r.expire(vset_key(scope), ttl, gt=True)      # a shorter TTL never cuts it short

def lookup(question, scope):
    found = r.vset().vsim(vset_key(scope), model.encode(question).tolist(),
                          with_scores=True, count=1)
    if not found:
        return "miss: nothing cached for this scope", None
    entry, score = next(iter(found.items()))
    cosine = 2 * score - 1                       # VSIM reports (1 + cosine) / 2
    raw = r.get(f"{PREFIX}ans:{entry.decode()}")
    if raw is None:                              # the answer expired: drop its vector too
        r.vset().vrem(vset_key(scope), entry)
        return "miss: expired", None
    cached = json.loads(raw)
    verdict = accept(cached["q"], question, cosine)     # Move 2's three checks
    return f"{verdict} (cosine {cosine:.3f})", cached["a"] if verdict == "accept" else None

acme, globex = scope_of("acme", ["all-staff"]), scope_of("globex", ["all-staff"])
store("Which SSO provider do we use?", "Acme signs in with Okta at acme.okta.com.", acme)
for question, scope, who in [("Which SSO provider do we use?", acme, "acme"),
                             ("Which SSO provider do we use?", globex, "globex"),
                             ("Which identity provider do we use for SSO?", acme, "acme"),
                             ("Which SSO protocol do we use?", acme, "acme")]:
    print(f"{who:7}", lookup(question, scope))

u17 = scope_of("acme", ["all-staff"], user="u17")      # personal answers: scope by user
store("How many vacation days do I have left?", "You have 12 days left.", u17)
u42 = scope_of("acme", ["all-staff"], user="u42")
print("u42    ", lookup("How many vacation days do I have left?", u42))

shared = scope_of("everyone")                   # the naive design: one cache for all tenants
store("Which SSO provider do we use?", "Acme signs in with Okta at acme.okta.com.", shared)
print("globex, shared cache:", lookup("Which SSO provider do we use?", shared))

acme    ('accept (cosine 0.999)', 'Acme signs in with Okta at acme.okta.com.')
globex  ('miss: nothing cached for this scope', None)
acme    ('accept (cosine 0.880)', 'Acme signs in with Okta at acme.okta.com.')
acme    ('not equivalent (cosine 0.857)', None)
u42     ('miss: nothing cached for this scope', None)
globex, shared cache: ('accept (cosine 0.999)', 'Acme signs in with Okta at acme.okta.com.')

The other tenant and the other user miss: different keys, nothing to compare against. Within the tenant, a rewording ("identity provider … for SSO", 0.880) hits, and the near miss ("SSO protocol", 0.857) passes the threshold and the slot check but not the verifier: SAML versus OIDC is a different answer. In the shared design, the other tenant gets Acme's answer at 0.999 (not 1.000 only because vector sets store int8-quantized vectors by default) on the very first repeat.

Choosing the scope

The scope is a property of the route. Decide it where you decide how to answer:

Key by the permission set, not by the user, when documents have per-group access. Users with the same groups may see the same answers, so they share entries and the hit rate stays useful, while a user who may read less never gets an answer built from documents they cannot read. The permission set must be the one the server resolved for the session: sorted, de-duplicated, never taken from the request body or a model's tool arguments.

Or check the sources at serve time. Store each answer with the ids of the chunks it used, share it more widely (say per tenant), and serve it only after one batched authorization check confirms the user may read every source. That matches the rule in Metadata & Filtering: anything derived from documents is visible only to users who may read all of its sources. You gain hits and pay an authorization call per hit, and when that service is down you must fail closed.

Revocation. A user who leaves a group gets a new permission-set hash as soon as the server resolves their groups again: on the next request if it resolves them per request, only when the token expires if they come from a token claim, and only when the cached lookup expires if you cache the directory call to save latency, so keep that lifetime short. Metadata & Filtering walks through the timeline. From then on, entries under the old hash are unreachable for them. A document whose access changes is different: answers built from it still sit under scopes that may no longer read it, and only dependency invalidation (Move 4) or the serve-time check removes them.

⚠️ A filter you can forget fails open. Many semantic-cache libraries scope entries with a metadata filter. RedisVL's SemanticCache.check() (0.27.2) takes an optional filter_expression, and without it, its docstring says, "the full cache will be searched". One code path that forgets the filter serves every tenant's answers. A scope in the key name cannot be forgotten, because without it you cannot build the key. If you use filters, build them in one function that raises when the scope is missing.

Test it like a security control. Keep leak tests in CI: store an answer under one tenant, permission set and user, then ask the identical question from every other combination and assert a miss. The identical question is the right test, because no threshold can stop it.

Move 4: When the documents change

Symptom: the refund window changed from 30 to 14 days at 10:00. At 15:00 the assistant still says 30 days, with a citation.

An answer keyed by the question skips retrieval, so it cannot notice that the documents changed. Five mechanisms bound how long it keeps serving the old answer, and they combine:

The last row shows where staleness lives. A key that contains the chunk text cannot serve an answer built on old text, because new text is a new key. The query-keyed caches (answer, semantic, retrieval result) go stale because they skip the step that would have seen the change.

Here is dependency invalidation, continuing from Move 1's block (r, cache_key, PREFIX):

import re, time

INDEX = {"refund-policy": "Annual plans are refundable within 30 days of purchase."}
VERSIONS = {"index": "docs-v42", "prompt": "answer-v7", "llm": "llm-2026-09"}
LLM_SECONDS = 0.0                              # how long the stub LLM takes

def generate(question, passages):              # stub LLM: repeats its source
    time.sleep(LLM_SECONDS)
    return passages[0]

def remember(key, value, sources, ttl):
    """Store an answer and record the documents it was built from, in one MULTI."""
    with r.pipeline() as p:
        p.set(key, value, ex=ttl)
        for s in sources:                       # reverse index: document -> answers built from it
            deps = f"{PREFIX}deps:{s}"
            p.sadd(deps, key)
            p.expire(deps, ttl, nx=True)        # a new set gets this TTL;
            p.expire(deps, ttl, gt=True)        # an existing one is only ever extended
        p.execute()

def answer(question, scope, ttl=24 * 3600):
    key = cache_key("ans", q=" ".join(question.lower().split()), scope=scope, **VERSIONS)
    if (hit := r.get(key)) is not None:
        return "cache", hit.decode()
    sources = ["refund-policy"]                 # retrieval stub: ids of the chunks used
    text = generate(question, [INDEX[s] for s in sources])
    remember(key, text, sources, ttl)
    return "fresh", text

def invalidate(doc_id):
    with r.pipeline() as p:                     # MULTI: read and clear the set in one step
        p.smembers(f"{PREFIX}deps:{doc_id}")
        p.delete(f"{PREFIX}deps:{doc_id}")
        keys, _ = p.execute()
    if keys:
        r.delete(*keys)

Q, SCOPE = "How long do I have to get a refund?", "acme:all-staff"
NEW = "Annual plans are refundable within 14 days of purchase."
days = lambda text: re.search(r"\d+ days", text).group()

print("1", *answer(Q, SCOPE))
invalidate("refund-policy")                     # wrong order: invalidate first...
print("2", *answer(Q, SCOPE))                   # ...a request lands before the re-index
INDEX["refund-policy"] = NEW                    # the re-index finishes
print("3", *answer(Q, SCOPE))

invalidate("refund-policy")                     # right order: re-index (done), then invalidate
print("4", *answer(Q, SCOPE))

Predict: what do lines 2 and 3 print?

1 fresh Annual plans are refundable within 30 days of purchase.
2 fresh Annual plans are refundable within 30 days of purchase.
3 cache Annual plans are refundable within 30 days of purchase.
4 fresh Annual plans are refundable within 14 days of purchase.

The rule: update the index, then invalidate. In the reverse order, a request between the two steps re-caches the old answer with a fresh TTL. The same race applies when invalidations arrive as events from another service: act only after the index write they refer to is visible.

Two details in remember() and invalidate() matter as much as the order:

• A dependency set must outlive every answer it lists. With a plain EXPIRE, writing a 2-second answer cuts the set that also lists a 24-hour answer down to 2 seconds, and the next invalidation silently misses the 24-hour answer (we reproduced exactly that). EXPIRE … NX followed by EXPIRE … GT (Redis 7 and later) only ever extends it. The set still collects keys of answers that have already expired; deleting those is harmless, and an occasional SSCAN sweep keeps it small.
• Read and clear the set in one step. As two separate calls, an answer written between SMEMBERS and DEL loses its record and survives the next edit. The MULTI makes the pair atomic.

The answer that was still being generated

The right order narrows the window; it does not close it. A request that read the old text before the index update and finishes generating after the invalidation writes the old answer back, with a full TTL:

slow request:  retrieve (14-day text) ── generate (1 s) ────────────── cache the 14-day answer, TTL 24 h
indexer:                  update index to 10 days ── invalidate (nothing to delete yet)
next request:                                                          cache hit: 14 days ✗

Facebook's memcache paper calls this a stale set and prevents it with leases: a token handed out on a miss, voided by a delete, and required for the write (Nishtala et al., NSDI 2013). Here the delete cannot name a key that the slow request has not written yet, so check freshness where it is decided, at serve time: store the version of each source with the answer, and serve a hit only while the index still holds that version.

import hashlib, json, threading

def version(text):                              # a source's version: the hash of its text
    return hashlib.sha256(text.encode()).hexdigest()[:16]

def answer_checked(question, scope, ttl=24 * 3600):
    key = cache_key("ans-checked", q=" ".join(question.lower().split()), scope=scope, **VERSIONS)
    if (hit := r.get(key)) is not None:
        entry = json.loads(hit)
        now = {s: version(INDEX[s]) for s in entry["sources"] if s in INDEX}
        if now == entry["sources"]:             # built from the text the index serves right now
            return "cache", entry["text"]
    sources = ["refund-policy"]
    texts = [INDEX[s] for s in sources]
    text = generate(question, texts)
    entry = {"text": text, "sources": {s: version(t) for s, t in zip(sources, texts)}}
    remember(key, json.dumps(entry), sources, ttl)
    return "fresh", text

def edit_while_generating(ask, question, new_policy):
    global LLM_SECONDS
    LLM_SECONDS = 1.0                           # a slow LLM call...
    slow = threading.Thread(target=ask, args=(question, SCOPE))
    slow.start()
    time.sleep(0.2)                             # ...that has already read the old policy
    INDEX["refund-policy"] = new_policy         # right order: re-index,
    invalidate("refund-policy")                 # then invalidate (nothing to delete yet)
    slow.join()                                 # the slow request caches its answer
    LLM_SECONDS = 0.0
    source, text = ask(question, SCOPE)
    print(f"{ask.__name__:14} {source:5} {days(text):<7}  (the policy now says {days(new_policy)})")

edit_while_generating(answer, "What is the refund window?",
                      "Annual plans are refundable within 10 days of purchase.")
edit_while_generating(answer_checked, "What is the refund window?",
                      "Annual plans are refundable within 7 days of purchase.")

Predict: in each run the policy changes while a slow request is generating, and the invalidation runs in the right order. Which run serves an outdated number?

answer         cache 14 days  (the policy now says 10 days)
answer_checked fresh 7 days   (the policy now says 7 days)

When the check reads the index itself, correctness no longer depends on when, or whether, the deletes arrive: an entry built from text the index no longer holds cannot be served, and a deleted source fails the check too. Hash each source's access list into its version as well, and a permission change on a document (Move 3's revocation case) fails the check just like an edit. The price is one batched lookup of the sources' current content hashes per hit, from the index payload, where ingestion keeps each document's content hash (Data Pipeline & Indexing), or from a small registry the indexer updates after each write; the same lookup can carry Move 3's serve-time permission check. Keep dependency invalidation as well: it frees memory at once, and erasure needs the cached text deleted, not merely unservable. If cached answers are your fallback when search is down (Production Deployment & Optimization), read the versions from the registry rather than the index, or decide in advance whether to serve unchecked hits, marked as possibly stale, during the outage.

Now TTL alone, run for real with a 2-second TTL and no invalidation:

INDEX["refund-policy"] = "Annual plans are refundable within 30 days of purchase."
VERSIONS["index"] = "docs-v43"                  # a new index version: every old key is unreachable
answer(Q, SCOPE, ttl=2)                         # cached with a 2-second TTL; nobody calls invalidate()
INDEX["refund-policy"] = NEW                    # the policy changes right after
start = time.monotonic()
for _ in range(6):
    source, text = answer(Q, SCOPE, ttl=2)
    print(f"t={time.monotonic() - start:.1f}s  {source:5}  {days(text)}")
    time.sleep(0.5)

t=0.0s  cache  30 days
t=0.5s  cache  30 days
t=1.0s  cache  30 days
t=1.5s  cache  30 days
t=2.0s  fresh  14 days
t=2.5s  cache  14 days

The stale answer lives until the entry expires, counted from its write, not from the document change. So choose a TTL by risk: it is the longest a wrong answer may survive if every other mechanism fails. For content that rarely changes but hurts when wrong (a dosing guideline, a legal deadline), a short TTL alone is the wrong tool: use dependency invalidation, or keep the route out of the answer cache.

Details that decide whether it works

• Deliver invalidations durably. Redis Pub/Sub delivers at most once: "If the subscriber is unable to handle the message (for example, due to an error or a network disconnect) the message is forever lost." Use a durable log with acknowledgements (Redis Streams, Kafka) or versions in the key, and keep the TTL as a backstop.
• Soften version bumps (see the exercise below): finer versions, warming, or validating instead of invalidating with the source check above.
• Cache chunk ids, not chunk text. A retrieval-result cache that stores ids and fetches the text by id through the permission-filtered store picks up deletes and access changes automatically.
• Negative answers have no sources. A cached "I couldn't find anything about the Enterprise plan" depends on no document, so dependency invalidation never reaches it when the Enterprise page is finally indexed. Give negative answers short TTLs.
• Deletes reach caches too. A deleted or erased document must also leave every cached answer that quotes it: the source check stops serving those answers, and the dependency record lets you delete them (Data Freshness & Lifecycle covers the wider lifecycle).
• Gate what you write. A cached wrong answer is served until it expires. Cache only answers that passed your checks (sources cited, every claim supported by the retrieved passages as checked in Grounding & Hallucination Control, no timeout or truncation), store the sources and versions with each entry, and let a thumbs-down on a cached answer evict it and flag it for review.

Your turn: you rebuild the index nightly and publish it at 02:00 as a new version, and every answer key contains the index version. What happens to the hit rate and to Redis memory at 02:00, and what are two ways to soften it?

Move 5: A stable prefix for provider prompt caching

Symptom: input tokens are most of the bill, and the provider's usage report shows zero cached tokens on every request.

Provider prompt caching cannot serve a wrong answer. The model still generates every time; the provider only reuses its processing of a prompt prefix it saw recently, and bills that prefix at a discount. As of September 2026:

• Anthropic (prompt caching docs): mark a breakpoint with cache_control, or put one cache_control at the top level of the request and let the breakpoint move forward as a conversation grows. Writes cost 1.25× the input price with the default 5-minute lifetime, 2× with a 1-hour lifetime; reads cost 0.1× on most models (0.05× on Claude Opus 5.5, 0.025× on Claude Fable 5.1). "The cache is refreshed for no additional cost each time the cached content is used." The minimum cacheable prompt is 512 to 4,096 tokens depending on the model; shorter prompts are silently not cached.
• OpenAI (prompt caching guide): caching "is enabled by default for supported OpenAI models", from 1,024 tokens on GPT-5.6 and later, with reads at 0.1× and writes at 1.25× the uncached input rate on those models. On those models OpenAI routes requests to the cache automatically; an optional prompt_cache_key keeps separate cache accounting per customer or user and helps prevent cache-hit probing across users (on earlier models, it helped route requests that share a prefix).

Both match the prefix exactly: Anthropic hashes everything up to the breakpoint in the order tools, system, messages, so any earlier change is a miss. Caches are isolated between organizations (and, on Anthropic's API, workspaces), so another provider customer never gets your prompts. Inside your organization (or workspace) the cache is shared: it cannot mix up answers, but a cached-token count or a fast first token can reveal that someone recently sent the same prefix, which is what OpenAI's per-user keys guard against.

Lay the prompt out by volatility

[ tool definitions ][ system prompt, rules, examples ][ retrieved passages ][ question ]
  never changes        changes with a release          changes per question
  ◀────────────── cached prefix ─────────────▶ breakpoint ◀── paid in full ───▶

Retrieved passages differ between questions, so they are usually paid in full; the stable part in front of them is what the cache saves. A 5,550-token prompt with 3,000 tokens of tools and instructions bills, on steady traffic, like 3,000 × 0.1 + 2,550 = 2,850 tokens: 49% less. What breaks the prefix:

• a timestamp or request id at the top of the system prompt;
• the user's name or tenant settings before the shared instructions (put them after, or accept one prefix per tenant);
• tool definitions or few-shot examples in a different order on each request;
• retrieved passages placed before the instructions.

Passages do get cached in two cases. In a multi-turn chat, turn 1's passages are part of the history on turn 2, so an automatic breakpoint caches them, as long as you append to the history instead of rewriting it. In an agent loop, each call starts with the previous call's transcript (Agentic RAG Systems prices that). Trimming old passages from the history saves tokens and breaks the prefix, so measure which wins.

Does it pay at your traffic?

A write costs extra, so a prefix that expires before its next use costs more than no caching. With random arrivals and a TTL that each hit refreshes, the chance a request finds the prefix expired is exp(−TTL / average gap):

import math

WRITE_5M, WRITE_1H, READ = 1.25, 2.00, 0.10   # x base input price (Anthropic, Sept 2026; reads vary by model)

def prefix_multiplier(minutes_between_requests, ttl_minutes, write):
    """Average price of the cached prefix per request, as a multiple of the uncached price.
    Assumes random (Poisson) arrivals and a TTL that each hit refreshes."""
    p_expired = math.exp(-ttl_minutes / minutes_between_requests)
    return p_expired * write + (1 - p_expired) * READ

print("one request every   5-min cache   1-hour cache")
for gap in (0.25, 1, 5, 12, 20, 60, 240):
    print(f"{gap:8g} min      {prefix_multiplier(gap, 5, WRITE_5M):6.2f}x"
          f"       {prefix_multiplier(gap, 60, WRITE_1H):6.2f}x")

one request every   5-min cache   1-hour cache
    0.25 min        0.10x         0.10x
       1 min        0.11x         0.10x
       5 min        0.52x         0.10x
      12 min        0.86x         0.11x
      20 min        1.00x         0.19x
      60 min        1.16x         0.80x
     240 min        1.23x         1.58x

With the 5-minute cache, a prefix pays off on average only if requests sharing it arrive more often than about every 20 minutes; the 1-hour cache stretches that to about every 80 minutes. Count arrivals per distinct prefix: 50,000 requests a day spread over 2,000 tenant-specific system prompts is one request per prefix every 58 minutes, where the 5-minute cache costs more than no caching.

The real call is short. This sketch needs the anthropic package and an API key; the model id is an example as of September 2026:

import anthropic

MODEL = "claude-sonnet-5"        # example id as of 2026-09; check the model list
client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

def answer_with_cached_prefix(system_prompt, passages, question):
    response = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=[{"type": "text", "text": system_prompt,        # stable: rules and examples
                 "cache_control": {"type": "ephemeral"}}],      # breakpoint after the stable part
        messages=[{"role": "user",
                   "content": f"<passages>\n{passages}\n</passages>\n\nQuestion: {question}"}],
    )
    u = response.usage
    print("read from cache:", u.cache_read_input_tokens,
          "| written:", u.cache_creation_input_tokens, "| uncached:", u.input_tokens)
    return response.content[0].text

Log those three fields on every call. A read share near zero on busy traffic means something in the prefix changes per request: diff two consecutive prompts.

Your turn: requests sharing one prefix arrive on average every 3 minutes. With the 5-minute cache, what does the prefix cost per request, as a multiple of its uncached price?

Move 6: Stampedes, expiry storms and full memory

Symptom: a new plan launches at 09:00, a banner links to the assistant, and within seconds hundreds of users ask the same question. Every request misses, and every request calls the LLM.

This is a cache stampede (a thundering herd, or dog-pile). Vattani, Chierichetti and Lowenstein (PVLDB 8(8), 2015) give the arithmetic: "if the cache item is accessed 10 times per second, and the recomputation of the item takes 3 seconds, then 30 requests will recompute the item." Here are 50 concurrent requests against Redis, first with plain cache-aside, then with request coalescing: one request computes, the others wait for its result.

import threading, time, uuid
import redis

r = redis.Redis(protocol=2)
PREFIX = "rag:"
calls = 0
counter = threading.Lock()

def expensive_answer(question):
    global calls
    with counter:
        calls += 1
    time.sleep(0.5)                          # stands in for retrieval + a 0.5 s LLM call
    return f"answer to {question}".encode()

def naive_get(key, question):
    value = r.get(key)
    if value is None:                        # every request that misses recomputes
        value = expensive_answer(question)
        r.set(key, value, ex=300)
    return value

def coalesced_get(key, question, lock_ms=5_000, wait_s=3.0):
    value = r.get(key)
    if value is not None:
        return value
    token = uuid.uuid4().hex
    if r.set(key + ":lock", token, nx=True, px=lock_ms):    # exactly one request wins
        try:
            value = expensive_answer(question)
            r.set(key, value, ex=300)
            return value
        finally:
            r.execute_command("DELEX", key + ":lock", "IFEQ", token)  # release only our lock
    deadline = time.monotonic() + wait_s
    while time.monotonic() < deadline:                      # the others wait for its result
        time.sleep(0.02)
        value = r.get(key)
        if value is not None:
            return value
    return expensive_answer(question)                       # the winner died: do it yourself

def burst(get, n=50):
    global calls
    calls = 0
    key = f"{PREFIX}ans:launch-{get.__name__}"
    r.delete(key)
    t0 = time.perf_counter()
    threads = [threading.Thread(target=get, args=(key, "Is the new plan available?")) for _ in range(n)]
    for t in threads: t.start()
    for t in threads: t.join()
    print(f"{get.__name__:14} {n} requests -> {calls} LLM calls in {time.perf_counter() - t0:.2f} s")
    r.delete(key)

burst(naive_get)
burst(coalesced_get)

naive_get      50 requests -> 50 LLM calls in 0.53 s
coalesced_get  50 requests -> 1 LLM calls in 0.53 s

Same wall time, 49 fewer LLM calls. The details carry the design:

• SET … NX PX is atomic: exactly one request gets the lock, and the lock expires on its own if its holder crashes. Set lock_ms above your p99 generation time.
• The release deletes the lock only if it still holds this request's token. DELEX … IFEQ (Redis 8.4 and later) is an atomic compare-and-delete; on older servers use a small Lua script. A plain DEL could delete a lock that expired and was taken by someone else.
• Waiters give up after wait_s. During a real outage, return a "busy, try again" response instead of computing, or the stampede just arrives one timeout later.
• Inside one process, a map from key to in-flight future does the same job; across replicas you need the shared lock.

Hot keys that expire cause the same burst at every expiry. The paper's XFetch refreshes early: a request recomputes when now + Δ·β·(−ln U) ≥ expiry, where Δ is the last recomputation time, U is uniform between 0 and 1, and β = 1 by default, so busy keys are usually refreshed by one request shortly before they expire. It helps only keys already cached; a cold key at launch still needs coalescing.

Expiry storms. Entries written together expire together: warming 10,000 questions with a fixed 1-hour TTL schedules all 10,000 misses for one window an hour later, as short as the warm-up was. Add jitter: ex = int(3600 * random.uniform(0.9, 1.1)).

Memory: set a limit and a policy

Measured on Redis 8.10: a vector-set element for a 384-dimension vector took about 1,100 bytes with the default int8 quantization and 2,250 bytes unquantized (graph links and element id included); a 1,071-character cached answer took 1,353 bytes; a float32 embedding-cache entry took 1,857 bytes. A semantic entry is roughly 2.5 KB, so a million of them is about 2.5 GB.

Now the trap. Redis's eviction docs say maxmemory 0 (no limit) "is the default behavior for 64-bit systems", and under noeviction "the server will return an error when you try to execute commands that cache new data". The Redis 8.10 server used for this lesson reports exactly maxmemory 0 and noeviction. An unbounded cache grows until the host runs out of memory; a bounded one without a policy starts refusing writes. For an instance that holds only caches:

• set maxmemory below the machine's RAM, leaving room for replication and persistence buffers;
• choose allkeys-lru (the docs' suggested default when some keys are much hotter than others) or allkeys-lfu;
• avoid volatile-* policies unless every cache key has a TTL, since they "behave like noeviction if no keys have an associated expiration";
• keep data you cannot lose (dependency sets, rate-limit counters) on another instance, so eviction never touches it. One MULTI then no longer covers an answer and its dependency record, so write the record first: a crash leaves a stray id in a set, not an answer that no invalidation can find.

Watch keyspace_hits, keyspace_misses, evicted_keys and expired_keys in INFO stats: many evictions with a low hit rate mean the cache is smaller than its working set.

Measure it: the cache dashboard

A cache nobody measures drifts into serving stale or wrong answers while its hit-rate chart looks healthy. Track per layer:

Before a new layer serves anyone, replay last week's traffic through it offline: the replay gives the hit rate, a labeled sample of its hits gives the false-hit rate, and the leak fixtures give the scope check. Sampling and confidence intervals are covered in Evaluation & Quality Metrics.

Final round: no label on the problem

Challenge 1: cold after every deploy

The exact answer cache reaches a 25% hit rate during the day, falls to zero after every deploy, and stays low on replicas added by autoscaling. Keys are built as f"ans:{hash(question)}:{tenant}".

Challenge 2: the intern and the salary bands

An HR assistant caches answers per tenant. A manager asks "What is the salary band for senior engineers?" and gets an answer built from a compensation document only managers may read. An hour later an intern asks the same question and gets the same answer. Retrieval applies the permission filter correctly.

Challenge 3: the discount that never came

A team moved 4,000 tokens of instructions into the system prompt to benefit from prompt caching. On steady traffic of 20 requests a minute, the usage report still shows cache_read_input_tokens: 0. The system prompt begins Assistant for {tenant_name}. Current time: {now}.

Cheat sheet: symptom → move → cost

Build in this order, least risk first: provider prompt caching (it cannot be wrong), the document-embedding cache (content-addressed), exact caches with complete keys, and a semantic answer cache last, only on routes where you have labeled pairs, a scope and a false-hit budget.

Before moving on, take one route of your own assistant and write down every input its answer depends on, which of them your cache key contains, the scope, how a source update reaches the cache and in which order, how a hit proves its sources are current, the TTL as "the longest a wrong answer may live", and the replay, false-hit and leak tests that must pass before it serves traffic. Ship the semantic layer only on routes where the measured false-hit rate stays within budget.

Next: Performance Optimization for the rest of the latency budget, Data Freshness & Lifecycle for change detection and deletes, Infrastructure & Security for access control and cost attribution across the system, and Production Deployment & Optimization for rolling out cache and index changes safely.