Production Deployment & Optimization

The health check that took search down

It is 14:02 on a Tuesday. The vector database behind your support assistant restarts after a node upgrade and is gone for about half a minute. Your API was built for exactly this: when search fails, it answers from a cache of popular answers and says that search is degraded. A customer asks "How do I reset my VPN token?" and gets neither the full answer nor the degraded one. The browser shows a connection error.

The API runs as three Kubernetes pods. Each pod has a readiness endpoint, /readyz, that returns 503 when the pod cannot reach the vector database, because "a pod that can't search isn't ready" sounded reasonable in code review. Kubernetes calls it every 5 seconds and takes a pod out of rotation after 2 failures in a row.

Predict first: the database stops at t = 6 s and comes back at t = 41 s. What does a client get at t = 10 s, at t = 20 s, and just after t = 41 s?

 time    ready pods   what the client got
  3.7s   3/3          200, full answer
  6.0s   --- vector DB stopped
  9.8s   3/3          200, degraded answer (the fallback works)
 12.5s   3/3          200, degraded answer
 15.2s   0/3          connection refused
   ...   0/3          connection refused, every call until 41.3s
 40.9s   --- vector DB started
 44.3s   1/3          200, full answer
 46.9s   3/3          200, full answer

Readiness is a question about one pod: can this pod serve a request right now? A shared dependency gives every pod the same answer at the same moment, so a readiness check on it turns a partial failure into a total one, and the fallback you wrote never receives a request. Change the check so that /readyz looks only at the pod's own state (is its encoder loaded?) and rerun the same outage:

  4.6s   --- vector DB stopped
  7.2s   3/3          200, degraded answer
   ...   3/3          200, degraded answer, every call
 34.1s   --- vector DB started
 36.5s   3/3          200, full answer

Retrieval did not change at all. What changed is how the system behaves when part of it fails, and that is what this lesson is about. A notebook pipeline answers one question. A production RAG service answers thousands an hour while its dependencies fail, while its index, prompts and models change every week, while its users' questions drift, and while every token is billed. It can be down, slow, wrong or too expensive, and each of those shows up on a different dashboard, or on none.

Here is the map. Each move starts from a symptom:

This is the map of the production section. Topology, secrets, access control, tracing and incident response are in Infrastructure & Security, and Qdrant's own operations (optimizer, quantization, sharding, snapshots, metrics) in Advanced Qdrant. The lesson assumes the pipeline and failure taxonomy of RAG Architecture & Implementation.

Before any move: write the SLOs

"Is the assistant working?" has no answer until you say what working means. A service level indicator (SLI) is a measured ratio of good events to all events. A service level objective (SLO) is a target for that ratio over a window. A RAG service needs four:

Three decisions hide in that table.

• A degraded answer counts as served, and you also count it separately. The customer in the hook would rather see cached answers than a connection error, so the fallback counts toward availability. But a system that quietly serves fallbacks all day is broken, so track the fallback share as its own number and alert on it.
• For a streamed answer, latency means time to first token. Give the full answer a separate, looser target. Answers held back until a check passes (the high-risk tier in Grounding & Hallucination Control) show nothing early, so give them their own target on the full answer, checks included. Measuring per stage and cutting the time is Performance Optimization's job.
• Quality cannot be read from live traffic alone, because live traffic has no labels. It comes from a labeled set you rerun (Move 3) and from judged samples (Move 4). The metrics themselves are taught in Evaluation & Quality Metrics.

Error budgets and burn rates

An SLO of 99.5% over 30 days is also a budget: 0.5% of requests may fail. At 1.5 million requests a month, that is 7,500 failed requests. The burn rate is how fast you spend it: the current error ratio divided by 0.5%. A burn rate of 1 spends exactly the whole budget in 30 days.

Paging on "error ratio above 1% for 5 minutes" wakes people for blips that cost nothing, and it misses slow leaks. Google's SRE workbook recommends starting with three alerts, each checking two windows:

The long window proves the problem is big enough to matter. The short window proves it is still happening, so the alert stops soon after recovery. Here is the first row as Prometheus rules, together with the other SLIs. We checked them with promtool 3.5 against synthetic series. The metric names are the ones your app exports, not a library's:

groups:
- name: rag-slo
  rules:
  - record: rag:errors:ratio_rate5m
    expr: sum(rate(rag_requests_total{outcome="error"}[5m])) / sum(rate(rag_requests_total[5m]))
  - record: rag:errors:ratio_rate1h
    expr: sum(rate(rag_requests_total{outcome="error"}[1h])) / sum(rate(rag_requests_total[1h]))
  - alert: RagErrorBudgetFastBurn        # 99.5% SLO: 14.4x burn spends 2% of 30 days' budget in 1 h
    expr: rag:errors:ratio_rate1h > 14.4 * 0.005 and rag:errors:ratio_rate5m > 14.4 * 0.005
    labels: {severity: page}
  - record: rag:fallback:ratio_rate1h    # served, but degraded
    expr: sum(rate(rag_requests_total{outcome="fallback"}[1h])) / sum(rate(rag_requests_total[1h]))
  - record: rag:ttft_within_slo:ratio_rate5m
    expr: sum(rate(rag_ttft_seconds_bucket{le="1.5"}[5m])) / sum(rate(rag_ttft_seconds_count[5m]))
  - record: rag:ttft_seconds:p95_5m
    expr: histogram_quantile(0.95, sum by (le) (rate(rag_ttft_seconds_bucket[5m])))
  - record: rag:cost_usd_per_1k_requests:1d
    expr: 1000 * sum(increase(rag_cost_usd_total[1d])) / sum(increase(rag_requests_total[1d]))
  - record: release:errors:ratio_rate30m # canary vs stable, side by side (Move 3)
    expr: sum by (release) (rate(rag_requests_total{outcome="error"}[30m])) / sum by (release) (rate(rag_requests_total[30m]))

Four details decide whether these numbers mean anything:

• sum before you divide. Without it, Prometheus matches series label by label, so each error series is divided by itself. The result is 1, "100% errors", whenever any error happened at all. We checked it with promtool: the unsummed expression returned 1 where the right answer was 0.0066.
• Put a histogram bucket exactly at the SLO threshold (here le="1.5"). The share of requests within the threshold is then exact, while a p95 from histogram_quantile is interpolated between buckets. In our test data, 90% of first tokens arrived within 1.5 s and the interpolated p95 was 2.44 s. A histogram is exported as _bucket, _sum and _count series, and histogram_quantile needs the per-le rates. Called on the bare metric name, it returns nothing.
• Export every label value from the start. A labeled counter series exists only after your code first touches it, so a canary with no errors yet has no outcome="error" series, and the per-release ratio returns nothing for it instead of 0: a gap exactly where the good news should be. Create each outcome at 0 when the pod starts (in the Python client, call .labels(...) once per outcome and release). We confirmed both cases with promtool: without the zero series only the stable release appeared; with it, the canary read 0.
• Label values must be bounded: outcome, route, release. Never a user id or the query text. Every distinct value creates a new series, and query text in metrics is a privacy problem (Infrastructure & Security covers logging prompts safely).

Your turn. An internal assistant has a 99.9% availability SLO over 30 days and a steady 20,000 questions a day. How many failed requests does the monthly budget allow? What error ratio triggers the 14.4× page? What share of the budget does a 40-minute total outage spend?

Move 1: Give every hop a timeout and a fallback

Symptom: one slow or failing dependency makes every request slow or failing.

Draw the request path with a latency budget and a failure plan on every hop:

question
   │
   ▼
gateway: auth, per-user rate limit ───────────────► over the limit: 429 + Retry-After
   │
   ▼
RAG API (stateless pods)          timeout         if it fails or runs out of time
   ├─ embed the query ........     100 ms   ──►   lexical leg only
   ├─ hybrid search ..........     150 ms   ──►   answer cache, or an honest "search is down"
   ├─ rerank 100 candidates ..     350 ms   ──►   skip it, keep the first-stage order
   └─ generate, first token ..     900 ms   ──►   search mode: top passages with links
   │
   ▼
answer + citations, with outcome (ok | fallback | error), release id and a trace

The timeouts add up to 1.5 s, the time-to-first-token SLO: when a hop runs out of time, the request takes the fallback branch instead of waiting. That works only if each timeout sits above the hop's p99 under load, not at its average; otherwise the fallback becomes the normal path. The reranker's hop depends on how fast your GPU scores pairs. Performance Optimization measured a laptop GPU reranking 100 (claim, abstract) pairs of about 350 tokens in 223 ms at the median, about 150,000 tokens per second; this lesson plans with that figure until you have measured the GPU you deploy. At that speed the course's usual pool of 100 takes about 233 ms of compute, which leaves about 117 ms for waiting in the GPU's queue. A pool of 150 would take the whole 350 ms by itself, so every request that had to wait at all would skip reranking: about 60% of requests in the busiest hour on one GPU, in the queue simulation of Move 6. A skipped rerank raises no error; the answers just get worse. The throughput decides the pool you can afford: at the cautious 50,000 tokens per second assumed in the worked example of Reranking Models, 100 pairs would take 700 ms, twice the timeout. Choose the pool size on your labeled set (Reranking Models shows how), measure the reranker's speed and each hop's p99 under load, and set the timeouts from those numbers.

The fallback column is where a RAG service differs from an ordinary web service:

⚠️ A different embedding model is not a fallback. If your queries are embedded by a hosted API, it is tempting to keep a small local encoder "in case the API is down". But the index was built with the API's model, and a query vector from another model lands in another space. If the dimensions differ, every query fails loudly. If they match, nothing errors and quality drops across the board; some queries still look right, which is why nobody notices (Vector Embeddings measures it). The fallbacks that keep the index meaningful are the lexical leg, or the same model served from somewhere else.

⚠️ Do not answer a private-knowledge question from the model's memory because retrieval failed. The model has never read your documents, so its answer is a guess that looks like an answer (Agentic RAG Systems makes the same point about answers given with zero retrieval).

Timeouts, retries and breakers

• Set every timeout and retry count yourself. SDK defaults are built for long jobs. Anthropic's Python SDK, for example, defaults to a 10-minute timeout and retries timed-out requests twice, so one hung call can hold a request for about 30 minutes (the timeout × 3 attempts, plus backoff). Set both for the hop: even a 0.9 s timeout with two retries can take 2.7 s. And the SDK's timeout applies to each network operation (connect, read, write), not to the whole call, so a stream that opens at once and then sends only keep-alive ping events never trips it. Enforce the first-token budget with your own deadline on the first text event.
• Retry only what is safe, and only within the budget. Idempotent reads (search, embedding) can be retried once with a jittered backoff. On a 429, honor retry-after. A 429 without retry-after, whose error.details.error_code is enforced_spend_limit_reached, means the organization has hit its monthly spend cap (Move 6): no retry succeeds until the month ends or the cap is raised, so go straight to the fallback and page someone. Never silently restart an answer the user has already started reading.
• Stop calling a dependency that is down. A circuit breaker counts failures. After enough of them in a row it opens, and calls go straight to the fallback without waiting for a timeout. After a cooldown it lets one trial call through (half-open): success closes it, failure opens it again.

import time

class Breaker:
    """CLOSED: calls go through. After `threshold` failures in a row -> OPEN.
    OPEN: fail fast to the fallback for `cooldown` seconds, then HALF_OPEN.
    HALF_OPEN: one trial call; success -> CLOSED, failure -> OPEN again."""
    def __init__(self, threshold=3, cooldown=30.0, clock=time.monotonic):
        self.threshold, self.cooldown, self.clock = threshold, cooldown, clock
        self.state, self.failures, self.opened_at = "CLOSED", 0, 0.0

    def call(self, fn, fallback):
        if self.state == "OPEN":
            if self.clock() - self.opened_at < self.cooldown:
                return fallback(), "fallback (fast)"
            self.state = "HALF_OPEN"              # cooldown over: let one trial through
        try:
            result = fn()
        except TimeoutError:
            self.failures += 1
            if self.state == "HALF_OPEN" or self.failures >= self.threshold:
                self.state, self.opened_at = "OPEN", self.clock()
            return fallback(), "fallback (after a 10 s timeout)"
        self.state, self.failures = "CLOSED", 0
        return result, "answer"

now = 0.0
llm_up = False

def generate():
    if not llm_up:
        raise TimeoutError("no response within 10 s")
    return "generated answer"

def search_mode():
    return "top passages with links"

breaker = Breaker(threshold=3, cooldown=30, clock=lambda: now)
for t, up in [(0, False), (5, False), (10, False), (15, False), (39, False),
              (41, False), (50, True), (72, True), (80, True)]:
    now, llm_up = t, up
    _, how = breaker.call(generate, search_mode)
    print(f"t={t:>2}s  LLM {'up  ' if up else 'down'}  {how:<32} -> {breaker.state}")

Predict first: the LLM is down until t = 45 s. The simulated clock does not move during a call. What do the users at t = 15, 41 and 50 s get, and which state is the breaker in after each?

t= 0s  LLM down  fallback (after a 10 s timeout)  -> CLOSED
t= 5s  LLM down  fallback (after a 10 s timeout)  -> CLOSED
t=10s  LLM down  fallback (after a 10 s timeout)  -> OPEN
t=15s  LLM down  fallback (fast)                  -> OPEN
t=39s  LLM down  fallback (fast)                  -> OPEN
t=41s  LLM down  fallback (after a 10 s timeout)  -> OPEN
t=50s  LLM up    fallback (fast)                  -> OPEN
t=72s  LLM up    answer                           -> CLOSED
t=80s  LLM up    answer                           -> CLOSED

This sketch is single-threaded. A production breaker also admits only one trial at a time when requests arrive concurrently, keeps one breaker per dependency (and per model), and counts only outage-shaped failures: timeouts, 5xx and 429. A 400 is a bug in your request, and opening the breaker will not fix it.

Keep the pods stateless

Rollouts replace pods, the autoscaler adds and removes them, and a failed node takes several at once. Anything a pod keeps only in its own memory disappears with it. So conversation history and caches belong in a shared store with a TTL, the index lives in the vector database with its own replicas and snapshots (Advanced Qdrant), and models are loaded at startup from a pinned artifact (Move 2). Sticky sessions that send a user's follow-ups to "the pod that remembers them" break the first time that pod is replaced.

Each box in the diagram can be a managed API or something you run: per-call prices and someone else's rate limits, or fixed capacity you must keep busy and operate. Move 5 prices both, and Vector DB Selection covers the database choice. For self-hosted LLM serving in 2026, the usual open-source engines are vLLM and SGLang; Hugging Face has put its TGI server into maintenance mode and recommends them.

Move 2: Tell the orchestrator the truth

Symptom: after a deploy, the new pods restart every minute and never serve. Or, as in the hook, all pods leave the load balancer together.

Kubernetes asks each container three different questions, and a failed answer has a different consequence for each (probe docs):

Readiness runs for the whole life of the container, not only at startup. And Kubernetes does not run a Dockerfile's HEALTHCHECK instruction (Docker and Compose do); only the probes in the pod spec count.

The model that never finished loading

Loading an encoder or a reranker takes time: reading weights from disk, or downloading them on a cold node. A FastAPI app usually loads it in its lifespan handler, and the server does not open its port until that handler returns. We checked with uvicorn 0.54: every connection was refused for the whole load. So during startup, every probe fails with "connection refused".

Predict first: the model takes 60 s to load. The Deployment has no startup probe. Its liveness probe has initialDelaySeconds: 30, periodSeconds: 10 and the default failureThreshold: 3. What happens?

initialDelaySeconds is the wrong knob because it is one fixed guess for every start. Too short, and a cold node gives you the loop above. Long enough for the worst case, and liveness is blind for that long after every start, even a fast one. A startup probe ends as soon as the app answers, gives up only after its full budget, and then hands over to a sensitive liveness probe. Size its budget from your slowest measured start, a cold node that must pull the image and the weights, not from your laptop.

The endpoints behind the probes are short. The point is what they leave out:

import time
from contextlib import asynccontextmanager
from fastapi import FastAPI, Response

state = {"encoder": None}

def load_encoder():
    time.sleep(0.1)          # stands in for SentenceTransformer(MODEL, revision=PINNED_COMMIT)
    return "encoder"

@asynccontextmanager
async def lifespan(app):
    state["encoder"] = load_encoder()   # the server accepts no connections until this returns
    yield

app = FastAPI(lifespan=lifespan)

@app.get("/livez")
def livez():
    return {"ok": True}                 # the process answers; nothing else is checked

@app.get("/readyz")
def readyz(response: Response):
    ready = state["encoder"] is not None   # this pod's own state only: no Qdrant, no LLM
    response.status_code = 200 if ready else 503
    return {"ready": ready}

Because the port stays closed until the encoder is loaded, every probe fails with "connection refused" during startup, and /readyz never gets to answer 503 then. Its 503 branch matters when the pod's own state can change while the port is open: a model loaded in the background after the server starts, or a model-server sidecar in the same pod that can crash.

The Deployment that uses them, checked against the Kubernetes 1.34 API schema with kubeconform:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-api
  labels: {app: rag-api}
spec:                      # no "replicas:" here: the autoscaler in Move 6 owns the count
  selector:
    matchLabels: {app: rag-api}
  strategy:
    type: RollingUpdate
    rollingUpdate: {maxSurge: 1, maxUnavailable: 0}   # a new pod is ready before an old one goes
  template:
    metadata:
      labels: {app: rag-api}                          # must match the selector
    spec:
      terminationGracePeriodSeconds: 90               # longest streamed answer + margin
      containers:
      - name: api
        image: registry.example.com/rag-api:2026.09.25
        ports:
        - {name: http, containerPort: 8000}
        resources:
          requests: {cpu: 500m, memory: 2Gi}
          limits: {memory: 2Gi}
        startupProbe:              # up to 30 x 5 s = 150 s to load the encoder
          httpGet: {path: /livez, port: http}
          periodSeconds: 5
          failureThreshold: 30
        livenessProbe:             # "is this process stuck?" and nothing else
          httpGet: {path: /livez, port: http}
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
        readinessProbe:            # "can THIS pod serve?", not "is Qdrant up?"
          httpGet: {path: /readyz, port: http}
          periodSeconds: 5
          failureThreshold: 2
        lifecycle:
          preStop:
            sleep: {seconds: 5}    # let endpoint removal spread before SIGTERM

What the rest of it encodes:

• Rollouts never lose capacity. With maxUnavailable: 0 and maxSurge: 1, a new pod must pass readiness before an old one is removed, so a version that never becomes ready stalls while the old pods keep serving. After progressDeadlineSeconds (600 s by default) the Deployment reports ProgressDeadlineExceeded. Kubernetes does not roll back on its own; your pipeline must (kubectl rollout undo).
• Leave replicas out when an autoscaler owns the count. Otherwise every kubectl apply resets it: deploy with replicas: 3 during a peak with the autoscaler at 10 pods, and 7 pods start terminating at once. maxUnavailable governs rollouts, not the replica count, so it does not stop this. To remove the field from a running Deployment, follow the HPA docs; a plain apply drops it to 1 pod once.
• Shutdown finishes the answers in flight. A deleted pod leaves the Service while its preStop hook runs; then it gets SIGTERM, stops accepting new connections and finishes the answers it is streaming. When terminationGracePeriodSeconds runs out, it is killed. The default is 30 s, counted from before the preStop hook. A 40-second answer cut at 30 s on every rollout shows up only as "answers sometimes stop mid-sentence". The server can cut it too: uvicorn waits for open requests by default, but --timeout-graceful-shutdown cancels them when it expires (with uvicorn 0.54 we saw a stream cut after 7 of its 10 chunks). If you set it, keep it above your longest stream and below the grace period minus the preStop pause.
• Requests for scheduling, a memory limit for safety. The scheduler places pods by their requests. A memory limit restarts a leaking pod instead of letting it starve its neighbors. Many teams skip the CPU limit on latency-sensitive services, because it throttles them exactly when they are busiest.
• Keep the encoder off the event loop. A CPU-bound encode() inside an async def handler blocks every request on that worker, /livez included. Under load, liveness fails, pods restart, the survivors get more load and fail too: the cascade the Kubernetes docs warn about. Run the model in a thread pool or a separate model server.
• Pin the model. SentenceTransformer(name, revision=...) accepts a commit id. Bake the pinned weights into the image or a versioned volume, so the query encoder cannot change under a running index and a cold start needs no download.

Your turn. Which of these belong in /readyz? (a) The encoder is loaded. (b) Qdrant answers a ping. (c) The tokenizer files are on local disk. (d) The LLM provider answers a one-token test prompt. (e) The pod's worker processes are running.

Move 3: Ship every change behind a gate

Symptom: a release makes answers worse, and every dashboard stays green.

Here is a typical incident. A release changes one number: the rerank pool goes from 100 candidates to 20, to take latency off the p95. At 350 tokens per pair and 150,000 tokens per second (Move 1), 100 pairs take about 233 ms and 20 take about 47 ms, so it saves about 190 ms. It works. The p95 falls, errors stay at 0.1%, every pod stays ready. Eleven days later a support lead notices that answers about returns keep quoting the wrong policy. For "Can I return opened headphones?", the current policy chunk ranks 37th in first-stage retrieval. With a pool of 100 the reranker lifted it to first. With a pool of 20 it never sees it, and a reranker cannot recover what it never sees (Reranking Models).

Infrastructure dashboards cannot see this class of failure. In a RAG service, most changes that hurt users raise no error: a new prompt template, a smaller candidate pool, a re-chunked index, an embedding model with the same dimension, a new generation model, even a model alias you call (a name the provider points at its newest snapshot) moving to a new snapshot. Each one needs the same care as a code change.

One manifest for everything that changes answers

Pin every behavior-changing artifact in one versioned file and deploy it as a unit:

release: 2026-09-25.2
embedding: {model: BAAI/bge-small-en-v1.5, revision: "<commit sha>", query_prefix: ""}
index: {alias: support, collection: support_v14, chunker: md-headings-v3}
retrieval: {k_candidates: 100, fusion: rrf, reranker: cross-encoder/ms-marco-MiniLM-L6-v2}
generation: {model: claude-sonnet-5, prompt: answer-v12, max_tokens: 800}

Every request logs the release id, and every metric carries it as a label (release in the rules above). A rollback then means redeploying the previous manifest, not remembering which of six settings someone changed.

Four gates, cheapest first

The offline gate: count the questions that changed. Most questions get the same verdict from both versions. Only the ones that flip carry information, and an exact sign test on them tells you whether the difference is more than noise:

from scipy.stats import binomtest

def paired_gate(fixed, broken):
    """Questions only one system gets right decide the comparison (an exact sign test)."""
    p = binomtest(fixed, fixed + broken, 0.5).pvalue
    return f"fixed {fixed}, broke {broken}, net {fixed - broken:+d}, two-sided p = {p:.2f}"

print(paired_gate(14, 10))    # 300 labeled questions; the rest unchanged
print(paired_gate(40, 18))

fixed 14, broke 10, net +4, two-sided p = 0.54
fixed 40, broke 18, net +22, two-sided p = 0.01

"+4 on 300 questions" is noise: if the two versions were equally good, a split at least that lopsided would happen about half the time. 40 against 18 is evidence. Read the broken questions either way. If all 10 broken ones are about returns, that is a slice regression even when the total improves. A useful gate states its rules in advance: not worse overall on a paired test, no slice worse by more than a margin you chose, no drop in correct abstentions on unanswerable questions, and p95 and cost per 1,000 queries within budget. The sign test above is the exact McNemar test of Evaluation & Quality Metrics and Generation Quality; paired bootstrap and randomization tests for graded scores are in Retrieval Metrics.

Shadow traffic tests the candidate on questions nobody labeled. Mirror a sample of live queries to it asynchronously; compare its top-10 overlap with production, its latency, tokens and errors; judge a sample of its answers. Mirroring 10% of traffic adds 10% to the generation bill. The shadow must have no side effects (no emails, no writes to shared caches, no analytics events counted as real), and it is under the same privacy rules as production, because it sees real questions.

A canary routes by a hash of the user id, so a conversation never switches versions halfway through. Compare canary and stable by the release label: errors, time to first token, fallback share, cost per 1,000 queries, answer length, thumbs-down. Roll back automatically when a guardrail breaks. Canaries show errors, latency and cost within minutes and quality only slowly, which is why the offline gate comes first.

Index versions behind aliases

A new embedding model means a new index, and so does a chunker change you want to evaluate before users see it (a small one can also roll through the normal sync in place: Data Pipeline & Indexing). The mechanics: build a new collection, fill it, evaluate it, catch it up with the source changes made during the build (Embedding Pipeline does it at scale), switch an alias atomically (Advanced Qdrant), and keep the old collection for rollback. Vector Embeddings explains why an in-place re-embed fails silently. Production adds three catches:

• An alias is global. Pointing support at the new collection moves all traffic, not a canary's share. A canary queries the versioned collection by name (support_v15) with its matching encoder. Only promotion switches the alias.
• The encoder and the collection switch together, in one manifest. An alias switch without the query-encoder switch is the same-dimension silent failure.
• Caches must not outlive the version they came from. A release is a global change, so key cached results and answers by index and release version, or a rollback keeps serving answers from the release you just rolled back. Single document edits are handled differently, by purging the answers that used the document (Caching Strategies).

Your turn. Which gate is the first that can catch each change? (a) A new prompt makes answers 40% longer. (b) A new chunker drops the tables out of product manuals. (c) The new version leaks memory, and pods run out of it after about three hours.

Move 4: Watch quality when nothing changed

Symptom: thumbs-down votes rise for three weeks. Nobody deployed anything, and every SLO dashboard is green.

A RAG system drifts without deploys, because three of its inputs keep moving:

• Users. New product names, a new market's language, seasonal questions.
• The corpus. A connector's credentials expire and a source stops updating. A permission sync hides a department's documents. A cleanup job deletes more than it should (Data Freshness & Lifecycle).
• Providers. A model alias can move to a new snapshot under you, which changes answers silently. A pinned model id does not change, but it is retired on a published date: OpenAI announced on 2026-04-22 that gpt-4 and gpt-3.5-turbo shut down on 2026-10-23, and from that day calls to them fail. That failure is loud; the quality risk is a rushed switch to a replacement that nobody evaluated. Pin the most specific model id your provider offers, watch its deprecation page, and put every model switch through the offline gate well before the date.

No single signal sees all of that. Combine cheap signals that run on all traffic with expensive ones that run on a sample:

How big a sample? To estimate an unsupported-answer rate near 3% within ±1 percentage point at 95% confidence, you need about 1.96² × 0.03 × 0.97 / 0.01² ≈ 1,118 judged answers. Judging 1% of 50,000 questions a day gives 500 a day, so one estimate every two to three days. That is the resolution of your quality dashboard, so decide whether it is enough before you trust it. At the prices in Move 5, a judge call with 5,000 input and 200 output tokens on Claude Sonnet 5 costs USD 0.012, so judging 1% of traffic adds about USD 0.12 per 1,000 queries. Runtime checks that stop an unsupported answer before the user sees it are a different tool: Grounding & Hallucination Control.

Slice before you conclude. An average hides drift. Slice every signal by source, intent, language and tenant, and use the failure taxonomy of RAG Architecture & Implementation to decide where to look. A falling top-1 score points at retrieval. A steady score with more unsupported answers points at generation. A falling chunk count points at ingestion.

Your turn. The "not found" rate jumps from 4% to 11% overnight. There was no deploy, p95 is flat, and top-1 score percentiles are flat for most traffic. Sliced by source, the HR source went from 42,000 chunks to 0 at 02:10. What happened, and what do you add so that it cannot happen silently again?

Move 5: Price 1,000 queries

Symptom: the bill grows faster than traffic, or a feature that looked cheap in a demo is not.

Foundations of Modern AI Search promised this computation: the cost per 1,000 queries, worked out before shipping. It takes an evening with a spreadsheet, or twenty lines of Python. Two kinds of cost go in. Pay-per-use (tokens, API calls) grows with traffic. Provisioned capacity (pods, database nodes, GPUs) is billed every hour, whether anyone asks anything or not.

The assumptions below are dated 2026-09-25. Token prices are list prices: Claude Sonnet 5 at USD 2 per million input tokens and USD 10 per million output tokens, and Claude Haiku 4.5 at USD 1 and USD 5 (Anthropic pricing; Sonnet 5's price was introductory until August 2026 and has since been made standard). Token counts are as measured with Sonnet 5's tokenizer. The query encoder (the pinned BAAI/bge-small-en-v1.5 of Move 3's release manifest) runs inside the API pods, so its cost is part of theirs. Node prices are AWS on-demand Linux prices in us-east-1 as listed on instances.vantage.sh that day: c7g.xlarge (4 vCPU, 8 GiB) USD 0.145 per hour, r7g.xlarge (4 vCPU, 32 GiB) USD 0.2142, and g6.xlarge (one NVIDIA L4 GPU) USD 0.8048. Every number is a placeholder for yours.

# Cost per 1,000 queries. Every number is an assumption dated 2026-09-25: replace it with yours.
HOURS = 730                                   # hours in an average month
PRICE = {                                     # USD per million tokens (input, output), list prices
    "claude-sonnet-5": (2.00, 10.00),
    "claude-haiku-4-5-20251001": (1.00, 5.00),
}
NODES = {                                     # USD per hour, on-demand list prices, always on
    "API pods + encoder, 3 x c7g.xlarge": 3 * 0.145,
    "vector DB, 3 x r7g.xlarge": 3 * 0.2142,
    "reranker, 2 x g6.xlarge (L4 GPU)": 2 * 0.8048,
}

def tokens(model, tokens_in, tokens_out=0):   # USD per 1,000 queries
    p_in, p_out = PRICE[model]
    return 1000 * (tokens_in * p_in + tokens_out * p_out) / 1e6

def per_1000(queries_per_month=1_500_000, chunks=8, llm_rerank=0):
    prompt = 1_100 + chunks * 350 + 100       # instructions + labeled chunks + question
    bill = {"answer generation": tokens("claude-sonnet-5", prompt, 350)}
    if llm_rerank:                            # one small-model call per candidate
        bill["LLM reranker"] = tokens("claude-haiku-4-5-20251001", llm_rerank * 520, llm_rerank * 5)
    for name, usd_per_hour in NODES.items():
        if llm_rerank and "reranker" in name:
            continue                          # the LLM replaces the GPU cross-encoder
        bill[name] = usd_per_hour * HOURS / (queries_per_month / 1000)
    return bill

scenarios = {"baseline: 1.5M queries/month, 8 chunks": per_1000(),
             "20 chunks in every prompt": per_1000(chunks=20),
             "LLM reranker on 30 candidates": per_1000(llm_rerank=30),
             "traffic falls to 150k/month": per_1000(queries_per_month=150_000)}
for title, bill in scenarios.items():
    total = sum(bill.values())
    print(f"{title}: USD {total:.2f} per 1,000 queries")
    for name, usd in sorted(bill.items(), key=lambda kv: -kv[1]):
        print(f"    {name:<34} {usd:8.4f}  {usd / total:6.1%}")

baseline: 1.5M queries/month, 8 chunks: USD 12.81 per 1,000 queries
    answer generation                   11.5000   89.8%
    reranker, 2 x g6.xlarge (L4 GPU)     0.7833    6.1%
    vector DB, 3 x r7g.xlarge            0.3127    2.4%
    API pods + encoder, 3 x c7g.xlarge   0.2117    1.7%
20 chunks in every prompt: USD 21.21 per 1,000 queries
    answer generation                   19.9000   93.8%
    reranker, 2 x g6.xlarge (L4 GPU)     0.7833    3.7%
    vector DB, 3 x r7g.xlarge            0.3127    1.5%
    API pods + encoder, 3 x c7g.xlarge   0.2117    1.0%
LLM reranker on 30 candidates: USD 28.37 per 1,000 queries
    LLM reranker                        16.3500   57.6%
    answer generation                   11.5000   40.5%
    vector DB, 3 x r7g.xlarge            0.3127    1.1%
    API pods + encoder, 3 x c7g.xlarge   0.2117    0.7%
traffic falls to 150k/month: USD 24.58 per 1,000 queries
    answer generation                   11.5000   46.8%
    reranker, 2 x g6.xlarge (L4 GPU)     7.8334   31.9%
    vector DB, 3 x r7g.xlarge            3.1273   12.7%
    API pods + encoder, 3 x c7g.xlarge   2.1170    8.6%

How to read it:

• Generation is 90% of the baseline, and the query embedding is a rounding error. Here the encoder shares the three API pods, USD 0.21 for everything they do. Had you used a hosted model such as OpenAI's text-embedding-3-small (USD 0.02 per million tokens, as in Vector Embeddings), 20 tokens per query would add USD 0.0004 per 1,000 queries. Optimize where the money is.
• Long prompts multiply the bill. Twenty chunks instead of eight raise the total by 66%, because every chunk is paid on every query. First check whether the extra chunks raise the gold-in-prompt rate at all (Context Augmentation).
• A per-candidate LLM reranker more than doubles it (2.2 times), even on a pool of only 30 candidates and a model with half the generator's price, because it multiplies calls by the pool size. A cross-encoder on GPUs costs USD 0.78 per 1,000 queries here; the quality side of that choice is in Reranking Models.
• Provisioned capacity gets cheaper per query as traffic grows. At a tenth of the traffic, the same GPUs cost ten times as much per query and the total nearly doubles. Low or spiky traffic favors pay-per-use; high, steady traffic favors provisioned capacity (Vector DB Selection computes the break-even).
• Agent loops multiply model calls again. Agentic RAG Systems computed a 3-call loop at 2.4 times the cost of a single pass.

The model leaves things out: storage and backups, network egress, logs and traces, evaluation and judge calls (about USD 0.12 per 1,000 queries in Move 4), shadow traffic (Move 3), retries, and people. Add whichever are large for you, then emit the result as the rag_cost_usd_total counter from the SLO rules, computed per request from the token counts the API returns. The dashboard number and the spreadsheet number should agree within a few percent. If they do not, one of them is wrong.

The levers, and what each costs you:

• Fewer, better chunks. You risk dropping the chunk that held the answer, so measure the gold-in-prompt rate before and after.
• Cache the fixed prefix. The 1,100-token instruction block is identical on every call. Anthropic bills cache reads at 0.1× the input price on most models and 5-minute cache writes at 1.25×, and Claude Sonnet 5 caches only prefixes of at least 1,024 tokens. With steady traffic this saves up to 1,100 × 0.9 × USD 2 per million = USD 1.98 per 1,000 queries, about 15%. The retrieved chunks change on every query and are paid in full. The price is a prefix you must keep byte-for-byte stable (Caching Strategies).
• Route easy questions to a smaller model, but only after measuring its accuracy on your labeled set.
• Cache whole answers for repeated questions. A hit saves everything, but you must purge the answers that used a document once its new version is indexed (in the reverse order a request can re-cache the old answer), and never share an entry across permission boundaries (Caching Strategies).
• Batch offline work such as nightly evaluation or enrichment. Anthropic's Batch API bills 50% of the standard price, in exchange for results that arrive later.

Your turn. You move the generator from Claude Sonnet 4.6 (USD 3 input and USD 15 output per million tokens) to Claude Sonnet 5 (USD 2 and USD 10). Anthropic's pricing page says its models from Claude 4.7 on use a tokenizer that produces approximately 30% more tokens for the same text; assume 30% for your prompts and answers. A prompt of yours measured 4,000 input and 350 output tokens on Sonnet 4.6's tokenizer. What is the change in generation cost per 1,000 queries?

Move 6: Plan capacity for the peak

Symptom: on launch day, time to first token goes from 1.1 s to 9 s, the LLM provider answers with 429 errors, and the autoscaler adds pods, which makes it worse.

Capacity is set by the resource that is slowest to scale, and in a RAG service the pods are the easy part. Start from Little's law: the number of requests in the system equals the arrival rate times the time each one stays. At 20 questions per second and 6 seconds per streamed answer, 120 requests are open at any moment. Each one holds a connection and some memory, and together they set the requests and tokens per minute that the LLM provider sees.

import math

# Launch-week assumptions (dated 2026-09-26; replace with your forecast and your measurements)
PEAK_QPS = 20                 # questions per second in the busiest minute
SECONDS_IN_SYSTEM = 6.0       # mean time a request stays open (streamed answer)
PROMPT, CACHED_PREFIX, ANSWER = 4_000, 1_100, 350     # tokens per question
USD_IN, USD_OUT = 2.00, 10.00 # Claude Sonnet 5 list prices, USD per million tokens (Move 5)
RERANK_TOKENS = 100 * 350     # 100 (question, chunk) pairs of 350 tokens
GPU_TOKENS_PER_S = 150_000    # cross-encoder tokens/s on one GPU (Move 1; measure yours)
TARGET_UTIL = 0.70            # plan to run hot at 70%, not 100%
IN_FLIGHT_PER_POD = 30        # the HPA target from the load test

in_flight = PEAK_QPS * SECONDS_IN_SYSTEM                       # Little's law: L = lambda * W
pods = math.ceil(in_flight / IN_FLIGHT_PER_POD) + 1            # + 1 to survive losing a node
gpus = math.ceil(PEAK_QPS * RERANK_TOKENS / (GPU_TOKENS_PER_S * TARGET_UTIL))
print(f"in flight at peak: {in_flight:.0f} requests -> {pods} API pods, {gpus} reranker GPUs")

need = {"RPM": PEAK_QPS * 60,
        "ITPM, no caching": PEAK_QPS * 60 * PROMPT,
        "ITPM, prefix cached": PEAK_QPS * 60 * (PROMPT - CACHED_PREFIX),
        "OTPM": PEAK_QPS * 60 * ANSWER}
tiers = {"Start": (1_000, 2_000_000, 400_000), "Build": (5_000, 5_000_000, 1_000_000),
         "Scale": (10_000, 10_000_000, 2_000_000)}
for name, value in need.items():
    col = 0 if name == "RPM" else 2 if name == "OTPM" else 1
    use = "  ".join(f"{t} {value / lim[col]:5.0%}" for t, lim in tiers.items())
    print(f"{name:<20} {value:>10,}   {use}")

usd_per_question = (PROMPT * USD_IN + ANSWER * USD_OUT) / 1e6
print(f"generation spend: USD {PEAK_QPS * 3600 * usd_per_question:,.0f} per peak hour, "
      f"USD {1_500_000 * usd_per_question:,.0f} per month at Move 5's traffic")

in flight at peak: 120 requests -> 5 API pods, 7 reranker GPUs
RPM                       1,200   Start  120%  Build   24%  Scale   12%
ITPM, no caching      4,800,000   Start  240%  Build   96%  Scale   48%
ITPM, prefix cached   3,480,000   Start  174%  Build   70%  Scale   35%
OTPM                    420,000   Start  105%  Build   42%  Scale   21%
generation spend: USD 828 per peak hour, USD 17,250 per month at Move 5's traffic

Three resources, three different answers:

• API pods: 5. They are the cheapest part and scale in a minute or two.
• Reranker GPUs: 7. At 20 questions per second, 100 pairs per question need 700,000 tokens per second, which keeps 7 GPUs 67% busy. At Move 5's traffic (1.5 million questions a month: about 0.6 per second on average, perhaps 1.7 in the busiest hour), one GPU would be only 40% busy, yet the simulation below sends 27% of its reranks past the 350 ms timeout; a second GPU brings that to about 2% in the busiest hour and 0.2% at average traffic, which is why Move 5 pays for two. Options for the launch: rent GPUs for the week, use a smaller reranker, or shrink the pool (50 pairs need 4 GPUs, 30 pairs need 2). Put a smaller pool through the offline gate first, because Move 3 showed what one can cost.
• The LLM provider's limits. For Claude Sonnet 5, Anthropic's rate-limit page listed on 2026-09-26: Start tier 1,000 requests, 2,000,000 input tokens and 400,000 output tokens per minute (RPM, ITPM, OTPM); Build tier 5,000, 5,000,000 and 1,000,000; Scale tier 10,000, 10,000,000 and 2,000,000. The limits are per organization and per model, and they are enforced as a token bucket, so a burst can hit them within the minute. For most current models, cache reads do not count toward ITPM. On Start, all three limits break. On Build, uncached input sits at 96%, and caching the fixed prefix brings it down to 70%.

The per-minute table hides a fourth limit. Each tier also has a monthly spend cap: USD 500 on Start, USD 1,000 on Build and USD 200,000 on Scale, and none on the Custom tier. At USD 828 an hour (USD 685 with the prefix cached), a Build organization hits its cap within 70 to 90 minutes of peak traffic, even if it had spent nothing yet that month, and an ordinary month at Move 5's traffic costs 17 times that cap. Once the cap is reached, the API answers 429 until the first day of the next month, with no retry-after header (Move 1). So this workload needs the Scale tier or a custom agreement, where the launch peak uses at most 48% of any per-minute limit. The same page lists two more traps. A sharp jump in traffic can hit acceleration limits, 429s below your tier's limits, so ramp a launch up gradually instead of switching everyone on at once. And a new organization may start in an Evaluation tier, below Start. Arrange the tier weeks before a launch; no autoscaler can raise it.

Utilization is not latency. A hop with a timeout is sized by its queue, not by its average load. This simulation sends Poisson arrivals to GPUs that each score one request at a time in a fixed time. That is a best case, because real service times vary:

import heapq, random

def share_late(qps, service_s, gpus, timeout_s=0.350, n=200_000, seed=7):
    """Poisson arrivals, fixed service time, first free GPU first: share of requests past the timeout."""
    rng, now, free, late = random.Random(seed), 0.0, [0.0] * gpus, 0
    for _ in range(n):
        now += rng.expovariate(qps)                   # the next question arrives
        start = max(now, heapq.heappop(free))         # it waits for the first free GPU
        heapq.heappush(free, start + service_s)
        late += start - now > timeout_s - service_s   # waited longer than the timeout leaves room for
    return late / n

for pairs, qps, gpus in [(150, 1.7, 1), (100, 1.7, 1), (100, 1.7, 2), (100, 0.6, 2), (100, 20, 7),
                         (100, 20, 8), (50, 20, 4), (30, 20, 2)]:
    service = pairs * 350 / 150_000                   # 350-token pairs at 150,000 tokens/s
    print(f"{pairs:>3} pairs, {qps:>4} q/s, {gpus} GPU(s): {qps * service / gpus:4.0%} busy, "
          f"{share_late(qps, service, gpus):5.1%} of reranks past 350 ms")

150 pairs,  1.7 q/s, 1 GPU(s):  60% busy, 59.6% of reranks past 350 ms
100 pairs,  1.7 q/s, 1 GPU(s):  40% busy, 26.5% of reranks past 350 ms
100 pairs,  1.7 q/s, 2 GPU(s):  20% busy,  1.9% of reranks past 350 ms
100 pairs,  0.6 q/s, 2 GPU(s):   7% busy,  0.2% of reranks past 350 ms
100 pairs,   20 q/s, 7 GPU(s):  67% busy,  3.1% of reranks past 350 ms
100 pairs,   20 q/s, 8 GPU(s):  58% busy,  0.6% of reranks past 350 ms
 50 pairs,   20 q/s, 4 GPU(s):  58% busy,  0.0% of reranks past 350 ms
 30 pairs,   20 q/s, 2 GPU(s):  70% busy,  0.4% of reranks past 350 ms

A pool of 150 under a 350 ms timeout skips the reranker for 60% of requests in an ordinary busy hour on one GPU (Move 1). With 100 pairs, seven GPUs at 67% still send about 3% of launch-day reranks down the silent fallback, and an eighth GPU brings that to 0.6%. Either rent the eighth or accept the 3% knowingly and watch the reranker's fallback share, and measure the real p99 under load before you trust any of these numbers.

Autoscale on the right signal

The Horizontal Pod Autoscaler computes desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric), ignores changes within a 10% tolerance, and by default applies a 300-second stabilization window before scaling down (HPA docs). A RAG API spends most of its time waiting for other services, so CPU is a poor signal for it. Requests in flight per pod is a better one:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-api
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: rag-api}
  minReplicas: 3
  maxReplicas: 12              # more pods do not raise the LLM provider's rate limit
  metrics:
  - type: Pods
    pods:
      metric: {name: rag_inflight_requests}   # needs a custom-metrics adapter
      target: {type: AverageValue, averageValue: "30"}

A per-pod custom metric reaches the autoscaler through the custom.metrics.k8s.io API, which needs an adapter such as Prometheus Adapter (KEDA does the same job through its own objects). This manifest and the Deployment in Move 2 pass kubeconform against the Kubernetes 1.34 schema; the Deployment leaves replicas out because this autoscaler owns it.

Predict first: four pods average 18% CPU and 70 requests in flight each, and time to first token is 7 s. What does an HPA targeting 70% CPU do? What does the one above do?

When the provider is the ceiling, add backpressure, not pods. Limit concurrent LLM calls so that the calls and tokens per minute stay inside your limits. Little's law converts one into the other: calls per minute = concurrent calls × 60 / seconds per call. At about 6 s per call, 120 concurrent calls make 1,200 calls and 4.8 million input tokens a minute, just under Build's 5 million. A per-pod cap multiplies with the replica count, so size it for maxReplicas, not for today's pods: 120 over 12 pods is 10 per pod, which leaves only 30 calls when the autoscaler runs its minimum of 3. A shared limiter, for example a token bucket in Redis, uses the whole limit at any pod count, at the price of one more dependency. Queue briefly with a deadline, and when the deadline passes, return 429 with Retry-After to your client instead of letting it time out. Keep batch work such as evaluation runs and re-enrichment out of the interactive budget: run it through the Batch API, which has its own rate limits, or give it a separate workspace with a lower limit (Anthropic supports per-workspace limits). A workspace limit caps what the batch job can take; it adds no capacity, because the organization's limits still apply to the total.

Load-test the real shape. Test at twice the forecast peak, with a realistic mix of questions and answer lengths. For the infrastructure test, use a stub LLM that streams tokens at your measured speed: a load test against the real provider is expensive and hits the very rate limit you are trying to plan around. Add a smaller test against the real provider to check the limit headroom. Measuring tail latency under load is covered in Performance Optimization.

Final round: no label on the incident

Real incidents do not say which move they need. For each one, name the move, the first thing you would check and the fix, with its cost.

Incident 1: the Monday cliff. Every Monday from 09:00 to 09:20, time to first token rises from 1.1 s to 9 s. The logs show 429 responses from the LLM provider with retry-after headers. The autoscaler goes from 4 pods to 12. A weekly evaluation job, run with the same API key, also starts at 09:00.

Incident 2: the rollout that never finished. After the node pool moves to a new instance type, a rollout has been stuck for 12 minutes. The new pods show 4 restarts each, and their events say Startup probe failed: connection refused. Their logs show downloading model weights (3.1 GB). The old pods are still serving.

Incident 3: quality slid, nothing shipped. Over two weeks the thumbs-down rate went from 3% to 6%. There were no deploys, and availability and latency look normal. The overall median top-1 retrieval score drifted from 0.60 to 0.55. Sliced by language, Spanish questions, now 30% of traffic (up from 2%), sit at a median of 0.44. Marketing launched in Mexico two weeks ago, and your encoder is English-only.

Incident 4: the canary that looked free. A new prompt ran as a 5% canary for two hours. Errors, time to first token and fallback share were identical to stable, so the team promoted it. A week later the daily cost alert fires: cost per 1,000 queries is up 25%. Answers now average 670 output tokens instead of 350.

Cheat sheet: symptom → move → cost

How would you measure it on your own data?

• Availability and latency: the SLO rules above, per release, plus a monthly drill that stops each dependency in staging and records what the client sees.
• Quality: the labeled set on every release (paired wins and losses, per slice), a judged sample of live answers every day, and human review of about 50 answers a week.
• Cost: cost per 1,000 queries per day and per release, compared with your spreadsheet.
• Capacity: a load test at twice the forecast peak, recording the in-flight requests per pod at which time to first token breaks its SLO, the rate-limit headroom at that load, and how many hours of that load your monthly spend cap would last.

Before moving on, take a RAG service you know and write down, from memory: its four SLOs, what each hop does when its dependency is down, what its readiness probe checks, which gate the last prompt change went through, what it costs per 1,000 queries, and which limit breaks first at twice today's peak. Every blank is a place to look.

Next: Performance Optimization to find and cut latency, Caching Strategies for every cache layer and its invalidation, and Infrastructure & Security for topology, access control, tracing and incident response.