Evaluation & Quality Metrics

The four-point win that shipped the wrong torque spec

A parts distributor runs a support assistant over its product documents: spec sheets, manuals, return and shipping policies. In August the team switches to a newer embedding model with the same dimension, and they do it properly: re-embed everything into a new collection, switch the alias, keep the old collection for rollback. Their evaluation set is the 50 questions they wrote during the pilot, each labeled with the chunk that answers it. Hit rate@5, the share of questions whose labeled chunk lands in the top 5, goes from 0.80 to 0.84. It ships on Thursday.

On Monday a field technician types "torque for XK-4471 bracket bolts?" and gets back "Tighten to 40 Nm in a cross pattern [spec-XK-4417]." The XK-4471 spec sheet says 25 Nm. The old model ranked that sheet first. The new one ranks it 7th, below the XK-4417 sheet, a near-twin that differs only in the part number, the bolt size and the torque.

Predict first: is 0.80 → 0.84 on 50 questions evidence that the new model is better? What would you have wanted to see before shipping?

Nothing was wrong with the metric. Four things were wrong with how it was used:

1. The set was written by the team during the pilot, not sampled from real traffic.
2. Nobody asked how large a difference 50 questions can detect.
3. The average mixed two kinds of queries moving in opposite directions, and nobody read the 3 losses.
4. Nothing between "merged" and "users" could catch it: no per-slice gate, no online check.

RAG Architecture & Implementation gave you the minimum: labeled questions, a trace for every run, one number per stage. This lesson turns it into a practice. It is the map of the Evaluation section; the metrics themselves live in its lessons:

• Retrieval Metrics: precision, recall and hit rate at k, relevance judgments and qrels, significance tests on retrieval runs.
• Ranking Quality: MRR, MAP, nDCG and comparing two rankers.
• Generation Quality: correctness, completeness and LLM-as-judge.
• Faithfulness Testing: is every claim supported by the context?
• Citation Coverage: is each sentence backed by the passages it cites, and does each citation support its sentence?
• Grounding & Hallucination Control: abstaining and checking answers at run time.

Each move here starts from a symptom in your numbers or your process:

This lesson uses the failure buckets of RAG Architecture & Implementation (not indexed, not retrieved, ranked out, lost in assembly, ignored or contradicted). The code uses Python 3.12 with numpy, scipy, rank_bm25 and sentence-transformers. Run the blocks in order in one session, because later blocks reuse earlier names.

Before any move: write the contract of one labeled question

Every number in this lesson is computed from records like this one, so design the record before you collect a thousand of them:

item = {
    "qid": "q-0412",
    "question": "torque for XK-4471 bracket bolts?",     # exactly as the user typed it
    "history": [],                                        # earlier turns, for follow-up questions
    "source": "logs-2026-08",                             # logs | synthetic | incident
    "slices": ["identifier", "en", "specs"],
    "answerable": True,
    "gold": [{"doc_id": "spec-XK-4471", "quote": "tighten the M8 mounting bolts to 25 Nm"}],
    "reference": "25 Nm, cross pattern",                  # the facts a correct answer must state
    "labeled_by": "field-eng-2", "label_version": 3, "corpus_snapshot": "2026-08-31",
}

def gold_chunk_ids(item, chunks):
    """Resolve gold quotes to whatever chunk ids the current index uses."""
    return sorted(cid for cid, (doc_id, text) in chunks.items()
                  for g in item["gold"]
                  if doc_id == g["doc_id"] and g["quote"].lower() in text.lower())

spec = ("Bracket XK-4471. Material: galvanized steel. Mounting: tighten the M8 mounting "
        "bolts to 25 Nm in a cross pattern. Re-check after 50 operating hours.")
v1 = {"spec-XK-4471#0": ("spec-XK-4471", spec)}                         # one chunk per doc
v2 = {f"spec-XK-4471#{i}": ("spec-XK-4471", part)                        # re-chunked by sentence
      for i, part in enumerate(s + "." for s in spec.rstrip(".").split(". "))}
print(gold_chunk_ids(item, v1), gold_chunk_ids(item, v2))
# ['spec-XK-4471#0'] ['spec-XK-4471#2']

Five decisions hide in that record:

• The question as typed, typos and all, and for a follow-up the turns before it. A cleaned-up question is a different and usually easier question, and a follow-up without its history is often unanswerable.
• Gold evidence as a document id plus a quote, not a chunk id. Chunk ids change every time you re-chunk; the quote survives, and you resolve it against the current index at evaluation time, as above. A quote that straddles a new chunk boundary matches nothing, so report unresolved labels as label errors, never as retrieval misses. Keep quotes short, or match on overlap.
• A reference that lists the facts a correct answer must state, not a model's full answer. A grader can check "25 Nm" and "cross pattern"; a reference paragraph invites grading by style.
• Unanswerable items ("answerable": False, no gold). The right outcome is "not in the sources". Without them you cannot measure abstention at all.
• Provenance: source, label_version and corpus_snapshot let you report synthetic items separately, and find the labels that went stale when a document changed (Data Freshness & Lifecycle).

Move 1: Measure every stage

Symptom: an answer is wrong, and nobody can say whether the index, retrieval, the prompt builder or the model lost it.

End-to-end accuracy tells you whether the system works. Stage metrics tell you where it fails. Collect one number per stage from the same labeled questions and the same logged traces:

The course's retrieval definitions: hit rate@k is the share of questions with at least one relevant item in the top k. Recall@k is relevant items in the top k divided by all relevant items for that question, averaged over questions. MRR averages 1 / rank of the first relevant item, counting 0 when none is within the cut-off. nDCG@k divides DCG@k by an IDCG computed from all judged relevant items, including ones the system never retrieved. Offline, sufficiency is the share of questions whose logged prompt contains every key fact of the reference, checked fact by fact with a string match or a judge. At run time there is no reference, so it becomes an LLM check (Agentic RAG Systems).

Read it like a funnel

Here are 200 answerable questions from a later run of the same assistant, counted at each checkpoint:

End to end, 128 of 200 (0.64). The biggest drop is 184 → 150: 34 questions whose gold chunk was in the pool but ranked below 5th. That is where next week goes, and reading those 34 tells you which fix. Spec sheets of different parts that differ mainly in the part number all have to stay in the index; they need exact identifier matching (Hybrid Retrieval Systems) and a reranker (Reranking Models). True near-duplicate chunks and superseded revisions call for deduplication and version filters instead. A perfect generator could win back at most 18 questions; fixing ranking, up to 34.

Two more numbers belong on the same report. On 25 unanswerable questions the assistant correctly said "not in the sources" 19 times, a false answer rate of 6/25 = 0.24, and it refused 9 of the 200 answerable ones, a false refusal rate of 4.5% (Generation Quality names both). Tuning a refusal rule moves those two numbers against each other; Grounding & Hallucination Control owns that trade-off.

🧠 Faithful is not the same as correct. A faithful answer is supported by its context; a correct answer matches the reference. An answer can faithfully repeat an outdated chunk and be wrong, or be right from the model's memory with no support at all. Report both, because they fail for different reasons.

⚠️ Same name, different metric. Evaluation tools reuse names for different quantities, so always say which one you mean:

Latency and cost are quality metrics too

A right answer that takes 12 seconds, or costs ten times the budget, is still a failure. Log the time and tokens of every stage and summarize them like any other metric. The timings below are simulated in the shape of a typical RAG trace; swap in your logged values:

import numpy as np

rng = np.random.default_rng(42)
n = 2_000                                   # simulated traced queries (ms per stage)
stages = {
    "embed":    rng.lognormal(np.log(15), 0.3, n),
    "retrieve": rng.lognormal(np.log(35), 0.4, n),
    "rerank":   rng.lognormal(np.log(110), 0.35, n),
    "generate": rng.lognormal(np.log(1400), 0.45, n),
}
total = sum(stages.values())
for name, ms in {**stages, "total": total}.items():
    print(f"{name:9} mean {ms.mean():6.0f}  p50 {np.percentile(ms, 50):6.0f}  p95 {np.percentile(ms, 95):6.0f}")
print("sum of stage p95s:", round(sum(np.percentile(ms, 95) for ms in stages.values())))

# Cost per 1,000 queries (round-number assumptions, not a price quote)
PRICE_IN, PRICE_OUT = 3.00, 15.00          # USD per million input / output tokens
tokens_in, tokens_out = 2_800, 300          # per query: prompt with 5 chunks, answer
print("USD per 1,000 queries:", (tokens_in * PRICE_IN + tokens_out * PRICE_OUT) / 1e6 * 1000)

embed     mean     15  p50     15  p95     24
retrieve  mean     38  p50     35  p95     68
rerank    mean    118  p50    111  p95    198
generate  mean   1540  p50   1362  p95   2974
total     mean   1711  p50   1531  p95   3146
sum of stage p95s: 3263
USD per 1,000 queries: 12.9

Three habits come out of that table:

• Report percentiles, not the mean. The mean total (1,711 ms) sits above the median (1,531 ms), pulled up by the slow tail.
• Measure end-to-end p95 directly. Stage p95s do not add up: slow stages rarely all coincide, so the sum (3,263 ms) overstates the real total p95 (3,146 ms). Keep the per-stage numbers to know where time goes; here, generation is about 90% of the total.
• Count failures in the tail. Record timed-out and failed requests with their durations, or p95 looks better than what users see.

At the assumed prices, 12.90 USD per 1,000 queries, the same as the classic pipeline in Agentic RAG Systems. A change that buys one point of accuracy by doubling p95 or cost is a trade-off to decide, not a win to announce.

Your turn. A reranker upgrade moves hit rate@5 from 0.75 to 0.82 and the gold-in-prompt rate from 0.73 to 0.80. Correct-and-supported answers barely move: 0.64 to 0.65. Where do you look?

Move 2: Build the set from real traffic

Symptom: the set is 50 questions the team wrote during the pilot. It scores 0.84, and the support tickets say otherwise.

People who build a system write questions it can answer, in its vocabulary. Users write "torque for XK-4471 bracket bolts?"

Sample, don't write

• Start from logs. Take a few months of queries and group near-duplicates, so the most popular question is labeled once instead of 30 times. Keep each question's count, though: the headline must still weight it by how often users ask it (or report head and tail as slices). Mask personal data before anyone reads them (Infrastructure & Security), and keep the set inside your deletion process: when a user's data is erased, their logged questions leave the set too (Data Freshness & Lifecycle).
• Stratify. Tag each query with its slices: query type (identifier, question, keyword phrase), language, product area, follow-up or standalone. Sample enough per slice to measure it (Move 3), and record each slice's share of traffic to weight the headline back (Move 4).
• Keep the tail (long, misspelled, informal, rare queries) and add questions the corpus cannot answer, such as escalations where support found no document.

Label it like data you will keep for years

• Domain experts label from a written guide with examples. Two of them label the same 50 to 100 questions, and you measure their agreement (Move 5); frequent disagreement means the guide or the question is ambiguous.
• Judge what the systems return. Label the union of the top results from the current system and the candidate (pooling). A result nobody judged is unknown, not irrelevant; count it as a miss and a system that finds new relevant documents gets punished for it. Retrieval Metrics covers pooling and qrels.

Synthetic questions: useful, and flattering

An LLM can write a question for every chunk in an afternoon, covering documents nobody has asked about yet, before you have any logs. The catch: a model that writes a question while looking at a chunk borrows the chunk's words. Here are eight chunks, a real query for each as users typed it, and a question an LLM would plausibly write from the chunk:

import re
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

CHUNKS = {
    "torque-4471": "Bracket XK-4471: tighten the M8 mounting bolts to 25 Nm in a cross pattern.",
    "torque-4417": "Bracket XK-4417: tighten the M10 mounting bolts to 40 Nm in a cross pattern.",
    "returns": "Unused parts can be returned within 30 days for a full refund; opened kits get store credit.",
    "shipping-eu": "Orders to EU countries ship from Rotterdam and arrive in 3 to 5 business days.",
    "warranty": "Pumps carry a two-year warranty that covers manufacturing defects, not wear parts.",
    "seal-kit": "Seal kit SK-220 fits pump models P40 and P45; replace seals every 2,000 operating hours.",
    "invoice": "Invoices are emailed as PDF when the order ships; VAT numbers must be added before checkout.",
    "frost": "Drain the pump housing before temperatures fall below freezing to prevent cracks.",
}
REAL = {   # as users typed them
    "torque-4471": "how tight do the xk-4471 bolts go",
    "torque-4417": "XK-4417 torque?",
    "returns": "can i send back a kit i already opened",
    "shipping-eu": "how long does delivery to germany take",
    "warranty": "is a worn impeller covered",
    "seal-kit": "when do seals need changing on a P45",
    "invoice": "where do i find my invoice",
    "frost": "winter storage pump tips",
}
SYNTH = {  # written while looking at the chunk
    "torque-4471": "What torque should the M8 mounting bolts of bracket XK-4471 be tightened to?",
    "torque-4417": "To what torque are the M10 mounting bolts of bracket XK-4417 tightened?",
    "returns": "Within how many days can unused parts be returned for a full refund?",
    "shipping-eu": "From where do orders to EU countries ship, and how many business days do they take?",
    "warranty": "What does the two-year pump warranty cover?",
    "seal-kit": "Which pump models does seal kit SK-220 fit?",
    "invoice": "When are invoices emailed as PDF?",
    "frost": "Why should the pump housing be drained before temperatures fall below freezing?",
}
IDS = list(CHUNKS)
tok = lambda s: re.findall(r"[a-z0-9]+", s.lower())
bm25 = BM25Okapi([tok(CHUNKS[i]) for i in IDS])
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunk_vecs = model.encode([CHUNKS[i] for i in IDS])

def overlap(q, gold):
    q, c = set(tok(q)), set(tok(CHUNKS[gold]))
    return len(q & c) / len(q)          # share of question words found in the gold chunk

def hit_at_1(questions):
    bm = np.mean([IDS[int(np.argmax(bm25.get_scores(tok(q))))] == g for g, q in questions.items()])
    qv = model.encode(list(questions.values()))
    dense = np.mean([IDS[int(np.argmax(chunk_vecs @ v))] == g for g, v in zip(questions, qv)])
    return bm, dense

for name, qs in (("real", REAL), ("synthetic", SYNTH)):
    ov = np.mean([overlap(q, g) for g, q in qs.items()])
    bm, dense = hit_at_1(qs)
    print(f"{name:9}  word overlap {ov:.2f}  BM25 hit@1 {bm:.2f}  dense hit@1 {dense:.2f}")

real       word overlap 0.30  BM25 hit@1 0.62  dense hit@1 1.00
synthetic  word overlap 0.67  BM25 hit@1 1.00  dense hit@1 1.00

On average, two thirds of a synthetic question's words come from its answer chunk (from 0.38 to 1.00 per question), against under a third for real queries. On the synthetic set BM25 ties with dense retrieval at 1.00, and you might decide to skip the embedding model. On real queries BM25 finds 5 of 8. The three misses share nothing with the right chunk except "a" or "to", which half the chunks contain, so rank_bm25 gives them zero weight: "invoice" against "Invoices", "germany" against "EU countries", "worn impeller" against "wear parts". (Dense scores 1.00 on both because eight chunks are an easy corpus.) Eight questions prove nothing statistically; the mechanism is the point, and the overlap number is a check you can run on your own synthetic set.

The other risks of synthetic sets:

• Wrong distribution. One question per chunk, spread evenly, and no unanswerable ones unless you ask. Real traffic piles onto a few documents and asks multi-part questions.
• The same model family on both sides. Rahmani et al. (SIGIR 2024) found that LLM-built test collections (synthetic queries and judgments) could evaluate retrieval systems reliably, and studied the risk that such collections favor systems built on LLMs. Check it on yours: compare how systems rank on the synthetic set and on a real sample.
• Wrong references. Review a sample of generated reference answers before grading against them.

Use synthetic questions for coverage, smoke tests and stress slices (paraphrases, typos). Tag them "source": "synthetic", report them separately, and keep them out of the headline.

Two sets and a split

• The representative set is sampled from traffic and estimates quality. Refresh it from new logs on a schedule, and version it so every score says which set produced it.
• The regression suite holds every incident with a must-pass rule: the XK-4471 question goes in on Monday. Keep it separate; a set enriched with past failures no longer estimates production quality.
• Dev and test. Tune on a dev split and decide on a test split you do not look at while tuning, the same advice Hybrid Retrieval Systems gave for choosing α.
• Stale labels. When a policy changes, its reference answer may become wrong. Tie each label to the document version it was made from, and re-check it when that document changes.

Your turn. Your synthetic set's mean word overlap is 0.71 and your 60 log queries' is 0.24. Hybrid retrieval scores hit@5 0.93 on the synthetic set and 0.71 on the log queries. Which number goes on the dashboard, and what happens to the synthetic set?

Move 3: Put an interval on it, and compare in pairs

Symptom: +0.04 on 50 questions, and a meeting about whether it is real.

Rebuild the hook's two runs from their counts and put numbers on the doubt:

import numpy as np
from scipy.stats import binomtest

def outcomes(slice_name, both, only_new, only_old, neither):
    """Per-question rows (slice, old_hit, new_hit) rebuilt from the counts of a paired run."""
    return ([(slice_name, 1, 1)] * both + [(slice_name, 0, 1)] * only_new +
            [(slice_name, 1, 0)] * only_old + [(slice_name, 0, 0)] * neither)

RUNS = (outcomes("identifier", both=3, only_new=0, only_old=3, neither=0) +
        outcomes("natural language", both=34, only_new=5, only_old=0, neither=5))
old = np.array([o for _, o, _ in RUNS])
new = np.array([n for _, _, n in RUNS])

for name, hits in (("old", old), ("new", new)):
    ci = binomtest(int(hits.sum()), len(hits)).proportion_ci(method="wilson")
    print(f"{name}: hit rate@5 {hits.mean():.2f}, 95% CI [{ci.low:.2f}, {ci.high:.2f}]")

fixed = int(((old == 0) & (new == 1)).sum())
broken = int(((old == 1) & (new == 0)).sum())
p = binomtest(fixed, fixed + broken).pvalue           # exact McNemar test
print(f"fixed {fixed}, broken {broken}, exact paired p = {p:.2f}")

rng = np.random.default_rng(0)
idx = rng.integers(0, len(old), size=(10_000, len(old)))   # resample questions, keep pairs
diffs = new[idx].mean(axis=1) - old[idx].mean(axis=1)
print("paired bootstrap 95% CI of new - old:", np.percentile(diffs, [2.5, 97.5]).round(2))

old: hit rate@5 0.80, 95% CI [0.67, 0.89]
new: hit rate@5 0.84, 95% CI [0.71, 0.92]
fixed 5, broken 3, exact paired p = 0.73
paired bootstrap 95% CI of new - old: [-0.06  0.16]

Each system's 95% interval is more than 0.2 wide. That is what 50 questions buy: one question is worth 0.02, and the interval spans about ten of them.

Why pairing works

Both systems answered the same 50 questions, and most of the variation between questions is difficulty. The 37 questions both got right and the 5 both got wrong say nothing about which system is better. All the evidence sits in the 8 that changed. McNemar's exact test asks one question: if the systems were equally good, how surprising would a 5–3 split of the changed questions be? Not at all (p = 0.73).

The paired bootstrap resamples whole questions, both systems' outcomes together, and reads an interval off the resampled differences: anywhere from 6 points worse to 16 points better. Resampling each system on its own throws the pairing away; on these data it widens the interval to about −0.11 to +0.19.

The same +0.04 can be convincing. On 400 questions where the new system fixes 28 and breaks 12, the exact paired p is 0.017.

How many questions do you need?

It depends on how many questions a change flips. Assume 10% of questions change outcome between two versions (measure it on one real change), a two-sided test at 5% and 80% power, and the standard sample-size approximation for McNemar's test:

Changes that flip more questions need more: at 16% flipped, 400 questions detect about 5.6 points. The rule of thumb in Foundations of Modern AI Search (with 50 queries, a change of five is worth a closer look, a change of one is noise) is consistent with this table: at 50 questions, only large changes deserve attention, and even they need their losses read. Decide the smallest difference worth acting on before you label. If the answer is 900 questions and you can afford 200, accept that offline you will only see large effects, and measure small ones online (Move 6).

Scores that are not 0 or 1

For per-question scores such as nDCG@10, use a paired bootstrap or a randomization (permutation) test on the per-question differences. Smucker, Allan & Carterette (CIKM 2007) compared significance tests on TREC runs and found "little practical difference between the randomization, bootstrap, and t tests", while the Wilcoxon and sign tests could miss real differences in means or report false ones. For 0/1 outcomes McNemar's exact test is fine: it is a sign test on the changed questions, and there the mean difference is exactly the net change divided by n. Retrieval Metrics runs these tests on real retrieval runs with ranx.

Noise you make yourself

• Measure the noise floor. LLM pipelines and LLM judges are not deterministic. Run the unchanged pipeline twice (an A/A run) and count the questions whose verdict flips. If 6 of 80 flip between identical runs, a candidate that fixes 4 and breaks 2 has shown you nothing.
• Do not pick the winner on the test set. Try 20 chunk sizes on the same 200 questions and keep the best, and its lead is partly luck on those questions. Tune on the dev split and confirm the winner once on the test split.

Your turn. On 120 questions a candidate fixes 9 and breaks 2. What is the difference, what does the exact paired test say, and do you ship?

Move 4: Slice, and read the losses

Symptom: the average went up, and one kind of query broke.

Continuing from the block above, split the hook's runs by slice and weight each slice by its share of real traffic:

TRAFFIC = {"identifier": 0.30, "natural language": 0.70}   # share of real queries, from logs

for name, col in (("old", 1), ("new", 2)):
    by_slice = {s: float(np.mean([r[col] for r in RUNS if r[0] == s])) for s in TRAFFIC}
    weighted = sum(TRAFFIC[s] * by_slice[s] for s in TRAFFIC)
    shown = ", ".join(f"{s} {v:.2f}" for s, v in by_slice.items())
    print(f"{name}: {shown} | traffic-weighted {weighted:.3f}")

old: identifier 1.00, natural language 0.77 | traffic-weighted 0.841
new: identifier 0.50, natural language 0.89 | traffic-weighted 0.770

Identifiers were 6 of 50 questions (12%) in the pilot set and are 30% of traffic. Weighted by traffic, the estimate flips: 0.841 for the old model against 0.770 for the new one. On these estimates, the new model wins only while identifier queries stay below about 18.5% of traffic.

Why it works. A mean is a weighted sum over whatever mix of questions you happened to label. Two slices moving in opposite directions cancel in it, and the weights decide which one wins. Those weights should be your traffic's, not an accident of labeling.

Weighting fixes the headline. It does not fix the sample size. The slice's 3 losses out of 6 are not statistically significant (a 3–0 split of changed questions gives p = 0.25), and neither is the weighted gap of 0.07, whose standard error is also about 0.07. Neither needs to be. Read the losses:

All three are one failure: near-twin documents for different parts, which differ mainly in an identifier and its values. Both twins are correct documents and must stay in the index, so deduplication is not the fix; exact identifier matching is. That belongs to Hybrid Retrieval Systems (a lexical leg matches XK-4471 exactly). The protection belongs to Move 7: all three questions join the regression suite, next to the user's own phrasing from Monday.

Slice hygiene:

• Declare slices before you look at results, from logs and product knowledge: query type, language, product area, tenant size, answerable or not, head or tail, follow-up or standalone. Slicing after the fact until something looks bad (or good) finds noise.
• Show counts and intervals next to every slice number. A slice of 6 questions is a list to read, not a rate.
• Read losses first. Every run, print the questions that went from right to wrong before you look at any average. Semantic Search Principles put it this way: read the queries that got worse as carefully as the ones that got better. A mean cannot tell a lost typo query from a wrong torque spec.

Your turn. English has 150 questions and 80% of traffic; Turkish has 50 questions and 20%. A new model moves English from 0.82 to 0.86 and Turkish from 0.80 to 0.70. Compute the unweighted and traffic-weighted headlines. Do you ship?

Move 5: Check the judge before you trust its numbers

Symptom: the LLM judge says 91% of answers are correct. The support team's spot check says 80%.

Retrieval metrics compare ids. Generation metrics need a grader: a person, a string rule, an embedding or an LLM. Each is a measuring instrument, and has to be checked against people before its readings mean anything. Some fail that check outright. Embedding similarity to the reference cannot tell right from wrong. Against "Tighten the XK-4471 mounting bolts to 25 Nm.", all-MiniLM-L6-v2 gives the same sentence with 40 Nm a cosine of 0.97, and with "Do not" in front 0.93, but a correct rewording ("For the XK-4471 bracket, use 25 Nm on the mounting bolts.") only 0.74. An embedding measures what a text is about, and a wrong torque is about exactly the same thing as the right one. Generation Quality runs that test on word-overlap and embedding metrics, and covers the graders that do read meaning: key-fact checks and LLM judges with rubrics.

Validate an LLM judge against people

Take a random sample of answers from a real run (100 to 300), have reviewers label them, run the judge on the same answers, and count its mistakes by kind. In the symptom's run, reviewers marked 160 of 200 answers correct. The judge failed 3 of those and passed 25 of the 40 wrong ones: 160 − 3 + 25 = 182 of 200, the 91% on the dashboard. Its agreement with the reviewers is 0.86, which sounds fine until you notice that a judge that says "correct" to everything already agrees 80% of the time. Cohen's kappa corrects for chance agreement and says 0.45.

The number to act on is the false-pass rate, the share of wrong answers the judge passes: here 25 of 40. (Generation Quality reports its complement, the catch rate: the share of wrong answers the judge fails.) Tighten the rubric so the judge must find the reference's key facts ("the answer must state the torque value"), re-run it on the same 200, and track the false-pass rate directly. Rules of thumb that call a kappa "moderate" or "excellent" are labels, not pass marks: decide which error your use case can tolerate. A lenient judge is dangerous for torque specs and harmless for a tone check. For 1–5 ratings, where how far off matters, use quadratic-weighted kappa or Krippendorff's alpha; Generation Quality works through both, and shows how to correct a pass rate for a judge's known errors.

Keep the ruler fixed

• Pin the judge: model id (a dated snapshot where offered), prompt, rubric and settings, versioned with the evaluation set. A new judge is a new ruler: re-score the baseline with it and re-validate it against human labels before comparing numbers across judge versions. Do not count on temperature=0 to pin it: several current models reject a non-default temperature, and even 0 was never fully deterministic (Classic RAG Pipeline). Measure the judge's own A/A flips instead.
• Know its biases. Zheng et al. (NeurIPS 2023) documented position, verbosity and self-enhancement biases in LLM judges. Swap the order in pairwise comparisons, check whether scores rise with answer length, and validate before letting a model grade its own family.
• Correct for known error. ARES (Saad-Falcon et al., NAACL 2024) combines an automated judge with a few hundred human annotations, using prediction-powered inference, to put confidence intervals on judge-based scores.

Judging faithfulness claim by claim, and checking that citations support their sentences, are covered in Faithfulness Testing and Citation Coverage.

Move 6: Take it online

Symptom: offline says the new version is better; users do not agree.

Offline and online disagree for ordinary reasons: the labeled set is not the traffic (Move 2), labels measure relevance while users also care about speed, length and trust, and latency or the interface changed. Only real users settle it.

A/B tests

• Randomize by user or account, not per query. A conversation that switches systems halfway cannot be credited to either one, and one user's queries are correlated. Analyze at the unit you randomized.
• Pick one primary metric before you start, such as the share of conversations resolved without a support ticket within 24 hours. Add guardrails that must not get worse: p95 latency, cost per 1,000 queries, abstention rate, thumbs-down share.
• Size it up front. If 30% of conversations escalate today, detecting a drop to 28% (two-sided 5%, 80% power) needs about 8,100 conversations per arm. At 2,000 conversations a day split evenly, that is about 8 days. Round up to two full weeks, so the result covers whole weekly cycles and any novelty effect has time to fade.
• Do not stop the first time p dips below 0.05. Repeated peeking inflates false positives unless you use a sequential method designed for it.

Know your signals' biases. Thumbs are sparse and skew toward unhappy users. A reformulation (asking again within a minute) signals a miss. A click on a citation means trust or doubt. Escalations are the costliest outcome and the most honest.

Interleaving, when the output is a ranked list

If users see a list (a search page, a sources panel) and click on it, you can compare two rankers on the same query. Merge both lists into one, remember which ranker contributed each result, and credit the clicks. Team-draft interleaving (Radlinski, Kurup & Joachims, CIKM 2008) works like picking sides for a match: the ranker with fewer picks chooses next, a coin flip breaks ties, and each takes its highest-ranked result not already on the list.

import random
from scipy.stats import binomtest

def team_draft(a, b, k, rng):
    """Interleave rankings a and b (team-draft); remember which side picked each result."""
    merged, team, picks = [], {}, {"A": 0, "B": 0}
    while len(merged) < k:
        left = {s: [d for d in r if d not in team] for s, r in (("A", a), ("B", b))}
        if not left["A"] or not left["B"]:
            break
        # The side with fewer picks goes next; a coin flip breaks ties.
        if picks["A"] < picks["B"] or (picks["A"] == picks["B"] and rng.random() < 0.5):
            side = "A"
        else:
            side = "B"
        doc = left[side][0]
        merged.append(doc); team[doc] = side; picks[side] += 1
    return merged, team

def credit(clicked, team):
    a = sum(team.get(d) == "A" for d in clicked)
    b = sum(team.get(d) == "B" for d in clicked)
    return "A" if a > b else "B" if b > a else "tie"

current  = ["faq-12", "spec-4417", "spec-4471", "manual-7", "faq-3"]
new_rank = ["spec-4471", "faq-12", "spec-4471-rev", "spec-4417", "manual-7"]
rng = random.Random(7)
shown, team = team_draft(current, new_rank, k=5, rng=rng)
for pos, doc in enumerate(shown, 1):
    print(pos, doc, team[doc])
print("user clicks spec-4471 ->", credit(["spec-4471"], team))

# After a week: searches where the clicks favored one side (searches without clicks are ignored)
wins_new, wins_cur, ties = 540, 460, 1_000
print("sign test p =", round(binomtest(wins_new, wins_new + wins_cur).pvalue, 3))

1 faq-12 A
2 spec-4471 B
3 spec-4417 A
4 spec-4471-rev B
5 manual-7 B
user clicks spec-4471 -> B
sign test p = 0.012

Every user sees one ordinary-looking list, and each search with clicks is a small paired comparison. Chapelle, Joachims, Radlinski & Yue (ACM TOIS 2012) validated interleaving on two commercial search engines and arXiv.org. On their Yahoo! data they compared it with clicks on the top result, the most reliable absolute click metric there: interleaving was more sensitive for all but one pair of rankers, "significantly reducing the number of queries needed by over an order of magnitude".

⚠️ Interleaving compares rankings that users click. A generated answer is not a list. To compare two answer generators online, run an A/B test, or show two answers side by side and let users pick, which is a different interface with its own biases.

Shadow traffic

Send live queries to the candidate too, without showing its output. You get latency, cost and error rates on real traffic, plus real questions to judge offline, but no user outcomes. It pays twice for the share of traffic you mirror (all of it, if you mirror everything), and the shadow path must not write anything users can see, such as a shared cache. Rollout mechanics (canaries, index versions behind aliases) → Production Deployment & Optimization.

Move 7: Gate every change

Symptom: a prompt edit broke refund questions and shipped on a Friday.

Anything that can change an answer or its score triggers an evaluation run: the model or its version, the prompt, chunking, the embedding model, index settings, the reranker, retrieval parameters, the corpus snapshot, and the judge itself. Run in two tiers:

The full run's verdict should not be "did the mean drop?". Gate on non-inferiority: a slice passes if the lower end of a one-sided 95% interval for (new − old) stays above −0.03, meaning the candidate is probably not more than 3 points worse. Continuing with numpy, outcomes and the paired bootstrap idea from Move 3:

def paired_lower_bound(old, new, reps=10_000, seed=0):
    """5th percentile of the bootstrap distribution of mean(new - old), questions resampled in pairs."""
    rng = np.random.default_rng(seed)
    d = np.asarray(new) - np.asarray(old)
    idx = rng.integers(0, len(d), size=(reps, len(d)))
    return float(np.percentile(d[idx].mean(axis=1), 5))

def gate(runs, p95_ms, usd_per_1k, margin=0.03, min_n=40,
         p95_budget_ms=2500, usd_budget=15.0):
    failures = []
    groups = {"overall": runs}
    for s in sorted({s for s, _, _ in runs}):
        groups[s] = [r for r in runs if r[0] == s]
    for name, rows in groups.items():
        old = [o for _, o, _ in rows]; new = [n for _, _, n in rows]
        diff = np.mean(new) - np.mean(old)
        if len(rows) < min_n:
            print(f"{name:10} n={len(rows):3}  diff {diff:+.3f}  too small to gate: read its losses")
            continue
        lb = paired_lower_bound(old, new)
        verdict = "ok" if lb >= -margin else "FAIL"
        print(f"{name:10} n={len(rows):3}  diff {diff:+.3f}  lower bound {lb:+.3f}  {verdict}")
        if verdict == "FAIL":
            failures.append(name)
    if p95_ms > p95_budget_ms:
        failures.append(f"p95 {p95_ms} ms")
    if usd_per_1k > usd_budget:
        failures.append(f"cost {usd_per_1k} USD per 1,000")
    return failures

runs = (outcomes("refunds", both=44, only_new=1, only_old=9, neither=6) +
        outcomes("shipping", both=120, only_new=9, only_old=3, neither=28) +
        outcomes("products", both=136, only_new=10, only_old=4, neither=30) +
        outcomes("warranty", both=20, only_new=2, only_old=1, neither=7))
print(gate(runs, p95_ms=2140, usd_per_1k=13.2))

overall    n=430  diff +0.012  lower bound -0.012  ok
products   n=180  diff +0.033  lower bound +0.000  ok
refunds    n= 60  diff -0.133  lower bound -0.217  FAIL
shipping   n=160  diff +0.037  lower bound +0.000  ok
warranty   n= 30  diff +0.033  too small to gate: read its losses
['refunds']

The prompt edit is +0.012 overall, so a gate on the overall number would have shipped it. The refunds slice lost 9 questions and gained 1 out of 60, and it fails. Warranty has only 30 questions, so it is printed for reading instead of gated. Latency and cost sit on the same report, so a quality win that blows the p95 budget fails too.

Why "fail on any drop" is the wrong gate. Scores wobble between identical runs (Move 3's noise floor), so a gate that fails whenever the mean dips fails on changes that change nothing. Teams learn to click "re-run", and then to ignore the gate. Take the noise out where you can: cache retrieval results and answers for stages the change did not touch, pin the judge, cache verdicts keyed by (question, answer text, judge version) so only changed answers are re-judged, and set the margin above the A/A noise you measured.

What the gate itself costs. At Move 1's prices, answering 500 questions costs 6.45 USD, and two judge calls per answer (about 3,200 input and 200 output tokens each) cost 12.60 USD: judging costs twice as much as answering. Nightly for a month, that is 571.50 USD. With verdict caching and 15% of answers changing on a typical night, a run drops to about 8.34 USD.

Also run the full set on a schedule when nothing changed: providers update models behind aliases, the corpus drifts and labels go stale. Monitoring quality drift in production → Production Deployment & Optimization.

Your turn. The PR author says: "Overall is +0.012 and the refunds slice is only 60 questions. The gate is too strict." Answer them.

Tools, as of September 2026

Everything above runs in plain Python; tools add scale, dashboards and ready-made judges. Versions checked on PyPI and npm on 2026-09-25:

Three cautions. The same metric name can mean different formulas in different tools (Move 1). Every LLM-judged score depends on the judge model you configure, so pin and validate it (Move 5). And check where your questions and answers are sent before you point a hosted tool at production logs.

Final round: no label on the problem

Real evaluation problems do not say which move they need. Read the symptom, then pick.

Challenge 1: the dashboard that never moves

Hit rate@5 has read 0.93, give or take 0.01, for six months, while complaints doubled. The set was sampled from logs at launch and never refreshed. Since then a new product line launched, and 40% of queries are about it.

Challenge 2: the winning prompt

A team tries 30 prompt variants on the same 150 questions and ships the best, which scored 5 points above the current prompt. On next month's freshly sampled questions, it is level with the old prompt.

Challenge 3: great offline, flat online

A reranker gains 0.04 nDCG@10 offline. A two-week A/B test with 3,000 conversations per arm shows no change in the escalation rate (30% in both arms).

Challenge 4: the judge that loved long answers

A new prompt makes answers 60% longer. The LLM judge's helpfulness score rises from 3.6 to 4.0 on a 1–5 scale. In a blind comparison, reviewers prefer the old answers.

Cheat sheet: symptom → move → cost

Before moving on, take the last change you shipped and write down: the stage it touched and the stage metric that should have moved; how many questions changed outcome and whether that exceeds your A/A noise; which slices moved and in which direction, weighted by traffic; which judge scored it and when that judge was last checked against people; and which gate would have blocked it had it been worse. If you cannot fill in a line, that is your next piece of evaluation work.

How would you measure it on your own data? Start small and honest. Sample 100 to 200 questions from logs, stratified by the slices you care about. Label each with a document id and quote, key facts, and "none" for questions the corpus cannot answer. Run the current system twice to learn your noise floor. Then compare every change in pairs, per slice, with its latency and cost next to it.

Next: Retrieval Metrics for qrels and significance tests on real runs, Generation Quality for judging answers, and Grounding & Hallucination Control for what to do at run time when the evidence is not there.