Retrieval Metrics

The dashboard said 1.0

A customer asks your support bot: "Can we pay by invoice on the Team plan, and when is the invoice due?" The bot replies: "Yes, Team plans can pay by invoice. Payment is due 14 days after the invoice date." The first half is right. The second half is invented. Invoices are due in 30 days, and the "14 days" came from a chunk about retrying failed card payments.

Two chunks answer that question: billing-invoice (invoicing is available on Team and Enterprise) and billing-terms (net 30). The retriever put the first one at rank 1 and never found the second. Nobody noticed for three weeks, because the evaluation dashboard showed this question at recall@5 = 1.0. Here is the labeled set and the function behind the dashboard:

QRELS = {  # question -> the chunks that answer it
    "q1": {"billing-invoice", "billing-terms"},   # invoice on Team? when is it due?
    "q2": {"sso-plans"},                          # which plans include SSO?
    "q3": {"data-export"},                        # export all data before cancelling
    "q4": {"api-limits", "api-increase"},         # Team rate limit? how to get more?
    "q5": {"refund-annual", "refund-timing"},     # annual plans refundable? how long?
    "q6": {"audit-log"},                          # who changed a user's role?
}
RUN = {  # question -> top 5 chunks from the retriever, best first
    "q1": ["billing-invoice", "plans-team", "billing-card", "plans-compare", "refund-annual"],
    "q2": ["plans-compare", "sso-plans", "sso-setup", "plans-team", "billing-invoice"],
    "q3": ["data-export", "account-delete", "data-retention", "api-limits", "sso-setup"],
    "q4": ["api-limits", "plans-team", "plans-compare", "api-keys", "data-export"],
    "q5": ["refund-annual", "billing-card", "refund-timing", "billing-terms", "plans-compare"],
    "q6": ["roles-permissions", "sso-setup", "account-delete", "plans-team", "api-keys"],
}

def recall_at_k(ranked, relevant, k):          # the dashboard's version
    return float(any(doc in relevant for doc in ranked[:k]))

print(round(sum(recall_at_k(RUN[q], QRELS[q], 5) for q in QRELS) / len(QRELS), 3))
print(recall_at_k(RUN["q1"], QRELS["q1"], 5))

Predict first: what do the two lines print? And what should they print?

Nothing crashed. A one-line definition bug hid exactly the failure users saw. Every retrieval metric rests on four decisions, and each one can lie to you:

• what you count: relevant chunks, queries with a hit, or documents;
• at what depth: the top 5 the prompt reads, or the top 100 the reranker sees;
• against which labels: who judged what, and what happens to results nobody judged;
• over which queries, and whether a difference is bigger than the noise.

The six moves below take these in turn. Each starts from a symptom you can see:

This lesson assumes you know the labeled query set from Foundations of Modern AI Search and the failure buckets of RAG Architecture & Implementation. Rank-aware metrics in depth (MAP, nDCG variants, ERR) belong to Ranking Quality. The wider evaluation program (sampling queries from logs, synthetic questions, online tests, gates in CI) belongs to Evaluation & Quality Metrics.

Move 1: Count the right thing

Symptom: a metric says "perfect" for a question whose answer is half wrong.

Four numbers from one ranked list

Take one query with R relevant items and a ranked list of results. The course defines:

Each one is computed per query and then averaged over queries. Earlier lessons write hit rate as hit@k. trec_eval calls it success, and ranx calls it hit_rate. Here are the correct versions, run on the same six questions (the block continues from the one above):

def precision_at_k(ranked, relevant, k):
    return sum(doc in relevant for doc in ranked[:k]) / k

def recall_at_k(ranked, relevant, k):
    return sum(doc in relevant for doc in ranked[:k]) / len(relevant)

def hit_at_k(ranked, relevant, k):
    return float(any(doc in relevant for doc in ranked[:k]))

def r_precision(ranked, relevant):
    return precision_at_k(ranked, relevant, len(relevant))

for q in QRELS:
    print(q, len(QRELS[q]), precision_at_k(RUN[q], QRELS[q], 5), recall_at_k(RUN[q], QRELS[q], 5),
          hit_at_k(RUN[q], QRELS[q], 5), r_precision(RUN[q], QRELS[q]))

The six printed rows, with the means added:

Read the columns one at a time.

• Hit rate is never below recall, per query or on average. The two agree only when every query finds all of its relevant items or none of them. q1 and q4 split theirs, and that is the gap. A multi-part question needs recall, because one hit does not answer it.
• Precision has a ceiling. precision@k can never exceed min(R, k) ÷ k. q2, q3 and q6 each have one relevant chunk, so their precision@5 tops out at 0.2, and the best possible mean on this set is 0.3. A target such as "precision@5 of at least 0.5" is impossible for a bot whose questions each have one answer chunk. Report precision next to its ceiling, or count the irrelevant chunks that reach the prompt instead. Recall has a ceiling too: when R is larger than k, recall@k can be at most k ÷ R.
• R-precision sets the depth per query. q2's answer is at rank 2: perfect recall@5, but R-precision 0, because a question with one relevant chunk is judged on its first slot only. R-precision equals recall@R. It helps when R varies a lot across queries.

Decide the unit before you count

• Chunks or documents. Suppose the labels name documents and the retriever returns chunks. Three chunks of the same relevant document in the top 5 count three times toward precision@5 (0.6) and once toward document recall. Map each chunk to its document and keep only the first occurrence before you compute document-level metrics.
• Near-duplicates count twice. A help page and its PDF export are both "relevant", so precision goes up while the prompt gains nothing. Removing duplicates is the job of Context Augmentation. Your metric should not reward you for skipping it.
• Average per query, and say so. The course averages per query (a macro average). Adding up all queries' counts first (a micro average) lets queries with many relevant items dominate. One query that finds 9 of its 10 relevant items plus one that finds 0 of 1 gives a per-query mean of 0.45 but a pooled 9/11 = 0.82. The libraries in Move 4 report the per-query mean.
• Questions with no answer. A question the corpus cannot answer has R = 0, so its recall is 0 ÷ 0. Keep such questions in your set, because they test abstention. But leave them out of the retrieval averages and measure them separately. Move 4 shows what the libraries do if you forget.

Precision against recall: the cut-off decides

So far k was fixed. Many RAG systems instead keep every chunk that scores above a cut-off, so the number kept varies from question to question. Now precision and recall pull against each other. Here is the sweep procedure from Cosine Similarity & Distance Metrics, run on new numbers small enough to check by hand. There are four answerable questions and one the corpus cannot answer:

# Each question: (score, is_relevant) for its top 6 chunks, and how many relevant chunks exist.
SCORED = {
    "qa": ([(0.71, 1), (0.64, 0), (0.58, 1), (0.52, 0), (0.47, 0), (0.41, 0)], 2),
    "qb": ([(0.66, 0), (0.62, 1), (0.55, 0), (0.49, 0), (0.44, 0), (0.40, 0)], 1),
    "qc": ([(0.59, 1), (0.57, 0), (0.53, 0), (0.48, 1), (0.45, 0), (0.42, 0)], 2),
    "qd": ([(0.52, 1), (0.50, 0), (0.46, 0), (0.43, 0), (0.40, 0), (0.38, 0)], 1),
    "qe": ([(0.61, 0), (0.56, 0), (0.51, 0), (0.46, 0), (0.42, 0), (0.39, 0)], 0),  # not answerable
}

def sweep(cutoff):
    kept = rel_kept = 0
    recalls, refused, kept_unanswerable = [], 0, 0
    for scored, n_rel in SCORED.values():
        keep = [rel for score, rel in scored if score >= cutoff]
        kept += len(keep)
        rel_kept += sum(keep)
        if n_rel == 0:
            kept_unanswerable += bool(keep)
        else:
            recalls.append(sum(keep) / n_rel)
            refused += not keep
    return rel_kept / kept, sum(recalls) / len(recalls), refused, kept_unanswerable

for cutoff in (0.40, 0.50, 0.55, 0.60, 0.65):
    p, r, refused, leaked = sweep(cutoff)
    print(f"cutoff {cutoff:.2f}  precision {p:.3f}  recall {r:.3f}  "
          f"answerable with nothing kept {refused}/4  unanswerable with something kept {leaked}/1")

cutoff 0.40  precision 0.214  recall 1.000  answerable with nothing kept 0/4  unanswerable with something kept 1/1
cutoff 0.50  precision 0.333  recall 0.875  answerable with nothing kept 0/4  unanswerable with something kept 1/1
cutoff 0.55  precision 0.400  recall 0.625  answerable with nothing kept 1/4  unanswerable with something kept 1/1
cutoff 0.60  precision 0.400  recall 0.375  answerable with nothing kept 2/4  unanswerable with something kept 1/1
cutoff 0.65  precision 0.500  recall 0.125  answerable with nothing kept 2/4  unanswerable with something kept 0/1

Precision here is pooled over all kept chunks, because a question with nothing kept has no precision of its own. Say that whenever you report it.

• Raising the cut-off buys precision (0.214 → 0.500) with recall (1.000 → 0.125), and it leaves more answerable questions with nothing to answer from.
• The unanswerable question keeps a chunk until the cut-off passes 0.61. Meanwhile qd's only relevant chunk scores 0.52. No single cut-off serves both, because similarity scores are not comparable from one query to the next.
• F1 (the harmonic mean of precision and recall) squeezes this table into one number and hides which side you gave up. For a decision, keep the columns, and choose by the requirement: for example, "kept chunks at least 80% precise, with the fewest answerable questions left empty".

💡 Measure it on your data. Run the same sweep on a few hundred labeled questions, including some the corpus cannot answer. Pick the lowest cut-off that meets your precision target, and write down the refusal rate it costs. What the system should do with an empty result belongs to Grounding & Hallucination Control.

Your turn. Two systems share one index. (1) A contract-review tool must surface every clause that matters to a lawyer, who reads up to 50 results. (2) A chat assistant puts 4 chunks into its prompt, and most of its questions have exactly one answer chunk. Which metric at which depth would you gate each one on? What is the best precision@4 the assistant could score?

Move 2: Measure at the depth that reads the list

Symptom: the right chunk sits at rank 7, the prompt reads 5 chunks, and the dashboard reports recall@10.

Two depths, two readers

A two-stage retriever has two readers, and each deserves its own number:

index ──▶ first stage ──top 100──▶ reranker ──top 5──▶ prompt ──▶ answer
             recall@100                recall@5
        (can the reranker win?)   (can the LLM answer?)

• recall@k_candidates (here @100) is the reranker's ceiling. A chunk the first stage never returns cannot be promoted (Reranking Models).
• recall@k_context (here @5, after reranking) is what the LLM gets to read.

A metric at a depth that no component reads, such as recall@10 when the prompt takes 5, can stay green while answers fail. Measure at the depths your pipeline actually uses. Report both numbers whenever you change either stage. The deeper number needs labels judged that deep (Move 5).

Order inside the depth: MRR and nDCG in brief

recall@5 does not care whether the answer sits at rank 1 or rank 5. When order matters (a person clicks the first link, or the prompt is too tight to include everything), use a rank-aware metric. Two are enough here, and Ranking Quality covers them and their relatives (MAP, ERR, graded variants) in depth.

• MRR (mean reciprocal rank): for each query, 1 ÷ the rank of the first relevant item, or 0 if none appears within the cut-off; then average over queries. Always state the cut-off. MRR@10 gives 0 to a query whose first relevant item is at rank 12.
• nDCG@k: each item at rank r earns its gain × 1 / log2(r + 1). DCG@k is the sum over the top k. Divide by IDCG@k, the DCG of the ideal ordering of all judged relevant items for the query, including the ones the system never retrieved. Always state the gain: linear (the grade itself) or exponential (2^rel − 1).

Here is one query with graded labels: 3 = answers it, 2 = partly answers it, 1 = related but no answer, 0 = irrelevant.

import math

GRADES = {"key-rotate": 3, "deploy-no-downtime": 2, "key-revoke": 2, "api-keys": 1}
ranked = ["api-keys", "key-rotate", "sso-setup", "key-revoke", "api-limits"]

def dcg(gains):
    return sum(g / math.log2(rank + 1) for rank, g in enumerate(gains, start=1))

def ndcg_at_k(ranked, grades, k):
    gains = [grades.get(doc, 0) for doc in ranked[:k]]
    ideal = sorted(grades.values(), reverse=True)[:k]     # every judged item, retrieved or not
    return dcg(gains) / dcg(ideal)

print(round(dcg([GRADES.get(d, 0) for d in ranked]), 3), round(dcg([3, 2, 2, 1]), 3),
      round(ndcg_at_k(ranked, GRADES, 5), 3))
# 3.754 5.693 0.659

DCG@5 = 1/log2(2) + 3/log2(3) + 2/log2(5) = 1 + 1.893 + 0.861 = 3.754 (ranks 3 and 5 earn nothing). The ideal ordering 3, 2, 2, 1 gives IDCG@5 = 5.693, so nDCG@5 = 0.659. Two traps are hiding in this one query:

• IDCG from the retrieved items only. Sort just the five retrieved grades (3, 2, 1, 0, 0) and IDCG becomes 4.762, so nDCG rises to 0.788: the system gets rewarded for never retrieving deploy-no-downtime. The ideal ordering must include every judged relevant item. Ranking Quality works through this trap and the choice of gain in depth.
• Where "relevant" starts. If grade 1 counts as relevant, MRR for this query is 1.0, because the "related" chunk is first. If only grades 2 and 3 count, MRR is 0.5. For a RAG prompt, a chunk that is related but holds no answer usually does not help, so a threshold of 2 is often the honest one. Move 4 shows how to set it in the libraries. With exponential gain the same ranking scores 0.620, so the gain choice moves the number too.

When one hit is not enough

For a multi-part question, even recall flatters you: a question that needs two chunks and has recall 0.5 is still unanswerable from the context. So add the all-found rate@k: the share of questions whose relevant chunks are all in the top k. It is not a standard library metric; compute it as the share of queries with recall@k = 1. It is the retrieval side of what Agentic RAG Systems calls sufficiency. If you label a chunk for every fact the answer needs, the all-found rate tells you how often the context could support a complete answer. Whether the answer actually used those chunks is a question for Citation Coverage. In the hook, the all-found rate@5 is 3/6 = 0.5, while hit rate@5 said 0.833. Move 6 measures the same gap on real data.

Your turn. A pipeline retrieves 150 hybrid candidates, a cross-encoder reranks them, and the top 6 go into the prompt. The team reports "recall@10 = 0.91" and calls retrieval done. What two numbers do you ask for instead, and why is 0.91 the wrong number?

Move 3: Measure each stage, not just the system

Symptom: retrieval looks healthy and the answer is still wrong. Or the other way round.

Four different questions hide behind "is retrieval working?", and each has its own number:

ANN recall is not retrieval recall

ANN recall asks how much the index approximation costs you. Retrieval recall asks how far the whole system is from the truth. On the test bench in ANN Algorithms, brute force scored ANN recall@10 = 1.000 by definition, yet retrieval recall@10 = 0.950. HNSW with ef = 10 lost 5.4 points of ANN recall (0.946) but only 2.6 points of retrieval recall (0.924). Tuning the index can only close the gap to exact search; it cannot fix what the embedding model gets wrong with exact search. When someone reports "recall 0.99" after tuning HNSW, ask which recall they mean.

Gold-in-prompt rate

The gold-in-prompt rate is the share of labeled questions whose gold chunk text appears in the final prompt. It can be lower than recall@k_context, because the prompt assembler can drop, reorder or cut what the retriever found: a token budget, deduplication, or a filter applied late. Check it against the text of the logged prompt, not against the list of selected ids. A chunk cut off by the budget still has the right id and only half its text.

Here is a trace log of five more questions to the same bot, each with one gold chunk. They are not the hook's six, so they are numbered t1 to t5:

CHUNKS = {
    "billing-terms": "Invoices are due 30 days after the invoice date (net 30).",
    "sso-plans": "SSO with SAML is included in the Business and Enterprise plans.",
    "data-export": "Admins can export all workspace data as JSON from Settings > Export.",
    "api-increase": "To raise your API limit, open a ticket with your expected peak requests per minute.",
    "audit-log": "The audit log records every role change with the actor, time and old and new role.",
}
TRACES = {  # gold chunks, first-stage pool (top 50, cut short here), reranked top 5, logged prompt
    "t1": ({"billing-terms"}, ["billing-invoice", "billing-terms", "billing-card"],
           ["billing-invoice", "billing-terms", "plans-team", "billing-card", "plans-compare"],
           "...Invoices are due 30 days after the invoice date (net 30)...."),
    "t2": ({"sso-plans"}, ["plans-compare", "sso-setup", "sso-plans"],
           ["sso-setup", "plans-compare", "plans-team", "sso-plans", "billing-card"],
           "...SSO with SAML is included in the Business and Enter"),          # cut by the token budget
    "t3": ({"data-export"}, ["account-delete", "data-retention", "data-export"],
           ["account-delete", "data-retention", "api-limits", "sso-setup", "plans-team"],
           "...Deleting an account removes..."),
    "t4": ({"api-increase"}, ["api-limits", "api-keys", "plans-team"],
           ["api-limits", "api-keys", "plans-team", "plans-compare", "data-export"],
           "...Team plans allow 600 requests per minute..."),
    "t5": ({"audit-log"}, ["roles-permissions", "audit-log", "sso-setup"],
           ["audit-log", "roles-permissions", "sso-setup", "account-delete", "plans-team"],
           "...The audit log records every role change with the actor, time and old and new role...."),
}

def stage(gold, pool, context, prompt):
    if not gold <= set(pool):
        return "not retrieved"
    if not gold <= set(context):
        return "ranked out"
    if not all(CHUNKS[g] in prompt for g in gold):     # check the text, not the id
        return "lost in assembly"
    return "in prompt"

for tid, trace in TRACES.items():
    print(tid, stage(*trace))
n = len(TRACES)
print("recall@50", sum(t[0] <= set(t[1]) for t in TRACES.values()) / n,
      "recall@5", sum(t[0] <= set(t[2]) for t in TRACES.values()) / n,
      "gold-in-prompt", sum(stage(*t) == "in prompt" for t in TRACES.values()) / n)

t1 in prompt
t2 lost in assembly
t3 ranked out
t4 not retrieved
t5 in prompt
recall@50 0.8 recall@5 0.6 gold-in-prompt 0.4

Read it as a funnel: recall@50 = 0.8, then recall@5 = 0.6, then gold-in-prompt = 0.4. Each drop is one bucket of the failure map in RAG Architecture & Implementation, and each bucket has a different fix. t4 was never retrieved: fix the first stage. t3 was ranked out: look at the reranker or the cut-off. t2 was lost in assembly, which is usually a bug that costs almost nothing to fix. With several gold chunks per question, this code requires all of them at every stage; report the "any" version too if your questions vary.

⚠️ Same name, different metric. Some evaluation tools use "context recall" for something else. In the ragas documentation (checked September 2026), the LLM-based context recall is the share of the reference answer's claims that the retrieved context supports. Its ID-based variant is ordinary recall, and its context precision is a rank-weighted average of precision@k, not precision@k. When you report a number, name the definition.

Predict: on 200 labeled questions, each with one gold chunk, recall@5 is 0.91 and gold-in-prompt is 0.58. Where do you look first?

Move 4: Let a library count, and check it by hand

Symptom: your script says one number, the library says another, and the vendor's harness says a third.

Two widely used Python libraries do this job. As of September 2026 they are ranx (version 0.3.21 here) and pytrec_eval (the pytrec-eval-terrier package, 0.5.10 here), which wraps NIST's trec_eval. Both take labels ("qrels") as query → doc → grade and a run as query → doc → score. That is a dictionary of scores, not a ranked list. Continuing with the hook's six questions:

import pytrec_eval
from ranx import Qrels, Run, evaluate

qrels = {q: {doc: 1 for doc in docs} for q, docs in QRELS.items()}
run = {q: {doc: len(docs) - i for i, doc in enumerate(docs)} for q, docs in RUN.items()}  # 5, 4, 3, 2, 1

print(evaluate(Qrels(qrels), Run(run), ["precision@5", "recall@5", "hit_rate@5", "r-precision", "mrr@5"]))

per_query = pytrec_eval.RelevanceEvaluator(qrels, {"P_5", "recall_5", "success_5", "Rprec", "recip_rank"}).evaluate(run)
for m in ["P_5", "recall_5", "success_5", "Rprec", "recip_rank"]:
    print(m, round(sum(v[m] for v in per_query.values()) / len(per_query), 3))

Both print 0.2, 0.667, 0.833, 0.417 and 0.75 (ranx shows more decimals). These match the hand-computed table from Move 1, and the MRR is (1 + 0.5 + 1 + 1 + 1 + 0) ÷ 6. Agreement on a case you checked by hand is what lets you trust the library on 300 queries you cannot check.

Five traps, each verified in these versions

1. Ties are ordered by the library, not by your list. trec_eval re-sorts every run by score and breaks ties by document id in reverse string order. The comparator in form_res_rels.c ends with strcmp(p2->docno, p1->docno). ranx 0.3.21 kept small groups of tied documents (up to 15 in our tests) in the order you inserted them, but not larger ones: among 16 tied documents, the one inserted first came out 15th. Treat tie order in any library as undefined. With tied scores, the two libraries can report different numbers, and your own runs can change from one day to the next. Ties bite inside retrieval too: the QDrant lesson saw its MRR change between identical runs because two chunks tied at the cut. Fix it by writing strictly decreasing scores from your final ranking, as the block above does with len(docs) - i.
2. Missing queries vanish. pytrec_eval returns results only for queries that appear in the run. A query where the retriever returned nothing (a filter matched nothing, or a timeout) silently leaves your average. ranx refuses to evaluate until you pass make_comparable=True, which scores missing queries as 0:

import pytrec_eval
from ranx import Qrels, Run, evaluate

qrels = {"q1": {"a": 1}, "q2": {"b": 1}}
run = {"q1": {"a": 0.8, "m": 0.8, "z": 0.8}}   # you ranked a first; q2 returned nothing

per_query = pytrec_eval.RelevanceEvaluator(qrels, {"recip_rank"}).evaluate(run)
print(per_query)
print(sum(v["recip_rank"] for v in per_query.values()) / len(per_query))
try:
    evaluate(Qrels(qrels), Run(run), "mrr")
except AssertionError as err:
    print("ranx:", str(err)[:48])
print(evaluate(Qrels(qrels), Run(run), "mrr", make_comparable=True))

{'q1': {'recip_rank': 0.3333333333333333}}
0.3333333333333333
ranx: Qrels and Run query ids do not match. Pass `make
0.5

Both traps show up at once. trec_eval put z and m ahead of a, so q1 scored 1/3 instead of 1, and q2 disappeared from the average. The honest MRR for what you meant is (1 + 0) ÷ 2 = 0.5. ranx reached 0.5 here only because this tie group is small enough that it kept your insertion order.

3. Unanswerable questions are scored anyway. ranx gives a query with no relevant labels 0 on every metric. pytrec_eval skips a query that has no judgments at all, but scores 0 for one whose judgments are all "not relevant". Remove such questions before computing retrieval metrics (Move 1).
4. Cut-offs and thresholds are parameters. trec_eval's recip_rank has no cut-off: a first hit at rank 12 earns 1/12, while ranx's mrr@10 gives 0. Cut the run to 10 before calling pytrec_eval if you mean MRR@10. With graded labels, binary metrics count grade 1 and above as relevant by default. Pass relevance_level=2 to pytrec_eval, or add -l2 in ranx (precision@5-l2, mrr@10-l2), when "related" should not count. The libraries then disagree on nDCG. pytrec_eval ignores relevance_level for nDCG and keeps every grade as a gain, while ranx applies -l2 to nDCG too and gives grades below 2 no gain: ndcg@5-l2 scores Move 2's query 0.523 instead of 0.659. For the TREC convention (binary metrics from grade 2, every grade as an nDCG gain), put the level only on the binary metrics in ranx. Both libraries' default nDCG (ndcg in ranx, ndcg_cut in trec_eval) uses linear gain, which reproduces Move 2's 0.659. ranx's ndcg_burges uses 2^rel − 1.
5. Per-query arrays follow the library's order. evaluate(..., return_mean=False) in ranx returns an array sorted by query id as a string (q1, q10, q2), not in your order. Zip it with your own list of ids and every per-query comparison is silently misaligned. Read run.scores[metric], a dictionary keyed by query id, as Move 6 does.

🧠 Keep a hand-checked test file. Five tiny cases cover most bugs: a hit at rank 1, a miss, a multi-relevant partial, a query with an empty run, and a tie. Compute them by hand once, then assert them every time the harness changes. When the harness and your hand arithmetic disagree, the harness is wrong until proven otherwise.

A real run: two embedding models on SciFact

SciFact is a BEIR dataset: 5,183 scientific abstracts, 300 test claims used as queries, and 339 relevance labels. Earlier lessons measured it too, and this block reproduces their numbers. It compares all-MiniLM-L6-v2 with e5-small-v2, using the query: / passage: prefixes its model card asks for. It downloads 2.8 MB of data (plus the two models on first use) and embeds the corpus twice, which took about 80 seconds on an Apple M2 Max.

import io, json, urllib.request, zipfile
import numpy as np
from sentence_transformers import SentenceTransformer

URL = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip"
z = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(URL).read()))
corpus = [json.loads(line) for line in z.open("scifact/corpus.jsonl")]
queries = {q["_id"]: q["text"] for q in map(json.loads, z.open("scifact/queries.jsonl"))}
qrels = {}
for line in z.open("scifact/qrels/test.tsv").read().decode().splitlines()[1:]:
    qid, doc_id, grade = line.split("\t")
    qrels.setdefault(qid, {})[doc_id] = int(grade)

doc_ids = [d["_id"] for d in corpus]
texts = [(d["title"] + " " + d["text"]).strip() for d in corpus]   # BEIR's format
qids = list(qrels)                                                  # the 300 test queries

def dense_run(model_name, q_prefix="", d_prefix="", k=100):
    model = SentenceTransformer(model_name)
    D = model.encode([d_prefix + t for t in texts], batch_size=64, normalize_embeddings=True)
    Q = model.encode([q_prefix + queries[q] for q in qids], normalize_embeddings=True)
    S = Q @ D.T                                       # cosine, since rows are unit length
    run = {}
    for i, q in enumerate(qids):
        top = np.argsort(-S[i])[:k]
        run[q] = {doc_ids[j]: float(S[i, j]) for j in top}
    return run

runs = {"MiniLM": dense_run("sentence-transformers/all-MiniLM-L6-v2"),
        "e5-small": dense_run("intfloat/e5-small-v2", "query: ", "passage: ")}

from ranx import Qrels, Run, evaluate

METRICS = ["hit_rate@10", "recall@10", "ndcg@10", "recall@100"]
per_query = {}
for name, r in runs.items():
    run = Run(r, name=name)
    means = evaluate(Qrels(qrels), run, METRICS)
    print(name, {m: round(float(v), 3) for m, v in means.items()})
    per_query[name] = {m: dict(run.scores[m]) for m in METRICS}   # keyed by query id

MiniLM {'hit_rate@10': 0.793, 'recall@10': 0.783, 'ndcg@10': 0.645, 'recall@100': 0.925}
e5-small {'hit_rate@10': 0.817, 'recall@10': 0.806, 'ndcg@10': 0.687, 'recall@100': 0.928}

MiniLM's recall@10 of 0.783 and nDCG@10 of 0.645 match what Cosine Similarity & Distance Metrics measured. e5-small's 0.687 matches Semantic Search Principles. Reproducing someone else's number before you trust your own harness costs one run and catches most of the traps above. Moves 5 and 6 reuse runs, qrels and per_query.

Your turn. A vendor's harness reports recall@5 = 0.94 on your pilot queries. Your ranx run on the same queries and labels says 0.71. Name three explanations you could check in ten minutes.

Move 5: Build labels that hold up

Symptom: a retriever that looks better to everyone who reads its results scores worse on your labels.

The labels (qrels) are the slowest and most valuable part of all this. Earlier lessons suggest 50 to 200 labeled queries sampled from real traffic. How to sample them, and the risks of synthetic questions, belong to Evaluation & Quality Metrics. This move is about the labels themselves.

Write the judging guide before the first label

• Pick the scale. Binary ("does this chunk contain the answer or part of it?") is fast and is all that recall and hit rate need. Graded labels let nDCG separate "answers it" from "mentions it". A tested example is the TREC 2019 Deep Learning passage scale (Craswell et al., 2020). 3, perfectly relevant: "dedicated to the query and contains the exact answer". 2, highly relevant: "has some answer for the query, but the answer may be a bit unclear". 1, related: "seems related to the query but does not answer it". 0, irrelevant. The track counted only grades 2 and 3 as relevant for binary measures, and still gave "related" a gain of 1 in nDCG.
• Give examples from your own corpus, two or three per grade, especially near the boundaries.
• For multi-part questions, record which part each chunk answers. That is what makes the all-found rate (Move 2) computable.
• Label something that survives re-chunking. Chunk ids change every time you change the chunker (Smart Chunking), and qrels keyed by old ids then point at different text or at nothing. Store the document id plus the answering passage (a quote or a character span). At evaluation time, count any chunk that contains the passage as relevant.
• Version the labels with the corpus. When a policy changes, the old chunk stops being the answer (Data Freshness & Lifecycle).
• Label and run as the user. Run each labeled query with the permissions and filters the real user would have (Metadata & Filtering). A chunk this user cannot see is not an answer for them. An unfiltered evaluation run gives credit for results the user never receives.

Pooling: whose results get judged

You cannot judge every chunk for every query, so you judge a pool: the union of the top results from several systems. TREC 2019 Deep Learning started from top-10 pools across all submitted runs, then used active learning to find more relevant items. Budget it as arithmetic with stated assumptions. 100 queries, 3 systems and a top 10 each give 3,000 slots. If 60% of them are unique, that leaves 1,800 judgments, and at 30 seconds each, 15 hours of judging.

Unjudged is not irrelevant. Every metric in this lesson treats an unjudged result as not relevant. SciFact has 339 relevant labels for 300 queries and no labeled negatives, so about 91% of each model's top 10 has never been judged (judged@10 = 0.088 and 0.091, printed below). Precision@10 of about 0.09 is a floor, not a measurement. The TREC overview says the same of MS MARCO's training labels: "sparse, with no negative labels and often only one positive label per query".

Pool bias, measured. What if SciFact's labels had come only from judging MiniLM's top 10? This block keeps a label only if a pool would have surfaced it, then scores both models (it continues from Move 4's run):

def top(name, q, k=10):
    return sorted(runs[name][q], key=runs[name][q].get, reverse=True)[:k]

def judged_from(systems):          # keep only the labels a pooled judging would have found
    out = {}
    for q, labels in qrels.items():
        pool = set().union(*(top(s, q) for s in systems))
        found = {doc: g for doc, g in labels.items() if doc in pool}
        if found:                  # nothing relevant in the pool: the query drops out
            out[q] = found
    return out

for label, labels in [("full labels", qrels),
                      ("pool: MiniLM top 10", judged_from(["MiniLM"])),
                      ("pool: both top 10s", judged_from(["MiniLM", "e5-small"]))]:
    row = {name: evaluate(Qrels(labels), Run({q: r[q] for q in labels}), "recall@10")
           for name, r in runs.items()}
    print(f"{label:20} {len(labels)} queries  " + "  ".join(f"{n} {v:.3f}" for n, v in row.items()))

judged = {name: np.mean([sum(d in qrels[q] for d in top(name, q)) / 10 for q in qids]) for name in runs}
print("judged@10:", {n: round(float(v), 3) for n, v in judged.items()})

full labels          300 queries  MiniLM 0.783  e5-small 0.806
pool: MiniLM top 10  238 queries  MiniLM 1.000  e5-small 0.960
pool: both top 10s   254 queries  MiniLM 0.934  e5-small 0.963
judged@10: {'MiniLM': 0.088, 'e5-small': 0.091}

Judge only MiniLM's results, and MiniLM scores a perfect recall@10 by construction. The better model now looks worse, and the 62 queries where MiniLM found nothing relevant vanish from the set. Pooling both models restores the order, but both numbers are still inflated: relevant abstracts that neither model found lost their labels, and 46 queries dropped out entirely. Four rules follow:

• Pool from every system you intend to compare.
• When a new system arrives, judge its unjudged top k before you trust a comparison.
• Judge as deep as you score. Labels pooled from top-10 lists can score recall@10, not recall@100. Relevant chunks at ranks 11 to 100 were never judged, and every relevant chunk a system put in its own top 10 is inside its top 100 by construction, so recall@100 comes out inflated. Pool deeper (on a sample of queries if the budget is tight), or report the number as an upper bound.
• Track judged@k for each system: the share of its top k that has any label. A newcomer with a much lower judged@k than the incumbent is being scored on guesses. pytrec_eval's judged_docs_only_flag=True drops unjudged results before scoring, which is a useful cross-check but inflates absolute numbers.

LLM labels, calibrated

Labeling with an LLM is now common, with conditions. At Bing, Thomas, Spielman, Craswell and Mitra (2023) tuned a prompt against careful labels from real searchers. They found LLM labels "as good as human labellers" and better than third-party workers "for a fraction of the cost", and also that "simple paraphrases" of the prompt changed accuracy. UMBRELA (Upadhyay et al., 2024) is an open-source reproduction of that assessor, built to support the TREC 2024 RAG Track. Clarke and Dietz (2024) argue that LLM judgments still cannot replace human ones, and show a circularity risk: a system that itself uses an LLM reranker can look better to an LLM judge without being better. In practice:

• Keep a human-labeled core (100 queries is a reasonable start) as the judge's answer key. Measure the judge's agreement with it before you use the judge, and again whenever its prompt or model changes.
• Never let the judge be the model under test, and avoid judging an LLM-reranked system with the same model family.
• Let the LLM label the pools. Have people review its disagreements with the core and a random sample. Calibrating judges in general is covered in Generation Quality.

Do two judges agree?

Have two people label the same sample of pairs. Cohen's kappa is agreement corrected for the agreement you would get by chance. For graded labels, a weighted kappa gives partial credit when judges are one grade apart.

from sklearn.metrics import cohen_kappa_score

# The same 16 (question, chunk) pairs, graded 0-3 by two people
ana = [3, 2, 0, 1, 3, 0, 2, 1, 0, 3, 2, 0, 1, 2, 0, 3]
ben = [3, 1, 0, 1, 2, 0, 2, 2, 0, 3, 1, 0, 0, 2, 1, 3]

binary = lambda grades: [int(g >= 2) for g in grades]      # "answers it, fully or partly"
print("raw agreement on grades ", sum(a == b for a, b in zip(ana, ben)) / len(ana))
print("raw agreement, binary   ", sum(a == b for a, b in zip(binary(ana), binary(ben))) / len(ana))
print("kappa, binary           ", round(cohen_kappa_score(binary(ana), binary(ben)), 2))
print("kappa, grades (weighted)", round(cohen_kappa_score(ana, ben, weights="quadratic"), 2))

raw agreement on grades  0.625
raw agreement, binary    0.8125
kappa, binary            0.62
kappa, grades (weighted) 0.86

The number is less useful than the disagreements behind it. All three binary disagreements (pairs 2, 8 and 11) are a 1 against a 2: "related" against "partly answers". That boundary in the guide is unclear, so add examples for it and relabel. Sixteen pairs make a smoke test, not an estimate. Double-label 50 to 100 pairs, and compare kappa across revisions of your own guide rather than against a universal threshold.

Your turn. Last year's labels came from judging a dense retriever's top 10. You add BM25 and fuse the two lists. The hybrid's recall@10 on those labels is 0.03 lower than dense alone, yet engineers reading the results prefer the hybrid. What do you check, and what do you do before believing either number?

Move 6: Is it real, and for whom?

Symptom: a change moved an average by a few points, and someone wants to ship it.

Compare per query, in pairs

Both systems answered the same queries, so compare them query by query. The difference on a single query cancels out how hard that query is, which is usually the biggest source of noise. Three tools cover it:

• Wins, losses, ties: how many queries got better, got worse or did not change.
• A paired bootstrap confidence interval: resample the queries with replacement 10,000 times, recompute the mean difference each time, and read off the 2.5th and 97.5th percentiles.
• A paired randomization test: if the two systems were interchangeable, the sign of each per-query difference would be a coin flip. Flip the signs at random 10,000 times. The p-value is the share of those flipped means at least as far from zero as the real one.

Smucker, Allan and Carterette (CIKM 2007) compared tests on TREC runs. The randomization test, the bootstrap and the paired t-test gave similar p-values. The Wilcoxon and sign tests could both miss real differences and report false ones, and the authors recommend the randomization test. Continuing with the SciFact run from Move 4:

def paired(metric, qs=qids, a="MiniLM", b="e5-small", n=10_000, seed=0):
    d = np.array([per_query[b][metric][q] - per_query[a][metric][q] for q in qs])
    rng = np.random.default_rng(seed)
    boot = d[rng.integers(0, len(d), (n, len(d)))].mean(axis=1)   # resample whole queries
    flips = rng.choice([-1, 1], (n, len(d)))                     # a world with no difference
    p = (np.abs((flips * d).mean(axis=1)) >= abs(d.mean())).mean()
    lo, hi = np.percentile(boot, [2.5, 97.5])
    print(f"{metric} over {len(d)} queries: {d.mean():+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  "
          f"p = {p:.3f}  better {(d > 0).sum()}, worse {(d < 0).sum()}, tied {(d == 0).sum()}")

paired("recall@10")
paired("ndcg@10")

recall@10 over 300 queries: +0.022  95% CI [-0.010, +0.056]  p = 0.190  better 18, worse 10, tied 272
ndcg@10 over 300 queries: +0.042  95% CI [+0.014, +0.071]  p = 0.003  better 74, worse 48, tied 178

As a cross-check, ranx's compare(..., stat_test="fisher") (its randomization test) gave p ≈ 0.19 and ≈ 0.004 on the same runs, and SciPy's paired t-test gave 0.179 and 0.0045. Randomization p-values wobble: over 12 calls, ranx's ranged over 0.19–0.22 and 0.002–0.009 with its default 1,000 permutations, even with its default seed, and over 0.19–0.20 and 0.003–0.006 with n_permutations=10_000. Use the larger setting, and don't read the third decimal.

Same two models, same 300 queries, two different verdicts. On nDCG@10, e5-small ranks better: the interval excludes zero. On recall@10 you cannot tell: 272 of 300 queries tied, because recall@10 only moves when a relevant abstract crosses rank 10. A cut-off metric is coarse. So the decision depends on the reader from Move 2. If the prompt takes all ten abstracts and their order does not matter, this run gives no significant evidence that e5-small helps. If order matters, the nDCG@10 result is significant evidence that it does. This is the kind of check to run before you retire the old collection after a model migration (Advanced Qdrant).

How many queries?

The width of that interval depends on the spread of the per-query differences and on the number of queries. What would a smaller labeled set have told you?

d = np.array([per_query["e5-small"]["ndcg@10"][q] - per_query["MiniLM"]["ndcg@10"][q] for q in qids])
print("SD of the per-query differences:", round(d.std(ddof=1), 3))
rng = np.random.default_rng(1)
for n in (30, 50, 100):
    samples = np.array([d[rng.choice(len(d), n, replace=False)] for _ in range(5000)])
    means = samples.mean(axis=1)
    half = 1.96 * samples.std(axis=1, ddof=1) / np.sqrt(n)     # rough 95% interval
    print(f"{n:3} queries: e5-small not ahead in {(means <= 0).mean():.0%} of samples, "
          f"interval includes zero in {(means - half <= 0).mean():.0%}")

SD of the per-query differences: 0.257
 30 queries: e5-small not ahead in 17% of samples, interval includes zero in 86%
 50 queries: e5-small not ahead in 11% of samples, interval includes zero in 80%
100 queries: e5-small not ahead in 2% of samples, interval includes zero in 64%

With 30 randomly chosen labeled queries, roughly one team in six would have seen e5-small level or behind. Worse, about six teams in seven could not have shown any difference, because their 95% interval would include zero. Even with 100 queries, about two teams in three could not. The half-width of that interval is roughly 1.96 × SD ÷ √n (the block computes it per sample). With SD = 0.257 that gives about ±0.03 at 300 queries, ±0.05 at 100 and ±0.07 at 50. A true difference as large as that half-width is detected only about half the time. To detect one reliably (80% power, two-sided 5%), it has to be about 2.8 × SD ÷ √n, as in Ranking Quality: about 0.10 nDCG@10 at 50 queries for this pair of models, while the gap measured on all 300 queries is 0.042. Your SD will differ, so compute it from a pilot. For per-query metrics that are 0 or 1 (hit rate, all-found), one query moves the mean by 1 ÷ n: 0.02 at 50 queries.

Don't grade your own homework

• Tune on one split, report on another. If you pick α, a prompt, a pool size or a model by trying many options on one query set, the winner's score on that set is the best of several noisy scores, and it is inflated. Hybrid Retrieval Systems makes the same point about choosing α.
• Decide the primary metric before you look. Test 10 independent metrics or slices at p below 0.05 when nothing really changed, and you expect 10 × 0.05 = 0.5 false "significant" results (half of them false wins, half false losses), with a 1 − 0.95¹⁰ ≈ 40% chance of at least one.

Slice before you celebrate

An average can hide a slice that got worse. Slice by what your users do: identifiers against prose, language, product area, or number of relevant items. Report n and an interval for each slice. SciFact's natural slice is the number of relevant abstracts per query:

single = [q for q in qids if len(qrels[q]) == 1]
multi = [q for q in qids if len(qrels[q]) > 1]
for label, qs in [("1 relevant", single), ("2+ relevant", multi)]:
    for name in runs:
        m = {k: np.mean([per_query[name][k][q] for q in qs]) for k in METRICS}
        all_found = np.mean([per_query[name]["recall@10"][q] == 1 for q in qs])
        print(f"{label:11} n={len(qs):3} {name:8}  hit {m['hit_rate@10']:.3f}  "
              f"recall {m['recall@10']:.3f}  all found {all_found:.3f}  nDCG {m['ndcg@10']:.3f}")
paired("ndcg@10", qs=single)
paired("ndcg@10", qs=multi)

1 relevant  n=277 MiniLM    hit 0.787  recall 0.787  all found 0.787  nDCG 0.651
1 relevant  n=277 e5-small  hit 0.805  recall 0.805  all found 0.805  nDCG 0.688
2+ relevant n= 23 MiniLM    hit 0.870  recall 0.739  all found 0.609  nDCG 0.576
2+ relevant n= 23 e5-small  hit 0.957  recall 0.813  all found 0.652  nDCG 0.678
ndcg@10 over 277 queries: +0.037  95% CI [+0.007, +0.068]  p = 0.017  better 60, worse 43, tied 174
ndcg@10 over 23 queries: +0.102  95% CI [+0.041, +0.169]  p = 0.003  better 14, worse 5, tied 4

• With one relevant abstract, hit rate, recall and all-found are the same number. The whole gap lives in the 23 multi-relevant queries. MiniLM finds at least one relevant abstract for 87% of them, but all of them for only 61%. That is Move 1's hook, measured on real data.
• e5-small's gain is larger on the small slice, and even with 23 queries the interval excludes zero. A small slice can show a large effect. It cannot show a small one: its interval is twice as wide as the big slice's.
• A slice that gets worse deserves the same paired test within the slice, and then a reading of its losing queries. Reranking Models describes a reranker that raised the English average while Turkish fell.

💡 Write the decision rule before the run. For example: "Switch if nDCG@10 on the held-out set improves with a 95% interval above zero, and no slice with at least 50 queries shows a significant loss." Then run it and follow it.

Your turn. A teammate tried 12 reranker instructions on the team's 40 labeled queries. The best one beats the current instruction by +0.03 MRR@10, and they want to ship it today. What do you ask for?

Final round: no label on the problem

Real reports don't say which move they need. Read each one, name the move, and decide what to measure next.

Challenge 1: the dashboard that improved itself

After tenant filters were added, the retrieval dashboard (built on pytrec_eval) shows recall@10 rising from 0.78 to 0.83 on 200 labeled queries. Support tickets about "no results" rose the same week.

Challenge 2: better order, same answers

A new reranker raises MRR@10 from 0.62 to 0.71 on 400 queries, and the interval excludes zero. recall@5 stays at 0.74, and answer accuracy doesn't move. The prompt takes 5 chunks.

Challenge 3: the judge that liked itself

To save labeling time, the team labels relevance with the same LLM that powers the new LLM reranker. nDCG@10 of the reranked system jumps by 0.12. The old system barely moves.

Challenge 4: the recall that never moves

You try three embedding models on 300 labeled queries. recall@10 is 0.812, 0.815 and 0.809, all "within noise". Users keep complaining that answers are incomplete. 30% of your questions have three or more parts.

Cheat sheet: symptom → move → cost

Before moving on, take one metric your team reports today and write down: its exact definition (unit, depth, relevance threshold, averaging); where the labels came from and which systems' results were judged; how many queries it averages over and what one query is worth; and which component actually reads the list at that depth. If you cannot fill in all four, the number is not yet telling you what you think.

How would you measure it on your own data? Sample 50 to 200 real queries, more if you need to see small differences. Label them against a written guide, as document id plus answering passage, graded or binary, with unanswerable questions kept aside. Build the pools from every system you compare, judged as deep as the metrics you compute. Compute recall and hit rate at both depths, the all-found rate for multi-part questions, MRR@10 or nDCG@10 where order matters, and gold-in-prompt from logged prompts. Use ranx or pytrec_eval with distinct scores and hand-checked cases. Compare systems in pairs, with an interval and a randomization test on the primary metric, per slice. Re-run all of it on every change of model, chunking, fusion, filter or prompt assembly.

Next: Ranking Quality for MRR, MAP, nDCG and ERR in depth. Evaluation & Quality Metrics for building and running the whole evaluation program. Generation Quality for judging what the model does with the chunks you retrieved.