Ranking Quality

The checklist that outranked the how-to

You run search for a cloud platform's developer docs. The same ranked list feeds the search box on the docs site and the in-product assistant. Last month the team replaced BM25 with hybrid retrieval plus a cross-encoder reranker. The offline report looked great: MRR@10 up sharply and recall@10 no worse. It shipped.

Two weeks later, support traces three production outages to the same search. Each customer typed rotate api key without downtime, opened the first result and did what it said. The first result was the Security checklist: "Rotate API keys every 90 days." It says when, not how. So they revoked the old key, created a new one, and every running service lost access until the next deploy. The page they needed, Rotate API keys with zero downtime (create a second key, deploy it everywhere, then revoke the first), sat right below it at rank 2.

The labels were not the problem. A docs writer had graded this query: 3 for the how-to, 2 for the API-key reference (it says an account can hold two active keys), 1 for the checklist, 0 for the rest. Here are the logged rankings and the two numbers the report used:

GRADES = {  # a docs writer's labels for "rotate api key without downtime"
    "rotate-keys": 3,         # Rotate API keys with zero downtime: the steps
    "api-keys": 2,            # API keys reference: an account can hold two active keys
    "security-checklist": 1,  # "Rotate keys every 90 days", no steps
    "webhook-secret": 0, "ssh-keys": 0, "maintenance": 0, "billing-api": 0,
}
old = ["webhook-secret", "maintenance", "security-checklist", "ssh-keys",
       "api-keys", "billing-api", "rotate-keys"]                    # BM25
new = ["security-checklist", "rotate-keys", "api-keys", "webhook-secret",
       "ssh-keys", "maintenance", "billing-api"]                    # hybrid + reranker
want = ["rotate-keys", "api-keys", "security-checklist", "webhook-secret",
        "ssh-keys", "maintenance", "billing-api"]                   # what the reader needed

def recall_at_k(ranking, grades, k, min_grade=1):
    relevant = {d for d, g in grades.items() if g >= min_grade}
    return len(relevant & set(ranking[:k])) / len(relevant)

def reciprocal_rank(ranking, grades, k=10, min_grade=1):
    """1 / rank of the first item with grade >= min_grade in the top k; 0 if none."""
    for rank, doc in enumerate(ranking[:k], start=1):
        if grades.get(doc, 0) >= min_grade:
            return 1 / rank
    return 0.0

for name, ranking in [("old", old), ("new", new), ("want", want)]:
    print(f"{name:5} recall@10 {recall_at_k(ranking, GRADES, 10):.2f}   "
          f"RR {reciprocal_rank(ranking, GRADES):.2f}")

old   recall@10 1.00   RR 0.33
new   recall@10 1.00   RR 1.00
want  recall@10 1.00   RR 1.00

Predict first: the report said new beats old, and it does. But would precision@3, hit rate@1 or average precision have told you that new is worse than want?

A ranking metric is a model of its reader: how far they read, when they stop, and how much a better document is worth to them. Pick the metric that reads your list the way your reader does. Feed it labels that carry the difference you care about. Then compute it exactly as defined, because the tools disagree more often than you would think. Each move below starts from a symptom in your numbers:

This lesson assumes the set metrics (precision@k, recall@k, hit rate@k) and a labeled query set, both from Retrieval Metrics. The code runs on Python 3.12 with numpy, scikit-learn, ranx 0.3.21 and pytrec_eval 0.5.10 (the pytrec-eval-terrier package, Python bindings for NIST's trec_eval). Later blocks reuse the functions and data from earlier ones.

Before any metric: who reads the list, and how deep?

Precision@k, recall@k and hit rate@k are set metrics. They ask which items made the top k and ignore the order inside it. Rank metrics, the subject of this lesson, also ask where each item sits. In the failure taxonomy of RAG Architecture & Implementation, they watch the ranked out bucket: relevant chunks that were found, then beaten by others.

Predict first: a reranker takes the first stage's top 100 and reorders them. You report recall@100 before and after. What can that number do?

The reader decides both the metric and the depth:

⚠️ "RAG chatbot, so use MRR" is a popular shortcut, and MRR is usually the wrong first metric there. The generator reads the whole context set, so first ask whether the right chunks got in: recall@k_context, and the gold-in-prompt rate after assembly (Context Augmentation). A rank metric earns its place in RAG when:

• labels are graded and slots are scarce, so it matters which relevant chunk gets one of the three slots;
• a token budget cuts the context at a variable depth;
• a person also reads the list, as in the hook.

Two habits for the rest of the lesson. Always write the depth: nDCG@10, never "nDCG", because nDCG@3 and nDCG@100 of the same run answer different questions. And write the reader next to the metric in your evaluation config, so the next person who changes k knows what they are changing.

Move 1: Reward the first hit — RR and MRR

Symptom: one answer is enough, and your metric can't tell whether it came first or third.

The reciprocal rank (RR) of a query is 1 / rank of the first relevant item. MRR@k averages it over queries and scores 0 when nothing relevant is in the top k. This is the definition the earlier lessons used. It is also what the MS MARCO passage ranking task evaluates with, at k = 10.

The curve is steep at the top and flat below. Moving from rank 1 to rank 2 costs 0.5; moving from rank 10 to rank 20 costs 0.05 (under MRR@20). That is the metric working as intended for a reader who stops at the first good result. It also has consequences you must handle yourself.

def mean_rr(runs, qrels, k=10, min_grade=1):
    """MRR@k over EVERY labeled query. A query with no hit in the top k,
    or missing from the run, scores 0 and stays in the denominator."""
    return sum(reciprocal_rank(runs.get(q, []), grades, k, min_grade)
               for q, grades in qrels.items()) / len(qrels)

print(reciprocal_rank(new, GRADES, min_grade=2), reciprocal_rank(want, GRADES, min_grade=2))

qrels = {"q1": {"a": 1}, "q2": {"b": 1}, "q3": {"c": 1}}
runs = {"q1": ["a", "x"], "q2": ["x", "b"], "q3": ["x", "y"]}   # q3 finds nothing
print(round(mean_rr(runs, qrels), 3), round(mean_rr(runs, {q: qrels[q] for q in ("q1", "q2")}), 3))

0.5 1.0
0.5 0.75

Keep the misses. q3 found nothing, so it scores 0 and MRR = (1 + 0.5 + 0) / 3 = 0.5. Drop it "because that's a retrieval failure, not a ranking one" and MRR becomes 0.75: you are now measuring a system that never fails. A query missing from the run (a timeout, a crash) also scores 0. The function averages over the labeled queries, not over whatever came back. A query with no relevant document in the corpus at all is a different case, handled in Move 3.

Choose the threshold on purpose. The first line printed shows that with min_grade=2, new gets 0.5 and want gets 1.0. With the default threshold of 1, both got 1.0. Chapelle et al. saw the same effect in a commercial search engine's logs (CIKM 2009, Table 3). RR computed with "good" documents counted as relevant agreed far less with click behavior than RR counting only the top two grades: a correlation of 0.172 against 0.366 with the clicks' maximum reciprocal rank. When partial answers can pass for answers, set the threshold above them.

Predict: on three queries, the first relevant item sits at ranks 1, 1 and 10. What is the MRR, and what is the average rank?

Polishing versus rescuing. On 100 labeled queries, change P moves 10 queries from rank 2 to rank 1, which adds 10 × 0.5 / 100 = 0.05 to MRR. Change Q moves 10 queries from rank 30 (0 under MRR@10) to rank 3, which adds 10 × 0.333 / 100 = 0.033. MRR prefers P. On a results page where rank 2 is still on screen, Q is probably the change users notice: it turns ten dead ends into answers, and it raises hit rate@10 by 0.10 while P leaves hit rate@10 unchanged. Report MRR next to hit rate@k, and look at which queries moved.

MRR fits a single-answer reader. It ignores everything after the first hit, and with binary labels it cannot tell a passing mention from the answer. The next two moves fix one weakness each.

Move 2: Reward every hit — AP and MAP

Symptom: the question needs several documents, and your metric says 1.0 as soon as the first one arrives.

A compliance assistant gets "Do our EU and US rules both require us to delete chat logs?" A complete answer needs three chunks: the EU retention rule, the US retention rule and the chat-log policy. MRR is 1.0 as soon as the EU rule ranks first, whether or not the other two ever show up.

Average precision (AP) of one query: at every rank where a relevant item appears, take the precision at that rank. Sum those precisions and divide by R, the number of relevant items for the query, retrieved or not. MAP is the mean of AP over queries.

def average_precision(ranking, grades, k=None, min_grade=1):
    """Sum of precision@rank at each relevant hit, divided by ALL relevant items."""
    relevant = {d for d, g in grades.items() if g >= min_grade}
    hits, total = 0, 0.0
    for rank, doc in enumerate(ranking[:k], start=1):
        if doc in relevant:
            hits += 1
            total += hits / rank
    return total / len(relevant) if relevant else 0.0

needs = {"eu-retention": 1, "us-retention": 1, "chat-log-policy": 1}
ranking = ["eu-retention", "x1", "us-retention", "x2", "x3",
           "chat-log-policy", "x4", "x5", "x6", "x7"]
print(round(average_precision(ranking, needs), 3), reciprocal_rank(ranking, needs))
needs["retention-2026-update"] = 1          # a fourth relevant chunk, never retrieved
print(round(average_precision(ranking, needs), 3), recall_at_k(ranking, needs, 10))

0.722 1.0
0.542 0.75

The relevant hits sit at ranks 1, 3 and 6, with precisions 1/1, 2/3 and 3/6. So AP = (1 + 0.667 + 0.5) / 3 = 0.722, while RR is a flat 1.0. Then a fourth relevant chunk turns up: a 2026 update to the retention rule that the system never retrieved. AP drops to 2.167 / 4 = 0.542 and recall@10 to 0.75.

⚠️ The denominator is R, not the hits you found. A common hand-rolled version divides by the number of relevant items retrieved. It gives 0.722 in both cases and never notices the missing update. AP folds recall into one number only because R counts what you missed.

AP@k is not one thing. ranx's map@k and trec_eval's map_cut_k divide by R (checked in the ranx 0.3.21 source and on these examples). The widely copied apk function from the Kaggle-era ml_metrics package divides by min(R, k). Take R = 4 with three relevant items filling the top 3: the first convention gives 0.75, the second 1.0. Neither is wrong. They answer different questions ("how much of what you needed?" against "how much of what fit in k?"). Say which one you report, and never compare numbers from the two.

Two more facts:

• With exactly one relevant item per query, AP equals RR and MAP equals MRR, because each precision sum has a single term, 1 / rank.
• AP is top-heavy too. A relevant item at rank 1 adds a full 1 / R, while the first relevant item at rank 10 adds only 0.1 / R.

Where it fits: exhaustive search, where the reader needs everything and reads deep (e-discovery, patent search, compliance review). It also works as a one-number summary across depths when you tune the order of a long list. For a RAG context set, recall@k_context answers the question more directly: are all the needed chunks in? And MAP is binary, like MRR, so it still cannot see the hook's problem.

Move 3: Grade the gains, discount the ranks — nDCG

Symptom: some relevant documents are worth far more than others, and your metric can't tell them apart. That was the hook.

Discounted cumulative gain (DCG) adds up the value of each result, discounted by how far down it sits:

DCG@k  = sum over ranks i = 1..k of  gain(grade_i) / log2(i + 1)
gain   = grade            linear: trec_eval's ndcg, ranx's ndcg, scikit-learn
       = 2**grade - 1     exponential: ranx's ndcg_burges, LightGBM and XGBoost defaults
IDCG@k = DCG@k of the ideal ordering of ALL judged items for the query
nDCG@k = DCG@k / IDCG@k    (1.0 = the best order the labels allow)

The discount at each rank:

Here is new from the hook at k = 3, with linear gain:

DCG@3 = 3.893. The ideal order is 3, 2, 1, so IDCG@3 = 3 + 2 × 0.631 + 1 × 0.5 = 4.762, and nDCG@3 = 3.893 / 4.762 = 0.817.

import math

def dcg(gains, k):
    return sum(g / math.log2(rank + 1) for rank, g in enumerate(gains[:k], start=1))

def ndcg_at_k(ranking, grades, k, gain="linear"):
    """The ideal comes from ALL judged items, not just the retrieved ones."""
    to_gain = (lambda g: g) if gain == "linear" else (lambda g: 2 ** g - 1)
    got = [to_gain(grades.get(doc, 0)) for doc in ranking]   # unjudged counts as 0
    ideal = sorted((to_gain(g) for g in grades.values()), reverse=True)
    return dcg(got, k) / dcg(ideal, k) if dcg(ideal, k) > 0 else None

for name, ranking in [("old", old), ("new", new), ("want", want)]:
    print(f"{name:5} nDCG@3 linear {ndcg_at_k(ranking, GRADES, 3):.3f}   "
          f"exponential {ndcg_at_k(ranking, GRADES, 3, 'exp'):.3f}")

old   nDCG@3 linear 0.105   exponential 0.053
new   nDCG@3 linear 0.817   exponential 0.736
want  nDCG@3 linear 1.000   exponential 1.000

Check a hand-rolled metric against a library before you trust it:

import pytrec_eval
from ranx import Qrels, Run, evaluate

def as_run(ranking):                  # both tools want scores, higher = better
    return {d: float(len(ranking) - i) for i, d in enumerate(ranking)}

run = {"q": as_run(new)}
print(evaluate(Qrels({"q": GRADES}), Run(run), ["ndcg@3", "ndcg_burges@3"]))
print(pytrec_eval.RelevanceEvaluator({"q": GRADES}, {"ndcg_cut_3"}).evaluate(run))

{'ndcg@3': np.float64(0.8174935137996165), 'ndcg_burges@3': np.float64(0.7363636171343382)}
{'q': {'ndcg_cut_3': 0.8174935137996165}}

Why it works. The gain says what a document is worth. The discount says how likely the reader is to get that far. Dividing by the ideal turns "how much value" into "how close to the best this query allows", so a query with one great document and a query with ten can share an average.

The gain is a product decision. Linear gain says a grade 3 is worth 1.5 times a grade 2. Exponential gain says 7 / 3, about 2.3 times.

Predict: the judged documents are a (grade 3), b and c (grade 2 each). System P returns [b, c, x] and system Q returns [a, x, y]. Which wins on nDCG@3?

The 1.0 that missed the answer

Suppose a ranker returns [security-checklist, webhook-secret, ssh-keys], and your evaluation builds the ideal from those three results:

from sklearn.metrics import ndcg_score

shown = ["security-checklist", "webhook-secret", "ssh-keys"]
grades_shown = [GRADES[d] for d in shown]                           # [1, 0, 0]
wrong_ideal = dcg(sorted(grades_shown, reverse=True), 3)            # ideal of what came back
print(round(dcg(grades_shown, 3) / wrong_ideal, 3), round(ndcg_at_k(shown, GRADES, 3), 3))
print(ndcg_score([grades_shown], [[3, 2, 1]], k=3))                 # sklearn, shown items only

1.0 0.21
1.0

The home-made version sorted what came back (1, 0, 0) and called it ideal. So a list that missed both the how-to and the reference scores a perfect 1.0. With the ideal built from every judged document (3, 2, 1), it scores 0.21. scikit-learn's ndcg_score (version 1.9.1) builds the ideal from the y_true you pass, so it makes the same mistake if you hand it only the retrieved items. Pass every judged document, giving the unretrieved ones the lowest scores, and it also returns 0.21. ranx and trec_eval build the ideal from the qrels, which is what you want.

What else to know about nDCG

• The discount is gentle. At rank 10 an item still earns 0.289 of its rank-1 value, so rank 1 is worth only 3.5 times rank 10, where RR says 10 times. That was deliberate. Järvelin & Kekäläinen introduced cumulated gain with graded judgments (SIGIR 2000), and in the extended version (ACM TOIS, 2002) they explain that the discount should fall "not too steeply (e.g., as division by rank) to allow for user persistence". Their original formula left the first ranks undiscounted: with base 2, ranks 1 and 2 counted in full. The 1/log2(rank + 1) form that tools use today follows Burges et al. (ICML 2005).
• Scores compare only on the same queries and labels. The ideal normalizes each query, but query difficulty and the number of relevant documents still set the level. The same all-MiniLM-L6-v2 model scores nDCG@10 of 0.318 on NFCorpus (Move 7) and 0.645 on SciFact (Retrieval Metrics). Neither number is "good" or "poor" on its own. Compare systems on the same set.
• A query with no relevant documents has no nDCG. Its ideal DCG is 0, so nDCG is 0/0. ndcg_at_k above returns None, while ranx and trec_eval quietly return 0 (checked), which drags the average down for queries no ranking could fix. Decide explicitly: take unanswerable queries out of the ranking metrics and track them where they belong, as abstention cases (Grounding & Hallucination Control). Do not confuse them with Move 1's misses. A query whose relevant document exists but was not retrieved scores 0 and stays.
• Near-duplicates inflate it. Three copies of a relevant chunk earn three gains, though the reader learns nothing from the second and third. Diversity-aware metrics such as α-nDCG (Clarke et al., SIGIR 2008) discount repeated information. In a RAG pipeline, the practical fix is deduplicating before assembly (Context Augmentation).

Move 4: Model the reader who stops — ERR

Symptom: your readers stop at the first great result, yet nDCG prefers a page of near-misses.

Chapelle et al. made the case with an example (CIKM 2009). Compare a list of 20 "good" documents with a list of one "perfect" document followed by 19 bad ones. DCG prefers the first list: grading bad to perfect as 0 to 4, with exponential gain, DCG@20 scores them 21.1 and 15.0. But a reader who wants the destination page clicks the perfect result, is satisfied and stops. They never see the 19 bad ones. On the first list, they wade through partial answers. The paper names DCG's weakness: a document at a given position "has always the same gain and discount independently of the documents shown above it".

The cascade model (Craswell et al., WSDM 2008) describes that reader. They scan from the top. At each result they are satisfied with a probability that depends on its grade, and then they stop; otherwise they move on. Expected reciprocal rank (ERR) is the expected value of 1 / rank at the point where they stop:

R(grade) = (2**grade - 1) / 2**max_grade     chance that this result satisfies the reader
ERR@k    = sum over r = 1..k of  (1/r) * R_r * product over i < r of (1 - R_i)

The formula and a brute-force simulation of that reader should agree:

import random

def err_at_k(ranking, grades, k, max_grade=3):
    """Expected reciprocal rank: a reader scans down and stops at rank r
    with probability R = (2**grade - 1) / 2**max_grade."""
    p_reach, total = 1.0, 0.0
    for rank, doc in enumerate(ranking[:k], start=1):
        R = (2 ** grades.get(doc, 0) - 1) / 2 ** max_grade
        total += p_reach * R / rank
        p_reach *= 1 - R
    return total

def simulate(ranking, grades, readers=200_000, max_grade=3, seed=0):
    """The same thing, by brute force: send simulated readers down the list."""
    rng, total = random.Random(seed), 0.0
    for _ in range(readers):
        for rank, doc in enumerate(ranking, start=1):
            if rng.random() < (2 ** grades.get(doc, 0) - 1) / 2 ** max_grade:
                total += 1 / rank
                break
    return total / readers

for name, ranking in [("old", old), ("new", new), ("want", want)]:
    print(f"{name:5} ERR@10 {err_at_k(ranking, GRADES, 10):.3f}   simulated {simulate(ranking, GRADES):.3f}")

grades = {"A": 3, "B": 2, "C": 2, "D": 2, "E": 2}
one_perfect = ["A", "junk1", "junk2", "junk3", "junk4"]
four_partial = ["B", "C", "D", "E", "A"]
for name, ranking in [("one perfect", one_perfect), ("four partial", four_partial)]:
    print(f"{name:12} nDCG@5 {ndcg_at_k(ranking, grades, 5):.3f}   ERR@5 {err_at_k(ranking, grades, 5):.3f}")

old   ERR@10 0.176   simulated 0.176
new   ERR@10 0.521   simulated 0.521
want  ERR@10 0.902   simulated 0.901
one perfect  nDCG@5 0.435   ERR@5 0.875
four partial nDCG@5 0.911   ERR@5 0.591

Read the output:

• The simulation matches the formula to within 0.001. ERR really is the average of 1 / rank at the point where a simulated reader stops, which also makes the simulation a good unit test for your implementation.
• On the hook, ERR@10 separates new from want sharply: 0.521 against 0.902, a wider gap than nDCG@3's 0.817 against 1.000. The how-to satisfies 7 readers in 8. In new, the checklist stands in front of it: one reader in eight stops there and never sees the how-to (the outage), and everyone else reaches it at rank 2, which halves the 1 / rank they earn.
• On the two five-item lists, the metrics disagree. nDCG@5 prefers four partial answers (0.911 against 0.435). ERR@5 prefers the single perfect one (0.875 against 0.591). Which is right depends on the reader. For a person who wants the one page, trust ERR. For an LLM that reads all five chunks, four useful chunks may well beat one great chunk plus four pieces of junk, and ERR's stopping reader is the wrong model.

Three facts before you use it:

• ERR is not normalized. With max_grade = 3, a perfect document at rank 1 scores 0.875, and even an endless list of perfect documents reaches only 7 × ln(8/7) ≈ 0.935. Compare ERR only at the same grade scale and depth.
• RR is ERR's special case. If every relevant result satisfies the reader for certain (R = 1) and every other result never does (R = 0), the reader stops at the first relevant item and ERR equals RR, as Chapelle et al. point out. The default mapping is gentler: with max_grade = 1, a relevant result satisfies only half the readers. ERR is Move 1 generalized to graded, uncertain satisfaction.
• The evidence comes from one search engine's clicks. On a commercial engine's logs, Chapelle et al. found that ERR correlated better with click metrics than DCG, nDCG, AP or RR, with the largest gap on easy, frequent queries. That is a finding about web searchers, not a law about your readers. Rank-biased precision (Moffat & Zobel, 2008) models a simpler reader, who moves on with a fixed probability whatever they have just seen.

Neither ranx 0.3.21 nor trec_eval computes ERR. The ten-line function above is enough, and the simulation tests it.

Move 5: Find the holes in your labels

Symptom: a new retriever loses offline, then wins with users.

Labels come from judging the documents that earlier systems returned (building them is covered in Retrieval Metrics). A new retriever, such as a new embedding model, learned sparse retrieval or a query rewriter, surfaces documents that nobody judged. ranx and trec_eval treat an unjudged document as not relevant, so the new system gets no credit for the relevant documents it finds. One way to see how much that matters is to score the candidate a second way, on a condensed list: drop the unjudged documents and let the judged ones move up.

def judged_at_k(ranking, grades, k=10):
    """Share of the top k that has any label at all."""
    return sum(doc in grades for doc in ranking[:k]) / k

def condensed(ranking, grades):
    """Drop unjudged documents; the judged ones move up."""
    return [doc for doc in ranking if doc in grades]

labels = {"A": 3, "B": 2, "C": 1, "D": 0, "E": 0, "F": 0, "G": 0, "H": 0, "I": 0, "J": 0}
current = ["B", "A", "D", "C", "E", "F", "G", "H", "I", "J"]
candidate = ["A", "U1", "U2", "B", "U3", "C", "U4", "D", "E", "F"]   # U = never labeled
print("judged@10 ", judged_at_k(current, labels), judged_at_k(candidate, labels))
for name, view in [("holes as 0", lambda r: r), ("condensed ", lambda r: condensed(r, labels))]:
    print(name, round(ndcg_at_k(view(current), labels, 10), 3), round(ndcg_at_k(view(candidate), labels, 10), 3))
labels |= {"U1": 2, "U2": 0, "U3": 1, "U4": 0}                      # label the holes
print("labeled   ", round(ndcg_at_k(current, labels, 10), 3), round(ndcg_at_k(candidate, labels, 10), 3))

judged@10  1.0 0.6
holes as 0 0.908 0.886
condensed  0.908 1.0
labeled    0.711 0.965

The last three rows are nDCG@10 with linear gain. The candidate puts four unlabeled documents in its top 10, so only 60% of its top 10 has labels. With the holes counted as not relevant, it loses, 0.886 against 0.908. On the condensed list, it wins, 1.000 against 0.908 (the current system has no holes, so its score does not change). Two estimates, two different winners. Label those four documents, two of which turn out to be relevant, and the candidate wins, 0.965 against 0.711. The current system's score fell too: the two new relevant documents join the ideal, and they are documents it failed to find.

Predict: is the true score always somewhere between the two estimates?

graded = {"A": 3, "B": 2, "C": 1} | {d: 0 for d in "DEFGHI"}
top10 = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "U1"]
print(round(ndcg_at_k(top10, graded, 10), 3), round(ndcg_at_k(condensed(top10, graded), graded, 10), 3))
print(round(ndcg_at_k(top10, graded | {"U1": 3}, 10), 3))     # the hole was the best document

1.0 1.0
0.89

What to do about holes:

• Measure them. Report judged@k next to every rank metric. When a candidate's judged@10 is well below the baseline's, its score rests on the assumption that every hole is irrelevant. It is an estimate, not a measurement.
• Score them both ways. Report the condensed-list score next to the holes-as-0 score. Sakai (SIGIR 2007) found condensed-list metrics a better answer to incomplete labels than bpref, a measure that Buckley & Voorhees (SIGIR 2004) designed for the purpose. The condensed score is never below the holes-as-0 score, but neither one bounds the truth, and the baseline's score can move too (0.908 became 0.711 above). When the two estimates pick different winners, as here, the holes decide the comparison.
• Fill them. Before comparing, label the top k of every system under test, not only the baseline's, then score all of them again. LLM judges make this cheap. Thomas et al. at Microsoft Bing reported that LLMs can label relevance at close to human quality, and UMBRELA (Upadhyay et al., 2024) is an open-source reproduction built to help assess the TREC 2024 RAG Track. Check any LLM judge against a human-labeled sample of your own data before you rely on it. For graded labels, measure agreement with weighted kappa (or Krippendorff's alpha) rather than raw percent agreement.

Watch for datasets that list only positives. Many BEIR qrels, including NFCorpus's (Move 7), list only relevant documents, so every unlisted document counts as not relevant. Others, such as TREC-COVID's, also list judged non-relevant documents. When labels were pooled from many different systems' results, as in TREC, treating the rest as not relevant is a tolerable assumption. NFCorpus's labels come from NutritionFacts.org's links to the papers it cites, so they miss relevant abstracts that no page linked to. They were not collected from any system in Move 7, so the gaps do not favor one of those systems by construction, but a system that finds an unlinked relevant abstract still gets no credit for it. Positives-only labels are riskiest on your own 50 queries, labeled from one system's top 5: every other system is then scored against that one system's view of the corpus.

Move 6: Compute it the way it is defined

Symptom: two tools report two numbers for the same run.

Most disagreements are not bugs in the tools. They are different defaults meeting your data. Here are three you will hit:

import pytrec_eval
from ranx import Qrels, Run, evaluate

qrels = {"q1": {"rotate-keys": 1}, "q2": {"sso-setup": 1}}
trec = pytrec_eval.RelevanceEvaluator(qrels, {"recip_rank"})

# 1. Two chunks tie at the top.
tied = {"q1": {"rotate-keys": 0.91, "security-checklist": 0.91}, "q2": {"sso-setup": 0.8}}
print("tie      pytrec_eval", trec.evaluate(tied)["q1"]["recip_rank"],
      "| ranx", evaluate(Qrels(qrels), Run(tied), "mrr", return_mean=False)[0])

# 2. The run lost a query (a timeout, say).
partial = {"q1": {"rotate-keys": 0.9}}
res = trec.evaluate(partial)
print("missing  pytrec_eval mean", sum(v["recip_rank"] for v in res.values()) / len(res),
      "| ranx", evaluate(Qrels(qrels), Run(partial), "mrr", make_comparable=True))

# 3. Ranks written into the score field (1 = best).
as_ranks = {"q1": {"rotate-keys": 1, "api-keys": 2, "sso-setup": 3}, "q2": {"sso-setup": 1}}
print("ranks    pytrec_eval", trec.evaluate(as_ranks)["q1"]["recip_rank"])

tie      pytrec_eval 0.5 | ranx 1.0
missing  pytrec_eval mean 1.0 | ranx 0.5
ranks    pytrec_eval 0.3333333333333333

1. Ties. Both tools re-sort each query's results by score. trec_eval, and so pytrec_eval, breaks ties by document id; in our tests it put ids in reverse alphabetical order, whatever order you passed. So security-checklist comes before rotate-keys. ranx kept the order you inserted in this two-document tie, but only because the tie group is small: Retrieval Metrics found larger groups reordered. Same run, RR 0.5 or 1.0, so treat tie order in either tool as undefined. Ties are more common than you think. Duplicate documents get identical scores, sigmoid outputs saturate at 1.0, and pytrec_eval compared scores at single precision in our tests, so scores that agree to about seven significant digits also tie. On Move 7's dense run, 39 of the 323 test queries have a tie among their top-10 scores (NFCorpus contains some abstracts twice), and nDCG@10 comes out 0.3179 in pytrec_eval against 0.3182 in ranx. Before scoring, rewrite each query's scores as len(results) - rank in the order you actually served. Then both tools report 0.3182.
2. Missing queries. pytrec_eval returns results only for queries present in the run, so averaging its output silently drops the query that timed out: 1.0 instead of 0.5. ranx refuses to evaluate (it raises an AssertionError) unless you pass make_comparable=True, which scores the missing query 0. Either way, average over your qrels' query ids.
3. Direction. Both tools read the score as "higher is better". Put ranks (1 = best) or distances in the score field and the list is scored upside down: RR 0.333 instead of 1.0. Negate distances from an L2 index, or from Qdrant's Euclid metric, before scoring.

Four quieter ones:

• Cut-off. trec_eval's recip_rank looks at the whole run; ranx's mrr@10 stops at 10. On Move 7's dense run, that is 0.512 against 0.508. MS MARCO-style MRR@10 is the second.
• Relevance threshold. Both tools count grade 1 and up as relevant for binary metrics by default. In ranx you write mrr@10-l2; in pytrec_eval you pass relevance_level=2.
• Gain. trec_eval's ndcg and ranx's ndcg use linear gain. ranx's ndcg_burges, LightGBM's default label_gain and XGBoost's default ndcg_exp_gain use 2**grade - 1. scikit-learn's ndcg_score is linear and by default averages over tied scores: a relevant and an irrelevant item tied at the top score 0.815, rather than 1.0 or 0.631.
• Averaging. Every number in this lesson is a macro average, where each query counts once. Weighting by traffic answers a different question (how the average search went) and lets frequent queries hide the long tail. Report per-slice numbers either way.

💡 Hand-check one query per metric. Before trusting a harness, compute one query by hand, as in Move 3's table, and assert that the tool agrees to four decimals. Then feed it a tie, a missing query and a query with no relevant documents on purpose. It takes ten minutes.

Move 7: Compare rankers query by query

Symptom: "+0.02 nDCG@10", and nobody knows whether it is real.

An average hides what decides whether a difference is real: how consistently it shows up across queries. Here is a real comparison on NFCorpus (Boteva et al., ECIR 2016) in its BEIR version. It has 3,633 medical abstracts and 323 test queries taken from NutritionFacts.org pages. The labels are graded (2 and 1) and come from the site's links, not from human assessors. A dev split of 324 queries is used in Move 8. There are three systems: BM25 (rank_bm25's BM25Okapi with its defaults, k1 = 1.5 and b = 0.75), dense retrieval with all-MiniLM-L6-v2, and the MiniLM-L6 cross-encoder reranking the dense top 100. It takes a few minutes on a laptop CPU, mostly for the reranker.

import re
import numpy as np
import pandas as pd
from huggingface_hub import hf_hub_download
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder, SentenceTransformer

def hf(repo, path):
    return hf_hub_download(repo, path, repo_type="dataset")

corpus = pd.read_parquet(hf("BeIR/nfcorpus", "corpus/corpus-00000-of-00001.parquet"))
queries = pd.read_parquet(hf("BeIR/nfcorpus", "queries/queries-00000-of-00001.parquet"))
query_text = dict(zip(queries["_id"], queries["text"]))
doc_ids = corpus["_id"].tolist()
docs = (corpus["title"] + ". " + corpus["text"]).tolist()

def load_qrels(split):
    """{query_id: {doc_id: grade}} from BEIR's TSV (grades 1 and 2 here)."""
    df = pd.read_csv(hf("BeIR/nfcorpus-qrels", f"{split}.tsv"), sep="\t", dtype=str)
    qrels = {}
    for q, d, grade in df.itertuples(index=False):
        qrels.setdefault(q, {})[d] = int(grade)
    return qrels

def tokenize(text):
    return re.findall(r"\w+", text.lower())

bm25 = BM25Okapi([tokenize(d) for d in docs])
bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
doc_vecs = bi.encode(docs, batch_size=64, normalize_embeddings=True)

def build_runs(qrels, k_candidates=100):
    """Each run is {query_id: {doc_id: score}}, higher = better.
    pools keeps (doc_id, cosine, reranker score) for the dense top k_candidates."""
    runs, pools = {"bm25": {}, "dense": {}, "rerank": {}}, {}
    for q in sorted(qrels):
        text = query_text[q]
        lex = bm25.get_scores(tokenize(text))
        top = np.argsort(-lex)[:k_candidates]
        runs["bm25"][q] = {doc_ids[i]: float(lex[i]) for i in top}
        cos = doc_vecs @ bi.encode(text, normalize_embeddings=True)
        top = np.argsort(-cos)[:k_candidates]
        runs["dense"][q] = {doc_ids[i]: float(cos[i]) for i in top}
        ce_scores = ce.predict([(text, docs[i]) for i in top], batch_size=100)
        pools[q] = [(doc_ids[i], float(cos[i]), float(s)) for i, s in zip(top, ce_scores)]
        runs["rerank"][q] = {d: s for d, _, s in pools[q]}
    return runs, pools

qrels = {split: load_qrels(split) for split in ("dev", "test")}
runs, pools = {}, {}
for split in qrels:
    runs[split], pools[split] = build_runs(qrels[split])
print(len(docs), "abstracts;", {split: len(q) for split, q in qrels.items()}, "labeled queries")

from ranx import Qrels, Run, evaluate

METRICS = ["ndcg@10", "mrr@10", "map@100", "recall@100"]
for name, run in runs["test"].items():
    scores = evaluate(Qrels(qrels["test"]), Run(run), METRICS)
    print(f"{name:7}", "  ".join(f"{m} {v:.3f}" for m, v in scores.items()))

3633 abstracts; {'dev': 324, 'test': 323} labeled queries
bm25    ndcg@10 0.306  mrr@10 0.509  map@100 0.138  recall@100 0.242
dense   ndcg@10 0.318  mrr@10 0.508  map@100 0.144  recall@100 0.315
rerank  ndcg@10 0.343  mrr@10 0.561  map@100 0.165  recall@100 0.315

ranx's ndcg@10 uses linear gain and the 1/log2(rank + 1) discount, so a grade-2 abstract is worth twice a grade-1 one; this is also how BEIR reports nDCG@10 (it calls pytrec_eval's ndcg_cut). Your last digit may differ on other hardware. Three things to notice:

• The recall@100 column is the "Before any metric" point in real data: reranking leaves it at 0.315.
• MRR@10 calls BM25 and dense a tie (0.509 against 0.508), while nDCG@10 gives dense a lead of 0.012.
• The reranker adds 0.025 to nDCG@10 and 0.053 to MRR@10.

Which of these differences would survive a fresh sample of queries? Compare per query. Both systems answered the same queries, so one query's difficulty affects both scores, and taking the difference cancels it.

from scipy.stats import binomtest

rng = np.random.default_rng(0)

def per_query(qrels, run, metric="ndcg@10"):
    """One score per query, in sorted query order, via ranx."""
    assert set(run) == set(qrels), "score every labeled query"
    r = Run(run)
    evaluate(Qrels(qrels), r, metric)
    return np.array([r.scores[metric][q] for q in sorted(qrels)])

def randomization_p(a, b, n=20_000):
    """Paired, two-sided: flip the sign of each query's difference at random
    and count how often the mean moves at least as far as the one observed."""
    diff = a - b
    null = (rng.choice([-1.0, 1.0], size=(n, len(diff))) * diff).mean(axis=1)
    return float((np.abs(null) >= abs(diff.mean()) - 1e-12).mean())

def bootstrap_ci(a, b, n=10_000):
    """95% interval for the mean difference, resampling queries with replacement."""
    diff = a - b
    means = diff[rng.integers(0, len(diff), size=(n, len(diff)))].mean(axis=1)
    return np.percentile(means, [2.5, 97.5])

for metric in ("ndcg@10", "mrr@10"):
    s = {name: per_query(qrels["test"], run, metric) for name, run in runs["test"].items()}
    for a, b in [("dense", "bm25"), ("rerank", "dense")]:
        diff = s[a] - s[b]
        up, down, same = (diff > 0).sum(), (diff < 0).sum(), (diff == 0).sum()
        lo, hi = bootstrap_ci(s[a], s[b])
        print(f"{metric} {a} - {b}: {diff.mean():+.3f}  up/down/same {up}/{down}/{same}  "
              f"sign p={binomtest(up, up + down).pvalue:.3f}  "
              f"randomization p={randomization_p(s[a], s[b]):.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")

ndcg@10 dense - bm25: +0.012  up/down/same 117/102/104  sign p=0.344  randomization p=0.290  95% CI [-0.010, +0.034]
ndcg@10 rerank - dense: +0.025  up/down/same 116/90/117  sign p=0.081  randomization p=0.004  95% CI [+0.008, +0.042]
mrr@10 dense - bm25: -0.001  up/down/same 76/63/184  sign p=0.309  randomization p=0.971  95% CI [-0.038, +0.035]
mrr@10 rerank - dense: +0.053  up/down/same 60/48/215  sign p=0.290  randomization p=0.001  95% CI [+0.022, +0.085]

Read it line by line:

• Dense against BM25: dense is better on 117 queries and worse on 102. The randomization test gives p = 0.29, and the 95% interval runs from −0.010 to +0.034, which includes zero. On 323 queries you cannot tell them apart. The near-even split does say that each system wins on different queries, which is the situation Hybrid Retrieval Systems exploits.
• Reranker against dense: +0.025 with p = 0.004 and an interval from +0.008 to +0.042. The gain is real and modest. Report the interval, not just the p-value, because it tells you how big the gain might plausibly be.
• The sign test is weak. It counts only ups and downs (116 against 90) and ignores their size, so the same data gives it p = 0.081. (For 0/1 per-query outcomes such as hit rate there is no size to ignore, and the sign test is exactly McNemar's test, the one Evaluation & Quality Metrics uses.) Smucker, Allan & Carterette compared significance tests on TREC runs (CIKM 2007). They found "little practical difference between the randomization, bootstrap, and t tests", while the Wilcoxon and sign tests "have a poor ability to detect significance" and can produce false detections. Use the paired randomization test or the paired t-test for the p-value, and the bootstrap for the interval. ranx's compare() runs the randomization test (stat_test="fisher") or the paired t-test ("student", its default) across several runs at once.
• MRR@10 changed on only a third of the queries (108 of 323). The other 215 kept their first relevant rank. The reranker's MRR gain is concentrated, which is worth knowing when you read the queries that moved.

How many queries do you need? Here the per-query differences between reranker and dense have a standard deviation of 0.157. To detect a true gain δ with a two-sided test at the 5% level and 80% power, you need roughly n = (2.8 × sd / δ)² queries: (2.8 × 0.157 / 0.025)² ≈ 310. As a check, we drew 50 of the 323 test queries at random, 4,000 times. In about 11% of the draws the reranker looked no better than dense, and only about 15% of the draws reached p < 0.05 (paired t-test, which agrees with the randomization test here: p = 0.005 on all 323 queries). A 50-query set cannot see a gain of this size. Evaluation & Quality Metrics covers sizing an evaluation set, and online checks (A/B tests, interleaving) that confirm offline wins with real users.

⚠️ Twenty tries make a winner. Test 20 configurations that are all really equal to the baseline, each at p < 0.05, and the chance that at least one "wins" is 1 − 0.95²⁰ ≈ 64% if the tests are independent. Pick the configuration on one split and run one test on another (Move 8), or correct for the number of tests.

Move 8: Tune on one split, report on another

Symptom: the reranker has knobs, and you tune them on the same queries you report.

Rerankers come with knobs: the pool size (Reranking Models sizes one on SciFact), an instruction for LLM rerankers, a score cut-off, and how much to trust the first stage. The recipe for all of them:

1. Fix the target first: the metric and depth your reader uses, plus the ceiling (the pool's recall).
2. Tune on dev; report on test, once.
3. Report the gain over the untuned reranker with an interval, and read the queries that got worse.

The knob here is a blend of the reranker's score with the first stage's cosine. Raw logits and cosines live on different scales, so both are min-max normalized per query first; Hybrid Retrieval Systems shows what happens when you skip that. The target is nDCG@10 with linear gain, as in Move 7.

def minmax(x):
    x = np.asarray(x, dtype=float)
    span = x.max() - x.min()
    return (x - x.min()) / span if span else np.zeros_like(x)

def blended(pools, lam):
    """lam = weight on the reranker; both scores min-max normalized per query."""
    run = {}
    for q, pool in pools.items():
        dense = minmax([c for _, c, _ in pool])
        rerank = minmax([s for _, _, s in pool])
        run[q] = {d: float(lam * r + (1 - lam) * c)
                  for (d, _, _), r, c in zip(pool, rerank, dense)}
    return run

lams = [round(x * 0.1, 1) for x in range(11)]
for split in ("dev", "test"):
    row = {lam: evaluate(Qrels(qrels[split]), Run(blended(pools[split], lam)), ["ndcg@10", "mrr@10"])
           for lam in lams}
    print(split, "nDCG@10", " ".join(f"{lam}:{m['ndcg@10']:.3f}" for lam, m in row.items()))
    print(split, "MRR@10 ", " ".join(f"{lam}:{m['mrr@10']:.3f}" for lam, m in row.items()))
    if split == "dev":
        chosen = max(lams, key=lambda lam: row[lam]["ndcg@10"])
        print("chosen on dev: lam =", chosen)

test_scores = {lam: per_query(qrels["test"], blended(pools["test"], lam)) for lam in (chosen, 1.0)}
lo, hi = bootstrap_ci(test_scores[chosen], test_scores[1.0])
print(f"test, blend {chosen} - reranker alone: {(test_scores[chosen] - test_scores[1.0]).mean():+.3f}  "
      f"randomization p={randomization_p(test_scores[chosen], test_scores[1.0]):.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")

dev nDCG@10 0.0:0.301 0.1:0.309 0.2:0.313 0.3:0.322 0.4:0.328 0.5:0.334 0.6:0.338 0.7:0.340 0.8:0.339 0.9:0.333 1.0:0.328
dev MRR@10  0.0:0.502 0.1:0.511 0.2:0.512 0.3:0.527 0.4:0.536 0.5:0.539 0.6:0.548 0.7:0.550 0.8:0.552 0.9:0.548 1.0:0.546
chosen on dev: lam = 0.7
test nDCG@10 0.0:0.318 0.1:0.323 0.2:0.327 0.3:0.334 0.4:0.342 0.5:0.345 0.6:0.351 0.7:0.350 0.8:0.348 0.9:0.348 1.0:0.343
test MRR@10  0.0:0.508 0.1:0.512 0.2:0.514 0.3:0.523 0.4:0.532 0.5:0.535 0.6:0.549 0.7:0.551 0.8:0.551 0.9:0.568 1.0:0.561
test, blend 0.7 - reranker alone: +0.007  randomization p=0.035  95% CI [+0.001, +0.014]

Here λ = 0 is dense alone and λ = 1 is the reranker alone. Read it:

• Dev picks λ = 0.7 (nDCG@10 0.340, against 0.328 for the reranker alone and 0.301 for dense alone).
• Test confirms it, barely: 0.350 against 0.343, a gain of 0.007 with p = 0.035 and an interval from +0.001 to +0.014. It is probably real, and tiny.
• Tuning on test flatters you. On test, λ = 0.6 scores 0.351, and you would have reported a number no fresh sample would reproduce. The gap is small here only because the curve is flat near its peak.
• The metric picks the knob. MRR@10 peaks at λ = 0.8 on dev. On test, it rates λ = 0.7 slightly below the reranker alone (0.551 against 0.561). If your reader wants the one best answer, the blend is not worth a knob. If they read a top 10 with graded labels, it is a small free gain. Decide the metric before you look at the sweep.

Is +0.007 worth shipping? The blend costs nothing at query time, because both scores already exist. But it adds one more setting that must be re-tuned whenever either model changes. That maintenance cost belongs in the decision next to the interval.

Your turn. A teammate reranks only the dense top 10 and reports "reranking hurts: MAP@100 falls from 0.144 to 0.125". nDCG@10 went from 0.318 to 0.333. What is going on, and what should they report?

Final round: no label on the problem

Real reports don't say which move they need. Find the reader first, then the symptom.

Challenge 1: the form filler

An agent step searches the product catalog and copies a value (a plan's price, an API rate limit) from the top result into a form. After an embedding-model upgrade, the team reports nDCG@10 up by 0.04, yet the rate of wrong form values hasn't changed.

Challenge 2: the quiet improvement

A teammate removes 40 of 300 labeled queries "where retrieval failed completely, probably label noise", and MRR@10 rises from 0.58 to 0.67.

Challenge 3: the notebook and CI disagree

A notebook reports nDCG@10 = 0.91 for a new retriever. The CI job, using ranx on the same queries, reports 0.47.

Challenge 4: the league table

Two teams report nDCG@10 for their retrievers. The support team's hybrid retriever scores 0.71 on 400 support tickets. The docs team's dense retriever scores 0.46 on 250 documentation questions. Leadership proposes putting the support team's retriever behind docs search, since it "scores 50% higher".

Cheat sheet: symptom → move → trap

Before moving on, take one ranking your system produces and write down: who reads it and how deep, the metric and k, the gain and the relevance threshold, how misses and unanswerable queries are handled, which tool computes it and how that tool breaks ties. Compute one query by hand and make the tool agree.

How would you measure it on your own data? Take real queries from your logs, sliced by type, and split them into dev and test. Size the test split with Move 7's arithmetic: with per-query differences as spread out as NFCorpus's, seeing a gain of 0.025 takes about 300 test queries. Grade the top 10 of every system you compare on a 0–3 scale, with a written definition of each grade (humans, or an LLM judge checked against a human sample). Tune on dev. On test, report the reader's metric at the reader's depth, judged@k, and a paired test with a bootstrap interval, per slice. Then read the ten queries that got worse.

Next: Retrieval Metrics for the set metrics and building labels, Reranking Models for the stage these metrics most often tune, and Generation Quality for judging what the model writes once the right chunks are in its prompt.