Citation Coverage

Cited, confident and mostly unbacked

Orbit Labs runs an internal assistant over its travel and expense policy. An engineer asks:

The conference hotel is sold out. Can I book another hotel above the cap, and when do I get my money back?

Retrieval works. The context set holds four passages, labelled S1 to S4 after the final ordering, exactly as Context Augmentation teaches:

The model answers:

If the conference hotel is sold out, you may book the nearest available hotel [S2]. The nightly cap is 220 euros, or 300 euros in London, Paris and Zurich [S1][S3]. To be reimbursed above the cap, get written approval from your manager before you book [S2]. Approved reports are paid within 5 business days [S3][S4]. You need an itemized hotel receipt for each night [S3]. Reports submitted after the 20th are paid a month later. Let me know if you need help with the report.

It looks well sourced. The shape check from Classic RAG Pipeline passes: there are citations, every label exists, no refusal is glued on. Five of the six factual sentences carry a marker.

Predict first: for how many of those six factual sentences does the passage they cite actually say what the sentence says?

This is not a toy failure. Liu, Zhang and Liang (2023) audited four generative search engines of the day (Bing Chat, NeevaAI, perplexity.ai and YouChat). All of them answered with in-line citations, and still only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported their sentence.

A citation is the model's claim about its source, not proof (RAG Architecture & Implementation). This lesson turns that sentence into numbers you can compute on every answer: whether each label resolves, which sentences needed a source, whether each cited passage supports its sentence, and whether each citation earns its place. Each move starts from a symptom:

Here is the checker you will have built by the end, for one answer:

answer text
  |  split into sentences; a marker after the full stop joins the sentence before it
  |  check-worthy?  offers, questions to the user, the refusal ------> not counted
  |  resolve each label against THIS turn's context set -----------> unknown label
  |  do the cited passages support the sentence? (a judge) --------> supported
  |  if not: does another passage in the context set? --------------> wrong citation: re-cite
  v                                            nothing supports it ---> unsupported claim
per-sentence verdicts -> attachment rate, citation recall, citation precision

The code in this lesson runs in one Python session, like a notebook. This block holds the logged turn and a parser that later blocks reuse. It handles the two marker habits you will meet: [S1][S3] and the malformed [S1, S3].

import re

# One logged turn of the expense assistant: label -> (source, passage text).
CONTEXT = {
    "S1": ("Travel policy v4 > Lodging",
           "The nightly hotel cap is 220 euros, or 300 euros in London, Paris and Zurich. "
           "A stay above the cap needs written approval from your manager before you book."),
    "S2": ("Travel policy v4 > Exceptions",
           "If the official conference hotel is sold out, you may book the nearest available "
           "hotel. Any amount above the cap is reimbursed only with the approval described "
           "under Lodging."),
    "S3": ("Expenses FAQ > Reimbursement",
           "Approved expense reports are paid with the next monthly payroll. Reports submitted "
           "after the 20th of the month are paid one month later."),
    "S4": ("Expense tool guide > Receipts",
           "Upload an itemized hotel receipt for every night. A card statement is not accepted "
           "as a receipt."),
}

ANSWER = ("If the conference hotel is sold out, you may book the nearest available hotel [S2]. "
          "The nightly cap is 220 euros, or 300 euros in London, Paris and Zurich [S1][S3]. "
          "To be reimbursed above the cap, get written approval from your manager before "
          "you book [S2]. "
          "Approved reports are paid within 5 business days [S3][S4]. "
          "You need an itemized hotel receipt for each night [S3]. "
          "Reports submitted after the 20th are paid a month later. "
          "Let me know if you need help with the report.")

MARKERS = re.compile(r"\[(S\d+(?:\s*,\s*S\d+)*)\]")      # [S1][S3], and the malformed [S1, S3]
LEADING = re.compile(r"^((?:\[[^\]]*\]\s*)+)")          # markers that open a piece

def labels_in(text):
    return [label.strip() for group in MARKERS.findall(text) for label in group.split(",")]

def parse_answer(answer):
    """Return [(sentence without markers, [labels it cites])]. A marker written after the
    full stop ("... payroll. [S3] Reports ...") belongs to the sentence before it."""
    parsed = []
    for piece in re.split(r"(?<=[.!?])\s+", answer.strip()):
        lead = LEADING.match(piece)
        if lead and parsed:
            parsed[-1][1].extend(labels_in(lead.group(1)))
            piece = piece[lead.end():]
        text = re.sub(r"\s+([.!?])", r"\1", MARKERS.sub("", piece)).strip()
        if text:
            parsed.append((text, labels_in(piece)))
    return parsed

for n, (text, labels) in enumerate(parse_answer(ANSWER), 1):
    print(n, labels, text)

1 ['S2'] If the conference hotel is sold out, you may book the nearest available hotel.
2 ['S1', 'S3'] The nightly cap is 220 euros, or 300 euros in London, Paris and Zurich.
3 ['S2'] To be reimbursed above the cap, get written approval from your manager before you book.
4 ['S3', 'S4'] Approved reports are paid within 5 business days.
5 ['S3'] You need an itemized hotel receipt for each night.
6 [] Reports submitted after the 20th are paid a month later.
7 [] Let me know if you need help with the report.

The sentence splitter is the simple one from Context Augmentation, and it has the usual limits: "e.g. the" or "2.5 days" will split in the wrong place. For production, use a real sentence segmenter and test it on your own answers.

Before any metric: the unit, the rule and the denominator

Three decisions come before any number, and most citation dashboards that disagree with each other disagree on one of them.

The unit is the sentence. ALCE (Gao et al., EMNLP 2023) segments output "into statements by sentence boundaries", and so does this lesson. A sentence can carry two claims ("the cap is 220 euros and receipts are required"); splitting sentences into atomic claims is finer and costlier, and Faithfulness Testing covers it. Whatever unit you choose, keep it fixed, or this month's number will not be comparable with last month's.

The rule for what needs a source. Liu et al. took the position that every generated statement about the external world is "verification-worthy", even the obvious ones, and listed the exceptions they met in practice: statements about the system itself and questions posed to the user. This lesson calls such sentences check-worthy. "Let me know if you need help with the report" is not; "Reports submitted after the 20th are paid a month later" is, even though it is true. Being true is not the same as being checkable. ALCE did not filter: it counted every sentence, noting that almost all the sentences the models wrote in its tasks needed a citation. So an uncited "Would you like me to open a request?" scores 0 under ALCE and is left out here. On chatty answers the two denominators give different numbers; say which one you use.

The denominator. Four numbers, four different denominators:

Citation recall is what this lesson means by citation coverage: the share of check-worthy sentences that are backed by what they cite. ALCE and Liu et al. both define recall by support, not by the presence of a marker. The marker count has its own name here, attachment rate, so the two cannot be confused.

Two relations are worth knowing by heart, because they catch dashboard bugs:

• Citation recall ≤ attachment rate. A sentence without a valid citation cannot be supported by it. If your recall is above your attachment rate, one of them is computed wrongly.
• Support rate of cited sentences = citation recall ÷ attachment rate. This is the "share of cited sentences that the cited passage really supports" that Context Augmentation asks you to measure. Different denominator, same judgments.

A fifth number sits next to these: faithfulness, whether the claim is supported by something in the context set, cited or not. Faithfulness Testing scores it per atomic claim; this lesson scores it per check-worthy sentence, so that it lines up with citation recall. Citations are the stricter test. An answer can be mostly faithful with bad citations, like the hook answer (five of its six sentences). The reverse cannot happen: if a sentence's cited passages support it, the context set does too, because the cited passages are part of it. So citation recall is never above faithfulness.

Your turn: compute the four numbers and faithfulness for the hook answer from the table in the reveal above. There are six check-worthy sentences and seven citations.

Move 1: Resolve every citation against the turn's context set

Symptom: a marker points to a passage that was not in the prompt, or a citation link opens the wrong page.

Resolution is the cheapest check, and the one to run on every answer before it is shown, whatever else you sample. It asks one question: does this label name a passage that was in the prompt that produced this answer? Five things break it:

1. An invented label. The context set has S1 to S4 and the answer cites [S6].
2. A stale label. In a chat, turn 2 retrieves a new context set and numbers it from S1 again. The model cites [S4] from turn 1.
3. A real id that was not in the prompt. If the model cites raw document ids or URLs, it can name a document that exists in your index but was never in its context. It cannot have read it. Resolve against the logged context set, never against the index.
4. A map built at the wrong time. Labels numbered before reordering point at the wrong passages (Context Augmentation, Challenge 1).
5. Markers copied from the passages. A chunk whose body contains "[S7]" or a numbered reference list teaches the model to copy it (Classic RAG Pipeline).

Cases 1, 3 and 5 are fabrications: the model names something it never read. Cases 2 and 4 are mix-ups: the label names a real passage, just not the one the model read.

Log the context set of every turn with its label map, and resolve against that log. That catches cases 1, 3 and 5. (A copied "[S7]" slips through only if this turn happens to have an S7.) A checker that resolves against the index instead passes case 3, because the id exists there. Per-turn resolution has two blind spots, though:

• Case 2, when the stale number is in range. It is caught in the example below only because turn 2 has two passages. If turn 2 had four, the stale [S4] would resolve to turn 2's own S4, a different and plausible passage. The fix is labels that are unique across the conversation: continue the numbering in turn 2 (S5, S6, …), or derive labels from the turn and the chunk id.
• Case 4, always. The label is in the map; the map points to the wrong passage. The fix is to build labels and map in one pass after the last reordering, and to test that each label's text in the prompt is the text its map entry points to (Context Augmentation, Challenge 1).

The support check of Move 3 is the backstop for all five: a label that points to the wrong passage usually fails it. This block also produces the structural numbers of the previous section:

REFUSAL = "I can't find that in the travel policy."
NOT_A_CLAIM = re.compile(r"^(let me know|would you like|i can also|happy to help)|\?$", re.I)

def check_worthy(sentence):
    """Every statement about the world needs a source. Offers, questions to the user and
    the refusal sentence do not. A crude filter on purpose: measure its errors."""
    return sentence != REFUSAL and not NOT_A_CLAIM.search(sentence)

def rate(part, whole):
    return round(part / whole, 3) if whole else None     # None: nothing to measure (a refusal)

def structural_report(answer, context):
    parsed = parse_answer(answer)
    cited = [label for _, labels in parsed for label in labels]
    claims = [(text, labels) for text, labels in parsed if check_worthy(text)]
    attached = [text for text, labels in claims if any(l in context for l in labels)]
    return {
        "citations": len(cited),
        "unknown labels": sorted({l for l in cited if l not in context}),
        "check-worthy sentences": len(claims),
        "attachment rate": rate(len(attached), len(claims)),
        "uncited claims": [text for text, labels in claims if text not in attached],
    }

print(structural_report(ANSWER, CONTEXT))
print(structural_report(REFUSAL, CONTEXT))

{'citations': 7, 'unknown labels': [], 'check-worthy sentences': 6, 'attachment rate': 0.833, 'uncited claims': ['Reports submitted after the 20th are paid a month later.']}
{'citations': 0, 'unknown labels': [], 'check-worthy sentences': 0, 'attachment rate': None, 'uncited claims': []}

A refusal has nothing to check, so its rate is None, not a crash and not a perfect score. Count refusals separately.

Now the follow-up question, "And train tickets?". The chat keeps one label map for the whole conversation, and new labels overwrite old ones:

TURN_2 = {   # the follow-up "And train tickets?" retrieved a new context set, labelled from S1 again
    "S1": ("Travel policy v4 > Rail",
           "Book second class for trips under four hours. First class is allowed for longer trips."),
    "S2": ("Expense tool guide > Tickets",
           "Upload the e-ticket or the booking confirmation as the receipt."),
}
answer_2 = ("Book second class for trips under four hours [S1]. "
            "Keep the itemized receipt for every ticket [S4].")

chat_map = {**CONTEXT, **TURN_2}      # the bug: one label map for the whole conversation
print("resolved against the chat:", structural_report(answer_2, chat_map)["unknown labels"])
print("resolved against turn 2:  ", structural_report(answer_2, TURN_2)["unknown labels"])

resolved against the chat: []
resolved against turn 2:   ['S4']

Predict: with the chat-wide map, what does the user see under "Keep the itemized receipt for every ticket"?

Don't ask the generator whether its sources exist. In Mata v. Avianca (S.D.N.Y., 2023), lawyers filed a brief citing court cases that ChatGPT had invented. When doubts arose, they asked ChatGPT whether the cases were real, and it assured them that they could be found in legal databases. The judge sanctioned them with a USD 5,000 fine (case summary). There was no retrieval in that story, so no context set to resolve against, and that is the point: a citation is checked against the record, never against the system that wrote it.

What an unknown label should do in production. Never render it as a link. Either replace the answer with your refusal and a route to a human, or drop the sentence and re-check the rest. Which one is a runtime policy for Grounding & Hallucination Control.

Cost: a set lookup per citation. Measure: the unknown-label rate per 1,000 answers, charted daily. A step change almost always follows a template, model or labelling change, so alert on it.

Move 2: Count only check-worthy sentences

Symptom: the uncited-sentence check from Context Augmentation flags "Let me know if you need help with the report" in every other answer, the dashboard says 71% of sentences are cited, and nobody believes it. Meanwhile "Reports submitted after the 20th are paid a month later" goes uncited, and it is exactly the kind of fact someone will act on.

Two fixes, both already in the code above.

Count only check-worthy sentences. check_worthy drops offers, questions to the user and the exact refusal sentence. It is a regular expression, so it will be wrong sometimes. "Would you like the form? It is on the intranet." hides a claim in the second sentence, and the filter handles that correctly only because it judges each sentence separately. Hand-label 100 sentences from real answers as check-worthy or not, and count the filter's mistakes in both directions. An LLM classifier is the next step if the regular expression misses too much.

Attach markers to the right sentence. Models put markers before the full stop ("... payroll [S3].") or after it ("... payroll. [S3]"). A naive split gives the second style to the next sentence.

Predict: a naive parser splits on sentence boundaries and gives each piece the markers it contains. What does it produce for "Approved reports are paid with the next monthly payroll. [S3] Reports submitted after the 20th are paid a month later."?

Attachment tells you what share of the claims a reader could check, not whether the check would succeed: the hook answer scores 0.83 here and 0.33 on citation recall. Report it as a smoke test next to the judged numbers, never instead of them.

Measure: attachment rate on all traffic (it is free), the check-worthiness filter's error rate on a labelled sample, and the rate of markers in the wrong position.

Move 3: Check support, sentence by sentence

Symptom: the cited passage is about the right topic, and it does not say what the sentence says. [S2] under "written approval before you book" is the classic: it mentions approval and never says "written" or "before".

This is the check the published lessons promised: does the passage the sentence cites support it? Three kinds of judge can answer it.

An NLI (natural language inference) model reads a premise and a hypothesis and returns probabilities for entailment, neutral and contradiction. Only entailment counts as support. Neutral means "the passage doesn't say", contradiction means "it says otherwise", and both fail. The label order differs between models, so read it from the model's config instead of hard-coding it. And like reranker scores, these "probabilities" are not calibrated: tune the threshold on sentences people have labelled.

The judge below uses cross-encoder/nli-deberta-v3-base, a small open model that reads at most 512 tokens per pair. It scores each claim against windows of one or two consecutive sentences inside each passage and keeps the best score. That is the idea behind SummaC (Laban et al., TACL 2022): segment the source into sentence units and aggregate pair scores. The next section shows why this beats scoring the whole passage in one pair. A window never joins the end of one passage to the start of the next: such a window would be an accident of citation order, and the score would change when the model wrote [S2][S1] instead of [S1][S2].

from sentence_transformers import CrossEncoder

nli = CrossEncoder("cross-encoder/nli-deberta-v3-base")      # a small open NLI model, 512-token input
ENTAIL = next(i for i, name in nli.config.id2label.items() if name == "entailment")

def sentences(text):
    return [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]

def support(passages, claim):
    """Best entailment probability for `claim`, over windows of one or two consecutive
    sentences inside each passage (the SummaC idea): short premises, nothing truncated.
    No window crosses a passage boundary, so this is the best single passage's score."""
    windows = []
    for p in passages:
        sents = sentences(p)
        windows += sents + [a + " " + b for a, b in zip(sents, sents[1:])]
    if not windows:
        return 0.0
    probs = nli.predict([(w, claim) for w in windows], apply_softmax=True)[:, ENTAIL]
    return float(probs.max())

THRESHOLD = 0.5      # a starting point; tune it on sentences people have labelled

def verdicts(answer, context):
    rows = []
    for text, labels in parse_answer(answer):
        if not check_worthy(text):
            continue
        known = [l for l in labels if l in context]
        cited = support([context[l][1] for l in known], text)
        best = max(context, key=lambda l: support([context[l][1]], text))
        found = support([context[best][1]], text) >= THRESHOLD
        rows.append(dict(text=text, labels=labels, known=known, cited=cited,
                         supported=cited >= THRESHOLD, elsewhere=best if found else None))
    return rows

rows = verdicts(ANSWER, CONTEXT)
for r in rows:
    print(f"{'+'.join(r['labels']) or '-':6} {r['cited']:.3f} "
          f"{'SUPPORTED' if r['supported'] else 'no':9} context: {r['elsewhere'] or '-':3} {r['text'][:44]}")

S2     0.996 SUPPORTED context: S2  If the conference hotel is sold out, you may
S1+S3  0.992 SUPPORTED context: S1  The nightly cap is 220 euros, or 300 euros i
S2     0.000 no        context: S1  To be reimbursed above the cap, get written 
S3+S4  0.000 no        context: -   Approved reports are paid within 5 business 
S3     0.000 no        context: S4  You need an itemized hotel receipt for each 
-      0.000 no        context: S3  Reports submitted after the 20th are paid a 

The judge agrees with the human table in the hook, sentence for sentence. The last column, the best single passage in the whole context set, splits the failures into kinds that need different fixes:

for r in rows:
    if r["supported"]:
        continue
    if r["elsewhere"]:
        print(f"cite as [{r['elsewhere']}]: {r['text']}")
    else:
        print(f"no passage supports it: {r['text']}")

cite as [S1]: To be reimbursed above the cap, get written approval from your manager before you book.
no passage supports it: Approved reports are paid within 5 business days.
cite as [S4]: You need an itemized hotel receipt for each night.
cite as [S3]: Reports submitted after the 20th are paid a month later.

• Wrong citation, right fact. The claim is in the context set, and the pointer is wrong. You can repair it after generation by re-citing the passage the judge found, as long as the judge is reliable enough on your data to be trusted with it. Measure its false passes first: a re-cited sentence can pass against a passage whose condition it dropped, so keep automatic re-citing out of high-risk answers (Grounding & Hallucination Control).
• Unsupported claim. Nothing in the context set says "5 business days". No re-citing fixes this. The sentence must go, or the answer must be regenerated or refused; that policy belongs to Grounding & Hallucination Control. This is the case Context Augmentation warned about: "5 business days" carries two valid labels, so every label check passes it, and only a support check catches it.

Three ways a small NLI judge fails

Long premises get truncated, silently. The model reads 512 tokens per pair, claim included. Real chunks are often 300 to 500 tokens, and a merged neighbour span is longer. Here is the Lodging rule buried at the end of a 666-token section:

def one_pair(premise, claim):
    """What most tutorials do: the whole premise in one NLI call."""
    probs = nli.predict([(premise, claim)], apply_softmax=True)[0]
    return {nli.config.id2label[i]: round(float(p), 3) for i, p in enumerate(probs)}

cap = "The nightly cap is 220 euros, or 300 euros in London, Paris and Zurich."
print("S1 alone:         ", one_pair(CONTEXT["S1"][1], cap))
print("S1 + S3 joined:   ", one_pair(CONTEXT["S1"][1] + " " + CONTEXT["S3"][1], cap))

TOPICS = ["rail tickets", "flights", "taxis", "car rental", "mileage", "parking", "meals",
          "per diems", "visas", "vaccinations", "travel insurance", "conference fees",
          "laundry", "hotel internet", "phone roaming", "tips", "client dinners", "gifts",
          "currency exchange", "cash advances", "corporate cards", "lost receipts",
          "cancellations", "loyalty points", "upgrades", "weekend stays", "training courses",
          "coworking desks", "luggage fees", "airport lounges", "ferries", "bicycle rental",
          "language courses", "medical costs abroad", "late checkout", "minibar charges",
          "trade fair booths", "team offsites", "relocation", "customs duties"]
section = " ".join(f"Section 5.{i} explains which {t} costs are covered and who approves them."
                   for i, t in enumerate(TOPICS, 1)) + " " + CONTEXT["S1"][1]
print("tokens in the long section:", len(nli.tokenizer(section)["input_ids"]))
print("long section, one pair:", one_pair(section, cap)["entailment"])
print("long section, windows: ", round(support([section], cap), 3))

S1 alone:          {'contradiction': 0.0, 'entailment': 0.992, 'neutral': 0.008}
S1 + S3 joined:    {'contradiction': 0.0, 'entailment': 0.001, 'neutral': 0.999}
tokens in the long section: 666
long section, one pair: 0.002
long section, windows:  0.992

The one-pair score dropped to 0.002. predict cut the pair at 512 tokens without an error, and the sentence that supports the claim was in the part it cut. (The length warning you may see comes from the line that counts tokens, not from predict.) Windows find the sentence at 0.992. The fix is not to truncate or summarize the passage first, since both remove evidence or add new text. Split it and score the pieces.

Joining passages can dilute a small model. The second line of output is the surprise. S1 supports the claim at 0.992; S1 followed by the unrelated S3 drops to 0.001. ALCE judges a sentence against its cited passages concatenated, with a much larger judge (TRUE, an 11-billion-parameter T5 model fine-tuned for NLI). With a small model like this one, joining passages is not safe. Windows sidestep it: each window is short, stays inside one passage, and the best one wins.

Implicit and multi-source support. Windows only see one or two adjacent sentences of one passage, so a claim that combines facts from two passages, or needs a small inference, fails. That is the job of the LLM judge. Here is the rubric, with a stub standing in for the model call and quotes you can verify by string matching:

import json

JUDGE_PROMPT = """You check citations in answers about company policy.

Passages:
{passages}

Sentence: {sentence}

Would a careful reader agree with "According to the passages above, {sentence}"?
Say "supported" only if the passages state or directly imply every part of the sentence,
"partial" if they support only some parts, and "not_supported" otherwise.
Quote the exact words you rely on, with the passage label."""

def generate(prompt: str) -> str:
    """Stands in for one LLM call, so the block runs offline. The real call is below."""
    return json.dumps({"verdict": "supported", "quotes": [
        {"label": "S2", "quote": "Any amount above the cap is reimbursed only with the approval"},
        {"label": "S1", "quote": "needs written approval from your manager before you book"}]})

def normalize(text):
    return re.sub(r"\s+", " ", text.replace("’", "'")).strip().lower()

def quote_found(label, quote, context):
    return label in context and normalize(quote) in normalize(context[label][1])

def llm_verdict(sentence, labels, context):
    passages = "\n".join(f"[{l}] {context[l][1]}" for l in labels if l in context)
    reply = json.loads(generate(JUDGE_PROMPT.format(passages=passages, sentence=sentence)))
    reply["quotes_verified"] = all(quote_found(q["label"], q["quote"], context)
                                   for q in reply["quotes"])
    return reply

multi = ("When the conference hotel is sold out, anything above the cap is reimbursed only "
         "if your manager approved it in writing before you booked.")
print("NLI windows:", round(support([CONTEXT["S2"][1], CONTEXT["S1"][1]], multi), 3))
print("LLM judge:  ", llm_verdict(multi, ["S2", "S1"], CONTEXT))

NLI windows: 0.001
LLM judge:   {'verdict': 'supported', 'quotes': [{'label': 'S2', 'quote': 'Any amount above the cap is reimbursed only with the approval'}, {'label': 'S1', 'quote': 'needs written approval from your manager before you book'}], 'quotes_verified': True}

The rubric's wording comes from Liu et al., who asked annotators whether a generic reader would affirm "According to cited webpages, s" (the AIS framework of Rashkin et al.). Asking the judge for quotes costs output tokens, and it buys you something: a judge that invents evidence is caught by a substring check. The real call, as of September 2026 with Anthropic's Python SDK, needs an API key and was not run for this lesson:

# Sketch: needs `pip install anthropic` and an API key. Not run for this lesson.
# It replaces the stub `generate` above; skip this block when you run the lesson offline.
import anthropic

JUDGE_MODEL = "claude-sonnet-5"      # an example id as of 2026-09; pick a current model
VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": ["supported", "partial", "not_supported"]},
        "quotes": {"type": "array", "items": {
            "type": "object",
            "properties": {"label": {"type": "string"}, "quote": {"type": "string"}},
            "required": ["label", "quote"], "additionalProperties": False}},
    },
    "required": ["verdict", "quotes"],
    "additionalProperties": False,
}

def generate(prompt: str) -> str:
    client = anthropic.Anthropic()                       # reads ANTHROPIC_API_KEY
    response = client.messages.create(
        model=JUDGE_MODEL, max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
        output_config={"format": {"type": "json_schema", "schema": VERDICT_SCHEMA}},
    )
    return next(block.text for block in response.content if block.type == "text")

output_config makes the reply match the schema (structured outputs). Batch all the sentences of one answer into one call in production. How to choose, calibrate and de-bias an LLM judge is the subject of Generation Quality.

Measure the judge before you trust the metric

A judge is a classifier, and its errors land in your metric. Label a few hundred sentences by hand, run the judge, and count both kinds of mistake: the false-pass rate (unsupported sentences it judges supported) and the false-fail rate (supported sentences it judges unsupported), the names Generation Quality uses for any judge. Eight labelled sentences show the method:

# 8 sentences a person labelled: (cited labels, sentence, supported?)
LABELED = [
    (["S1"], "In Zurich the nightly cap is 300 euros.", True),
    (["S1"], "In Berlin the nightly cap is 220 euros.", True),     # Berlin is "any other city"
    (["S1"], "The cap is 300 euros in every city.", False),
    (["S1"], "A stay above the cap needs no approval from your manager.", False),
    (["S3"], "Reports submitted after the 20th are paid a month later.", True),
    (["S3"], "Approved reports are paid within 5 business days.", False),
    (["S4"], "A card statement does not count as a receipt.", True),
    (["S2", "S1"], multi, True),                                   # needs both passages
]
agree = false_ok = false_alarm = 0
for labels, sentence, human in LABELED:
    judged = support([CONTEXT[l][1] for l in labels], sentence) >= THRESHOLD
    agree += judged == human
    false_ok += judged and not human
    false_alarm += human and not judged
    if judged != human:
        print("disagree:", sentence)
print(f"agreement {agree}/{len(LABELED)}, judged supported but not: {false_ok}, "
      f"judged unsupported but is: {false_alarm}")

disagree: In Berlin the nightly cap is 220 euros.
disagree: When the conference hotel is sold out, anything above the cap is reimbursed only if your manager approved it in writing before you booked.
agreement 6/8, judged supported but not: 0, judged unsupported but is: 2

On this tiny set the small judge is strict: a false-pass rate of 0 of 3 and a false-fail rate of 2 of 5. It caught the wrong number, the wrong scope and the negation. All three were blunt (an invented figure, a widened scope, an inserted "no"); subtler numeric reasoning and negation are classic hard cases for NLI models, and eight sentences prove nothing about them. It missed an inference ("Berlin" means "any other city") and a two-passage synthesis. A strict judge under-reports citation recall and sends false alarms to your review queue; a lenient one ships unsupported claims. Which error costs more depends on the product, so measure both, per slice (numbers, negations, multi-source sentences). A common setup routes what the NLI judge rejects to the LLM judge, so most sentences never need the expensive call. Faithfulness Testing goes deeper on claim decomposition and on dedicated hallucination detectors.

Move 4: Score every citation

Symptom: after a prompt change that says "cite all relevant sources", sentences arrive with two or three markers each. Citation recall goes up. Readers click through to passages that have nothing to do with the sentence.

Recall alone rewards padding: as Liu et al. put it, a response that cites every webpage on the Internet for each statement would have high citation recall and low precision. Citation precision scores each citation. ALCE's rule, per citation, is:

• a citation is irrelevant if (a) it cannot support the sentence on its own and (b) removing it does not change whether the remaining citations support the sentence;
• a citation scores 1 if its sentence is supported (recall 1) and the citation is not irrelevant;
• precision is averaged over all citations.

Rule (b) is what keeps honest multi-source citations: if two passages each supply half of a sentence, neither supports it alone, but removing either one breaks the support, so both score 1. Liu et al.'s version is similar: a partially supporting citation counts when the union of the sentence's citations fully supports it and no single citation does. ALCE also allowed at most three citations per statement, because "more citations usually do not help".

def precision_flags(row, context):
    """ALCE's rule, per citation: 1 if the sentence is supported by its citations together,
    and this citation is not irrelevant. Irrelevant = it cannot support the sentence alone,
    and removing it leaves the rest still supporting the sentence."""
    flags = {}
    for label in row["labels"]:
        if label not in context or not row["supported"]:
            flags[label] = 0
            continue
        alone = support([context[label][1]], row["text"]) >= THRESHOLD
        rest = [context[l][1] for l in row["known"] if l != label]
        still = bool(rest) and support(rest, row["text"]) >= THRESHOLD
        flags[label] = int(alone or not still)
    return flags

def citation_metrics(rows, context):
    n = len(rows)
    flags = [f for r in rows for f in precision_flags(r, context).values()]
    attached = sum(bool(r["known"]) for r in rows)
    supported = sum(r["supported"] for r in rows)
    return {                                   # rate() from Move 1: None when nothing to count
        "attachment rate": rate(attached, n),
        "citation recall": rate(supported, n),
        "citation precision": rate(sum(flags), len(flags)),
        "support rate of cited sentences": rate(supported, attached),
        "faithfulness (any passage)": rate(sum(r["elsewhere"] is not None for r in rows), n),
    }

for r in rows:
    print(precision_flags(r, CONTEXT), r["text"][:40])
print(citation_metrics(rows, CONTEXT))

{'S2': 1} If the conference hotel is sold out, you
{'S1': 1, 'S3': 0} The nightly cap is 220 euros, or 300 eur
{'S2': 0} To be reimbursed above the cap, get writ
{'S3': 0, 'S4': 0} Approved reports are paid within 5 busin
{'S3': 0} You need an itemized hotel receipt for e
{} Reports submitted after the 20th are pai
{'attachment rate': 0.833, 'citation recall': 0.333, 'citation precision': 0.286, 'support rate of cited sentences': 0.4, 'faithfulness (any passage)': 0.833}

These are the numbers you computed by hand. Both judged numbers in this code rest on the best single passage, because no window crosses a passage boundary. So the code's citation recall means "some cited passage supports the sentence on its own", stricter than ALCE's "the cited passages together", and its faithfulness means "some single passage in the context set supports it". A sentence that needs two passages fails both here; that is the LLM judge's job.

Your turn: a sentence reads "Hotels above the cap need written approval before booking, and the nearest available hotel is allowed when the conference hotel is sold out [S1][S2][S4]." S1 supports the first half, S2 the second, S4 neither. What are its recall and the precision of each citation?

Why precision matters to users. Every irrelevant citation is a link that fails the reader who clicks it, and teaches them that clicking is pointless. Measure: precision on the same judged sample as recall, plus the distribution of citations per sentence. A jump in the share of sentences with three or more markers usually means the prompt is encouraging padding.

Move 5: Choose a citation format you can verify

Symptom: checking one claim means opening a 40-page PDF and searching. The format of a citation decides how much of the checking a machine can do.

Two rules hold across all of them. Finer pointers make every later step cheaper: a quoted span is a short premise for the judge and a precise highlight for the reader. And a pointer that is valid is still not proof. Every format in the table still needs Move 3.

Quotes you can check with string matching

When the model quotes, quote_found from Move 3 checks each quote against its passage after normalizing whitespace, case and curly apostrophes. A paraphrased "quote" fails, which is the right result: it is not what the source says, word for word. Do not loosen this to fuzzy matching without measuring what the looser match lets through.

Provider-native citations

Several LLM APIs return citations themselves. As of September 2026:

• Anthropic Citations: you send document blocks (plain text, PDF or custom content) with citations enabled. Plain text and PDFs are chunked into sentences, and custom content is cited block by block. The response is a list of text blocks, and each cited block carries its citations: cited_text, document_index, document_title and a location. For plain text that is a 0-indexed character range with an exclusive end; for PDFs, 1-indexed pages; for custom content, block indices. The docs say that citations "are guaranteed to contain valid pointers to the provided documents" and that cited_text does not count toward output tokens. For RAG there is also a search_result block (a source, a title and a list of text blocks), which you can put in the user message or return from your own search tool. Its citations carry a search_result_index, counted over all search results in the request (across messages and tool results), and a range of text blocks. Citations on either kind of block cannot be combined with structured outputs (the API returns a 400 error).
• OpenAI's web search tool returns url_citation annotations with start_index, end_index, url and title. Its docs require inline citations from web results to be "clearly visible and clickable" in your interface.
• Cohere's chat with documents returns citations with start and end indices, the cited text span and its sources. An accurate mode (the default) adds citations after the whole answer is generated, and a fast mode emits them inline while streaming.
• Google's Gemini API also returns cited text ranges with source URLs when you ground answers with Google Search; check its current docs for the field names.

Here is what checking a native response looks like. The response below has the shape Anthropic's docs describe. It comes from turn 2 of a chat in which, in turn 1, the user attached their hotel invoice as a document.

# A response in the shape Anthropic's Citations docs describe (content trimmed to text blocks).
# Turn 1 of this chat sent the user's hotel invoice as a document, so it is document 0.
content = [
    {"type": "text", "text": "If the conference hotel is sold out, you may book the nearest available hotel",
     "citations": [{"type": "char_location", "document_index": 2,
                    "document_title": "Travel policy v4 > Exceptions",
                    "cited_text": "If the official conference hotel is sold out, you may book the nearest available hotel. ",
                    "start_char_index": 0, "end_char_index": 88}]},
    {"type": "text", "text": ". "},
    {"type": "text", "text": "Above the cap, you need your manager's written approval before booking",
     "citations": [{"type": "char_location", "document_index": 1,
                    "document_title": "Travel policy v4 > Lodging",
                    "cited_text": "A stay above the cap needs written approval from your manager before you book.",
                    "start_char_index": 78, "end_char_index": 156}]},
    {"type": "text", "text": ". "},
    {"type": "text", "text": "Approved reports are paid with the next monthly payroll",
     "citations": [{"type": "char_location", "document_index": 3,
                    "document_title": "Expenses FAQ > Reimbursement",
                    "cited_text": "Approved expense reports are paid with the next monthly payroll. ",
                    "start_char_index": 0, "end_char_index": 65}]},
    {"type": "text", "text": ", usually within 5 business days."},
]

def check_native(content, documents):
    """documents: (title, text) in the order of ALL document blocks in the request,
    across every message. Returns the answer text and the pointers that do not match."""
    answer, problems = "", []
    for block in content:
        for c in block.get("citations") or []:
            i = c["document_index"]
            if not 0 <= i < len(documents):
                problems.append((i, "no such document"))
                continue
            title, text = documents[i]
            span = text[c["start_char_index"]:c["end_char_index"]]
            if title != c["document_title"] or span.strip() != c["cited_text"].strip():
                problems.append((c["document_index"], title))
        answer += block["text"]
    return answer, problems

this_turn = [CONTEXT[l] for l in ("S1", "S2", "S3", "S4")]
invoice = ("User upload: hotel invoice", "Hotel Lindenhof, 3 nights, 267 euros per night.")
print("indexed from this turn: ", check_native(content, this_turn)[1])
answer, problems = check_native(content, [invoice] + this_turn)
print("indexed across the chat:", problems)
print([t for t in re.split(r"(?<=[.!?])\s+", answer)])
third = "Approved reports are paid with the next monthly payroll, usually within 5 business days."
print("support for the third sentence:", round(support([CONTEXT["S3"][1]], third), 3))

indexed from this turn:  [(2, 'Expenses FAQ > Reimbursement'), (1, 'Travel policy v4 > Exceptions'), (3, 'Expense tool guide > Receipts')]
indexed across the chat: []
['If the conference hotel is sold out, you may book the nearest available hotel.', "Above the cap, you need your manager's written approval before booking.", 'Approved reports are paid with the next monthly payroll, usually within 5 business days.']
support for the third sentence: 0.0

Two lessons in four lines of output:

• The API's pointers were valid; the app's map was not. Anthropic numbers documents across all messages in the request. An app that numbers from the current turn shifts every citation by one, and every highlight lands on the neighbouring passage. A citation of the last document (index 4 here) falls off the end of the current-turn list, which check_native reports as "no such document" instead of crashing. Comparing document_title and cited_text with your own record catches all of it before a user does. The same check catches any mapping bug on your side, which the API cannot see.
• A cited span is not a supported sentence. The third sentence carries a valid citation, so its attachment is perfect. But the citation covers only "paid with the next monthly payroll". The tail, "usually within 5 business days", is uncited and unsupported, and the support check scores the whole sentence at 0.0. With span-level citations, you can also measure the share of answer characters inside a cited span, a finer attachment rate.

Measure: whatever the format, the same four numbers on the same judged sample. That also lets you compare formats: run your labelled questions through label-only, label-plus-quote and native citations, and compare citation recall, precision, tokens and latency.

Move 6: Design the citation UI for checking

Symptom: users trust answers with citations and almost never click them.

The evidence on this is consistent:

• Pew Research Center's browsing data from 900 U.S. adults (March 2025) showed users clicking a link inside a Google AI summary in 1% of visits to pages that had one (Pew, July 2025).
• In a live question-answering experiment, Ding et al. (AAAI 2025) found that trust rose significantly when citations were present, even when the citations were random, and fell when participants checked them.
• Liu et al. found that across the four engines they audited, citation precision was inversely correlated with perceived utility (r = −0.96): the answers that seemed most helpful were often the ones whose citations were least accurate.

Put together: a citation badge buys trust whether or not it is earned, and readers rarely audit it. So your system must do the checking (Moves 1–4), and the interface should make the reader's occasional check cheap and honest:

• Put the marker right after the sentence it supports, not in a list at the end.
• On hover or tap, show the supporting span, highlighted inside its passage, with title, section, date and status (the header fields from Context Augmentation). A native span or a verified quote gives you that span for free.
• Deep-link to the place, meaning the page of a PDF or the anchor of a section, not the top of a 40-page document.
• Never show a badge the checker rejected. An unknown label is never a link. A sentence your judge found unsupported loses its marker, or the answer goes to fallback (Move 1). A re-cited sentence shows the repaired source.
• Collapse duplicates. "[S1][S1]" becomes one marker, and a sentence with two genuine sources shows both.
• Show conflicts with both sources. When two current passages disagree, the answer says so and cites both (Context Augmentation, Move 1).
• Let readers report a wrong source. Each report is a labelled example for your evaluation set.

⚠️ Hiding unsupported sentences improves the look of the answer, not its truth. Count how often the checker rewrites or blocks answers, per 1,000 answers, and review a sample. A spike means retrieval or the prompt changed.

Measure: citation click rate and reports of wrong sources per 1,000 answers, next to the judged metrics. Clicks alone are a poor quality signal. They are rare, and a reader who never clicks may be trusting a bad citation.

Measure it: a citation eval you can rerun

Everything above becomes one evaluation you rerun after every change to the model, prompt, citation format or retrieval.

The labelled set. Take 100–200 real questions and the answers your system gives. For each check-worthy sentence, a person records whether its cited passages support it (yes, partial, no) and, for each citation, whether it supports the sentence fully, partly or not at all. Liu et al.'s question makes a good instruction: would a reader agree with "according to the cited passages, …"? Two labellers on a shared subset tell you how clear your guidelines are. The labels measure the system and the judge at the same time.

Micro or macro, say which. Averaging over all sentences (micro) lets long answers dominate. Averaging per answer, then over answers (macro), gives each answer one vote. Take one answer with 10 check-worthy sentences, 9 of them supported, and one with 2 sentences, none supported. Micro citation recall is 9/12 = 0.75; macro is (0.9 + 0.0)/2 = 0.45. The short, wrong answer almost disappears from the micro number.

Report per slice, not only overall: question type, answer length, sentences with numbers, sentences that cite two or more sources. Comparing two systems on the same questions needs a paired test; Evaluation & Quality Metrics covers confidence intervals and significance.

Was the evidence there at all?

If the context set lacks a fact the answer needs, a complete answer cannot reach citation recall 1 honestly. The right move is to say that the sources don't cover that part: recall stays honest, and the answer stays incomplete. The bad moves are citing a passage that doesn't say it, or stating the fact uncited. Citation metrics catch the bad moves but not the gap, so measure the evidence too. Agentic RAG Systems calls this sufficiency: does the returned set contain every fact the answer needs? Offline, with a reference answer split into key facts, the same support judge computes it:

# Labeled once, offline: the facts a complete answer to the hotel question needs.
KEY_FACTS = [
    "If the conference hotel is sold out, you may book the nearest available hotel.",
    "A stay above the cap needs written approval from your manager before you book.",
    "Approved expense reports are paid with the next monthly payroll.",
    "Reports submitted after the 20th are paid one month later.",
]

def evidence_sufficiency(facts, context):
    """Share of the reference answer's facts that some passage in the context set supports."""
    found = [any(support([text], f) >= THRESHOLD for _, text in context.values()) for f in facts]
    return sum(found) / len(facts), [f for f, ok in zip(facts, found) if not ok]

print(evidence_sufficiency(KEY_FACTS, CONTEXT))
ranked_out = {l: v for l, v in CONTEXT.items() if l != "S1"}    # S1 fell below the cut-off
print(evidence_sufficiency(KEY_FACTS, ranked_out))

(1.0, [])
(0.75, ['A stay above the cap needs written approval from your manager before you book.'])

Evidence sufficiency is 1.0 for the logged turn, so no failure in the hook happened upstream: every fact the answer needed reached the prompt. What went wrong happened after that, in the pointers and in "5 business days". Without S1, it drops to 0.75, and no prompt change can make the approval rule citable. A question whose evidence sufficiency is 1.0 is sufficient, and the sufficiency rate in Evaluation & Quality Metrics is the share of questions that are. Some tools report an LLM-based version of evidence sufficiency as "context recall", computed against a reference answer. That is a different quantity from retrieval recall@k and from the gold-in-prompt rate of Context Augmentation, which check labelled gold chunks rather than the facts of a reference answer (Retrieval Metrics). At run time you have no reference answer; the sufficiency check of Agentic RAG Systems is the online stand-in.

Keep the "coverage" words apart: citation coverage in this lesson is citation recall; index coverage in Evaluation & Quality Metrics is the share of gold documents indexed in their current version; and Grounding & Hallucination Control uses coverage for the share of answerable questions that get an answer.

Where the fix lives

The failure taxonomy of RAG Architecture & Implementation follows the gold chunk: not indexed, not retrieved, ranked out, lost in assembly, or ignored or contradicted. Some citation failures fit one of its buckets. Others are pointer errors: the evidence reached the prompt, and only the citation is wrong. Those sit outside the taxonomy, and saying so keeps them from being "fixed" in retrieval.

What the checking costs

Assume an assistant with 20,000 answers a day and 6 check-worthy sentences per answer, each cited passage about 15 sentences long (so about 30 windows per sentence):

• NLI judge on all traffic: 20,000 × 6 × 30 = 3.6 million pairs a day. We timed the model above on an Apple M2 Max laptop, scoring pairs of 20 to 55 tokens (a one- or two-sentence window plus a claim) in batches of 32. It ran about 550 pairs a second on the GPU and about 120 on the CPU (8 threads) with nothing else running, and about 400–500 and 50–70 while another job kept several CPU cores busy. Measure yours. That is about 1.8 to 2.5 GPU-hours, or 8 to 20 CPU-hours, a day. Affordable, on hardware you plan for.
• LLM judge: assume one call per answer, 2,500 input tokens (passages, sentences, rubric) and 400 output tokens (verdicts and quotes), at an assumed USD 2 per million input and USD 10 per million output tokens (the list price of the judge model in the code above in September 2026; check your provider's price page). That is USD 9 per 1,000 answers: USD 180 a day on all traffic, or USD 9 a day on a 5% sample.

A common split: resolution and attachment on 100% of answers (free), the NLI judge on 100% or a large share, the LLM judge on what the NLI judge rejects and on a daily random sample, and people on a weekly sample that keeps the judges honest.

Final round: no label on the problem

Challenge 1: the one-source rule

To reduce clutter, a team changes the prompt from "cite the passages that support each sentence" to "cite exactly one passage per sentence". On the 150-question eval (micro-averaged, unknown-label rate 0 before and after), attachment stays at 0.97, citation precision rises from 0.62 to 0.68, faithfulness stays at 0.88, and citation recall falls from 0.81 to 0.66. Readers file no more complaints than before. Ship it?

Challenge 2: the popover one passage off

After a release that lets users attach files to the chat, about 30% of citation popovers highlight a neighbouring policy passage instead of the one the sentence cites, and a few popovers do not open at all. The app uses Anthropic's native citations. The unknown-label rate is 0, and no API errors are logged.

Challenge 3: the green dashboard

The weekly report shows NLI-judged citation recall of 0.94. A reviewer reads 100 random answers and finds that 71% of check-worthy sentences are supported by what they cite. The support function in production is called like this: support([text for _, text in context.values()], sentence).

Cheat sheet: symptom → move → cost

Before moving on: measure it on your own data

Pull 100 answers from your logs, with the logged context set of each turn. Label every check-worthy sentence: supported by its citations or not, and, if not, whether another passage in the prompt supports it. Then compute the unknown-label rate, attachment rate, citation recall, citation precision and faithfulness, micro and macro, and put your judge's verdicts next to the labels. If citation recall is far below faithfulness, your citations are the problem, not your facts. If faithfulness is low too, check evidence sufficiency: below 1.0 points upstream, and 1.0 means the model ignored or contradicted what it had.

Then explain one challenge from this lesson aloud from a blank page: which number moved, which denominator it uses, what the judge sees, where the fix lives, and which measurement would prove it worked.

Next: Grounding & Hallucination Control decides what to do at run time with a sentence that fails these checks, Faithfulness Testing goes deeper into judging support claim by claim, and Generation Quality covers the other dimensions of a good answer.