Faithfulness Testing

The refund that was never on offer

Larkspur (a fictional company) sells a team workspace app, and its help-center assistant is a RAG system. Every night an evaluation job replays yesterday's questions and computes a "groundedness" score for each answer: the cosine similarity between the answer and the passage it cites. Anything below 0.7 raises an alert. The alert has never fired.

A customer asks: "If I cancel our Team plan today, do we get this month's fee back?" Retrieval does its job and puts the right passage in the prompt. Here is that passage, the answer the assistant gave last week (A), the answer it gives since a prompt change (B), and a third answer (C) a support agent wrote by hand:

from sentence_transformers import SentenceTransformer

passage = ("Team plans are billed monthly and can be cancelled at any time, but the current "
           "month is not refunded. Annual Enterprise contracts can be refunded in full within "
           "14 days of signing; after that they are not refundable.")
answers = {
    "A": "You can cancel your Team plan at any time, but the current month is not refunded.",
    "B": "You can cancel your Team plan at any time, and the current month is refunded.",
    "C": "You can stop your Team subscription whenever you like, but you will not get "
         "this month's payment back.",
}
embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
p = embedder.encode(passage)                 # unit-length vectors: dot product = cosine
for name, text in answers.items():
    print(name, round(float(embedder.encode(text) @ p), 3))

Predict first: B promises a refund the passage explicitly rules out. Does the 0.7 alert catch it? Which answer scores lowest?

A 0.769
B 0.765
C 0.548

The metric is not broken; it measures the wrong thing. An embedding summarises what a text is about, and A and B are about the same thing. One "not" barely moves the vector (Semantic Search Principles shows why). The question you actually care about is different: does the answer follow from the passage? Answering it takes a model that reads both texts together, and a procedure around it that you can trust. That procedure is this lesson.

What "faithful" means in this course

An answer is faithful to its context when every claim it makes is supported by the context set that was actually in the prompt.

Read the context set from the prompt log, not by re-running retrieval. You are testing what the model saw. Faithfulness is a relation between the answer and that context, not between the answer and the world, and it can disagree with correctness in both directions:

RAG Architecture & Implementation sorts RAG failures into buckets by where the gold chunk got lost. A faithfulness check is the test for one of them: ignored or contradicted, where the right chunk was in the prompt and the answer went its own way. It says nothing about the other buckets. When the right chunk was never retrieved, the model may faithfully answer "I can't find that", which some tools score as perfectly faithful (Move 3) and which helps no one. So you always read faithfulness next to retrieval metrics (Retrieval Metrics) and answer quality (Generation Quality).

This lesson is about offline testing: a fixed, labeled set of questions that you run after every change and on a schedule, where a slow check and a human reviewer are affordable. Blocking or repairing a bad answer while the customer waits is runtime control, with a latency budget and different trade-offs; it lives in Grounding & Hallucination Control. Faithfulness asks whether each claim is supported by anything in the context. The stricter, citation-level question (are a sentence's own cited passages enough to support it, and does each citation help?) is citation recall and citation precision, in Citation Coverage.

Here is the map. Each move starts from a symptom you can see in scores or reviews:

This lesson assumes the RAG pipeline and its failure buckets from the lessons linked above.

Move 1: Ask a model that reads both texts

Symptom: a similarity or word-overlap score cannot tell "is refunded" from "is not refunded".

Natural language inference (NLI) is the task built for this question. Given a premise (the passage) and a hypothesis (a claim), a model says whether the premise entails the hypothesis, contradicts it, or is neutral (does not settle it). Public NLI models were trained on sentence pairs from datasets such as SNLI and MultiNLI. Architecturally they are cross-encoders, like the rerankers in Reranking Models: both texts go through the transformer together, so "not" in one can interact with "refunded" in the other. The difference is the head: three labels instead of one relevance score.

Continuing from the block above, with a small public model, cross-encoder/nli-deberta-v3-base (Apache 2.0):

import numpy as np
from sentence_transformers import CrossEncoder

nli = CrossEncoder("cross-encoder/nli-deberta-v3-base")
LABELS = [nli.model.config.id2label[i] for i in range(3)]   # read the order; never assume it
print(LABELS)

def nli_probs(premise, hypothesis):
    logits = nli.predict([(premise, hypothesis)])[0]
    p = np.exp(logits - logits.max())
    return {label: round(float(x), 3) for label, x in zip(LABELS, p / p.sum())}

for name, text in answers.items():
    print(name, nli_probs(passage, text))

['contradiction', 'entailment', 'neutral']
A {'contradiction': 0.0, 'entailment': 0.997, 'neutral': 0.003}
B {'contradiction': 0.993, 'entailment': 0.002, 'neutral': 0.005}
C {'contradiction': 0.0, 'entailment': 0.995, 'neutral': 0.005}

That is the separation similarity could not give: A and C entailed, B contradicted, each by a wide margin.

⚠️ Read the label order from the model. This model's outputs are ordered contradiction, entailment, neutral. facebook/bart-large-mnli, another widely used NLI model, orders them contradiction, neutral, entailment. Swap models while keeping a hard-coded index, and you silently read one label's probability as another's. Nothing raises an error, and the new numbers look just as plausible (the final round has an example). model.config.id2label is the source of truth.

Turning labels into verdicts:

The three numbers come from a softmax. They are useful for ranking and for thresholds you tune on labeled data. They are not calibrated probabilities, for the same reason reranker scores are not.

Predict: a support agent writes the answer more casually: "Yes, cancel whenever you want. This month's charge stays, though." Does the model call it entailed?

print(nli_probs(passage, "Yes, cancel whenever you want. This month's charge stays, though."))
# {'contradiction': 0.0, 'entailment': 0.159, 'neutral': 0.841}

Move 2: Split the answer into claims

Symptom: one verdict for a whole answer. The wrong detail passes because the rest of the answer is right.

A new question: "We're moving our 8-person team to the Team plan. What will it cost, and what do we get?" The context set now holds four help-center passages. check_claim below keeps every premise short: it tests the claim against sentence windows (one sentence, or two adjacent sentences of the same passage) and keeps the best entailment. These toy passages have one or two sentences, so a window is often the whole passage; on real 300-token chunks it is not, and Move 4 shows why that matters. Continuing from above:

import re

PASSAGES = {
    "plans": "The Team plan costs €12 per user per month and includes up to 10 users. "
             "Single sign-on (SSO) is available only on the Enterprise plan.",
    "billing": passage,
    "storage": "Each user on the Team plan gets 50 GB of storage, pooled across the workspace.",
    "export": "Workspace owners can export all data as a ZIP file from Settings. "
              "Export links expire after 7 days.",
}

def split_sentences(text):
    return [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]

def windows(passages):
    """(passage id, text) for every sentence and every pair of adjacent sentences.
    No window crosses from one passage into the next."""
    out = []
    for pid, text in passages.items():
        sents = split_sentences(text)
        out += [(pid, s) for s in sents] + [(pid, a + " " + b) for a, b in zip(sents, sents[1:])]
    return out

def check_claim(claim, passages=PASSAGES, threshold=0.5):
    """Supported if some sentence window of some passage entails the claim."""
    wins = windows(passages)
    logits = nli.predict([(text, claim) for _, text in wins])
    probs = np.exp(logits - logits.max(axis=1, keepdims=True))
    entail = (probs / probs.sum(axis=1, keepdims=True))[:, LABELS.index("entailment")]
    best = int(entail.argmax())
    verdict = "supported" if entail[best] >= threshold else "not supported"
    return verdict, wins[best][0], round(float(entail[best]), 3)

answer = ("The Team plan costs €12 per user per month, so your 8 users would pay €96 a month. "
          "It includes up to 10 users and SSO. "
          "Each user gets 50 GB of storage, pooled across the workspace. "
          "You can cancel at any time and get the current month refunded.")

print("whole answer:", nli_probs(" ".join(PASSAGES.values()), answer))
for sentence in split_sentences(answer):
    print(check_claim(sentence), sentence)

The answer has two errors: SSO is Enterprise-only, and the current month is not refunded. It also has one correct calculation (12 × 8 = 96) that no passage states. Predict: does the whole-answer check catch it? Which sentences pass?

whole answer: {'contradiction': 0.231, 'entailment': 0.615, 'neutral': 0.155}
('not supported', 'plans', 0.003) The Team plan costs €12 per user per month, so your 8 users would pay €96 a month.
('supported', 'plans', 0.573) It includes up to 10 users and SSO.
('supported', 'storage', 0.991) Each user gets 50 GB of storage, pooled across the workspace.
('not supported', 'billing', 0.017) You can cancel at any time and get the current month refunded.

Two things went wrong at the sentence level. A sentence can carry several facts, and a verdict on the sentence is a verdict on some blend of them. That is partial support. And a sentence can lean on its neighbours: "It" means the Team plan only because of the sentence before. The fix for both is to verify atomic claims: short, standalone statements of one fact each, with pronouns replaced by what they refer to.

This is the idea behind FActScore (Min et al., EMNLP 2023), which scores long generations by "the percentage of atomic facts supported by a reliable knowledge source", and it is the first step of the faithfulness metrics in Ragas and DeepEval (TruLens, in Move 3, splits into sentences instead). Ragas's statement generator, for example, is told to "Break down each sentence into one or more fully understandable statements. Ensure that no pronouns are used in any statement."

A rule can split sentences. Splitting facts inside a sentence needs a language model. Here is a decomposer running a small open model locally through one provider-agnostic function. Qwen/Qwen3-4B-Instruct-2507 (Apache 2.0) is an example as of 2026, not a recommendation; it downloads about 8 GB. Swap the body of generate for any API client and nothing else changes:

import json
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

GEN_MODEL = "Qwen/Qwen3-4B-Instruct-2507"      # pick your own; keep it in one constant
device = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu")
tok = AutoTokenizer.from_pretrained(GEN_MODEL)
llm = AutoModelForCausalLM.from_pretrained(GEN_MODEL, dtype=torch.bfloat16).to(device)

def generate(prompt: str, max_new_tokens: int = 800) -> str:
    enc = tok.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True,
                                  return_tensors="pt", return_dict=True).to(device)
    out = llm.generate(**enc, max_new_tokens=max_new_tokens, do_sample=False)   # greedy
    return tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)

def parse_json(text):
    return json.loads(re.sub(r"^```(?:json)?\s*|\s*```$", "", text.strip()))

DECOMPOSE = """Break the answer into atomic claims: short, standalone sentences that each state one fact.
Replace pronouns with the names they refer to. Keep numbers, conditions and negations exactly.
Each claim must contain exactly one fact: if a sentence lists several things or joins facts with
"and", "so" or a comma, write one claim per thing. Do not add anything the answer does not say.

Example
Answer: The Basic plan is free and includes 3 projects and email support. You can upgrade later and keep your files.
Claims: ["The Basic plan is free.", "The Basic plan includes 3 projects.", "The Basic plan includes email support.",
"You can upgrade the Basic plan later.", "You keep your files when you upgrade the Basic plan."]

Question: {question}
Answer: {answer}
Claims:"""

question = "We're moving our 8-person team to the Team plan. What will it cost, and what do we get?"
for claim in parse_json(generate(DECOMPOSE.format(question=question, answer=answer))):
    print("-", claim)

- The Team plan costs €12 per user per month.
- Eight users would pay €96 a month.
- The Team plan includes up to 10 users.
- The Team plan includes SSO.
- Each user gets 50 GB of storage.
- The storage is pooled across the workspace.
- You can cancel the Team plan at any time.
- You get the current month refunded when you cancel the Team plan.

That prompt took two revisions. Without the explicit "one claim per thing" rule and an example that lists features, the model kept "The Team plan includes up to 10 users and SSO" in one piece: exactly the partial-support trap from above. The decomposer is a model too, and its mistakes are invisible downstream. A missed claim is never checked; a merged claim is judged on its easier half; a claim with an added detail can flip a verdict. Here it added "the Team plan" to the cancellation claims, which happens to be right because the passage covers Team plans. Measure it: decompose 20–30 real answers, and have a person count missed, merged and invented claims.

Now verify each claim:

# the eight claims printed above (paste them if you skipped the model)
claims = [
    "The Team plan costs €12 per user per month.",
    "Eight users would pay €96 a month.",
    "The Team plan includes up to 10 users.",
    "The Team plan includes SSO.",
    "Each user gets 50 GB of storage.",
    "The storage is pooled across the workspace.",
    "You can cancel the Team plan at any time.",
    "You get the current month refunded when you cancel the Team plan.",
]
verdicts = [check_claim(c)[0] for c in claims]
for c, v in zip(claims, verdicts):
    print(f"{v:14s} {c}")

def faithfulness(verdicts):
    """Supported claims / all claims. Undefined (NaN) when there are no claims."""
    return sum(v == "supported" for v in verdicts) / len(verdicts) if verdicts else float("nan")

print("faithfulness:", faithfulness(verdicts))

supported      The Team plan costs €12 per user per month.
not supported  Eight users would pay €96 a month.
supported      The Team plan includes up to 10 users.
not supported  The Team plan includes SSO.
supported      Each user gets 50 GB of storage.
supported      The storage is pooled across the workspace.
supported      You can cancel the Team plan at any time.
not supported  You get the current month refunded when you cancel the Team plan.
faithfulness: 0.625

Both false claims are caught, and each failure now points at one fact you can show a developer. A careful reader agrees with every verdict but one: €96 is correct arithmetic on a stated price, and the model cannot multiply. By the reader's labels the answer scores 6/8 = 0.75; the checker says 5/8 = 0.625. Keep that gap in mind for Move 4.

Move 3: Say exactly what the score is

Symptom: two evaluation tools report different faithfulness for the same answers, and the team argues about which is right.

The course's definition, for one answer:

faithfulness(answer) = claims judged supported / all claims in the answer

Every word in it hides a choice: who extracts the claims, what counts as "supported", and what happens when there are no claims. Common tools make those choices differently. Here is what their current source code does (read on 2026-09-25):

The DeepEval default is the one that surprises people. Its docs say it plainly: a claim is truthful if it "does not contradict any facts presented in the retrieval_context". So by default it measures non-contradiction, not support. An answer that invents a plausible extra fact loses nothing.

Your turn. An answer to "Can I export our data?": "Workspace owners can export all data as a ZIP file from Settings. Exports are encrypted with AES-256. Export links expire after 7 days." The first and third claims are in the export passage; the second is in no passage. Compute the score under the Ragas definition, under DeepEval's default, and with penalize_ambiguous_claims=True. Then TruLens, assuming its judge scores the three sentences 3, 0 and 3.

Averages hide the answer that matters

A test set has many answers, so you need an aggregate, and the two obvious ones disagree. Take two answers: one with a single claim, unsupported; one with 19 claims, all supported.

• Mean of per-answer scores (macro): (0 + 1) / 2 = 0.50.
• Share of all claims supported (micro): 19 / 20 = 0.95.

Neither is wrong. Say which one you report.

Neither is a good gate, either. Suppose every answer in a release has five claims and one of them is unsupported. Every answer scores 0.8, so a gate at "mean ≥ 0.8" passes the release, and every single answer contains something the sources do not say. One wrong refund promise per answer is not "80% fine" to the customer who gets it. For gating, count answers:

unsupported-answer rate = share of answers with at least one claim that is not supported.

Report the claim-level score for trends and debugging, and gate on the answer-level rate.

Refusals need a rule, too. An answer that says "I can't find that in the help center" has no factual claims. Ragas returns NaN, DeepEval returns 1.0, and TruLens gives abstentions full marks (or 0.0, if its trivial-statement filter drops the refusal first). Exclude refusals from the faithfulness average, so that none of them counts as a perfect answer, and report next to the score how many answers were refusals.

Excluding them does not stop the quieter trick. A system that refuses exactly the questions it would have got wrong raises its score: ten answers, eight fully supported and two at 0.5, average 0.90; refuse those two, and the eight that are left average 1.00. So a faithfulness score that rises while refusals rise is not an improvement until you know whether the refusals were right. That takes an "answerable?" label per question: offline, the false refusal and false answer rates in Generation Quality; at run time, abstention in Grounding & Hallucination Control.

Move 4: Know where claim checking breaks

Symptom: the checker passes answers your reviewers flag, or flags answers they accept.

The company a fact keeps

check_claim tests short sentence windows and keeps the best entailment. Why not hand the model a whole passage, or the whole context at once? Continuing from above:

NOISE = [
    "Guests can view shared boards but cannot edit them.",
    "Admins can require two-factor authentication for all members.",
    "Deleted items stay in the trash for 30 days.",
    "The mobile app supports offline editing.",
    "Invoices are emailed to the billing contact on the first day of each month.",
    "API requests are limited to 100 per minute per workspace.",
    "Webhooks retry failed deliveries up to five times.",
    "Audit logs are kept for one year on the Enterprise plan.",
    "Custom domains require DNS verification.",
    "Support replies within one business day on the Team plan.",
]
claim = "Annual Enterprise contracts can be refunded in full within 14 days of signing."
blob = " ".join(NOISE[:5] + [PASSAGES["billing"]] + NOISE[5:])
print("billing passage alone:", nli_probs(PASSAGES["billing"], claim))
print("inside 10 other lines:", nli_probs(blob, claim))
print("40 lines, no refunds: ", nli_probs(" ".join(NOISE * 4), claim))
print("all four passages:    ", nli_probs(" ".join(PASSAGES.values()), "SSO requires the Enterprise plan."))
print("plans passage alone:  ", nli_probs(PASSAGES["plans"], "SSO requires the Enterprise plan."))

billing passage alone: {'contradiction': 0.0, 'entailment': 0.997, 'neutral': 0.003}
inside 10 other lines: {'contradiction': 0.0, 'entailment': 0.02, 'neutral': 0.979}
40 lines, no refunds:  {'contradiction': 0.004, 'entailment': 0.907, 'neutral': 0.089}
all four passages:     {'contradiction': 0.0, 'entailment': 0.004, 'neutral': 0.996}
plans passage alone:   {'contradiction': 0.0, 'entailment': 0.991, 'neutral': 0.009}

The claim is copied word for word from the premise, yet surrounded by ten unrelated lines (164 tokens) it scores neutral 0.979. It fails in the other direction too: the ten unrelated lines repeated four times (437 tokens, not one word about refunds) "entail" it at 0.907. A bigger model fails differently, not never. With cross-encoder/nli-deberta-v3-large in the same code, the ten-line case comes out right (entailment 0.957), and the 40 unrelated lines still "entail" the claim at 0.739.

This is a known problem with a name. SummaC (Laban et al., TACL 2022) describes "a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level)" and fixes it by splitting documents into sentences and aggregating over sentence pairs. Two practical rules follow:

• Verify against sentence windows, one sentence or two adjacent sentences inside one passage, and keep the best entailment. That is what check_claim does. The cost is claims × windows model calls, each on a short pair.
• Mind the input limit. These DeBERTa models take 512 tokens for premise and claim together, and CrossEncoder truncates longer pairs without an error. A fact at the end of a long passage can be cut off before the model ever sees it. A sentence window is far below the limit.

⚠️ Do not read "contradiction" from passages about other topics. The storage passage "contradicts" "SSO requires the Enterprise plan" with 0.998, and the export passage "contradicts" "You can cancel at any time." with 0.956. NLI models often say contradiction for off-topic pairs. That is why check_claim decides on entailment alone. If you want "contradicted" versus "not found" as a diagnostic, take it from the windows of the passage the claim cites, or from the judge in Move 5.

Ten claims that separate checkers

HARD_CASES = [
    # (case, claim, what a careful reader says)
    ("paraphrase", "SSO requires the Enterprise plan.", "supported"),
    ("number in words", "Every Team user gets fifty gigabytes of storage.", "supported"),
    ("number swap", "The Team plan includes up to 12 users.", "contradicted"),
    ("negation flip", "SSO is available on the Team plan.", "contradicted"),
    ("added fact", "Exports are encrypted with AES-256.", "not in the sources"),
    ("dropped condition", "Annual Enterprise contracts can be refunded in full.", "misleading"),
    ("derived number", "A Team workspace with 8 users has 400 GB of pooled storage.", "supported (8 × 50)"),
    ("derived number, wrong", "A Team workspace with 8 users has 500 GB of pooled storage.", "contradicted"),
    ("needs two passages", "Team plan users pay €12 per month each and get 50 GB of storage each.", "supported"),
    ("two facts, one wrong", "The Team plan includes up to 10 users and SSO.", "partly contradicted"),
]
for case, claim, human in HARD_CASES:
    verdict, _, p = check_claim(claim)
    print(f"{case:22s} {verdict:14s} {p:5.3f}   reader: {human}")

paraphrase             supported      0.994   reader: supported
number in words        supported      0.998   reader: supported
number swap            not supported  0.001   reader: contradicted
negation flip          not supported  0.007   reader: contradicted
added fact             not supported  0.000   reader: not in the sources
dropped condition      supported      0.997   reader: misleading
derived number         not supported  0.035   reader: supported (8 × 50)
derived number, wrong  not supported  0.010   reader: contradicted
needs two passages     not supported  0.000   reader: supported
two facts, one wrong   supported      0.994   reader: partly contradicted

We ran the same ten claims through two dedicated detectors, on the same sentence windows, and through the LLM judge from Move 5, which reads the four passages whole, one call per claim. HHEM-2.1-Open is Vectara's open hallucination detector (flan-t5-base, Apache 2.0, English, one consistency score from 0 to 1). MiniCheck-Flan-T5-Large is a fact-checker from Tang, Laban & Durrett (EMNLP 2024). For both detectors, 0.5 is the pass mark. nli-deberta-v3-large gave the same ten verdicts as the base model. HHEM's custom model code failed to load on transformers 5.17, so both classifiers ran on transformers 4.57.

Ten hand-made claims are not a benchmark, and none of these tools is being ranked here. The table shows why you run candidates on your hard cases before trusting one. Read it by row:

• Numbers. A swapped number is usually caught, but HHEM and MiniCheck both passed "up to 12 users" (0.65 and 0.54). No classifier can do arithmetic: each flagged the correct 400 GB exactly like the wrong 500 GB. The small judge did worse and got both backwards. Derived numbers need a calculator, not a better classifier: flag any number in a claim that no passage contains, and have a person or code verify the calculation.
• Negation. NLI and MiniCheck caught the SSO flip. HHEM passed it at 0.79, and also passed "When you cancel a Team plan, the current month is refunded" at 0.94, the very error this lesson opened with.
• Paraphrase. Fine here, but recall the casual answer from Move 1 that NLI called neutral.
• Partial support. Every classifier passed the "10 users and SSO" claim. Only the judge caught it. Good decomposition fixes this for all of them.
• Needs two passages. Each window holds half the claim, so every classifier flags it. The judge, which reads all four passages, passes it. Joining the passages looks like the fix for the classifiers; the next section shows why it is not.
• Dropped condition. Everything passes it, the judge included. "Annual Enterprise contracts can be refunded in full" is entailed by the passage; what makes it misleading is the question it answers. Claim-level checking cannot see this by construction.

Joining passages lets false claims through

If each window holds half of a claim, why not hand the classifier both passages at once, but only for claims that every window rejected? Try it on the true two-passage claim and on a claim that looks like it:

joined = PASSAGES["plans"] + " " + PASSAGES["storage"]
for c in ["Team plan users pay €12 per month each and get 50 GB of storage each.",
          "Team plan users get SSO and 50 GB of storage each."]:
    print(check_claim(c)[2], nli_probs(joined, c)["entailment"], c)

Predict: the second claim is half false (SSO is Enterprise-only). Does the joined premise catch it?

0.0 0.998 Team plan users pay €12 per month each and get 50 GB of storage each.
0.058 0.993 Team plan users get SSO and 50 GB of storage each.

So keep every window inside one passage, and send claims that no window supports to the LLM judge of Move 5, which reads all the passages and can combine them. Citation Coverage reaches the same rule for citations: with a small NLI model, joining passages is not safe.

Dropped conditions need the question

To catch an answer that is true but leaves out the limit that matters, give a checker the question, the answer and the sources, and ask what is missing. Continuing with generate from Move 2:

OMIT = """A support answer may be misleading when it leaves out a condition, limit or exception
that the sources attach to it. Read the question, the answer and the sources.
List each condition, limit or exception in the sources that changes the answer to this question
and that the answer leaves out. List nothing that the answer already states or that does not apply.

Question: {question}
Answer: {answer}
Sources:
{passages}

Reply with a JSON list of strings only (an empty list if nothing is missing)."""

def missing_conditions(question, answer, passages=PASSAGES):
    sources = "\n".join(f"[{k}] {v}" for k, v in passages.items())
    return parse_json(generate(OMIT.format(question=question, answer=answer, passages=sources)))

print(missing_conditions("Can we get a refund on our annual Enterprise contract?",
                         "Yes, annual Enterprise contracts can be refunded in full."))

['Annual Enterprise contracts can only be refunded in full within 14 days of signing; after that they are not refundable.']

It found the 14-day limit. Before you celebrate, look at its false alarms. We ran it on four more answers that a person would accept, and it flagged all four. Two flags do not apply: it attached the Enterprise refund rule to a question about cancelling a Team plan, and it warned the Enterprise customer, whose answer already stated the 14-day limit, that the rule does not cover Team plans. Two are debatable, and a reviewer would have to decide: the storage answer "Each user gets 50 GB of storage" leaves out that the storage is pooled, and the export answer does not mention that export links expire after 7 days. One real catch in five answers, four flags to review, two of them useless. Use a check like this to fill a review queue, not a gate, and measure its precision on labeled answers before you let it block anything.

Faithful to the wrong passage is the last hard case, and no faithfulness check will ever catch it. If the prompt held last year's refund policy and the answer quotes it, every claim is supported. That failure is fixed where the stale passage came from (Data Freshness & Lifecycle) and where conflicting versions are assembled into one prompt (Context Augmentation).

Your turn: write numbers_not_in_sources(claim), returning the numbers in a claim that appear in no passage. What does it return for "Eight users would pay €96 a month." and for "The Team plan includes up to 12 users."? Which of the two is a real error?

def numbers_not_in_sources(claim, passages=PASSAGES):
    source_numbers = set(re.findall(r"\d+(?:\.\d+)?", " ".join(passages.values())))
    return [n for n in re.findall(r"\d+(?:\.\d+)?", claim) if n not in source_numbers]

print(numbers_not_in_sources("Eight users would pay €96 a month."))       # ['96']
print(numbers_not_in_sources("The Team plan includes up to 12 users."))   # []

Move 5: Bring in an LLM judge where NLI runs out

Symptom: the NLI checker flags correct claims that need reading across passages, and it cannot tell "contradicted" from "not found" reliably.

An LLM judge is the grader most teams reach for, and the one the Ragas, DeepEval and TruLens metrics use under the hood. It reads the whole context, handles informal wording, and can label why a claim failed. It is also slower, costs money per call, and is a model with its own biases. The way to use it well is the same as for NLI: claim by claim, with labels you define and output you validate. Continuing from above:

JUDGE = """You check whether claims are supported by source passages.
Label each claim:
SUPPORTED: the passages state it, or it follows from them (arithmetic on stated numbers counts).
CONTRADICTED: the passages state something incompatible with it.
NOT_FOUND: the passages do not settle it either way.
Use only the passages, never your own knowledge. Quote the words you relied on.

Passages:
{passages}

Claims:
{claims}

Reply with a JSON list only, one object per claim, in order:
[{{"claim": "...", "label": "SUPPORTED|CONTRADICTED|NOT_FOUND", "quote": "..."}}]"""

def judge(claims, passages=PASSAGES):
    prompt = JUDGE.format(passages="\n".join(f"[{k}] {v}" for k, v in passages.items()),
                          claims="\n".join(f"{i}. {c}" for i, c in enumerate(claims, 1)))
    verdicts = parse_json(generate(prompt))        # raises on invalid JSON: never a silent pass
    assert len(verdicts) == len(claims), "judge skipped or merged claims"
    return [v["label"] for v in verdicts]

print(judge(claims))

['SUPPORTED', 'NOT_FOUND', 'SUPPORTED', 'CONTRADICTED', 'SUPPORTED', 'SUPPORTED', 'SUPPORTED', 'CONTRADICTED']

On supported versus not supported, the judge agrees with the NLI checker on all eight claims, and it adds the diagnosis: SSO and the refund are contradicted, not merely missing. It also shares the checker's mistake. It calls €96 NOT_FOUND, although its prompt says arithmetic counts, and sent alone the same claim came back CONTRADICTED. The prompt design choices each exist for a reason:

• Labels defined in the prompt, with NOT_FOUND spelled out, so "the passage does not say" cannot drift into a pass. (DeepEval's "borderline" is exactly that drift, made the default.)
• Only the passages. A judge that uses its own knowledge turns the faithfulness test into a trivia test.
• A quote per verdict, so a reviewer can check the judge in seconds.
• Validated output. While we tuned a variant of this prompt, the same small model returned JSON with a trailing comma and, for a one-claim request, two verdicts. Parse strictly, check the count, and treat a malformed reply as an error to retry or review, never as "supported".

What it costs. On an M2 Max laptop's GPU, this 4B model took about 18 seconds for the eight-claim call and about 3 seconds per claim with one call per claim; with other jobs running on the machine we measured up to three times that. An API judge is billed by tokens instead: each call carries the context set plus the claims, so 1,000 answers with 1,500 context tokens each is at least 1.5 million input tokens per run in batched mode, and roughly six times that if you send each of six claims separately. Multiply by your model's current price. The NLI checker, for comparison, scored 300 to 550 claim–window pairs (15 to 60 tokens each) per second on the same laptop's GPU and 40 to 120 on its CPU, varying from run to run; a 460-token pair, a whole long passage, ran at only 10 to 18 per second on the GPU.

A cascade keeps most of the judge's accuracy for a fraction of its calls: run NLI on every claim, and send only the claims NLI does not support to the judge. On our answer that is 3 claims of 8, and it is where claims that need two passages and derived numbers end up. The catch is what the cascade never re-checks: claims NLI wrongly passed. In the hard-case table, that is the merged "10 users and SSO" claim, which the judge alone would have caught, and Move 6 finds a whole fault type that NLI passes. A cascade therefore depends on good decomposition, and you measure its miss rate per fault type on the injected faults of Move 6, not just the judge's.

Is the judge any good?

You find out the way you found out for NLI: compare it with people on labeled claims. Generation Quality covers calibrating judges in depth. For faithfulness specifically:

• Label 100–200 claims by hand from real answers, with two people labeling independently. Measure their agreement with Cohen's κ; where they disagree, write the rule down ("a derived number is supported if the arithmetic is right") and put that rule into the judge prompt.
• Report the judge's precision and recall for "not supported" against those labels, per hard-case type (recall for "not supported" is what Generation Quality calls the catch rate). A judge that is right on average can still fail every derived number, as ours did.
• Prefer a judge from a different model family than the generator. Zheng et al. (2023) list self-enhancement among the biases of LLM judges, alongside position and verbosity bias. Panickssery, Bowman & Feng (2024) found that LLMs can recognise their own generations and that this ability correlates linearly with how strongly they favour them. How large the effect is depends on the models and the task, so measure your pair against human labels rather than assuming a number.
• Pin what can change a verdict: the judge model version, the prompt, and whether claims go one per call or in a batch. In our run, batching alone changed the verdicts on both derived numbers (NOT_FOUND together, CONTRADICTED and SUPPORTED one at a time). Change any of them, and rerun the agreement check.

Dedicated detectors sit between NLI and a general LLM judge: small models trained for exactly this pair-wise check. HHEM-2.1-Open's model card reports balanced accuracy on RAGTruth and AggreFact subsets; MiniCheck's authors report GPT-4-level accuracy on their LLM-AggreFact benchmark "for 400x lower cost" with a 770M-parameter model. Ragas 0.4.3 even ships a FaithfulnesswithHHEM variant, only on its legacy ragas.metrics import path, which warns that it will be removed in v1.0. It uses HHEM instead of the LLM for the verdicts and rounds each score at 0.5. Treat all of these as candidates to test on your hard cases, as in the table above.

Move 6: Test the tester

Symptom: the dashboard says "faithfulness 0.96". Nobody can say which errors the checker would have missed.

A checker is code, so test it like code. Borrow mutation testing: take claims a person has confirmed as faithful, apply small edits that produce the error types you fear, and count how many the checker catches. The originals measure false alarms, the mutants measure detection. (As rates, these are what Generation Quality calls the false-fail rate and the catch rate.) Continuing from above:

FAITHFUL = [   # checked by a person: every one is supported by PASSAGES
    "The Team plan costs €12 per user per month.",
    "The Team plan includes up to 10 users.",
    "SSO is available only on the Enterprise plan.",
    "Team plans are billed monthly.",
    "Team plans can be cancelled at any time.",
    "The current month is not refunded when a Team plan is cancelled.",
    "Annual Enterprise contracts can be refunded in full within 14 days of signing.",
    "Annual Enterprise contracts are not refundable after 14 days.",
    "Each Team user gets 50 GB of storage.",
    "Storage is pooled across the workspace.",
    "Workspace owners can export all data as a ZIP file.",
    "Export links expire after 7 days.",
]

def swap_number(c):
    m = re.search(r"\d+", c)
    return c[:m.start()] + str(int(m.group()) * 2) + c[m.end():] if m else None

def flip_negation(c):
    if " not " in c:
        return c.replace(" not ", " ", 1)
    for old, new in [(" can ", " cannot "), (" is ", " is not "), (" are ", " are not ")]:
        if old in c:
            return c.replace(old, new, 1)
    return None

def swap_plan(c):
    if "Team" in c:
        return c.replace("a Team", "an Enterprise").replace("Team", "Enterprise")
    return c.replace("Enterprise", "Team") if "Enterprise" in c else None

def drop_condition(c):
    out = re.sub(r" (within|after) \d+ days( of signing)?", "", c)
    return out if out != c else None

MUTATORS = [swap_number, flip_negation, swap_plan, drop_condition]

def run_suite(threshold):
    flagged = lambda claim: check_claim(claim, threshold=threshold)[0] != "supported"
    row = [f"{sum(map(flagged, FAITHFUL))}/{len(FAITHFUL)}"]
    for mutate in MUTATORS:
        mutants = [m for m in map(mutate, FAITHFUL) if m]
        row.append(f"{sum(map(flagged, mutants))}/{len(mutants)}")
    return row

print("threshold | false alarms |", " | ".join(m.__name__ for m in MUTATORS))
for t in (0.5, 0.9, 0.98):
    print(t, "|", " | ".join(run_suite(t)))

threshold | false alarms | swap_number | flip_negation | swap_plan | drop_condition
0.5 | 0/12 | 6/6 | 8/8 | 5/9 | 0/3
0.9 | 0/12 | 6/6 | 8/8 | 8/9 | 0/3
0.98 | 0/12 | 6/6 | 8/8 | 9/9 | 1/3

Predict before reading on: at 0.5 the checker "caught" 6 of 6 number swaps. Is that 6 of 6 errors?

With the mutants labeled by a person, 24 of the 26 are real faults. Here they are next to the LLM judge from Move 5, which we ran on the same claims, one call per claim:

Now the table says something you can act on. NLI is solid on numbers and negation, weak on entity swaps: it passed "Annual Team contracts are not refundable after 14 days" at 0.979 and "Enterprise plans are billed monthly" at 0.814. It matched the sentence shape and let the plan name slide. Neither checker catches dropped conditions, as Move 4 predicted. The judge's better row cost a model call per claim, seconds each on the laptop, against a few hundredths of a second for the NLI pairs.

The cascade from Move 5 inherits NLI's blind spot. The four plan swaps NLI passed never reach the judge, so the cascade catches 18 of the 24 real faults, where the judge alone catches 22. Routing with a stricter bar helps: send every claim NLI scores below 0.98 to the judge, and on this suite the cascade catches 22 of 24 with 24 judge calls instead of 38. Whether 0.98 holds up is a threshold question, and the next section says what it takes to answer one.

Choosing a threshold, honestly

Raising the entailment threshold to 0.9 caught three more plan swaps, and 0.98 caught all nine plus one dropped condition ("Annual Enterprise contracts are not refundable.", 0.976), with no false alarm on the 12 true claims. It is tempting to ship 0.98. Look at the sample sizes first. At 0.98, with 9 of 9 plan swaps caught, the 95% Wilson interval for the true detection rate runs from 0.70 to 1.00; with 5 of 5 real number swaps, from 0.57 to 1.00. With 0 false alarms in 12 true claims, the false-alarm rate could still be as high as 0.24. The margin is thin, too: the lowest of the 12 true claims scored 0.992, and a true claim from Move 2's answer, "The storage is pooled across the workspace.", scores 0.95, so 0.98 would flag it. To pin a detection rate near 0.9 to within ±0.05, you need about 1.96² × 0.9 × 0.1 / 0.05² ≈ 139 labeled faults per fault type. Suites of a dozen claims find bugs in the checker; they cannot choose a threshold.

So build it at the size the decision needs:

• Real answers first. Sample a few hundred answers from logs and have people label every claim: supported, contradicted, not in the sources. Real failures teach you which fault types matter. RAGTruth (Niu et al., 2024) is a public example of this kind of labeling: nearly 18,000 RAG responses with word-level hallucination annotations.
• Mutants for coverage. Add mutants for the fault types that are rare in logs but costly (a wrong refund window, a swapped plan name), label them, and track detection per type.
• Sweep the threshold on that set, plot false alarms against detection, and pick the point your review team can afford. A false alarm costs a reviewer's minute; a missed fault costs whatever the wrong answer costs your customer.

Run it as a regression gate

Keep the labeled questions fixed and rerun the suite whenever the prompt, generator model, retriever, chunking or checker changes. Freeze the context sets when you want to test the generator and the checker alone. Also rerun it end to end, with live retrieval, on a schedule: a hosted model behind an alias can change without a deploy on your side, and the corpus behind your retriever changes daily.

Compare two versions on the same questions, answer by answer. Suppose that on 200 questions the old prompt produced 31 answers with an unsupported claim and the new one 20. What decides whether that is real is the 25 answers that changed state: 18 got fixed, 7 got broken. Under "no real difference", each changed answer is a coin flip, and an exact two-sided binomial test on 7 of 25 gives p ≈ 0.043. That is a paired test (McNemar's, in its exact form). Read the 7 newly broken answers before you celebrate: they are the regressions your users will meet. Evaluation & Quality Metrics covers CI gates and paired statistics in depth.

Final round: no labels on the problem

Real reports do not say which move they need. Find the symptom, then pick.

Challenge 1: the average went up

After a prompt change, mean faithfulness on the 300-question suite rose from 0.94 to 0.95. The same week, support tickets about wrong refund windows doubled. The checker is check_claim plus the claim decomposer from this lesson. Give two reasons the suite missed it, and what you would add.

Challenge 2: the model swap that improved everything

A teammate swaps the NLI model in check_claim for facebook/bart-large-mnli, "a bigger model, should be more accurate". Their version of the code reads entailment as probs[:, 1]. Overnight, the unsupported-answer rate on the suite falls from 18% to 0%. What do you check first?

Challenge 3: the flagged multiplication

The checker keeps flagging claims like "Eight users would pay €96 a month." A colleague proposes lowering the entailment threshold to 0.01 so these stop showing up. Respond.

Cheat sheet: symptom → move → cost

Before moving on, take ten answers from your own system and do the whole loop by hand once. Split each answer into claims, label every claim against its logged context set, run check_claim and one judge, and write down each disagreement and its cause: premise, number, negation, decomposition, or a dropped condition. That list is the first version of your hard-case suite, and it tells you which move your system needs first.

Next: Citation Coverage checks that each sentence cites a passage that supports it. Grounding & Hallucination Control moves these checks into the request path, with abstention and latency budgets. Generation Quality covers the other dimensions of a good answer, and how to calibrate judges.